{"id":232297,"date":"2025-06-17T11:46:40","date_gmt":"2025-06-17T18:46:40","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/java\/?p=232297"},"modified":"2025-06-17T11:46:40","modified_gmt":"2025-06-17T18:46:40","slug":"connect-spring-ai-to-local-ai-models-with-foundry-local","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/java\/connect-spring-ai-to-local-ai-models-with-foundry-local\/","title":{"rendered":"Connect Spring AI to Local AI Models with Foundry Local"},"content":{"rendered":"

What is Azure AI Foundry and Foundry Local?<\/h2>\n

Azure AI Foundry is Microsoft’s comprehensive platform for enterprise AI development and deployment,\nenabling organizations to build, customize, and operate AI solutions at scale. It provides tools,\nservices, and infrastructure to develop, fine-tune and deploy AI models in production environments\nwith enterprise-grade security and compliance.<\/p>\n

Foundry Local is the desktop companion to Azure AI Foundry that brings powerful AI model inference\nto your local machine. It’s an open-source tool that provides an OpenAI-compatible API, allowing\ndevelopers to run, test, and integrate large language models directly on their own hardware without\nsending data to the cloud. This makes it ideal for development, testing, and scenarios requiring\ndata privacy or offline operation.<\/p>\n

Why Use Foundry Local with Spring AI?<\/h2>\n

Running AI models locally has become increasingly valuable for Java developers who want to reduce latency,\navoid network dependency, and eliminate API cost overhead. Foundry Local makes this seamless by offering\nan OpenAI-compatible API, allowing Spring AI to interact with local models as if they were hosted in the cloud.<\/p>\n

In this guide, we\u2019ll walk you through setting up Foundry Local, integrating it with a Spring Boot project using\nSpring AI, and building a simple REST API to interact with the AI model for chat and summarization tasks. This\nenables powerful local inference with models like Phi-3.5, Qwen, or DeepSeek, using standard Spring idioms.<\/p>\n

Quick Setup<\/h2>\n

Step 1: Install Foundry Local<\/h3>\n

Foundry Local can be installed via Homebrew for macOS users:<\/p>\n

$ brew tap microsoft\/foundrylocal\r\n$ brew install foundrylocal\r\n$ foundry --help<\/pre>\n
This sets up the CLI tools necessary to run and manage language models locally.<\/p>\n
Step 2: Download and Start a Model<\/h3>\n
Before interacting with any model, you need to list available models, download the one you want,\nand load it into memory:<\/p>\n
# Discover available models\r\n$ foundry model list\r\n\r\n# Download the Phi-3.5 mini model\r\n$ foundry model download phi-3.5-mini\r\n\r\n# Load it into memory for serving\r\n$ foundry model load phi-3.5-mini\r\n\r\n# Check model and service status\r\n$ foundry service status<\/pre>\n
Once loaded, the model will be available through a local HTTP endpoint using an OpenAI-compatible interface.<\/p>\n
Create a Spring Boot Project<\/h2>\n
You can use Spring Initializr to quickly generate a project with the required dependencies.<\/p>\n
Run the following cURL command to download a Spring Starter project with the required dependencies:<\/p>\n
curl -G https:\/\/start.spring.io\/starter.zip \\\r\n    -d javaVersion=21 \\\r\n    -d baseDir=demo \\\r\n    -d language=java \\\r\n    -d bootVersion=3.5.0 \\\r\n    -d type=maven-project \\\r\n    -d dependencies=web,spring-ai-openai \\\r\n    -o demo.zip<\/pre>\n
Now unzip the project and open Visual Studio Code.<\/p>\n
unzip demo.zip\r\ncode demo\/<\/pre>\n
If you\u2019re adding Spring AI to an existing project, use the following Maven dependencies:<\/p>\n
<dependencies>\r\n    <dependency>\r\n        <groupId>org.springframework.boot<\/groupId>\r\n        <artifactId>spring-boot-starter-web<\/artifactId>\r\n    <\/dependency>\r\n    <dependency>\r\n        <groupId>org.springframework.ai<\/groupId>\r\n        <artifactId>spring-ai-starter-model-openai<\/artifactId>\r\n        <version>1.0.0<\/version>\r\n    <\/dependency>\r\n<\/dependencies>\r\n<\/pre>\n
These dependencies enable the REST interface and Spring AI\u2019s OpenAI integration capabilities.<\/p>\n
Connect Spring AI to Foundry Local<\/h2>\n
Step 1: Configure Properties<\/h3>\n
Update application.properties to point Spring AI to your locally running Foundry instance:<\/p>\n
# Azure AI Foundry Local Configuration\r\nspring.ai.openai.api-key=not-used\r\nspring.ai.openai.base-url=http:\/\/localhost:8081\r\nspring.ai.openai.chat.options.model=Phi-3.5-mini-instruct-generic-gpu\r\n\r\n# Server Configuration\r\nserver.port=8080<\/pre>\n
Even though an API key is required by the config, it won\u2019t be used for local models.<\/p>\n
Step 2: Configuration Class<\/h3>\n
This configuration class sets up the OpenAI-compatible API client and initializes the chat model with your local settings:<\/p>\n
@Configuration\r\npublic class FoundryLocalConfig {\r\n\r\n    @Value(\"${spring.ai.openai.base-url}\") private String baseUrl;\r\n    @Value(\"${spring.ai.openai.api-key}\")  private String apiKey;\r\n    @Value(\"${spring.ai.openai.chat.options.model}\") private String modelName;\r\n\r\n\t@Bean\r\n\tpublic OpenAiApi openAiApi() {\r\n\t\treturn OpenAiApi.builder().baseUrl(baseUrl).apiKey(apiKey).build();\r\n\t}\r\n\r\n\t@Bean\r\n\tpublic OpenAiChatModel chatModel(OpenAiApi openAiApi) {\r\n\t\tOpenAiChatOptions options = OpenAiChatOptions.builder()\r\n\t\t\t\t.model(modelName)\r\n\t\t\t\t.temperature(0.7)\r\n\t\t\t\t.build();\r\n\t\treturn OpenAiChatModel.builder().openAiApi(openAiApi).defaultOptions(options).build();\r\n\t}\r\n}<\/pre>\n
This is the bridge that connects Spring AI\u2019s abstraction to the local model served by Foundry.<\/p>\n
Step 3: Implement a Service Layer<\/h3>\n
Create a service class to interact with the chat model. This encapsulates the logic for calling the model:<\/p>\n
@Service\r\npublic class AIService {\r\n\r\n    private final OpenAiChatModel chatModel;\r\n\r\n    public AIService(OpenAiChatModel _chatModel) { chatModel = _chatModel;\r\n    }\r\n\r\n    public String chat(String message) {\r\n        try {\r\n            return chatModel.call(message);\r\n        } catch (Exception e) {\r\n            throw new RuntimeException(\"Error calling AI model: \" + e.getMessage(), e);\r\n        }\r\n    }\r\n\r\n    public String summarizeText(String text) {\r\n        String prompt = \"Please provide a concise summary of the following text:\\n\\n\" + text;\r\n        return chatModel.call(prompt);\r\n    }\r\n}<\/pre>\n
The `chat()` and `summarizeText()` methods abstract out the logic so your controller stays clean.<\/p>\n
Step 4: Create REST Endpoints<\/h3>\n
Now expose those methods via HTTP endpoints:<\/p>\n
@RestController\r\n@RequestMapping(\"\/api\/ai\")\r\npublic class AIController {\r\n\r\n    private final AIService aiService;\r\n\r\n    public AIController(AIService _aiService) { aiService = _aiService; }\r\n\r\n    @PostMapping(\"\/chat\")\r\n    public ResponseEntity<Map<String, String>> chat(@RequestBody Map<String, String> request) {\r\n        String message = request.get(\"message\");\r\n        if (message == null || message.trim().isEmpty()) {\r\n            return ResponseEntity.badRequest().body(Map.of(\"error\", \"Message is required\"));\r\n        }\r\n        \r\n        String response = aiService.chat(message);\r\n        return ResponseEntity.ok(Map.of(\"response\", response));\r\n    }\r\n\r\n    @PostMapping(\"\/summarize\")\r\n    public ResponseEntity<Map<String, String>> summarize(@RequestBody Map<String, String> request) {\r\n        String text = request.get(\"text\");\r\n        if (text == null || text.trim().isEmpty()) {\r\n            return ResponseEntity.badRequest().body(Map.of(\"error\", \"Text is required\"));\r\n        }\r\n        \r\n        String summary = aiService.summarizeText(text);\r\n        return ResponseEntity.ok(Map.of(\"summary\", summary));\r\n    }\r\n\r\n    @GetMapping(\"\/health\")\r\n    public ResponseEntity<Map<String, String>> health() {\r\n        try {\r\n            aiService.chat(\"Hello\");\r\n            return ResponseEntity.ok(Map.of(\"status\", \"healthy\"));\r\n        } catch (Exception e) {\r\n            return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)\r\n                .body(Map.of(\"status\", \"unhealthy\", \"error\", e.getMessage()));\r\n        }\r\n    }\r\n}\r\n<\/pre>\n
With these endpoints, your app becomes a simple gateway for local AI-powered inference.<\/p>\n
Running the Application<\/h2>\n
Step 1: Start Foundry Local<\/h3>\n
Configure Foundry to serve the model on port 8081:<\/p>\n
foundry service set --port 8081\r\nfoundry service start\r\nfoundry model load phi-3.5-mini<\/pre>\n
Make sure this service is running and healthy before starting your Spring Boot app.<\/p>\n
Step 2: Start Spring Boot<\/h3>\n
Use Maven to build and run the app:<\/p>\n
mvn clean compile\r\nmvn spring-boot:run<\/pre>\n
Step 3: Test Your Endpoints<\/h3>\n
Perform a health check<\/p>\n
curl http:\/\/localhost:8080\/api\/ai\/health<\/pre>\n
Send a chat message:<\/p>\n
curl -X POST http:\/\/localhost:8080\/api\/ai\/chat \\\r\n  -H \"Content-Type: application\/json\" \\\r\n  -d '{\"message\": \"What is Spring AI?\"}'<\/pre>\n
Summarize a block of text:<\/p>\n
curl -X POST http:\/\/localhost:8080\/api\/ai\/summarize \\\r\n  -H \"Content-Type: application\/json\" \\\r\n  -d '{\"text\": \"Your long text here...\"}'<\/pre>\n
Conclusion<\/h2>\n
In this post, we demonstrated how to build a fully local AI-powered application using Spring AI and Foundry Local.<\/p>\n
Thanks to Foundry\u2019s compatibility with OpenAI\u2019s APIs, Spring AI required no special integration work\u2014we simply changed the base URL to localhost. This local-first approach has multiple advantages:<\/p>\n