Introducing new Ollama Connector for Local ModelsĀ
As a long-awaited feature we are happy to bring a dedicated Ollama Connector. In the core implementation of this connector, we are also using the Ollama Sharp library which allows you to manipulate some native Ollama APIs as well as using it with our connector for breaking glass scenarios (provide the Ollama Sharp client directly to the AddOllama(Service) extensions).Ā
This connector is currently in alpha pre-release and supports Chat Completion, Text Embeddings and Text Generation capable models.Ā
The biggest benefit of having a dedicated connector for Ollama is that it allows us to support Semantic Kernel features that targeted for Ollama deployed models.Ā
What is Ollama?Ā
Ollama is an open-source MIT license platform that facilitates the local operation of AI models directly on personal or corporate hardware. It supports a variety of models from different sources, such as Phi-3, Llama-3, Mistral, and many others, allowing users to run these models on their local machines without the need for continuous internet connectivity. This local deployment secures sensitive data and provides complete control over the AI models and their operation.
The platform is quite simple to use and setup and has been largely adopted by the AI community for edge scenarios and local AI inference tasks.
Get Started
- Deploy a model like Phi-3. For more details, see the Ollama AI Models library.
- Install the Connectors.Ollama pre-release package in your existing project from NuGet.
Add the following code to your application to start making requests to your local AI model. Make sure to provide the correct model Id (phi3, llama3.1, etc.) and the endpoint if different from the default āhttp://localhost:11434ā.
var chatService = new OllamaChatCompletionService(
modelId: "phi3");
var chatHistory = new ChatHistory("You are a helpful assistant that knows about AI.");
chatHistory.AddUserMessage("Hi, I'm looking for book suggestions");
var reply = await chatService.GetChatMessageContentAsync(chatHistory);For more scenarios, see the samples below:
- Chat Completion.
- Streaming Chat Completion.
- Text Generation
- Streaming Text Generation
- Text Embeddings
Conclusion
We’re excited to see what you build! Try out the new Ollama Connector and give us your feedback. Please reach out if you have any questions or feedback through ourĀ Semantic Kernel GitHub Discussion Channel. We look forward to hearing from you!Ā We would also love your support, if youāve enjoyed using Semantic Kernel, give us a star onĀ GitHub.
Does it support function calling?
I understand it is being implemented during these days: https://github.com/microsoft/semantic-kernel/pull/9275
This is exciting news guys! I canāt wait to see more features added to the connector library. Good job šš»
Soon, the PR is going through the review process – https://github.com/microsoft/semantic-kernel/pull/9275
I realized this was the answer to my question above š