{"id":57146,"date":"2025-06-17T12:30:00","date_gmt":"2025-06-17T19:30:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/dotnet\/?p=57146"},"modified":"2025-06-17T12:29:31","modified_gmt":"2025-06-17T19:29:31","slug":"multimodal-vision-intelligence-with-dotnet-maui","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/dotnet\/multimodal-vision-intelligence-with-dotnet-maui\/","title":{"rendered":"Multimodal Vision Intelligence with .NET MAUI"},"content":{"rendered":"

Expanding the many ways in which users can interact with our apps is one of the most exciting parts of working with modern AI models and device capabilities. With .NET MAUI, it’s easy to enhance your app from a text-based experience to one that supports voice<\/strong>, vision<\/strong>, and more.<\/p>\n

Previously I covered adding voice<\/strong> support<\/a> to the “to do” app from our Microsoft Build 2025 session<\/a>. Now I’ll review the vision<\/strong> side of multimodal intelligence. I want to let users capture or select an image and have AI extract actionable information from it to create a project and tasks in the Telepathic<\/a> sample app. This goes well beyond OCR scanning by using an AI agent to use context and prompting to produce meaningful input.<\/p>\n

\"Screenshots<\/p>\n

See what I see<\/h2>\n

From the floating action button menu on MainPage<\/code> the user selects the camera button immediately transitioning to the PhotoPage<\/code> where MediaPicker<\/code> takes over. MediaPicker<\/code> provides a single cross-platform API for working with photo gallery, media picking, and taking photos. It was recently modernized in .NET 10 Preview 4.<\/p>\n

The PhotoPageModel<\/code> handles both photo capture and file picking, starting from the PageAppearing<\/code> lifecycle event that I’ve easily tapped into using the EventToCommandBehavior<\/code> from the Community Toolkit for .NET MAUI<\/a>.<\/p>\n

<ContentPage.Behaviors>\n    <toolkit:EventToCommandBehavior\n        EventName=\"Appearing\"\n        Command=\"{Binding PageAppearingCommand}\"\/>\n<\/ContentPage.Behaviors><\/code><\/pre>\n
The PageAppearing<\/code> method is decorated with [RelayCommand]<\/code> which generates a command thanks to the Community Toolkit for MVVM<\/a> (yes, toolkits are a recurring theme of adoration that you’ll hear from me). I then check for the type of device being used and choose to pick or take a photo. .NET MAUI’s cross-platform APIs for DeviceInfo<\/code> and MediaPicker<\/code> save me a ton of time navigating through platform-specific idiosyncrasies.<\/p>\n
if (DeviceInfo.Idiom == DeviceIdiom.Desktop)\n{\n    result = await MediaPicker.PickPhotoAsync(new MediaPickerOptions\n    {\n        Title = \"Select a photo\"\n    });\n}\nelse\n{\n    if (!MediaPicker.IsCaptureSupported)\n    {\n        return;\n    }\n\n    result = await MediaPicker.CapturePhotoAsync(new MediaPickerOptions\n    {\n        Title = \"Take a photo\"\n    });\n}<\/code><\/pre>\n
Another advantage of using the built-in MediaPicker<\/code> is giving users the native experience for photo input they are already accustomed to. When you’re implementing this, be sure to perform the necessary platform-specific setup as documented<\/a>.<\/p>\n
Processing the image<\/h2>\n
Once an image is received, it’s desplayed on screen along with an optional Editor<\/code> field to capture any additional context and instructions the user might want to provide. I build the prompt with StringBuilder<\/code> (in other apps I like to use Scriban templates), grab an instance of the Microsoft.Extensions.AI<\/code>‘s IChatClient<\/code><\/a> from a service, get the image bytes, and supply everything to the chat client using a ChatMessage<\/code><\/a> that packs TextContent<\/code><\/a> and DataContent<\/code><\/a>.<\/p>\n
private async Task ExtractTasksFromImageAsync()\n{\n    \/\/ more code\n\n    var prompt = new System.Text.StringBuilder();\n    prompt.AppendLine(\"# Image Analysis Task\");\n    prompt.AppendLine(\"Analyze the image for task lists, to-do items, notes, or any content that could be organized into projects and tasks.\");\n    prompt.AppendLine();\n    prompt.AppendLine(\"## Instructions:\");\n    prompt.AppendLine(\"1. Identify any projects and tasks (to-do items) visible in the image\");\n    prompt.AppendLine(\"2. Format handwritten text, screenshots, or photos of physical notes into structured data\");\n    prompt.AppendLine(\"3. Group related tasks into projects when appropriate\");\n\n    if (!string.IsNullOrEmpty(AnalysisInstructions))\n    {\n        prompt.AppendLine($\"4. {AnalysisInstructions}\");\n    }\n    prompt.AppendLine();\n    prompt.AppendLine(\"If no projects\/tasks are found, return an empty projects array.\");\n\n    var client = _chatClientService.GetClient();\n    byte[] imageBytes = File.ReadAllBytes(ImagePath);\n\n    var msg = new Microsoft.Extensions.AI.ChatMessage(ChatRole.User,\n    [\n        new TextContent(prompt.ToString()),\n        new DataContent(imageBytes, mediaType: \"image\/png\")\n    ]);\n\n    var apiResponse = await client.GetResponseAsync<ProjectsJson>(msg);\n\n    if (apiResponse?.Result?.Projects != null)\n    {\n        Projects = apiResponse.Result.Projects.ToList();\n    }\n\n    \/\/ more code\n}<\/code><\/pre>\n
Human-AI Collaboration<\/h2>\n
Just like with the voice experience, the photo flow doesn’t blindly assume the agent got everything right. After processing, the user is shown a proposed set of projects and tasks for review and confirmation.<\/p>\n
This ensures users remain in control while benefiting from AI-augmented assistance. You can learn more about designing these kinds of flows using best practices in the HAX Toolkit<\/a>.<\/p>\n
Resources<\/h2>\n