December 4th, 2024
heartlikeintriguing49 reactions

How to develop AI Apps and Agents in Azure – A Visual Guide

As organizations explore new AI-powered experiences and automated workflows, there’s a growing need to move beyond experiments and proofs-of-concept to production-ready applications. This guide walks you through the essential steps and decisions for building robust AI applications in Azure, focusing on reliability, security, and enterprise-grade quality.

Why Choose Azure’s Managed Services?

It’s easy to experiment with generative AI models and create proof-of-concept demos, but building production-ready applications that can scale reliably is a different challenge entirely. When deploying AI-powered applications for real business use, you need infrastructure that provides consistent performance, robust security, and reliable operations. Did you know that OpenAI’s ChatGPT, GitHub Copilot, and Microsoft’s Copilots, are all deployed on Azure’s managed services? Managed services reduce uncertainty when deploying AI agents with specific goals and guardrails, making them accessible to organizations of all sizes.

In this article I am going to provide you with a visual map to help you decide which Azure AI service is best for your use case. Let’s get started:

Image MS AzureBlog DecisionTree 12 03 24 decision tree

Need a Quick Start?

The quickest way to get started is by going to Azure AI Foundry at https://ai.azure.com, which serves as your central hub for AI development, offering:

Image ai foundry

  • Playgrounds: Add your data and ground models to your content through managed RAG (Retrieval Augmented Generation) with just a few clicks. Deploy production-ready chat experiences quickly without complex setup
  • Prompt Flow: Enables you to do evaluation-driven development, tune prompts, integrate with tools, and provides built-in observability and troubleshooting
  • Agent Service: Enable secure, scalable single-purpose agents with managed RAG, managed function calling, and bring-your-own customization options. Seamlessly integrate with enterprise systems

Pro Tip: Start Your AI Journey with Azure AI Foundry

Azure AI Foundry provides everything you need to kickstart your AI application development journey. It offers an intuitive platform with built-in development tools, essential AI capabilities, and ready-to-use models (1800+!). As your needs grow, you can seamlessly integrate additional Azure managed services to enhance your AI solutions further. This makes it the ideal starting point for both beginners and experienced developers.

Need More Control? Let’s Build Your Stack

What Kind of Model Do You Need?

Selecting the right AI model is a critical decision that impacts your application’s capabilities, performance, and cost-effectiveness. Azure offers a comprehensive range of models to address different requirements, from multimodal reasoning to specialized tasks. Here’s a guide to help you choose the most suitable model for your specific needs:

Image MS AzureBlog DecisionTree 12 03 24 section 4

Requirement Options
Multimodal reasoning (text + images)
  • Azure OpenAI GPT-4o: Latest multimodal model for understanding both text and images
  • Llama models: Open-source multi-modal foundation model, designed for various natural language processing tasks, offering flexibility and customization for developers
Sensitive to latency and cost
  • Smaller LLMs like 4o-mini: optimized for lower latency and cost, making them suitable for applications where quick responses and resource efficiency are critical.
  • Choose from 1800+ models in Azure AI Model Catalog: Specialized models for specific tasks
Embeddings for search or classification
  • Azure OpenAI’s text-embedding-3 family: provides embeddings that capture semantic meanings of text (vector representations), enabling effective search, classification, and clustering tasks.
  • Cohere embeddings: Alternative text embedding models with strong multilingual support
Working with images
  • Azure OpenAI’s CLIP model + AI Search: enables vector-based image search by understanding and associating images with textual descriptions, enhancing image retrieval capabilities.
Advanced reasoning (System-2)
  • o1-preview: designed for complex problem-solving with built-in reflection mechanisms, enabling advanced reasoning and decision-making processes.
  • o1-mini: offers efficient reasoning capabilities, providing a balance between performance and resource utilization for applications requiring streamlined decision-making.

Pro Tip: Choosing the Right Model

Start with Azure AI Foundry’s model catalog to explore and experiment with different models. For most enterprise applications, consider using GPT-4o for complex multimodal tasks, while leveraging specialized models like 4o-mini for latency-sensitive operations. When building RAG applications, pair embedding models with your LLM – Azure OpenAI’s text-embedding-3 family works seamlessly with GPT models. Remember that you can always switch or combine models as your needs evolve, so focus on finding the right balance between capability and efficiency for your specific use case.

How Will Your Agent Remember Things?

When building AI applications, choosing the right storage solution is crucial for managing different types of data effectively. Here’s a guide to help you select the appropriate memory solution for your needs:

Image MS AzureBlog DecisionTree 12 03 24 section 2

Requirement Options
Search Capabilities
  • Azure AI Search: Enterprise-grade search service with built-in AI capabilities including:
    • Multi-modal semantic search
    • OCR and image analysis
    • Translation services
    • Rich integrations with Azure AI services
Frequently Accessed Knowledge
  • Cosmos DB: Globally distributed database with multi-model support
  • Azure Redis Cache: In-memory data store for high-performance scenarios requiring low latency, which is also integrated with APIM’s Gen-AI Gateway for semantic caching
  • Azure AI Search: Combines search capabilities with knowledge storage
Episodic Memory (interaction history) and Knowledge Graphs
Operational Data with Semantic Retrieval
  • NoSQL preference → Cosmos DB with DiskANN: Ideal for applications needing global distribution and vector search
  • SQL preference → PostgreSQL with pgvector: Best for applications requiring both traditional SQL capabilities and vector operations
  • MongoDB preference → MongoDB vCore: Fully managed MongoDB service with vector search capabilities

Pro Tip: Choosing the Right Memory Solution

Start by evaluating your search needs – Azure AI Search provides comprehensive multi-modal search capabilities with built-in AI services integration. For frequent data access, consider combining Azure Redis Cache for performance-critical operations with a persistent storage solution like Cosmos DB. When building knowledge graphs, leverage the GraphRAG solution accelerators available for both Cosmos DB and PostgreSQL to simplify implementation.

Where Will You Run Your Application?

Choosing the right runtime environment and frontend infrastructure is crucial for your AI application’s performance, scalability, and maintainability. Azure offers various options to match your specific deployment needs, from simple web apps to complex containerized solutions. Here’s a guide to help you select the most appropriate runtime configuration:

Image MS AzureBlog DecisionTree 12 03 24 section 1

Requirement Options
Web Applications
  • Azure App Service: Fully managed platform for building, deploying, and scaling web apps
    • Built-in CI/CD integration
    • Automatic scaling and load balancing
    • Enterprise-grade security and compliance
Serverless and Event-Driven
  • Azure Container Apps: Fully managed serverless container service for AI workloads
    • Serverless GPUs with scale-to-zero
    • Dynamic Sessions for secure code interpretation with Hyper-V isolation
    • Built-in data governance (data never leaves container boundaries)
    • Enterprise features like private endpoints and planned maintenance
  • Azure Functions: Serverless compute with Azure OpenAI integration
    • OpenAI triggers and bindings for chat assistants and RAG patterns
    • Pay-per-execution pricing with Flex consumption
    • Support for vector stores (AI Search, Cosmos DB MongoDB, ADX)
    • Managed identity support for secure service access
Container Orchestration
  • Azure Kubernetes Service (AKS): Managed Kubernetes service for complex container orchestration
    • Full container orchestration control
    • Multi-container deployments
    • Enterprise-grade security features
    • Advanced networking and scaling options
Communication Features
  • Azure Communication Services: Comprehensive platform for adding communication capabilities
    • Voice and video calling
    • SMS and chat functionality
    • Easy integration with existing applications

Pro Tip: Choosing the Right Runtime Environment

Consider starting with Azure App Service for straightforward web applications. For event-driven workloads, both Azure Container Apps and Functions offer serverless capabilities with automatic scaling – choose Container Apps when you need container flexibility or GPU support, and Functions for lightweight compute with AI bindings. If you need full container orchestration control, AKS provides enterprise-grade Kubernetes management.

How Will Your AI Agent Take Action?

When building AI applications that need to interact with the real world, you’ll need tools that enable your agents to take actions, process information, and integrate with enterprise systems. Azure provides a comprehensive set of tools that let your AI agents create real-world impact while maintaining security and control. With the AI Agent Service in Azure AI Foundry, integrating these tools has become even more streamlined. Here’s a guide to help you choose the right tools for your AI application:

Image MS AzureBlog DecisionTree 12 03 24 section 3

Requirement Options
Plugins and Workflows (Function Calling)
  • Logic Apps: The primary tool for enabling AI agents to take actions
    • Native integration with AI Agent Service for seamless function calling
    • Support for On-Behalf-Of (OBO) flows
    • 200+ pre-built connectors for enterprise systems
    • Visual workflow designer for complex orchestrations
  • Azure Functions: Serverless compute for custom tool implementations
    • Custom function calling implementations
    • Integration with AI services via bindings
AI Services (via APIs)
  • Content Understanding: Process and structure any content type
    • Unified processing of documents, images, videos, and audio
    • Field extraction with configurable schema
    • Built-in confidence scoring and source grounding
    • Ideal for automation, RAG, and analytics workflows
  • Document Intelligence: Extract and analyze information from documents
  • Vision: Process and analyze images and videos
  • Language: Natural language processing and understanding
  • Speech & Avatar: Voice interaction and digital human experiences
Code Interpreter
  • Azure Container Apps Dynamic Sessions: Secure environment for running AI-generated code
    • Isolated execution environment
    • Support for multiple programming languages
    • Integration with AI Agent Service

Pro Tip: Leveraging AI Agent Service for Tool Integration

The AI Agent Service in Azure AI Foundry significantly simplifies tool integration for your AI applications. It provides managed function calling capabilities and seamless integration with Logic Apps, making it easier to implement complex workflows and system interactions. When building AI agents that need to take actions, start with Logic Apps for orchestration and leverage the AI Agent Service’s built-in support for OBO flows and enterprise system integration.

How Will You Ensure Quality and Safety?

Enterprise AI applications require comprehensive quality controls across safety, evaluation, security, and reliability dimensions. Azure provides integrated services to help you build AI applications that meet the highest quality standards. Here’s a guide to help you implement the right quality attributes:

Image MS AzureBlog DecisionTree 12 03 24 section 5

Requirement Options
Quality Attributes & Reliability
AI Safety
  • Azure AI Content Safety: Comprehensive content safety service
    • Detect harmful content in text and images
    • Built-in Prompt Shields for LLM attack protection
    • Support for custom safety categories
    • Integration with Microsoft Defender for Cloud for threat protection
Evaluation & LLMOps
  • Azure AI Foundry Evaluations: Integrated evaluation platform
    • Built-in metrics for quality, safety, and performance
    • AI-assisted and NLP-based evaluation methods
    • Support for custom evaluation flows
    • Comprehensive evaluation metrics library
  • Prompt Flow (integrated with AI Foundry): Evaluation-driven development
    • Flow-based evaluation orchestration
    • Built-in observability and troubleshooting
Security

Pro Tip: Building Enterprise-Grade AI Applications

Start with Azure AI Foundry’s evaluation capabilities to assess your application’s quality and safety. Use Content Safety service to protect against harmful content and integrate with API Management’s GenAI Gateway for production-grade reliability. Implement Microsoft Defender for Cloud to ensure comprehensive security coverage for your AI workloads. This layered approach helps create AI applications that meet enterprise requirements for quality, safety, and reliability.

Need Additional Development Support?

When building AI applications, you can accelerate your development by leveraging battle-tested frameworks that provide abstracted design patterns and pre-built integrations with the above managed services:

Framework Capabilities
Semantic Kernel Microsoft’s open-source SDK that integrates LLMs with conventional programming languages (C#, Python, Java). Ideal for enterprise applications requiring tight integration with existing code.
AutoGen Framework for building multi-agent applications, enabling sophisticated agent-to-agent interactions and complex task completion.
Langchain Popular framework for building LLM applications with ready-to-use components for common patterns like RAG, agents, and chains.
LlamaIndex Data framework specialized in connecting custom data with LLMs, offering advanced RAG capabilities and data connectors.

Ready to Start Building?

Choose your path based on your needs:

  1. Want the quickest start? Head to Azure AI Foundry for a guided experience with built-in best practices and patterns.
  2. Need more control? Start with AI App templates for common patterns, or build your stack from scratch by selecting your models, memory solutions, and deployment options from the choices above.
  3. Looking for development frameworks? Use battle-tested frameworks like Semantic Kernel, AutoGen, or LangChain that provide abstracted design patterns and pre-built integrations for rapid development.

Remember: Major players like ChatGPT, GitHub Copilot, and Microsoft’s Copilots all run on these same services – you’re building on proven infrastructure. To accelerate your development:

  • Landing Zone Reference Architectures: Ready-to-deploy infrastructure templates that follow best practices for security, scaling, and governance
  • AI App Templates: Quickly customize existing AI applications for your specific business needs using production-tested patterns

Prerequisites to Get Started

Before you begin, ensure you have:

  • An Azure subscription
  • User role with Azure AI Developer permissions
  • Azure AI Inference Deployment Operator permissions (if models aren’t already deployed)

This guide will continue to evolve as Azure’s AI capabilities expand. Start building today and transform your AI experiments into production-ready applications!

Author

Govind Kamtamneni
Principal Research Engineer, Applied AI

Principal Research Engineer, Applied AI

Priyanka Vergadia
Sr. Director Developer Engagement

3 comments

Leave a comment

Your email address will not be published. Required fields are marked *

Sort by :
  • Your comment is awaiting moderation.
    Michaelsoymn

    Нужен аккмулятор? магазин аккумуляторов по выгодной цене с подбором под ваш автомобиль. В наличии аккумуляторы популярных брендов, услуги установки, диагностика аккумулятора, прием старой АКБ и оперативная доставка по Санкт-Петербургу.

  • Your comment is awaiting moderation.
    Alberttaw

    Мировые новости https://trawa-moscow.ru в режиме реального времени: политика, экономика, общество, технологии, наука, культура, спорт и международные события. Следите за главными событиями дня, аналитикой, интервью и эксклюзивными материалами со всего мира.

  • Your comment is awaiting moderation.
    KirbyHAL

    Ищешь аккумулятор? магазин аккумуляторов спб продажа автомобильных аккумуляторов в Санкт-Петербурге для любых марок автомобилей. Подберите АКБ по характеристикам, емкости и пусковому току, оформите заказ с доставкой или самовывозом, получите гарантию и помощь специалистов.

  • Your comment is awaiting moderation.
    AntonioGefar

    Ищешь аккумулятор? аккумуляторы спб продажа автомобильных аккумуляторов в Санкт-Петербурге для любых марок автомобилей. Подберите АКБ по характеристикам, емкости и пусковому току, оформите заказ с доставкой или самовывозом, получите гарантию и помощь специалистов.

  • Your comment is awaiting moderation.
    Michaelsoymn

    Нужен аккмулятор? аккумуляторы по выгодной цене с подбором под ваш автомобиль. В наличии аккумуляторы популярных брендов, услуги установки, диагностика аккумулятора, прием старой АКБ и оперативная доставка по Санкт-Петербургу.

  • Your comment is awaiting moderation.
    VictorBurce

    Top title checker title brand check follows the VIN even after a title has been washed.

  • Your comment is awaiting moderation.
    AntonioGefar

    На сайті https://rest.od.ua ви знайдете багато корисної інформації для кожного одесита: театральна афіша Одеси, карта та схема проїзду до всіх театрів та концертних майданчиків міста

  • Your comment is awaiting moderation.
    DavidBop

    Fraud-aware shoppers rebuilt title check run a title check to catch washed salvage titles.

  • Your comment is awaiting moderation.
    PatrickDrowl

    лента нихромовая х20н80 лента 29нк

  • Your comment is awaiting moderation.
    DavidKar 24 hours ago

    Детский центр https://run.org.ua развития и здоровья с комплексными программами для детей разных возрастов. Развивающие занятия, логопед, психолог, подготовка к школе, творческие кружки, физическое развитие, диагностика и индивидуальный подход к каждому ребенку.

  • Your comment is awaiting moderation.
    JarrodPag 1 day ago

    Женский онлайн-журнал https://lolitaquieretemucho.com с интересными материалами о красоте, здоровье, стиле, модных тенденциях, косметике, фитнесе, воспитании детей, домашнем уюте, карьере и личностном развитии.

  • Your comment is awaiting moderation.
    JohnnieJolum 1 day ago

    Ежедневно публикуем https://presslook.com.ua полезные статьи для женщин о здоровье, красоте, моде, психологии, любви, семье, кулинарии, саморазвитии, путешествиях, финансах и современных тенденциях образа жизни.

  • Your comment is awaiting moderation.
    DarronBor 1 day ago

    Все самое интересное https://ramledlightings.com для женщин в одном месте. Советы по уходу за собой, обзоры косметики, секреты красоты, идеи стильных образов, рекомендации по здоровью, отношениям и воспитанию детей.

  • Your comment is awaiting moderation.
    JamesPoone 1 day ago

    Откройте для себя https://icz.com.ua мир красоты, здоровья и вдохновения. Читайте полезные статьи о моде, уходе за собой, психологии, отношениях, семье, правильном питании, путешествиях и гармоничной жизни современной женщины.

  • Your comment is awaiting moderation.
    Davidtyday 1 day ago

    Актуальные статьи https://godwood.com.ua для женщин о красоте, здоровье, отношениях, беременности, воспитании детей, моде, косметике, фитнесе, правильном питании, путешествиях и современных лайфхаках.

  • Your comment is awaiting moderation.
    Thomaszenny 1 day ago

    Современный портал https://horoscope-web.com для женщин с интересными статьями, экспертными советами и обзорами. Узнавайте больше о красоте, здоровье, моде, отношениях, материнстве, уюте, саморазвитии и вдохновляющих историях.

  • Your comment is awaiting moderation.
    DonaldKEd 1 day ago

    Женский информационный https://gratransymas.com портал с полезными материалами о моде, уходе за собой, психологии, семейной жизни, здоровье, кулинарии, хобби, карьере, отдыхе и личностном развитии.

  • Your comment is awaiting moderation.
    LouisTheld 1 day ago

    Познавательный портал https://detiwki.com.ua для детей с интересными статьями, развивающими заданиями, научными фактами, играми, головоломками, творческими идеями, опытами, рассказами о природе, космосе, животных, истории и окружающем мире.

  • Your comment is awaiting moderation.
    Edwinemott

    Откройте мир полезных https://beautyadvice.kyiv.ua советов для женщин: уход за лицом и телом, стиль, мода, здоровье, психология, рецепты, воспитание детей, финансы, саморазвитие и вдохновение для счастливой жизни.

  • Your comment is awaiting moderation.
    JamesTof

    Строительный портал https://vasha-opora.com.ua для тех, кто строит, ремонтирует и благоустраивает. Новости рынка, обзоры строительных материалов, пошаговые инструкции, рекомендации специалистов, идеи для дома, квартиры и загородного участка.

  • Your comment is awaiting moderation.
    Arnoldosen

    Узнайте больше https://poradnik.com.ua о строительстве и ремонте: полезные статьи, экспертные рекомендации, обзоры строительных материалов, современные технологии, инженерные решения, советы по отделке и эксплуатации частных домов.

  • Your comment is awaiting moderation.
    LarryZet

    Самые важные новости https://tvk-avto.com.ua автомобильной отрасли, обзоры автомобилей, рейтинги, тест-драйвы, экспертные статьи, советы по обслуживанию, выбору шин, аккумуляторов, масел, аксессуаров и уходу за автомобилем.

  • Your comment is awaiting moderation.
    Thomasbax

    Smith Jonsi’s articles https://vocal.media/authors/smit-jonsy on Vocal cover modern AI applications and digital technologies. Independent reviews and honest comparisons of features, capabilities, pricing, voice quality, privacy, and user experience.

  • Your comment is awaiting moderation.
    KennethQuift

    Расширенная статья здесь: https://elicebeauty.com/jean-patou/

  • Your comment is awaiting moderation.
    DanielAbate

    Мир автомобилей https://road.kyiv.ua без лишней информации: свежие новости, обзоры популярных моделей, тест-драйвы, советы по эксплуатации, ремонту, обслуживанию, выбору запчастей и актуальные материалы для каждого автовладельца.

  • Your comment is awaiting moderation.
    RobertBed

    Автомобильный портал https://proauto.kyiv.ua с актуальными статьями, аналитикой и обзорами. Узнавайте о новых моделях, изменениях на авторынке, современных технологиях, сервисном обслуживании, ремонте, эксплуатации и выборе автомобиля.

  • Your comment is awaiting moderation.
    BrianRix

    Автомобильный портал https://troeshka.com.ua с ежедневными публикациями о новых моделях, электромобилях, гибридах, внедорожниках, кроссоверах, технологиях, автоспорте, ремонте, тюнинге и полезными рекомендациями для водителей.

  • Your comment is awaiting moderation.
    Thomaselets

    Узнавайте первыми https://setbook.com.ua о новинках автомобильного рынка. Новости производителей, тесты автомобилей, сравнение комплектаций, советы по покупке, ремонту, страхованию, обслуживанию и безопасной эксплуатации транспорта.

  • Your comment is awaiting moderation.
    DavidFak

    Все самое интересное https://black-star.com.ua из мира автомобилей: свежие новости, обзоры новинок, тест-драйвы, рекомендации по выбору машины, обслуживанию, экономии топлива, уходу за кузовом и подготовке автомобиля к разным сезонам.

  • Your comment is awaiting moderation.
    ChesterHog

    Информационный автомобильный https://avtonews.kyiv.ua портал с ежедневными публикациями о новых автомобилях, технологиях, электрокарах, автоспорте, правилах дорожного движения, ремонте, диагностике, тюнинге и полезных советах для автовладельцев.

  • Your comment is awaiting moderation.
    JesusmoK

    Следите за новостями https://prp.org.ua Украины онлайн: оперативная информация, аналитика, интервью, обзоры, комментарии экспертов и репортажи о политике, экономике, международных событиях, технологиях и общественной жизни.

  • Your comment is awaiting moderation.
    Billyswimb

    Следите за главными https://avtomobilist.kyiv.ua событиями автомобильного рынка. Новости производителей, обзоры новых моделей, экспертные статьи, тест-драйвы, рейтинги автомобилей, советы по ремонту, обслуживанию и безопасной эксплуатации.

  • Your comment is awaiting moderation.
    Billypound

    Все важные события https://novosti24.com.ua Украины и мира в удобном формате. Новости бизнеса, финансов, политики, общества, транспорта, науки, медицины, культуры, спорта и других сфер с ежедневным обновлением материалов.

  • Your comment is awaiting moderation.
    DavidZet

    Портал о строительстве https://stroy-portal.kyiv.ua с ежедневными публикациями о современных технологиях, ремонте, проектировании, выборе материалов, строительной технике, инструментах, ландшафтном дизайне и обустройстве участка.

  • Your comment is awaiting moderation.
    MichaelMut

    Актуальная информация https://vitamax.dp.ua о строительстве, ремонте и благоустройстве. Новости отрасли, технологии, строительные материалы, проекты домов, советы по эксплуатации зданий, инженерным решениям и организации строительных работ.

  • Your comment is awaiting moderation.
    MartinRib

    Все для строительства https://valkbolos.com дома и ремонта квартиры: статьи, инструкции, обзоры материалов, советы по выбору инструментов, монтажу инженерных коммуникаций, утеплению, кровельным и отделочным работам.

  • Your comment is awaiting moderation.
    Robertvug

    Строительный портал https://smallbusiness.dp.ua с практическими рекомендациями по строительству, ремонту и отделке. Обзоры инструментов, материалов, оборудования, инженерных систем, технологии монтажа, советы специалистов и строительные лайфхаки.

  • Your comment is awaiting moderation.
    DavidFus

    Читайте полезные https://mr.org.ua материалы о строительстве домов, ремонте квартир, выборе строительных материалов, инженерных системах, дизайне интерьера, благоустройстве участка, современных технологиях и профессиональных строительных решениях.

  • Your comment is awaiting moderation.
    CalvinFup

    Женский портал https://nicegirl.kyiv.ua для тех, кто ценит красоту, здоровье и комфорт. Полезные советы по уходу за собой, обзоры косметики, идеи образов, секреты гармоничных отношений, домашнего уюта и активного образа жизни.

  • Your comment is awaiting moderation.
    Charlietew 2 days ago

    Онлайн-журнал https://mirlady.kyiv.ua для женщин с актуальными статьями о моде, красоте, здоровье, семье, детях, фитнесе, правильном питании, косметике, карьере, вдохновении и современных тенденциях.