November 20th, 2026
0 reactions

Serve open models in production with Foundry Managed Compute

Foundry Managed Compute is generally available with 150+ open models, Global and US or EU Data Zone deployments, and dedicated NVIDIA and AMD GPUs.

We are excited to announce the general availability of Managed Compute in Microsoft Foundry. When we introduced Managed Compute in preview at Microsoft Build in June, we set out to make serving open models as approachable as calling a managed model API. You choose the model that fits your application; we provision the GPUs, maintain the containers, and operate the serving runtime.

Since Build, we’ve scaled serving capacity, improved prompt reuse, and expanded GPU and deployment options, including US and EU Data Zones. Customer feedback helped us refine the experience and give teams a clearer view of how their deployments are running. We’ve also strengthened the software supply chain and added more open models to the catalog. The result is a managed platform for serving open models on dedicated GPUs in the Foundry experience you already use.

Getting started with open models on Foundry Managed Compute

Sign into the Foundry portal, navigate to the model catalog and filter by “Managed Compute” under “Deployment options” to discover the 150+ open models available. Select a model, then choose a deployment template that fits your workload. The template defines the context length, serving runtime, and GPU configuration. Depending on the model, you can choose NVIDIA GPUs (A100, H100, or H200) or AMD MI300X GPUs. Start with one instance, review the hourly cost in the deployment wizard, and test a few prompts in the Playground once the deployment is ready.

Watch the Ignite breakout session From open weights to workload: Operationalizing at enterprise scale (BRK246) from 7:54 to learn how to get started with Foundry Managed Compute.

Serve open models in production with Foundry Managed Compute

Here’s how to deploy DeepSeek V4.1 Flash (1017) from Python. The example creates one billable instance on four MI300X GPUs. You’ll need a Foundry project, permission to deploy and call models, and enough Managed Compute GPU quota.

from azure.identity import DefaultAzureCredential
from azure.mgmt.cognitiveservices import CognitiveServicesManagementClient

credential = DefaultAzureCredential()
account_name = "<your-foundry-account>"
deployment_name = "deepseek-1017-demo"
registry = "azureml://registries/azure-huggingface"

with CognitiveServicesManagementClient(credential, "<your-subscription-id>") as management:
    deployment = management.managed_compute_deployments.begin_create_or_update(
        resource_group_name="<your-resource-group>",
        account_name=account_name,
        deployment_name=deployment_name,
        resource={
            "sku": {"name": "GlobalManagedCompute", "capacity": 1},
            "properties": {
                "model": (
                    f"{registry}/models/deepseek-ai--deepseek-v4.1-flash-1017/versions/5"
                ),
                "deploymentTemplate": (
                    f"{registry}/deploymenttemplates/"
                    "deepseek-ai--deepseek-v4-1-flash-1017--1m-amd-4xmi300x/labels/latest"
                ),
                "acceleratorType": "MI300_192GB",
            },
        },
    ).result()

state = deployment.properties.provisioning_state
if state != "Succeeded":
    raise RuntimeError(f"Deployment ended in state: {state}")

Once the deployment is ready, call it from the same Python session. In the Chat Completions request, set model to the deployment name:

from azure.identity import get_bearer_token_provider
from openai import OpenAI

client = OpenAI(
    base_url=f"https://{account_name}.services.ai.azure.com/openai/v1/",
    api_key=get_bearer_token_provider(credential, "https://ai.azure.com/.default"),
)
response = client.chat.completions.create(
    model=deployment_name,
    messages=[{"role": "user", "content": "Explain GPU memory in two sentences."}],
    max_completion_tokens=1024,
)
print(response.choices[0].message.content)

Open models in the catalog

Since we opened the public preview in June 2026, we’ve expanded the Hugging Face Collection in Foundry to 150+ models for Managed Compute. Recent additions for reasoning and coding include DeepSeek V4.1 Flash (1017), Reflection Beam, and Meta’s Muse Spark Open. The NVIDIA Nemotron 3.5 family includes Embed and Rerank for search, Parse for extracting text, tables, and layout from documents, and Transcribe for speech. For structured decisions, the collection includes Laya Typed-Decisions from ConvAIInnovations.

“Open models unlock incredible opportunities for enterprises to build their own AI with the flexibility and control they need. Through our partnership with Microsoft, we’re bringing that opportunity to life with Managed Compute in Microsoft Foundry, making secure, high-performance deployment of open models a one-click experience.”

— Jeff Boudier, VP of Product, Hugging Face

The secure model and runtime supply chain

Before we add a model to the collection, we review its license, screen for untrusted code, and validate its SafeTensors weights. We stage those weights in Azure so your deployment can load them without connecting to Hugging Face Hub. We also build, scan, and sign the serving images and roll out runtime updates to existing deployments. Canonical’s Ubuntu Pro helps us maintain the OS packages in those images.

“Developers running open models on Microsoft Foundry Managed Compute shouldn’t have to track vulnerabilities in the software underneath. When Microsoft builds the serving images, it applies security patches to the Ubuntu Universe packages with Ubuntu Pro Expanded Security Maintenance.”

— Jehudi Castro-Sierra, Public Cloud Alliance Director, Canonical

Scaling production workloads

As traffic grows, add model instances behind the endpoint you already use. Each instance uses the number of GPUs specified in the template. With the DeepSeek 1017 template above, changing capacity from one to five takes the deployment from 4 to 20 GPUs. You pay hourly for each accelerator.

We sent the same load to two DeepSeek 1017 deployments: one with one instance and one with five. Over the same interval, the larger deployment processed about five times as many tokens and returned fewer HTTP 429 responses. To size your own deployment, run a representative workload and compare token throughput, request status codes, and streaming latency in Azure Monitor.

If your requests begin with the same system instructions or document context, prompt caching can save work where the model and runtime support it. With prompt-aware routing, Foundry prefers to send requests with matching prefixes to the same instance. With load-aware routing, Foundry can send requests to other instances when that instance is busy. You can supply prompt_cache_key as an affinity hint for related requests. When the runtime reports cache usage, usage.prompt_tokens_details.cached_tokens shows how many input tokens were reused.

For leading open models with a high-performing draft model, we enable speculative decoding to improve inference efficiency and reduce the effective cost per token.

We partnered with Signal65 to benchmark popular open models on AMD Instinct MI300X GPUs and compare Foundry with systems Signal65 tuned in its lab. In the study, one instance of deepseek-v4.1-flash-1017 supported 128 concurrent agents on four MI300X GPUs and matched the performance of Signal65’s tuned self-hosted serving environment on MI300X virtual machines. The study summary and full report (PDF) describe the workloads and test setup.

AT&T runs open models on Foundry Managed Compute

AT&T added Managed Compute alongside the model-serving platform it built on Azure Kubernetes Service. While developing OTel 2.0, the team used Phi-4, GPT-OSS-120B, and Gemma 4 to process about a trillion tokens of telecom data for training . Review the blog post AT&T and Microsoft scale trillion-token workloads with Microsoft Foundry and AMD to learn more.

“When you are processing hundreds of billions of tokens, infrastructure becomes part of the problem you solve. Foundry Managed Compute gave us access to GPU capacity at scale so our teams could focus on advancing OTel2.0 instead of managing infrastructure.”

— Mark Austin, Vice President, Data Science and AI at AT&T

Watch the AT&T segment of the Ignite breakout session around the 26 minute mark to learn how AT&T built its own serving stack for open models on Azure and used Foundry Managed Compute to expand model serving capacity for their production workloads.

Serve open models in production with Foundry Managed Compute

Announcing custom model serving in preview

  • LoRA adapters from Foundry post-training: Deploy registered adapters produced by Foundry fine-tuning to a matching base-model version and LoRA-enabled template. Multiple adapters share the target’s GPU capacity, each with its own deployment name. Review the announcement blog to learn more and get started.
  • Bring Your Own Weights (limited public preview): Register and deploy custom full-weight SafeTensors models, including weights trained outside Foundry, using a compatible catalog base model and Microsoft-provided deployment template.

Getting started resources

Start with the Managed Compute overview and follow the deployment guide to launch your first model. Once you’re serving traffic, see how to scale production workloads and monitor deployment performance. If you’re working with adapters, the LoRA adapter deployment overview explains which deployments can host them and how they share GPU capacity.

0 comments