{"id":5798,"date":"2026-08-27T07:57:24","date_gmt":"2026-08-27T14:57:24","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/agent-framework\/?p=5798"},"modified":"2026-08-27T07:57:24","modified_gmt":"2026-08-27T14:57:24","slug":"agent-harness-making-your-claw-production-ready","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/agent-framework\/agent-harness-making-your-claw-production-ready\/","title":{"rendered":"Agent Harness: Making your claw production-ready"},"content":{"rendered":"

Part 4 of Build your own claw and agent harness with Microsoft Agent Framework<\/a>.<\/em><\/p>\n

Over the last three parts our personal finance assistant grew from a single tool into a genuinely capable agent: it plans, reads your portfolio, asks before it trades, remembers what matters, loads skills on demand, reorganizes files with a shell, computes with CodeAct, and fans research out to background agents. It works on your machine. But “works on my machine” isn’t the same as ready to run for other people<\/em>.<\/p>\n

This final part closes that gap along four axes:<\/p>\n

    \n
  1. Observability<\/strong> – see what the claw is doing with OpenTelemetry traces, token usage, and tool calls.<\/li>\n
  2. Governance<\/strong> – screen prompts and responses through Microsoft Purview<\/strong>, so data access is discoverable, classified, and auditable for a regulated finance context.<\/li>\n
  3. Deployment<\/strong> – host the claw as a Foundry Hosted Agent<\/strong>.<\/li>\n
  4. Evals<\/strong> – measure quality with local finance checks and hosted Foundry evals, then use the results to tune prompts, tools, and skills.<\/li>\n<\/ol>\n

    To get there we make one structural change. Up to now each step was a single program. A production agent is usually run more than one way<\/em> – interactively while you develop, hosted in the cloud for real use, and inside an eval harness in CI. So we split the claw into a shared agent factory<\/strong> plus three thin hosts that consume it: a console<\/strong>, a hosted<\/strong> service, and an evals<\/strong> runner. The agent is defined once; only the host around it changes.<\/p>\n

    Define once: the shared agent<\/h2>\n

    Everything that makes the claw ours<\/em> – instructions, file access, the valuation and risk skills, memory, approvals, the shell, CodeAct, and the background research agent – now lives in one factory. Each host just calls it.<\/p>\n

    In .NET<\/strong> the factory returns the built agent plus the resources the host should dispose:<\/p>\n

    await using ClawAgentBuild build = await ClawAgentFactory.CreateAsync(new ClawAgentFactoryOptions\r\n{\r\n    Log = Console.WriteLine,\r\n});\r\n\r\n\/\/ build.Agent is the same claw from Part 3 - skills, shell, CodeAct, background agents, approvals.<\/code><\/pre>\n
    In Python<\/strong> it’s an async<\/code> factory that returns an async context manager, so the shell and any MCP skills sessions are torn down cleanly:<\/p>\n
    agent = await build_claw_agent(credential=AzureCliCredential())\r\nasync with agent:\r\n    # \u2026 run the agent \u2026<\/code><\/pre>\n
    With one definition feeding every host, observability, governance, deployment, and evals all apply to the same agent<\/em> – not three subtly different copies.<\/p>\n
    See what it’s doing: observability<\/h2>\n
    An agent that reads files, runs code, and calls tools is a small distributed system. When something goes wrong – a skill misfires, a tool loops, a response costs 10x what you expected – you need to see<\/em> it. The harness already emits OpenTelemetry spans, metrics, and logs for model calls, tool invocations, and token usage; you just wire up an exporter.<\/p>\n
    The agent carries a single OpenTelemetry source name<\/strong> so hosts can subscribe to exactly its signals.<\/p>\n
    In .NET<\/strong> that’s one option on the harness:<\/p>\n
    AIAgent agent = chatClient.AsHarnessAgent(new HarnessAgentOptions\r\n{\r\n    OpenTelemetrySourceName = ClawAgentFactory.OpenTelemetrySourceName,\r\n    \/\/ \u2026 file access, skills, shell, CodeAct, background agents \u2026\r\n});<\/code><\/pre>\n
    When OTEL_EXPORTER_OTLP_ENDPOINT<\/code> is set, the console<\/strong> host stands up trace and metric providers pointed at that source and sends their telemetry to the configured OTLP collector:<\/p>\n
    var otlpEndpoint = Environment.GetEnvironmentVariable(\"OTEL_EXPORTER_OTLP_ENDPOINT\");\r\nvar telemetryEnabled = !string.IsNullOrWhiteSpace(otlpEndpoint);\r\n\r\nusing var tracerProvider = telemetryEnabled\r\n    ? Sdk.CreateTracerProviderBuilder()\r\n        .AddSource(ClawAgentFactory.OpenTelemetrySourceName)\r\n        .AddOtlpExporter(options => options.Endpoint = new Uri(otlpEndpoint!))\r\n        .Build()\r\n    : null;<\/code><\/pre>\n
    In Python<\/strong>, instrumentation is on by default – a single call wires the providers from environment variables (OTLP endpoint, console exporters, sensitive-data capture):<\/p>\n
    from agent_framework.observability import configure_otel_providers, get_tracer\r\n\r\nconfigure_otel_providers()\r\nwith get_tracer().start_as_current_span(\"Claw Console Session\"):\r\n    agent = await build_claw_agent(credential=AzureCliCredential())\r\n    async with agent:\r\n        # \u2026 run the agent; spans, metrics, and logs flow to your collector \u2026<\/code><\/pre>\n
    What the harness does for you vs. what you wire by hand:<\/strong> the harness produces<\/em> the telemetry – spans for each tool call and model turn, token-usage metrics, structured logs. You choose where it goes<\/em>: an OTLP collector, the console, or Azure Monitor \/ Application Insights<\/strong>. Locally you wire the exporters yourself; when hosted on Foundry you wire nothing at all<\/strong> – the hosting runtime registers the exporter pipeline and Foundry injects APPLICATIONINSIGHTS_CONNECTION_STRING<\/code> for you (see the deployment section below).<\/p><\/blockquote>\n
    Keep it governed: Purview<\/h2>\n
    A finance assistant touches sensitive material. In a regulated setting you need prompts and responses screened against organizational policy – credit-card numbers, confidential holdings, disallowed content – with an audit trail. Microsoft Purview<\/strong> does exactly that, and the integration is a thin wrapper around the chat client, so it composes with everything else the claw does.<\/p>\n
    We make it opt-in<\/strong>: when PURVIEW_CLIENT_APP_ID<\/code> is set the factory adds Purview; otherwise it runs unchanged. In .NET<\/strong> it’s a builder step on the chat client:<\/p>\n
    if (!string.IsNullOrWhiteSpace(purviewClientAppId))\r\n{\r\n    chatClient = chatClient\r\n        .AsBuilder()\r\n        .WithPurview(browserCredential, new PurviewSettings(\"Claw\"))\r\n        .Build();\r\n}<\/code><\/pre>\n
    In Python<\/strong> it’s chat middleware handed to the FoundryChatClient<\/code>:<\/p>\n
    from agent_framework.microsoft import PurviewChatPolicyMiddleware, PurviewSettings\r\n\r\nmiddleware = []\r\nif client_app_id := os.environ.get(\"PURVIEW_CLIENT_APP_ID\"):\r\n    credential = InteractiveBrowserCredential(client_id=client_app_id)\r\n    middleware = [PurviewChatPolicyMiddleware(credential, PurviewSettings(app_name=\"Claw\"))]\r\n\r\nclient = FoundryChatClient(credential=..., middleware=middleware)<\/code><\/pre>\n
    Now every prompt is checked before it reaches the model and every response before it reaches the user; blocked content is replaced with a policy message, and the interaction is logged for audit. Purview needs a Microsoft 365 E5 tenant with the right Graph permissions – see the AgentWithPurview<\/code><\/a> and purview_agent<\/code><\/a> samples for the full setup.<\/p>\n
    Ship it: deploy as a Foundry Hosted Agent<\/h2>\n
    Creating the agent host application<\/h3>\n
    Because the agent is defined once, hosting it is mostly wiring<\/em>, not rewriting. The hosted<\/strong> host takes the same build.Agent<\/code> and exposes it over the Responses protocol so Foundry can run it. In .NET<\/strong> the entire host is a thin ASP.NET app:<\/p>\n
    using Azure.Core;\r\nusing Azure.Identity;\r\nusing ClawAgent;\r\nusing Microsoft.Agents.AI.Foundry.Hosting;\r\n\r\n\/\/ A specific credential is preferable in production (e.g. ManagedIdentityCredential); the chained\r\n\/\/ credential below tries a dev token first (for local Docker debugging), then DefaultAzureCredential.\r\nTokenCredential credential = new ChainedTokenCredential(\r\n    new DevTemporaryTokenCredential(),\r\n    new DefaultAzureCredential());\r\n\r\nawait using ClawAgentBuild build = await ClawAgentFactory.CreateAsync(new ClawAgentFactoryOptions\r\n{\r\n    ProjectEndpoint = Environment.GetEnvironmentVariable(\"FOUNDRY_PROJECT_ENDPOINT\"),\r\n    DeploymentName = Environment.GetEnvironmentVariable(\"FOUNDRY_MODEL\"),\r\n    Credential = credential,\r\n\r\n    \/\/ Disable filesystem and shell access on the hosted container (see the risk note below).\r\n    EnableFileAccess = false,\r\n    EnableShell = false,\r\n});\r\n\r\nvar builder = WebApplication.CreateBuilder(args);\r\n\r\n\/\/ Registers the Responses API host for the agent AND auto-applies OpenTelemetry.\r\nbuilder.Services.AddFoundryResponses(build.Agent);\r\n\r\nvar app = builder.Build();\r\n\r\n\/\/ The endpoint that live Foundry calls.\r\napp.MapFoundryResponses();\r\n\r\napp.Run();<\/code><\/pre>\n
    In Python<\/strong> it’s the responses host server:<\/p>\n
    from agent_framework_foundry_hosting import ResponsesHostServer\r\n\r\nagent = await build_claw_agent(\r\n    credential=DefaultAzureCredential(),\r\n    enable_file_access=False,   # off on the hosted container\r\n    enable_shell=False,         # off on the hosted container\r\n)\r\nawait ResponsesHostServer(agent).run_async()<\/code><\/pre>\n
    Core telemetry collection and export are automatically configured when hosted.<\/span><\/strong>\u00a0There are no exporters to configure in either language. In .NET<\/strong>, AddFoundryResponses<\/code> automatically wraps the agent with OpenTelemetryAgent<\/code>, and the Foundry hosting runtime registers the OTLP exporter pipeline. In Python<\/strong>, Agent Framework is natively instrumented (on by default) and the hosting runtime collects and exports its spans \u2014 so the hosted host makes no<\/em> configure_otel_providers()<\/code> call at all (unlike the local console). When your agent runs on Foundry, it injects APPLICATIONINSIGHTS_CONNECTION_STRING<\/code> automatically, so traces, metrics, and logs land in Application Insights with zero configuration. To capture prompt and response content in those traces (off by default), set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true<\/code> in .NET<\/strong> or ENABLE_SENSITIVE_DATA=true<\/code> in Python<\/strong>.<\/p>\n
    Turn off risky features<\/h3>\n
    For the hosted agent, we turn off file access and shell. This is the one place we deliberately<\/em> diverge from the console. In a shared, hosted environment, giving the model arbitrary read\/write access to the container filesystem, or letting it run shell commands, is a serious security risk – data exfiltration, tampering, and persistence – even behind a deny-list. So the hosted build sets EnableFileAccess = false<\/code> and EnableShell = false<\/code>. Background agents stay on. If you do<\/em> enable file access or shell on a hosted container, treat it as a production security decision and scope it tightly.<\/p>\n
    If you genuinely need file access when hosted, don’t reach for the container disk – supply an external AgentFileStore<\/code><\/strong> instead, for example one backed by Azure Blob Storage. In .NET<\/strong>:<\/p>\n
    await using ClawAgentBuild build = await ClawAgentFactory.CreateAsync(new ClawAgentFactoryOptions\r\n{\r\n    \/\/ ...\r\n    EnableFileAccess = true,\r\n    FileStore = new MyBlobAgentFileStore(blobContainerClient),\r\n});<\/code><\/pre>\n
    In Python<\/strong> it’s the file_access_store<\/code> argument:<\/p>\n
    agent = await build_claw_agent(\r\n    credential=DefaultAzureCredential(),\r\n    enable_file_access=True,\r\n    file_access_store=MyBlobAgentFileStore(blob_container_client),\r\n)\r\nasync with agent:\r\n    # \u2026 run the agent \u2026<\/code><\/pre>\n
    The claw reads and writes through the store’s abstraction, so files live in blob storage (governed, durable, shared) rather than on ephemeral container disk.<\/p>\n
    Enabling CodeAct in the container<\/h3>\n
    This is the second deliberate divergence. Locally, CodeAct runs on Hyperlight<\/strong>, which isolates guest code in a VM-backed micro-sandbox.\u00a0 For the hosted build, we pass a CodeActProvider<\/code> backed by LocalCodeAct<\/code><\/strong>, which runs the generated Python in a child process and leans on the hosted container itself as the sandbox \u2014 the same approach as the canonical Hosted-LocalCodeAct<\/code> sample (the container image installs python3<\/code> for it). In .NET<\/strong>:<\/p>\n
    await using ClawAgentBuild build = await ClawAgentFactory.CreateAsync(new ClawAgentFactoryOptions\r\n{\r\n    \/\/ ...\r\n    CodeActProvider = new LocalCodeActProvider(\r\n        Environment.GetEnvironmentVariable(\"LOCAL_CODEACT_PYTHON\") ?? \"python3\"),\r\n});<\/code><\/pre>\n
    LocalCodeAct<\/code> is not itself a sandbox \u2014 it executes model-generated Python, so only run it inside an externally sandboxed environment such as a hosted-agent container. To drop CodeAct entirely instead, set EnableCodeAct = false<\/code>.<\/p>\n
    Build and deploy<\/h3>\n
    Both hosts ship an agent.manifest.yaml<\/code> and an agent.yaml<\/code>, but the two files have different jobs. The manifest<\/strong> is the template passed to azd ai agent init<\/code>: it carries the agent’s name, metadata, protocol, and configurable parameters. The agent definition<\/strong> describes what Foundry runs – the Responses protocol, CPU and memory, and any environment variables the container needs. See the actual .NET manifest<\/a>, .NET agent definition<\/a>, Python manifest<\/a>, and Python agent definition<\/a>.<\/p>\n
    The deployment path then differs by language. Python<\/strong> uses Foundry’s default code (ZIP) deployment: azd<\/code> uploads the self-contained sample folder, installs requirements.txt<\/code>, and starts hosted.py<\/code>. Pass that entry point when you initialize the project:<\/p>\n
    cd python\/samples\/02-agents\/harness\/build_your_own_claw\/claw_step04_production_ready\r\nazd ai agent init -m agent.manifest.yaml --entry-point hosted.py\r\nazd up      # or `azd deploy` on subsequent pushes<\/code><\/pre>\n
    The .NET<\/strong> sample deploys as a container image because it builds against the Agent Framework repo source (ProjectReference<\/code>) and uses repo-level Central Package Management. A ZIP deployment uploads only the project folder, so a server-side restore cannot resolve those out-of-folder references or Directory.Packages.props<\/code>. Publish locally first, then let azd<\/code> build and push the image:<\/p>\n
    # From ClawAgent.Hosted:\r\ndotnet publish -c Release -f net10.0 -r linux-x64 --self-contained false -o out\r\n\r\n# First time only:\r\nazd ai agent init -m agent.manifest.yaml --deploy-mode container\r\nazd up      # or `azd deploy` after republishing on subsequent pushes<\/code><\/pre>\n
    The .NET Dockerfile<\/code> copies the pre-published out\/<\/code> into a python3<\/code>-enabled aspnet:10.0<\/code> image. For the complete prerequisites and identity assignments, follow the .NET deployment guide<\/a> or the Python deployment guide<\/a>.<\/p>\n
    Prove it’s good: evals<\/h2>\n
    Before and after you deploy, you want to evaluate whether the claw is actually good<\/em>. You may also want to ensure that no regressions follow any changes, or test different prompts to find the most optimal version. The evals<\/strong> host builds the agent and runs it against a small set of finance queries with two layers of checks.<\/p>\n
    Local checks<\/strong> are plain functions – fast, free, and runnable in CI. In .NET<\/strong>:<\/p>\n
    LocalEvaluator localEvaluator = new(\r\n    FunctionEvaluator.Create(\"numeric_valuation\", item =>\r\n        !item.Query.Contains(\"Value MSFT\", StringComparison.OrdinalIgnoreCase)\r\n        || Regex.IsMatch(item.Response, @\"\\d\")));\r\n\r\nAgentEvaluationResults results = await build.Agent.EvaluateAsync(queries, localEvaluator);\r\nConsole.WriteLine($\"Passed: {results.Passed}\/{results.Total}\");<\/code><\/pre>\n
    In Python<\/strong> the same idea with the @evaluator<\/code> decorator:<\/p>\n
    @evaluator(name=\"numeric_valuation_answer\")\r\ndef numeric_valuation_answer(query: str, response: str) -> bool:\r\n    return \"msft\" not in query.lower() or any(c.isdigit() for c in response)\r\n\r\nlocal = LocalEvaluator(numeric_valuation_answer)\r\nresults = await evaluate_agent(agent=agent, queries=queries, evaluators=local)\r\nprint(f\"{results[0].passed}\/{results[0].total}\")<\/code><\/pre>\n
    Hosted Foundry evals<\/strong> add model-graded quality scores (relevance, coherence) and a shareable report – gated on FOUNDRY_PROJECT_ENDPOINT<\/code> so the local checks always run:<\/p>\n
    \/\/ .NET\r\nFoundryEvals foundryEvals = new(projectClient, deploymentName, FoundryEvals.Relevance, FoundryEvals.Coherence);\r\nAgentEvaluationResults quality = await build.Agent.EvaluateAsync(queries, foundryEvals);<\/code><\/pre>\n
    # Python\r\nfrom agent_framework.foundry import FoundryChatClient, FoundryEvals\r\n\r\nfoundry = FoundryEvals(\r\n    client=FoundryChatClient(credential=credential),\r\n    evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.COHERENCE],\r\n)\r\nquality = await evaluate_agent(agent=agent, queries=queries, evaluators=foundry)<\/code><\/pre>\n
    Run these on every change and the score tells you whether a new instruction, tool, or skill helped or hurt – the tuning loop that keeps a deployed agent honest.<\/p>\n
    Run it<\/h2>\n
    .NET<\/strong> – run the console, the evals, or the hosted service:<\/p>\n
    cd dotnet\r\ndotnet run --project samples\/02-agents\/Harness\/BuildYourOwnClaw\/Claw_Step04_ProductionReady\/ClawAgent.Console\r\ndotnet run --project samples\/02-agents\/Harness\/BuildYourOwnClaw\/Claw_Step04_ProductionReady\/ClawAgent.Evals\r\ndotnet run --project samples\/02-agents\/Harness\/BuildYourOwnClaw\/Claw_Step04_ProductionReady\/ClawAgent.Hosted<\/code><\/pre>\n
    Python<\/strong><\/p>\n
    uv run python\/samples\/02-agents\/harness\/build_your_own_claw\/claw_step04_production_ready\/console.py\r\nuv run python\/samples\/02-agents\/harness\/build_your_own_claw\/claw_step04_production_ready\/evals.py\r\nuv run python\/samples\/02-agents\/harness\/build_your_own_claw\/claw_step04_production_ready\/hosted.py<\/code><\/pre>\n
    The console behaves exactly like Part 3’s claw – now with telemetry flowing to your collector.<\/p>\n