Building an operations assistant on Azure that waits for approval before it acts

Grounding an assistant in runbooks is the easy half. The interesting engineering is in the boundary between what a model proposes and what an authenticated person actually authorizes.

An operations assistant stops being a demo the moment it can act. Answering “which runbook covers this alert” is a search problem. Opening an incident from that same conversation is a control-plane problem, and that is where a helpful prototype quietly turns into something you would not want on call.

I built a deliberately narrow reference app to sit on that boundary and see what breaks. It runs on Azure Container Apps, retrieves Markdown runbooks through Azure AI Search, calls Azure OpenAI, and reads the user identity from Container Apps authentication. The assistant can call a deterministic read tool right away. When it wants to open an incident, it does not. It records a pending action and returns a preview. A separate endpoint executes that action only after the same signed-in person confirms it.

I validated the full loop in a browser: Microsoft sign-in returned to the chat UI, retrieval produced three citations, the metrics tool answered, and an incident appeared only after the confirmation click. The incident system itself is mocked on purpose. The point was to exercise the controls around a mutation, not to file real tickets while debugging them.

This piece is about the decisions, not the deployment sequence. The full walkthrough lives in the step-by-step tutorial, and the runnable code, Bicep, and exact App Registration commands are in the personal assistant lab.

The shape of it

The app is small enough to hold in your head:

Bicep provisions Container Apps, Azure Container Registry, Azure OpenAI, Azure AI Search, Log Analytics, and Application Insights. Azure Developer CLI builds the image remotely in ACR, so I never needed Docker on my workstation for the Azure path.

Conversation history and pending actions live in memory, and the Container App is fixed at one replica. That is a design choice I want to be honest about, not a limitation I am hiding. Both stores have to become shared and durable before this scales past one process, and I will come back to why that ordering matters.

Retrieve the evidence before you ask the model anything

At startup the lab can create an Azure AI Search index and ingest the sample runbooks. Each document stores its title, source, content, and a 1,536-dimensional embedding from text-embedding-3-small. The vector field is searchable but neither retrievable nor stored, because the model needs the readable fields for grounding and citations, not the raw floats.

Queries combine lexical and vector retrieval, then apply semantic ranking:

payload = {
    "search": query,
    "queryType": "semantic",
    "semanticConfiguration": "runbook-semantic",
    "select": "id,title,source,content",
    "top": 3,
    "vectorQueries": [
        {
            "kind": "vector",
            "vector": query_vector,
            "fields": "content_vector",
            "k": 50,
            "weight": 2,
        }
    ],
}

top=3 caps what lands in the prompt. k=50 gives ranking a wider candidate set to work from. Azure AI Search fuses the lexical and vector lists with Reciprocal Rank Fusion before semantic reranking; the hybrid search docs and vector index docs cover the query behavior and schema. The agent gets those documents as context and returns citations from the same result. It is never asked to invent a source label.

The HTTP 207 ingestion trap

Batch ingestion has a failure mode that is easy to miss. Azure AI Search returns HTTP 207 when a request holds a mix of successful and failed document operations. A plain raise_for_status() sails right past that and tells you nothing about whether every runbook actually landed.

So the adapter reads the body, collects the keys that succeeded, records each failure, and compares against the full set of expected IDs:

indexed_ids = {
    str(result.get("key"))
    for result in results
    if result.get("status") is True and result.get("key")
}
failures = [
    result for result in results
    if result.get("status") is not True
]
missing_ids = sorted(expected_document_ids - indexed_ids)

if failures or missing_ids:
    raise RuntimeError("Azure AI Search document ingestion was incomplete.")

There is a test where one document in a 207 batch succeeds and another fails. Treating that as a clean bootstrap would leave retrieval quietly dependent on whichever files happened to make it in.

BOOTSTRAP_RAG_ON_STARTUP=true keeps the first deploy convenient, but it also hands the Container App identity write roles on Search. In production I would move index creation and ingestion into a job or pipeline with Search Service Contributor and Search Index Data Contributor, and leave the runtime with only Search Index Data Reader. The Azure AI Search RBAC guidance lists the roles.

Keep the read path and the write path apart

The metrics tool is read only. In the sample it derives repeatable CPU, memory, and error-rate values from a hash of the resource name, which keeps local tests stable and lets an Azure Monitor adapter drop in later without touching the agent loop. Tool arguments still cross an untrusted boundary, so Pydantic rejects extra fields, holds the metrics window to 5 through 60 minutes, bounds string lengths, and restricts severity to sev0 through sev4.

Incident creation runs a different route. A model tool call does not reach the incident adapter. It records the proposed title, summary, severity, session, and requester under a fresh action ID and returns a preview:

record = pending_action_service.create_incident_request(
    session_id=session_id,
    actor=actor,
    title=validated.title,
    summary=validated.summary,
    severity=validated.severity,
)

Confirmation requires {"confirm": true}. The service then checks that the action exists, is still pending, and belongs to the authenticated actor before it calls the mock adapter. This split matters because model intent and user authorization are two different facts. A prompt like “investigate the CPU alert and do what is needed” can push a model toward an incident tool. That is a reasonable proposal. It is not informed consent to mutate an external system.

Approval invariants

The approval boundary relies on a few invariants that should remain true regardless of the storage or ITSM implementation:

• The backend executes the server-side payload stored with the pending action, not a payload resubmitted by the browser.
• The authenticated actor confirming the action must match the actor who requested it.
• Only an action in the pending state can be claimed for execution.
• The action ID remains the idempotency key across retries and downstream calls.

These controls are necessary but not sufficient for every production operation. Pending actions also need an expiration and cancellation policy. Browser-based confirmation should include CSRF protection, and high-risk mutations may require step-up authentication, separation of duties, or approval from a second operator.

Put identity where the authorization decision is

Container Apps has built-in authentication for Microsoft Entra ID. With auth required, the platform validates the session or bearer token before the request reaches FastAPI and passes identity headers such as X-MS-CLIENT-PRINCIPAL-ID and X-MS-CLIENT-PRINCIPAL-NAME.

The app treats the principal ID as the stable actor identifier, falling back to it for the audit name when an app-only token has no user name. Missing identity is tolerated only in local development; in Azure mode a request with no principal ID gets a 401. The actor ID then shows up in two places that are easy to overlook:

memory_session_id = f"{actor.actor_id}:{session_id}"

That keeps two users who pick the same session_id from reading each other’s history. The pending action stores requested_by_id, and confirmation compares it against the current actor, because holding an opaque action UUID is not the same as being allowed to run it.

The whole trust model depends on Container Apps actually requiring authentication. Publishing the API anonymously and reading optional headers in app code would look similar and protect nothing. /healthz is the one deliberate exception so platform probes can run without a user session. Microsoft documents the setup under Container Apps authentication with Microsoft Entra ID.

Let managed identity carry the credentials

No Azure OpenAI or Search keys live in the runtime. Both resources have local auth disabled. In development DefaultAzureCredential can ride the developer’s Azure CLI session; in Azure the code selects ManagedIdentityCredential:

def build_token_credential(config):
    if config.is_development:
        return DefaultAzureCredential()

    return ManagedIdentityCredential(
        client_id=config.managed_identity_client_id
    )

The system-assigned Container App identity gets Cognitive Services OpenAI User plus the Search roles the bootstrap needs. A separate user-assigned identity holds AcrPull for the image, and the registry keeps its admin user disabled. This pulls long-lived keys out of code and environment variables, and it makes least privilege legible in Bicep: once ingestion moves out of the runtime, the Search write roles leave with it. Reference material sits in managed identities in Container Apps and Azure OpenAI authentication with managed identity.

Easy Auth still needs an App Registration credential for the browser provider. The template stores it as a Container App secret, but Azure Developer CLI also writes the value to the local .azure/<environment>/.env file in the clear. The documented flow clears that local copy after provisioning and expects rotation before expiry.

Idempotency and concurrency are backend problems, not model problems

Confirmation gets retried. People double-click, clients time out, networks drop the response after the server already did the work. The sample handles concurrent confirmation inside one process with a lock that atomically flips an action from pending to executing, and the incident adapter caches results by action_id used as the idempotency key, so replaying a completed confirmation returns the same incident.

Those controls prove the contract for one process. They do not turn an in-memory store into a distributed or durable one. A real implementation needs an external store with an atomic claim or compare-and-set, a unique idempotency constraint, and a recovery policy for actions stuck in executing after a crash, and the eventual ITSM call should receive that same stable key when its API accepts one.

This is why I would not touch maxReplicas first. With two replicas, /chat could write a pending action in one process while /confirm hits another that has never seen it. State has to move to shared durable storage before scaling out. Azure Managed Redis is a plausible target for short-lived shared state, and the code already exposes store protocols, but I am not claiming that integration is finished.

Telemetry and audit answer different questions

The app wires up Azure Monitor OpenTelemetry when APPLICATIONINSIGHTS_CONNECTION_STRING is set, with custom spans around chat.request, rag.search, and tool.execute. Attributes capture the backend, result count, tool name, and whether a pending action came out of the turn. The user message is intentionally not among them, because copying prompts into general telemetry only widens the set of places operational detail can surface.

The audit stream has a separate job. It records pending_action_created, pending_action_confirmed, and pending_action_result with the actor and action ID. Titles and severities help an investigation but can carry resource names, so their retention and access should reflect that. Traces explain how the system behaved. Audit records establish who asked for a mutation and who confirmed it. In production those often belong in different destinations with different retention and access policies. The Application Insights OpenTelemetry docs cover the SDK setup.

What this slice does not pretend to solve

One replica, in-memory history, in-memory pending actions, deterministic mock metrics, and a mock ITSM adapter. No long-term memory, no document-level ACLs, no private endpoints, no retrieval evaluations, no rate limiting. The sample also does not implement pending-action expiration, cancellation, CSRF protection, step-up authentication, payload signing, or risk-based approval policies. Those controls should be selected according to the impact of each tool rather than applied uniformly to every operation.

That scope was the whole point. It let me validate retrieval, model tool selection, authenticated approval, deployment, and telemetry without chasing several distributed systems at the same time. The order I would take toward production is fixed: externalize state and ingestion first, then add expiration and recovery for pending actions, define approval policies by risk, and finally connect Azure Monitor and a real ITSM system with scoped credentials, durable idempotency, and tested failure recovery.

If you want to run it, the fake mode needs Python 3.12 and no Azure subscription, and the Azure path is written up end to end. The companion README’s deploy section has the ordered steps, and the full tutorial walks through provisioning, the App Registration, and the browser validation in the same order I ran them.