{"id":1208,"date":"2025-09-16T08:00:45","date_gmt":"2025-09-16T15:00:45","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/foundry\/?p=1208"},"modified":"2025-09-15T16:04:04","modified_gmt":"2025-09-15T23:04:04","slug":"announcing-computer-use-tool-preview-in-azure-ai-foundry-agent-service","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/foundry\/announcing-computer-use-tool-preview-in-azure-ai-foundry-agent-service\/","title":{"rendered":"Announcing Computer Use tool (Preview) in Azure AI Foundry Agent Service"},"content":{"rendered":"<h2 aria-level=\"2\"><span data-contrast=\"none\">Overview<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:160,&quot;335559739&quot;:80}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">We are excited to announce <\/span><b><span data-contrast=\"auto\">Computer Use<\/span><\/b><span data-contrast=\"auto\">\u2014are now available in preview in <\/span><b><span data-contrast=\"auto\">Azure AI Foundry Agent Service<\/span><\/b><span data-contrast=\"auto\">. It brings feature parity with the Azure OpenAI Responses API, but with the added advantage of seamless integration into the Foundry agent runtime and enterprise security. With this release, developers can create agents that not only reason over text, retrieve knowledge, or call APIs, but also directly interact with computer interfaces through natural language instructions. At launch, it is accessible through <\/span><b><span data-contrast=\"auto\">REST API and SDK<\/span><\/b><span data-contrast=\"auto\">, giving developers the flexibility to embed them directly into their applications, pipelines, and automation workflows.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<p><span data-ccp-props=\"{}\"> <a href=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/09\/computer-use-preview.gif\"><img decoding=\"async\" class=\"alignnone size-full wp-image-1209\" src=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/09\/computer-use-preview.gif\" alt=\"computer use preview image\" width=\"1920\" height=\"1080\" \/><\/a><\/span><\/p>\n<p><span data-contrast=\"auto\">By adding Computer Use, Foundry Agent Service expands the scope of what an agent can achieve in multi-step conversations. If you want to navigate complex enterprise applications, it opens the door to new, creative, and highly practical scenarios for enterprise AI.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<p aria-level=\"2\"><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:160,&quot;335559739&quot;:80}\">\u00a0<\/span><\/p>\n<h2 aria-level=\"2\"><span data-contrast=\"none\">Use cases<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:160,&quot;335559739&quot;:80}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">Computer Use addresses distinct, high-value scenarios.\u00a0 Computer Use allows agents to interact with applications in the same way a human user would. It enables agents that not only reason and respond but also take meaningful actions across digital environments.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<p><b><span data-contrast=\"auto\">Computer Use<\/span><\/b><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"5\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Web &amp; desktop automation:<\/span><\/b><span data-contrast=\"auto\"> Fill out forms, upload\/download artifacts, or complete tasks in apps without APIs.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<\/ul>\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"5\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" data-aria-posinset=\"2\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Operational copilots:<\/span><\/b><span data-contrast=\"auto\"> Help employees triage tickets or manage workflows across multiple enterprise dashboards.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<\/ul>\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"5\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" data-aria-posinset=\"3\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Legacy integration:<\/span><\/b><span data-contrast=\"auto\"> Interact with older desktop apps by simulating clicks and keystrokes.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<\/ul>\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"5\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" data-aria-posinset=\"4\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Human-in-the-loop workflows:<\/span><\/b><span data-contrast=\"auto\"> Require users to approve sensitive or high-risk steps before they are executed.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<\/ul>\n<h2 aria-level=\"2\"><span data-contrast=\"none\">How it works<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:160,&quot;335559739&quot;:80}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">Computer Use operates as a continuous loop of suggested actions, execution, and screenshot feedback.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<p><b><span data-contrast=\"auto\">Computer Use<\/span><\/b><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"4\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Action loop:<\/span><\/b><span data-contrast=\"auto\"> The agent requests actions (e.g., click, type, screenshot) from the computer-use-preview model.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<\/ul>\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"4\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" data-aria-posinset=\"2\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Execution environment:<\/span><\/b><span data-contrast=\"auto\"> Your code performs the action in a controlled environment (browser or desktop) and captures a screenshot.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<\/ul>\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"4\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" data-aria-posinset=\"3\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Screenshot feedback:<\/span><\/b><span data-contrast=\"auto\"> The screenshot is sent back to the model to inform the next step.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<\/ul>\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"4\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" data-aria-posinset=\"4\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Pixel-based reasoning:<\/span><\/b><span data-contrast=\"auto\"> Unlike Browser Automation, Computer Use interprets <\/span><b><span data-contrast=\"auto\">raw pixels<\/span><\/b><span data-contrast=\"auto\">, enabling adaptation to unfamiliar or dynamic UIs.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<\/ul>\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"4\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" data-aria-posinset=\"5\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Safety checks:<\/span><\/b><span data-contrast=\"auto\"> Malicious instructions, irrelevant domains, or sensitive domains trigger warnings that require human acknowledgment before proceeding.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><span data-contrast=\"auto\">Compared with Browser Automation, Computer Use offers richer visualization, broader cross-application support, and more human-like interaction patterns. However, because of its power, we strongly recommend using it only on <\/span><b><span data-contrast=\"auto\">low-privilege virtual machines<\/span><\/b><span data-contrast=\"auto\"> that do not contain sensitive data or credentials.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<table data-tablestyle=\"MsoNormalTable\" data-tablelook=\"1184\" aria-rowcount=\"8\">\n<tbody>\n<tr aria-rowindex=\"1\">\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<td data-celllook=\"4369\"><b><span data-contrast=\"auto\">Browser Automation\u202f<\/span><\/b><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<td data-celllook=\"4369\"><b><span data-contrast=\"auto\">Computer Use<\/span><\/b><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<\/tr>\n<tr aria-rowindex=\"2\">\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">Model support\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">Most models supported by Foundry Agent Service<\/span><\/td>\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">Computer-use-preview model\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<\/tr>\n<tr aria-rowindex=\"3\">\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">Visualize\u202fwhats\u202fhappening\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">No\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">Yes\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<\/tr>\n<tr aria-rowindex=\"4\">\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">How it understands the screen\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">Parse the HTML or XML pages into DOM documents\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">Raw pixel data from screenshots\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<\/tr>\n<tr aria-rowindex=\"5\">\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">How it acts\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">A list of actions provided by the model\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">Virtual keyword and mouse\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<\/tr>\n<tr aria-rowindex=\"6\">\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">Multi-step?\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">Yes\u202f\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">Yes\u202f\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<\/tr>\n<tr aria-rowindex=\"7\">\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">Interfaces\u202f\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">Browser\u202f\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">Computer and browser\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<\/tr>\n<tr aria-rowindex=\"8\">\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">Can I bring my own resource\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">BYO Playwright resource and store keys in connection\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/td>\n<td data-celllook=\"4369\"><span data-contrast=\"auto\">No resource required, we recommend using <b>low-privilege virtual machines<\/b>\u00a0<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<h2 aria-level=\"2\"><span data-contrast=\"none\">Security and responsible use<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:160,&quot;335559739&quot;:80}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"none\">WARNING:<\/span><span data-ccp-props=\"{&quot;335557856&quot;:16777215,&quot;335559738&quot;:240,&quot;335559739&quot;:0}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"none\">The Computer Use tool comes with significant security and privacy risks, including prompt injection attacks. Learn more about intended uses, capabilities, limitations, risks, and considerations when choosing a use case in the<\/span><a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/ai-foundry\/responsible-ai\/openai\/transparency-note?tabs=image%22\"><span data-contrast=\"none\"> Azure OpenAI transparency note<\/span><\/a><span data-contrast=\"none\">.\u00a0\u00a0<\/span><\/p>\n<p><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<h2 aria-level=\"2\"><span data-contrast=\"none\">Code samples<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:160,&quot;335559739&quot;:80}\">\u00a0<\/span><\/h2>\n<pre class=\"prettyprint language-py\"><code class=\"language-py\"># pylint: disable=line-too-long,useless-suppression\r\n# ------------------------------------\r\n# Copyright (c) Microsoft Corporation.\r\n# Licensed under the MIT License.\r\n# ------------------------------------\r\n\r\n\"\"\"\r\nDESCRIPTION:\r\n    This sample demonstrates how to use agent operations with the Computer Use tool (preview)\r\n    using a synchronous client. This sample uses fake screenshot to demonstrate how output actions work,\r\n    but the actual implementation would involve mapping the output action types to their corresponding\r\n    API calls in the user's preferred managed environment framework (e.g. Playwright or Docker).\r\n\r\n    NOTE: Usage of the computer-use-preview model currently requires approval. Please see\r\n    https:\/\/learn.microsoft.com\/azure\/ai-foundry\/openai\/how-to\/computer-use for more information.\r\n\r\nUSAGE:\r\n    python sample_agents_computer_use.py\r\n\r\n    Before running the sample:\r\n\r\n    pip install azure-ai-agents --pre\r\n    pip install azure-ai-projects azure-identity\r\n\r\n    Set these environment variables with your own values:\r\n    1) PROJECT_ENDPOINT - The Azure AI Project endpoint, as found in the Overview\r\n       page of your Azure AI Foundry portal.\r\n    2) MODEL_DEPLOYMENT_NAME - The deployment name of the AI model, as found under the \"Name\" column in\r\n       the \"Models + endpoints\" tab in your Azure AI Foundry project.\r\n\r\n    Optional:\r\n    - To target a specific environment, set COMPUTER_USE_ENVIRONMENT to one of: windows, mac, linux, browser\r\n      Otherwise defaults to 'browser'.\r\n\"\"\"\r\n\r\nimport os, time, base64\r\nfrom typing import List\r\nfrom azure.ai.agents.models._models import ComputerScreenshot, TypeAction\r\nfrom azure.ai.projects import AIProjectClient\r\nfrom azure.ai.agents.models import (\r\n    MessageRole,\r\n    RunStepToolCallDetails,\r\n    RunStepComputerUseToolCall,\r\n    ComputerUseTool,\r\n    ComputerToolOutput,\r\n    MessageInputContentBlock,\r\n    MessageImageUrlParam,\r\n    MessageInputTextBlock,\r\n    MessageInputImageUrlBlock,\r\n    RequiredComputerUseToolCall,\r\n    SubmitToolOutputsAction,\r\n)\r\nfrom azure.identity import DefaultAzureCredential\r\n\r\n\r\ndef image_to_base64(image_path: str) -&gt; str:\r\n    \"\"\"\r\n    Convert an image file to a Base64-encoded string.\r\n\r\n    :param image_path: The path to the image file (e.g. 'image_file.png')\r\n    :return: A Base64-encoded string representing the image.\r\n    :raises FileNotFoundError: If the provided file path does not exist.\r\n    :raises OSError: If there's an error reading the file.\r\n    \"\"\"\r\n    if not os.path.isfile(image_path):\r\n        raise FileNotFoundError(f\"File not found at: {image_path}\")\r\n\r\n    try:\r\n        with open(image_path, \"rb\") as image_file:\r\n            file_data = image_file.read()\r\n        return base64.b64encode(file_data).decode(\"utf-8\")\r\n    except Exception as exc:\r\n        raise OSError(f\"Error reading file '{image_path}'\") from exc\r\n\r\n\r\nasset_file_path = os.path.abspath(os.path.join(os.path.dirname(__file__), \"..\/assets\/cua_screenshot.jpg\"))\r\naction_result_file_path = os.path.abspath(os.path.join(os.path.dirname(__file__), \"..\/assets\/cua_screenshot_next.jpg\"))\r\nproject_client = AIProjectClient(endpoint=os.environ[\"PROJECT_ENDPOINT\"], credential=DefaultAzureCredential())\r\n\r\n# Initialize Computer Use tool with a browser-sized viewport\r\nenvironment = os.environ.get(\"COMPUTER_USE_ENVIRONMENT\", \"windows\")\r\ncomputer_use = ComputerUseTool(display_width=1026, display_height=769, environment=environment)\r\n\r\nwith project_client:\r\n\r\n    agents_client = project_client.agents\r\n\r\n    # Create a new Agent that has the Computer Use tool attached.\r\n    agent = agents_client.create_agent(\r\n        model=os.environ[\"MODEL_DEPLOYMENT_NAME\"],\r\n        name=\"my-agent-computer-use\",\r\n        instructions=\"\"\"\r\n            You are an computer automation assistant. \r\n            Use the computer_use_preview tool to interact with the screen when needed.\r\n            \"\"\",\r\n        tools=computer_use.definitions,\r\n    )\r\n\r\n    print(f\"Created agent, ID: {agent.id}\")\r\n\r\n    # Create thread for communication\r\n    thread = agents_client.threads.create()\r\n    print(f\"Created thread, ID: {thread.id}\")\r\n\r\n    input_message = (\r\n        \"I can see a web browser with bing.com open and the cursor in the search box.\"\r\n        \"Type 'movies near me' without pressing Enter or any other key. Only type 'movies near me'.\"\r\n    )\r\n    image_base64 = image_to_base64(asset_file_path)\r\n    img_url = f\"data:image\/jpeg;base64,{image_base64}\"\r\n    url_param = MessageImageUrlParam(url=img_url, detail=\"high\")\r\n    content_blocks: List[MessageInputContentBlock] = [\r\n        MessageInputTextBlock(text=input_message),\r\n        MessageInputImageUrlBlock(image_url=url_param),\r\n    ]\r\n    # Create message to thread\r\n    message = agents_client.messages.create(thread_id=thread.id, role=MessageRole.USER, content=content_blocks)\r\n    print(f\"Created message, ID: {message.id}\")\r\n\r\n    run = agents_client.runs.create(thread_id=thread.id, agent_id=agent.id)\r\n    print(f\"Created run, ID: {run.id}\")\r\n\r\n    # create a fake screenshot showing the text typed in\r\n    result_image_base64 = image_to_base64(action_result_file_path)\r\n    result_img_url = f\"data:image\/jpeg;base64,{result_image_base64}\"\r\n    computer_screenshot = ComputerScreenshot(image_url=result_img_url)\r\n\r\n    while run.status in [\"queued\", \"in_progress\", \"requires_action\"]:\r\n        time.sleep(1)\r\n        run = agents_client.runs.get(thread_id=thread.id, run_id=run.id)\r\n\r\n        if run.status == \"requires_action\" and isinstance(run.required_action, SubmitToolOutputsAction):\r\n            print(\"Run requires action:\")\r\n            tool_calls = run.required_action.submit_tool_outputs.tool_calls\r\n            if not tool_calls:\r\n                print(\"No tool calls provided - cancelling run\")\r\n                agents_client.runs.cancel(thread_id=thread.id, run_id=run.id)\r\n                break\r\n\r\n            tool_outputs = []\r\n            for tool_call in tool_calls:\r\n                if isinstance(tool_call, RequiredComputerUseToolCall):\r\n                    print(tool_call)\r\n                    try:\r\n                        action = tool_call.computer_use_preview.action\r\n                        print(f\"Executing computer use action: {action.type}\")\r\n                        if isinstance(action, TypeAction):\r\n                            print(f\"  Text to type: {action.text}\")\r\n                            # (add hook to input text in managed environment API here)\r\n\r\n                            tool_outputs.append(\r\n                                ComputerToolOutput(tool_call_id=tool_call.id, output=computer_screenshot)\r\n                            )\r\n                        if isinstance(action, ComputerScreenshot):\r\n                            print(f\"  Screenshot requested\")\r\n                            # (add hook to take screenshot in managed environment API here)\r\n\r\n                            tool_outputs.append(\r\n                                ComputerToolOutput(tool_call_id=tool_call.id, output=computer_screenshot)\r\n                            )\r\n                    except Exception as e:\r\n                        print(f\"Error executing tool_call {tool_call.id}: {e}\")\r\n\r\n            print(f\"Tool outputs: {tool_outputs}\")\r\n            if tool_outputs:\r\n                agents_client.runs.submit_tool_outputs(thread_id=thread.id, run_id=run.id, tool_outputs=tool_outputs)\r\n\r\n        print(f\"Current run status: {run.status}\")\r\n\r\n    print(f\"Run completed with status: {run.status}\")\r\n    if run.status == \"failed\":\r\n        print(f\"Run failed: {run.last_error}\")\r\n\r\n    # Fetch run steps to get the details of the agent run\r\n    run_steps = agents_client.run_steps.list(thread_id=thread.id, run_id=run.id)\r\n    for step in run_steps:\r\n        print(f\"Step {step.id} status: {step.status}\")\r\n        print(step)\r\n\r\n        if isinstance(step.step_details, RunStepToolCallDetails):\r\n            print(\"  Tool calls:\")\r\n            run_step_tool_calls = step.step_details.tool_calls\r\n\r\n            for call in run_step_tool_calls:\r\n                print(f\"    Tool call ID: {call.id}\")\r\n                print(f\"    Tool call type: {call.type}\")\r\n\r\n                if isinstance(call, RunStepComputerUseToolCall):\r\n                    details = call.computer_use_preview\r\n                    print(f\"    Computer use action type: {details.action.type}\")\r\n\r\n                print()  # extra newline between tool calls\r\n\r\n        print()  # extra newline between run steps\r\n\r\n    # Optional: Delete the agent once the run is finished.\r\n    agents_client.delete_agent(agent.id)\r\n    print(\"Deleted agent\")<\/code><\/pre>\n<h2><span data-contrast=\"none\">Getting started<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:160,&quot;335559739&quot;:80}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">To start using Computer Use in Foundry Agent Service:<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"1\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Request access<\/span><\/b><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<\/ul>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"o\" data-font=\"Courier New\" data-listid=\"1\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:1440,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Courier New&quot;,&quot;469769242&quot;:[9675],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;o&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" data-aria-posinset=\"2\" data-aria-level=\"2\"><span data-contrast=\"auto\"><a href=\"https:\/\/aka.ms\/oai\/cuaaccess\">computer-use-preview for Computer Use<\/a> (available in <\/span><b><span data-contrast=\"auto\">East US 2<\/span><\/b><span data-contrast=\"auto\">, <\/span><b><span data-contrast=\"auto\">Sweden Central<\/span><\/b><span data-contrast=\"auto\">, and <\/span><b><span data-contrast=\"auto\">South India<\/span><\/b><span data-contrast=\"auto\">)<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"1\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" data-aria-posinset=\"2\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Create deployments<\/span><\/b><span data-contrast=\"auto\"> in your Azure OpenAI resource once access is approved.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<\/ul>\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"1\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" data-aria-posinset=\"3\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Configure your agent<\/span><\/b><span data-contrast=\"auto\"> through REST API or SDK:<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<\/ul>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"o\" data-font=\"Courier New\" data-listid=\"1\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:1440,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Courier New&quot;,&quot;469769242&quot;:[9675],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;o&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" data-aria-posinset=\"2\" data-aria-level=\"2\"><i><span data-contrast=\"auto\">Computer Use<\/span><\/i><span data-contrast=\"auto\">: specify environment (browser, windows, mac, ubuntu) \u2192 implement the action loop (execute actions, send screenshots, continue until complete).<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"1\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" data-aria-posinset=\"4\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Apply best practices<\/span><\/b><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<\/ul>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"o\" data-font=\"Courier New\" data-listid=\"1\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:1440,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Courier New&quot;,&quot;469769242&quot;:[9675],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;o&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"2\"><span data-contrast=\"auto\">Run Computer Use agents only on <\/span><b><span data-contrast=\"auto\">low-privilege, isolated machines<\/span><\/b><span data-contrast=\"auto\">.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><b><span data-contrast=\"auto\">Learn more<\/span><\/b><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" data-aria-posinset=\"2\" data-aria-level=\"1\"><span data-contrast=\"auto\">\ud83d\udcd8<\/span><a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/ai-foundry\/agents\/how-to\/tools\/computer-use\"><span data-contrast=\"auto\"> Computer Use tool in Azure AI Foundry<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/a><\/li>\n<\/ul>\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" data-aria-posinset=\"3\" data-aria-level=\"1\"><span data-contrast=\"auto\">\ud83d\udcd8<\/span> <a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/ai-foundry\/responsible-ai\/agents\/transparency-note\"><span data-contrast=\"none\">Transparency notes for action tools<\/span><\/a><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><span data-contrast=\"auto\">With these steps and resources, you can quickly enable agents that combine text reasoning, image creation, and real-world computer interaction.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<p><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Overview\u00a0 We are excited to announce Computer Use\u2014are now available in preview in Azure AI Foundry Agent Service. It brings feature parity with the Azure OpenAI Responses API, but with the added advantage of seamless integration into the Foundry agent runtime and enterprise security. With this release, developers can create agents that not only reason [&hellip;]<\/p>\n","protected":false},"author":166509,"featured_media":1563,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1208","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-microsoft-foundry"],"acf":[],"blog_post_summary":"<p>Overview\u00a0 We are excited to announce Computer Use\u2014are now available in preview in Azure AI Foundry Agent Service. It brings feature parity with the Azure OpenAI Responses API, but with the added advantage of seamless integration into the Foundry agent runtime and enterprise security. With this release, developers can create agents that not only reason [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/posts\/1208","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/users\/166509"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/comments?post=1208"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/posts\/1208\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/media\/1563"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/media?parent=1208"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/categories?post=1208"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/tags?post=1208"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}