Computer Use (preview) — Azure OpenAI

What it is

• Computer Use is a specialized Azure OpenAI capability that lets a model operate a computer by “seeing” screenshots and emitting UI actions (for example, click, type, key press, move) to achieve a goal.
• It runs in a closed loop: the model proposes the next action; your app executes it in a real environment (browser/OS), captures a screenshot, and sends that back for the next step until completion or human stop.

How it works (high level)

• Access via the Responses API on your Azure OpenAI resource.
• Include the Computer Use tool in the request with display size and environment (browser, mac, windows, ubuntu).
• The model returns computer_call items with suggested actions. Your code performs them and replies with computer_call_output plus a fresh screenshot (as base64 data URL).

Model and deployment

• Model name to deploy: computer-use-preview (limited access; preview).
• Service: Azure OpenAI (in Azure AI Foundry/AI Studio).
• Regions (preview): eastus2, swedencentral, southindia.
• Request access: https://aka.ms/oai/cuaaccess

Minimal request shape

POST https://{resource}.openai.azure.com/openai/v1/responses
{
  "model": "<your-deployment-name>",
  "input": [{"type":"message","role":"user","content":"Check the latest AI news on bing.com."}],
  "tools": [{
    "type": "computer_use_preview",
    "display_width": 1024,
    "display_height": 768,
    "environment": "browser"
  }]
}

Loop step (send action result back):

{
  "model": "<your-deployment-name>",
  "tools": [{
    "type": "computer_call_output",
    "call_id": "<from last computer_call>",
    "output": [{
      "type": "input_image",
      "image_url": "data:image/png;base64,<screenshot-base64>"
    }]
  }]
}

Safety and supervision

• The API can flag pending_safety_checks; hand control to a human when these appear.
• Always run in a controlled/test environment; avoid exposing sensitive data.

Notes

• Preview feature; availability and behavior may change.
• Billing follows Azure OpenAI usage for the deployed model.

Back to top ↑

---

Last built with the static site tool.