How to Give an LLM Agent a Browser

of useful work still happens inside web interfaces.

To resolve a ticket, the support team needs to use an admin console. An operations team relies on dashboards to keep track of alarms. Before speaking with a customer, a sales rep will probably check the CRM first.

For LLM agents to become useful in these workflows, they need to work directly through the browser.

In this post, we’ll build a browser-using agent with OpenAI Agents SDK and Playwright MCP. We’ll first look at the loop behind browser use, and then put it into practice through a concrete case study.

1. The Mental Model

At a high level, a browser-using agent is best understood as an LLM placed in an interaction loop with a browser.

The loop goes like this: the agent starts with a task and the browser’s current state. The agent then interprets that state, decides what to do next, and sends an action back to the browser. That action would produce a new browser state, which becomes the input for the next decision.

This loop continues until the agent believes that the task is complete.

To make the loop work, we need two connections between the agent and the browser:

1. An observation channel, so that the agent can receive the current browser state.
2. An action channel, so that the agent can interact with the browser.

For the observation channel, common choices include screenshots, structured page information (e.g., Document Object Model or accessibility tree), or a combination of both.

For the action channel, the agent may use mouse and keyboard controls, or target specific page elements, or issue higher-level browser commands.

The choices of observation and action are usually independent, but in practice, two pairings are commonly seen:

1. screenshot + coordinate-based mouse and keyboard actions
2. structured page state + element-targeted browser actions

The first pairing leans towards general computer use, which extends beyond the browser to other desktop applications. The second one is more specific to browser use, as it takes advantage of the structure already available inside a web page.

In this post, we’ll focus on structured page observations together with element-targeted browser actions.

2. Case Study: Resolving a Customer Support Request

For our case study, we’ll build a browser-using agent that resolves a customer request through a web-based support console.

2.1 Preparing the Support Console

To create the browser environment, I vibe-coded a small customer support console. It’s a static web application built with plain HTML, CSS, and JavaScript. There is no application backend or database: all the data lives in the browser.

We can serve the console locally with Python’s built-in HTTP server:

cd toy_app
python -m http.server 8000 --bind 127.0.0.1

This makes the console available at http://127.0.0.1:8000, which we later pass to the agent as APP_URL:

The left side of the console works like a support inbox. Once a case is selected, the rest of the console shows the related order, customer context, and resolution policies. From there, the case can be resolved with an internal note. The console also has an audit log that keeps track of the update.

The agent’s task is simple: investigate one incoming support case and carry it through to resolution, entirely through this console interface.

2.2 Defining the Browser-Using Agent

Next, we configure the browser-using agent.

For the tech stack, here we’ll use the OpenAI Agents SDK to power the agent runtime and Playwright MCP to connect the agent to the browser.

Our final, configured agent looks like this:

# pip install openai-agents
from agents import Agent, ModelSettings
from openai.types.shared import Reasoning

agent = Agent(
    name="Support Console Browser Agent",
    model="gpt-5.4",
    model_settings=ModelSettings(
        reasoning=Reasoning(effort="medium"),
    ),
    instructions=AGENT_INSTRUCTIONS,
    mcp_servers=[playwright_server],
)

There are three pieces we need to unpack here, i.e., the LLM client, the agent instruction, and the browser tools.

First, we connect the Agents SDK to Azure OpenAI:

import os
from openai import AsyncAzureOpenAI
from agents import (
    set_default_openai_api,
    set_default_openai_client,
)

azure_client = AsyncAzureOpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    api_version=os.environ["OPENAI_API_VERSION"],
    azure_endpoint=os.environ["OPENAI_API_BASE"],
)

set_default_openai_client(azure_client)
set_default_openai_api("responses")

We register the client with the Agents SDK and configure it to use the Responses API.

Then, we have the following instruction, which is intentionally minimal:

AGENT_INSTRUCTIONS = """
You are an agent that can interact with a web browser.
""".strip()

We only set up the agent’s role. The actual task will come later in the prompt sent to the agent.

Next, we need to set up the Playwright MCP.

What is Playwright MCP?
Playwright is a browser automation library. It drives a real browser on your behalf and performs clicking, typing, navigating, and reading whatever’s on the page.

MCP (Model Context Protocol) is a standard way to expose tools to an LLM, so an agent can call them directly. Playwright MCP is what you get when you put the two together, i.e., Playwright’s browser capabilities, exposed as tools an agent can pick up and use.

What makes it interesting is how it shows the agent the page. Instead of a screenshot, Playwright MCP defaults to an accessibility snapshot, which is basically a structured read of what’s on the page. Interactive elements like links, buttons, and input fields each get a reference ID the agent can target directly. That’s exactly the structured observation and element-targeted action pairing we talked about earlier.

Playwright MCP runs through Node.js. To install Node.js, on Windows:

winget install OpenJS.NodeJS.LTS

On macOS:

brew install node

And on Linux:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.5/install.sh | bash
\. "$HOME/.nvm/nvm.sh"
nvm install --lts

Then verify the installation:

node --version
npx --version

We can configure the Agents SDK to start the MCP server through the following npx command:

from agents.mcp import MCPServerStdio

playwright_server = MCPServerStdio(
    name="Playwright MCP",
    params={
        "command": "npx",
        "args": [
            "-y",
            "@playwright/mcp@latest",
            "--browser",
            "chrome",
        ],
    },
)

Here, npx retrieves and runs the latest Playwright MCP package. The -y flag automatically accepts the npx confirmation prompt, while --browser chrome tells Playwright which browser to launch. Chrome is also Playwright MCP’s default browser, and if it is already installed, no separate browser installation is required.

MCPServerStdio configures the Agents SDK to launch Playwright MCP as a local process. When the agent first calls a browser tool, Playwright MCP opens a visible Chrome window and executes the requested browser action.

2.3 Running the Agent

Now we can give the agent a concrete task:

APP_URL = "http://127.0.0.1:8000"

TASK = f"""
Open {APP_URL} and resolve the support case for order ORD-1042.

The customer says they received the wrong item. Use the information available in the
application to determine and apply the appropriate resolution. Add a concise internal
note and make sure the resolution was successfully recorded.

Report what you did when the task is complete.
""".strip()

In the task prompt, we described how to access the application and the desired outcome.

We then run the agent with:

from agents import Runner

async with playwright_server:
    result = await Runner.run(
        agent,
        TASK,
        max_turns=20,
    )

print(result.final_output)

The async with block starts the Playwright MCP process and keeps it connected while the agent is running. We use max_turns to set an upper limit on how many turns the agent can take.

Once started, Chrome would open, and we can watch the agent work through the support console.

The final response summarized the outcome correctly:

Resolved CASE-4107 for order ORD-1042 with Replacement.

The case now shows Resolved, the recorded action is Replacement,
and the audit log contains the corresponding resolution entry.

If you like, you can also inspect the browser-tool calls and their outputs this way:

for item in result.new_items:
    print(type(item).__name__, item)

In my run, the agent found the case associated with ORD-1042 and it inspected the order, customer request, inventory status, and relevant resolution policy. It then concluded that a replacement was appropriate, added an internal note, and submitted the resolution. Finally, it checked the updated case and audit log to confirm that the action had been recorded.

This is exactly the agentic behavior we want.

3. From Browser Use to Computer Use

What we built in this case study is a browser-using agent. Still, the underlying pattern, that is, the observe, decide, act, and repeat loop, naturally extends to general computer use.

What changes are the observation and action channels.

In our case, Playwright MCP provides the agent with structured page information and allows it to target individual web elements. A more general computer-use agent may instead work from screenshots and control the mouse and keyboard by coordinates.

You can find the repo for our case study here: https://github.com/ShuaiGuo16/llm-browser-agent/tree/main