ai-explained

Browser & Vision Agents

A browser agent navigates web pages like a human: click, type, scroll, read. A vision agent perceives the world through images. Together, they form the embodied frontier — agents that act in real (digital) environments rather than calling tidy APIs.

Browser agents

Use cases:

  • Web research that requires JavaScript-rendered pages.
  • Filling forms.
  • Booking flights, scheduling.
  • Scraping behind login walls.
  • QA automation.
  • Accessibility automation.

Two architectural styles

DOM-based

The agent sees the page’s HTML/accessibility tree:

<button id="submit-btn">Submit</button>
<input id="email" placeholder="Email" />

Agent issues actions like click("submit-btn") or type("email", "user@x.com").

Pros:

  • Reliable selectors.
  • Works on any DOM-renderable page.
  • Cheap (no vision model needed for understanding).

Cons:

  • Modern apps obfuscate IDs (auto-generated CSS classes).
  • Some content is canvas/visual-only.
  • Accessibility trees can be incomplete.

Tools: Playwright, Selenium, Puppeteer, browser-use.

Vision-based

The agent sees a screenshot of the page; clicks at pixel coordinates.

[Screenshot]
Action: click(x=423, y=187)

Powered by vision-language models (GPT-4-vision, Claude with vision, Gemini, Qwen-VL). The model identifies UI elements visually and outputs coordinates.

Pros:

  • Works on any visual interface — including non-web (desktop apps, mobile, games).
  • Resilient to DOM obfuscation.
  • Closer to how humans use software.

Cons:

  • Slower (vision models are bigger).
  • More expensive.
  • Pixel-precision can be flaky.

Tools: Anthropic’s Computer Use API (Sonnet’s computer_* tools), OpenAI Operator, Google’s Project Mariner.

Hybrid approaches

The current state-of-the-art is hybrid:

  • Vision to understand the page semantically and identify what to interact with.
  • DOM for precise, fast actions (click an aria-label-tagged element rather than a pixel).

Both Anthropic’s Claude Computer Use and the OpenAI Operator family use this hybrid approach.

A typical browser agent loop

def browser_agent(task: str):
    page = browser.new_page()
    messages = [{"role": "user", "content": task}]

    for step in range(50):
        screenshot = page.screenshot()
        accessibility = page.accessibility.snapshot()

        response = llm(
            messages=messages,
            tools=[click_tool, type_tool, scroll_tool, navigate_tool, finish_tool],
            attachments=[screenshot, accessibility],
        )

        if response.is_finish():
            return response.result
        execute_action(page, response.action)
        messages.append(response.message)
    raise TimeoutError("Agent exceeded steps")

The structure is the same agent loop from earlier — just with browser-specific tools.

Common challenges

Captchas

Pretty much by design, captchas defeat automated agents. Handle by:

  • Pausing, surfacing to a human.
  • Using a captcha-solving service (with appropriate ethical/legal review).
  • Avoiding sites with captchas.

Anti-bot measures

Many sites detect automation:

  • Use stealth-mode browsers (playwright-stealth).
  • Throttle requests; emulate human pacing.
  • Respect robots.txt and ToS.

Drift

A page’s layout changes; the agent’s selectors break. Vision-based agents partially mitigate this; you’ll still need to handle dynamic content gracefully.

Authentication

Sessions expire; some flows require multi-factor.

  • Persist cookies/sessions where appropriate.
  • Surface auth challenges to a human.
  • Use account credentials owned by the agent’s user, not service accounts.

Multi-step workflows

Sequential clicks across many pages; one mistake derails everything.

  • Break into smaller checkpoints.
  • Verify state after each major step (URL match, expected text on page).
  • Make actions idempotent where possible.

Vision agents beyond the browser

A vision agent operates on any pixel grid:

  • Desktop automation: GUI tools, legacy apps.
  • Mobile automation: Android, iOS via screencast.
  • Game playing: research and entertainment.
  • Robotics (closing the gap with AI for the physical world).
  • Document understanding: PDFs, images of forms, receipts.
  • Accessibility tools: voice/keyboard control of any app.

The frontier in 2026:

  • Anthropic Computer Use: production-ish agent that operates a desktop.
  • OpenAI Operator: web agent with vision.
  • Google’s Project Mariner: web agent.
  • Apple Intelligence on-device VLM: smaller scope, on-device.
  • Open models: Qwen-VL, InternVL, LLaVA — increasingly capable.

Document understanding as a special case

Reading a PDF is closer to vision than to text:

  • Tables with merged cells, footnotes, multi-column layouts.
  • Diagrams, charts, equations.
  • Scanned images (no embedded text layer).

Modern VLMs handle most of this well. Tools:

  • Claude with vision for general docs.
  • Mathpix, Azure Document Intelligence for specialized OCR.
  • Unstructured, LlamaParse for layout-aware extraction.

For RAG over images/PDFs, you can either:

  • Extract text via OCR/parsing → embed text → retrieve → include in prompt.
  • Use multimodal embeddings (CLIP-style) that map images directly into the embedding space.

Performance considerations

Browser/vision agents are slow (seconds per step) and expensive (vision tokens). For latency-critical workflows:

  • Cache page state when possible.
  • Use API-based shortcuts when available (don’t browser-automate Gmail when the API works).
  • Pre-compute embeddings and screenshots offline if the workflow is repeatable.

Safety considerations specific to browser agents

  • Phishing risk: an agent might be tricked into entering credentials on a fake page. Restrict to trusted domains.
  • Auto-purchase: confirm large transactions out-of-band.
  • Session hijacking: agent’s browser cookies are sensitive; don’t expose them.
  • Public exposure: agent screenshots may capture PII; redact before logging.

Treat the browser session as a sensitive vault, not a screenshot dumping ground.

Eval

Browser/vision agent evals are hard:

  • Task completion rate: does the agent finish the requested task?
  • Step efficiency: how many actions did it take? (Lower is often better.)
  • Cost / latency: average and p99.
  • Robustness: same task on slightly different pages.

Benchmarks: WebArena, VisualWebArena, OSWorld, MiniWob++, AndroidWorld.

These benchmarks correlate weakly with production performance — your specific task is unique. Build a domain-specific eval set.

See also