Optimize your website for AI browsers

What does this guide cover?

This guide is the hands-on manual. You go through your site area by area and fix what AI browsers get stuck on. What Agentic Engine Optimization means in principle and how to think about agents as a new kind of visitor is covered in the guide Agentic Engine Optimization. Which AI browsers exist and how they are built is shown in AI browsers compared.

The agent clicks like a human and is easily overwhelmed

Perplexity Comet, Claude in Chrome, Gemini in Chrome and the browser in ChatGPT look at the rendered page and click through it the way a person would. That is flexible but error-prone. One ambiguous button, a form field without a label or a layout that jumps while loading, and the task fails. You can be at the top of the AI answer and still lose the sale because the agent gets stuck in checkout.

The good news is that the sources of error repeat. Most fixes are small, targeted changes that do not need a relaunch.

How does an AI browser read your page?

“Like a human” is only true of the result. The agent does not see your page, it reads it in three steps.

1. Read the page. It fetches the accessibility tree, the structure of roles, names and states of every interactive element, and takes a screenshot when needed.
2. Match the element. It decides which element fits the task, for example “this field is called Email, so the address goes here”.
3. Trigger the action. It clicks or types into exactly that element.

Anthropic describes exactly this for Claude. Claude reads pages through their structure, meaning the accessibility tree, elements, forms and tabs, and through screenshots with coordinates. The documentation advises reading the tree first and only then taking a screenshot. Claude falls back to screenshots and coordinate clicks for canvas interfaces, heavily virtualised lists and elements inside cross-origin iframes. Perplexity writes that the Comet assistant takes in the visible text, headings and metadata of the open page with every request. OpenAI names screenshots and the rendered state of the page for the ChatGPT cloud browser. For Gemini in Chrome, Google only documents that auto browse clicks, scrolls and fills in forms.

Whatever is marked up cleanly in the structure, an agent finds reliably. An element that merely looks like a button remains guesswork based on the screenshot, slower and more error-prone. That is where most stumbling blocks arise.

Where do the agents stop by themselves?

Some interruptions are intentional. Vendors make their agents stop at sensitive points and let the person decide. Plan these handovers into your flows.

The table reflects the status on November 3, 2026. OpenAI discontinued the ChatGPT Atlas browser on August 9, 2026. Since then, the browser features live in the ChatGPT Work cloud browser, in the desktop app's built-in browser and in the Chrome extension. Google discontinued Project Mariner in May 2026, and its technology now sits in Gemini. What matters for you as a website owner is therefore that every agent can operate your site.

Common mistakes at a glance

In April 2026, Google published recommendations for agent-friendly websites on web.dev. Together with the patterns agents fail on in practice, they result in this list. The right-hand column tells you how to find each issue on your site.

Google adds two more points. Clickable elements should be recognizable as clickable via cursor: pointer. Interactive elements must be visible and larger than 8 square pixels, otherwise the agent filters them out when analyzing the image.

Clean up the accessibility tree

Open Chrome DevTools, choose the Accessibility tab under Elements and click through your most important elements. Every interactive element needs a role (button, link, textbox, checkbox) and a name. If either is missing, the agent only sees an image at that point. The Lighthouse category “Agentic Browsing” checks this in the audit “agent-accessibility-tree”.

This is what the difference looks like in code.

Forms an agent can fill in

Forms are where tasks fail most often, because every field has to be recognized correctly. These rules come from the form recommendations on web.dev and the WCAG 2.2 techniques.

• One label per field, linked via for and id. Placeholder text disappears when typing and therefore does not replace the label.
• Set autocomplete values, such as email, name, street-address, postal-code, username, current-password, cc-number. This tells the browser what belongs in which field.
• Suitable field types. type="email" and type="tel" for contact details, and for card numbers and ZIP codes type="text" with inputmode="numeric" instead of type="number".
• One name field instead of separate fields where possible, and no drop-down lists for dates.
• Errors as text next to the field. Mark the field with aria-invalid="true" and link the message via aria-describedby. No agent recognizes a red border on its own.
• Build custom widgets to a pattern. For choices such as size or color, the WAI-ARIA Authoring Practices offer the Combobox and Listbox patterns. A native <select> is usually the most robust option.

Cookie banners, pop-ups and chat widgets

Overlays are doubly hard for agents. They cover the elements underneath, and the agent first has to work out how to close them. In the 2025 BrowserArena study, removing pop-up banners was one of the three typical failures of the agents tested, alongside CAPTCHAs. A study of deliberately placed pop-ups (ACL 2025) found that agents clicked such pop-ups in 86 % of cases on average and that their task success rate dropped by 47 %. That study looked at targeted attacks. A comparable measurement for cookie banners does not exist yet.

1. Build the cookie banner as a dialog with the role dialog, a heading as its name and real buttons with clear text such as “Accept all” and “Reject”. The “Dialog (Modal)” pattern in the WAI-ARIA Authoring Practices shows how focus and closing work.
2. Do not place invisible or semi-transparent layers over the content that remain after closing.
3. Keep checkout, cart and forms free of newsletter pop-ups and chat widgets that sit on top of buttons.
4. Check the page once with the banner open and once with it closed. Both states must be usable.

The agent must not invent consent on the user's behalf. A clearly labeled banner helps it pass the question to the person correctly or answer it as instructed.

Tune bot protection for agents

Many AI browsers run in the user's own browser and look like normal Chrome to your server. Others run in the provider's cloud on data center IPs and quickly end up in a blanket block. These cloud agents now have an identifier.

• The ChatGPT cloud browser signs its requests with Web Bot Auth, meaning HTTP Message Signatures under RFC 9421. It sends the headers Signature, Signature-Input and Signature-Agent with the value "https://chatgpt.com". OpenAI's help explains how to allow it on Cloudflare, Akamai, HUMAN and Vercel, and says that the website decides whether to allow this traffic.
• Since 1 July 2026, Cloudflare lists such signed agents as part of Verified Bots and distinguishes direct access from access through an intermediary.
• For Claude in Chrome, Comet and Gemini in the local browser, the vendors document no identifier of their own. They arrive with the user's IP and browser.

So use CAPTCHAs selectively, for example at login after several failed attempts. According to Anthropic, Claude in Chrome does not solve CAPTCHAs, and ChatGPT hands over to the user. A CAPTCHA in the wrong place ends the task. The guide Bot protection and SEO explains how to set up bot protection so that it slows down attackers and lets wanted bots and customers through.

Layout, navigation and rendering

• Load stably. Give images and ad slots fixed dimensions so nothing jumps. You measure CLS via the Core Web Vitals.
• Same elements in the same place. The add-to-cart button should sit in the same position on all product pages.
• Menus via click and focus. A menu that only opens on hover also needs a button with aria-expanded.
• A clear end. For long lists, offer pages with their own URLs or a “Load more” button.
• Content in the HTML. Product name, price and availability belong in the delivered HTML. The JavaScript crawling study shows where source code and rendered HTML diverge.

WebMCP as the next step

Removing stumbling blocks means staying out of the agent's way. WebMCP goes one step further. Your page describes its most important actions, such as search or cart, directly as tools. The agent then does not have to guess what a button does.

WebMCP is a W3C Community Group draft and not a standard. Chrome offers it as an origin trial from version 149, and Edge also runs it as an origin trial. The interface is now called document.modelContext. For forms there is a declarative variant with the attributes toolname and tooldescription on the <form>. By default, the user still has to click submit. The first major agent to use such tools is the built-in browser of the ChatGPT desktop app, since 31 August 2026; ChatGPT's Chrome extension does not.

The guide WebMCP explains the setup, the imperative variant and how to measure agent calls.

How much does it matter?

The A11y-CUA study by researchers at UC Berkeley and the University of Michigan, presented at CHI 2026, had a current computer-use agent complete 60 everyday tasks while changing the mode of interaction. In the default mode, which relies on sight and mouse, it completed 78.3 % of tasks. Using only the keyboard, the same path screen readers and the accessibility tree use, it managed 41.67 %, and with a screen magnifier 28.3 %.

The success rate almost halves as soon as the agent has to rely on the structure. Clean markup spares it the guesswork via screenshot, and that often decides whether the task succeeds.

Test, don't guess

1. Run through with real agents. The agent in Perplexity Comet is free. Let it walk your most important paths, such as search, cart, booking and inquiry. Test several agents, for example Claude in Chrome and ChatGPT as well, because they behave differently.
2. Set up a fixed test case. Build the most important tasks into a repeatable, automated test. That way you notice when a release or a new feature breaks a path for agents.
3. Use Lighthouse “Agentic Browsing”. The category is available from Chrome 150. It checks the accessibility tree for agents, WebMCP, layout stability, llms.txt and, since Lighthouse 13.5, a file for Agent Resource Discovery. It is experimental and shows the share of passed audits instead of a score from 0 to 100.
4. Look at the accessibility tree. In Chrome DevTools under Elements → Accessibility you see the structure the agent encounters and spot missing names or roles.

Common mistakes

• The team optimizes for one AI browser that is discontinued or rebuilt shortly afterwards.
• The cookie banner sits as an overlay over the checkout.
• Variant selection such as size or color is a custom widget without roles.
• Form errors are shown only as a color on the field.
• The firewall blocks data center IPs across the board and also hits the cloud agents of real customers.
• After a relaunch, labels that existed before are missing and nobody checks.

The same effort pays off several times

Almost all of these improvements, semantic HTML, clear labels, content in the HTML and a low CLS score, also make your site better for screen readers and search engines. You already track CLS in PageSpeed Insights and Search Console. You are strengthening foundations that pay off across several channels.

Frequently asked questions