The open standard for agent readiness

§ the guide

Make your site readable by AI agents

A practical guide to making your website work for the agents that now read it, based on real agent runs and a controlled experiment.

Developed by ora.ai and Vercel · v1.0 · last updated August 2026

AI agents visit websites on users’ behalf to answer questions, integrate APIs, and complete tasks. Together with the ora.ai research lab, we measured which site features help them succeed.

In the studies, two changes caused agents to fail: hiding answers behind JavaScript and blocking access. Other practices shaped where agents went and which pages their answers were built from. This guide follows the order in which agents encounter a site: find, read, then act. Each stage depends on the previous one, and unlinked files are unlikely to be discovered.

All recommendations in this guide share one prerequisite: the agent must be able to reach the page and find the answer in the returned HTML.

§ find

Be reachable and citable

Agents can cite only pages they can discover and fetch. They may receive or recall a URL, follow a link, or find a page through a crawl-built search index. Pages blocked from crawlers do not appear in those indexes at all. Blocking can also stop an agent from fetching a page while answering a user.

To make information citable, publish it on one canonical, current first-party docs page. The number above is why: your docs are the page agents build their answers from.

That covers what to publish. The crawl policy covers who may read it. Write one rule for all agents instead of an allowlist of named crawlers:

# Apply the default policy to current and future agents.
User-agent: *
# Block low-value public paths only. Do not list secret paths.
Disallow: /search/

Sitemap: https://example.com/sitemap.xml

The wildcard still leaves you one deliberate choice. The model providers run a separate crawler for each job:

• Training crawlers (GPTBot, ClaudeBot)
• Search-index crawlers (OAI-SearchBot, Claude-SearchBot)
• User-initiated fetchers (ChatGPT-User, Claude-User)

Each obeys its own robots.txt rule, so you can block a training crawler and opt out of training without dropping out of search indexes or agent answers. Two catches: a more specific group such as User-agent: GPTBot overrides your * group, so audit existing rules; and user-initiated fetchers may not consult robots.txt at all.

Then fetch a page with an agent user-agent string to inspect the response:

curl -iL -A 'Claude-User/1.0' https://example.com/your-page

The request should end with a 200 response; intermediate 301 or 308 redirects are acceptable. The returned HTML should contain the answer. A 403, empty app shell, or JavaScript-only answer prevents fetch-only agents from reading the page.

The curl does not prove much either way: bot blockers weigh where a request comes from more than its user-agent string, so your laptop may pass where the real agent is blocked, or the reverse. The dependable test is your server logs. OpenAI and Anthropic publish their crawlers’ IP ranges (separate per-bot lists from OpenAI, one combined list from Anthropic). Find those requests and confirm they return 200.

llms.txt belongs in the reading section because an agent can request it only after discovering and reaching the site.

§ read

Put the answer where agents read

After reaching a page, an agent must be able to extract the answer from the returned content. In the studies, agents used the same homepage and docs pages as human visitors. The agent-specific files (llms.txt, markdown mirrors, JSON-LD) help agents find, fetch, and use those same answers.

If your site is client-rendered, getting answers into the initial HTML is a real migration (SSR, prerendering, or a hybrid), but it is the prerequisite for agents that fetch rather than run JavaScript. The machinery beyond that (content negotiation, .md endpoints, agent detection, frontmatter) did not change whether agents succeeded in the studies. Links also matter: agents rarely found pages or files that were not linked from content they had already reached.

Agents read the same pages people do

Where agents go on your site, and what lets them pull the answer once there:

Link the files that describe your site

STUDIES Agents typically opened these resources after visiting the homepage. Depending on the file type, 86–100% of visits came through links rather than guessed paths. Link each resource from content agents already visited, such as the footer, page metadata like <link rel="alternate"> in the head, or a resources section.

When shipping an llms.txt, make it a lean, described index:

# Acme
> Acme ships a payments API for developers.

## Docs
- [Quickstart](https://example.com/docs/quickstart): first charge in one request
- [Authentication](https://example.com/docs/auth): API keys, scopes, rotation

## API
- [OpenAPI](https://example.com/openapi.json): full request and response contract

Inject JSON-LD at strategic places, starting with the homepage, so an agent fetching the raw HTML gets your key facts and routes in one machine-readable block.

Agents request markdown when offered, so make it reachable

STUDIES Agents requested markdown on about 65% of web fetches. When they specified a format, they chose markdown 96% of the time. Format did not determine task success in the studies, so ensure the HTML works before adding a .md mirror.

<!-- A mirror nothing points to is a file nothing fetches. -->
<link rel="alternate" type="text/markdown" href="/docs/quickstart.md">

The same relation can also be sent as an HTTP Link: header, so a CDN can advertise the mirror without touching page heads.

Return accurate HTTP status codes

Return 404 for a missing path or a valid redirect to a replacement. Do not return 200 for a missing page, even if the body suggests alternatives; agents cannot distinguish that response from a live page. In one observed case, a single-page app returned 200 for every path, making dead URLs appear valid.

The same applies when you throttle: answer with a 429 and a Retry-After header (RFC 6585) rather than a silent challenge page. Agents and crawlers treat 429 as a back-off signal and will return.

By the end of this layer the agent has found you and read you. Whether it can act on what it reads is the final layer: usability.

§ act

Let agents act, not just read

Reading documentation does not let an agent complete a task. To act, an agent needs an operable interface such as an API, MCP server, SDK, or CLI, plus authentication it can complete. These interfaces still depend on discoverable and readable documentation.

That is the full journey: find, read, act. The last step happens only if the first two did.

§ beyond this guide

Beyond this guide

This guide covers the main practices. The ora scanner covers additional agent-readiness checks through its public methodology, per-check breakdown, and leaderboard. The protocol changes as new agent behavior is measured, so checks and weights may change between guide versions.

Scan any domain at ora.ai, or from the CLI:

npx @ora-ai/ax audit https://example.com

The same scan is available from the API:

curl -X POST https://ora.ai/api/scan \
  -H 'Content-Type: application/json' \
  -d '{"url": "example.com"}'

{
  "score": 72,
  "grade": "B",
  "layers": [
    { "id": "accessibility", "score": 43, "maxScore": 62, "checks": [
      { "id": "content-no-js", "status": "pass", "score": 3, "maxScore": 3 },
      { "id": "markdown-link-alternate", "status": "fail", "score": 0, "maxScore": 1,
        "recommendation": "Advertise a markdown twin with
          <link rel=\"alternate\" type=\"text/markdown\"> …" }
    ] },
    …
  ]
}

Full documentation for the scanner, CLI, and API is at ora.ai/docs.

§ references

• What agents actually reach: the field report behind the trace numbers in this guide, drawing on part of the data used for this spec.
• The state of agent readiness: the scanner report behind the reachability figure.
• ora research: the ongoing research program behind the protocol.
• The agent readiness dataset: the runs and probes behind every [STUDIES] number, with a per-claim breakdown to reproduce them.

§ changelog

v1.0 · August 2026 · first public version.