Agent policy file

Status: draft 0.1 (agent-policy/0.1). Covers friction #21 (robots.txt says yes or no, but nothing about terms or pricing) and publishes prices for #24. Reference implementation: agent_index/sitepolicy.py.

robots.txt answers "may a crawler fetch this path". Agents also need to know what they may do with the content, what it costs, and how to credit it. A site publishes /.well-known/agent-policy.json:

{
  "version": "agent-policy/0.1",
  "default": {
    "allow": ["read", "index", "summarize", "quote"],
    "attribution": "Link to the article",
    "rate_limit_per_minute": 60
  },
  "paths": [
    {"match": "/premium/*", "allow": ["read", "summarize"], "prices": {"read": {"amount": 25, "currency": "USD"}}},
    {"match": "/account/*", "allow": ["read", "act"], "requires_identity": true},
    {"match": "/archive/*", "allow": ["read"], "requires_signature": true}
  ]
}

Uses

Use	Meaning
`read`	Fetch and read to answer the person's question.
`index`	Store and search later (what agent-index does).
`summarize`	Restate in the agent's own words.
`quote`	Reproduce passages verbatim, with attribution.
`train`	Use for model training.
`act`	Use the site's actions on the person's behalf.

Rules

The first paths entry whose match (a glob on the URL path) fits wins, merged over default. With no match, default applies.
A use not listed in allow is not allowed.
prices maps a use to a price per request in minor units. Paying works as in payments.
requires_identity: true means the use needs a person's grant (agent identity).
requires_signature: true admits agents that sign their requests (Web Bot Auth or a grant) and turns away anonymous ones with auth_required.
rate_limit_per_minute is enforced per agent key (per address for anonymous callers) with 429 rate_limited and Retry-After.
robots.txt still applies. This file adds terms; it never grants access robots.txt denies.

Enforcement

On the site, AgentFriendly(policy=...) serves the file and enforces it: uses, identity, signatures, rate limits and prices.

On the agent side, agent-index reads the file (cached for a day, shared across agents) before every fetch:

If read isn't allowed, the fetch is refused with not_allowed and the site is never asked.
If index isn't allowed, the page is cached for reuse but kept out of search.
The result carries terms: the allowed uses, the attribution the site asks for, and the price, so the agent can credit and pay correctly.

The crawler skips paths that don't allow index and paces itself by the rate limit. The site_policy tool answers questions about any URL and use.