agent-index

A web index and fetch cache built for AI agents. It turns pages built for eyes into clean, cited, deduplicated Markdown, shares one cache across every agent, and serves it all as MCP tools.

It runs on one machine: one SQLite file, no services to stand up.

What it fixes

The issue numbers come from the Agent Friction Matrix, which ranks 29 agent inefficiencies by priority and difficulty. The roadmap shows where all 29 stand. The table below lists the ones the index fixes on its own.

#	Friction	Fix here
1	HTML noise, ads and banners eat tokens	`extract.clean` returns main content as Markdown plus title, author, dates and language. About 95% fewer tokens on a typical page.
4	Context fills with repeated tool output	Pass a `reader` id and a repeat read returns only what changed, or nothing.
5	JavaScript pages need a headless browser	Empty app shells are detected and rendered once in headless Chromium. Every later read uses the snapshot.
2	Search ranks for clicks, not answers	Hybrid search over passages: keyword (BM25) and vector matching merged by rank fusion, then weighted for freshness and spam.
12	Over-exploration when an answer already exists	`search` over everything already indexed, before any fetch.
13	No freshness, canonical or spam signal	Every record carries fetched-at, last-changed, canonical URL, duplicate-of, and a spam score.
14	Agents duplicate each other's fetches	One shared cache with ETag revalidation, plus `Fetcher.memo` for any expensive tool call.
15	Memory between sessions is ad hoc	A portable agent memory file.
19	Long tasks have no checkpoints	`checkpoint_save` / `checkpoint_load`, atomic writes.
25	Pagination hides data	`next_url` from `rel=next` or "Next" links, and `crawl` follows them.

The rest need websites, payment networks or other agents to take part. For those, agent-index is the agent side and the AgentFriendly middleware is the site side, both working over real HTTP, so any Python site can adopt them today:

#	Friction	What works
3, 8, 16	Borrowed sessions, anonymous agents, coarse scopes	Agent identity: the person signs one-site, one-hour grants (`agent-index grant`); the agent uses the narrowest one per request with RFC 9421 signatures; the person can revoke at the site (`agent-index revoke`)
9	Bot detection blocks legitimate agents	Web Bot Auth: every request signed, key directory from `agent-index key-directory`, verified by the middleware
7, 17, 24, 28	Spending limits, human-only checkout, micropayments, receipts	Agent payments: mandates checked by the wallet and the merchant, HTTP 402 paid within the approval policy, durable hash-chained ledger, receipts, refunds and receipt audits
6, 11, 22, 23	Pixel clicking, no action APIs, discovery, multi-step flows	Action manifests: sites hand the middleware one function per action; agents use `call_action` and `run_flow`; `find_action` searches every manifest the index has seen
26	Errors are red text	Agent errors: one code and one `retry` hint for every failure, including ordinary sites' error pages
21	robots.txt has no terms or prices	Agent policy file, followed by `fetch` and the crawler and enforced by the middleware
18	All-or-nothing approvals	Graded approvals on every spend, send and delete
10	Weak self-verification	Claim checking against the cited source (`verify.py`)
27, 29	Trust and negotiation	Source accuracy, signed attestations, and price negotiation over HTTP

See it work

agent-index demo

Runs a whole errand offline against in-memory sites (a news site, a shop, a mirror and the agent's key directory), over the same protocols as the real web: crawl and index politely, search with citations, share a cached page between agents, catch a copied page with a changed figure, get a structured error from an ordinary 404 page, read a site's terms and get into a signed-agents-only archive with Web Bot Auth, pay 25 cents for a premium page from a mandate, negotiate a price, buy through the shop's actions under a narrow grant with one approval, get a refund, audit the receipts, and revoke the grant. Each step names the friction it removes.

Install

pip install -e '.[render,dev]'

Rendering uses Playwright with any Chromium already on the machine (AGENT_INDEX_CHROMIUM to point at one), or run playwright install chromium.

Search modes

search defaults to hybrid. The default embedder needs no downloads and matches spelling and word-form variants. For matching by meaning, install the semantic extra and set AGENT_INDEX_EMBEDDER=fastembed:BAAI/bge-small-en-v1.5; existing passages are re-embedded automatically the next time the index opens.

Use it from an agent

Add the MCP server to your client:

{
  "mcpServers": {
    "agent-index": { "command": "agent-index", "args": ["serve"], "env": { "AGENT_INDEX_READER": "my-agent" } }
  }
}

Tools: search, fetch, crawl, index_site, index_stats, site_policy, discover_actions, find_action, call_action, run_flow, negotiate, receipts, request_refund, wallet_add, wallet_list, wallet_forget, check_claim, corroborate, report_source, source_accuracy, approval_check, whoami, attest, record_attestation, reputation, memory_add, memory_query, memory_forget, checkpoint_save, checkpoint_load.

Acting for a person

agent-index whoami                     # the agent's key id
agent-index person-key                 # the person's key (keep it on the person's device)
agent-index grant shop.example --scope cart --purpose "Buy printer paper" --max-total 4000 --install
agent-index wallet list
agent-index revoke shop.example <grant id>

The person's approval policy lives in approvals.json in the data folder (default: spending up to $20 goes ahead, more asks). Set AGENT_INDEX_SIGNATURE_AGENT to the URL hosting agent-index key-directory to sign every request with Web Bot Auth.

Data lives in ~/.agent-index (set AGENT_INDEX_HOME to change it). Point several agents at the same folder and they share the cache, index, memory and trust log. The folder also holds this agent's signing key (agent.key, owner-only permissions), used by attest and shown by whoami.

Safety

The fetcher only fetches http and https URLs on public addresses, and checks every redirect. Loopback, private, link-local and cloud metadata addresses are refused, so an agent can't be steered into reading internal services. Set AGENT_INDEX_ALLOW_PRIVATE=1 to index an intranet on purpose.
Only text is stored: HTML, Markdown, plain text and JSON (as a code block). Other types, such as PDFs and images, return an error.
The crawler waits the longest of its default delay, the site's robots.txt Crawl-delay, and the site policy's rate_limit_per_minute.

Use it from the command line

agent-index fetch https://pypi.org/project/trafilatura/
agent-index search "extract main text as markdown"
agent-index crawl https://example.com/blog --max-pages 5
agent-index index-site https://example.com --max-pages 100   # robots.txt, sitemap, agent policy, 1 request/second
agent-index stats

How a fetch works

Normalize the URL (lower-case host, drop tracking parameters and fragments).
If the cached copy is younger than max_age, return it. No network.
Otherwise revalidate with If-None-Match / If-Modified-Since. A 304 costs almost nothing.
On new content, extract the clean view. If the page is an empty JavaScript shell, render it once.
Compute provenance: content hash, SimHash for near-duplicates, canonical URL, last-changed date, spam score.
Store the page and a version, and re-index its passages for search.
If the reader has seen an older version, return a diff instead of the whole page.
If the site is down, serve the last good copy marked stale.

Make your own site agent-friendly

Wrap any ASGI app (Starlette, FastAPI, Django ASGI) and it adopts the specs in one step:

from agent_index.site import AgentFriendly

app = AgentFriendly(app, host="example.com",
                    policy=POLICY,              # served at /.well-known/agent-policy.json and enforced
                    actions=ACTIONS,            # served at /.well-known/agent-actions.json
                    handlers={"checkout": fn},  # run those actions: input checked, grant and scope verified
                    trusted_principals=KEYS,    # people whose grants you accept
                    merchant=Merchant("example.com", KEYS, state=SiteState("agents.db")),  # 402s, ledger, refunds
                    bot_keys=DirectoryResolver(fetch=httpsig.public_fetch),  # recognise Web Bot Auth agents
                    negotiator=limits_for_item) # haggle within your limits

Clients that ask for text/markdown get the agent view generated from your own HTML, with front matter and an ETag. Agents acting for a person reach your app with scope["agent"] set to who they are, who they act for, and why; agents that only sign get scope["signed_agent"]. Your app's own error pages reach agents as structured errors. agent-index's fetcher prefers a site's own agent view when one is offered.

Website

site/ builds guavaindex.com into one Cloudflare Worker. Every page is served as HTML to people and as Markdown with front matter to agents (Accept: text/markdown or a .md path), with ETags, an llms.txt and an agent policy file.

pip install -e '.[site]'
python site/build.py && node site/check.mjs           # build and smoke-test
npx wrangler deploy --config site/wrangler.toml       # deploy

Every push to main deploys automatically: the guavaindex Worker is connected to this repo through Cloudflare Workers Builds (build pip install markdown && python3 site/build.py, deploy npx wrangler deploy --config site/wrangler.toml). The GitHub workflow only builds and smoke-tests the site.

Tests

pytest

The render test drives real headless Chromium against a local page and is skipped when no browser is available.

Specs

Agent view: how a site can serve the clean view itself.
Agent memory: the portable memory file format.
Handoff and checkpoints: structured task state for resuming and for passing work between agents.
Agent identity, payments, actions, errors, site policy, approvals, trust and negotiation.
Roadmap: status of all 29 issues.