---
title: "agent-index docs"
canonical: https://guavaindex.com/docs
modified: 2026-11-10T08:36:22Z
lang: en
---

# agent-index

A web index and fetch cache built for AI agents. It turns pages built for eyes into clean, cited, deduplicated Markdown, shares one cache across every agent, and serves it all as MCP tools.

It runs on one machine: one SQLite file, no services to stand up.

## What it fixes

The issue numbers come from the Agent Friction Matrix, which ranks 29 agent inefficiencies by priority and difficulty. [The roadmap](/specs/roadmap) shows where all 29 stand. The table below lists the ones the index fixes on its own.

| # | Friction | Fix here |
|---|---|---|
| 1 | HTML noise, ads and banners eat tokens | `extract.clean` returns main content as Markdown plus title, author, dates and language. About 95% fewer tokens on a typical page. |
| 4 | Context fills with repeated tool output | Pass a `reader` id and a repeat read returns only what changed, or nothing. |
| 5 | JavaScript pages need a headless browser | Empty app shells are detected and rendered once in headless Chromium. Every later read uses the snapshot. |
| 2 | Search ranks for clicks, not answers | Hybrid search over passages: keyword (BM25) and vector matching merged by rank fusion, then weighted for freshness and spam. |
| 12 | Over-exploration when an answer already exists | `search` over everything already indexed, before any fetch. |
| 13 | No freshness, canonical or spam signal | Every record carries fetched-at, last-changed, canonical URL, duplicate-of, and a spam score. |
| 14 | Agents duplicate each other's fetches | One shared cache with ETag revalidation, plus `Fetcher.memo` for any expensive tool call. |
| 15 | Memory between sessions is ad hoc | A portable [agent memory file](/specs/agent-memory). |
| 19 | Long tasks have no checkpoints | `checkpoint_save` / `checkpoint_load`, atomic writes. |
| 25 | Pagination hides data | `next_url` from `rel=next` or "Next" links, and `crawl` follows them. |

The rest need websites, payment networks or other agents to take part. For those, agent-index is the agent side and the `AgentFriendly` middleware is the site side, both working over real HTTP, so any Python site can adopt them today:

| # | Friction | What works |
|---|---|---|
| 3, 8, 16 | Borrowed sessions, anonymous agents, coarse scopes | [Agent identity](/specs/agent-identity): the person signs one-site, one-hour grants (`agent-index grant`); the agent uses the narrowest one per request with RFC 9421 signatures; the person can revoke at the site (`agent-index revoke`) |
| 9 | Bot detection blocks legitimate agents | Web Bot Auth: every request signed, key directory from `agent-index key-directory`, verified by the middleware |
| 7, 17, 24, 28 | Spending limits, human-only checkout, micropayments, receipts | [Agent payments](/specs/agent-payments): mandates checked by the wallet and the merchant, HTTP 402 paid within the approval policy, durable hash-chained ledger, receipts, refunds and receipt audits |
| 6, 11, 22, 23 | Pixel clicking, no action APIs, discovery, multi-step flows | [Action manifests](/specs/agent-actions): sites hand the middleware one function per action; agents use `call_action` and `run_flow`; `find_action` searches every manifest the index has seen |
| 26 | Errors are red text | [Agent errors](/specs/agent-errors): one code and one `retry` hint for every failure, including ordinary sites' error pages |
| 21 | robots.txt has no terms or prices | [Agent policy file](/specs/agent-policy), followed by `fetch` and the crawler and enforced by the middleware |
| 18 | All-or-nothing approvals | [Graded approvals](/specs/approvals) on every spend, send and delete |
| 10 | Weak self-verification | Claim checking against the cited source (`verify.py`) |
| 27, 29 | Trust and negotiation | [Source accuracy, signed attestations, and price negotiation over HTTP](/specs/trust-and-negotiation) |

## See it work

```bash
agent-index demo
```

Runs a whole errand offline against in-memory sites (a news site, a shop, a mirror and the agent's key directory), over the same protocols as the real web: crawl and index politely, search with citations, share a cached page between agents, catch a copied page with a changed figure, get a structured error from an ordinary 404 page, read a site's terms and get into a signed-agents-only archive with Web Bot Auth, pay 25 cents for a premium page from a mandate, negotiate a price, buy through the shop's actions under a narrow grant with one approval, get a refund, audit the receipts, and revoke the grant. Each step names the friction it removes.

## Install

```bash
pip install -e '.[render,dev]'
```

Rendering uses Playwright with any Chromium already on the machine (`AGENT_INDEX_CHROMIUM` to point at one), or run `playwright install chromium`.

## Search modes

`search` defaults to hybrid. The default embedder needs no downloads and matches spelling and word-form variants. For matching by meaning, install the `semantic` extra and set `AGENT_INDEX_EMBEDDER=fastembed:BAAI/bge-small-en-v1.5`; existing passages are re-embedded automatically the next time the index opens.

## Use it from an agent

Add the MCP server to your client:

```json
{
  "mcpServers": {
    "agent-index": { "command": "agent-index", "args": ["serve"], "env": { "AGENT_INDEX_READER": "my-agent" } }
  }
}
```

Tools: `search`, `fetch`, `crawl`, `index_site`, `index_stats`, `site_policy`, `discover_actions`, `find_action`, `call_action`, `run_flow`, `negotiate`, `receipts`, `request_refund`, `wallet_add`, `wallet_list`, `wallet_forget`, `check_claim`, `corroborate`, `report_source`, `source_accuracy`, `approval_check`, `whoami`, `attest`, `record_attestation`, `reputation`, `memory_add`, `memory_query`, `memory_forget`, `checkpoint_save`, `checkpoint_load`.

### Acting for a person

```bash
agent-index whoami                     # the agent's key id
agent-index person-key                 # the person's key (keep it on the person's device)
agent-index grant shop.example --scope cart --purpose "Buy printer paper" --max-total 4000 --install
agent-index wallet list
agent-index revoke shop.example <grant id>
```

The person's approval policy lives in `approvals.json` in the data folder (default: spending up to $20 goes ahead, more asks). Set `AGENT_INDEX_SIGNATURE_AGENT` to the URL hosting `agent-index key-directory` to sign every request with Web Bot Auth.

Data lives in `~/.agent-index` (set `AGENT_INDEX_HOME` to change it). Point several agents at the same folder and they share the cache, index, memory and trust log. The folder also holds this agent's signing key (`agent.key`, owner-only permissions), used by `attest` and shown by `whoami`.

### Safety

- The fetcher only fetches `http` and `https` URLs on public addresses, and checks every redirect. Loopback, private, link-local and cloud metadata addresses are refused, so an agent can't be steered into reading internal services. Set `AGENT_INDEX_ALLOW_PRIVATE=1` to index an intranet on purpose.
- Only text is stored: HTML, Markdown, plain text and JSON (as a code block). Other types, such as PDFs and images, return an error.
- The crawler waits the longest of its default delay, the site's robots.txt `Crawl-delay`, and the site policy's `rate_limit_per_minute`.

## Use it from the command line

```bash
agent-index fetch https://pypi.org/project/trafilatura/
agent-index search "extract main text as markdown"
agent-index crawl https://example.com/blog --max-pages 5
agent-index index-site https://example.com --max-pages 100   # robots.txt, sitemap, agent policy, 1 request/second
agent-index stats
```

## How a fetch works

1. Normalize the URL (lower-case host, drop tracking parameters and fragments).
2. If the cached copy is younger than `max_age`, return it. No network.
3. Otherwise revalidate with `If-None-Match` / `If-Modified-Since`. A 304 costs almost nothing.
4. On new content, extract the clean view. If the page is an empty JavaScript shell, render it once.
5. Compute provenance: content hash, SimHash for near-duplicates, canonical URL, last-changed date, spam score.
6. Store the page and a version, and re-index its passages for search.
7. If the reader has seen an older version, return a diff instead of the whole page.
8. If the site is down, serve the last good copy marked `stale`.

## Make your own site agent-friendly

Wrap any ASGI app (Starlette, FastAPI, Django ASGI) and it adopts the specs in one step:

```python
from agent_index.site import AgentFriendly

app = AgentFriendly(app, host="example.com",
                    policy=POLICY,              # served at /.well-known/agent-policy.json and enforced
                    actions=ACTIONS,            # served at /.well-known/agent-actions.json
                    handlers={"checkout": fn},  # run those actions: input checked, grant and scope verified
                    trusted_principals=KEYS,    # people whose grants you accept
                    merchant=Merchant("example.com", KEYS, state=SiteState("agents.db")),  # 402s, ledger, refunds
                    bot_keys=DirectoryResolver(fetch=httpsig.public_fetch),  # recognise Web Bot Auth agents
                    negotiator=limits_for_item) # haggle within your limits
```

Clients that ask for `text/markdown` get the agent view generated from your own HTML, with front matter and an ETag. Agents acting for a person reach your app with `scope["agent"]` set to who they are, who they act for, and why; agents that only sign get `scope["signed_agent"]`. Your app's own error pages reach agents as structured errors. agent-index's fetcher prefers a site's own agent view when one is offered.

## Website

`site/` builds [guavaindex.com](https://guavaindex.com) into one Cloudflare Worker. Every page is served as HTML to people and as Markdown with front matter to agents (`Accept: text/markdown` or a `.md` path), with ETags, an `llms.txt` and an agent policy file.

```bash
pip install -e '.[site]'
python site/build.py && node site/check.mjs           # build and smoke-test
npx wrangler deploy --config site/wrangler.toml       # deploy
```

Every push to `main` deploys automatically: the `guavaindex` Worker is connected to this repo through Cloudflare Workers Builds (build `pip install markdown && python3 site/build.py`, deploy `npx wrangler deploy --config site/wrangler.toml`). The GitHub workflow only builds and smoke-tests the site.

## Tests

```bash
pytest
```

The render test drives real headless Chromium against a local page and is skipped when no browser is available.

## Specs

- [Agent view](/specs/agent-view): how a site can serve the clean view itself.
- [Agent memory](/specs/agent-memory): the portable memory file format.
- [Handoff and checkpoints](/specs/handoff): structured task state for resuming and for passing work between agents.
- [Agent identity](/specs/agent-identity), [payments](/specs/agent-payments), [actions](/specs/agent-actions), [errors](/specs/agent-errors), [site policy](/specs/agent-policy), [approvals](/specs/approvals), [trust and negotiation](/specs/trust-and-negotiation).
- [Roadmap](/specs/roadmap): status of all 29 issues.