AIFeed lets any site publish a signed, verifiable declaration of what AI agents may do with its content — training, indexing, summarization, citation — and lets agents prove, on every fetch, that the permission is real, current, and authorized. This guide covers the problem, the protocol, the security layers, both content profiles, the measured savings, revocation, and how to adopt it today.
robots.txt and similar files are unsigned text. Anyone can edit them,
nothing binds them to a domain, and there is no way to tell an honest declaration from a
forged one. Publishers cannot prove what they allowed; AI builders cannot prove what they
were allowed to use; both sides pay for blind re-crawling.
An unsigned file carries no cryptographic evidence of who wrote it or whether it was tampered with in transit.
A permission granted last year may have been revoked. Without a revocation check, agents act on stale policy.
Agents re-download full HTML — menus, ads, scripts — for pages that did not change, wasting bytes on both sides.
The publisher runs one tool. The rest is standard web infrastructure: a well-known path, a DNS record, and signatures anyone can verify offline.
Ed25519 key pair. The private key never leaves the server.
Identity, per-use permissions, crawl limits, contact, and license.
Ed25519 over JCS canonical JSON — deterministic bytes, verifiable offline.
At /.well-known/ai.json, with signature sidecars and content indexes.
A _aifeed TXT record pins the key to the domain, preventing manifest forgery.
Agents verify TLS → domain → signature → anchor, then re-check revocation on every use.
Ed25519 signatures over JCS-canonical JSON. Canonicalization is strict: duplicate keys, floats, and oversized integers are rejected, so verification is deterministic across languages.
TLS plus the _aifeed DNS anchor. A stolen manifest from another domain cannot be replayed because the key is pinned to the origin's own DNS.
Status lives in a multi-signature registry. Agents re-check it; offline bundles carry the evidence so audits work without network access.
Content profiles may only restrict the manifest, never widen it. Crawl limits and rate policy keep enforcement bounded and explainable.
text/aifeed+markdown, extension .aifeed.mdtext/mako+markdown, the external MAKO trust profileMeasured in a local HTTP harness with the same edge policy that ships as nginx/Caddy templates. Scenario S3: training crawlers blocked, non-compliant crawlers throttled, compliant clients served signed content with delta indexes.
| Side | Metric | Result | Label |
|---|---|---|---|
| Web owner / host | Egress bytes saved | −55.19% | measured-simulation |
| Web owner / host | Origin CPU saved | −56.23% | measured-simulation |
| Web owner / host | Peak connections reduced | −88.24% | measured-simulation |
| AI side | Bytes received (all profiles) | −54.84% | measured-simulation |
| AI side | Bytes received (compliant clients) | −72.93% | measured-simulation |
| AI side | Unchanged pages skipped | 14 / 18 | measured-simulation |
| AI side | Verification cost | 0.70 ms/page | measured-simulation |
| Scale model | Projection per 1,000 tenants | see benchmark | model |
Honest baseline: with no enforcement (S0) nothing is blocked or throttled and the origin serves every byte to every profile — that is the cost this protocol removes. Figures are simulation, not a production CDN; the 30-day pilot kit exists to replace them with live data.
Open the interactive enforcement benchmark Animated process flow, per-scenario tables, and two-sided savings with embedded measured data./.well-known/ai.json, signatures, content index._aifeed TXT pins the key to the domain.Usage keys cover the real decisions agents make:
search, retrieval, input — discovery and groundingquote, summarize, reproduce, translate — derived outputmodify, embed, commercial_use — higher-risk usestraining — the decisive toggle for model trainingContent profiles can only narrow these grants. Crawl limits bound request rates; attribution settings travel with the content; and a denial is as machine-readable as an allowance.
Status documents require multiple signatures, so a single compromised key cannot revoke or restore a site.
active, suspended, revoked — with signed timestamps and reasons.
Manifest, signatures, revocation, and content hashes in one auditable archive for compliance reviews.
No. It is a technical signal: cryptographically verifiable permissions that complement law and contracts. It makes the declared policy auditable; it does not replace legal advice or enforcement.
Those are unsigned conventions. AIFeed binds permissions to a domain with Ed25519, JCS canonical bytes, a DNS anchor, and a revocation registry — and it serves the content itself in lean, agent-ready formats.
The protocol declares policy; enforcement happens at the edge (the shipped nginx/Caddy templates and adapter code). That gives compliant clients faster, cheaper access while non-compliant behavior is blocked or throttled with an honest baseline you can measure.
Publish a new key, update the DNS anchor, and use the revocation registry to invalidate the old key. Rotation and revocation are first-class operations, not afterthoughts.
The standard is open (CC BY 4.0) and the reference implementation is MIT, zero-dependency, and works on static sites, nginx, Caddy, Apache, Node, Next.js, PHP, Python, Go, and WordPress.
Release 1.0.0-draft: specs are not frozen yet, cross-language conformance is at 75 vectors (JS and Python), and an external cryptographic review is still pending. Wire versions in use: manifest 0.1/0.2, AIFeed Markdown 1.0, MAKO 0.2.
npx aifeed site build ./public generates the manifest, signed content, and indexes for Hugo, Jekyll, Astro, Next export, or plain HTML.
The plugin publishes the signed declaration and serves AIFeed Markdown + MAKO dual-stack, plus /llms.txt.
Adapters for nginx, Caddy, Apache, Node/Express, Next.js, PHP, Python ASGI, and Go — plus a GitHub Action.
npm install @aifeed/verify — fetch, verify, decide, and select pages with a zero-dependency SDK.