Most sites treat all machine traffic the same: block it, or serve everyone HTML. ADN asks a different question — what is this machine trying to do? — and serves the response shape that job needs.
The nine intent classes
Classification uses behaviour signals, user-agent heuristics, and verified bot categories — not credentials alone.
Bulk training crawlers
Who: GPTBot, ClaudeBot, CCBot, Bytespider
What they want: Entire site content for model training
Response shape: Bulk corpus export + delta feed
GET /llms-full.txt HTTP/1.1
User-Agent: GPTBot/1.0→ 200 OK, full corpus with navigation index
Live user-triggered fetches
Who: ChatGPT-User, Perplexity-User, Claude-User
What they want: A specific page, right now — a user asked an AI to check your pricing
Response shape: Fast Markdown or DocLang (latency-critical, never rate-limited)
GET /pricing.dclg HTTP/1.1
User-Agent: ChatGPT-User/1.0
Accept: application/vnd.doclang.document+xml→ 200 OK, structured DocLang in <200ms
Alternatively, request Markdown:
GET /pricing.md HTTP/1.1
User-Agent: ChatGPT-User/1.0
Accept: text/markdown→ 200 OK, clean Markdown in <200ms
Developer and IDE agents
Who: Cursor, Claude Code, GitHub Copilot
What they want: Structured access to docs and APIs for code context
Response shape: MCP tools + Markdown + llms.txt navigation
POST /mcp/v1 HTTP/1.1
Content-Type: application/json
{"method": "tools/call", "params": {"name": "search_content", "arguments": {"query": "auth"}}}→ Structured search results, not raw HTML
Legacy SEO and social
Who: Googlebot, facebookexternalhit, Slackbot
What they want: Unmodified HTML for indexing and link previews
Response shape: Passthrough untouched (hard safety rule)
This is non-negotiable. Serving Markdown to Googlebot would break SEO.
Five response shapes
| Shape | When | Format |
|---|---|---|
| Passthrough | Legacy SEO/social bots | Unmodified origin HTML |
| Markdown | Live fetches, IDE agents | Clean text/markdown |
| DocLang | Live fetches, structured parsers | XML (.dclg) with semantic elements |
| Bulk corpus | Training crawlers | llms-full.txt + delta feed |
| MCP tools | Developer agents | JSON-RPC over HTTP |
| Blocked | Abusive traffic | 403 with machine-readable reason |
The classification pipeline
1. Hard safety rule — Googlebot, OG unfurlers → passthrough immediately
2. Verified bot signals — Cloudflare verified bot category maps to intent class
3. User-agent heuristics — 200+ signatures in the fixture corpus
4. Behaviour scoring — rate, path diversity, robots.txt honouring, 304 respect
5. Declared intent — optional X-ADP-Intent header (gateway may override)
Anonymous curl and Scrapy requests get graded by behaviour, not blocked.
Graceful degradation
When policy limits apply, ADN degrades in order — never walls without explanation:
1. Full content
2. Summary + link to full version
3. Sample + machine-readable terms
4. 429 Too Many Requests with retry-after
5. 403 Forbidden with upgrade instructions
Every degradation response includes a path to better access — register your agent, accept terms, upgrade identity tier.
See it on your site
Run the free efficiency scan at https://www.agent-delivery.network to see which intent classes are hitting your site today, what response shape they'd get, and how many tokens you're wasting on HTML.
Ready to deploy? https://app.agent-delivery.network