# robots.txt — aicontentdrop.com # # Content Signals (https://contentsignals.org) declare how this site's content # may be used. We allow all three for crawlers that can route a reader back to # us: we WANT to be read, indexed, cited, and learned from. Our content is the # funnel. # # search = yes — build a search index and link back to us # ai-input = yes — ground AI answers in our content (RAG / live retrieval) # ai-train = yes — train or fine-tune generative models on our content # # The exception is TRAINING-ONLY crawlers (CCBot, Bytespider below): they feed # corpora with no citation or referral path, so no reader ever arrives from # them. They get a plain Disallow and no Content-Signal line, by the same # take-without-sending-back test the edge-blocked list at the bottom of this # file already applies. No group in this file refuses search, AI input, or AI # training for the site as a whole: being read, cited and learned from is the # funnel, and every signal line below says yes to all three. # # Agent-friendly entry points: # /llms.txt — curated overview of the site # /llms-full.txt — full text of every public article # /.md — markdown twin of any public page # Accept: text/markdown — markdown of any page via content negotiation # # Shared agent runs (/make/share/) are disallowed in every group below, # and the page carries X-Robots-Tag: noindex from the server as well. The token # is the authorisation and the owner can revoke it at any moment — what an index # copy would add is a snapshot that outlives the revocation, which is the one # thing a revocable link must not have. User-agent: * Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /api/ Disallow: /auth/callback Disallow: /oauth/ Allow: /mcp$ Disallow: /mcp/docs Disallow: /mcp/sse Disallow: /mcp/docs/sse Disallow: /a2a Disallow: /stitch-* Disallow: /engine Disallow: /make Disallow: /make/share/ # AI Crawlers — explicitly allow public content User-agent: GPTBot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /api/ Disallow: /auth/callback Disallow: /oauth/ Allow: /mcp$ Disallow: /mcp/docs Disallow: /mcp/sse Disallow: /mcp/docs/sse Disallow: /a2a Disallow: /stitch-* Disallow: /engine Disallow: /make Disallow: /make/share/ User-agent: ChatGPT-User Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /api/ Disallow: /auth/callback Disallow: /oauth/ Allow: /mcp$ Disallow: /mcp/docs Disallow: /mcp/sse Disallow: /mcp/docs/sse Disallow: /a2a Disallow: /stitch-* Disallow: /engine Disallow: /make Disallow: /make/share/ User-agent: Google-Extended Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /api/ Disallow: /auth/callback Disallow: /oauth/ Allow: /mcp$ Disallow: /mcp/docs Disallow: /mcp/sse Disallow: /mcp/docs/sse Disallow: /a2a Disallow: /stitch-* Disallow: /engine Disallow: /make Disallow: /make/share/ User-agent: GoogleOther Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /api/ Disallow: /auth/callback Disallow: /oauth/ Allow: /mcp$ Disallow: /mcp/docs Disallow: /mcp/sse Disallow: /mcp/docs/sse Disallow: /a2a Disallow: /stitch-* Disallow: /engine Disallow: /make Disallow: /make/share/ User-agent: Meta-ExternalAgent Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /api/ Disallow: /auth/callback Disallow: /oauth/ Allow: /mcp$ Disallow: /mcp/docs Disallow: /mcp/sse Disallow: /mcp/docs/sse Disallow: /a2a Disallow: /stitch-* Disallow: /engine Disallow: /make Disallow: /make/share/ User-agent: Meta-ExternalFetcher Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /api/ Disallow: /auth/callback Disallow: /oauth/ Allow: /mcp$ Disallow: /mcp/docs Disallow: /mcp/sse Disallow: /mcp/docs/sse Disallow: /a2a Disallow: /stitch-* Disallow: /engine Disallow: /make Disallow: /make/share/ User-agent: ClaudeBot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /api/ Disallow: /auth/callback Disallow: /oauth/ Allow: /mcp$ Disallow: /mcp/docs Disallow: /mcp/sse Disallow: /mcp/docs/sse Disallow: /a2a Disallow: /stitch-* Disallow: /engine Disallow: /make Disallow: /make/share/ User-agent: PerplexityBot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /api/ Disallow: /auth/callback Disallow: /oauth/ Allow: /mcp$ Disallow: /mcp/docs Disallow: /mcp/sse Disallow: /mcp/docs/sse Disallow: /a2a Disallow: /stitch-* Disallow: /engine Disallow: /make Disallow: /make/share/ # User-triggered agents. # # These fetch a page because a person asked about us seconds earlier, which # makes them the highest-intent traffic here - a block turns away a live # prospect mid-question. They are already allowed by the `*` group above; they # are named individually so that a reader checking for them by name gets an # answer instead of an inference. User-agent: ChatGPT-User/2.0 Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /api/ Disallow: /auth/callback Disallow: /oauth/ Allow: /mcp$ Disallow: /mcp/docs Disallow: /mcp/sse Disallow: /mcp/docs/sse Disallow: /a2a Disallow: /stitch-* Disallow: /engine Disallow: /make Disallow: /make/share/ User-agent: Claude-User Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /api/ Disallow: /auth/callback Disallow: /oauth/ Allow: /mcp$ Disallow: /mcp/docs Disallow: /mcp/sse Disallow: /mcp/docs/sse Disallow: /a2a Disallow: /stitch-* Disallow: /engine Disallow: /make Disallow: /make/share/ User-agent: Claude-SearchBot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /api/ Disallow: /auth/callback Disallow: /oauth/ Allow: /mcp$ Disallow: /mcp/docs Disallow: /mcp/sse Disallow: /mcp/docs/sse Disallow: /a2a Disallow: /stitch-* Disallow: /engine Disallow: /make Disallow: /make/share/ User-agent: Perplexity-User Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /api/ Disallow: /auth/callback Disallow: /oauth/ Allow: /mcp$ Disallow: /mcp/docs Disallow: /mcp/sse Disallow: /mcp/docs/sse Disallow: /a2a Disallow: /stitch-* Disallow: /engine Disallow: /make Disallow: /make/share/ User-agent: OAI-SearchBot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /api/ Disallow: /auth/callback Disallow: /oauth/ Allow: /mcp$ Disallow: /mcp/docs Disallow: /mcp/sse Disallow: /mcp/docs/sse Disallow: /a2a Disallow: /stitch-* Disallow: /engine Disallow: /make Disallow: /make/share/ # Training-only crawlers — disallowed. # # CCBot (Common Crawl) and Bytespider (ByteDance) build model-training corpora # only. Neither powers an answer engine that cites or links back, so unlike the # crawlers above, no reader ever arrives from them. # # `Disallow: /` is the whole policy for these two. They deliberately carry NO # Content-Signal line, and that omission is the point: a crawler that fetches # nothing has no content for a usage signal to govern, so the line changed # nothing about what these two may do - while a reader scanning this file for a # site-wide policy found the refusing signal here and reported it as ours. It # was never ours. Every crawler that can send a reader back is welcome to # index, cite, ground answers in, and train on this content, and each group # above says so explicitly. # # If a future group here ever needs a restrictive signal line, put the group at # the END of the file and expect the same misreading. User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / # Blocked at the edge — kept in sync deliberately. # # These crawlers are blocked by a Cloudflare WAF rule ("AI Crawl Control - Block AI # bots by User Agent"). They take content without sending readers back, so we don't # pay the bandwidth. Listing them here is not a second layer of enforcement — it is # honesty: a robots.txt that advertises "Allow: /" to a crawler we 403 at the edge is # a lying invitation, and Cloudflare counts every such fetch as a robots.txt # violation against us. # # If you unblock any of these in AI Crawl Control, delete its group here in the same # change. Verified 2026-08-13: CCBot and Perplexity-User were 403 while listed as # allowed. See .planning/AI_AGENT_READINESS_ULTRAPLAN.md (P0). # # "Anchor Browser" is also blocked at the edge but cannot be expressed here — a # robots.txt user-agent token has no space in it. User-agent: Amazonbot Disallow: / User-agent: PetalBot Disallow: / User-agent: Timpibot Disallow: / User-agent: Novellum Disallow: / User-agent: ProRataInc Disallow: / User-agent: Terracotta Disallow: / # Schema Feeds (NLWeb) — structured-data feeds an answer engine can ingest # directly instead of re-deriving JSON-LD from every page. # # Commented out on purpose: "Schemamap" is not a robots.txt directive any # crawler parses, and every validator that reads this file (Lighthouse's # robots-txt audit, Semrush, Ahrefs) reports it as a syntax error — which costs # the SEO score on every page of the site for a line no consumer acts on. The # feed itself is unchanged and stays discoverable at the URL below and from # /.well-known/ai-catalog.json. # Schemamap: https://aicontentdrop.com/schema-map.xml # Sitemaps Sitemap: https://aicontentdrop.com/sitemap.xml Sitemap: https://aicontentdrop.com/sitemap-tools.xml Sitemap: https://aicontentdrop.com/sitemap-video.xml # Agent endpoints (no credential needed to read) # /.well-known/ai-catalog.json — Agentic Resource Discovery catalogue # /.well-known/mcp.json — MCP server manifest (POST /mcp) # /.well-known/agent-card.json — A2A agent card (POST /a2a) # /.well-known/did.json — did:web identity behind the catalogue # /ask — NLWeb natural-language query endpoint # /?mode=agent — machine-readable homepage # /developers/llms.txt — scoped context for the API surface # /openapi.json — REST API specification # /graphql — GraphQL API, introspection open # /graphql/schema.graphql — the schema as SDL # /docs — developer documentation (every page has a .md twin) # /v1/batch — many read operations in one request # LLM Context # See https://llmstxt.org for the llms.txt standard # Provides structured product information for AI model training and retrieval # Full-text corpus for LLMs # https://aicontentdrop.com/llms.txt — curated overview # https://aicontentdrop.com/llms-full.txt — every public article, full text