# robots.txt for consult-with-roy.com
#
# Replaces what this zone served until 2026-09-07: Cloudflare's Content Signals
# *preamble* and nothing after it — twenty-six lines explaining what a content
# signal means, then no signal, no User-agent, no Allow, no Disallow, no
# Sitemap. It returned 200 with `X-Matched-Path: /404` because the app had no
# robots route at all, so the boilerplate was the entire file. It restricted
# nothing and disclosed nothing.
#
# A static file rather than app/robots.ts on purpose: Next's MetadataRoute.Robots
# type emits only User-agent/Allow/Disallow/Crawl-delay/Sitemap and has no way to
# express the Content-Signal line below. tests/seo.spec.ts asserts the Disallow
# list here still matches PRIVATE_PATH_PREFIXES in lib/seo/site.ts, so this
# cannot quietly drift from the per-page robots metadata.

# Policy (Roy, 2026-10-01): indexing yes, AI answers yes, AI training yes.
# Supersedes 2026-09-07, which declined training. Everything public here is
# marketing the practice wants repeated; being in a model's weights is reach.
#
# ONE GROUP, ON PURPOSE. A crawler obeys only the single most specific group
# that names it (RFC 9309 §2.2.1) and never falls back to `*`. Until 2026-10-01
# this file gave Googlebot, Bingbot, OAI-SearchBot, ClaudeBot and others their
# own groups holding nothing but `Allow: /` — so none of them ever saw the
# Disallow list below, /r/ included. With every crawler welcome there is no
# reason to name any of them, and every reason not to: a named group is a group
# that silently drops the closures. tests/seo.spec.ts fails if one reappears.
# If a crawler ever needs blocking, give it `Disallow: /`, which needs nothing
# repeated; anything less than a full block must restate every rule below.
#
# Content-Signal is a Cloudflare/IETF proposal, not part of the original robots
# grammar, so a crawler that does not understand the line ignores it and obeys
# the ordinary directives below.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

# Closed surfaces. /r/ is the one that matters most: those URLs carry a
# single-use token that resumes a client's session, so an indexed /r/ URL is a
# published credential rather than merely an awkward privacy leak. The rest are
# authenticated or per-client views with nothing generic to rank for.
Disallow: /admin
Disallow: /admin/
Disallow: /api
Disallow: /api/
Disallow: /r/
Disallow: /consult
Disallow: /consult/
Disallow: /verify
Disallow: /verify/
Disallow: /chat
Disallow: /chat/

Sitemap: https://consult-with-roy.com/sitemap.xml