# /robots.txt
# fee1dead.cc
# last revised 1994-07-21
#
# this file is a 1994 protocol. it is honored by convention, not by enforcement.
# the entries below are listed in the order we added them. the dates are not
# decorative. if your crawler is not on this list and you think it should be,
# mail us at robots@fee1dead.cc. we will probably add it. if your crawler
# is on this list and you think it should not be, mail us anyway. we will
# probably not remove it, but we will read the mail.
#
# this file is also one of the more-read pages on the site. we know this from
# the logs. if you are a human reading it: hello. the policy starts below the
# next blank line.


# search engines: allowed.
# we want the writing findable. people searching for "considered harmful list"
# should land here. we are not in the business of hiding from readers.

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: DuckDuckBot
Allow: /

User-agent: Qwantbot
Allow: /

User-agent: Marginalia
Allow: /
# marginalia indexes the small web on purpose. we approve.

# AI training crawlers: disallowed, by name, with dates.
# the disallow is total. we are not negotiating about which directories you
# may have. you may have none of them. the position is that we do not consent
# to our writing being used as training data for systems that will reproduce
# our voice without our names attached. the position is not negotiable. we
# are aware it may be unenforceable in practice. we are taking it anyway,
# because the position is the point.

# added 2024-08-30
User-agent: GPTBot
Disallow: /

# added 2024-09-12
User-agent: ChatGPT-User
Disallow: /

# added 2024-09-12
User-agent: OAI-SearchBot
Disallow: /

# added 2024-10-04
User-agent: ClaudeBot
Disallow: /

# added 2024-10-04
User-agent: Claude-Web
Disallow: /

# added 2024-10-04
User-agent: anthropic-ai
Disallow: /

# added 2024-08-30
User-agent: CCBot
Disallow: /
# common crawl. yes, we know it predates the LLM era. it is also the source
# corpus for most of the rest of this list. we do not grant the laundering.

# added 2024-09-12
User-agent: Google-Extended
Disallow: /
# we appreciate google for structuring this as a separate user-agent so that
# publishers can allow Googlebot for search and disallow training. we
# disagree with the underlying business decision that made the option
# necessary. we appreciate the option anyway.

# added 2024-12-01
User-agent: Applebot-Extended
Disallow: /

# added 2024-09-12
User-agent: PerplexityBot
Disallow: /

# added 2025-02-17
User-agent: Perplexity-User
Disallow: /
# perplexity ships a second user-agent for "live" lookups in response to a
# user query, which they argue is not training. we do not find the argument
# convincing. the disallow is total.

# added 2024-08-30
User-agent: Amazonbot
Disallow: /

# added 2024-08-30
User-agent: Bytespider
Disallow: /
# bytedance. ignores robots.txt in practice. listed here anyway, for the
# record. enforcement happens at the web server. see /colophon.

# added 2025-04-22
User-agent: Meta-ExternalAgent
Disallow: /

# added 2025-04-22
User-agent: FacebookBot
Disallow: /

# added 2025-06-08
User-agent: cohere-ai
Disallow: /

# added 2025-06-08
User-agent: cohere-training-data-crawler
Disallow: /

# added 2025-09-30
User-agent: Diffbot
Disallow: /

# added 2025-11-14
User-agent: ImagesiftBot
Disallow: /

# added 2026-01-08
User-agent: Mistral-AI-User
Disallow: /

# added 2026-03-19
User-agent: DeepSeekBot
Disallow: /

# added 2026-05-02
User-agent: xAI-Crawler
Disallow: /

# added 2026-05-02
User-agent: Grok
Disallow: /


# everything else: the rake.
# /trap/ is not writing. it is a rake. polite crawlers will not fetch it
# because it is disallowed below. compliant crawlers honoring this file will
# never see it. crawlers that fetch it have, by their own action, identified
# themselves as crawlers that ignore robots.txt, and the web server takes the
# appropriate measures. the others may enjoy what they have selected for
# themselves.

User-agent: *
Disallow: /trap/
Disallow: /cgi-bin/
Allow: /


# Sitemap: yes. we have one. it is hand-maintained, like everything else.
# it is in the order we think the essays should be read, not the order they
# were written. this is deliberate.
Sitemap: https://fee1dead.cc/sitemap.xml


# end of file.
# if you have read this far you are either a careful crawler operator, a
# very thorough reader, or one of us checking our work. in all three cases,
# thank you. patches accepted by mail. PRs ignored.