ai crawlers

choose, for each site, which AI crawlers may read it. two boxes, both on by default:

• let AI companies train on my site: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent, Bytespider and Amazonbot copy pages to build their models
• let AI assistants read my site to answer questions: ChatGPT, Claude and Perplexity open your pages when someone asks about them, and link to them

where to set it#

• on the web: site settings on tinydot.com, the AI crawlers section
• on the Mac: Edit Site, under custom domain
• on the iPhone: Edit Site, AI crawlers

what happens when you turn one off#

• your site's /robots.txt asks those crawlers to stay away, and says the same thing as a Content-Signal line (ai-train=no, ai-input=no)
• if they come anyway under their own name, they are turned away
• with answers off, /llms.txt and /llms-full.txt are no longer published
• search engines are not affected. Google and Bing keep indexing your site

your own robots.txt#

put a robots.txt in the top folder of your site and visitors get that file exactly as you wrote it. the boxes still turn away the crawlers you block.

what it cannot do#

• a scraper that pretends to be a browser still gets through. no robots file stops that
• Google's AI Overviews read pages with Googlebot, the same crawler as search. blocking it would take your site out of Google