Neatnik Notes · Gotta block ’em all
While we’re playing whack-a-mole, let’s poison these rodents.
Link tags: robots.txt
7
Blocking bots – Manu
Blocking the bots is step one.
Perplexity AI Is Lying about Their User Agent • Robb Knight
See, this is exactly why we need to poison these bots.
AI Agents robots.txt Builder | Dark Visitors
A handy resource for keeping your blocklist up to date in your robots.txt file.
Though the name of the website is unfortunate with its racism-via-laziness nomenclature.
Robots.txt - Jim Nielsen’s Blog
I realized why I hadn’t yet added any rules to my
robots.txt: I have zero faith in it.
Documentation for GPTBot - OpenAI API
Now that the horse has bolted—and ransacked the web—you can shut the barn door:
To disallow GPTBot to access your site you can add the GPTBot to your site’s robots.txt:
User-agent: GPTBot Disallow: /
http://google.com/robots.txt
User-agent: zombies Disallow: /brains