Link tags: scrapers

2 AI Agents robots.txt Builder | Dark Visitors

A handy resource for keeping your blocklist up to date in your robots.txt file.

Though the name of the website is unfortunate with its racism-via-laziness nomenclature.

I realized why I hadn’t yet added any rules to my robots.txt: I have zero faith in it.