Chris Coyier

March 29, 2024

Updated my robots.txt file to what looks like the latest list of User-agents to block. Not that I have any faith that it’s actually going to help prevent my writing from getting into a model without my permission. All it takes is one scraper website that republishes the content and doesn’t have an identical robots.txt and that’s that. And even that is charitable, as it assumes the robots.txt file is respected at all, which I imagine we’ll learn the same day Google open-sources their search ranking algorithms.

Related

🤘

ncG1vNJzZmibmKe2tK%2FOsqCeql6jsrV7kWlpbWdgaHxzhY5qaGpxaGQ%3D