Article URL: https://www.theverge.com/24067997/robots-txt-ai-text-file-web-crawlers-spiders
Comments URL: https://news.ycombinator.com/item?id=39420845
Points: 18
# Comments: 5
With the rise of AI, web crawlers are suddenly controversial leephillips on February 18, 2024 at 11:14 Hacker News: Front Page
