Post

LLM training and scraping has been blocked across MSPFA, as well as Google LLM summarization! Note: Some (especially lesser-known) bots can get around user agent blocks, and other search engines don't seem to have a setting to block LLM search result summarization, but do correct me if I'm wrong.

19/10/26, 4:30 pm1 repost15 likes

Yash Jain

@earanyash.bsky.social

Robots.txt and user-agent blocks only stop the bots that bother to identify themselves. The ones ignoring it entirely are the bigger problem, and there's no real enforcement mechanism yet besides naming and shaming.