Post

Newsrooms should not block the Internet Archive. Any public service publisher should not block the Internet Archive. If anything, we should all be directly supporting the Internet Archive.

I am annoyed that it keeps getting brought up as an “AI scraping” issue when it’s clearly a greedy paywall thing

Most of the businesses that complain about "AI scraping" really mean that they want to CHARGE for AI scraping. See eg. Reddit's sanctimonious bleating about how everyone has to lose all their useful tools so the CEO can make a profit by selling data to AI firms.

Why not use bot blockers and have internet archive as an exception

It's not about that - it's about the Internet Archive itself being scraped. So running a bot blocker on your own site wouldn't prevent that. Newsroom content is flowing from the Archive.

And my info may be dated but trying to block just bots is a challenge in itself

Could news sites use a “sign up/sign in to read” to block AI scrapers (as I’ve seen some have) instead of taking stuff off of internet archive? I’m just curious, I’m not a tech person rlly

The AI scrapers used to do things that were easily fingerprinted as being an AI tool, but this requires them to be cooperative and good citizens. Unfortunately their customers tend to get pissed when they couldn't just ask it for the news every day or to do something with a web page.

i believe the issue is that AI scrapers are crawling the Internet Archive. The NYT can try to block AI scrapers from their own site, but they can’t force the Archive to block AI scrapers from accessing archived NYT content

Spoke with my newspaper publisher wife. She maintains it’s also a business move - papers are looking for any revenue generation, and being paid for your archives is part of that. Dallas Morning News doing this in 2017 was a big change AI may be accelerating this, but it’s not the only reason

I just feel a lot of things about this because I DO NOT want to subscribe to the NYT for a variety of reasons and get sent articles and stuff for inspo all the time and now have no way of reading them outside using a subscriber friend’s login (which I’m sure they’ll crack down on) or a gift link.

No judgment but if you keep going back sounds like if you still value their journalism then you should consider supporting it :)