Post

Newsrooms should not block the Internet Archive. Any public service publisher should not block the Internet Archive. If anything, we should all be directly supporting the Internet Archive.

407 reposts1.9K likes

The Internet Archive (a.k.a. The Wayback Machine) is a vital record that should be forever protected and maintained. Some uses: 1) when people try to cover up malicious posts, including those by a corrupt govt; 2) …

2) when your hosting company is bought by someone else and they erroneously take down your site and all backups ( actually happened to me) it saves you 3) it’s an important historical record, etc etc. it’s a vital public resource.

That one’s really problematic - they altered an archived article based on a conflict the owner had. It’s a little bit sus.

I agree and also there needs to be a bit more thought about a) shared governance (which could lead to better results and product over time) and b) IA not doing cowboy shit that jeopardizes the whole project.

Most of the businesses that complain about "AI scraping" really mean that they want to CHARGE for AI scraping. See eg. Reddit's sanctimonious bleating about how everyone has to lose all their useful tools so the CEO can make a profit by selling data to AI firms.

It's not about that - it's about the Internet Archive itself being scraped. So running a bot blocker on your own site wouldn't prevent that. Newsroom content is flowing from the Archive.

Could news sites use a “sign up/sign in to read” to block AI scrapers (as I’ve seen some have) instead of taking stuff off of internet archive? I’m just curious, I’m not a tech person rlly

The AI scrapers used to do things that were easily fingerprinted as being an AI tool, but this requires them to be cooperative and good citizens. Unfortunately their customers tend to get pissed when they couldn't just ask it for the news every day or to do something with a web page.

i believe the issue is that AI scrapers are crawling the Internet Archive. The NYT can try to block AI scrapers from their own site, but they can’t force the Archive to block AI scrapers from accessing archived NYT content

Spoke with my newspaper publisher wife. She maintains it’s also a business move - papers are looking for any revenue generation, and being paid for your archives is part of that. Dallas Morning News doing this in 2017 was a big change AI may be accelerating this, but it’s not the only reason

There is a lot to it. I don’t disagree, but I stand by what I said to. AI scrapers and those who’d pay for archives are different users/customers. Hard to dig deep on this platform.

I just feel a lot of things about this because I DO NOT want to subscribe to the NYT for a variety of reasons and get sent articles and stuff for inspo all the time and now have no way of reading them outside using a subscriber friend’s login (which I’m sure they’ll crack down on) or a gift link.