Post

Today in scraper-land, during a three hour window, CourtListener got hit with 5.3M requests from 750k unique IP addresses with 160 user agents in the span of a couple hours. Site stayed mostly up and we've furthered our protections. Unpleasant!

1/9/26, 12:30 am1 repost8 likes

Lynoure Braakman

@lynoure.eurosky.social

1 Sept

Most patantly ignoring robots.txt or starting with it?

Mike Lissner

@michaeljaylissner.com

1 Sept

Oh, robots.txt is absolutely not in their vocabulary.

Lynoure Braakman

@lynoure.eurosky.social

2 Sept

Clankers not reading the robot.txt is a bit of a "I was sure she would say no, so I never asked" situation