Post

Jacob Matson
@matsonj.com

One of the notions we think about quite a bit at @motherduck.com is "handling data before it becomes big" and I think this example from @tigrisdata.com really nails it. The trick here is parsing the JSON on insert so its really quick to query it.

Tigris Data@tigrisdata.com

AI scrapers can overload your app before you even see them coming. Anubis uses proof-of-work + honeypots to fight back—paired with @motherduck.com + @tigrisdata.com for real-time pattern detection at scale. 🧠 Detect bots. 🚫 Stop downtime. 🔗 www.tigrisdata.com/blog/anubis/

tigrisdata.comFending off scrapers with Tigris and DuckDB | Tigris Object StorageAnubis is a Web AI Firewall Utility that helps administrators keep their sites up in the wake of abusive scrapers. In order to make Anubis filter things better, you need data. Here's how Xe got that d...
Jacob Matson
@matsonj.com

Another good strategy is partitioning your writes to object storage by date / time (i.e. by hour) so you can easily write naive queries for analysis and backfills, like `select * from 's3://my_bucket/year/month/day/hour/*.json'`

0 reposts2 likes

Techaro
@techaro.lol

Not mentioned in the article: the files in the bucket are named after UUIDV7s. This lets you do limited time based lexographic querying. The bundles have at least one pope Greg minute of request metadata too. Could probably make the bundles bigger, but it makes round trips for testing slower.