Nikhil Benesch

Nikhil Benesch

@benesch.bsky.social

Systems engineer @turbopuffer.bsky.social. Former CTO @materialize.com.

1.3K followers606 following218 posts

They’re calling it our most boring feature yet. boring(n): mundane; works exactly as expected; the highest praise you can give a database

turbopuffer@turbopuffer.bsky.social

by popular demand, now introducing one-dimensional vectors 🎈

I've been dreaming about conditional writes on S3 for years. I couldn't have asked for a better way to celebrate than getting to ship tpuf on AWS. 🐑

turbopuffer@turbopuffer.bsky.social

now available: turbopuffer AWS regions ☁️ come find us in us-west-2 and ap-southeast-2

It’s a rare day that my love of going to battle with build systems pays off like this. Kudos to GCP for a very impressive new SKU. πŸ‘πŸ’¨

turbopuffer@turbopuffer.bsky.social

GCP released a new ARM machine type with NVMe SSDs recently 70% increase in end-to-end production indexing throughput at 20-33% lower machine cost!

Couldn't be more excited to be joining the team at @turbopuffer.bsky.social. πŸ‘πŸ’¨

turbopuffer@turbopuffer.bsky.social

welcoming two excellent additions to team tpuf this week: @arash11gt as a customer engineer, and @nikhilbenesch to the DB team from pasts at Materialize & Cockroach. we're locking in a p999 eng team to build the best search engine for scale.

Just catching up on my NULL BITMAPS and this is easily the best intuition for write skew I've ever seen described. The analogy to merge skew in a codebase is genius.

Well well well: www.crunchydata.com/blog/pg_incr... Incremental pipelines come to Postgres via Crunchy Data! This is like "dbt incremental", not true incremental view maintenance like @materialize.com or Snowflake's dynamic tables, but it's a neat step towards IVM.

crunchydata.compg_incremental: Incremental Data Processing in Postgres | Crunchy Data BlogWe are excited to release a new open source extension called pg_incremental. pg_incremental works with pg_cron to do incremental batch processing for data aggregations, data transformations, or import...
Nikhil Benesch@benesch.bsky.social

Maybe EDB or Crunchy could pull it off, a la Crunchy Data Warehouse. But it’s not clear the business case is sound. IVM means more efficient read workloads, which means customers spend *less* money on Postgres.

Nikhil Benesch reposted

Nikhil Benesch reposted

For anyone keeping score in the open table format wars, AWS launching their s3 table bucket with only Iceberg support seems like a really big deal. Nothing is named in such a way that they can't add support for Delta down the line, but this feels like a huge endorsements of Iceberg (vs. others)

Nikhil Benesch reposted

Replying to @benesch.bsky.social

Aurora DSQL uses most of the SQL layer (parser, planner, optimizer, etc) from PostgreSQL itself. The storage layer, concurrency control, replication, etc are ground-up.

I'll be curious to see how GCP goes about this. Google has historically built its own tech ground up (see Spanner), but increasingly the game seems to be 100% compatibility with Postgres, which is only technically feasible if you start by forking the Postgres source code (see Aurora, Alloy, Neon).

S3 gets *native* Iceberg support. 🀯 From the AWS blog: aws.amazon.com/blogs/aws/ne... We knew something like this was coming (bsky.app/profile/bene...), but it's an even deeper integration than I expected.

aws.amazon.comNew Amazon S3 Tables: Storage optimized for analytics workloads | Amazon Web ServicesAmazon S3 Tables optimize tabular data storage (like transactions and sensor readings) in Apache Iceberg, enabling high-performance, low-cost queries using Athena, EMR, and Spark.
Nikhil Benesch@benesch.bsky.social

Something interesting is brewing in Iceberg-on-S3 land. πŸ‘€ lists.apache.org/thread/v7x65... cc @eatonphil.bsky.social

Jeff Barr's instructions so far: > I ... invoked the Spark shell with command-line arguments to use the Amazon S3 Tables Catalog for Apache Iceberg package... As far as I can tell the "Amazon S3 Tables Catalog for Apache Iceberg" package has yet to be published.

Awesome work from @antiguru.bsky.social that can cut @materialize.com memory use by 50% on a common class of workload. Most importantly for our customers: Materialize is now 50% cheaper on these workloads!

Moritz Hoffmann@antiguru.bsky.social

Sometimes it's easier to mitigate a problem than solve it: a feature I recently implemented reduces the memory and CPU @materialize.com requires to maintain future updates. Some details why solving the problem is hard and the mitigation easy-ish. materialize.com/changelog/20...