Jan Kaul

Started a new Reddit community r/dataengineeringvault. Join if you like my content & genuine news from the open-source data engineering ecosystem. I'll share the latest, but I'm happy to get your blogs. I actively encourage links/blogs (not AI-generated). sh.reddit.com/r/dataengine....

At long last, @chris.blue and I have submitted the final manuscript of Designing Data-Intensive Applications, second edition, to the publisher. There is always more that could be improved but at some point we just have to call it done. Now it goes into production; probably shipping in ~4 months.

So #databs, who wants to try S3Tables? I've created a Datafusion distribution with support for S3Tables that you can run from the command line. This is all you need: frostbow -u arn:aws:s3tables:us-east-1:123456789:bucket/my-bucket-prefix Check it out the example: github.com/JanKaul/fros...

Does anybody have an idea how S3 Tables can achieve 3x performance compared to regular Iceberg tables on S3? Does it work similar to the object-store file layout? iceberg.apache.org/docs/latest/... #databs

Join me tomorrow for my talk: "Dashtool - A data build tool to use Iceberg Materialized Views for data transformations" as part of the Open Source Analytics Community Series. December 3rd, 15:00 UTC us02web.zoom.us/w/8299438519... Passcode: 299889

This has been a long time coming. After the concerted efforts many engineers from across companies and across continents, posted some great benchmark results datafusion.apache.org/blog/2024/11...

There is a discussion on the Top 2025 Data Trends with Matt Housley today. Could be interesting. #databs www.linkedin.com/events/top20...

I recently gave a talk about Iceberg Materialized Views at Chill Data Summit London. I think Iceberg Materialized Views can really revolutionize the way we approach data transformations. Watch the recording here: youtu.be/bDxUeReHyHQ?...

Databricks wants to merge Apache Iceberg and Delta Lake...

Bluesky now has over 10 million users, and I was #2.062.888!