Julien Le Dem

Julien Le Dem

@julien.ledem.net

Principal Engineer, Founder, Angel, Advisor, OSS. LFAI&data: OpenLineage, Marquez, ASF: Parquet, Arrow, Iceberg, 🐖 he/him. Me: https://julien.ledem.net/ Blog: https://sympathetic.ink

5.9K followers722 following344 posts

Julien Le Dem reposted

Last year, I gave a keynote at AI Council about The Deconstructed Database: a composable and extensible database design enabled by the cloud and a collection of standard open source formats and components. (Parquet, Arrow, Substrait, Iceberg, OpenLineage, Calcite, DataFusion).

After the talk I was asked how we apply those principles at Datadog. Well, thank you for your patience, this year, Pierre Lacave and I will go over the evolution of the data platform at Datadog from the very beginning to today and the challenges ahead. Come see our talk in San Francisco!

File formats! Great references 😎

Andy Pavlo@andypavlo.bsky.social

I've posted my latest recap of the world of databases: www.cs.cmu.edu/~pavlo/blog/... All the hot topics from the last year: • More Postgres action! • MCP for everyone! • MongoDB gets litigious with FerretDB! • File formats! • Market movements! • The richest person in the history of the world!

In the past few years, we’ve seen a cambrian explosion of new columnar formats, challenging the hegemony of Parquet. Presumably, the design of yore is not going to cut it moving forward. I spent some time to understand a bit better how things actually changed. sympathetic.ink/2025/12/11/C...

sympathetic.inkColumn Storage for the AI EraIn the past few years, we’ve seen a cambrian explosion of new columnar formats, challenging the hegemony of Parquet: Lance, Fastlanes, Nimble, Vortex, AnyBlox, F3 (File Format for the Future). The thi...
View full thread
View full thread

The components of databases, distributed or not, have been commoditized as individual parts that anyone can compose into use-case specific engines. Define your constraints and build a query engine that solves your problem.

It’s been incredible to see the adoption of key components like Parquet, Arrow, Iceberg, and DataFusion. They provide an interoperability layer that enables using data without creating silos and duplication.

Over the past decade, the big data ecosystem has matured and evolved from a melting pot of competing projects into a composable ecosystem organized around a few open source standards.

Netflix: "We capture over a trillion events a day." Datadog: www.datadoghq.com/blog/enginee....

Gergely Orosz@gergely.pragmaticengineer.com

What’s it like to work as a software engineer at Netflix? In this special episode recorded at Netflix’s headquarters in Los Gatos, I sat down with Elizabeth Stone, CTO at the company - in the signature Netflix director chairs (and with a pro Netflix camera crew!) (cont'd)