Jack Vanlightly

Durable functions have many names across frameworks, but it reduces to 3 forms: stateless functions, sessions, actors. Explainer here: jack-vanlightly.com/blog/2025/12...

New post: The Durable Function Tree. Durable execution engines all end up building some form of function tree with suspension points shaped by local vs remote side effects. I look at why, the trade-offs, and where orchestration should (and shouldn’t) be used. jack-vanlightly.com/blog/2025/12...

📝 Blogged: "On Idempotency Keys" Discussing several options for ensuring exactly-once processing in distributed systems using idempotency keys, from UUIDs to monotonically increasing sequences. 👉 www.morling.dev/blog/on-idem...

New blog post: Demystifying Determinism in Durable Execution Why do durable execution frameworks care so much about determinism? I unpack the underlying mechanics. Post: jack-vanlightly.com/blog/2025/11...

New blog post about Qbeast and how it brings a multidimensional spatial index to Iceberg/Delta. 🔹 Hypercube-based layout 🔹 Index used by writers, invisible to engines 🔹 Better locality + pruning, adaptive layout Lots of innovation ahead in lakehouses. jack-vanlightly.com/blog/2025/11...

Stream-order vs batch-order in Iceberg: * Flink wants temporal locality. * Spark wants value locality. Same table, conflicting physics. New post: jack-vanlightly.com/blog/2025/11...

Three KIPs (1150, 1176, 1183) all target Kafka’s cross-AZ replication costs but there is a wider question at stake. My new post explains the KIPs, the trade-offs between reusing old abstractions vs. embracing stateless compute over S3. jack-vanlightly.com/blog/2025/10...

New post: why I’m not a fan of “zero-copy” Iceberg tables for Apache Kafka. From a systems design view, it trades storage savings for coupling and complexity. Sometimes, duplication is cheaper than coupling. jack-vanlightly.com/blog/2025/10...

Why don’t Iceberg or Delta Lake have secondary indexes? Because analytics workloads and OLTP workloads optimize for opposite I/O patterns. See my dive into data layout, pruning, and what “indexing” really means in open table formats: jack-vanlightly.com/blog/2025/10...

New deep dive: Understanding Apache Fluss I spent August reverse-engineering Fluss, Alibaba’s new table storage engine for Flink (partially forked from Kafka). This post covers its architecture, tiering, and how it tackles changelogs & low-latency state. jack-vanlightly.com/blog/2025/9/...

New blog post: A Conceptual Model for Storage Unification. The post defines what storage unification means, defines terminology and evaluates different building blocks and approaches to doing it. jack-vanlightly.com/blog/2025/8/...

In a future of autonomous AI agents, we can't limit ourselves to error prevention and error detection, we must also include remediation. jack-vanlightly.com/blog/2025/7/...

Science moves slowly because wrong theories waste decades. Engineering is careful because failures kill people. Software moves fast because mistakes are cheap, the expensive error isn't making the wrong choice, it's taking too long to make any choice. jack-vanlightly.com/blog/2025/7/...

Where does reliability begin, and where does it end? In distributed business architectures, the answer is responsibility boundaries. New post: jack-vanlightly.com/blog/2025/7/...

ChatGPT thought it was Tuesday, so I made fun of it and it admitted it was Wednesday. So I made fun of it again, and it admitted it was...Wednesday. But sure, AI agents are gonna steal my job 🤔

My musical evolution continues, discovered deep hypnotic drone music today. No drugs required 😄 The Hypnus Records label is great.

How to reliably distribute work across microservices, stream processors, durable execution, event-driven, orchestration and now AI agents? Coordinated Progress is a 4 part series that explores the common structure behind reliable distributed systems. jack-vanlightly.com/blog/2025/6/...

I took a break from social media and my blog for a couple of months. ND burnout. But I'm tentatively back, probably just to post my writing here for now. HOTDS is on pause. Getting back to writing is therapeutic though. I'll post something this week that I've been working on.

Another Humans of the Data Sphere is out, with issue 10! In this issue people are talking fsyncs, tips for running ClickHouse at scale, the problems with MCP and more. Plus I dig up a classic paper from 1962. www.hotds.dev/p/humans-of-...

Wow, I just discovered gamma wave music. Wrote non-stop for three hours.

Any Principal Engineers out there with ADHD or creative wiring — who don’t thrive in the tasks of project coordination, alignment meetings, and people management, but thrive on strategy, system design, writing, and shaping direction through ideas? Curious how you navigate the role.

A new disaggregated log replication survey post is out. How does the combination of Apache Pulsar with Apache BookKeeper divide and conquer the responsibilities of log replication? jack-vanlightly.com/blog/2025/3/...

I think I have an issue with tabs, it's grown to 371. My workstation is struggling to open chrome after a restart now.

Another Humans of the Data Sphere is out, with issue #9! In this issue, we also look at whether software engineers can learn from mechanical engineering, and looking at table formats as a form of virtualization. www.hotds.dev/p/humans-of-...

A new log replication disaggregation survey post is out! The Kafka Replication Protocol: 🔹Separation of control plane from data plane. 🔹Role separation with minimal coupling. 🔹Kafka’s alignment with Paxos roles. jack-vanlightly.com/blog/2025/2/...

I updated my How to Disaggregate a Log Replication Protocol to include "separating ordering from IO". Basically I couldn't ignore CORFU as a way of separating responsibilities! So now we have A-F of ways of breaking apart the monolith. jack-vanlightly.com/blog/2025/2/...

Table virtualization, stream-table materialization redefine how we think about data sharing, composability, and interoperability. The OTFs (Iceberg, Delta Lake), metadata separation, & cloud storage are paving the way for modular, composable data platforms. jack-vanlightly.com/blog/2025/2/...

So many databases claim to be compatible with #Postgres these days, but what does that really mean? @pgdba.bsky.social tries to provide an objective answer with the Postgres Compatibility Index, which verifies compatibility in a wide range of criteria. Loving to see this effort!

The latest Humans of the Data Sphere is out, with issue #8! Additional topics in this post include systems correctness practices at AWS, Datadog's Husky compaction and solving for the distributed case first in ambitious systems projects. www.hotds.dev/p/humans-of-...

This reminds me of Paxos vs Raft. Paxos formalized the responsibilities of reaching consensus and acting on the agreed values into *distinct roles* (proposer, acceptor, learner). These roles can be put in a monolith or distributed. The basis for the Paxos family of protocols are these roles.

Confluent + Databricks next-level partnership 💪 Bi-directional flow between Confluent and Databricks. Kafka topics appearing as Delta tables in Databricks. Delta tables appearing as Kafka topics in Confluent. Simply amazing.

New dist sys post on log replication! In this one, I classify five approaches to disaggregating log replication protocols—laying the groundwork for a survey of real-world systems through the lens of disaggregation. jack-vanlightly.com/blog/2025/2/...

Yesterday, I covered Virtual Consensus—today, let’s dive deeper into a crucial distinction: 👉 Failure-free ordering vs. Fault-tolerant consensus Decoupling these concepts can change how we think about consensus protocols. jack-vanlightly.com/blog/2025/2/...

New distributed systems protocol write-up! This write-up dives into the Virtual Consensus in Delos paper and why it makes sense as the default log replication protocol in the era of object storage and hybrid environments. jack-vanlightly.com/blog/2025/2/...

Speculation is growing that Snowflake is planning to acquire Redpanda—but why? What justifies the rumored high price tag? When you consider market trends, the Snowflake vs. Databricks rivalry, and the AI shift, the rationale starts to become clear. Here’s my take. jack-vanlightly.com/blog/2025/2/...

Humans of the Data Sphere issue #7 is out. Obviously, DeepSeek happened but there have been plenty of other conversations happening. In this issue we also look at well and ill-conditioned APIs and the challenge of contextual data quality in AI-powered data pipelines. www.hotds.dev/p/humans-of-...

I read this blogpost 3 times, not interested in the investment aspect, but there's so many valuable insights in here on current development of AI. Also the writing alone deserves its own separate praise. youtubetranscriptoptimizer.com/blog/05_the_...