Weston Pace

Weston Pace

@westonpace.bsky.social

Software developer working on all things arrow and columnar storage, currently, Lance.

143 followers367 following274 posts

Weston Pace reposted

This release is cool for a lot of reasons but my favorite is that the new Observable does database access with ADBC and returns results as Arrow. And, as a bonus, it uses @columnar.tech 's dbc (dbc.how) to install drivers.

Observable@observablehq.com

For more than a year, we’ve been quietly building an all-new Observable. Today, that new Observable is live on observablehq.com. Read Mike @ocks.org’s announcement: observablehq.com/@observableh...

It's pretty awesome to look back and see how far this has come in 10 years!

Apache Arrow@arrow.apache.org

Apache Arrow is 10 years old 🎉 The first git commit landed on 2016-02-05. Now Arrow has 10+ official implementations and a growing third-party ecosystem. Read our celebratory blog post for some tidbits of Arrow history, and our outlook on the present and future. arrow.apache.org/blog/2026/02...

Lot's of work being done on columnar file formats lately. I count 5 new formats so far (Lance, Nimble, Vortex, FastLanes, F3). It's definitely something we follow at LanceDB and it can be confusing to track. So here is my very biased head-canon (trying to stay positive)

View full thread

F3 is from a joint project between CMU and Tsinghua University. They have tackled the "forwards compatibility" problem by storing WASM decoders with the data so that old readers can read data written by futuristic writers.

Lot's of work being done on columnar file formats lately. I count 5 new formats so far (Lance, Nimble, Vortex, FastLanes, F3). It's definitely something we follow at LanceDB and it can be confusing to track. So here is my very biased head-canon (trying to stay positive)

View full thread

Vortex comes from SpiralDB. They've done a good job explaining what they do and writing about it. They've made a big focus on compression but, especially, on pushing down compute to run against compressed data.

FastLanes comes from CWI. They're the group that's designed some of the new lightweight compression algorithms (e.g. FSST). They definitely focus on compression and they likely have the best layout for processing data already in memory.

Lot's of work being done on columnar file formats lately. I count 5 new formats so far (Lance, Nimble, Vortex, FastLanes, F3). It's definitely something we follow at LanceDB and it can be confusing to track. So here is my very biased head-canon (trying to stay positive)

I work on Lance! So I'm most biased here. We focus on balancing random access and full scans. All formats have focused on better random access / large data, but none to the extent that we have, especially for tensors / embeddings.

Nimble comes from Meta, and there has sadly not been much written about it publicly. The best I can say at the moment is that Nimble has made perhaps the biggest emphasis of all the formats on extremely wide schemas (again, all formats have done some here).

We're trying to figure out "Substrait coordinates" (e.g. organization, name, version tuples) for Substrait functions. Is anyone out there actually passionate about the topic or have any lessons or advice? At the moment, leaning towards `organization:name:version` (and forbid colon in each field)

Weston Pace reposted

It was fun to let an Arrow C++ contributor highlight their work here. Thanks to Rossi for all the work on the post and of course all the reviewers of the PRs themselves.

Apache Arrow@arrow.apache.org

New post up on the Arrow blog about some of the recent improvements to the embedded query engine inside Arrow C++: arrow.apache.org/blog/2025/07... #apachearrow

Me: Do this thing Claude Code: I tried but I'm not allowed to cd to that directory for security reasons. Claude Code: Hmm... Claude Code: I can use xargs to work around this limitation, how does that sound? Me: 🤨

Are there any technical words or terminology for the difference between "a dependency" and "a dependency that's part of your public API"? Or do any dependency management systems treat these differently? Because I think there's a huge functional difference between those two things.

The most realistic part of Ready Player One is when IOI refuses to let Parzival work from home when they make their offer for him to consult. Despite the entire job being in the oasis.