Ryan Abernathey

Ryan Abernathey

@rabernat.bsky.social

Scientist and Startup Founder Co-Founder and CEO @Earthmover @pangeo_data steering council member ex-Professor @Columbia @LamontEarth

1.6K followers692 following44 posts

Big launch from @earthmover.io today! Now you can access high-quality ARCO data directly from trusted providers via our Data Marketplace.

Earthmover@earthmover.io

Announcing the Earthmover Data Marketplace: cloud native access to high-velocity gridded weather, climate, and E/O datasets. Get seamless access to data from trusted providers, including forecasts from @ecmwf.int, @noaa.gov, and exciting new AI-based forecasts! Visit: app.earthmover.io/marketplace

Ryan Abernathey reposted

Ryan Abernathey reposted

1/ Our team just got back from an inspiring week at the @cloudnativegeo.org conference in Snowbird, Utah, and one thing is clear: 𝐙𝐚𝐫𝐫 𝐚𝐝𝐨𝐩𝐭𝐢𝐨𝐧 𝐢𝐬 𝐬𝐮𝐫𝐠𝐢𝐧𝐠 𝐚𝐜𝐫𝐨𝐬𝐬 𝐠𝐞𝐨𝐬𝐩𝐚𝐭𝐢𝐚𝐥.

Ryan Abernathey reposted

📣 Our team is heading to the @cloudnativegeo.org Conference in Snowbird, Utah. We’ll be sharing our latest open-source work on Xarray, Zarr, VirtualiZarr, Icechunk, and leading a hands-on workshop, 𝑍𝑎𝑟𝑟, 𝐼𝑐𝑒𝑐ℎ𝑢𝑛𝑘, & 𝑋𝑎𝑟𝑟𝑎𝑦 𝑓𝑜𝑟 𝐶𝑙𝑜𝑢𝑑-𝑛𝑎𝑡𝑖𝑣𝑒 𝐺𝑒𝑜𝑠𝑝𝑎𝑡𝑖𝑎𝑙 𝐷𝑎𝑡𝑎-𝑐𝑢𝑏𝑒 𝐴𝑛𝑎𝑙𝑦𝑠𝑖𝑠. Read more: bit.ly/4lRWNAJ

Ryan Abernathey reposted

💡 Another banger technical blog post by @functionth.bsky.social! This one explains Icechunk's transaction system via the example of bank account transfers. Having true ACID transactions for the array data model is such a game changer!

Earthmover@earthmover.io

1/ 🚨 New Blog Post Alert: "𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔 𝐴𝑏𝑜𝑢𝑡 𝐼𝑐𝑒𝑐ℎ𝑢𝑛𝑘 𝐶𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦 𝑤𝑖𝑡ℎ 𝑎 𝐶𝑙𝑖𝑐ℎ𝑒́𝑑 𝑏𝑢𝑡 𝐼𝑛𝑠𝑡𝑟𝑢𝑐𝑡𝑖𝑣𝑒 𝐸𝑥𝑎𝑚𝑝𝑙𝑒" 🏦🔁 👉 Read it here: earthmover.io/blog/learnin...

Imagine if all you had to do to serve geopspatial data over OGC APIs was to ingest @zarr.dev datasets into cloud object storage...and the platform handled the rest. Scaling, logging, metrics, auth, etc. That's Flux! Join our webinar if you want to learn more.

Earthmover@earthmover.io

1/ Today we are launching Flux, a powerful new addition to the Earthmover platform. Flux is a high-performance gateway for exploring, querying, and visualizing geospatial data via standards-compliant APIs (EDR, WMS, and OPeNDAP).

I have to admit that I too was confused about how S3 prefixes and scalability work before reading this post by @functionth.bsky.social.

Earthmover@earthmover.io

📣 Blog post alert! 𝐄𝐱𝐩𝐥𝐨𝐫𝐢𝐧𝐠 𝐈𝐜𝐞𝐜𝐡𝐮𝐧𝐤 𝐬𝐜𝐚𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲: 𝐮𝐧𝐭𝐚𝐧𝐠𝐥𝐢𝐧𝐠 𝐒𝟑'𝐬 𝐩𝐫𝐞𝐟𝐢𝐱 𝐬𝐭𝐨𝐫𝐲. This technical post by @functionth.bsky.social dives deep into the internals of how S3 shards data, showing that distributed Icechunk can easily perform 230,000 object reads/sec and beyond. earthmover.io/blog/explori...

This one was a lot of fun to work on! Why @xarray.bsky.social and @zarr.dev are faster than DuckDB and Parquet for gridded data...from first principles.

Earthmover@earthmover.io

⭐ We just released the first post in our Fundamentals series. This one is called 𝐓𝐞𝐧𝐬𝐨𝐫𝐬 𝐯𝐬. 𝐓𝐚𝐛𝐥𝐞𝐬 - 𝐖𝐡𝐲 𝐭𝐚𝐛𝐮𝐥𝐚𝐫 𝐭𝐨𝐨𝐥𝐬 𝐭𝐫𝐢𝐩 𝐨𝐯𝐞𝐫 𝐠𝐫𝐢𝐝𝐝𝐞𝐝 𝐝𝐚𝐭𝐚. earthmover.io/blog/tensors...

I am so excited to share this new project with NASA and @developmentseed.org . By applying Icechunk and the "virtualization" approach, we can make Petabytes of archival data accessible to high performance analytics. Unlocking NASA's cloud archives will have immense downstream benefits for the world.

Earthmover@earthmover.io

1/ 🚀 Solving #NASA ’s cloud data dilemma: Icechunk unlocks 100x faster access to archival data formats We're thrilled to publish results from our pilot project with NASA and @developmentseed.org to enable high-performance cloud-native access for NASA’s 100s of petabytes of Earth observation data.

Ryan Abernathey reposted

Here on Bluesky, I follow two topics: 1) software & data infrastructure and 2) cycling. I mostly post about software but today, I can share a bit about a fun cycling adventure of my own 🚵‍♂️. Last week, I got to ride in the 24 Hours in the Old Pueblo - placing 4th in the Open Men Duo category:

One reason we started @earthmover.io was to create a home for amazing open source devs building towards a shared vision for a cloud-native future. So I couldn't be more excited to have @tegnicholas.bsky.social joining the team! 🚀 Tom's capabilities as an engineer are unparalleled!!!

Earthmover@earthmover.io

We’re thrilled to welcome @tegnicholas.bsky.social to the team! Tom is a prolific contributor to @xarray.bsky.social and @pangeo.io. He created Xarray Datatree and VirtualiZarr. Tom is working as a Forward Engineer, supporting our customers and the broader open-source community.

Have you ever waited a long time for @xarray.bsky.social to open up a large @zarr.dev dataset in cloud storage? Upgrade to Zarr 3 for a massive performance improvement! 🚀

Earthmover@earthmover.io

1/ Check out our latest blog post earthmover.io/blog/xarray-... to learn about the dramatic improvement and performance of Xarray’s Zarr backend. We achieved improved the “time to first byte” metric, building on Zarr-Python’s new asyncio internals.

Ryan Abernathey reposted

Register for tomorrow’s webinar, Data Version Control for Arrays with Icechunk. bit.ly/3WVPSf2 @rabernat.bsky.social will discuss Icechunk’s application in AI/ML model development, how it adds safety + flexibility to data pipelines, & how it enables git-style version control to Zarr Arrays + Groups

Next week I'll be giving a webinar on one of my favorite parts of Icechunk--the ability to apply git-style version control to Zarr arrays and Groups. This capability makes data workflows so more safe, robust, and reproducible. Come check it out! (Registration link below.)

Earthmover@earthmover.io

Join our webinar, Data Version Control for Arrays with #Icechunk. @rabernat.bsky.social will explain how Icechunk’s transactions, snapshots, tags, + branches can add safety & flexibility to data pipelines and workflows. Register here: bit.ly/3WVPSf2

Ryan Abernathey reposted

This is a golden opportunity to educate the public about how basic research works in this country, and the benefits it brings across the board - including to businesses. How we became the most dynamic economy in the world through research investment...

Ryan Abernathey reposted

NYC folks — @rabernat.bsky.social will be leading a technical workshop, Navigating Weather and Climate Data, on Tuesday as part of the #GeospatialRiskSummit. Ryan will teach attendees the current state-of-the-art technology for analyzing weather and climate data and share visions of the future.

Ryan Abernathey reposted

View full thread
View full thread

Meanwhile, the centrality of GRIB files in operational meteorology systems will continue to create friction for practitioners by requiring slow and costly data transformations. Will 2025 be the year we see operational forecasts in Zarr from a public-sector data provider?

View full thread

I/O bottlenecks are surpassing compute bottlenecks for many teams. Ingestion, curation, and optimization of training and evaluation datasets is becoming a major priority. In this new world, modeling is starting to look a lot more like data analytics and visualization in terms of infrastructure.

3. The @pangeo.io stack, built on the foundation of Xarray and Zarr, continues to expand its impact in this new AI-centric world. Nearly every team training AI weather models is doing so from Zarr data. However, practices vary widely in terms of on-disk data layout and data loader architecture...

View full thread

2. AI is changing the requirements for data systems. Whereas the main infrastructure requirement for weather forecasting used to be a hard-core HPC system for compute-bound workloads, AI requires smaller, GPU-based clusters, plus a massive trove of clean, AI-ready data upon which to train.

1. The private-sector presence in atmospheric science will continue to grow. On the “data provider” side, there is an incredible amount of private-sector innovation happening around AI-driven weather forecasting, from big tech companies (Google, NVIDIA, @microsoft.com) to startups...

(like @brightband.com, Excarta, Silurian, Salient, Zeus AI, Jua, etc), to more established companies breaking into this field (Spire, Tomorrow). On the “end user” side, we’re seeing growing sophistication by commodities and energy traders in their use of weather data, with many companies...

Ryan Abernathey reposted

When my house and community burned - where was the satellite information? A very raw blog post I am putting up so folks in the satellite, geospatial and tech fields can listen firsthand why I think we are failing the situations that need us the most. www.linkedin.com/pulse/when-o...

linkedin.comWhen our community burned where was the satellite information?Video of our home burned down taken by a neighbor after the Palisades Fire. I am writing this as a follow-up to my brief post last week showing a video of our home destroyed by the Palisades fire.

Ryan Abernathey reposted

Stop by booth 353 to chat with our team and grab some stickers! #AMS2025

Ryan Abernathey@rabernat.bsky.social

The @earthmoverhq.bsky.social booth is ready for #AMS25! Come by and chat with us about your cloud data projects! 🤓

Ryan Abernathey reposted

We’re thrilled to announce the release of @zarr.dev-Python 3! This major release brings full support for Zarr v3 specification, including the new chunk-sharding extension, major performance enhancements, and a thoroughly modernized codebase. Read more: earthmover.io/blog/zarr3-r...

earthmover.ioZarr-Python 3 is here! | EarthmoverZarr-Python 3 is here! This release brings support for Zarr's v3 specification, new extensions, and major performance improvements.

Ryan Abernathey reposted

Releasing geo-index 0.2: Fast, memory-efficient, zero-copy spatial indexes (RTree & KDTree) for Python & Rust. - Range, nearest-neighbor queries - Persist indexes as bytes, reuse them later - Supports float32 for 2x less memory - Enables GeoParquet spatial partitioning github.com/kylebarron/g...

github.comGitHub - kylebarron/geo-index: A Rust crate and Python library for packed, immutable, zero-copy spatial indexes.A Rust crate and Python library for packed, immutable, zero-copy spatial indexes. - kylebarron/geo-index

Ryan Abernathey reposted

If you’ll be at #AMS25 and love weather data, join us for a @pangeo.io Community Happy Hour! We’ll have drinks, appetizers, and great conversation with friends old and new about technology, weather & climate data, and open-source software! Register: lu.ma/ddtba5f5

lu.maPangeo Community Happy Hour at AMS 2025 · LumaWe're hosting a small gathering for the Pangeo Community at AMS 2025. Join us for drinks and light appetizers, and catch up with our community of data and…

I’m still getting over the AGU Flu 🤧 but AMS is just around the corner! We’ve got some exciting updates technology updates from @earthmoverhq.bsky.social planned. Who’s going to be there?

Earthmover@earthmover.io

🌤️ #AMS2025 is just around the corner! We are taking AMS by storm with an exhibitor booth (booth 353), talks from @jhamman.bsky.social and @rabernat.bsky.social , and hosting a @pangeo.io Community Happy Hour (register here: lu.ma/ddtba5f5)!

Amazing new results from the SWOT mission. I didn’t even realize that mapping bathymetry was possible with the SWOT data.

Tamlin Pavelsky@globalhydro.bsky.social

This paper is incredible: one year of SWOT data gives us a much better picture of global ocean floor topography than all other data ever collected. And this will only keep getting better with more SWOT data. This one result may justify the cost of launching SWOT. www.science.org/doi/10.1126/...