Earthmover

Earthmover

@earthmover.io

the cloud platform for weather and climate data teams https://earthmover.io/ find us more frequently on LinkedIn: https://www.linkedin.com/company/earthmover

430 followers152 following95 posts

Today, napari users canโ€™t fully leverage Xarrayโ€™s labeled metadata, as slider names, units & dimensions often get out of sync. Thatโ€™s changing. At SciPy 2025, napari, Xarray & CellProfiler devs began a collab to build true metadata-aware visualization across sciences. earthmover.io/blog/scienti...

earthmover.ioScientific Data Visualization with Xarray and Napari - EarthmoverIan Hunt-Isaak Xarray Community Developer Tim Monko Neuroscience PhD & napari community manager This blog was also published on the xarray blog and the napari blog. TL;DR Making Napari and Xarray work...

Zarr is great for big multidimensional dataโ€”but not for collaboration. Our latest post by Lindsey Nield shows how Icechunk adds Git-like transactions + version control, so teams can safely share and scale Zarr workflows. earthmover.io/blog/multi-p...

earthmover.ioMulti-Player Mode: Why Teams That Use Zarr Need Icechunk - EarthmoverIcechunk brings Git-like version control to Zarr. Ensure atomic updates, consistent reads, and scalable collaboration for shared datasets.

August updates: How to access 50 years of ERA5 data with Earthmover, the Icechunk 1.0 release, and an invitation to our August webinar: How To Turn Weather Radar Archives into Usable Science with Xarray DataTree and Zarr

linkedin.comAugust Newsletter๐ŸŒ 50 Years of ERA5 Access in Earthmover: The Arraylake / Icechunk edition is an analysis-ready dataset that retains only the single-level surface fieldsโ€”18 hourly variables such as 2 m temperature, 10...

๐Ÿ“ก From Raw Scans to Structured Science โ€” Radar Data Gets a Standard In our latest blog post, Alfonso Ladino-Rincon explores the progress made with FM-301/CfRadial 2.1, the World Meteorological Orgโ€™s official global data standard for weather radar and lidar. earthmover.io/blog/from-fi...

earthmover.ioFrom Files to Datasets: FM-301 and the Future of Radar Interoperability - EarthmoverAt Earthmover, weโ€™re interested in weather radar data for two reasons: Weโ€™re obsessed with data like thisโ€“which doesnโ€™t fit into either traditional relational databases or into the standard โ€œgeospatia...

Today at SciPy 2025 we released Icechunk 1.0, an open source package and specification that enables database-style transactions against petabyte-scale array datasets using only cloud object storage as infrastructure. Read about it on our blog earthmover.io/blog/icechun..., or visit earthmover.io

earthmover.ioIcechunk 1.0: Production-Grade Cloud-Native Array Storage Is Here - EarthmoverA year ago, we made an important internal decision which set Earthmover on a new courseโ€”we decided to refactor and open source our core technology for storing array-based data in the cloud. This took ...

Watch Naomi's talk from CNG to learn more about how CTrees is using the Earthmover Platform

CNG@cloudnativegeo.org

Naomi Provost's #CNG2025 talk is now live! She shares CTrees_org's approach to transforming high-resolution global forest data from science models into interactive product platforms using cloud-native #geospatial tech โ–ถ๏ธ youtu.be/Ti4xvUSsUiY

While Flux gives access to many standard APIs to access your data, it is not always clear which methods are right for a given application. Matt will dive into building real-world web applications using different Flux APIs to create rich experiences and discuss best practices.

๐ป๐‘œ๐‘ค ๐‘‘๐‘œ๐‘’๐‘  ๐ผ๐‘๐‘’๐‘โ„Ž๐‘ข๐‘›๐‘˜ ๐‘Ž๐‘ฃ๐‘œ๐‘–๐‘‘ ๐‘Ÿ๐‘’๐‘‘๐‘ข๐‘›๐‘‘๐‘Ž๐‘›๐‘ก ๐‘ ๐‘ก๐‘œ๐‘Ÿ๐‘Ž๐‘”๐‘’ ๐‘๐‘’๐‘ก๐‘ค๐‘’๐‘’๐‘› ๐‘‘๐‘Ž๐‘ก๐‘Ž ๐‘ฃ๐‘’๐‘Ÿ๐‘ ๐‘–๐‘œ๐‘›๐‘ ? Icechunk stores only new or changed chunks for each version โ€”no redundant copies or rewrites. You get instant time travel, branching, and efficient updates, all with negligible storage overhead. More: bit.ly/3F1XFST

earthmover.ioIcechunk: Efficient storage of versioned array data - EarthmoverWe recently got an interesting question in Icechunkโ€™s community Slack channel (thank you Iury Simoes-Sousa for motivating this post): Iโ€™m new to Icechunk. How is the storage managed for redundant info...

Our small but mighty team has spent decades wrangling scientific data in academia and industry. Along the way, weโ€™ve spotted the same anti-patterns cropping up everywhere: data swamps, code duplication, bespoke and fragile pipelines, and gatekeepers.

Our latest blog post dives into the chaos of the status quo - where every tweak means regeneratingย the ๐‘คโ„Ž๐‘œ๐‘™๐‘’ ๐‘‘๐‘Ž๐‘ก๐‘Ž๐‘ ๐‘’๐‘กย and collaboration and experimentation is often stifled by silos and secret knowledge. Check out the full post: earthmover.io/blog/tensoro...

earthmover.ioTensorOps: Scientific Data Doesn't Have to Hurt - EarthmoverCurious how your team scores on the "Data Pain Survey"? Wondering why your teams are building Rube Goldberg machines just to put some data on a map? Or just want to see our plan to bring order to your...

1/ Our team just got back from an inspiring week at the @cloudnativegeo.org conference in Snowbird, Utah, and one thing is clear:ย ๐™๐š๐ซ๐ซ ๐š๐๐จ๐ฉ๐ญ๐ข๐จ๐ง ๐ข๐ฌ ๐ฌ๐ฎ๐ซ๐ ๐ข๐ง๐  ๐š๐œ๐ซ๐จ๐ฌ๐ฌ ๐ ๐ž๐จ๐ฌ๐ฉ๐š๐ญ๐ข๐š๐ฅ.

View full thread

3/ As we navigate a future increasingly reliant on massive volumes of EO data, Zarr is poised to become a cornerstone of the cloud-native geospatial ecosystem, unlocking new frontiers for scientific discovery and operational applications.

1/ Our team just got back from an inspiring week at the @cloudnativegeo.org conference in Snowbird, Utah, and one thing is clear:ย ๐™๐š๐ซ๐ซ ๐š๐๐จ๐ฉ๐ญ๐ข๐จ๐ง ๐ข๐ฌ ๐ฌ๐ฎ๐ซ๐ ๐ข๐ง๐  ๐š๐œ๐ซ๐จ๐ฌ๐ฌ ๐ ๐ž๐จ๐ฌ๐ฉ๐š๐ญ๐ข๐š๐ฅ.

๐Ÿ“ฃ Our team is heading to the @cloudnativegeo.org Conference in Snowbird, Utah. Weโ€™ll be sharing our latest open-source work on Xarray, Zarr, VirtualiZarr, Icechunk, and leading a hands-on workshop, ๐‘๐‘Ž๐‘Ÿ๐‘Ÿ, ๐ผ๐‘๐‘’๐‘โ„Ž๐‘ข๐‘›๐‘˜, & ๐‘‹๐‘Ž๐‘Ÿ๐‘Ÿ๐‘Ž๐‘ฆ ๐‘“๐‘œ๐‘Ÿ ๐ถ๐‘™๐‘œ๐‘ข๐‘‘-๐‘›๐‘Ž๐‘ก๐‘–๐‘ฃ๐‘’ ๐บ๐‘’๐‘œ๐‘ ๐‘๐‘Ž๐‘ก๐‘–๐‘Ž๐‘™ ๐ท๐‘Ž๐‘ก๐‘Ž-๐‘๐‘ข๐‘๐‘’ ๐ด๐‘›๐‘Ž๐‘™๐‘ฆ๐‘ ๐‘–๐‘ . Read more: bit.ly/4lRWNAJ

1/ ๐Ÿšจ New Blog Post Alert: "๐ฟ๐‘’๐‘Ž๐‘Ÿ๐‘›๐‘–๐‘›๐‘” ๐ด๐‘๐‘œ๐‘ข๐‘ก ๐ผ๐‘๐‘’๐‘โ„Ž๐‘ข๐‘›๐‘˜ ๐ถ๐‘œ๐‘›๐‘ ๐‘–๐‘ ๐‘ก๐‘’๐‘›๐‘๐‘ฆ ๐‘ค๐‘–๐‘กโ„Ž ๐‘Ž ๐ถ๐‘™๐‘–๐‘โ„Ž๐‘’ฬ๐‘‘ ๐‘๐‘ข๐‘ก ๐ผ๐‘›๐‘ ๐‘ก๐‘Ÿ๐‘ข๐‘๐‘ก๐‘–๐‘ฃ๐‘’ ๐ธ๐‘ฅ๐‘Ž๐‘š๐‘๐‘™๐‘’" ๐Ÿฆ๐Ÿ” ๐Ÿ‘‰ Read it here: earthmover.io/blog/learnin...

earthmover.ioLearning about Icechunk consistency with a clichรฉd but instructive example - EarthmoverIn this post weโ€™ll show what can happen when more than one process write to the same Icechunk repository concurrently, and how Icechunk uses transactions and conflict resolution to guarantee consisten...
View full thread

3/ We use bank account transfers to explore how ๐ˆ๐œ๐ž๐œ๐ก๐ฎ๐ง๐ค ensures ๐ญ๐ซ๐š๐ง๐ฌ๐š๐œ๐ญ๐ข๐จ๐ง๐š๐ฅ ๐œ๐จ๐ง๐ฌ๐ข๐ฌ๐ญ๐ž๐ง๐œ๐ฒ in concurrent environments. This isnโ€™t about banking; itโ€™s about understanding how Icechunk handles conflicting writes, retries, rebases, and optimistic concurrency control for @zarr.dev data.

4/ Youโ€™ll learn: โ€ข How concurrent processes can cause consistency issues โ€ข Why Icechunk transactions are ๐š๐ญ๐จ๐ฆ๐ข๐œ ๐š๐ง๐ ๐ข๐ฌ๐จ๐ฅ๐š๐ญ๐ž๐ ๐›๐ฒ ๐๐ž๐Ÿ๐š๐ฎ๐ฅ๐ญ โ€ข How ๐ซ๐ž๐›๐š๐ฌ๐ข๐ง๐  ๐ฌ๐ž๐ฌ๐ฌ๐ข๐จ๐ง๐ฌ unlock faster, conflict-aware concurrency โ€ข The role of ๐Œ๐•๐‚๐‚ ๐š๐ง๐ ๐ ๐š๐ซ๐›๐š๐ ๐ž ๐œ๐จ๐ฅ๐ฅ๐ž๐œ๐ญ๐ข๐จ๐ง in Icechunkโ€™s design

1/ ๐Ÿšจ New Blog Post Alert: "๐ฟ๐‘’๐‘Ž๐‘Ÿ๐‘›๐‘–๐‘›๐‘” ๐ด๐‘๐‘œ๐‘ข๐‘ก ๐ผ๐‘๐‘’๐‘โ„Ž๐‘ข๐‘›๐‘˜ ๐ถ๐‘œ๐‘›๐‘ ๐‘–๐‘ ๐‘ก๐‘’๐‘›๐‘๐‘ฆ ๐‘ค๐‘–๐‘กโ„Ž ๐‘Ž ๐ถ๐‘™๐‘–๐‘โ„Ž๐‘’ฬ๐‘‘ ๐‘๐‘ข๐‘ก ๐ผ๐‘›๐‘ ๐‘ก๐‘Ÿ๐‘ข๐‘๐‘ก๐‘–๐‘ฃ๐‘’ ๐ธ๐‘ฅ๐‘Ž๐‘š๐‘๐‘™๐‘’" ๐Ÿฆ๐Ÿ” ๐Ÿ‘‰ Read it here: earthmover.io/blog/learnin...

earthmover.ioLearning about Icechunk consistency with a clichรฉd but instructive example - EarthmoverIn this post weโ€™ll show what can happen when more than one process write to the same Icechunk repository concurrently, and how Icechunk uses transactions and conflict resolution to guarantee consisten...

2/ Could you run a bank ledger on #Icechunk? A multidimensional array store designed for scientific data probably isnโ€™t the first thing that comes to mind for this application... But surprisingly, it totally works!