TEGNicholas.bsky.social

I've been thinking a lot recently about how one thing science needs generally is a **social network for sharing big data**, so I wrote this. hackmd.io/wKKm4cIDR6a9... 🧵

"[Matt Yglesias] can write whole essays claiming that fracking is good and we need fossil fuel friendly energy policies, dismissing progressives as childish, while never engaging with the scientific literature on the consequences of climate change"

New blog post on Obstore, fast, multi-provider cloud storage access for Python: developmentseed.org/blog/2025-08...

Open source, open science for earth, climate and geospatial science? Coming to #AGU25? Build tools in #Python @jupyter.org? Submit an abstract for this session and come meet us and like minded scientists!

Will Alpine co-wrote Microsoft's manifesto on how AI will be a powerful force for good for climate change In a new interview, Alpine disavows the manifesto, saying he believes Microsoft used his work to distract from the much larger climate harms the company enables through contracts with Big Oil

NASA is being told to cancel 19 *active* missions to save $6B, which looks to be less than the ICE *hiring/retention* budget going forward. I need people to let that sentence sink into their bones for a minute.

I've been adding new accounts to the Open Source Geospatial starter pack. Who else wants on or off? #gischat #geosky go.bsky.app/PGYLmPG

It's outrageous that NASA GISS, one of the best earth & space science labs in the world, is being kicked out of its Columbia home. The outstanding scientists who work there can't say that publicly, but I can. And so can you --- call your reps, esp. (but not only) if you live in NYC or NY state.

𝐻𝑜𝑤 𝑑𝑜𝑒𝑠 𝐼𝑐𝑒𝑐ℎ𝑢𝑛𝑘 𝑎𝑣𝑜𝑖𝑑 𝑟𝑒𝑑𝑢𝑛𝑑𝑎𝑛𝑡 𝑠𝑡𝑜𝑟𝑎𝑔𝑒 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑑𝑎𝑡𝑎 𝑣𝑒𝑟𝑠𝑖𝑜𝑛𝑠? Icechunk stores only new or changed chunks for each version —no redundant copies or rewrites. You get instant time travel, branching, and efficient updates, all with negligible storage overhead. More: bit.ly/3F1XFST

Excellent post by Brian Davis laying out why doing "Open Science" for data-driven workflows is almost impossible in practice, at least without much better data pipeline tools.

The proposed cuts to NOAA cold have profound consequences not just for climate change, but for our national security and the entire economy. Here's what I learned: www.propublica.org/article/trum...

It's fun to work with real hardcore software engineers like @functionth.bsky.social who can teach you about database consistency and transactions and all that Scientific data infrastructure should be built on solid foundations like this instead of on piles of janky code written by postdocs...

the fact that I've never once thought about making a range request, and yet make them constantly for extremely targeted data pulls, is absolutely an invisible technical miracle

Hard to overstate this plan's reach, which touches nearly every aspect of NOAA's work - dissolving its research arm, gutting climate science, diminishing sat observations, boosting fossil fuels. With amazing colleagues Daniel Cusick and @scottpwaldman.bsky.social : www.politico.com/news/2025/04...

You could also do this for arbitrarily large scientific array datasets using Xarray + Icechunk + R2/Tigris juhache.substack.com/p/0-data-dis...

📣 Blog post alert! 𝐄𝐱𝐩𝐥𝐨𝐫𝐢𝐧𝐠 𝐈𝐜𝐞𝐜𝐡𝐮𝐧𝐤 𝐬𝐜𝐚𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲: 𝐮𝐧𝐭𝐚𝐧𝐠𝐥𝐢𝐧𝐠 𝐒𝟑'𝐬 𝐩𝐫𝐞𝐟𝐢𝐱 𝐬𝐭𝐨𝐫𝐲. This technical post by @functionth.bsky.social dives deep into the internals of how S3 shards data, showing that distributed Icechunk can easily perform 230,000 object reads/sec and beyond. earthmover.io/blog/explori...

Several times some database comp sci nerd has suggested to me that you could just do everything in array land using tabular database tools. Whilst they are technically correct that you _could_, this article convincingly shows why you _should not_ - that would be horribly inefficient.

I am so excited to share this new project with NASA and @developmentseed.org . By applying Icechunk and the "virtualization" approach, we can make Petabytes of archival data accessible to high performance analytics. Unlocking NASA's cloud archives will have immense downstream benefits for the world.

Very proud to be working on enabling this - the potential improvements for NASA (and eventually all fields of science) are enormous, benefiting users and internal staff.

Who else is going to @cloudnativegeo.org conference this year?

Register to attend our team's workshop at @cloudnativegeo.org! The workshop will teach attendees how to use the @zarr.dev data format model for cloud-native geospatial datacube analysis. 🗓️ Apr 30-May 2 📍Snowbird, Utah 🔗 Register here: 2025-ut.cloudnativegeo.org

With Flux all the data engineers serving climate and weather data for maps, timeseries, and scientific analysis will no longer need to run their own servers

You nailed it a while back.

As a non-US-citizen living in NYC on an O-1 visa, previously at Columbia University, who has been outspoken on Climate Change, I must say this kind of authoritarian repression has me worried. It makes me much more likely to take my skills and USCIS-approved "Exceptional Ability" to another country.

if you find yourself reluctant to support the guy at Columbia because you find some of the politics he espoused distasteful or even abhorrent, you should know that this is exactly the reaction that they hoped for in choosing him as a target for unlawful detention based on political speech.

lotta people trying very very hard to win the 2024 election instead of adapting to the rapidly shifting environment

Banger.

All websites that depend on NOAA data should display something like this. earth.nullschool.net

We now have a public slack channel for VirtualiZarr development and questions, feel free to join! earthmover-community.slack.com/archives/C08...

I joined Earthmover! Excited to focus on building tools for sharing scientific data full-time. This is a natural continuation of my work on open-source projects over the last ~6 years, which itself was motivated by seeing the same frustrations in every field of science I've ever worked in.

Relatively simple but very important quality-of-life improvement when accessing big climate datasets such as ERA5 and CMIP.

Next week I'll be giving a webinar on one of my favorite parts of Icechunk--the ability to apply git-style version control to Zarr arrays and Groups. This capability makes data workflows so more safe, robust, and reproducible. Come check it out! (Registration link below.)

Recording of the talk + lively community discussion is up! youtu.be/GZvJ0H89G0A If you're interested in these data cataloging questions then please also sign up to receive updates here forms.gle/yJY4Xp1Gvjd6...

Join us today, February 12 at 4 PM EST for @tegnicholas.bsky.social's presentation and moderated discussion on FROST: Federated Registry of Scientific Things! Everyone is welcome to show up via the zoom link - hope to see you there! discourse.pangeo.io/t/pangeo-sho...

I'm giving a talk tomorrow about what's technically required to make a decentralized social network for scientific data a reality! See this discourse link for zoom details: (discourse.pangeo.io/t/pangeo-sho...)

People may not understand just how important federal science research funding is to the prosperity of the nation. I tried to explain it, because it's under unprecedented and direct assault.

A grand challenge (from @al.merose.com ) alex.merose.com/pbfft/

Indivisible is urging people to visit their senators' local offices between now and Thursday to demand that they use every tool—including denying a quorum and blocking unanimous consent—to fight the confirmation of Project 2025 architect Russell Vought for head of OMB docs.google.com/document/d/1...

So many of my best stories have come from tips. I investigate sketchy technology, very broadly defined: privacy, tech entrepreneurs, surveillance, intersections w/ govt. Read 👇🏽 + then get in touch.

as always we need to be careful about what it is that we are using criticisms to actually do in the world and not just what political aesthetic we are trying to express

Must-read - by far the most clear-eyed description I've seen of the severity of what is happening. "[big tech executives] have quickly lined up to negotiate and pay bribes to the president that would allow their companies to operate unimpeded" "separatist province known as California"