Post

Apologies! I used the wrong words - I meant conda-forge. I have been a bit confused about the use of conda-* for R packages, since I strongly associate conda with Python, and I'm still not really sure what problem conda-forge is solving? Is it building binaries for specific operating systems?

I think the "conda-forge -> Python" track is the hardest thing to shake in people's brain (which I guess is due to the huge success it had in building gnarly wheels). @conda-forge.org is cyberinfrastructure to build and distribute conda packages and conda is a language agnostic packaging format.

conda packages also implement a bootstrapping system (requiring all dependencies to also be conda packages) all the way down to the low-level C library requirements (e.g. glibc for Unix) so that they effectively are operating system agnostic for a particular OS family.

So conda-forge builds for operating system and computing architecture computing platforms (e.g. linux-64, osx-arm64, linux-aarch64, linux-riscv64) from a single recipe file which for the end-user very nicely avoids having to now be concerned about linux-distribution specific packaging.

That combined with having everything on conda-forge be built in a controlled manner where the whole ecosystem advances through automated rebuild migrations means that you can get an entire polyglot system trivially that has been built in mostly optimized ways the whole way down to the compilers.

While conda-forge is fundamentally "just another" conda "channel" (aka, conda package index) it underpins other more field-specific channels that allow for their builds to sit atop it. So instead of everyone having to rebuild their own things over and over, conda-forge can build (most of) the world.

Which means with @prefix.dev's Pixi I can do: ``` pixi init example && cd example pixi workspace platform add linux-64 osx-arm64 win-64 time pixi add --no-install r-sf ``` and in just barely over 2 seconds have a fully resolved lock file down to the digest level for all 3 platforms for all deps.

That was in a clean Docker container with no cache so that resolution was the slowest it will ever get. Similarly, on my local WiFi to actually install with no previous cache ``` # time pixi install The default environment has been installed. real 0m11.016s user 0m11.621s sys 0m3.915s ```

So that sounds awesome! I can imagine that this is really great in situations where you want to install packages quickly and have all the dependencies resolve. Like if I was an on a HPC system? However, most R users just want to run install.packages("new-package") and be done with it.

It sounds like the problem conda-forge is solving is making pkg installation easier and faster? I don't mean to be playing devils advocate too hard, but I am curious - what problem conda-forge is solving? The average R user doesn't really reach for the terminal to manage R pkg installation.

> Like if I was an on a HPC system? I would say that, yes, having optimized builds that are maybe only a bit behind thought-out Spack builds is a boon across all computing (laptop, HTC, HPC, cloud).

I will also give you that to an R user the idea of controlling your software environment _outside_ of your session is strange. I think it is just a difference between how R and Julia approach the concept of computing and most other languages and software env management tools do this.

Though github.com/eddelbuettel... is the same sort of stangeness for an R user, no? Though "run install ...and be done with it" doesn't really happen with the default of CRAN if you're on Linux. If you want binary installs you need to modify your setup to use P3M and whatever r-universe(s) you need?

As to "what is the problem being solved?" I think conda-forge solve multiple which I can pitch as questions: * How can I get binary builds of all the tools in my environment? * How can I distribute my software for multiple computing platforms through a single index?

* How can I get powerful domain tools without having to build from source? * How can I be guaranteed to get builds that will be compatible with custom constraints and _all_ other software in my computing environment? * How can I get a digest-level reproducible software env without extra tools?

Those might not be things that you actually care about though if a person isn't hitting them in their daily workflow. So I'll also provide an R user's perspective from Chris Toney, maintainer of gdalraster: firelab.github.io/gdalraster/a... cc @mdsumner.bsky.social given a past discussion too

I've been meaning to compare my docker layers hypertidy/gdal-r-ci (latest) to conda, I could not find any other way to align versions than to do the Linux side all myself (but I don't even have conda recipes for my own packages like Chris recently added).

It's just not in my view and being all-system or all-pak/all-r2u seemed the only options (can control oak if your careful but it could have a "not-list" rather than carte blanche imo (I've scoped a PR)

If conda really aligns versions also across platforms and is recent that's very attractive but I can't see cran moving from its own builds for macos/windows

The history for me was that about ten years ago "anaconda solves everything" was forced down our throats by local admins and it was so blatantly not fit for purpose. I just didn't go back it was so ridiculous. Appreciate that that be as experience coloured my views too long 🙏

> If conda[-forge] aligns versions ...across platforms and is recent that's very attractive Yes, see some of the points in Chris's Zulip server response (linked in bsky.app/profile/matt...).

> ten years ago ... it was so blatantly not fit for purpose. I just didn't go back it was so ridiculous. I had a very similar experience in 2014 and I described myself as a "conda hater". Then in 2023 Pixi was created, I learned about it, I tried it out and I fully changed my views.

I realized that the conda packaging standards are great and I had really bounced off hard of the tooling and infra at the time. I'm now a big proponent of conda-forge (as you can tell by me not shutting up about this). :P

nvm, on it now - at the higher layers mine should be a conda env. Bug refactoring coming over the next week to gdal-r-ci