NixPkgs supports multiple versions for certain blessed packages. Node, Python, etc each have a canonical “latest” version as well as historical versions. However, most packages do not work this way. The questions for us to answer then are:
Do we need to provide multiple versions of packages?
Why not provide multiple versions for all packages?
What is the best way to provide these packages?
As you may guess from reading the latter two questions, the first’s answer is: yes. We do have to have multiple versions for at least packages with multiple parallel supported releases. These packages are often the ones that NixPkgs handled specially (Node, Python, etc).
Then, if we are providing multiple versions, why stop at just these packages? Wouldn’t it be a more pleasant experience to be able to say something like pkgs.bat.v1_12_0 instead of having to reference an old commit (with all the atrocious tooling that relies on)? I believe we can improve the story here for multi-version management of packages. This would also make it possible for specific problematic packages to be linked using prior versions of a dependency when absolutely necessary (otherwise defaulting to the latest).
The final question is the most difficult to answer: how. How do we organize things such that we can provide these packages with multiple versions. As I see it there are some problems that may need addressing:
Consistent accessors for versioned packages (eg. package.v1_0, package.versions.v1_0, or similar)
Different versions may depend on different dependencies
Different versions may require different build steps
Information for all versions and their install requirements must be tracked
… likely more unknowns …
I’d like to use this thread to brainstorm ideas for handling this case. Please feel free to add your thoughts to the discussion!
consistency might be complicated a bit as there will be a handful of packages that use their own esoteric versioning scheme. The package.version syntax you described would be fine for this, but we should be careful not to make any assumptions about semver
also would it even be possible to provide versioned packages for rolling release projects, or would we have to just provide whatever the latest build is at the time of that aux packages revision
Supporting multiple versions for all or at least most packages is something that I’ve always missed from nixpkgs, myself. And while it’s true that you can just pin multiple commits to achieve a similar effect, doing that always felt very hacky… Not to mention that it adds more fuel to the fire that is having thousands of instances of nixpkgs.
I personally think this is a pretty good idea, I really like the package.versions.v1_0 syntax for specifying a specific version, while just package refers to the latest version. Implementation wise, I think it would make sense to keep multiple derivations for multiple versions, but trying to reduce duplication as much as possible.
For example, if versions v1.0 through v1.5 of a package can be built with mostly the same expression but with a couple of minor changes (such as changing the src and/or adding a small command to post-build, etc) then that should be a single file, then if version v2.0 changes massively and requires a new expression entirely, that should be a new file entirely. The package default.nix could then just point to the latest file, and pass through some information about the other versions
As for caching, I don’t think every version needs to be cached. This should be evaluated on a case-per-case basis, but generally I think caching the latest version and any versions used by a currently supported release is enough.
I’ve been pondering around the general question of multiple versions recently myself as I was thinking about ways how we might be able to achieve a better end experience regarding security updates.
A few (maybe random) thoughts along that path
As versions change, so can inputs/ dependencies and a known vulnerability status. I feel there is a need to be able to have derivations for each version that differ in more than just the version number and the hash.
Relying/ forcing semantic versioning is not a great idea. The versioning scheme is typically given by the creators of the software
Some cool benefits
Maybe we could have a 9999 auto-created version tag (like gentoo has) which pulls the latest sources from the upstream repo everytime it is build. Not sure how to handle the hash in that situation though.
The fetchers in nixpkgs all require hashes but most of them are based on builtins.fetchTarball which verifies the hash if you provide one but doesn’t require it.
The downside being that Nix is not as flexible with name characters so we would have to use quotes everywhere like packages."bat@3.0". While we don’t want to use patterns like with a bunch, it can still be useful for cases like package lists which would not be possible with this pattern.
Hm. I may be misunderstanding the Guix docs, but I understood the name@version syntax to be something that’s handled by the CLI. That is, in Guix a package is actually a list of packages, where each element in this list is a different version of the package, e.g hello = [<package definition for hello@1>, <package defintion for hello@2>, ...] - and the CLI just selects the right version.
In contrast, today’s nixpkgs usually has an attribute set for packages with multiple version, e.g. hello = { hello_1 = <package defintion for hello@1>, hello_2 = <package definition for hello@2>, ...}.
I think I prefer the latter, since it’s easier to pick a specific version. Obviously the exact layout of the attribute set is up for discussion, but I aggre that
I’m happy to say, I think these answers are great. Stuff like python38 has long bothered me.
I don’t have any feedback, other than I care about versioning a lot. My first nixpkgs issue, before I had done a single project with nix, was with versions. I couldn’t figure out how to install a specific version of nodejs. Ever since then, there’s been a github issue “Canonicalizing version strings” (or something like that) which I revisit each year. Basically all my other work (automatedly refactoring nixpkgs, redesigning nixpkgs/flakehub to be a distributed system, the shared-inputs problem, solving the LD_LIBRARY_PATH issue, etc) all of it originates from that one issue.
Last year the little nix version search CLI I threw together supported doing nvs --install python@3.10. Which was a huge win for me after so many years. So I love the direction this is thread has been going. I really think nix needs to be as easy to use as asdf even if we need to move mountains to get there.
For those that did read the sig_sources/registry/ecosystem design, I want to mention pkgs.${pkgName}.versions."v${version}" will work fine in ecosystem, but registry can’t have that format while keeping all the mentioned benefits. For discoverability/automation every derivation in registry needs to be only 1 attribute deep. It’ll be something worth discussing more, but I need to get to bed.
Assuming they don’t provide their own versioning scheme, we could use some sort of Calendar Versioning scheme. Something like package.20240515? Or if the time is important, package.20240515_0912
They do so by indexing all historical versions of nixpkgs and pulling older package sets to get specific versions of packages. For example the python page on nixhub shows the nixpkgs hashes for the different versions:
Ok that’s about the approach I’d expect. I was hacking on a terrible solution for this using git bisect and git tags before
One more thing for thought - I’ve seen the arbitrary string normalization thing before, I think it was Ruby’s Bundler that had to normalize FQDNs into environment variable keys. You want as lossless a conversion as possible lest you get clashes. They went with a special conversion of dashes into triple underscores.
