First of: that’s a very impressive write-up!
I’ll put my thoughts below quotes that are (hopefully) indicative of the exact place in your text (for lack of a better term) I’m talking about.
I can confirm this is not possible. Python’s R module has the entire R language (and R modules) as a dependency. Other languages have python as a dependency.
Hmm, I’m not yet convinced that this means we have to introduce circular dependencies (which I think this really is about)
For example for the node-gyp thing (Jupyter Notebook (python) ← node-gyp (node) ← python) thing,
could be solved by having a python/js core (the naming is up for bikeshed, e.g. ‘bootstrap’ might better convey the purpose) repo
which only sets up these cross-language concern,
and a pure python repo on top.
After fully reading your proposal: Nice, I belive this is what you already came up with, lol.
Here is a structure that I think could get put-to-work this week and later.
- core - a reorganized attrset of minimum-packages needed to build the nix cli (which I know how to create thanks to nixpkgs-skeleton)
why focus on nix cli for core? I.e. what is the thing that differentiates nix-cli from other packages (e.g. compilers) (and with lix planning to depend on rust: this would pull in the SIG Rust stuff, depending on exact repo layout. How would we handle that? I.e. what is core and what is SIG <language/ecosystem>)
I’m vary on automating such ‘core’ things, as the registry. I’m scared this makes it much more difficult to fix/understand if stuff breaks (a similar concern was raised in SIG Repos: How should they work? - #30 by getchoo, albeit with a possible more complex scheme behind it (very first paragraph))
- Lib - Should probably be split into stdlib (aka just lib, always forward-compatible) and internalSupport (aka stuff the aux-monorepo(s) might need but the rest of the world might not need, and also might break forward compatibility)
I have hard time imagining that we can break forward compatability that easy, even if it’s just internally.
I might be wrong, but I think nixpkgs had problems with this in a monorepo - and we’re talking about multiple repos needing to synchronise on this here.
While still complicated, making these stages explicit is, I think, the only way to make this stuff even barely manageable. Just imagine the difference between “Error: autoconf_stage2 failed to build” compared to “Error: autoconf (one of multiple generated at runtime) failed to build”.
Someone who has more experience with how nixpkgs currently bootstraps needs to wheigh in here, but to me this sounds extremely reasonable.
… Cross Compilation …
I don’t quite understand why a cross-compiled package has more dependencies. In my mind it just has different packages (different compiler mainly, since the target arch is different).
How can we solve this without subjective “code-feel” guidelines? Two rules.
2.
I have a hard time following here and don’t quite get what you’re arguing for in the end. It would be really kind if you (or somebody else) could try to reword this, in order to help me understand this. But it’s not that important, in the end the whole think still kinda makes sense to me.
Home Manager probably would live in the ecosystem space
Food for thought: are we conflating things that should be kept apart here? I.e. package building/configuration and system (in the sense of system (host services, networking, boot, filesystems, etc) but also user configuration (dotfiles, user services, i.e. home manager, etc) configuration?
I think we should keep these apart from each other, i.e. the configuration stuff (todays nixos and home manager) should be kept different from packaging stuff (todays bootstrapping (the different stages), pythonPackages, rPackages, luaPackages, vscode, neovim, etc).
And this discussion should focus solely on the packaging aspect (which is then consumend by the configuration stuff).
Enable stuff like the dev-shell mixin experience (ex: ecosystems.aux.tools.mkDev [ ecosystems.nodejs.envTooling.typescript ecosystems.python.envTooling.poetry ecosystems.latex.envTooling.basics ])
And in spirit with the last point: where do we see the dev env story?
Is it something that needs to be kept together with packaing (for technical/maintenance reasons)?
Or is it both plausible and sensible to separate it too?
While registry needs to be rigourously consistent in order to be automated, ecosystems only need to be consistent to help with ergonomics.
To make sure I understand this all correctly:
- lib: generic pure nix ‘tooling’
- core: bootstrapping of build minimum amount of required toolchain(s)
- sig_sources: this is the bootstrapping of SIG/ecosystem toolchains AND ecosystem/SIG specific (pure) nix ‘tooling’?
- ecosystem: (language (or finer grained)/)SIG specific package sets + nix tooling building upon the bootstrapping in sig_sources
- registry: combines sig_sources + ecosystem together in a clever way to avoid the circular dependency issues
- polypkgs: tie everything together, to have a one-stop shop for those who want it
Then it also makes sense why you list these functions under ecosystems, and not sig_sources to me.
Aux
- Having one layer before getting into packages is important for future expansion
I don’t quite get the example, but I belive what you’re getting at is that we need an abstraction layer between the actual layout of things
and how we present them to users, so that when our layout changes, we can maintain backwards compatability in this abstraction layer.
With that I fully agree. And I like the idea to have specific place for this layer!
Polypkgs
- Its own repo so that tarball-urls are easy drop-in replacements for nixpkg tarball urls
- If nixpkgs gets a commit, we generate a new flake.lock
I think we ended up somewhere that it’s best to actually fully separate our public
package sets from nixpkgs,
since providing nixpkgs is increased maintenance complexity for use,
but not much benefit to our users (they can just pull in nixpkgs themselves).
(this was the discussion in On the future of our nixpkgs fork)
(Not fully sure this is the right conclusion, now that I think about it:
I’m not knowledgable enough to know if this is also just as easy when not using flakse,
and us pulling in the nixpkgs has the (implicit) guaranteee that the nixpkgs
we provide is compatible with auxpkgs - which the user pulling it in doesn’t have)
All in all, I feel this really goes in the right direction; while I think there’s some discussion to have on certain details, the overall architecture/broader idea seems very solid to me, and I especially like the idea on making the bootstrapping and library parts of what today is all “nixpkgs” more distinguished in however we will call the different parts of auxolot packaging (I.e. going even further than just distinguishing nix the build tool, nixos the system configuration and nixpkgs the package set).
P.S.: I’m slowly starting to doubt that the complexity of it all can be appropriately handled in a discourse thread. Yesterday someone shared the CQ2 tool on in the matrix chat. But I really don’t wanna rush introducing yet another tool without proper discussion & consideration & approval of the current interim leadership, so this best stays here in the P.S. for now, and if we come to the point where we really feel productive discussion is simply not possible due to the complexity exceeding what discourse’s format can handle, we can revisit this.
PPS: For some details on the ecosystem/polypkgs package sets I think we should also take the thoughts on how to structure our new packages in On the future of our nixpkgs fork - #13 by getchoo into account (stuff like: do we want to namespace packages similar to e.g. gentoo or aur).