SIG Repos: How should they work?

[Jeff]

Okay finally a full response for you @liketechnik (and I think this will help others too)

@liketechnik

How can we solve this without subjective “code-feel” guidelines? Two rules.
2.

I have a hard time following here and don’t quite get what you’re arguing for in the end

So imagine two pictures 1. A spider web 2. A tree (a normal/hierarchical tree)

• Right now nixpkg’s control flow is like a spiderweb, any piece of code is allowed to call/depend anything else at any time, including itself.
• The proposed repo structure (e.g the registry) allows (but doesn’t enforce) de-tangling the spider web.
• Those two rules, at least partially, enforce turning the spider web into a hierarchical tree.

Without those rules (or better rules) we would more than likely start making our own custom spiderweb.

@liketechnik

• registry - automated.

I’m vary on automating such ‘core’ things, as the registry.

I agree we should be cautious about automating things. This automation is for one file, and the only thing that is automated is the order of the attributes. It is effectively a sort function. No advanced logic or “intelligent” dependency detection.

• sig_sources and flakehub packages will say (or be annotated to say) “this X derivation depends on A,B, and C” (it’ll do this inside of json/toml or a yet-to-be-determined structured way)
• the automated system says “okay, I’ll make sure X appears below A,B, and C.” (And report an error if that is impossible because of circular dependencies)

That’s the only automated thing.

Doing the sorting by hand would be extremely time consuming and error prone considering multiple people editing upstream and downstream sources.

@liketechnik

To make sure I understand this all correctly:

• lib: generic pure nix ‘tooling’
• core: bootstrapping of build minimum amount of required toolchain(s)
• sig_sources: this is the bootstrapping of SIG/ecosystem toolchains AND ecosystem/SIG specific (pure) nix ‘tooling’?
• ecosystem: (language (or finer grained)/)SIG specific package sets + nix tooling building upon the bootstrapping in sig_sources
• registry: combines sig_sources + ecosystem together in a clever way to avoid the circular dependency issues
• polypkgs: tie everything together, to have a one-stop shop for those who want it

Thanks for echoing those. I think that’s the best way to identify confusion.

• core: (some room for discussion, but nbd)

• sig sources:

• registry: No, the registry doesn’t import ecosystem(s). If sig_repos were analogous to fishing, lumber, and mining organizations, then the registry would be an assembly line, and ecosystems would be the big-box retail stores and restaurants (lumber → assembly line → retail store, never the reverse order). I would also go as far as saying it’s not clever at all. It’s the most boring un-clever way possible to organize packages; a flat list in (effectively) chronological build order. Converting the highly-clever recursive categorically-grouped nested nixpkgs structure into the most dumb flat structure ever is what is really hard.

• ecosystems: doesn’t have derivation definitions. It exposes user-facing nix functions and it points to derivation definitions stored in the registry. If the user-facing nix function is ONLY user facing, it can be defined in ecosystem. If a function is used by maintainers to build packages and also happens to be user facing, then it needs to go in sig_sources and merely be imported into ecosystem.

• the aux namespace repo: will be like 3 files and ~20 lines of nix code possibly forever. “Update the flake lock and commit” will be its whole workflow. Ideally users use this endpoint and know nothing about the other repos. If they want lib, use aux.lib instead of pulling directly from the lib repo.

• polypkgs: while “tie everything together” is not necessarily wrong, it is probably misleading. Polypkgs is a drop-in replacement for nixpkgs, but we want to use polypkgs as little as possible. Sadly “as little as possible” is probably still going to mean “all over the place for everything” at first but oh well. However, unlike the aux repo, polypkgs is going to need a notable amount of nix code. At first it will be simple, but as we overlay more and more things it is likely to grow. Sometimes overrides need extra logic to work correctly.

@liketechnik

I have hard time imagining that we can break forward compatability that easy, even if it’s just internally.
I might be wrong, but I think nixpkgs had problems with this in a monorepo - and we’re talking about multiple repos needing to synchronise on this here.

Think of it like this: we deprecate a function in the internal lib, and we do a quick search to see if that deprecated function name exists in any aux repo. If the answer is no, we can safely remove the deprecated function from the internal lib. Nice and clean. With the public lib we can never ever ever ever remove/rename a function because it could break an unknown number of other peoples nix code (which is supposed to not bitrot). We see this already with deprecated junk like toPath in nix. We can’t move very fast when changing lib because every mistake is a permanent mistake. For the internal lib we could move a little faster knowing there is room to fix our mess/mistakes later.

@liketechnik

not that important, in the end the whole think still kinda makes sense to me.

Home Manager probably would live in the ecosystem space

Food for thought: are we conflating things that should be kept apart here?

@liketechnik

Enable stuff like the dev-shell mixin experience (ex: ecosystems.aux.tools.mkDev [ ecosystems.nodejs.envTooling.typescript ecosystems.python.envTooling.poetry ecosystems.latex.envTooling.basics ])

And in spirit with the last point: where do we see the dev env story?

Sorry if my mention of home manager and dev tooling was confusing. They were supposed to be examples to explain the intuition.

Key idea: all packages (derivations) are fully defined before ecosystem even begins. If it has to do with packages then it should be handled in SIG sources, flakehub (e.g. external), or registry (and of course core packages will be handled in core).

Is [dev env] something that needs to be kept together with packaing (for technical/maintenance reasons)?

In practice, I don’t know! We will have to try and see. If it doesn’t get used for packaging, then it can stay in ecosystem. If it does get used for packaging (or if the maintenance team simply finds it helpful) then most of the code will live in sig_sources, then ecosystem will just reference it.

why focus on nix cli for core?

TLDR: it’s a well defined starting point. Its not a must have. We can talk about it more later.

@liketechnik

Polypkgs

• Its own repo so that tarball-urls are easy drop-in replacements for nixpkg tarball urls
• If nixpkgs gets a commit, we generate a new flake.lock

I think we ended up somewhere that it’s best to actually fully separate our public
package sets from nixpkgs,
since providing nixpkgs is increased maintenance complexity for use,
but not much benefit to our users (they can just pull in nixpkgs themselves).
(this was the discussion in On the future of our nixpkgs fork)
(Not fully sure this is the right conclusion, now that I think about it:
I’m not knowledgable enough to know if this is also just as easy when not using flakse,
and us pulling in the nixpkgs has the (implicit) guaranteee that the nixpkgs
we provide is compatible with auxpkgs - which the user pulling it in doesn’t have)

I’m a bit confused on what this part is saying. My understanding is

1. We need polypkgs internally while we bootstrap
2. The easiest way for us to test aux will be to start swapping out nixpkgs with polypkgs in our own projects
3. Other people might like our maintenance. For example, if we fix/maintain an unmaintained package on nixpkgs, then others might use ploypkgs to get that fix and (inadvertently) help us out my testing our code and maybe even contributing.