SIG Repos: How should they work?

[getchoo]

@imadnyc

In my example, it would create an overlay of the derivation and copy it over, so you’re not importing a new version of the repo you’re copying from. In the example, python isn’t instantiating a new version of JS SIG, but overlaying the input to C with a local derivation that now resides in Python SIG. This would be then copied back over the JS SIG and deleted from python by the bot keeping track, so it wouldn’t have another input to JS SIG’s top level that wasn’t already imported in Python’s top level. Of course, this doesn’t solve the issue of flake import circles.

i think this proposal is a very good example of exactly why we should be avoiding circular dependencies. throwing automatically created overlays around and ensuring their creation and deletion across multiple repositories is not just a massive technical hurdle in both the context of nix and creating a bot to do this in the first place, but also a huge point of failure. if any step in this process were to go wrong, it would most likely require manual intervention that i’m betting very few people would even be able to perform given the extreme complication of this process. i don’t see why we should increase our bus factor and inter-tree workflow complication just to support circular dependencies

@imadnyc

The issue with making sure that each SIG is “pure” is that it creates a ton of maintainer burden

i’m not sure how? the SIGs with the most work to go through (language specific ones) would only need to worry about PRs being sent to them, and occasionally updating core for packages deemed important enough to be shared at our lowest (highest? i still haven’t made up my mind!) tier at core. i’m pretty sure that regardless of any solution we choose, dealing with core and your own PRs will be a consistent burden; this isn’t introducing anything new.

as i said in my proposal, the only place this would possibly increase maintenance burden in top level. quoting myself from there:

the main cost here would be the overriding of inputs in top-level. this would require some additional testing in top-level, as we can no longer guarantee perfect reproducibility from the original repo. i think this could create a good place to introduce an equivalent to nixpkgs/nixos’ hydra jobsets though, as individual repos could act more as a master , nixpkgs-unstable , or nixos-unstable-small while top-level would be more akin to nixos-unstable .

i think top-level acting as nixos-unstable would make sense regardless of our solution here, as it is the “final stop” so to speak and what any nixos do we have an actual alternative name for our fork yet? i haven’t been keeping up on some things should be using

@imadnyc

and slowly creeps towards a monorepo, no?

from my response to vlinkz voicing the same concern:

is this not already the case? top-level is quite literally all of our repos combined. for all intents and purposes, it is inherently a monorepo

@imadnyc

For each combination we need to pass it off to a specific SIG (JS + Python SIG, JS + Python + Rust SIG)

this should never need to happen. i’ve been very explicit about language specific repositories not depending on each other. this should be reserved for SIGs larger than a single language like gnome and plasma

@imadnyc

or we give it to top-level, which will increase in size as time goes on

this will be happening regardless, given it’s a combination of all of our repos. unlike nixpkgs though, it has a practical cap on how large it’s individual package count can get, as any language specific package (like a vast majority of nodePackages, pythonPackages, haskellPackages in nixpkgs) will not be allowed in it. higher (lower? will i ever decide?) tiers preceding top-level like gnome and plasma will also take even more of the burden off top-level (something i didn’t fully realize the potential of until vlinkz mentioned it, so thanks!)

@imadnyc

And each SIG gets smaller and smaller as the dependency graph gets more hairy no?

im a bit confused by this point, but i think it’s continuing with the assumption of python depending on javascript, etc? i can’t really provide anything here without more clarification

@imadnyc

If not I’d appreciate more clarification, I admit I’m having a little trouble picturing the graph in my head.

i would highly recommend not just you, but anyone who wants a bit more insight into what exactly i’m suggesting to read my original proposal here along with the follow-up to vlinkz’s concerns here. i feel some points may have been missed with these conversations happening in parallel across two platforms, as a few of these concerns have already been brought up and addressed there