Another option could be something like josh to keep a monorepo but make contributing to it easier. This would probably require some restructuring though.
I like the idea of using Aux to achieve technical advancements but I would not like to end up building upon sand pilars due to rushing over the implementation.
josh sounds excellent if we go with a monorepo setup! I think this is what TVL uses for their monorepo too if anyone wants to see it in action:
- all git clone instructions in depot: context:global repo:^dep… - Sourcegraph
- the josh module: josh.nix - depot - Sourcegraph
I’m still in support of moving away from a monorepo. I haven’t used josh before so I don’t have much of an opinion on it. Either way restructuring would be a huge task. I feel like in the restructuring process we have two main methods of going about it:
- Fork nixpkgs and break it up into its new structure
- Create a new repo that depends on nixpkgs as a flake input and work towards migrating everything into the new repo (or set of repos) to the point that we can drop the nixpkgs dependence
Edit: I feel the latter option may be more manageable as it allows us to slowly ramp up to maintaining our set of packages instead of maintaining 80000+ right at the start
Create a new repo that depends on nixpkgs as a flake input
I think this is going to be the only way to maintain momentum.
We can do both decentralized and a monorepo. The monorepo is just a collection of pinned versions that work well together.
auxpkgs (curated, not top-level)
packages
(flat, all values are derivations)packageManagers
(exactly two deep, ex: rustCargo.tokio)- mark each package derivation, in the meta field, as both an aux package and with the status (properly maintained, ported but up to aux-standards, or patched by aux for functionality but not compliant with standards)
- every package is given a directory with default.nix
- and the default.nix is always a function that takes an attrset argument and returns a derivation
- it always imports a flake from another repo
- the repo URL is always imported from a serialized file (like json or toml) so that changes can be automated
top level
- inherit nixpkgs
- override with auxpkgs
Start with just using nixpkgs, and experiment with overriding packages as aux maintainers maintain them.
Breaking things into core can be a separate project.
I think for Stage 1, since we inherit a monorepo anyways, the simplest option is to give each SIG a branch/fork and then have the CI try to perform wave merges. This still gives SIGs the ability to merge at their pace.
@_deleted I’m really curious about this, and I think its relevent.
Lets say auxpkg wants to overwrite package abc
in nixpkgs. What kind of problems could come up?
Like assuming the worst possible senario, that abc
is heavily used all over nixpkgs and maybe even inside of bootstrapping.
I think overwriting is the wrong way to go about it. Instead, just have flake output for each. Still working it out, but something like this:
{
inputs = {
nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
};
outputs = { self, nixpkgs, ... }:
let
lib = import ./lib;
forAllSystems = lib.genAttrs lib.systems.flakeExposed;
in
{
nixPackages = forAllSystems (system:
(import nixpkgs { inherit system; })
);
auxPackages = forAllSystems (system:
(import ./. { inherit system nixPackages; })
);
};
}
So you can depend on nixPackages.<pkg>
until there is a suitable replacement
Well lets say openssl has a vulnerability. The Aux maintainers patch it so auxPackages.openssl
has the fix. Well everything in nixPackages doesn’t get the fix, and other stuff in auxPackages
might use nixPackages.python, which in turn uses nixPackages.openssl instead of the patched auxPackages.openssl
.
I think we will have to do an override on nixPackages. Which is recursive and going to make stuff hard to debug, so we should design carefully. But I think its necessary to be able to maintain without full direct forking.
As a temporary bandaid fix yes, but I don’t want to set a precedent of auxPackages
just being overrides of nixPackages
. A better ideal goal in that situation is bring python
into auxPackages
Patching openssl is going to mean we need to pull everything that uses openssl into auxPackages. Which is on the order of half of all packages including core. I don’t think that’s realistically within our ability.
Agreed, it’s not. Overriding may become necessary for security reasons. The long-term goal however should be to not depend on nixpkgs at all, so I believe when possible we should package ourselves rather than overlay. You’re right for something like openssl that may not be possible
I agree. Aux should absolutely not be advertised as “an overlay for Nix”. Its just a way for us to maintain core components with O(1) sized increments (which is still potentially huge) instead of O(n) sized increments.
What we can do is establish policies like, when a package is pulled into auxPackages, we do a build and trace every nixpkgs attribute that the build touches. (e.g. auxPackages.python would have nixpkgs.gcc and nixpkgs.openssl on that “touched” list). We would mark/record this list in the meta attribute, and once all the things on that list have been overridden with pure auxPackages (e.g. nixpkgs.gcc == auxPackages.gcc
, and auxPackages.gcc.meta.pureAux == true
) then we can change the meta and say auxPackages.python.meta.pureAux = true
and do basically a find-and-replace on the code (swapping the nixpkgs var out with auxPackages var)
Then eventually, there won’t be anything left pulling in nixpkgs.
As much as I dislike using fetchers instead of inputs this is probably the best solution here (at least until lazy inputs are a thing). This would pull a minimal amount of code for the user and is reasonably easy to update. In fact I’m pretty sure we could use Drift to do it automatically.
Re: docs location
I think this is the direction things were leaning in. Is that right @minion & @coded?
Also I recognize your name, glad to have you around
Correct, I wholeheartedly agree
As this is only talking about the packages itself AFAICS: What’s the stance on all the modules aka. nixos/modules/
?
Should/will they be separate from the packages entirely?
Are there already any ideas regarding that? Since that will come into importance rather sooner than later too.
I believe we will be doing the same with NixOS, moving it to a separate repository and creating a group responsible for its work.
(sorry for the wall of text, I tried to make it legible)
Until something like loomio is up, maybe using the wiki as a stopgap could help with a discourse topic for each proposal? I’m a bit lost with the proposals.
So far I see:
- mono repo:
- josh: Monorepo with a git proxy in front? Never used it myself, but I do wonder what pull/merge requests will look like
- branch per SIG: not sure if
master
will be periodic merges from SIGs?
- multi repo:
- per domain
- per programming language: some projects use multiple languages?
- mixed:
- monorepo = collection of pinned versions: packages in their own repos/domains/languages and mono repo pins them?
Probably missed others… discourse’s lack of a threaded view isn’t helpful
I’m not entirely convinced multi-repo is a good option. It feels more natural e.g a language has a standard library and new projects using the standard library don’t end up in the same repo.
However how are dependencies tracked? And if a dependency has to be updated that might require pull requests across multiple repositories. At best it’s tedious, at worst, it things can be forgotten and security fixes aren’t propagated.
Maybe I missed it, but my major question is What are we trying to fix? Which difficulties is the monorepo providing?
Each proposal should be investigated for their weighed strengths and weaknesses. Something might be a negative but a minor inconvenience, some things might be positive and a massive boon. Having a scale and a method of counting to reach a score could be quite helpful.
As an example, a scale of:
- 0 : Unimportant
- 1: Convenient
- 2: Important
- 3: Very important
- 1000: Dealbreaker
Feature/Risk | Weight |
---|---|
Looks good | 1 |
Allows domain ownership | 2 |
Tooling exists | 2 |
Supported by forge | 3 |
Feature/Risk | Proposal Jaba | Proposal Doobie |
---|---|---|
Looks good | yes | no |
Allows domain ownership | yes | yes |
Tooling exists | yes | no |
Supported by forge | no | yes |
Total | 6 | 5 |
The total is calculated by simply converting “yes” to 1
and “no” to 0
, multiplying it by the feature weight, and summing it up → sum([ weight * boolean for weight, boolean in column.jaba])
.
This would double as a historic record and help during decision making by allowing to see:
- What are the problems
- How important are they to the community
- Which proposals there are
- Which feature/risk every proposal has
- The mathematical “victor”