Musings on a monorepo versus developer oriented distribution

General
21

Create a new repo that depends on nixpkgs as a flake input

I think this is going to be the only way to maintain momentum.

We can do both decentralized and a monorepo. The monorepo is just a collection of pinned versions that work well together.

auxpkgs (curated, not top-level)

  • packages (flat, all values are derivations)
  • packageManagers (exactly two deep, ex: rustCargo.tokio)
  • mark each package derivation, in the meta field, as both an aux package and with the status (properly maintained, ported but up to aux-standards, or patched by aux for functionality but not compliant with standards)
  • every package is given a directory with default.nix
    • and the default.nix is always a function that takes an attrset argument and returns a derivation
    • it always imports a flake from another repo
    • the repo URL is always imported from a serialized file (like json or toml) so that changes can be automated

top level

  • inherit nixpkgs
  • override with auxpkgs

Start with just using nixpkgs, and experiment with overriding packages as aux maintainers maintain them.

Breaking things into core can be a separate project.

4 Likes
22

I think for Stage 1, since we inherit a monorepo anyways, the simplest option is to give each SIG a branch/fork and then have the CI try to perform wave merges. This still gives SIGs the ability to merge at their pace.

1 Like
23

@_deleted I’m really curious about this, and I think its relevent.

Lets say auxpkg wants to overwrite package abc in nixpkgs. What kind of problems could come up?
Like assuming the worst possible senario, that abc is heavily used all over nixpkgs and maybe even inside of bootstrapping.

24

I think overwriting is the wrong way to go about it. Instead, just have flake output for each. Still working it out, but something like this:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  };

  outputs = { self, nixpkgs, ... }:
    let
      lib = import ./lib;
      forAllSystems = lib.genAttrs lib.systems.flakeExposed;
    in
    {
      nixPackages = forAllSystems (system:
        (import nixpkgs { inherit system; })
      );

      auxPackages = forAllSystems (system:
        (import ./. { inherit system nixPackages; })
      );
    };
}

So you can depend on nixPackages.<pkg> until there is a suitable replacement

1 Like
25

Well lets say openssl has a vulnerability. The Aux maintainers patch it so auxPackages.openssl has the fix. Well everything in nixPackages doesn’t get the fix, and other stuff in auxPackages might use nixPackages.python, which in turn uses nixPackages.openssl instead of the patched auxPackages.openssl.

I think we will have to do an override on nixPackages. Which is recursive and going to make stuff hard to debug, so we should design carefully. But I think its necessary to be able to maintain without full direct forking.

26

As a temporary bandaid fix yes, but I don’t want to set a precedent of auxPackages just being overrides of nixPackages. A better ideal goal in that situation is bring python into auxPackages

1 Like
27

Patching openssl is going to mean we need to pull everything that uses openssl into auxPackages. Which is on the order of half of all packages including core. I don’t think that’s realistically within our ability.

1 Like
28

Agreed, it’s not. Overriding may become necessary for security reasons. The long-term goal however should be to not depend on nixpkgs at all, so I believe when possible we should package ourselves rather than overlay. You’re right for something like openssl that may not be possible

29

I agree. Aux should absolutely not be advertised as “an overlay for Nix”. Its just a way for us to maintain core components with O(1) sized increments (which is still potentially huge) instead of O(n) sized increments.

What we can do is establish policies like, when a package is pulled into auxPackages, we do a build and trace every nixpkgs attribute that the build touches. (e.g. auxPackages.python would have nixpkgs.gcc and nixpkgs.openssl on that “touched” list). We would mark/record this list in the meta attribute, and once all the things on that list have been overridden with pure auxPackages (e.g. nixpkgs.gcc == auxPackages.gcc, and auxPackages.gcc.meta.pureAux == true) then we can change the meta and say auxPackages.python.meta.pureAux = true and do basically a find-and-replace on the code (swapping the nixpkgs var out with auxPackages var)

Then eventually, there won’t be anything left pulling in nixpkgs.

2 Likes
31

As much as I dislike using fetchers instead of inputs this is probably the best solution here (at least until lazy inputs are a thing). This would pull a minimal amount of code for the user and is reasonably easy to update. In fact I’m pretty sure we could use Drift to do it automatically.

1 Like
32

Re: docs location

I think this is the direction things were leaning in. Is that right @minion & @coded?

34

Also I recognize your name, glad to have you around :slight_smile:

1 Like
35

Correct, I wholeheartedly agree

36

As this is only talking about the packages itself AFAICS: What’s the stance on all the modules aka. nixos/modules/?
Should/will they be separate from the packages entirely?

Are there already any ideas regarding that? Since that will come into importance rather sooner than later too.

2 Likes
37

I believe we will be doing the same with NixOS, moving it to a separate repository and creating a group responsible for its work.

5 Likes
38

(sorry for the wall of text, I tried to make it legible)

Until something like loomio is up, maybe using the wiki as a stopgap could help with a discourse topic for each proposal? I’m a bit lost with the proposals.

So far I see:

Probably missed others… discourse’s lack of a threaded view isn’t helpful :confused:

I’m not entirely convinced multi-repo is a good option. It feels more natural e.g a language has a standard library and new projects using the standard library don’t end up in the same repo.
However how are dependencies tracked? And if a dependency has to be updated that might require pull requests across multiple repositories. At best it’s tedious, at worst, it things can be forgotten and security fixes aren’t propagated.

Maybe I missed it, but my major question is What are we trying to fix? Which difficulties is the monorepo providing?
Each proposal should be investigated for their weighed strengths and weaknesses. Something might be a negative but a minor inconvenience, some things might be positive and a massive boon. Having a scale and a method of counting to reach a score could be quite helpful.

As an example, a scale of:

  • 0 : Unimportant
  • 1: Convenient
  • 2: Important
  • 3: Very important
  • 1000: Dealbreaker
Feature/Risk Weight
Looks good 1
Allows domain ownership 2
Tooling exists 2
Supported by forge 3
Feature/Risk Proposal Jaba Proposal Doobie
Looks good yes no
Allows domain ownership yes yes
Tooling exists yes no
Supported by forge no yes
Total 6 5

The total is calculated by simply converting “yes” to 1 and “no” to 0, multiplying it by the feature weight, and summing it up → sum([ weight * boolean for weight, boolean in column.jaba]).

This would double as a historic record and help during decision making by allowing to see:

  • What are the problems
  • How important are they to the community
  • Which proposals there are
  • Which feature/risk every proposal has
  • The mathematical “victor”
2 Likes
39

could help with a discourse topic for each proposal? I’m a bit lost with the proposals.

I’m struggling myself too, but let me try to shed some light onto the current discussion in this area:

I do think we’re starting to debate ‘the same’ topic in three different places now, or rather three so closely connected topics, that they heavily influence each other:

  • in this topic, we’re debating mono- vs multi-repo
  • in SIG Repos: How should they work? we’re already discussing how our multi-repo approach “will look like” (in the sense that ‘mono-repo’ isn’t an option that’s part of the options in that specific topic)
  • in On the future of our nixpkgs fork, we’re discussing how we will (not) integrate nixpkgs into our own package set(s); but with the base assumption also kind of being that we have a multi-repo approach

Nevertheless, you raise some excellent questions.

At some point, (part of) the answer was (or in my impression the answer was):

Let me try to summarize the current discussion and extract the main points still in discussion:

  • Goals (from user perspective):
    • SIG repos can be used individually, reducing size and eval time
    • top-level still provides what previously ‘Nixpkgs’ was - a one-stop location for all packages
  • Goals (from SIG/maintainer perspective):
    • reduce maintainer burden
    • scoped repos, with less noise
    • less moving parts in each SIG repo

I’m very unsure how to proceed here, with the discussion being spread across 3 different topics,
with wildly varying underlying assumptions - but also a lot of thoughts and discussion already spent on each of them.

1 Like
40

Which isn’t to say that we shouldn’t have this discussion - I’m not the one to decide this, I don’t want to decide this and I do think we should have this discussion.

But maybe we should prioritize seeking consensus of the underlying assumption of what we want to achieve and how to achieve that, for now. (Like there are already so many amazing ideas in here - but unfortunately, not all of them are compatible with each other; and what I’d like is that we realize that as soon as possible, so we’re not starting to work in opposite directions of each other.)

1 Like
41

We are trying to fix that we dont have enough core people to maintain and publish everything ourselves. Distributed allows for outsourcing the job to a large degree. For example usining the flakehub flakes maintained by other people right now.

42

Agreed. For now I’m focusing on the “Sig Repos: How should they work”

1 Like