Lib Adoption Proposal

Labs
1

Hey there everyone! I am excited to announce my request for the adoption of a new standalone Nix library: Aux Lib.

What is Aux Lib?

This library is a full recreation of Nixpkgs’ library, but has zero dependencies. It operates on pure Nix concepts and is intended to be used as a foundational piece to support the creation of more complex structures such as package sets and operating systems. A few key features of the library include:

  • Module system
  • Library extensions
  • DAG support
  • Normalized naming and organization using namespaces

I have tried to build this library in a way that is as intuitive as possible for people to use. Rather than slapping everything on the root attribute set, everything is contained within relevant namespaces. This means that any time you want to work with a string, you know that you’ll find your helper in lib.strings.*. In addition to this organization, common patterns have been standardized to make it (hopefully) easier for users to find what they are looking for. Validation logic, for example, is contained within a validate attribute set under the relevant namespace such as lib.strings.validate.* to allow for performing different validations on a particular data type.

The module system has also been ported, but contains several distinct changes which will make it less confusing for users.

  • Renaming module imports to includes so it does not collide with the actual import keyword
  • Renaming module disabledModules to excludes to match
  • Making it clear that certain things are internal to Aux Lib using __double_underscores__, which should make it distinguishable from Nix builtins which use only __prefixed_underscore
  • Directed acyclic graph support ported from Home-Manager for more expressive and complex structures
  • Normal modules do not support shorthand syntax which should help users understand that their system configuration is in fact a module rather than some json-like config file
  • Module arguments are now dynamically supplied so you no longer have to add ...
  • Dynamic and static module arguments are now distinguished as config.__module__.args.dynamic and config.__module__.args.static rather than the prior config._module.args + specialArgs combination

What is Aux Lib not?

Notably, Aux Lib only operates on pure Nix concepts. There are no mentions of a package, package metadata, builders, etc. This is intentional as it allows the library to remain independent of a particular package set. These concepts can be built atop this library in another project such as Aux Foundation or Aux Tidepool and should not be included in Aux Lib itself.

Additionally, Aux Lib is not indented to answer every problem. The scope of the library has been trimmed down from what Nixpkgs offers with the intent to simplify its use and maintenance, realizing that most of the time we do not need all of the extra pieces. Any additions to Aux Lib should be thoughtfully considered and care should be taken to not bloat the library, possibly making it harder for users to find what they are looking for or increasing evaluation time.

Adoption

In order for this project to move onto its next phase, pulling it out of Aux Labs, it needs a proper home and team responsible for its upkeep and use. I believe that @sig_core is the right place for this project as it is a central point from which all of our other projects can/will consume. As SIG Core is intended to manage core projects such as the core package set, I believe that the library enabling that package set also falls under its domain. To proceed with this process we require the owners of SIG Core to accept this adoption proposal and work together to create a transition plan. I believe @isabel is the touch-point here :slight_smile:

Before accepting this proposal, please take a look at the code in the repository linked at the beginning of this post. We do not have any outstanding known bugs (other than one recently discovered by @austreelis), but the library will require some documentation and testing love. Currently about half of the project has tests and while there is code documentation it is rather bare-bones. This will be a collaboration effort with @sig_documentation to make the project ready for end-users. @coded and @minion can be reached for input on documentation.

Anything I missed?

This has been in the works for a little while and I may have missed some things in this post. If so, please feel free to ask questions or let me know! I am excited to finally have this library moving forward now that it has been proven to function well within Aux Foundation and Aux Tidepool for a few months.

11 Likes
2

Glad this is finally getting released! I’ll take a look into getting some docs started this upcoming week :slight_smile: Thanks to all the contributors for the work y’all have done on this.

3 Likes
3

I’m in favor of adopting aux-lib, I believe it has some pleasing features, and it is quite well made, thanks for the work ! (Especially the dag is a godsend)

I’ve been personally using it as a replacement of nixpkgs’ lib for things like devshells (I have been replacing my flake devShells with some, numtide/devshell-inspired, custom module system implemented with aux-lib + npins, something I wished to do with nixpkgs’ lib for some time) and it is working great. I’m not explicitely looking for bugs but I do go for some weird cases, and the lib looks pretty robust.

I think I have two minor things to point out:

First, I think tests are a tiny bit lacking right now. For instance I have to parse their result to programatically check if they succeed (instead of relying on say an exit code). Ideally I would like structured output too (like JSON), but that’s really just a nice-to-have.

Second, I am wondering how will adding functions will play with the nested namespace approach: say we already have a lists.sort, which could implement a slower stable sort, but we later discover we can provide a faster unstable sort. I several ways this can play out:

  1. Renaming lists.sort to lists.sort.stable to allow adding lists.sort.unstable. This is a breaking change (booo)
  2. Adding a new namespace under lists, which has a similar name to “sort” but wouldn’t clash, e.g. lib.sorts.unstable. This kind of is the current approach (e.g. types.string for a simple string merging equal options, and types.strings.lines for merged newline-separated strings), but I’m wondering if this will ever be a burden. I think it’s okay if the phenomenon is limited, but I would be sad if aux-lib was riddled with pairs of similarly-named attributes because we (didn’t) realised we needed a variant of an already-existing function.
  3. Making lists.sort a functor that has a unstable attribute (the new sort) but behaves like the old lists.sort if called. This is non-breaking but it makes the code less readable, and could be confusing ? (I personally think it’s pretty natural but I’m expecting to be a minority ?)
  4. Wait a long enough time before stabilizing an API that we would be confident an attribute will not have to be broken up. If combined with 2, this could be a nice compromise.

I think those two points do not prevent adoption of aux-lib, though.

4 Likes
4

I’ve been personally using it […]

Awesome!! If you have any projects visible publicly, please do send me links! I’d love to see what you are doing with the library.

[…] tests are a tiny bit lacking right now.

I agree. Not just with the amount of tests, but, like you mention, the testing infrastructure itself is rather naive. At the time my goal with it was to have something “good enough” to allow for validating that the library works as expected while I was building it. Now that we are this far along, I think a proper unit testing setup is in order. Perhaps a separate project can be spun up to build out a good solution?

[…] how will adding functions will play with the nested namespace approach […]-

This is a difficult problem for us to solve and I’m not sure we will be able to fully. Undoubtedly software gets to a point where some change becomes worth a breaking change. I think our goal should be to try and adopt features in a way that prevents as many breaking changes as possible. For the example of a sort function we can look at how sorting is organized in the library today. Aux Lib has a lib.lists.sort.* attribute set for much the same reason it has a lib.lists.validate.* attribute set. These common actions can perform similar, but distinct actions and it is helpful to separate them. As it turns out, breaking things down at this level also helps with discovery. Since people will find lib.lists.sort and then learn that they can choose from different sorting methods. A cautious approach to feature adoption would serve us well here.

I think that I align most with your suggested #4:

Wait a long enough time before stabilizing an API that we would be confident an attribute will not have to be broken up. If combined with 2, this could be a nice compromise.

Referencing #2:

Adding a new namespace under lists, which has a similar name to “sort” but wouldn’t clash, e.g. lib.sorts.unstable. This kind of is the current approach (e.g. types.string for a simple string merging equal options, and types.strings.lines for merged newline-separated strings), but I’m wondering if this will ever be a burden. I think it’s okay if the phenomenon is limited, but I would be sad if aux-lib was riddled with pairs of similarly-named attributes because we (didn’t) realised we needed a variant of an already-existing function.

4 Likes
5

This is dope, lots of interesting changes, especially the DAG stuff!

Are there any plans to have API versioning of lib? Would be nice to have some reassurance that minor revisions won’t require downstream changes, similar to how nixpkgs keeps options stable within a release, but for libs.

4 Likes
6

I expect all of our projects (that are not in the experimental stage) to follow versioning best practices. I think everyone I have spoken to has been in favor of SemVer so we should be good to go :slight_smile:

4 Likes
7

Great work Jake! I love the concept you applied to the lib, especially the namespacing makes a lot of sense to me personally. I love when reading code clearly states e.g. that it’s now validating a string :+1:

I wonder how much/ if not all of this can be solved by versioning the lib?
I personally love to be able to just automatically update my flake inputs, but a guarantee on certain things not including breaking changes is really sweet :wink:

Re adoption:
I haven’t looked at the code, but I assume it’s not a drop-in replacement to nixpkgs.lib !?
Are there any features/ functions from nixpkgs.lib that are currently missing or intentionally not implemented?

1 Like
8

I’ve been using the lib quite a bit, and there’s ime there’w 3 kinds of missing bits:

  • Intentionally left out because they’re already builtins.
  • Intentionally left out, at least for now, because considered out of scope for a general-purpose lib. nixpkgs’ lib.system is an example of that, but an equivalent is in labs/tidepool’s lib.
  • Not deemed needed right now. Things like nixpkgs’ lib.debug for instance are simply not implemented. I think it’s important to let us time to get some of those right.

Also, it’s definitely not a drop-in replacement for nixpkgs’ lib, in its current state. I personally don’t think it should be, I find it very usable this way.

Labs’ lib also generally feel like a “language standard generic lib”, while nixpkgs’ has some specialized tools for package management (versionAtLeast for instance but there’s other that are way more specific).

2 Likes
9

I think it maybe better to create a SIG specifically for the topic of lib, its already pretty depthy and a bit complex, so a specialised team maybe better then adding onto core’s work. Although, I do apricate the proposal as I thought nixpkgs always lacked this when it came to me only needing lib and not the entire nixpkgs repo.

2 Likes
10

Thanks for the run-down, highly appreciated :+1:
Since I’m in the middle of a massive re-write I might just add another todo to the never ending list :wink:

2 Likes
11

This is brilliant. I am in favor.