In the meantime, thereās a few other things with modules I think are worth talking about.
Imports and mkOption (from nix modules) are great to the point that Iād consider them necessary in a re-do of nixpkgs. That said, Iām not as sold using nixos modules as-is.
For a design I think we should mention the tasks we want to accomplish, and then see how we can craft a module structure to fit those tasks. For example, it would be great if I could hear examples of tasks you were trying to accomplish when overriding phases.
Iāve got a few tasks for indexing/discoverability and detangling. But Iāll have to defer to other peopleās input for tasks related to services, NixOS system stuff, and hydra.
Hereās things I think packages/modules should handle:
- For making a searchable index, we should know (without building a module):
-
- What config options are available
-
- What values they are allowed to be (enum values, boolean, positive numbers, package inputs)
- What maintained versions of the package are available
- (Boolean) Does the module have a test suite
- What systems (OS triples) passed the test suite in CI
- Definite system requirements for building (if the maintainer added any)
- Dependency constraints (e.g. node needing to be version 10, if the maintainer added any such constraints)
- Default dependencies (flake locks do this for flakes)
- Is / is-not cached (canāt be just a boolean, will need to be a structure specifying ācached for this set of config options and systemā)
- Which dependencies are build-time only
- A name that is broad enough for substitutions (ex: ācoreutilsā could be satisfied by gnu coreutils or rust coreutils)
- What licenses does the package use
- (Boolean) Foss/non-foss flag
- Maintainers (with options for names, email, site+username)
- URLs for source code
- Hashes of code (there are different hash funcs but also differences in filesystem or compressed format (zip, tar) )
- Known cveās
- What binaries are part of the output
- (Maybe) commits for previously-maintained versions (ex: { āv3.5.0ā = ā[commit-hash]ā; } )
- build requires internet (true, false, or null=unknown)
- What specifications the package provides (ex: āshell:posixā, āv3.5.0ā, āfeature:python-minimalā,)
- It would be nice to have an optional icon url
- (Maybe) env stuff
-
- Any services it would like to set or depend on
- Any ENV vars it would like to modify
- Shell hooks it would like to modify (bashrc extensions)
- At install time of a module, some users tasks are:
-
- Easy way to set config options
- Easy way to list config options
- Easy way to list direct dependencies
- An easy way to override direct dependencies
- A way to override any level of the dependency tree (see Denoās import map system!)
- An obvious/easy way for users to report package specific issues. Like a āreport issueā button on the aux.search.org.
Beyond that though, thereās an important question that seems easy but can actually be really hard. What is āthe same moduleā vs a different module?
What can make this hard is, for example, redis (the database), changed licenses. So, if we want a license to be an attribute of a module, then modules would need it to be a specific version (or subset of versions, e.g. versions before the license change) instead all the of versions. Another āwhat is a moduleā question is; can a module have dynamic dependencies? (I think, with a NixOS module import list, the answer is ānoā, which IMO is good for many tasks)
Towards answering that āwhatās a moduleā question, Iād like to propose a smaller and larger definition. Iāll call the larger one tools and the smaller one modules ā with tools being user-facing, and modules focusing more on being well-defined little packets that are faster to eval. Tools could contain multiple modules under this definition.
To make that idea more concrete, Modules could be defined as
- Single set of dependencies (on other modules)
- Has one active set of licenses/description/cve list (doesnāt contain two versions of a package that have a different license and/or description and/or CVE list)
- Has a static list of versions it can build
- Has some form of āIf youāre looking for coreutils, I am one implementation of coreutilsā (e.g. a certificate system for solving the shared inputs problem)
- Has a derivation function (that doesnāt necessarily use mkDerivation)
For eval time (for indexing/search tasks), I think it would be hard to do better than serializing as much module info as possible. Either to json/toml, or for real performance: convert the json to bson. Compared to high performance parsers that integrate directly into databases, full Nix eval is just heavy and slow. Letting the nix runtime focus purely on building derivations instead of repeatedly generating static info at runtime would almost certainly be runtime-optimal. Nix can also import json/toml so its not like the info is outside of nixās grasp. We might need two files: pre-build info (version, name, licence, etc) and post-build (are tests passing, is there a cached version available, CVE list, names of contributed binaries, output hash values, etc).
However, with one license, one cve list, etc modules would necessarily end up fragmented (Like python38, python39, etc). That isnāt great for user experience. So āToolsā could be for when a user is looking for something: when they search python they get the āpythonā tool, not a fragmented list of python2, python38, python39, pythonMinimal, etc. I think it would be a much better experience to simply select python, select the version, and be presented with config options for that version, than to see a million different variants of python at the top level of search results. We could store info such as description, icon, and website homepage at the tool level instead of the module/package level.