At one point we flirted with the idea of replacing bash with something else. This post is meant as a wishlist of what such language should provide, how it should handle stuff, what features it should focus on, etc. I’ll start, but please share your thoughts, regardless of if you think replacing bash is a good idea! This post is not meant to discuss that topic, nor do I think it’s realistic for us to go with a completely home-grown solution.
In no particular order:
easily bootstrappable (even if in primitive form, should be one of the first packages we’re able to build)
first class string operations (splitting, joining, filtering, et al.)
first class path operations
sane argument handling (e.g. never split arguments by default. something like mkdir $out should never create more than one directory, regardless what the value of $out is)
first class environment management
Things we potentially could omit that a regular shell can’t afford (feel free to dispute those):
job control - we probably don’t want to run stuff in the background etc? this shouldn’t be our responsibility?
types other than strings and collections of strings - we don’t want to be a general purpose language, specifically a “command runner”. If you need to do math to build your package you either need a build system or a script in a more capable language
This is not to say the scope wouldn’t grow to a general-purpose shell (in fact I’m personally quite fed up with the viable shell options right now so at some point I’d like to be able to use this hypothetical language as a primary way to interact with my computers)
oo, thanks for @'ing me! yes, this looks like a good list to me.
nuanced ways to do string substitution and multi-line literals are important things for us but they are handled nix-side, I don’t see any reason the shell needs to do those parts.
I do feel like some of these are at odds with terse syntax. like, the interface between nix and the shell is still going to be text files, so your $out example is always going to be a little beyond the shell’s capabilities to handle without help from the nix side… (for those who don’t have this fresh in their minds: the problem is that if the value contains a space, the shell interprets it as multiple words)
unless we create a rule that every shell argument needs to be in a double-quoted string, or something of that nature, which is an option, but I’d like to hear your thoughts?
I mean, the $out case is handled by nix passing that value as a single environment value. The problems start with more complex structures, which could regardless of the approach you take, get pretty ugly pretty fast.
The issue is, this does make passing in values between shell and nix a bit more problematic. Instead of simply doing cp "${pkgs.foo}/*" ./ you’d need to pass in foo as a variable, and refer to it in the script, like cp $foo/* ./.
As for “structured attrs” (nixpkgs reference intended), I do like that nix already has first-class support for JSON. So adding in automagically parsing JSON variables and unpacking them into more complex variables is something we could do, and also make a first-class citizen, in contrast to how structuredAttrs work in nixpkgs, where they have to shoehorn JSON into bash
hmmm… yeah, it almost makes me want to address things with shell-ification helpers on the nix side, to convert complex data structures into a syntax that copies over cleanly… but then, you’re right, we should be using structured attrs or something similar for that, instead of textual substitution… hmm.
your JSON idea is a good one; I don’t feel like I have all my questions answered in terms of how this would actually be better than bash syntactically, but I also think we probably need to sketch it out in more detail so those concerns become apparent
I think the biggest advantage would be not having to rely on third party tools and a bunch of scripts to parse json into environment variables. Assuming a map would be one of the first-class types, passing in { foo = "bar"; } as variable a would mean in the script one could access $a.foo and use it as any other variable in the script (no commitment to the syntax itself).
That’d also mean you’d have to be purposeful with expanding arguments. Sure, in bash you can already do it if you pass in a string as a variable, but you still have to quote it, and even if you do there could be issues, and you can’t just pass in anything you want as an environment variable as you please because it may clash with the stdenv script etc.
A lot of that can be improved on without a new language, but some stuff, like what types bash supports or its defaults, can’t.
I’m happy to brainstorm some example scripts in that hypothetical language if you have some examples of more complex scripts, especially if they require string manipulation, multiline shenanigans (this is an important one, impossible to get right with bash!), and others. The more challenging and complex, the better.
take a look at the stage1 init script in nixpkgs. that’s the most complex one I’m aware of, and also it runs at a time when it can’t rely on most external tools, so I think it makes a good example.
Great, thank you. This immediately brings up another, unique to nix, feature we’d probably want - first class templating. So that instead of @foo@ and risking writing arbitrary bash, having to actively think about escaping etc, we could provide functionality akin to prepared sql statements.
oh fascinating. that’s a really neat idea. what would the interface between the shell and nix actually be, then? would we still use a text file as the means of getting things from nix to shell?
First idea that comes to mind is that the script could declare ahead of time what variables it needs substituted at “compile-time”, and then we’d produce a blob with the variables already substituted in. Ideally the blob would be viewable in any text editor, just with some binary junk concatenated somewhere, with big warning comments about the file being only for viewing.
It definitely needs more time in the oven, but the primary interface would still be textual and would consist of writing scripts as we’re used to (for the most part), at least I think so.
E.g. (inspired by fish, again, not committing to anything)
#!/usr/bin/env super-cool-lang
needs verbose
needs extraUtils
set targetRoot /mnt-root
set console tty1
# extraUtils below is already escaped and substituted by the time the script is running
set -x PATH $extraUtils/bin
ln -s $extraUtils/bin /bin
ln -s $extraUtils/bin /sbin
...
which then would be compiled with something like super-cool-lang-prepare stage-1-init.template.x stage-1-init.x, and verbose and extraUtils taken from the environment at compile-time
EDIT: Yes, this does not account for $verbose being an env variable supplied at runtime. This is solvable with dedicated syntax or maybe having a special variable like $const being a collection of substituted values. Alas
I do think it’s a useful property that text files can be inspected by eye, especially when debugging. I’d lean towards JSON or another textual format over a binary one, but the idea of providing the structured stuff off to the side somewhere seems reasonable. I definitely agree that the interface should continue to be file-based.