This extends the `error: cannot coerce a TYPE to a string` message
to print the value that could not be coerced. This helps with debugging
by making it easier to track down where the value is being produced
from, especially in errors with deep or unhelpful stack traces.
Do this if we want to do `--hash-algo` everywhere, and not `--algo` for
hash commands.
The new `nix hash convert` is updated. Deprecated new CLI commands are
left as-is (`nix hash path` needs to be redone and is also left as-is).
Good to document these formats separately from commands that happen to
use them.
Eventually I would like this and `builtins.derivation` to refer to a
store section on derivations that is authoritative, but that doesn't yet
exist, and will take some time to make. So I think we're just best off
merging this now as is.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
A small step towards https://github.com/NixOS/nix/issues/6507
I believe this incomplete definition is one that can be agreed on.
It would be nice to define more, but considering that the issue
also proposes changes to the design, I believe we should hold off
on those.
As for the wording, we're dealing with some very general and vague
terms, that have to be treated with exactly the right amount of
vagueness to be effective.
I start out with a fairly abstract definition of package.
1. to establish a baseline so we know what we're talking about
2. so that we can go in and clarify that we have an extra, Nix-specific
definition.
"Software" is notoriously ill-defined, so it makes a great qualifier
for package, which we don't really want to pin down either, because
that would just get us lost in discussion.
We can come back to this after we've done 6057 and a few years in a
desert cave.
Then comes the "package attribute set" definition.
I can already hear Valentin say "That's not even Nix's responsibility!"
and on some days I might even agree.
However, in our current reality, we have `nix-env`, `nix-build` and
`nix profile`, which query the `outputName` attribute - among others -
which just don't exist in the derivation.
For those who can't believe what they're reading:
$ nix-build --expr 'with import ./. {}; bind // {outputName = "lib";}' --no-out-link
this path will be fetched (1.16 MiB download, 3.72 MiB unpacked):
/nix/store/rfk6klfx3z972gavxlw6iypnj6j806ma-bind-9.18.21-lib
copying path '/nix/store/rfk6klfx3z972gavxlw6iypnj6j806ma-bind-9.18.21-lib' from 'https://cache.nixos.org'...
/nix/store/rfk6klfx3z972gavxlw6iypnj6j806ma-bind-9.18.21-lib
and let me tell you that bind is not a library.
So anyway, that's also proof of why calling this a "derivation attrset" would be wrong, despite the type attribute.
The Nix team has requested that this output format remain unchanged.
I've added a warning to the man page explaining that `nix-instantiate
--eval` output will not parse correctly in many situations.
channels make everything more stateful, and therefore more complicated
and potentially confusing, but aren't needed for this task, so don't encourage their use.
* deduplicate installation instructions
- reorder sections to present pinned installation more prominently
- remove outdated notes on the macOS installer rework
- update instructions to handle the installer tarball
Co-authored-by: Travis A. Everett <travis.a.everett@gmail.com>
since `up` and `values` are both pointer-aligned the type field will
also be pointer-aligned, wasting 48 bits of space on most machines. we
can get away with removing the type field altogether by encoding some
information into the `with` expr that created the env to begin with,
reducing the GC load for the absolutely massive amount of single-entry
envs we create for lambdas. this reduces memory usage of system eval by
quite a bit (reducing heap size of our system eval from 8.4GB to 8.23GB)
and gives similar savings in eval time.
running `nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'`
before:
Time (mean ± σ): 5.576 s ± 0.003 s [User: 5.197 s, System: 0.378 s]
Range (min … max): 5.572 s … 5.581 s 10 runs
after:
Time (mean ± σ): 5.408 s ± 0.002 s [User: 5.019 s, System: 0.388 s]
Range (min … max): 5.405 s … 5.411 s 10 runs
a bunch of derivation strings contain no escape sequences. we can
optimize for this fact by first scanning for the end of a derivation
string and simply returning the contents unmodified if no escape
sequences were found. to make this even more efficient we can also use
BackedStringViews to avoid copies, avoiding heap allocations for
transient data.
before:
Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
Time (mean ± σ): 6.952 s ± 0.015 s [User: 5.294 s, System: 1.452 s]
Range (min … max): 6.926 s … 6.974 s 10 runs
after:
Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
Time (mean ± σ): 6.907 s ± 0.012 s [User: 5.272 s, System: 1.429 s]
Range (min … max): 6.893 s … 6.926 s 10 runs
This fixes a segfault on infinite function call recursion (rather than
infinite thunk recursion) by tracking the function call depth in
`EvalState`.
Additionally, to avoid printing extremely long stack traces, stack
frames are now deduplicated, with a `(19997 duplicate traces omitted)`
message. This should only really be triggered in infinite recursion
scenarios.
Before:
$ nix-instantiate --eval --expr '(x: x x) (x: x x)'
Segmentation fault: 11
After:
$ nix-instantiate --eval --expr '(x: x x) (x: x x)'
error: stack overflow
at «string»:1:14:
1| (x: x x) (x: x x)
| ^
$ nix-instantiate --eval --expr '(x: x x) (x: x x)' --show-trace
error:
… from call site
at «string»:1:1:
1| (x: x x) (x: x x)
| ^
… while calling anonymous lambda
at «string»:1:2:
1| (x: x x) (x: x x)
| ^
… from call site
at «string»:1:5:
1| (x: x x) (x: x x)
| ^
… while calling anonymous lambda
at «string»:1:11:
1| (x: x x) (x: x x)
| ^
… from call site
at «string»:1:14:
1| (x: x x) (x: x x)
| ^
(19997 duplicate traces omitted)
error: stack overflow
at «string»:1:14:
1| (x: x x) (x: x x)
| ^
Previously, IFDs would be built within the eval store, even though one
is typically using `--eval-store` precisely to *avoid* local builds.
Because the resulting Nix expression must be copied back to the eval
store in order to be imported, this requires the eval store to trust
the build store's signatures.
It might seem obnoxious to have yet more configure flags, but I found
controlling both the unit and functional tests with one flag was quite
confusing because they are so different:
- unit tests depending on building, functional tests don't (e.g. when
we test already-built Nix)
- unit tests can be installed, functional tests cannot
- unit tests neeed extra libraries (GTest, RapidCheck), functional
tests need extra executables (jq).
- unit tests are run by `make check`, functional tests are run by `make
installcheck`
Really on a technical level, they seem wholly independent. Only on a
human level ("they are both are tests") do they have anything in common.
I had messed up the logic in cross builds because of this. Now I
split the flag in two (and cleaned up a few other inconsistencies), and
the logic fixed itself.
Co-Authored-By: Robert Hensing <roberth@users.noreply.github.com>
* document `fetchTree`
* display experimental feature note at the top
we have to enable the new `fetchTree` experimental feature to render it
at all. this was a bug introduced when adding that new feature flag.
Co-authored-by: tomberek <tomberek@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Silvan Mosberger <github@infinisil.com>
* Print the value in `error: cannot coerce` messages
This extends the `error: cannot coerce a TYPE to a string` message
to print the value that could not be coerced. This helps with debugging
by making it easier to track down where the value is being produced
from, especially in errors with deep or unhelpful stack traces.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
This includes position information in more places, making debugging
easier.
Before:
```
$ nix-instantiate --show-trace --eval tests/functional/lang/eval-fail-using-set-as-attr-name.nix
error:
… while evaluating an attribute name
at «none»:0: (source not available)
error: value is a set while a string was expected
```
After:
```
error:
… while evaluating an attribute name
at /pwd/lang/eval-fail-using-set-as-attr-name.nix:5:10:
4| in
5| attr.${key}
| ^
6|
error: value is a set while a string was expected
```
Today, with the tests inside a `tests` intermingled with the
corresponding library's source code, we have a few problems:
- We have to be careful that wildcards don't end up with tests being
built as part of Nix proper, or test headers being installed as part
of Nix proper.
- Tests in libraries but not executables is not right:
- It means each executable runs the previous unit tests again, because
it needs the libraries.
- It doesn't work right on Windows, which doesn't want you to load a
DLL just for the side global variable . It could be made to work
with the dlopen equivalent, but that's gross!
This reorg solves these problems.
There is a remaining problem which is that sibbling headers (like
`hash.hh` the test header vs `hash.hh` the main `libnixutil` header) end
up shadowing each other. This PR doesn't solve that. That is left as
future work for a future PR.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
This makes for more useful manual table of contents, that displays the
information at a glance.
The `nix help-stores` command is kept as-is, even though it will show up
in the manual with the same information as these pages due to the way it
is written as a "`--help`-style" command. Deciding what to do with that
command is left for a later PR.
This change also lists all store types at the top of the respective overview page.
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems
this focuses on `nix-shell -p` and refers to search.nixos.org for
package search, which is currently the easiest and most effective way to
find program names.
- put the highlight box around all the relevant instructions
- simplify the wording
- make the link more prominent by using the whole phrase for the link text
- helps navigating the code as it highlights which files are generated
- makes it less error prone when working incrementally
(although this should be just fixed by building out of tree)
Previously many of the documentation targets were depending on
`$(bindir)/nix` which is the installed version. This meant that its
install rules would be triggered (which in chain would also trigger the
install of libraries, as reported in #5140). Therefore a build of the
documentation without an installation would not be possible (which apart
from doing unwanted operations it may also generate permission problems
for example).
The fix makes the rules depend on `$(nix_PATH)` instead, which is the
executable in the build tree.
`installcheck` doesn't yet work, but the rest of the build can now
happen mostly inside a separate build directory.
Progress on #9342
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
up to now, those were managed outside of this repo, which as
unsurprisingly a real hassle to deal with if one wanted to prevent URLs
from breaking when moving pages around. this change removes a large part
of the friction involved in moving content in the Nix manual.
possible next steps for further automation:
- check for content that moved and warn if it's not reachable from
links that were valid prior to a change
- create redirect rules automatically based on this information
nix-repl> bools = [ false true ]
nix-repl> combinations = builtins.concatMap (a: builtins.concatMap (b: map (c: { inherit a b c; }) bools) bools) bools
nix-repl> builtins.all ({ a, b, c }: (a -> b -> c) == (a -> (b -> c))) combinations
true
nix-repl> builtins.all ({ a, b, c }: (a -> b -> c) == ((a -> b) -> c)) combinations
false
The basic idea here is to separate a few intertwined notions:
1. Not all "run bash tests" are "install tests"
2. Not all "run bash tests" use `tests/functional/init.sh`, or any
pre-test initialization at all.
This will used in the next commit when we have a test that check unit
test golden master data.
Also, move our custom `PS4` from the test to the test runner, as it is
part of how we want to display the tests, not the test themselves.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
As discussed in our last meeting, we need a bit more time, but we are
"time boxing" the work left to do to ensure there is not unbounded
delay.
Rather than putting it back underneath `flakes`, though, put it
underneath its own `fetch-tree` experimental feature (which `flakes`
includes/implies). This signals our commitment to the plan to stabilize
it first without waiting to go through the rest of Flakes, and also will
give users a "release candidate" when we get closer to stabilization.
This reverts commit 4112dd1fc9.
Before it returned a list of JSON objects with store object information,
including the path in each object. Now, it maps the paths to JSON
objects with the metadata sans path.
This matches how `nix derivation show` works.
Quite hillariously, none of our existing functional tests caught this
change to `path-info --json` though they did use it. So just new
functional tests need to be added.
This adds publicKeys as an optional fetcher input attribute to flakes
and builtins.fetchGit to provide a nix interface for the json-encoded
`publicKeys` attribute of the git fetcher.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
update the glossary to point to the new page.
since this is a cross-cutting concern, it warrants its own section in
the manual.
Co-authored-by: John Ericson <git@JohnEricson.me>
* document the store concept and its purpose
reword the glossary to link to more existing information instead of
repeating it.
move the store documentation to the top of the table of contents, in
front of the Nix language. this will provide a natural place to
document other aspects of the store as well as the various store types.
move the package management section after the Nix language and before
Advanced Topics to follow the pattern to layer more complex concepts on
top of each other.
this structure of the manual will also nudge beginners to learn Nix
bottom-up and hopefully make more likely that they understand underlying
concepts first before delving into complex use cases that may or may not
be easy to implement with what's currently there.
[John adds this note] The sort of beginner who likes to dive straight into reference documentation should prefer this approach. Conversely, the sort of beginner who would prefer the opposite top-down approach of trying to solve problems before they understand everything that is going on is better off reading other tutorial/guide material anyways, and will just "random-access" the reference manual as a last resort. For such random-access the order doesn't matter, so this restructure doesn't make them any worse off.
Co-authored-by: John Ericson <git@JohnEricson.me>
Rather than having a misc tutorial page in the grab-bag "package management" section, this information should just be part of the S3 store docs.
---------
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
Single quotes are a basic feature of shell syntax that people expect to
work. They are also more convenient for writing literal code expressions
with less escaping.
this is the first thing most beginners see, and it misleads them into
assuming `nix-env` is appropriate for doing anything but setting and
reverting profile generations.
this chapter is the root of most evil around the ecosystem, and today we
finally close it for good.
* docker: publish images to ghcr.io
docker.com announced their intention to remove the free plan used by
OSS. The nixos/nix image is essential to various CI runs to build with
nix. To provide a continuity plan, this commit pushes the image to
ghcr.io as well.
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
MemoryInputAccessor is an in-memory virtual filesystem that returns
files like <nix/fetchurl.nix>. This removes the need for special hacks
to handle those files.
Add a new experimental `impure-env` setting that is a key-value list of
environment variables to inject into FOD derivations that specify the
corresponding `impureEnvVars`.
This allows clients to make use of this feature (without having to change the
environment of the daemon itself) and might eventually deprecate the current
behaviour (pick whatever is in the environment of the daemon) as it's more
principled and might prevent information leakage.
This introduces some shared infrastructure for our notion of protocols.
We can then define multiple protocols in terms of that notion.
We an also express how particular protocols depend on each other.
For example, we can define a common protocol and a worker protocol,
where the second depends on the first in terms of the data types it can
read and write.
The "serve" protocol can just use the common one for now, but will
eventually need its own machinary just like the worker protocol for
version-aware serialisers
For people working on Nix with `nix develop`, it's better to just use
`autoreconfPhase` and `configurePhase`, which is standard Nixpkgs / nix
shell make from Nixpkgs practice --- it is good to emphasize the degree
to which Nix is *just* a regular C++ project which can be worked on in
the regular way.
(For people running `nix-shell`, the story is similar, except
`configurePhase` would use non-writable store paths, which matters for
hte times we use output paths before `make install`, so I kept the
existing `./configure ...` instruction.)
For people building Nix without Nix (e.g. packaging it for another
distro) they also don't need `bootstrap.sh`, and can just run
`autoreconf -vfi` directly. (More likely, they have their own idioms to
do this just as we have `autoreconfPhase`.)
I was sleepy and confused that "interpolated expression" was a new type of thing at first. This nudges the reader to understand that its just a regular expression, and these conditions are imposed by the interpolation operation.
I think it is bad for these reasons when `tests/` contains a mix of
functional and integration tests
- Concepts is harder to understand, the documentation makes a good
unit vs functional vs integration distinction, but when the
integration tests are just two subdirs within `tests/` this is not
clear.
- Source filtering in the `flake.nix` is more complex. We need to
filter out some of the dirs from `tests/`, rather than simply pick
the dirs we want and take all of them. This is a good sign the
structure of what we are trying to do is not matching the structure
of the files.
With this change we have a clean:
```shell-session
$ git show 'HEAD:tests'
tree HEAD:tests
functional/
installer/
nixos/
```
the `term` output mode leaves inline HTML around verbatim, while `nroff`
mode (used for `man` pages) does not.
the correct solution would be to pre-render all output with a more
benign tool so we have less liabilities in our own code, but this has to
do for now.
In #4770 I implemented proper `nix-shell(1)` support for derivations
using `__structuredAttrs = true;`. Back then we decided to introduce two
new environment variables, `NIX_ATTRS_SH_FILE` for `.attrs.sh` and
`NIX_ATTRS_JSON_FILE` for `.attrs.json`. This was to avoid having to
copy these files to `$NIX_BUILD_TOP` in a `nix-shell(1)` session which
effectively meant copying these files to the project dir without
cleaning up afterwords[1].
On last NixCon I resumed hacking on `__structuredAttrs = true;` by
default for `nixpkgs` with a few other folks and getting back to it,
I identified a few problems with the how it's used in `nixpkgs`:
* A lot of builders in `nixpkgs` don't care about the env vars and
assume that `.attrs.sh` and `.attrs.json` are in `$NIX_BUILD_TOP`.
The sole reason why this works is that `nix-shell(1)` sources
the contents of `.attrs.sh` and then sources `$stdenv/setup` if it
exists. This may not be pretty, but it mostly works. One notable
difference when using nixpkgs' stdenv as of now is however that
`$__structuredAttrs` is set to `1` on regular builds, but set to
an empty string in a shell session.
Also, `.attrs.json` cannot be used in shell sessions because
it can only be accessed by `$NIX_ATTRS_JSON_FILE` and not by
`$NIX_BUILD_TOP/.attrs.json`.
I considered changing Nix to be compatible with what nixpkgs
effectively does, but then we'd have to either move $NIX_BUILD_TOP for
shell sessions to a temporary location (and thus breaking a lot of
assumptions) or we'd reintroduce all the problems we solved back then
by using these two env vars.
This is partly because I didn't document these variables back
then (mea culpa), so I decided to drop all mentions of
`.attrs.{json,sh}` in the manual and only refer to `$NIX_ATTRS_SH_FILE`
and `$NIX_ATTRS_JSON_FILE`. The same applies to all our integration tests.
Theoretically we could deprecated using `"$NIX_BUILD_TOP"/.attrs.sh` in
the future now.
* `nix develop` and `nix print-dev-env` don't support this environment
variable at all even though they're supposed to be part of the replacement
for `nix-shell` - for the drv debugging part to be precise.
This isn't a big deal for the vast majority of derivations, i.e.
derivations relying on nixpkgs' `stdenv` wiring things together
properly. This is because `nix develop` effectively "clones" the
derivation and replaces the builder with a script that dumps all of
the environment, shell variables, functions etc, so the state of
structured attrs being "sourced" is transmitted into the dev shell and
most of the time you don't need to worry about `.attrs.sh` not
existing because the shell is correctly configured and the
if [ -e .attrs.sh ]; then source .attrs.sh; fi
is simply omitted.
However, this will break when having a derivation that reads e.g. from
`.attrs.json` like
with import <nixpkgs> {};
runCommand "foo" { __structuredAttrs = true; foo.bar = 23; } ''
cat $NIX_ATTRS_JSON_FILE # doesn't work because it points to /build/.attrs.json
''
To work around this I employed a similar approach as it exists for
`nix-shell`: the `NIX_ATTRS_{JSON,SH}_FILE` vars are replaced with
temporary locations.
The contents of `.attrs.sh` and `.attrs.json` are now written into the
JSON by `get-env.sh`, the builder that `nix develop` injects into the
derivation it's debugging. So finally the exact file contents are
present and exported by `nix develop`.
I also made `.attrs.json` a JSON string in the JSON printed by
`get-env.sh` on purpose because then it's not necessary to serialize
the object structure again. `nix develop` only needs the JSON
as string because it's only written into the temporary file.
I'm not entirely sure if it makes sense to also use a temporary
location for `nix print-dev-env` (rather than just skipping the
rewrite in there), but this would probably break certain cases where
it's relied upon `$NIX_ATTRS_SH_FILE` to exist (prime example are the
`nix print-dev-env` test-cases I wrote in this patch using
`tests/shell.nix`, these would fail because the env var exists, but it
cannot read from it).
[1] https://github.com/NixOS/nix/pull/4770#issuecomment-836799719
* document "Import From Derivation"
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
make the example more realistic, since `headers` is not an output name
used in Nixpkgs
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
derivations are about data transformation, so the term "build" does not
add any information. there was also some feedback that "build task" is
not more helpful than "derivation" if you have no prior experience with
Nix or build systems, while existing associations may be misleading.
Was confused why `make html` didn't work while working on #9032, but
then I realized that after this section was written, the target was
renamed to `manual-html` in 6910f5dcb6.
Before they were an "ad-hoc" header with bold and a colon; now they are
a proper subheader.
For the man pages, this doesn't make much of a difference, but it will
help more on for the HTML manual, where things can be restyled. Again,
good separation of content vs presentation.
Behavior change:
Before we only showed uption if the command-specific options were
non-empty. But that is somewhat odd since we also show common options.
Now, we do everything based on the union of both sorts of options (with
hidden-categories filtered, as before).
Implementation change:
The JSON dumping once again includes all options; the filtering of
hidden categories is done in the Nix instead. This is better separation
of "content" vs "presentation", and prepare the way for the HTML manual
vs manpages / `--help` doing different things.
there is a very confusing warning in the Nixpkgs manual that
mischaracterises `nix-env` behavior, and this example shows what's
really happening.
note that it doesn't use `pkgs.runCommand` or other `pkgs.stdenv`
facilities, as deep down those set `meta.outputsToInstall` to very
particular defaults that do not generally apply to Nix.
We use the same nested map representation we used for goals, again in
order to save space. We might someday want to combine with `inputDrvs`,
by doing `V = bool` instead of `V = std::set<OutputName>`, but we are
not doing that yet for sake of a smaller diff.
The ATerm format for Derivations also needs to be extended, in addition
to the in-memory format. To accomodate this, we added a new basic
versioning scheme, so old versions of Nix will get nice errors. (And
going forward, if the ATerm format changes again the errors will be even
better.)
`parsedStrings`, an internal function used as part of parsing
derivations in A-Term format, used to consume the final `]` but expect
the initial `[` to already be consumed. This made for what looked like
unbalanced brackets at callsites, which was confusing. Now it consumes
both which is hopefully less confusing.
As part of testing, we also created a unit test for the A-Term format for
regular non-experimental derivations too.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Apply suggestions from code review
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
this removes a lot of noise from the web search, which precludes finding
the actual documentation.
some configuration settings have enough documentation to warrant
individual pages, so the alternative of including full setting
documentation in each command page doesn't make much sense here.
this change technically means that the command line flags to override
settings are "invisible", and not exported as JSON. this may or may not
be desirable. a more explicit approach would be adding a `hidden` field
to the flag's JSON output, but would also require adjusting
post-processing of that JSON for manual rendering.
this is a pure reformatting, contents were not changed
one sentence per line makes reviewing diffs and making suggestions much
more convenient. the indentation was an artifat of the DocBook
migration.
Continue with the characterization testing idioms begun in
c70484454f, but this time for unit tests.
Co-authored-by: Andreas Rammhold <andreas@rammhold.de>
In the Nix language, given a drv path, we should be able to construct
another string referencing to one of its output. We can do this today
with `(import drvPath).output`, but this only works for derivations we
already have.
With dynamic derivations, however, that doesn't work well because the
`drvPath` isn't yet built: importing it like would need to trigger IFD,
when the whole point of this feature is to do "dynamic build graph"
without IFD!
Instead, what we want to do is create a placeholder value with the right
string context to refer to the output of the as-yet unbuilt derivation.
A new primop in the language, analogous to `builtins.placeholder` can be
used to create one. This will achieve all the right properties. The
placeholder machinery also will match out the `outPath` attribute for CA
derivations works.
In 60b7121d2c we added that type of
placeholder, and the derived path and string holder changes necessary to
support it. Then in the previous commit we cleaned up the code
(inspiration finally hit me!) to deduplicate the code and expose exactly
what we need. Now, we can wire up the primop trivally!
Part of RFC 92: dynamic derivations (tracking issue #6316)
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
We want to be able to write down `foo.drv^bar.drv^baz`:
`foo.drv^bar.drv` is the dynamic derivation (since it is itself a
derivation output, `bar.drv` from `foo.drv`).
To that end, we create `Single{Derivation,BuiltPath}` types, that are
very similar except instead of having multiple outputs (in a set or
map), they have a single one. This is for everything to the left of the
rightmost `^`.
`NixStringContextElem` has an analogous change, and now can reuse
`SingleDerivedPath` at the top level. In fact, if we ever get rid of
`DrvDeep`, `NixStringContextElem` could be replaced with
`SingleDerivedPath` entirely!
Important note: some JSON formats have changed.
We already can *produce* dynamic derivations, but we can't refer to them
directly. Today, we can merely express building or example at the top
imperatively over time by building `foo.drv^bar.drv`, and then with a
second nix invocation doing `<result-from-first>^baz`, but this is not
declarative. The ethos of Nix of being able to write down the full plan
everything you want to do, and then execute than plan with a single
command, and for that we need the new inductive form of these types.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
When loading a derivation from a JSON, malformed input would trigger
cryptic "assertion failed" errors. Simply replacing calls to `operator []`
with calls to `.at()` was not enough, as this would cause json.execptions
to be printed verbatim.
Display nice error messages instead and give some indication where the
error happened.
*Before:*
```
$ echo 4 | nix derivation add
error: [json.exception.type_error.305] cannot use operator[] with a string argument with number
$ nix derivation show nixpkgs#hello | nix derivation add
Assertion failed: (it != m_value.object->end()), function operator[], file /nix/store/8h9pxgq1776ns6qi5arx08ifgnhmgl22-nlohmann_json-3.11.2/include/nlohmann/json.hpp, line 2135.
$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: [json.exception.type_error.302] type must be string, but is object
$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error: [json.exception.type_error.302] type must be object, but is string
```
*After:*
```
$ echo 4 | nix derivation add
error: Expected JSON of derivation to be of type 'object', but it is of type 'number'
$ nix derivation show nixpkgs#hello | nix derivation add
error: Expected JSON object to contain key 'name' but it doesn't
$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: Expected JSON value to be of type 'string' but it is of type 'number'
$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error:
… while reading key 'outputs'
error: Expected JSON value to be of type 'object' but it is of type 'string'
```
Over the last year or so I've run into several use cases where I need to
parse and/or serialize URLs for use by `builtins.fetchTree` or
`builtins.getFlake`, largely in order to produce _lockfile-like_ files
for lang2nix frameworks or tools which use `nix` internally to drive
builds.
I've gone through the painstaking process of emulating
`nix::FlakeRef::fromAttrs` and `nix::parseFlakeRef` several times with
mixed success; but these are difficult to create and even harder to
maintain if I hope to stay aligned with changes to the real
parser/serializer.
I understand why adding new `builtins` isn't something we want to do
flagrantly. I'm recommending this addition simply because I keep
encountering use cases where I need to parse/serialize these URIs in
`nix` expressions, and I want a reliable solution.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
the original change broke many pre-existing anchor links.
also change formatting of the constants listing slightly:
- the type should not be part of the anchor
- add highlight to the "impure only" note
* clarify wording on args@ default handling
Most importantly use shorter sentences and emphasize the key point that defaults aren't taken into account
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
this is a how-to guide which should not be in the reference manual.
it also refers to `nix-env`, which should not be the first thing readers
of the reference manual encounter, as it behaves very differently in
spirit from the rest of Nix.
slightly reword the documentation to be more concise and informative.
Grouping our tests should make it easier to understand the intent than
one long poorly-arranged list. It also is convenient for running just
the tests for a specific component when working on that component.
We need at least one test group so this isn't dead code; I decided to
collect the tests for the `ca-derivations` and `dynamic-derivations`
experimental features in groups. Do
```bash
make ca.test-group -jN
```
and
```bash
make dyn-drv.test-group -jN
```
to try running just them.
I originally did this as part of #8397 for being able to just the local
overlay store alone. I am PRing it separately now so we can separate
general infra from new features.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
* Lang now verifies errors and parse output
* Some new miscellaneous tests
* Easy way to update the tests
* Document workflow in manual
* Use `!` not `~` as separater char for sed
It is confusing to use `~` when we are talking about paths and home
directories!
* Test test suite itself (`test/lang-test/infra.sh`)
Additionally, run shellcheck on `tests/lang.sh` to help ensure it is
correct, now that is is more complex.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
This is done in roughly the same way builtin functions are documented.
Also auto-link experimental features for primops, subsuming PR #8371.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
- Improved API docs from comment
- Exit codes are for `nix-build`, not just `nix-store --release`
- Make note in tests so the magic numbers are not surprising
Picking up where #8387 left off.
Deleting store info corrected (there is a foot-gun in Nix with
`--delete-generations old`!)
Also a few things are cleaned up based on feedback.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Uninstall instructions were moved to their own page in #8267. The
overall section link was redirected in #8286, but platform-specific
links (which I give out frequently when I triage installer trouble)
weren't included.
This is generally a fine practice: Putting implementations in headers
makes them harder to read and slows compilation. Unfortunately it is
necessary for templates, but we can ameliorate that by putting them in a
separate header. Only files which need to instantiate those templates
will need to include the header with the implementation; the rest can
just include the declaration.
This is now documenting in the contributing guide.
Also, it just happens that these polymorphic serializers are the
protocol agnostic ones. (Worker and serve protocol have the same logic
for these container types.) This means by doing this general template
cleanup, we are also getting a head start on better indicating which
code is protocol-specific and which code is shared between protocols.
While this is not actually a notion in the implementation, it is
explicitly described in the thesis and quite important for understanding
how the store works.
Co-authored-by: John Ericson <git@JohnEricson.me>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Introduce what substituters "are" in the configuration option entry.
Remove arbitrary line breaks for easier editing in the future.
Link glossary some more.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
packages and configurations are not really a concept in Nix or the Nix language. the idea of transforming files into other files clearly captures what it's all about, and the new phrasing should make the term "derivation" more obvious both in terms of meaning and origin.
Add support to --list-generations
as another way to say
nix-env --profile /nix/var/nix/profiles/per-user/$USER/channels --list-generations
the way we did for nix-channel --rollback [generation id]
The primop `builtins.replaceStrings` currently always strictly evaluates the
replacement strings, however time and space are wasted for their computation
if the corresponding pattern do not occur in the input string. This commit
makes the evaluation of the replacement strings lazy by deferring their
evaluation to when the corresponding pattern are matched and memoize the result
for efficient retrieval on subsequent matches.
The testcases for replaceStrings was updated to check for lazy evaluation
of the replacements. A note was also added in the release notes to
document the behavior change.
This adds a new configuration option to Nix, `always-allow-substitutes`,
whose effect is simple: it causes the `allowSubstitutes` attribute in
derivations to be ignored, and for substituters to always be used.
This is extremely valuable for users of Nix in CI, where usually
`nix-build-uncached` is used. There, derivations which disallow
substitutes cause headaches as the inputs for building already-cached
derivations need to be fetched to spuriously rebuild some simple text
file.
This option should be a good middle-ground, since it doesn't imply
rebuilding the world, such as the approach I took in
https://github.com/NixOS/nixpkgs/pull/221048
it's more likely for readers to find it right there.
this also slightly rewords examples to make them stand out better.
in the long run there probably needs to be a dedicated section on formal syntax, and better highlighting of examples.
e.g. nix-env -e subversion => nix-env --uninstall subversion
The aim is to make the documentation less cryptic for newcomers and the
long options are more self-documenting.
The change was made with the following script:
<https://github.com/aschmolck/convert-short-nix-opts-to-long-ones>
and sanity checked visually.
This gives some more context and should clarify why it works that way.
Also link it from the section on `NIX_USER_CONF_FILES`.
Co-authored-by: John Ericson <git@JohnEricson.me>
`/etc/bash.bashrc` is backed up as `/etc/bash.bashrc.backup-before-nix`,
but since other changes might have been introduced in the meantime we can't
just tell the user to revert.
add anchor to `builtins.derivation` and list some built-in functions that are
exposed in the global scope.
I decided not to list everything, because we probably don't want to
encourage people using them that way.
* doc rendering: add functions to scope explicitly
this especially helps beginners with code readability, since the origin
of names is always immediately visible.
it's probably better not to show the manifest file documentation in the
command-specific pages, because these are implementation details that are not really practically useful.
this means no additional hassle for building the manual, but clutters
the table of contents a bit.
placed in a subsection of the binary install, the instructions are hard
to find. putting them in a separate page that is shown in the table of
contents should make it easier for users to find what they need when
they need it.
- Create a glossary entry for experimental features.
- Have the man page experimental feature notice link `nix-commmand`.
(Eventually this should be programmed, based on whether the command is
experimental, and if so what experimental feature does it depend on.)
- Document which installables depend on which experimental features.
I tried to use the same style (bold warning and block quote) that the
top of the man page uses.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
This function returns true or false depending on whether the Nix client
is trusted or not. Mostly relevant when speaking to a remote store with
a daemon.
We include this information in `nix ping store` and `nix doctor`
Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
They are put in the manual separate pages under the new overarching
description of experimental features.
The settings page just lists the valid experimental feature names (so
people know what a valid setting entry looks like), with links to those
pages. It doesn't attempt to describe each experimental feature as that
is too much information for the configuration settings section.
Add a page explaining what “experimental features” are, when and how they should be used
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
