`canonPath` and `absPath` work on native paths, and so should switch
between supporting Unix paths and Windows paths accordingly.
The templating is because `CanonPath`, which shares the implementation,
should always be Unix style. It is the pure "nix-native" path type for
virtual file operations --- it is part of Nix's "business logic", and
should not vary with the host OS accordingly.
The core `CanonPath` constructors were using `absPath`, but `absPath` in
some situations does IO which is not appropriate. It turns out that
these constructors avoided those situations, and thus were pure, but it
was far from obvious this was the case.
To remedy the situation, abstract the core algorithm from `canonPath` to
use separately in `CanonPath` without any IO. No we know by-construction
that those constructors are pure.
That leaves `CanonPath::fromCWD` as the only operation which uses IO /
is impure. Add docs on it, and `CanonPath` as a whole, explaining the
situation.
This is also necessary to support Windows paths on windows without
messing up `CanonPath`. But, I think it is good even without that.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Commit 83c067c0fa changed `builtins.pathExists`
to resolve symlinks before checking for existence. Consequently, if the path
refers to a symlink itself, existence of the target of the symlink (instead of
the symlink itself) was checked. Restore the previous behavior by skipping
symlink resolution in the last component.
No outward facing behavior is changed.
Older methods with same names that operate on on method + algo pair (for
old-style `<method>:algo`) are renamed to `*WithAlgo`.)
The functions are unit-tested in the same way the names for the hash
algorithms are tested.
for plain inherits this is really just a stylistic choice, but for
inherit-from it actually fixes an exponential size increase problem
during expr printing (as may happen during assertion failure reporting,
on during duplicate attr detection in the parser)
this also has the effect of sorting let bindings lexicographically
rather than by symbol creation order as was previously done, giving a
better canonicalization in the process.
When I started contributing to Nix, I found the mix of definitions and
names in `fmt.hh` to be rather confusing, especially the small
difference between `hintfmt` and `hintformat`. I've renamed many classes
and added documentation to most definitions.
- `formatHelper` is no longer exported.
- `fmt`'s documentation is now with `fmt` rather than (misleadingly)
above `formatHelper`.
- `yellowtxt` is renamed to `Magenta`.
`yellowtxt` wraps its value with `ANSI_WARNING`, but `ANSI_WARNING`
has been equal to `ANSI_MAGENTA` for a long time. Now the name is
updated.
- `normaltxt` is renamed to `Uncolored`.
- `hintfmt` has been merged into `hintformat` as extra constructor
functions.
- `hintformat` has been renamed to `hintfmt`.
- The single-argument `hintformat(std::string)` constructor has been
renamed to a static member `hintformat::interpolate` to avoid pitfalls
with using user-generated strings as format strings.
Pretty-print values in the REPL by printing each item in a list or
attrset on a separate line. When possible, single-item lists and
attrsets are printed on one line, as long as they don't contain a nested
list, attrset, or thunk.
Before:
```
{ attrs = { a = { b = { c = { }; }; }; }; list = [ 1 ]; list' = [ 1 2 3 ]; }
```
After:
```
{
attrs = {
a = {
b = {
c = { };
};
};
};
list = [ 1 ];
list' = [
1
2
3
];
}
```
Some tools which consume the "nix print-dev-env" rc script (such as
"nix-direnv") are sensitive to the use of unbound variables. They use
"set -u".
The "nix print-dev-env" rc script initially unsets "shellHook", then
loads variables from the derivation, and then evaluates "shellHook".
However, most derivations don't have a "shellHook" attribute.
So users get the error "shellHook: unbound variable". This can be
demonstrated with the command:
nix print-dev-env nixpkgs#hello | bash -u
This commit changes the rc script to provide an empty fallback value
for the "shellHook" variable.
Closes: #7951#8253
While preparing PRs like #9753, I've had to change error messages in
dozens of code paths. It would be nice if instead of
EvalError("expected 'boolean' but found '%1%'", showType(v))
we could write
TypeError(v, "boolean")
or similar. Then, changing the error message could be a mechanical
refactor with the compiler pointing out places the constructor needs to
be changed, rather than the error-prone process of grepping through the
codebase. Structured errors would also help prevent the "same" error
from having multiple slightly different messages, and could be a first
step towards error codes / an error index.
This PR reworks the exception infrastructure in `libexpr` to
support exception types with different constructor signatures than
`BaseError`. Actually refactoring the exceptions to use structured data
will come in a future PR (this one is big enough already, as it has to
touch every exception in `libexpr`).
The core design is in `eval-error.hh`. Generally, errors like this:
state.error("'%s' is not a string", getAttrPathStr())
.debugThrow<TypeError>()
are transformed like this:
state.error<TypeError>("'%s' is not a string", getAttrPathStr())
.debugThrow()
The type annotation has moved from `ErrorBuilder::debugThrow` to
`EvalState::error`.
As discussed in the maintainer meeting on 2024-01-29.
Mainly this is to avoid a situation where the name is parsed and
treated as a file name, mostly to protect users.
.-* and ..-* are also considered invalid because they might strip
on that separator to remove versions. Doesn't really work, but that's
what we decided, and I won't argue with it, because .-* probably
doesn't seem to have a real world application anyway.
We do still permit a 1-character name that's just "-", which still
poses a similar risk in such a situation. We can't start disallowing
trailing -, because a non-zero number of users will need it and we've
seen how annoying and painful such a change is.
What matters most is preventing a situation where . or .. can be
injected, and to just get this done.
Use `diff --color=always` to print colored output for language test
failures. I've also flipped the arguments so that expected lines missing
from the actual output will be marked with a red `-` and additional
lines found in the actual output will be marked with a green `+`.
Previously it was the other way around, which was very confusing.
This extends the `error: cannot coerce a TYPE to a string` message
to print the value that could not be coerced. This helps with debugging
by making it easier to track down where the value is being produced
from, especially in errors with deep or unhelpful stack traces.
More invariants are enforced in the type, and less state needs to be
stored in the main sink itself. The method here is roughly that known as
"session types".
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Low-hanging fruit in the spirit of #9753 and #9754 (means 9999years did
all the hard work already).
This basically prints out what was attempted to be called as function,
i.e.
map (import <nixpkgs> {}) [ 1 2 3 ]
now gives the following error message:
error:
… while calling the 'map' builtin
at «string»:1:1:
1| map (import <nixpkgs> {}) [ 1 2 3 ]
| ^
… while evaluating the first argument passed to builtins.map
error: expected a function but found a set: { _type = "pkgs"; AAAAAASomeThingsFailToEvaluate = «thunk»; AMB-plugins = «thunk»; ArchiSteamFarm = «thunk»; BeatSaberModManager = «thunk»; CHOWTapeModel = «thunk»; ChowCentaur = «thunk»; ChowKick = «thunk»; ChowPhaser = «thunk»; CoinMP = «thunk»; «18783 attributes elided»}
Do this if we want to do `--hash-algo` everywhere, and not `--algo` for
hash commands.
The new `nix hash convert` is updated. Deprecated new CLI commands are
left as-is (`nix hash path` needs to be redone and is also left as-is).
Add several tests for git fetching:
- shallow-cache-separation: can fetch the same repo shallowly and non-shallowly
- shallow-ignore-ref: ensure that ref gets ignored when shallow=true is set
- ssh-shallow: can fetch a git repo via ssh using shallow=1
Otherwise we get a stray `tests/functional/result`, which can cause
spurious failures later.
(I got a failure because the test temp dir effecting the store dir
changed. This caused a test later because Nix didn't want to remove the
old `result` because it wasn't pointing inside the new Nix store.)
Most of this is a `catch SysError` -> `catch SystemError` sed. This
is a rather pure-churn change I would like to get out of the way. **The
intersting part is `src/libutil/error.hh`.**
On Unix, we will only throw the `SysError` concrete class, which has
the same constructors that `SystemError` used to have.
On Windows, we will throw `WinError` *and* `SysError`. `WinError`
(which will be created in a later PR), will use a `DWORD` instead of
`int` error value, and `GetLastError()`, which is the Windows equivalent
of the `errno` machinery. Windows will *also* use `SysError` because
Window's "libc" (MSVCRT) implements the POSIX interface, and we use it
too.
As the docs describe, while we *throw* one of the 3 choices above (2
concrete classes or the alias), we should always *catch* `SystemError`.
This ensures no matter how the implementation changes for Windows (e.g.
between `SysError` and `WinError`) the catching logic stays the same
and stays correct.
Co-Authored-By volth <volth@volth.com>
Co-Authored-By Eugene Butler <eugene@eugene4.com>
Also fingerprint and some preparatory improvements.
Testing is still not up to scratch because lots of logic is duplicated
between the workdir and commit cases.
Enabled for fetchGit, which historically had this behavior,
among other behaviors we do not want in fetchGit.
fetchTree disables this parameter by default. It can choose the
simpler behavior, as it is still experimental.
I am not confident that the filtering implementation is future
proof. It should reuse a source filtering wrapper, which I believe
Eelco has already written, but not merged yet.
The Nix team has requested that this output format remain unchanged.
I've added a warning to the man page explaining that `nix-instantiate
--eval` output will not parse correctly in many situations.
Previously, there were two mostly-identical value printers -- one in
`libexpr/eval.cc` (which didn't force values) and one in
`libcmd/repl.cc` (which did force values and also printed ANSI color
codes).
This PR unifies both of these printers into `print.cc` and provides a
`PrintOptions` struct for controlling the output, which allows for
toggling whether values are forced, whether repeated values are tracked,
and whether ANSI color codes are displayed.
Additionally, `PrintOptions` allows tuning the maximum number of
attributes, list items, and bytes in a string that will be displayed;
this makes it ideal for contexts where printing too much output (e.g.
all of Nixpkgs) is distracting. (As requested by @roberth in
https://github.com/NixOS/nix/pull/9554#issuecomment-1845095735)
Please read the tests for example output.
Future work:
- It would be nice to provide this function as a builtin, perhaps
`builtins.toStringDebug` -- a printing function that never fails would
be useful when debugging Nix code.
- It would be nice to support customizing `PrintOptions` members on the
command line, e.g. `--option to-string-max-attrs 1000`.
solves #9388
This utilizes nixos vm tests to allow:
- writing tests for fetchTree and fetchGit involving actual networking.
- writing small independent test cases by automating local and remote repository setup per test case.
This adds:
- a gitea module setting up a gitea server
- a setup module that simplifies writing test cases by automating the repo setup.
- a simple git http test case
Other improvements:
For all nixos tests, add capability of overriding the nix version to test against.
This should make it easier to prevent regressions. If a new test is added it can simply be ran against any older nix version without having to backport the test.
For example, for running the container tests against nix 2.12.0:
`nix build "$(nix eval --raw .#hydraJobs.tests.containers --impure --apply 't: (t.forNix "2.12.0").drvPath')^*" -L`
This fixes a segfault on infinite function call recursion (rather than
infinite thunk recursion) by tracking the function call depth in
`EvalState`.
Additionally, to avoid printing extremely long stack traces, stack
frames are now deduplicated, with a `(19997 duplicate traces omitted)`
message. This should only really be triggered in infinite recursion
scenarios.
Before:
$ nix-instantiate --eval --expr '(x: x x) (x: x x)'
Segmentation fault: 11
After:
$ nix-instantiate --eval --expr '(x: x x) (x: x x)'
error: stack overflow
at «string»:1:14:
1| (x: x x) (x: x x)
| ^
$ nix-instantiate --eval --expr '(x: x x) (x: x x)' --show-trace
error:
… from call site
at «string»:1:1:
1| (x: x x) (x: x x)
| ^
… while calling anonymous lambda
at «string»:1:2:
1| (x: x x) (x: x x)
| ^
… from call site
at «string»:1:5:
1| (x: x x) (x: x x)
| ^
… while calling anonymous lambda
at «string»:1:11:
1| (x: x x) (x: x x)
| ^
… from call site
at «string»:1:14:
1| (x: x x) (x: x x)
| ^
(19997 duplicate traces omitted)
error: stack overflow
at «string»:1:14:
1| (x: x x) (x: x x)
| ^
Previously, IFDs would be built within the eval store, even though one
is typically using `--eval-store` precisely to *avoid* local builds.
Because the resulting Nix expression must be copied back to the eval
store in order to be imported, this requires the eval store to trust
the build store's signatures.
The profile manifest is now an object keyed on the name returned by
getNameFromURL() at installation time, instead of an array. This
ensures that the names of profile elements don't change when other
elements are added/removed.
On macOS in the `nix develop` shell, `make
tests/functional/logging.sh.test` errors:
++(logging.sh:18) mktemp
+(logging.sh:18) builder=/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/tmp.StuabKUhMh
+(logging.sh:19) echo -e '#!/bin/sh\nmkdir $out'
+++(logging.sh:22) mktemp -d
++(logging.sh:22) nix-build -E 'with import ./config.nix; mkDerivation { name = "fnord"; builder = /var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/tmp.StuabKUhMh; }' --out-link /var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/tmp.oaKcy0NXqC/result
error:
… while calling the 'derivationStrict' builtin
at <nix/derivation-internal.nix>:9:12:
8|
9| strict = derivationStrict drvAttrs;
| ^
10|
… while evaluating derivation 'fnord'
whose name attribute is located at «string»:1:42
… while evaluating attribute 'args' of derivation 'fnord'
at /Users/wiggles/nix/tests/functional/config.nix:23:7:
22| builder = shell;
23| args = ["-e" args.builder or (builtins.toFile "builder-${args.name}.sh" ''
| ^
24| if [ -e "$NIX_ATTRS_SH_FILE" ]; then source $NIX_ATTRS_SH_FILE; fi;
error: path '/var' is a symlink
+(logging.sh:22) outp=
++(logging.sh:22) onError
++(/Users/wiggles/nix/tests/functional/common/vars-and-functions.sh:237) set +x
logging.sh: test failed at:
main in logging.sh:22
This is because `mktemp` returns a path like
`/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/tmp.qDY24l6bIM`,
where `/var` is a symlink to `/private/var`.
Then, we attempt to use that path as a `builder`, which errors because
symlinks are impure or whatever.
Anyways, we can fix this by using `realpath "$(mktemp)"` instead of
`mktemp` directly.
NB: This error doesn't seem to happen when I run the tests through `nix
flake check`. I'm not sure if Nix does something to `TMP` in that case.
As part of the CLI stabilization effort, the last remaining checkbox (at
the moment) for `nix daemon` is that it "needs testing". This implements
the proposal of using `nix daemon` in place of `nix-daemon` in the test
suite.
`nix flake check` had these warnings:
trace: warning: Module argument `nodes.client.config` is deprecated. Use `nodes.client` instead.
trace: warning: Module argument `nodes.client.config` is deprecated. Use `nodes.client` instead.
trace: warning: The option `services.openssh.permitRootLogin' defined in `/nix/store/3m3hfpmbjdf4w39qfjami7ljhvhczay1-source/tests/nixos/nix-copy.nix' has been renamed to `services.openssh.settings.PermitRootLogin'.
trace: warning: Module argument `nodes.http_dns.config` is deprecated. Use `nodes.http_dns` instead.
trace: warning: Module argument `nodes.github.config` is deprecated. Use `nodes.github` instead.
trace: warning: Module argument `nodes.sourcehut.config` is deprecated. Use `nodes.sourcehut` instead.
This keeps hint messages, source location information, and source code
snippets grouped together, while making stack traces shorter (so that
more stack frames can be viewed on the same terminal).
Before:
error:
… while evaluating the attribute 'body'
at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:4:3:
3|
4| body = x "x";
| ^
5| }
… from call site
at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:4:10:
3|
4| body = x "x";
| ^
5| }
… while calling 'x'
at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:2:7:
1| let {
2| x = arg: assert arg == "y"; 123;
| ^
3|
error: assertion '(arg == "y")' failed
at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:2:12:
1| let {
2| x = arg: assert arg == "y"; 123;
| ^
3|
After:
error:
… while evaluating the attribute 'body'
at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:4:3:
3|
4| body = x "x";
| ^
5| }
… from call site
at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:4:10:
3|
4| body = x "x";
| ^
5| }
… while calling 'x'
at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:2:7:
1| let {
2| x = arg: assert arg == "y"; 123;
| ^
3|
error: assertion '(arg == "y")' failed
at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:2:12:
1| let {
2| x = arg: assert arg == "y"; 123;
| ^
3|
`eval-system` option overrides just the value of `builtins.currentSystem`.
This is more useful than overriding `system` since you can build these
derivations on remote builders which can work on the given system.
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
It works with both `ssh://` and `ssh-ng://` now since #9600 (and
`ssh-ng:// didn't work before that).
Also, by making the two tests share code, we nudge ourselves towards
making sure there is feature parity.
I don't love the way this code looks. There are two larger problems:
- eval, build/scratch, destination stores (#5025) should have different
types to reflect the fact that they are used for different purposes
and those purposes correspond to different operations. It should be
impossible to "use the wrong store" in my cases.
- Since drvs can end up in both the eval and build/scratch store, we
should have some sort of union/layered store (not on the file sytem
level, just conceptual level) that allows accessing both. This would
get rid of the ugly "check both" boilerplate in this PR.
Still, it might be better to land this now / soon after minimal cleanup,
so we have a concrete idea of what problem better abstractions are
supposed to solve.
* Print the value in `error: cannot coerce` messages
This extends the `error: cannot coerce a TYPE to a string` message
to print the value that could not be coerced. This helps with debugging
by making it easier to track down where the value is being produced
from, especially in errors with deep or unhelpful stack traces.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
This includes position information in more places, making debugging
easier.
Before:
```
$ nix-instantiate --show-trace --eval tests/functional/lang/eval-fail-using-set-as-attr-name.nix
error:
… while evaluating an attribute name
at «none»:0: (source not available)
error: value is a set while a string was expected
```
After:
```
error:
… while evaluating an attribute name
at /pwd/lang/eval-fail-using-set-as-attr-name.nix:5:10:
4| in
5| attr.${key}
| ^
6|
error: value is a set while a string was expected
```
In the process, partially undo e89b5bd0bf
in that the ancient < 2.4 version is now supported again by the
serializer again. `LegacySSHStore`, instead of also asserting that the
version is at least 4, just checks that `narHash` is set.
This allows us to better test the serializer in isolation for both
versions (< 4 and >= 4).
Today, with the tests inside a `tests` intermingled with the
corresponding library's source code, we have a few problems:
- We have to be careful that wildcards don't end up with tests being
built as part of Nix proper, or test headers being installed as part
of Nix proper.
- Tests in libraries but not executables is not right:
- It means each executable runs the previous unit tests again, because
it needs the libraries.
- It doesn't work right on Windows, which doesn't want you to load a
DLL just for the side global variable . It could be made to work
with the dlopen equivalent, but that's gross!
This reorg solves these problems.
There is a remaining problem which is that sibbling headers (like
`hash.hh` the test header vs `hash.hh` the main `libnixutil` header) end
up shadowing each other. This PR doesn't solve that. That is left as
future work for a future PR.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
`installcheck` doesn't yet work, but the rest of the build can now
happen mostly inside a separate build directory.
Progress on #9342
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
The basic idea here is to separate a few intertwined notions:
1. Not all "run bash tests" are "install tests"
2. Not all "run bash tests" use `tests/functional/init.sh`, or any
pre-test initialization at all.
This will used in the next commit when we have a test that check unit
test golden master data.
Also, move our custom `PS4` from the test to the test runner, as it is
part of how we want to display the tests, not the test themselves.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
As discussed in our last meeting, we need a bit more time, but we are
"time boxing" the work left to do to ensure there is not unbounded
delay.
Rather than putting it back underneath `flakes`, though, put it
underneath its own `fetch-tree` experimental feature (which `flakes`
includes/implies). This signals our commitment to the plan to stabilize
it first without waiting to go through the rest of Flakes, and also will
give users a "release candidate" when we get closer to stabilization.
This reverts commit 4112dd1fc9.
Enables shebang usage of nix shell. All arguments with `#! nix` get
added to the nix invocation. This implementation does NOT set any
additional arguments other than placing the script path itself as the
first argument such that the interpreter can utilize it.
Example below:
```
#!/usr/bin/env nix
#! nix shell --quiet
#! nix nixpkgs#bash
#! nix nixpkgs#shellcheck
#! nix nixpkgs#hello
#! nix --ignore-environment --command bash
# shellcheck shell=bash
set -eu
shellcheck "$0" || exit 1
function main {
hello
echo 0:"$0" 1:"$1" 2:"$2"
}
"$@"
```
fix: include programName usage
EDIT: For posterity I've changed shellwords to shellwords2 in order
not to interfere with other changes during a rebase.
shellwords2 is removed in a later commit. -- roberth
Users may select specific outputs using the ^output syntax or selecting
any output using ^*.
URL parsing currently doesn't support these kinds of output references:
parsing will fail.
Currently `queryRegex` was reused for URL fragments, which didn't
include support for ^. Now queryRegex has been split from fragmentRegex,
where only the fragmentRegex supports ^.
