This fixes a bug in commands like `nix eval' which would emit invalid attribute
sets if they contained reserved keywords such as "assert", "let", etc.
These keywords will not be quoted when printed, making them valid expressions.
All keywords recognized by the lexer are quoted except "or", which does not
require quotation.
fe5509df caused only wanted outputs to be passed to the
post-build-hook, which resulted in paths being built
without ever going into the hook.
This commit adds a (currently failing) test for this.
Previously, we relied on the `shutdown()` function to terminate `accept()`
calls on a listening socket. However, this approach did not work on macOS as
the waiting `accept()` call is not considered a connected socket, resulting in
an `ENOTCONN` error. Instead, we now close the listening socket to terminate
the `accept()` call.
Additionally, we fixed a resource management issue where we set the
`daemonSocket` variable to -1, triggering resource cleanup and causing the
`stopDaemon` function to be called twice. This resulted in errors as the socket
was already closed by the time the second `stopDaemon` call was made. Instead of
setting `daemonSocket` to -1, we now release the socket using the `release()`
method on a unique pointer. This properly transfers ownership and allows for
correct resource cleanup.
These changes ensure proper behavior and resource management for the
recursive-nix feature on macOS.
We finally test the status quo of remote build trust in a number of
ways. We create a new experimental feature on `nix-daemon` to do so.
PR #3921, which improves the situation with trustless remote building,
will build upon these changes. This code / tests was pull out of there
to make this, so everything is easier to review, and in particular we
test before and after so the new behavior in that PR is readily apparent
from the testsuite diff alone.
Issues:
1. Features gated on disabled experimental settings should warn and be
ignored, not silently succeed.
2. Experimental settings in the same config "batch" (file or env var)
as the enabling of the experimental feature should work.
3. For (2), the order should not matter.
These are analogous to the issues @roberth caught with my changes for
arg handling, but they are instead for config handling.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
This is somewhat hacky fix just for 2.15. I unintentionally hid them
from the manual, when no one wanted to hide them that (including
myself). I also required the experimental feature to be enabled in an
order-dependent way, which is not good.
The simplest fix for this immanent release is just to always show them,
and always allow them to be set.
Effectively undoes some changes from aa663b7e89
This function returns true or false depending on whether the Nix client
is trusted or not. Mostly relevant when speaking to a remote store with
a daemon.
We include this information in `nix ping store` and `nix doctor`
Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
This introduces the SourcePath type from lazy-trees as an abstraction
for accessing files from inputs that may not be materialized in the
real filesystem (e.g. Git repositories). Currently, however, it's just
a wrapper around CanonPath, so it shouldn't change any behaviour. (On
lazy-trees, SourcePath is a <InputAccessor, CanonPath> tuple.)
Prior to this, there was an ad-hoc whitelist in `main.cc`. Now, every
command states its stability.
In a future PR, we will adjust the manual to take advantage of this new
information in the JSON.
(It will be easier to do that once we have some experimental feature
docs to link too; see #5930 and #7798.)
Was probably an overlook of when the tests were first added, but that
now messes-up with the `nix-channel --update` that happens down the line
(and can't access the network since we're inside a Nix build)
See the note in the test.
We don't want these flags showing up for commands where they are
irrelevant.
Eventually, this needs a proper fix, but it need not be a blocker for
stabilize: for a quick-n-dirty punt, just put these flags behind the
`nix-command` unstable feature.
This is fine because they are only relevant for commands which we don't
need to stabilize for a while.
`legacyPackages` of nixpkgs trigger eval errors in `hasContent`, causing
the whole `legacyPackages` being skipped. We should treat it as
has-content in that case.
I noticed a regression in the lazy-trees branch, which I'm trying to
capture with this test. While the tests succeeds in master, the
lazy-trees branch gives the following error message:
error: access to path
'/build/nix-test/tests/flakes/flake-in-submodule/rootRepo/submodule/flake.nix'
is forbidden because it is not under Git control; maybe you should
'git add' it to the repository
'/build/nix-test/tests/flakes/flake-in-submodule/rootRepo'?
- Try not to put cryptic "99" in many places
Factor out `exit 99` into `skipTest` function
- Alows make sure skipping a test is done with a reason
`skipTest` takes a mandatory argument
- Separate pure conditionals vs side-effectful test skipping.
"require daemon" already had this, but "sandbox support" did not.
Add the --base64 and --sri flags for the Base64 and SRI format output.
Add the --base16 flag to explicitly specify the hexadecimal format.
Add the --to-base64 and --to-sri flag to convert a hash to the above
mentioned format.
Use `set -u` and `set -o pipefail` to catch accidental mistakes and
failures more strongly.
- `set -u` catches the use of undefined variables
- `set -o pipefail` catches failures (like `set -e`) earlier in the
pipeline.
This makes the tests a bit more robust. It is nice to read code not
worrying about these spurious success paths (via uncaught) errors
undermining the tests. Indeed, I caught some bugs doing this.
There are a few tests where we run a command that should fail, and then
search its output to make sure the failure message is one that we
expect. Before, since the `grep` was the last command in the pipeline
the exit code of those failing programs was silently ignored. Now with
`set -o pipefail` it won't be, and we have to do something so the
expected failure doesn't accidentally fail the test.
To do that we use `expect` and a new `expectStderr` to check for the
exact failing exit code. See the comments on each for why.
`grep -q` is replaced with `grepQuiet`, see the comments on that
function for why.
`grep -v` when we just want the exit code is replaced with `grepInverse,
see the comments on that function for why.
`grep -q -v` together is, surprise surprise, replaced with
`grepQuietInverse`, which is both combined.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
The release notes document the change in behavior, I don't include it
here so there is no risk to it getting out of sync.
> Motivation
>> Plumbing CLI should be simple
Store derivation installations are intended as "plumbing": very simple
utilities for advanced users and scripts, and not what regular users
interact with. (Similarly, regular Git users will use branch and tag
names not explicit hashes for most things.)
The plumbing CLI should prize simplicity over convenience; that is its
raison d'etre. If the user provides a path, we should treat it the same
way not caring what sort of path it is.
>> Scripting
This is especially important for the scripting use-case. when arbitrary
paths are sent to e.g. `nix copy` and the script author wants consistent
behavior regardless of what those store paths are. Otherwise the script
author needs to be careful to filter out `.drv` ones, and then run `nix
copy` again with those paths and `--derivation`. That is not good!
>> Surprisingly low impact
Only two lines in the tests need changing, showing that the impact of
this is pretty light.
Many command, like `nix log` will continue to work with just the
derivation passed as before. This because we used to:
- Special case the drv path and replace it with it's outputs (what this
gets rid of).
- Turn those output path *back* into the original drv path.
Now we just skip that entire round trip!
> Context
Issue #7261 lays out a broader vision for getting rid of `--derivation`,
and has this as one of its dependencies. But we can do this with or
without that.
`Installable::toDerivations` is changed to handle the case of a
`DerivedPath::Opaque` ending in `.drv`, which is new: it simply doesn't
need to do any extra work in that case. On this basis, commands like
`nix {show-derivation,log} /nix/store/...-foo.drv` still work as before,
as described above.
When testing older daemons, the post-build-hook will be run against the
old CLI, so we need the old version of the post-build-hook to support
that use-case.
Co-authored-by: Travis A. Everett <travis.a.everett@gmail.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Split `common.sh` into the vars and functions definitions vs starting
the daemon (and possibly other initialization logic). This way,
`init.sh` can just `source` the former. Trying to start the daemon
before `nix.conf` is written will fail because `nix daemon` requires
`--experimental-features 'nix-command'`.
`killDaemon` is idempotent, so it's safe to call when no daemon is
running.
`startDaemon` and `killDaemon` use the PID (which is now exported to
subshells) to decide whether there is work to be done, rather than
`NIX_REMOTE`, which might conceivably be set differently even if a
daemon is running.
`startDaemon` and `killDaemon` can save/restore the old `NIX_REMOTE` as
`NIX_REMOTE_OLD`.
`init.sh` kills daemon before deleting everything (including the daemon
socket).
`init.sh` is tested on its own. We used to do that. I deleted it in
4720853129 but I am not sure why. Better
to just restore it; at one point working on this every other test
passed, so seems good to check whether `init.sh` can be run twice.
We don't *need* to run `init.sh` twice, but I want to try to make our
tests as robust as possible so that manual debugging (where tests for
better or worse might be run ways that we didn't expect) is less
fragile.
It would be incorrect to say that the `sourceInfo` has an `outPath`
that isn't the root. `sourceInfo` is about the root, whereas only
the flake may not be about the root. Thanks Eelco for pointing that
out.
- `nixpkgsFor` does all of native, static, cross, and the different stdenvs.
- The main Nix derivation is no longer duplicated for static.
- DRY nixpkgs.lib and lib.genAttrs calls.
Adding a test to ensure there is no regression.
The tests that are split out of `tests/build.sh` are ones that don't yet
work with CA derivation. I have not yet evaluated whether they should or
not.
This behavior, reported missing in issue #4661, already got fixed in
PR #4818, but didn't get a test case then.
XDG Base Directory is a standard for locations for storing various
files. Nix has a few files which seem to fit in the standard, but
currently use a custom location directly in the user's ~, polluting
it:
- ~/.nix-profile
- ~/.nix-defexpr
- ~/.nix-channels
This commit adds a config option (use-xdg-base-directories) to follow
the XDG spec and instead use the following locations:
- $XDG_STATE_HOME/nix/profile
- $XDG_STATE_HOME/nix/defexpr
- $XDG_STATE_HOME/nix/channels
If $XDG_STATE_HOME is not set, it is assumed to be ~/.local/state.
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
Co-authored-by: Tim Fenney <kodekata@gmail.com>
Co-authored-by: pasqui23 <pasqui23@users.noreply.github.com>
Co-authored-by: Artturin <Artturin@artturin.com>
Co-authored-by: John Ericson <Ericson2314@Yahoo.com>
For frameworks it's important that structures are as lazy as possible
to prevent infinite recursions, performance issues and errors that
aren't related to the thing to evaluate. As a consequence, they have
to emit more attributes than strictly (sic) necessary.
However, these attributes with empty values are not useful to the user
so we omit them.
Previously, getDefaultNixPath was called too early: at initialisation
time, before CLI and config have been processed, when `restrictEval` and
`pureEval` both have their default value `false`. Call it when
initialising the EvalState instead, and use `setDefault`.
Add an `$` at the end of the `grep` regex. Without it, `checkRef foo`
would always imply `checkRef foo.drv`. We want to tell these situations
apart to more precisely test what is going on.
Allows checking directory entry type of a single file/directory.
This was added to optimize the use of `builtins.readDir` on some
filesystems and operating systems which cannot detect this information
using POSIX's `readdir`.
Previously `builtins.readDir` would eagerly use system calls to lookup
these filetypes using other interfaces; this change makes these
operations lazy in the attribute values for each file with application
of `builtins.readFileType`.
The original `builtins.getContext` test from
1d757292d0 would have caught this. The
problem is that b30be6b450 adding
`builtins.appendContext` modified that test to make it test too much at
once, rather than adding a separate test.
We now have isolated tests for both functions, and also a property test
showing everything put together (in the form of an eta rule for strings
with context). This is better coverage and properly reproduces the bug.
Rather than using `/nix/var/nix/{profiles,gcroots}/per-user/`, put the user
profiles and gcroots under `$XDG_DATA_DIR/nix/{profiles,gcroots}`.
This means that the daemon no longer needs to manage these paths itself
(they are fully handled client-side). In particular, it doesn’t have to
`chown` them anymore (removing one need for root).
This does change the layout of the gc-roots created by nix-env, and is
likely to break some stuff, so I’m not sure how to properly handle that.
Prior to this change, we had a bunch of ad-hoc string manipulation code
scattered around. This made it hard to figure out what data model for
string contexts is.
Now, we still store string contexts most of the time as encoded strings
--- I was wary of the performance implications of changing that --- but
whenever we parse them we do so only through the
`NixStringContextElem::parse` method, which handles all cases. This
creates a data type that is very similar to `DerivedPath` but:
- Represents the funky `=<drvpath>` case as properly distinct from the
others.
- Only encodes a single output, no wildcards and no set, for the
"built" case.
(I would like to deprecate `=<path>`, after which we are in spitting
distance of `DerivedPath` and could maybe get away with fewer types, but
that is another topic for another day.)
Adds a new boolean structured attribute
`outputChecks.<output>.unsafeDiscardReferences` which disables scanning
an output for runtime references.
__structuredAttrs = true;
outputChecks.out.unsafeDiscardReferences = true;
This is useful when creating filesystem images containing their own embedded Nix
store: they are self-contained blobs of data with no runtime dependencies.
Setting this attribute requires the experimental feature
`discard-references` to be enabled.
Without the change checks issue the fllowing warning:
$ nix flake check
trace: warning: The option `nix.useSandbox' defined in `makeTest parameters' has been renamed to `nix.settings.sandbox'.
trace: warning: The option `nix.useSandbox' defined in `makeTest parameters' has been renamed to `nix.settings.sandbox'.
trace: warning: The option `nix.maxJobs' defined in `makeTest parameters' has been renamed to `nix.settings.max-jobs'.
...
First, logic is consolidated in the shell script instead of being spread
between them and makefiles. That makes understanding what is going on a
little easier.
This would not be super interesting by itself, but it gives us a way to
debug tests more easily. *That* in turn I hope is much more compelling.
See the updated manual for details.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
This makes 'nix build' work on paths (which will be copied to the
store) and store paths (returned as is). E.g. the following flake
output attributes can be built using 'nix build .#foo':
foo = ./src;
foo = self.outPath;
foo = builtins.fetchTarball { ... };
foo = (builtins.fetchTree { .. }).outPath;
foo = builtins.fetchTree { .. } + "/README.md";
foo = builtins.storePath /nix/store/...;
Note that this is potentially risky, e.g.
foo = /.;
will cause Nix to try to copy the entire file system to the store.
What doesn't work yet:
foo = self;
foo = builtins.fetchTree { .. };
because we don't handle attrsets with an outPath attribute in it yet,
and
foo = builtins.storePath /nix/store/.../README.md;
since result symlinks have to point to a store path currently (rather
than a file inside a store path).
Fixes#7417.
This makes the position object used in exceptions abstract, with a
method getSource() to get the source code of the file in which the
error originated. This is needed for lazy trees because source files
don't necessarily exist in the filesystem, and we don't want to make
libutil depend on the InputAccessor type in libfetcher.
Fix#6209
When trying to run `nix log <installable>`, try first to resolve the derivation pointed to
by `<installable>` as it is the resolved one that holds the build log.
This has a couple of shortcomings:
1. It’s expensive as it requires re-reading the derivation
2. It’s brittle because if the derivation doesn’t exist anymore or can’t
be resolved (which is the case if any one of its build inputs is missing),
then we can’t access the log anymore
However, I don’t think we can do better (at least not right now).
The alternatives I see are:
1. Copy the build log for the un-resolved derivation. But that means a
lot of duplication
2. Store the results of the resolving in the db. Which might be the best
long-term solution, but leads to a whole new class of potential
issues.
This is a really old test case (which was originally written before the
proper Nix syntax). The tested deep comparison behavior was implemented
and reverted soon after due to performance problems, but it has been
restored in today's Nix again (thanks to the derivation comparison
optimization, presumably).
* Clarify the documentation of foldl': That the arguments are forced
before application (?) of `op` is necessarily true. What is important
to stress is that we force every application of `op`, even when the
value turns out to be unused.
* Move the example before the comment about strictness to make it less
confusing: It is a general example and doesn't really showcase anything
about foldl' strictness.
* Add test cases which nail down aspects of foldl' strictness:
* The initial accumulator value is not forced unconditionally.
* Applications of op are forced.
* The list elements are not forced unconditionally.
This commit adds a test covering the discrepancy between parseDrvName's
implementation and documentation (the discrepancy was eliminated in the previous
commit).
I just had a colleague get confused by the previous phrase for good
reason. "valid" sounds like an *objective* criterion, e.g. and *invalid
signature* would be one that would be trusted by no one, e.g. because it
misformatted or something.
What is actually going is that there might be a signature which is
perfectly valid to *someone else*, but not to the user, because they
don't trust the corresponding public key. This is a *subjective*
criterion, because it depends on the arbitrary and personal choice of
which public keys to trust.
I therefore think "trustworthy" is a better adjective to use. Whether
something is worthy of trust is clearly subjective, and then "trust"
within that word nicely evokes `trusted-public-keys` and friends.
This runs the installer in a QEMU VM. Unlike the old installer test
that ran inside a declaratively built RedHat/Debian image, this uses
an image from Vagrant.
`--override-input` id snarky because it takes two arguments, so it
doesn't play well when completed in the middle of the CLI (since the
argument just after gets interpreted as its second argument). So use
`--update-input` instead
I recently got fairly confused why the following expression didn't have
any effect
{
description = "Foobar";
inputs.sops-nix = {
url = github:mic92/sops-nix;
inputs.nixpkgs_22_05.follows = "nixpkgs";
};
}
until I found out that the input was called `nixpkgs-22_05` (please note
the dash vs. underscore).
IMHO it's not a good idea to not throw an error in that case and
probably leave end-users rather confused, so I implemented a small check
for that which basically checks whether `follows`-declaration from
overrides actually have corresponding inputs in the transitive flake.
In fact this was done by accident already in our own test-suite where
the removal of a `follows` was apparently forgotten[1].
Since the key of the `std::map` that holds the `overrides` is a vector
and we have to find the last element of each vector (i.e. the override)
this has to be done with a for loop in O(n) complexity with `n` being
the total amount of overrides (which shouldn't be that large though).
Please note that this doesn't work with nested expressions, i.e.
inputs.fenix.inputs.nixpkgs.follows = "...";
which is a known problem[2].
For the expression demonstrated above, an error like this will be
thrown:
error: sops-nix has a `follows'-declaration for a non-existant input nixpkgs_22_05!
[1] 2664a216e5
[2] https://github.com/NixOS/nix/issues/5790
- Don't use `printf` for the expected result, but just use bash's `$' '`
litteral strings
- Quote the `nix` call result
- Invert the order in the comparisons (just because it feels more
natural)
Basically an attempt to resume fixing #5543 for a breakage introduced
earlier[1]. Basically, when evaluating an older `nixpkgs` with
`nix-shell` the following error occurs:
λ ma27 [~] → nix-shell -I nixpkgs=channel:nixos-18.03 -p nix
error: anonymous function at /nix/store/zakqwc529rb6xcj8pwixjsxscvlx9fbi-source/pkgs/top-level/default.nix:20:1 called with unexpected argument 'inNixShell'
at /nix/store/zakqwc529rb6xcj8pwixjsxscvlx9fbi-source/pkgs/top-level/impure.nix:82:1:
81|
82| import ./. (builtins.removeAttrs args [ "system" "platform" ] // {
| ^
83| inherit config overlays crossSystem;
This is a problem because one of the main selling points of Nix is that
you can evaluate any old Nix expression and still get the same result
(which also means that it *still evaluates*). In fact we're deprecating,
but not removing a lot of stuff for that reason such as unquoted URLs[2]
or `builtins.toPath`. However this property was essentially thrown away
here.
The change is rather simple: check if `inNixShell` is specified in the
formals of an auto-called function. This means that
{ inNixShell ? false }:
builtins.trace inNixShell
(with import <nixpkgs> { }; makeShell { name = "foo"; })
will show `trace: true` while
args@{ ... }:
builtins.trace args.inNixShell
(with import <nixpkgs> { }; makeShell { name = "foo"; })
will throw the following error:
error: attribute 'inNixShell' missing
This is explicitly needed because the function in
`pkgs/top-level/impure.nix` of e.g. NixOS 18.03 has an ellipsis[3], but
passes the attribute-set on to another lambda with formals that doesn't
have an ellipsis anymore (hence the error from above). This was perhaps
a mistake, but we can't fix it anymore. This also means that there's
AFAICS no proper way to check if the attr-set that's passed to the Nix
code via `EvalState::autoCallFunction` is eventually passed to a lambda
with formals where `inNixShell` is missing.
However, this fix comes with a certain price. Essentially every
`shell.nix` that assumes `inNixShell` to be passed to the formals even
without explicitly specifying it would break with this[4]. However I think
that this is ugly, but preferable:
* Nix 2.3 was declared stable by NixOS up until recently (well, it still
is as long as 21.11 is alive), so most people might not have even
noticed that feature.
* We're talking about a way shorter time-span with this change being
in the wild, so the fallout should be smaller IMHO.
[1] 9d612c393a
[2] https://github.com/NixOS/rfcs/pull/45#issuecomment-488232537
[3] https://github.com/NixOS/nixpkgs/blob/release-18.03/pkgs/top-level/impure.nix#L75
[4] See e.g. the second expression in this commit-message or the changes
for `tests/ca/nix-shell.sh`.
nixos/nix#6290 introduced a regex pattern to account for tags when
resolving sourcehut refs. nixos/nix#4638 reafactored the code,
accidentally treating the pattern as a regular string, causing all
non-HEAD ref resolving to break.
This fixes the regression and adds more test cases to avoid future
breakage.
To quote Eelco in #5867:
> Unfortunately we can't do
>
> evalSettings.pureEval.setDefault(false);
>
> because then we have to do the same in main.cc (where
> pureEval is set to true), and that would allow pure-eval
> to be disabled globally from nix.conf.
Instead, a command should specify that it should be impure by
default. Then, `evalSettings.pureEval` will be set to `false;` unless
it's overridden by e.g. a CLI flag.
In that case it's IMHO OK to be (theoretically) able to override
`pure-eval` via `nix.conf` because it doesn't have an effect on commands
where `forceImpureByDefault` returns `false` (i.e. everything where pure
eval actually matters).
Closes#5867
- Test that without the XP feature things work as before
- Test that with or without the XP feature `--file file` works
- Test that with XP feature passing a flakeref works
- Test `:reload` with a flake
Add a new `file` fetcher type, which will fetch a plain file over
http(s), or from the local file.
Because plain `http(s)://` or `file://` urls can already correspond to
`tarball` inputs (if the path ends-up with a know archive extension),
the URL parsing logic is a bit convuluted in that:
- {http,https,file}:// urls will be interpreted as either a tarball or a
file input, depending on the extensions of the path part (so
`https://foo.com/bar` will be a `file` input and
`https://foo.com/bar.tar.gz` as a `tarball` input)
- `file+{something}://` urls will be interpreted as `file` urls (with
the `file+` part removed)
- `tarball+{something}://` urls will be interpreted as `tarball` urls (with
the `tarball+` part removed)
Fix#3785
Co-Authored-By: Tony Olagbaiye <me@fron.io>
'nix profile install' will now install all outputs listed in the
package's meta.outputsToInstall attribute, or all outputs if that
attribute doesn't exist. This makes it behave consistently with
nix-env. Fixes#6385.
Furthermore, for consistency, all other 'nix' commands do this as
well. E.g. 'nix build' will build and symlink the outputs in
meta.outputsToInstall, defaulting to all outputs. Previously, it only
built/symlinked the first output. Note that this means that selecting
a specific output using attrpath selection (e.g. 'nix build
nixpkgs#libxml2.dev') no longer works. A subsequent PR will add a way
to specify the desired outputs explicitly.
The produced path is then allowed be imported or utilized elsewhere:
```
assert (43 == import (builtins.toFile "source" "43")); "good"
```
This will still fail on write-only stores.
Pos objects are somewhat wasteful as they duplicate the origin file name and
input type for each object. on files that produce more than one Pos when parsed
this a sizeable waste of memory (one pointer per Pos). the same goes for
ptr<Pos> on 64 bit machines: parsing enough source to require 8 bytes to locate
a position would need at least 8GB of input and 64GB of expression memory. it's
not likely that we'll hit that any time soon, so we can use a uint32_t index to
locate positions instead.
The test illustrates failure in issue #5320. Here derivation and
it's built input have identical CA sotre path. As a result we generate
extraneout reference to build input:
$ make installcheck
...
ran test tests/selfref-gc.sh... [PASS]
ran test tests/ca/selfref-gc.sh... [FAIL]
...
deleting '/tmp/.../tests/ca/selfref-gc/store/iqciq1mpg5hc7p6a52fp2bjxbyc9av0v-selfref-gc'
deleting '/tmp/...tests/ca/selfref-gc/store/zh0kwpnirw3qbv6dl1ckr1y0kd5aw6ax-selfref-gc.drv'
error: executing SQLite statement
'delete from ValidPaths where path = '/tmp/.../tests/ca/selfref-gc/store/fsjq0k146r85lsh01l0icl30rnhv7z72-selfref-gc';':
constraint failed (in '/tmp/.../tests/ca/selfref-gc/var/nix/db/db.sqlite')
Requested by ppepino on the Matrix:
https://matrix.to/#/!KqkRjyTEzAGRiZFBYT:nixos.org/$Tb32BS3rVE2BSULAX4sPm0h6CDewX2hClOTGzTC7gwM?via=nixos.org&via=matrix.org&via=nixos.dev
This adds a new command, :bl, which works like :b but also creates
a GC root symlink to the various derivation outputs.
ckie@cookiemonster ~/git/nix -> ./outputs/out/bin/nix repl
Welcome to Nix 2.6.0. Type :? for help.
nix-repl> :l <nixpkgs>
Added 16118 variables.
nix-repl> :b runCommand "hello" {} "echo hi > $out"
This derivation produced the following outputs:
./repl-result-out -> /nix/store/kidqq2acdpi05c4a9mlbg2baikmzik44-hello
[1 built, 0.0 MiB DL]
ckie@cookiemonster ~/git/nix -> cat ./repl-result-out
hi
Impure derivations are derivations that can produce a different result
every time they're built. Example:
stdenv.mkDerivation {
name = "impure";
__impure = true; # marks this derivation as impure
outputHashAlgo = "sha256";
outputHashMode = "recursive";
buildCommand = "date > $out";
};
Some important characteristics:
* This requires the 'impure-derivations' experimental feature.
* Impure derivations are not "cached". Thus, running "nix-build" on
the example above multiple times will cause a rebuild every time.
* They are implemented similar to CA derivations, i.e. the output is
moved to a content-addressed path in the store. The difference is
that we don't register a realisation in the Nix database.
* Pure derivations are not allowed to depend on impure derivations. In
the future fixed-output derivations will be allowed to depend on
impure derivations, thus forming an "impurity barrier" in the
dependency graph.
* When sandboxing is enabled, impure derivations can access the
network in the same way as fixed-output derivations. In relaxed
sandboxing mode, they can access the local filesystem.
The new implementation relies on tab separting the hash and ref (this is how sourcehut does it). This fixes the integration test to use a tab instead of a space.
Don’t try and assume that we know the output paths when we’ve just built
with `--dry-run`. Instead make `--dry-run` follow a different code path
that won’t assume the knowledge of the output paths at all.
Fix#6275
```console
$ nix eval --expr '({ foo ? 1 }: foo) { fob = 2; }'
error: anonymous function at (string):1:2 called with unexpected argument 'fob'
at «string»:1:1:
1| ({ foo ? 1 }: foo) { fob = 2; }
| ^
Did you mean foo?
```
Not that because Nix will first check for _missing_ arguments before
checking for extra arguments, `({ foo }: foo) { fob = 1; }` will
complain about the missing `foo` argument (rather than extra `fob`) and
so won’t display a suggestion.
Make the evaluator show some suggestions when trying to access an
invalid field from an attrset.
```console
$ nix eval --expr '{ foo = 1; }.foa'
error: attribute 'foa' missing
at «string»:1:1:
1| { foo = 1; }.foa
| ^
Did you mean foo?
```
Fixes
nix-daemon: src/libstore/sqlite.cc:97: nix::SQLiteStmt::Use::Use(nix::SQLiteStmt&): Assertion `stmt.stmt' failed.
which happens because the daemon doesn't properly handle the case
where ca-derivations isn't enabled at daemon startup.
A few notes:
* The `echo hi` is needed to make sure that a file that can be read by
`nix log` is properly created (i.e. some output is needed). This is
known and to be fixed in #6051.
* We explicitly ignore the floating-CA case here: the `$out` of `input3`
depends on `$out` of `input2`. This means that there are actually two
derivations - I assume that this is because at eval time (i.e.
`nix-instantiate -A`) the hash of `input2` isn't known yet and the
other .drv is created as soon as `input2` was built. This is another
issue on its own, so we ignore the case here explicitly.
- Make sure that it starts even without the `nix-command` xp feature
- Fail if it doesn’t manage to start
This fixes a 30s wait for every test in `init.sh` as the daemon couldn’t
start, but the code was just waiting 30s and continuing as if everything
was all right.
Polling every 1 second means that even the simplest test takes at least
2 seconds. We can reasonably poll 1/10 of that to make things much
quicker (esp. given that most of the time 0.1s is enough for the
daemon to be started or stopped)
The tests are scheduled in the order they appear, so running the long
ones first slightly improves the scheduling.
On my machine, this decreases the time of `make install` from 40s to 36s
diff-index operates on the view that git has of the working tree,
which might be outdated. The higher-level diff command does this
automatically. This change also adds handling for submodules.
fixes#4140
Alternative fixes would be invoking update-index before diff-index or
matching more closely what require_clean_work_tree from git-sh-setup.sh
does, but both those options make it more difficult to reason about
correctness.
The .git/refs/heads directory might be empty for a valid
usable git repository. This often happens in CI environments,
which might only fetch commits, not branches.
Therefore instead we let git itself check if HEAD points to
something that looks like a commit.
fixes#5302
previously :a would override old bindings of a name with new values if the added
set contained names that were already bound. in nix 2.6 this doesn't happen any
more, which is potentially confusing.
fixes#6041
Bundlers are now responsible for correctly handling their inputs which
are no longer constrained to be (Drv->Drv)->Drv->Drv, but can be of
type (attrset->Drv)->attrset->Drv.
It’s totally valid to have entries in `NIX_PATH` that aren’t valid paths
(they can even be arbitrary urls or `channel:<channel-name>`).
Fix#5998 and #5980
if we defer the duplicate argument check for lambda formals we can use more
efficient data structures for the formals set, and we can get rid of the
duplication of formals names to boot. instead of a list of formals we've seen
and a set of names we'll keep a vector instead and run a sort+dupcheck step
before moving the parsed formals into a newly created lambda. this improves
performance on search and rebuild by ~1%, pure parsing gains more (about 4%).
this does reorder lambda arguments in the xml output, but the output is still
stable. this shouldn't be a problem since argument order is not semantically
important anyway.
before
nix search --no-eval-cache --offline ../nixpkgs hello
Time (mean ± σ): 8.550 s ± 0.060 s [User: 6.470 s, System: 1.664 s]
Range (min … max): 8.435 s … 8.666 s 20 runs
nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
Time (mean ± σ): 346.7 ms ± 2.1 ms [User: 312.4 ms, System: 34.2 ms]
Range (min … max): 343.8 ms … 353.4 ms 20 runs
nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
Time (mean ± σ): 2.720 s ± 0.031 s [User: 2.415 s, System: 0.231 s]
Range (min … max): 2.662 s … 2.780 s 20 runs
after
nix search --no-eval-cache --offline ../nixpkgs hello
Time (mean ± σ): 8.462 s ± 0.063 s [User: 6.398 s, System: 1.661 s]
Range (min … max): 8.339 s … 8.542 s 20 runs
nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
Time (mean ± σ): 329.1 ms ± 1.4 ms [User: 296.8 ms, System: 32.3 ms]
Range (min … max): 326.1 ms … 330.8 ms 20 runs
nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
Time (mean ± σ): 2.687 s ± 0.035 s [User: 2.392 s, System: 0.228 s]
Range (min … max): 2.626 s … 2.754 s 20 runs
This removes a dynamic stack allocation, making the derivation
unparsing logic robust against overflows when large strings are
added to a derivation.
Overflow behavior depends on the platform and stack configuration.
For instance, x86_64-linux/glibc behaves as (somewhat) expected:
$ (ulimit -s 20000; nix-instantiate tests/lang/eval-okay-big-derivation-attr.nix)
error: stack overflow (possible infinite recursion)
$ (ulimit -s 40000; nix-instantiate tests/lang/eval-okay-big-derivation-attr.nix)
error: expression does not evaluate to a derivation (or a set or list of those)
However, on aarch64-darwin:
$ nix-instantiate big-attr.nix ~
zsh: segmentation fault nix-instantiate big-attr.nix
This indicates a slight flaw in the single stack protection page
approach that is not encountered with normal stack frames.
Unless `--precise` is passed, make `nix why-depends` only show the
dependencies between the store paths, without introspecting them to
find the actual references.
This also makes it ~3x faster
There already existed a smoke test for the link content length,
but it appears that there exists some corruptions pernicious enough
to replace the file content with zeros, and keeping the same length.
--repair-path now goes as far as checking the content of the link,
making it true to its name and actually repairing the path for such
coruption cases.
nixpkgs can save a good bit of eval memory with this primop. zipAttrsWith is
used quite a bit around nixpkgs (eg in the form of recursiveUpdate), but the
most costly application for this primop is in the module system. it improves
the implementation of zipAttrsWith from nixpkgs by not checking an attribute
multiple times if it occurs more than once in the input list, allocates less
values and set elements, and just avoids many a temporary object in general.
nixpkgs has a more generic version of this operation, zipAttrsWithNames, but
this version is only used once so isn't suitable for being the base of a new
primop. if it were to be used more we should add a second primop instead.
When we check for disappeared overrides, we can get "false positives"
for follows and overrides which are defined in the dependencies of the
flake we are locking, since they are not parsed by
parseFlakeInputs. However, at that point we already know that the
overrides couldn't have possible been changed if the input itself
hasn't changed (since we check that oldLock->originalRef == *input.ref
for the input's parent). So, to prevent this, only perform this check
when it was possible that the flake changed (e.g. the flake we're
locking, or a new input, or the input has changed and mustRefetch ==
true).
When a variable is assigned in the REPL, make sure to remove any possible reference to the old one so that we correctly pick the new one afterwards
Fix#5706
Previously, when we were attempting to reuse the old lockfile
information in the computeLocks function, we have passed the parent of
the current input to the next computeLocks call. This was incorrect,
since the follows are resolved relative to the parent. This caused
issues when we tried to reuse oldLock but couldn't for some
reason (read: mustRefetch is true), in that case the follows were
resolved incorrectly.
Fix this by passing the correct parent, and adding some tests to
prevent this particular regression from happening again.
Closes https://github.com/NixOS/nix/issues/5697
If we’re in pure eval mode, then tell that in the error message rather
than (wrongly) speaking about restricted mode.
Fix https://github.com/NixOS/nix/issues/5611
When an input follows disappears, we can't just reuse the old lock
file entries since we may be missing some required ones. Refetch the
input when this happens.
Closes https://github.com/NixOS/nix/issues/5289
When setting flake-local options (with the `nixConfig` field), forward
these options to the daemon in case we’re using one.
This is necessary in particular for options like `binary-caches` or
`post-build-hook` to make sense.
Fix <343239fc8a (r44356843)>
Having the `post-build-hook` use `nix` from the client package can lead
to a deadlock in case there’s a db migration to do between both, as a
`nix` command running inside the hook will run as root (and as such will
bypass the daemon), so might trigger a db migration, which will get
stuck trying to get a global lock on the DB (as the daemon that ran the
hook already has a lock on it).
When running a `:b` command in the repl, after building the derivations
query the store for its outputs rather than just assuming that they are
known in the derivation itself (which isn’t true for CA derivations)
Fix#5328
Rather than having them plain strings scattered through the whole
codebase, create an enum containing all the known experimental features.
This means that
- Nix can now `warn` when an unkwown experimental feature is passed
(making it much nicer to spot typos and spot deprecated features)
- It’s now easy to remove a feature altogether (once the feature isn’t
experimental anymore or is dropped) by just removing the field for the
enum and letting the compiler point us to all the now invalid usages
of it.
The min bound written corresponds to the date of the commit that
introduced the change, but it only got merged on master some weeks
later. Since the version is essentially the commit date, that means that
there’s a whole range of commits on master (including the current
`nixUnstable`) that have a higher version but don’t contain the required
change.
This fixes a bug in the garbage collector where if a path
/nix/store/abcd-foo is valid, but we do a
isValidPath("/nix/store/abcd-foo.lock") first, then a negative entry
for /nix/store/abcd is added to pathInfoCache, so /nix/store/abcd-foo
is subsequently considered invalid and deleted.
The garbage collector no longer blocks other processes from
adding/building store paths or adding GC roots. To prevent the
collector from deleting store paths just added by another process,
processes need to connect to the garbage collector via a Unix domain
socket to register new temporary roots.
We now build the context (so this has the side-effect of making
builtins.{path,filterSource} work on derivations outputs, if IFD is
enabled) and then check that the path has no references (which is what
we really care about).
This actually bit me quite recently in `nixpkgs` because I assumed that
`nix-build --check` would also error out if hashes don't match anymore[1]
and so I wrongly assumed that I couldn't reproduce the mismatch error.
The fix is rather simple, during the output registration a so-called
`delayedException` is instantiated e.g. if a FOD hash-mismatch occurs.
However, in case of `nix-build --check` (or `--rebuild` in case of `nix
build`), the code-path where this exception is thrown will never be
reached.
By adding that check to the if-clause that causes an early exit in case
of `bmCheck`, the issue is gone. Also added a (previously failing)
test-case to demonstrate the problem.
[1] https://github.com/NixOS/nixpkgs/pull/139238, the underlying issue
was that `nix-prefetch-git` returns different hashes than `fetchgit`
because the latter one fetches submodules by default.
If the store path contains a flake, this means that a command like
"nix path-info /path" will show info about /path, not about the
default output of the flake in /path. If you want the latter, you can
explicitly ask for it by doing "nix path-info path:/path".
Fixes#4568.
When doing e.g.
nix-build -A package --keep-failed --option \
builders \
'ssh://mfhydra?remote-store=/home/bosch/store x86_64-linux - 10 4 big-parallel'
this doesn't work properly because this build-setting is ignored.
I changed this behavior by passing the `settings.keepFailed` through the
serve-protocol to remote machines to make sure that I can introspect the
build-directory (which is particularly helpful when I have to look at a
`config.log` from a failed build for instance).
When `NIX_DAEMON_PACKAGE` is set, make all the tests use the Nix daemon.
That way we can test every piece of Nix functionality both with and
without the daemon.
Tests for which using the daemon isn’t possible or doesn’t make sens can
selectively be disabled with `needLocalStore`
Some people want to avoid using registries at all on their system; Instead
of having to add --no-registries to every command, this commit allows to
set use-registries = false in the config. --no-registries is still allowed
everywhere it was allowed previously, but is now deprecated.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Fill `NIX_CONFIG` with the value of the current Nix configuration before
calling the nix subprocesses in the repl
That way the whole configuration (including the possible
`experimental-features`, a possibly `--store` option or whatever) will
be made available.
This is required for example to make `nix repl` work with a custom
`--store`
We need to support it for the “old” fetch* functions for backwards
compatibility, but we don’t need it for fetchTree (as it’s a new
function).
Given that changing the `name` messes-up the content hashing, we can
just forbid passing a custom `name` argument to it
Fixes this random failure:
error: hash mismatch in fixed-output derivation '/tmp/nix-shell.EUgAVU/nix-test/tests/check/store/sfps3l3c5n7dabpx34kigxnfhmrwk2h6-dummy.drv':
specified: sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=
got: sha256-0qhPS4tlCTfsj3PNi+LHSt1akRumTfJ0WO2CKdqASiY=
which happens because multiple tests were writing to ./dummy.
Add an access-control list to the realisations in recursive-nix (similar
to the already existing one for store paths), so that we can build
content-addressed derivations in the restricted store.
Fix#4353
This way no derivation has to expect that these files are in the `cwd`
during the build. This is problematic for `nix-shell` where these files
would have to be inserted into the nix-shell's `cwd` which can become
problematic with e.g. recursive `nix-shell`.
To remain backwards-compatible, the location inside the build sandbox
will be kept, however using these files directly should be deprecated
from now on.
This is needed to push the adoption of structured attrs[1] forward. It's
now checked if a `__json` exists in the environment-map of the derivation
to be openend in a `nix-shell`.
Derivations with structured attributes enabled also make use of a file
named `.attrs.json` containing every environment variable represented as
JSON which is useful for e.g. `exportReferencesGraph`[2]. To
provide an environment similar to the build sandbox, `nix-shell` now
adds a `.attrs.json` to `cwd` (which is mostly equal to the one in the
build sandbox) and removes it using an exit hook when closing the shell.
To avoid leaking internals of the build-process to the `nix-shell`, the
entire logic to generate JSON and shell code for structured attrs was
moved into the `ParsedDerivation` class.
[1] https://nixos.mayflower.consulting/blog/2020/01/20/structured-attrs/
[2] https://nixos.org/manual/nix/unstable/expressions/advanced-attributes.html#advanced-attributes
Resolve the derivation before trying to load its environment −
essentially reproducing what the build loop does − so that we can
effectively access our dependencies (and not just their placeholders).
Fix#4821
Make ca-derivations require a `ca-derivations` machine feature, and
ca-aware builders expose it.
That way, a network of builders can mix ca-aware and non-ca-aware
machines, and the scheduler will send them in the right place.
When the `keep-going` option is set to `true`, make `nix flake check`
continue as much as it can before failing.
The UI isn’t perfect as-it-is as all the lines currently start with a
mostly useless `error (ignored): error:` prefix, but I’m not sure what
the best output would be, so I’ll leave it as-it-is for the time being
(This is a bit hijacking the `keep-going` flag as it’s supposed to be a
build-time only thing. But I think it’s faire to reuse it here).
Fix https://github.com/NixOS/nix/issues/4450
When adding a path to the local store (via `LocalStore::addToStore`),
ensure that the `ca` field of the provided `ValidPathInfo` does indeed
correspond to the content of the path.
Otherwise any untrusted user (or any binary cache) can add arbitrary
content-addressed paths to the store (as content-addressed paths don’t
need a signature).
Similar to the nar-info disk cache (and using the same db).
This makes rebuilds muuch faster.
- This works regardless of the ca-derivations experimental feature.
I could modify the logic to not touch the db if the flag isn’t there,
but given that this is a trash-able local cache, it doesn’t seem to be
really worth it.
- We could unify the `NARs` and `Realisation` tables to only have one
generic kv table. This is left as an exercise to the reader.
- I didn’t update the cache db version number as the new schema just
adds a new table to the previous one, so the db will be transparently
migrated and is backwards-compatible.
Fix#4746
First, "XDG_CONFIG_HOME" shouldn't be named "home", as it may be
confusing compared with `$HOME`, which an upcoming test will be using.
Then, using a fixed location for the test is problematic. Use
`$TEST_ROOT` instead.
This requires adding `nix` to its own closure which is a bit unfortunate,
but as it is optional (the test will be disabled if `OUTER_NIX` is unset) it
shouldn't be too much of an issue.
(Ideally this should go in another derivation so that we can build Nix and run
the test independently, but as the tests are running in the same derivation
as the build it's a bit complicated to do so).
Requires a slight update to the test infra to work properly, but
having the possibility to group tests that way makes the whole thing
quite cleaner imho
This was
- Added in dbf96e10ec.
- Commented out in 07975979aa, which I
believe only reached master by mistake.
- Deleted in c32168c9bc, when
`tests/build-hook-ca.nix` was reused for a new test.
But the test works, and we ought to have it.
This is probably what most people expect it to do. Fixes#3781.
There is a new command 'nix flake lock' that has the old behaviour of
'nix flake update', i.e. it just adds missing lock file entries unless
overriden using --update-input.
This is technically a breaking change, since attempting to set plugin
files after the first non-flag argument will now throw an error. This
is acceptable given the relative lack of stability in a plugin
interface and the need to tie the knot somewhere once plugins can
actually define new subcommands.
When performing distributed builds of machine learning packages, it
would be nice if builders without the required SIMD instructions can
be excluded as build nodes.
Since x86_64 has accumulated a large number of different instruction
set extensions, listing all possible extensions would be unwieldy.
AMD, Intel, Red Hat, and SUSE have recently defined four different
microarchitecture levels that are now part of the x86-64 psABI
supplement and will be used in glibc 2.33:
https://gitlab.com/x86-psABIs/x86-64-ABIhttps://lwn.net/Articles/844831/
This change uses libcpuid to detect CPU features and then uses them to
add the supported x86_64 levels to the additional system types. For
example on a Ryzen 3700X:
$ ~/aps/bin/nix -vv --version | grep "Additional system"
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
It's now
at /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix:7:7:
instead of
at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix
The new format is more standard and clickable.
This change is to simplify [Trustix](https://github.com/tweag/trustix) indexing and makes it possible to reconstruct this URL regardless of the compression used.
In particular this means that 7c2e9ca597/contrib/nix/nar/nar.go (L61-L71) can be removed and only the bits that are required to establish trust needs to be published in the Trustix build logs.
For each known realisation, store:
- its output
- its output path
This comes with a set of needed changes:
- New `realisations` module declaring the types needed for describing
these mappings
- New `Store::registerDrvOutput` method registering all the needed informations
about a derivation output (also replaces `LocalStore::linkDeriverToPath`)
- new `Store::queryRealisation` method to retrieve the informations for a
derivations
This introcudes some redundancy on the remote-store side between
`wopQueryDerivationOutputMap` and `wopQueryRealisation`.
However we might need to keep both (regardless of backwards compat)
because we sometimes need to get some infos for all the outputs of a
derivation (where `wopQueryDerivationOutputMap` is handy), but all the
stores can't implement it − because listing all the outputs of a
derivation isn't really possible for binary caches where the server
doesn't allow to list a directory.
Otherwise https://cache.nixos.org is chosen by default, causing the OSX
testsuite to hang inside the sandbox.
(In a way, this is probably rugging an actual bug under the carpet as
Nix should be able to gracefully timeout in such a case, but that's
beyond mac OSX-fu)
Without setting HGPLAIN, the user's environment leaks into
hg invocations, which means that the output may not be in the
expected format.
HGPLAIN is the Mercurial-recommended solution for this in that
it's intended for uses by scripts and programs which are looking
to parse Mercurial's output in a consistent manner.
Although the non-resolved derivation will never get a cache-hit (it
doesn't have an output path to query the cache for anyways), we might
get one on the resolved derivation.
Having vm-test-run-unnamed for all the test derivation doesn't look very
nice, so in order to better distinguish them from their store path,
let's actually give them proper names.
Signed-off-by: aszlig <aszlig@nix.build>
Perl-based tests are deprecated since NixOS 20.03 and subsequently got
removed in NixOS 20.09, which effectively means that tests are going to
fail as soon as we build it with NixOS 20.09 or anything newer.
I've put "# fmt: off" at the start of every testScript, because
formatting with Black really messes up indentation and I don't think it
really adds anything in value or readability for inlined Python scripts.
Signed-off-by: aszlig <aszlig@nix.build>
In particular, this means that derivations can output derivations. But
that ramification isn't (yet!) useful as we would want, since there is
no way to have a dependent derivation that is itself a dependent
derivation.
tar(1) on FreeBSD does not use standard output or input when the -f flag
is not provided. Instead, it defaults to /dev/sa0 on FreeBSD.
Make this tar invocation a bit more robust and explicitly tell tar(1) to
use standard output.
This is one of the issues discovered while porting Nix to FreeBSD. It has
been tested and committed locally to FreeBSD ports:
https://svnweb.freebsd.org/ports/head/sysutils/nix/Makefile?revision=550026&view=markup#l108
Otherwise the result of the printing can't be parsed back correctly by
Nix (because the unescaped `${` will be parsed as the begining of an
anti-quotation).
Fix#3989
This seems more correct. It also means one can specify the features a
store should support with --store and remote-store=..., which is useful.
I use this to clean up the build remotes test.
Before, processConnection wanted to know a user name and user id, and
`nix-daemon --stdio`, when it isn't proxying to an underlying daemon,
would just assume "root" and 0. But `nix-daemon --stdio` (no proxying)
shouldn't make guesses about who holds the other end of its standard
streams.
Now processConnection takes an "auth hook", so `nix-daemon` can provide
the appropriate policy and daemon.cc doesn't need to know or care what
it is.
If a repo is dirty, it used to return a `rev` object with an "empty"
sha1 (0000000000000000000000000000000000000000). Please note that this
only applies for `builtins.fetchGit` and *not* for `builtins.fetchTree{
type = "git"; }`.
The original idea was to implement a git-fetcher in Nix's core that
supports content hashes[1]. In #3549[2] it has been suggested to
actually use `fetchTree` for this since it's a fairly generic wrapper
over the new fetcher-API[3] and already supports content-hashes.
This patch implements a new git-fetcher based on `fetchTree` by
incorporating the following changes:
* Removed the original `fetchGit`-implementation and replaced it with an
alias on the `fetchTree` implementation.
* Ensured that the `git`-fetcher from `libfetchers` always computes a
content-hash and returns an "empty" revision on dirty trees (the
latter one is needed to retain backwards-compatibility).
* The hash-mismatch error in the fetcher-API exits with code 102 as it
usually happens whenever a hash-mismatch is detected by Nix.
* Removed the `flakes`-feature-flag: I didn't see a reason why this API
is so tightly coupled to the flakes-API and at least `fetchGit` should
remain usable without any feature-flags.
* It's only possible to specify a `narHash` for a `git`-tree if either a
`ref` or a `rev` is given[4].
* It's now possible to specify an URL without a protocol. If it's missing,
`file://` is automatically added as it was the case in the original
`fetchGit`-implementation.
[1] https://github.com/NixOS/nix/pull/3216
[2] https://github.com/NixOS/nix/pull/3549#issuecomment-625194383
[3] https://github.com/NixOS/nix/pull/3459
[4] https://github.com/NixOS/nix/pull/3216#issuecomment-553956703
Originally, the test was only checking for different “real” storeDir.
That’s an easy case to handle, but the much harder one is if different
virtual store dirs are used. To do this, we need the SubstitutionGoal
to know about the ca, so it can recalculate the path to copy it over.
An important note here is that the store path passed to copyStorePath
needs to be one for srcStore - so that queryPathInfo works properly.
This also adds an error message when the store path from queryPathInfo
is different from the one we requested.
We can’t use custom name here because different names will have
different store paths. This is a limitation of the Store API’s
reliance on store paths.
We might be able to get around the above in the future by using a
dummy name for certain fixed output paths.
This fixes an issue where lockfile generation was not idempotent:
after updating a lockfile, a "follows" node would end up pointing to a
new copy of the node, rather than to the original node.
The `remote-store` test loads the `user-env` one to test nix-env when
using the daemon, but actually does it incorrectly because every test
starts (in `common.sh`) by resetting the value of `NIX_REMOTE`, meaning
that the `user-env` test will never use the daemon.
Fix this by setting `NIX_REMOTE_` before sourcing `user-env.sh` in the
`remote-store` test, so that `NIX_REMOTE` is correctly set inside the
test
The initial contents of the flake is specified by the
'templates.<name>' or 'defaultTemplate' output of another flake. E.g.
outputs = { self }: {
templates = {
nixos-container = {
path = ./nixos-container;
description = "An example of a NixOS container";
};
};
};
allows
$ nix flake init -t templates#nixos-container
Also add a command 'nix flake new', which is identical to 'nix flake
init' except that it initializes a specified directory rather than the
current directory.
The previous regex was too strict and did not match what git was allowing. It
could lead to `fetchGit` not accepting valid branch names, even though they
exist in a repository (for example, branch names containing `/`, which are
pretty standard, like `release/1.0` branches).
The new regex defines what a branch name should **NOT** contain. It takes the
definitions from `refs.c` in https://github.com/git/git and `git help
check-ref-format` pages.
This change also introduces a test for ref name validity checking, which
compares the result from Nix with the result of `git check-ref-format --branch`.
This makes 'nix flake' less cluttered and more consistent (it's only
subcommands that operator on a flake). Also, the registry is not
inherently flake-related (e.g. fetchTree could also use it to remap
inputs).
Motivation: maintain project-level configuration files.
Document the whole situation a bit better so that it corresponds to the
implementation, and add NIX_USER_CONF_FILES that allows overriding
which user files Nix will load during startup.
A test case for correct handling of temporary directory deletion that
was added to check.sh as part of PR #2689 was initially disabled for
Darwin because of a directory permission issue in PR #2688.
Now that the issue in PR #2688 is fixed, this commit enables the test
case for Darwin.
Temporarily add user-write permission to build directory so that it
can be moved out of the sandbox to the store with a .check suffix.
This is necessary because the build directory has already had its
permissions set read-only, but write permission is required
to update the directory's parent link to move it out of the sandbox.
Updated the related --check "derivation may not be deterministic"
messages to consistently use the real store paths.
Added test for non-root sandbox nix-build --check -K to demonstrate
issue and help prevent regressions.
Future editions of flakes or the Nix language can be supported by
renaming flake.nix (e.g. flake-v2.nix). This avoids a bootstrap
problem where we don't know which grammar to use to parse
flake*.nix. It also allows a project to support multiple flake
editions, in theory.
With --check and the --keep-failed (-K) flag, the temporary directory
was being retained regardless of whether the build was successful and
reproducible. This removes the temporary directory, as expected, on
a reproducible check build.
Added tests to verify that temporary build directories are not
retained unnecessarily, particularly when using --check with
--keep-failed.
This provides a pluggable mechanism for defining new fetchers. It adds
a builtin function 'fetchTree' that generalizes existing fetchers like
'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a
set of attributes, e.g.
fetchTree {
type = "git";
url = "https://example.org/repo.git";
ref = "some-branch";
rev = "abcdef...";
}
The existing fetchers are just wrappers around this. Note that the
input attributes to fetchTree are the same as flake input
specifications and flake lock file entries.
All fetchers share a common cache stored in
~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching
mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}).
This also adds support for Git worktrees (c169ea5904).
This allows querying the location of function arguments. E.g.
builtins.unsafeGetAttrPos "x" (builtins.functionArgs ({ x }: null))
=> { column = 57; file = "/home/infinisil/src/nix/inst/test.nix"; line = 1; }
This is now done in a single pass. Also fixes some issues when
updating flakes with circular dependencies. Finally, when using
'--recreate-lock-file --commit-lock-file', the commit message now
correctly shows the differences.
If you do a fetchTree on a Git repository, whether the result contains
a revCount attribute should not depend on whether that repository
happens to be a shallow clone or not. That would complicate caching a
lot and would be semantically messy. So applying fetchTree/fetchGit to
a shallow repository is now an error unless you pass the attribute
'shallow = true'. If 'shallow = true', we don't return revCount, even
if the repository is not actually shallow.
Note that Nix itself is not doing shallow clones at the moment. But it
could do so as an optimisation if the user specifies 'shallow = true'.
Issue #2988.
Worktrees[1] are a feature of git which allow you to check out a ref in
a different directory.
While playing around with flakes I realized that git repositories in a
worktree checkout break when trying to build a flake:
```
$ git worktree add ../nixpkgs-flakes nixpkgs-flakes
$ cd ../nixpkgs-flakes
$ nix build .#hello
error: opening directory '/home/ma27/Projects/nixpkgs-flakes/.git/refs/heads': Not a directory
```
This issue has been fixed by determining with `git rev-parse --git-common-dir`
where the actual `.git` directory is.
Please note that this issue only exists on the `flakes` branch, fetching
worktree checkouts with Nix master seems to work fine.
[1] https://git-scm.com/docs/git-worktree
This allows all supported fetchers to be used, e.g.
builtins.fetchTree {
type = "github";
owner = "NixOS";
repo = "nix";
rev = "d4df99a3349cf2228a8ee78dea320afef86eb3ba";
}
This improves reproducibility and may be faster than fetching from the
original source (especially for git/hg inputs, but probably also for
github inputs - our binary cache is probably faster than GitHub's
dynamically generated tarballs).
Unfortunately this doesn't work for the top-level flake since even if
we know the NAR hash of the tree, we don't know the other tree
attributes such as revCount and lastModified.
Fixes#3253.
Typical usage:
$ nix flake update ~/Misc/eelco-configurations/hagbard --update-input nixpkgs
to update the 'nixpkgs' input of a flake while leaving every other
input unchanged.
The argument is an input path, so you can do e.g. '--update-input
dwarffs/nixpkgs' to update an input of an input.
Fixes#2928.
Added a flag --no-update-lock-file to barf if the lock file needs any
changes. This is useful for CI systems if you're building a
checkout. Fixes#2947.
Renamed --no-save-lock-file to --no-write-lock-file. It is now a fatal
error if the lock file needs changes but --no-write-lock-file is not
given.
E.g.
$ nix flake update ~/Misc/eelco-configurations/hagbard \
--override-input 'dwarffs/nixpkgs' ../my-nixpkgs
overrides the 'nixpkgs' input of the 'dwarffs' input of the top-level
flake.
Fixes#2837.
When computing a lock file, we now respect the lock files of flake
inputs. This is important for usability / reproducibility. For
example, the 'nixops' flake depends on the 'nixops-aws' and
'nixops-hetzner' repositories. So when the 'nixops' flake is used in
another flake, we want the versions of 'nixops-aws' and
'nixops-hetzner' locked by the the 'nixops' flake because those
presumably have been tested.
This can lead to a proliferation of versions of flakes like 'nixpkgs'
(since every flake's lock file could depend on a different version of
'nixpkgs'). This is not a major issue when using Nixpkgs overlays or
NixOS modules, since then the top-level flake composes those
overlays/modules into *its* version of Nixpkgs and all other versions
are ignored. Lock file computation has been made a bit more lazy so it
won't try to fetch all those versions of 'nixpkgs'.
However, in case it's necessary to minimize flake versions, there now
are two input attributes that allow this. First, you can copy an input
from another flake, as follows:
inputs.nixpkgs.follows = "dwarffs/nixpkgs";
This states that the calling flake's 'nixpkgs' input shall be the same
as the 'nixpkgs' input of the 'dwarffs' input.
Second, you can override inputs of inputs:
inputs.nixpkgs.url = github:edolstra/nixpkgs/<hash>;
inputs.nixops.inputs.nixpkgs.url = github:edolstra/nixpkgs/<hash>;
or equivalently, using 'follows':
inputs.nixpkgs.url = github:edolstra/nixpkgs/<hash>;
inputs.nixops.inputs.nixpkgs.follows = "nixpkgs";
This states that the 'nixpkgs' input of the 'nixops' input shall be
the same as the calling flake's 'nixpkgs' input.
Finally, at '-v' Nix now prints the changes to the lock file, e.g.
$ nix flake update ~/Misc/eelco-configurations/hagbard
inputs of flake 'git+file:///home/eelco/Misc/eelco-configurations?subdir=hagbard' changed:
updated 'nixpkgs': 'github:edolstra/nixpkgs/7845bf5f4b3013df1cf036e9c9c3a55a30331db9' -> 'github:edolstra/nixpkgs/03f3def66a104a221aac8b751eeb7075374848fd'
removed 'nixops'
removed 'nixops/nixops-aws'
removed 'nixops/nixops-hetzner'
removed 'nixops/nixpkgs'
Fixes
error: derivation '/nix/store/klivma7r7h5lndb99f7xxmlh5whyayvg-zlib-1.2.11.drv' has incorrect output '/nix/store/fv98nnx5ykgbq8sqabilkgkbc4169q05-zlib-1.2.11-dev', should be '/nix/store/adm7pilzlj3z5k249s8b4wv3scprhzi1-zlib-1.2.11-dev'
As fromTOML supports \u and \U escapes, bring fromJSON on par. As JSON defaults
to UTF-8 encoding (every JSON parser must support UTF-8), this change parses the
`\u hex hex hex hex` sequence (\u followed by 4 hexadecimal digits) into an
UTF-8 representation.
Add a test to verify correct parsing, using all escape sequences from json.org.
Having a colon in the path may cause issues, and having the hash
function indicated isn't actually necessary. We now verify the path
format in the tests to prevent regressions.
This replaces the '(...)' installable syntax, which is not very
discoverable. The downside is that you can't have multiple expressions
or mix expressions and other installables.
Derivations that want to use recursion should now set
requiredSystemFeatures = [ "recursive-nix" ];
to make the daemon socket appear.
Also, Nix should be configured with "experimental-features =
recursive-nix".
This is an alternative to the IN_NIX_SHELL environment variable,
allowing the expression to adapt itself to nix-shell without
triggering those adaptations when used as a dependency of another
shell.
Closes#3147
Experimental features are now opt-in. There is currently one
experimental feature: "nix-command" (which enables the "nix"
command. This will allow us to merge experimental features more
quickly, without committing to supporting them indefinitely.
Typical usage:
$ nix build --experimental-features 'nix-command flakes' nixpkgs#hello
(cherry picked from commit 8e478c2341,
without the "flakes" feature)
Experimental features are now opt-in. There are currently two
experimental features: "nix-command" (which enables the "nix"
command), and "flakes" (which enables support for flakes). This will
allow us to merge experimental features more quickly, without
committing to supporting them indefinitely.
Typical usage:
$ nix build --experimental-features 'nix-command flakes' nixpkgs#hello
A command like
$ nix run nixpkgs#hello
will now build the attribute 'packages.${system}.hello' rather than
'packages.hello'. Note that this does mean that the flake needs to
export an attribute for every system type it supports, and you can't
build on unsupported systems. So 'packages' typically looks like this:
packages = nixpkgs.lib.genAttrs ["x86_64-linux" "i686-linux"] (system: {
hello = ...;
});
The 'checks', 'defaultPackage', 'devShell', 'apps' and 'defaultApp'
outputs similarly are now attrsets that map system types to
derivations/apps. 'nix flake check' checks that the derivations for
all platforms evaluate correctly, but only builds the derivations in
'checks.${system}'.
Fixes#2861. (That issue also talks about access to ~/.config/nixpkgs
and --arg, but I think it's reasonable to say that flakes shouldn't
support those.)
The alternative to attribute selection is to pass the system type as
an argument to the flake's 'outputs' function, e.g. 'outputs = { self,
nixpkgs, system }: ...'. However, that approach would be at odds with
hermetic evaluation and make it impossible to enumerate the packages
provided by a flake.
Instead of a list, inputs are now an attrset like
inputs = {
nixpkgs.uri = github:NixOS/nixpkgs;
};
If 'uri' is omitted, than the flake is a lookup in the flake registry, e.g.
inputs = {
nixpkgs = {};
};
but in that case, you can also just omit the input altogether and
specify it as an argument to the 'outputs' function, as in
outputs = { self, nixpkgs }: ...
This also gets rid of 'nonFlakeInputs', which are now just a special
kind of input that have a 'flake = false' attribute, e.g.
inputs = {
someRepo = {
uri = github:example/repo;
flake = false;
};
};
With this patch, and this file I called `log.py`:
#!/usr/bin/env nix-shell
#!nix-shell -i python3 -p python3 --pure
import sys
from pprint import pprint
stack = []
timestack = []
for line in open(sys.argv[1]):
components = line.strip().split(" ", 2)
if components[0] != "function-trace":
continue
direction = components[1]
components = components[2].rsplit(" ", 2)
loc = components[0]
_at = components[1]
time = int(components[2])
if direction == "entered":
stack.append(loc)
timestack.append(time)
elif direction == "exited":
dur = time - timestack.pop()
vst = ";".join(stack)
print(f"{vst} {dur}")
stack.pop()
and:
nix-instantiate --trace-function-calls -vvvv ../nixpkgs/pkgs/top-level/release.nix -A unstable > log.matthewbauer 2>&1
./log.py ./log.matthewbauer > log.matthewbauer.folded
flamegraph.pl --title matthewbauer-post-pr log.matthewbauer.folded > log.matthewbauer.folded.svg
I can make flame graphs like: http://gsc.io/log.matthewbauer.folded.svg
---
Includes test cases around function call failures and tryEval. Uses
RAII so the finish is always called at the end of the function.
This currently fails because we're using POSIX file locks. So when the
garbage collector opens and closes its own temproots file, it causes
the lock to be released and then deleted by another GC instance.
Passing `--post-build-hook /foo/bar` to a nix-* command will cause
`/foo/bar` to be executed after each build with the following
environment variables set:
DRV_PATH=/nix/store/drv-that-has-been-built.drv
OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev
This can be useful in particular to upload all the builded artifacts to
the cache (including the ones that don't appear in the runtime closure
of the final derivation or are built because of IFD).
This new feature prints the stderr/stdout output to the `nix-build`
and `nix build` client, and the output is printed in a Nix 2
compatible format:
[nix]$ ./inst/bin/nix-build ./test.nix
these derivations will be built:
/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv
building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'...
hello!
bye!
running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'...
post-build-hook: + sleep 1
post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: + sleep 1
post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: + sleep 1
post-build-hook: + printf 'very important stuff'
/nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
[nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix
my-example-derivation> hello!
my-example-derivation> bye!
my-example-derivation (post)> + sleep 1
my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> + sleep 1
my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> + sleep 1
my-example-derivation (post)> + printf 'very important stuff'
[1 built, 0.0 MiB DL]
Co-authored-by: Graham Christensen <graham@grahamc.com>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
