This is useful for diagnosing whether an evaluation is copying large
paths to the store. Example:
$ nix build .#packages.x86_64-linux.default --large-path-warning-threshold 1000000
warning: copied large path '/home/eelco/Dev/nix-master/' to the store (6271792 bytes)
warning: copied large path '«github:NixOS/nixpkgs/b550fe4b4776908ac2a861124307045f8e717c8e?narHash=sha256-7kkJQd4rZ%2BvFrzWu8sTRtta5D1kBG0LSRYAfhtmMlSo%3D»/' to the store (155263768 bytes)
warning: copied large path '«github:libgit2/libgit2/45fd9ed7ae1a9b74b957ef4f337bc3c8b3df01b5?narHash=sha256-oX4Z3S9WtJlwvj0uH9HlYcWv%2Bx1hqp8mhXl7HsLu2f0%3D»/' to the store (22175416 bytes)
warning: copied large path '/nix/store/z985088mcd6w23qwdlirsinnyzayagki-source' to the store (5885872 bytes)
This fixes https://github.com/NixOS/nixpkgs/pull/290775 by not expanding aliases
when sourcing the stdenv setup script. The way bash handles aliases is to expand
them when a function is defined, not when it is used. I.e.:
$ alias echo="echo bar "
$ echo foo
bar foo
$ xyzzy() { echo foo; }
$ shopt -u expand_aliases
$ xyzzy
bar foo
$ xyzzy2() { echo foo; }
$ xyzzy2
foo
The problem is that ~/.bashrc is sourced before the stdenv setup, and bashrc
commonly sets aliases for ‘cp’, ‘mv’ and ‘rm’ which you don’t want to take
effect in the stdenv derivation builders. The original commit introducing this
feature (5fd8cf7667) even mentioned this very
alias.
The only way to avoid this is to disable aliases entirely while sourcing the
stdenv setup, and reenable them afterwards.
Passing the commit message as an argument causes update failures on repositories with lots of flake inputs. In some cases, the commit message is over 250,000 bytes.
the old `copyFile` was just a wrapper that was calling the `copy`
function. This wrapper function is removed and the `copy` function is
renamed to `copyFile`.
Building derivations is a lot harder, but the downloading goals is
portable enough.
The "common channel" code is due to Volth. I wonder if there is a way we
can factor it out into separate functions / files to avoid some
within-function CPP.
Co-authored-by: volth <volth@volth.com>
In streaming mode, libarchive doesn't handle symlinks in zip files
correctly. So write the entire file to disk so libarchive can access
it in random-access mode.
Fixes#10649. This was broken in cabee98152.
* Document string context
Now what we have enough primops, we can document how string contexts
work.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Felix Uhl <iFreilicht@users.noreply.github.com>
builtins.strictDerivation returns an attribute set with drvPath and
output paths. For some reason, current implementation forbids drv
instead of drvPath.
Sometimes we read a directory with children we cannot stat. It's a pitty
we even try to stat at all (wasteful) in the `DT_UNKNOWN` case, but at
least this should get rid of the failure.
This was changed in #10611, which caused the derivation paths of
anything using builtin:fetchurl to change (i.e. all of
Nixpkgs). However, impureEnvVars doesn't actually do anything for
builtin:fetchurl, so we can just set it to its historical value.
This makes for shorter and more portable code.
The only tricky part is catching exceptions: I just searched for near by
`catch (Error &)` or `catch (SysError &)` and adjusted them to `catch
(std::filesystem::filesystem_error &)` according to my human judgement.
Good for windows portability; will help @siddhantk232 with his GSOC
project.
Different parts of the project honor different sets of proxy environment
variables. With this commit all parts of the project will honor the same
set of proxy environment variables.
---------
Co-authored-by: Your Name <you@example.com>
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
Now that SourcePath uses a SourceAccessor instead of an InputAccessor,
we can use it in function signatures instead of passing a
SourceAccessor and CanonPath separately.
Make sure that `extraSandboxProfile` is set before we check whether it's
empty or not (in the `sandbox=true` case).
Also adds a test case for this.
Co-Authored-By: Artemis Tosini <lix@artem.ist>
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
After the removal of the InputAccessor::fetchToStore() method, the
only remaining functionality in InputAccessor was `fingerprint` and
`getLastModified()`, and there is no reason to keep those in a
separate class.
Fix formatting violations, update blacklist to reflect moved files.
PR #10556 passed CI before the new formating rules were added, and our
CI has the race condition of allowing old results, resulting in master
getting broken.
This missing GC root wasn't much of a problem before, because the
heap would end up with a reference to the `baseEnv` pretty soon,
but when unit testing, the construction of `EvalState` doesn't
necessarily happen well before GC runs for the first time.
Found while unit testing the Rust bindings that currently reside
at https://github.com/nixops4/nixops4/tree/main/rust
Add a method to check if a value has been initialized. This helps avoid
segfaults when calling `type()`.
Useful in the context of the new C API.
Closes#10524
Windows now has some basic Unix Domain Socket support, see
https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/
Building `nix daemon` on Windows I've left for later, because the daemon
currently forks per connection but this is not an option on Windows. But
we can get the client part working right away.
With Linux kernel >=6.6 & glibc 2.39 a `fchmodat2(2)` is available that
isn't filtered away by the libseccomp sandbox.
Being able to use this to bypass that restriction has surprising results
for some builds such as lxc[1]:
> With kernel ≥6.6 and glibc 2.39, lxc's install phase uses fchmodat2,
> which slips through 9b88e52846/src/libstore/build/local-derivation-goal.cc (L1650-L1663).
> The fixupPhase then uses fchmodat, which fails.
> With older kernel or glibc, setting the suid bit fails in the
> install phase, which is not treated as fatal, and then the
> fixup phase does not try to set it again.
Please note that there are still ways to bypass this sandbox[2] and this is
mostly a fix for the breaking builds.
This change works by creating a syscall filter for the `fchmodat2`
syscall (number 452 on most systems). The problem is that glibc 2.39
and seccomp 2.5.5 are needed to have the correct syscall number available
via `__NR_fchmodat2` / `__SNR_fchmodat2`, but this flake is still on
nixpkgs 23.11. To have this change everywhere and not dependent on the
glibc this package is built against, I added a header
"fchmodat2-compat.hh" that sets the syscall number based on the
architecture. On most platforms its 452 according to glibc with a few
exceptions:
$ rg --pcre2 'define __NR_fchmodat2 (?!452)'
sysdeps/unix/sysv/linux/x86_64/x32/arch-syscall.h
58:#define __NR_fchmodat2 1073742276
sysdeps/unix/sysv/linux/mips/mips64/n32/arch-syscall.h
67:#define __NR_fchmodat2 6452
sysdeps/unix/sysv/linux/mips/mips64/n64/arch-syscall.h
62:#define __NR_fchmodat2 5452
sysdeps/unix/sysv/linux/mips/mips32/arch-syscall.h
70:#define __NR_fchmodat2 4452
sysdeps/unix/sysv/linux/alpha/arch-syscall.h
59:#define __NR_fchmodat2 562
I tested the change by adding the diff below as patch to
`pkgs/tools/package-management/nix/common.nix` & then built a VM from
the following config using my dirty nixpkgs master:
{
vm = { pkgs, ... }: {
virtualisation.writableStore = true;
virtualisation.memorySize = 8192;
virtualisation.diskSize = 12 * 1024;
nix.package = pkgs.nixVersions.nix_2_21;
};
}
The original issue can be triggered via
nix build -L github:nixos/nixpkgs/d6dc19adbda4fd92fe9a332327a8113eaa843894#lxc \
--extra-experimental-features 'nix-command flakes'
however the problem disappears with this patch applied.
Closes#10424
[1] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2031073804
[2] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2030844251
With Nix 2.3, it was possible to pass a subpath of a store path to
exportReferencesGraph:
with import <nixpkgs> {};
let
hello = writeShellScriptBin "hello" ''
echo ${toString builtins.currentTime}
'';
in
writeClosure [ "${hello}/bin/hello" ]
This regressed with Nix 2.4, with a very confusing error message, that
presumably indicates it was unintentional:
error: path '/nix/store/3gl7kgjr4pwf03f0x70dgx9ln3bhl7zc-hello/bin/hello' is not in the Nix store
At this point many features are stripped out, but this works:
- Can run libnix{util,store,expr} unit tests
- Can run some Nix commands
Co-Authored-By volth <volth@volth.com>
Co-Authored-By Brian McKenna <brian@brianmckenna.org>
This also reworks the Mercurial fetcher (which was still using the
old cache interface) to have two distinct cache mappings:
* A ref-to-rev mapping, which is store-independent.
* A rev-to-store-path mapping.
Code operating on store objects (including creating them) should, in
general, use `ContentAddressMethod` rather than `FileIngestionMethod`.
See also dfc876531f which included some
similar refactors.
See https://github.com/NixOS/nix/pull/8699#discussion_r1554312181
Casting a function pointer to `void*` is undefined behavior in the C
spec, since there are platforms with different sizes for these two kinds
of pointers. A safe alternative might be `void (*callback)()`
* add intermediate variables and clarifying comments
Co-authored-by: Alexander Groleau <alex@proof.construction>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
https://github.com/NixOS/nix/pull/10456 fixed the addition of symlink
store paths to the sandbox, but also made it so that the hardcoded
sandbox paths (like `/etc/hosts`) were now bind-mounted without
following the possible symlinks. This made these files unreadable if
there were symlinks (because the sandbox would now contain a symlink to
an unreachable file rather than the underlying file).
In particular, this broke FOD derivations on NixOS as `/etc/hosts` is a
symlink there.
Fix that by canonicalizing all these hardcoded sandbox paths before
adding them to the sandbox.
Like always declining; local builds only, as can be inferred from the
docs. (Not worth spending too many words on this pretty obvious
behavior, I think. Also, plans to remove it? https://github.com/NixOS/nix/issues/1221)
This requires moving resolveSymlinks() into SourceAccessor. Also, it
requires LocalStoreAccessor::maybeLstat() to work on parents of the
store (to avoid an error like "/nix is not in the store").
Fixes#10375.
Instead of relying on setup script to set output variables when
structured attributes are enabled, iterate over the values of an
outputs associative array.
See also
374fa3532e/pkgs/stdenv/generic/setup.sh (L23-L26)
Bind-mounting symlinks is apparently not possible, which is why the
thing was failing.
Fortunately, symlinks are small, so we can fallback to copy them at no cost.
Fix https://github.com/NixOS/nix/issues/9579
Co-authored-by: Artturin <Artturin@artturin.com>
Now that we have a few things identifying content address methods by
name, we should be consistent about it.
Move up the `parseHashAlgoOpt` for tidiness too.
Discussed this change for consistency's sake as part of #8876
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
This was used in only one place, namely builtins.fetchurl with an
expected hash. Since this can cause similar issues as described
in #9814 and #9905 with the "locked" flag for fetchTarball and fetchTree,
let's just remove it.
Note that if an expected hash is given and the hash algorithm is
SHA-256, then we will never do a download anyway if the resulting
store path already exists. So removing the "locked" flag will only
cause potentially unnecessary HTTP requests (subject to the tarball
TTL) for non-SHA-256 hashes.
This probably snuck in in a refactor using truthiness or so. The
trustedness flag was having the optional fullness checked, rather than
the actual contained trust level.
Also adds some tests.
```
m1@6876551b-255d-4cb0-af02-8a4f17b27e2e ~ % nix store ping
warning: 'nix store ping' is a deprecated alias for 'nix store info'
Store URL: daemon
Version: 2.20.4
Trusted: 0
m1@6876551b-255d-4cb0-af02-8a4f17b27e2e ~ % nix doctor
warning: 'doctor' is a deprecated alias for 'config check'
[PASS] PATH contains only one nix version.
[PASS] All profiles are gcroots.
[PASS] Client protocol matches store protocol.
[INFO] You are trusted by store uri: daemon
```
code blocks, if not surrounded by empty lines, have the language
tags (in these cases, always `nix`) show up in the output of :doc.
for example:
nix-repl> :doc builtins.parseFlakeRef
Synopsis: builtins.parseFlakeRef flake-ref
Parse a flake reference, and return its exploded form.
For example: nix builtins.parseFlakeRef
"github:NixOS/nixpkgs/23.05?dir=lib" evaluates to: nix { dir =
"lib"; owner = "NixOS"; ref = "23.05"; repo = "nixpkgs"; type =
"github"; }
is now instead:
nix-repl> :doc builtins.parseFlakeRef
Synopsis: builtins.parseFlakeRef flake-ref
Parse a flake reference, and return its exploded form.
For example:
| builtins.parseFlakeRef "github:NixOS/nixpkgs/23.05?dir=lib"
evaluates to:
| { dir = "lib"; owner = "NixOS"; ref = "23.05"; repo = "nixpkgs"; type = "github"; }
When querying all paths in a binary cache store, the path's representation
is `<hash>-x` (where `x` is the value of `MissingName`) because the .narinfo
filenames only contain the hash.
Before cc46ea1630 this worked correctly,
because the entire path info was read and the path from this
representation was printed, i.e. in the form `<hash>-<name>`. Since then
however, the direct result from `queryAllValidPaths()` was used as `path`.
Added a regression test to make sure the behavior remains correct.
This was part of approved PR #10021. Unfortunately that one is stalled
on a peculiar Linux test timeout, so trying to get bits of it merged
first to bisect failure.
Forcing a conditional include, vs making the headers content
conditional, I think is more maintainable.
It is also how the other platform-specific headers (like
`namespaces.hh`) have been adapted.
A possible use of them might have been to figure out the paths
(which can now be retrieved with maybePathsOut), but I have
not found evidence that it was used this way, and it would have
been broken, because non-CA outputs weren't recorded in the map.
It's a little weird we don't check the return status for these, but
changing that would introduce risk so I did not.
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
This introduces new utility functions to get elements from JSON — in an ergonomic way and with nice error messages if the expected type does not match.
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
This splits files and adds new identifiers in preperation for supporting
windows, but no Windows-specific code is actually added yet.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
This patch makes `makeDecompressionSink` strip only a single layer
of compression specified via method. This fixes erroneous decompression
of doubly-compressed NARs fetched with curl.
This function is nice for more than `PosixSourceAccessor`. We can make a
few things simpler with it.
Note that the error logic slightly changes in some of the call sites, in
that we also count `ENOTDIR` and not just `ENOENT` as not having the
file, but that should be fine.
HintFmt(string) invokes the HintFmt("%s", literal) constructor,
which is not what we want here. Add a constructor with a proper name
and call that.
Next step: rename all the other ones to HintFmt::literal(string).
Fixes https://github.com/NixOS/nix/issues/10238
Thunks are now overwritten by a helper function
`Value::finishValue(newType, payload)` (where `payload` is the
original anonymous union inside `Value`). This helps to ensure we
never update a value elsewhere, since that would be incompatible with
parallel evaluation (i.e. after a value has transitioned from being a
thunk to being a non-thunk, it should be immutable).
There were two places where this happened: `Value::mkString()` and
`ExprAttrs::eval()`.
This PR also adds a bunch of accessor functions for value contents,
like `Value::integer()` to access the integer field in the union.
Fixes an instance of
nix: src/libutil/util.cc:139: nix::Path nix::canonPath(PathView, bool): Assertion `path != ""' failed.
... which I've been getting in one of my shells for some reason.
I have yet to find out why TMPDIR was empty, but it's no reason for
Nix to break.
Fixes
terminate called after throwing an instance of 'boost::wrapexcept<boost::io::too_few_args>'
what(): boost::too_few_args: format-string referred to more arguments than were passed
Aborted (core dumped)
for type errors in AttrCursor.
In a daemon-based Nix setup, some options cannot be overridden by a
client unless the client's user is considered trusted.
Currently, if an untrusted user tries to override one of those
options, we are silently ignoring it.
This can be pretty confusing in certain situations.
e.g. a user thinks he disabled the sandbox when in reality he did not.
We are now sending a warning message letting know the user some options
have been ignored.
Related to #1761.
This is a cherry-pick of 9e0f5f803f.
The above commit has been reverted by
a59e77d9e5 to prevent spamming warnings
with experimental features, but these are now totally ignored on the
daemon side, so there's no reason for the revert any more.
Previously, `state.mkList()` would set the type of the value to tList
and allocate the list vector, but it would not initialize the values
in the list. This has two problems:
* If an exception occurs, the list is left in an undefined state.
* More importantly, for multithreaded evaluation, if a value
transitions from thunk to non-thunk, it should be final (i.e. other
threads should be able to access the value safely).
To address this, there now is a `ListBuilder` class (analogous to
`BindingsBuilder`) to build the list vector prior to the call to
`Value::mkList()`. Typical usage:
auto list = state.buildList(size);
for (auto & v : list)
v = ... set value ...;
vRes.mkList(list);
* Add regression test
* Fix 'no repo' test so it doesn't succeed if the data is still in cache
* Use git_revparse_single inside git-utils instead of reimplementing the same logic.
Previously, errors while printing values in `nix repl` would be printed
in `«error: ...»` brackets rather than displayed normally:
```
nix-repl> legacyPackages.aarch64-darwin.pythonPackages.APScheduler
«error: Package ‘python-2.7.18.7’ in /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/pkgs/development/interpreters/python/cpython/2.7/default.nix:335 is marked as insecure, refusing to evaluate.»
```
Now, errors will be displayed normally if they're emitted at the
top-level of an expression:
```
nix-repl> legacyPackages.aarch64-darwin.pythonPackages.APScheduler
error:
… in the condition of the assert statement
at /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/lib/customisation.nix:268:17:
267| in commonAttrs // {
268| drvPath = assert condition; drv.drvPath;
| ^
269| outPath = assert condition; drv.outPath;
… in the left operand of the OR (||) operator
at /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/pkgs/development/interpreters/python/passthrufun.nix:28:45:
27| if lib.isDerivation value then
28| lib.extendDerivation (valid value || throw "${name} should use `buildPythonPackage` or `toPythonModule` if it is to be part of the Python packages set.") {} value
| ^
29| else
(stack trace truncated; use '--show-trace' to show the full trace)
error: Package ‘python-2.7.18.7’ in /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/pkgs/development/interpreters/python/cpython/2.7/default.nix:335 is marked as insecure, refusing to evaluate.
```
Errors emitted in nested structures (like e.g. when printing `nixpkgs`)
will still be printed in brackets.
Strings are now printed directly when evaluated by `:print`, rather than
escaped. This makes it easier to debug multi-line strings or strings
containing quotes, like the results of `builtins.readFile`,
`lib.toShellArg`, and so on.
```
nix-repl> "cuppy\ndog\ncity"
"cuppy\ndog\ncity"
nix-repl> :p "cuppy\ndog\ncity"
cuppy
dog
city
```
Fixes this very long warning, which I'll only include the first line of:
/nix/store/8wrjhrycpshhc3b41xmjwvgqr2m3yajq-libcxx-16.0.6-dev/include/c++/v1/__memory/construct_at.h:66:5: warning: destructor called on non-final 'RegexMatcher' that has virtual functions but non-virtual destructor [-Wdelete-non-abstract-non-virtual-dtor]
__loc->~_Tp();
`nix eval` forces values and prints derivations as attribute sets, so
commands that print derivations (e.g. `nix eval nixpkgs#bash`) will
infinitely loop and segfault.
Printing derivations as `.drv` paths makes `nix eval` complete as
expected. Further work is needed, but this is better than a segfault.
This wasn't caught by CI because #10149 and #10152 pass
individually... It doesn't happen on lazy-trees either because we
never try to fetch relative path flakes (#10089).
we now keep not a table of all positions, but a table of all origins and
their sizes. position indices are now direct pointers into the virtual
concatenation of all parsed contents. this slightly reduces memory usage
and time spent in the parser, at the cost of not being able to report
positions if the total input size exceeds 4GiB. this limit is not unique
to nix though, rustc and clang also limit their input to 4GiB (although
at least clang refuses to process inputs that are larger, we will not).
this new 4GiB limit probably will not cause any problems for quite a
while, all of nixpkgs together is less than 100MiB in size and already
needs over 700MiB of memory and multiple seconds just to parse. 4GiB
worth of input will easily take multiple minutes and over 30GiB of
memory without even evaluating anything. if problems *do* arise we can
probably recover the old table-based system by adding some tracking to
Pos::Origin (or increasing the size of PosIdx outright), but for time
being this looks like more complexity than it's worth.
since we now need to read the entire input again to determine the
line/column of a position we'll make unsafeGetAttrPos slightly lazy:
mostly the set it returns is only used to determine the file of origin
of an attribute, not its exact location. the thunks do not add
measurable runtime overhead.
notably this change is necessary to allow changing the parser since
apparently nothing supports nix's very idiosyncratic line ending choice
of "anything goes", making it very hard to calculate line/column
positions in the parser (while byte offsets are very easy).
this needs a string comparison because there seems to be no other way to
get that information out of bison. usually the location info is going to
be correct (pointing at a bad token), but since EOF isn't a token as
such it'll be wrong in that this case.
this hasn't shown up much so far because a single line ending *is* a
token, so any file formatted in the usual manner (ie, ending in a line
ending) would have its EOF position reported correctly.
the parser treats a plain \r as a newline, error reports do not. this
can lead to interesting divergences if anything makes use of this
feature, with error reports pointing to wrong locations in the input (or
even outside the input altogether).
previously we reported the error at the beginning of the binding
block (for plain inherits) or the beginning of the attr list (for
inherit-from), effectively hiding where exactly the error happened.
this also carries over to runtime positions of attributes in sets as
reported by unsafeGetAttrPos. we're not worried about this changing
observable eval behavior because it *is* marked unsafe, and the new
behavior is much more useful.
we already normalize attr order to lexicographic, doing the same for
formals makes sense. doubly so because the order of formals would
otherwise depend on the context of the expression, which is not quite as
useful as one might expect.
the parser modifies its inputs, which means that sharing them between
the error context reporting system and the parser itself can confuse the
reporting system. usually this led to early truncation of error context
reports which, while not dangerous, can be quite confusing.
* Convert all InputScheme::fetch() methods to getAccessor().
* Add checkLocks() method for checking lock attributes.
* Rename fetch() to fetchToStore().
This is the case for e.g. dirty Git workdirs, where we would get
$ nix flake metadata
Resolved URL: git+file:///home/eelco/Dev/nix-master
Locked URL: git+file:///home/eelco/Dev/nix-master
The "lockedRef" field is a misnomer, since it can be unlocked
(e.g. for a dirty Git workdir). In that case, `nix profile upgrade`
needs to assume that the package can have changed, and perform an
upgrade.
When reviewing old PRs, I found that #9997 adds some code to ensure one
particular assert is always present. But, removing asserts isn't
something we do in our own release builds either in the flake here or in
nixpkgs, and is plainly a bad idea that increases support burden,
especially if other distros make bad choices of build flags in their Nix
packaging.
For context, the assert macro in the C standard is defined to do nothing
if NDEBUG is set.
There is no way in our build system to set -DNDEBUG without manually
adding it to CFLAGS, so this is simply a configuration we do not use.
Let's ban it at compile time.
I put this preprocessor directive in src/libutil.cc because it is not
obvious where else to put it, and it seems like the most logical file
since you are not getting a usable nix without it.
Directly fail if a flakeref points to something that isn't a directory
instead of falling back to the logic of trying to look up the hierarchy
to find a valid flake root.
Fix https://github.com/NixOS/nix/issues/9868
It is possible to exfiltrate a file descriptor out of the build sandbox
of FODs, and use it to modify the store path after it has been
registered.
To avoid that issue, don't register the output of the build, but a copy
of it (that will be free of any leaked file descriptor).
Instead, serialize as NAR and send that over, then rehash sever side.
This is alorithmically simpler, but comes at the cost of a newer
parameter to `Store::addToStoreFromDump`.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Part of RFC 133
Extracted from our old IPFS branches.
Co-Authored-By: Matthew Bauer <mjbauer95@gmail.com>
Co-Authored-By: Carlo Nucera <carlo.nucera@protonmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Florian Klink <flokli@flokli.de>
desugaring inherit-from to syntactic duplication of the source expr also
duplicates side effects of the source expr (such as trace calls) and
expensive computations (such as derivationStrict).
Flakes still reside in the Nix store (so there shouldn't be any change
in behaviour), but they are now accessed via the rootFS
accessor. Since rootFS implements access checks, we no longer have to
worry about flake.{nix,lock} or their parents being symlinks that
escape from the flake.
Extracted from the lazy-trees branch.
"hash type" -> "hash algorithm" in all comments, documentation, and
messages.
ht -> ha, [Hh]ashType -> [HhashAlgo] for all local variables and
function arguments. No API change is made.
Continuation of 5334c9c792 and 837b889c41.
The sandbox rule `(allow network* (local ip))` doesn't do what it
implies. Adding this rule permits all network traffic. We should be
matching on (remote ip "localhost:*")` instead.
This seems to have found one actual bug in fs-sink.cc: the symlink case
was falling into the regular file case, which can't possibly be
intentional, right?
When a file conflict arises during a package install a suggestion is
made to remove the old entry. This was previously done using the
installable URLs of the old entry. These URLs are quite verbose and
often do not equal the URL of the existing entry.
This change uses the recently introduced profile entry name for the
suggestion, resulting in a simpler output.
The improvement is easily seen in the change to the functional test.
- `nix store add` supports text hashing
With functional test ensuring it matches `builtins.toFile`.
- Factored-out flags for both commands
- Move all common reusable flags to `libcmd`
- They are not part of the *definition* of the CLI infra, just a usag
of it.
- The `libstore` flag couldn't go in `args.hh` in libutil anyways,
would be awkward for it to live alone
- Shuffle around `Cmd*` hierarchy so flags for deprecated commands don't
end up on the new ones
This PR reduces the creation of short-lived basic_json objects while
parsing flake.lock files. For large flake.lock files (~1.5MB) I was
observing ~60s being spent for trivial nix build operations while
after this change it is now taking ~1.6s.
It's better to just check whether the input has all the attributes
needed to consider itself locked (e.g. whether a Git input has an
'rev' attribute).
Also, the 'locked' field was actually incorrect for Git inputs: it
would be set to true even for dirty worktrees. As a result, we got
away with using fetchTree() internally even though fetchTree()
requires a locked input in pure mode. In particular, this allowed
'--override-input' to work by accident.
The fix is to pass a set of "overrides" to call-flake.nix for all the
unlocked inputs (i.e. the top-level flake and any --override-inputs).
This fixes warnings like
warning: Ignoring setting 'auto-allocate-uids' because experimental feature 'auto-allocate-uids' is not enabled
warning: Ignoring setting 'impure-env' because experimental feature 'configurable-impure-env' is not enabled
when using the daemon and the user didn't actually set those settings.
Note: this also hides those settings from `nix config show`, but that
seems a good thing.
`canonPath` and `absPath` work on native paths, and so should switch
between supporting Unix paths and Windows paths accordingly.
The templating is because `CanonPath`, which shares the implementation,
should always be Unix style. It is the pure "nix-native" path type for
virtual file operations --- it is part of Nix's "business logic", and
should not vary with the host OS accordingly.
The core `CanonPath` constructors were using `absPath`, but `absPath` in
some situations does IO which is not appropriate. It turns out that
these constructors avoided those situations, and thus were pure, but it
was far from obvious this was the case.
To remedy the situation, abstract the core algorithm from `canonPath` to
use separately in `CanonPath` without any IO. No we know by-construction
that those constructors are pure.
That leaves `CanonPath::fromCWD` as the only operation which uses IO /
is impure. Add docs on it, and `CanonPath` as a whole, explaining the
situation.
This is also necessary to support Windows paths on windows without
messing up `CanonPath`. But, I think it is good even without that.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
There is no longer an `importTarball` method. Instead, there is a
`unpackTarfileToSink` function (back in libutil). The caller can use
thisw with the `getParseSink` method we added in the last commit easily
enough.
In addition, tarball cache functionality is separated from `git-utils`
and moved into `tarball-cache`. This ensures we are separating mechanism
and policy.
There is now a separation of:
1. A `FileSystemObjectSink` for writing to git repos
2. Adapting libarchive to use that parse sink.
The prepares a proper separation of concerns.
A command like
rm -rf ~/.cache/nix/tarball-cache/ ~/.cache/nix/fetcher-cache-v1.sqlite*; nix flake metadata 'git+file:///home/eelco/Dev/nixpkgs?rev=9463103069725474698139ab10f17a9d125da859'
was spending about 84% of its runtime in lookup(), specifically in
git_tree_entry_bypath(). (The reading of blobs is less than 3%.)
It appears libgit2 doesn't do a lot of caching of trees, so we now
make sure that when we look up a path, we add all its parents, and all
the immediate children of the parents (since we have them in memory
anyway), to our own cache.
This speed up the command above from 17.2s to 7.8s on my machine.
Fixes (or at least should improve a lot) #9684.
Commit 83c067c0fa changed `builtins.pathExists`
to resolve symlinks before checking for existence. Consequently, if the path
refers to a symlink itself, existence of the target of the symlink (instead of
the symlink itself) was checked. Restore the previous behavior by skipping
symlink resolution in the last component.
No outward facing behavior is changed.
Older methods with same names that operate on on method + algo pair (for
old-style `<method>:algo`) are renamed to `*WithAlgo`.)
The functions are unit-tested in the same way the names for the hash
algorithms are tested.
* reword description of the `cores` setting
- be precise about the `builder` executable
- clearly distinguish between `builder` and job parallelism
- clarify the role of `mkDerivation` in the example
- remove prose for the default, it's shown programmatically
- mention relation to `max-jobs`
- move all reference documentation to the `builders` configuration setting
- reword documentation on machine specification, add examples
- disable showing the default value, as it rendered as `@/dummy/machines`, which is wrong
- highlight the examples
- link to the configuration docs for distributed builds
- builder -> build machine
Co-authored-by: Janik H <janik@aq0.de>
