This commit adds an optional `__impure` parameter to fetchurl.nix, which allows
the caller to use `libfetcher`'s fetcher in an impure derivation. This allows
nixpkgs' patch-normalizing fetcher (fetchpatch) to be rewritten to use nix's
internal fetchurl, thereby eliminating the awkward "you can't use fetchpatch
here" banners scattered all over the place.
See also: https://github.com/NixOS/nixpkgs/pull/188587
98e361ad4c introduced a regression where
previously stored attributes were replaced by placeholders. As a
result, a command like 'nix build nixpkgs#hello' had to be executed at
least twice to get caching.
This code does not seem necessary for suggestions to work.
Makes `printValueAsJSON` not copy paths to the store for `nix eval
--json`, `nix-instantiate --eval --json` and `nix-env --json`.
Fixes https://github.com/NixOS/nix/issues/5612
I recently got fairly confused why the following expression didn't have
any effect
{
description = "Foobar";
inputs.sops-nix = {
url = github:mic92/sops-nix;
inputs.nixpkgs_22_05.follows = "nixpkgs";
};
}
until I found out that the input was called `nixpkgs-22_05` (please note
the dash vs. underscore).
IMHO it's not a good idea to not throw an error in that case and
probably leave end-users rather confused, so I implemented a small check
for that which basically checks whether `follows`-declaration from
overrides actually have corresponding inputs in the transitive flake.
In fact this was done by accident already in our own test-suite where
the removal of a `follows` was apparently forgotten[1].
Since the key of the `std::map` that holds the `overrides` is a vector
and we have to find the last element of each vector (i.e. the override)
this has to be done with a for loop in O(n) complexity with `n` being
the total amount of overrides (which shouldn't be that large though).
Please note that this doesn't work with nested expressions, i.e.
inputs.fenix.inputs.nixpkgs.follows = "...";
which is a known problem[2].
For the expression demonstrated above, an error like this will be
thrown:
error: sops-nix has a `follows'-declaration for a non-existant input nixpkgs_22_05!
[1] 2664a216e5
[2] https://github.com/NixOS/nix/issues/5790
Prevents errors when running with UBSan:
/nix/store/j5vhrywqmz1ixwhsmmjjxa85fpwryzh0-gcc-11.3.0/include/c++/11.3.0/bits/stl_pair.h:353:4: runtime error: load of value 229, which is not a valid value for type 'AttrType'
Overrides for inputs with flake=false were non-sticky, since they
changed the `original` in `flake.lock`. This fixes it, by using the same
locked original for both flake and non-flake inputs.
Don’t explicitely give it a constructor, but use aggregate
initialization instead (also prevents having an implicit coertion, which
is probably good here)
* libexpr: fix builtins.split example
The example was previously indicating that multiple whitespaces would be
collapsed into a single captured whitespace. That isn't true and was
likely a mistake when being documented initially.
* Fix segfault on unitilized list when printing value
Since lists are just chunks of memory the individual elements in the
list might be unitilized when a programming error happens within Nix.
In this case the values are null-initialized (at least with Boehm GC)
and we can avoid a nullptr deref when printing them.
I ran into this issue while ensuring that new expression tests would
show the actual value on an assertion failure.
This is unlikely to cause any runtime performance regressions as
printing values is not really in the hot path (unless the repl is the
primary use case).
* Add operator<< for ValueTypes
* Add libexpr tests
This introduces tests for libexpr that evalulate various trivial Nix
language expressions and primop invocations that should be good smoke
tests wheter or not the implementation is behaving as expected.
'nix profile install' will now install all outputs listed in the
package's meta.outputsToInstall attribute, or all outputs if that
attribute doesn't exist. This makes it behave consistently with
nix-env. Fixes#6385.
Furthermore, for consistency, all other 'nix' commands do this as
well. E.g. 'nix build' will build and symlink the outputs in
meta.outputsToInstall, defaulting to all outputs. Previously, it only
built/symlinked the first output. Note that this means that selecting
a specific output using attrpath selection (e.g. 'nix build
nixpkgs#libxml2.dev') no longer works. A subsequent PR will add a way
to specify the desired outputs explicitly.
after #6218 `Symbol` no longer confers a uniqueness invariant on the
string it wraps, it is now possible to create multiple symbols that
compare equal but whose string contents have different addresses. this
guarantee is now only provided by `SymbolIdx`, leaving `Symbol` only as
a string wrapper that knows about the intricacies of how symbols need to
be formatted for output.
this change renames `SymbolIdx` to `Symbol` to restore the previous
semantics of `Symbol` to that name. we also keep the wrapper type and
rename it to `SymbolStr` instead of returning plain strings from lookups
into the symbol table because symbols are formatted for output in many
places. theoretically we do not need `SymbolStr`, only a function that
formats a string for output as a symbol, but having to wrap every symbol
that appears in a message into eg `formatSymbol()` is error-prone and
inconvient.
The produced path is then allowed be imported or utilized elsewhere:
```
assert (43 == import (builtins.toFile "source" "43")); "good"
```
This will still fail on write-only stores.
with position and symbol tables in place we can now shrink Attr by a full
pointer with some simple field reordering. since Attr is a very hot struct this
has substantial impact on memory use, decreasing GC allocations and heap size by
10-15% each. we also get a ~15% performance improvement due to reduced GC
loading.
pure parsing has taken a hit over the branch base because positions are now
slightly more expensive to create, but overall we get a noticeable improvement.
before (on memory-friendliness):
Benchmark 1: nix search --no-eval-cache --offline ../nixpkgs hello
Time (mean ± σ): 6.960 s ± 0.028 s [User: 5.832 s, System: 0.897 s]
Range (min … max): 6.886 s … 7.005 s 20 runs
Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
Time (mean ± σ): 328.1 ms ± 1.7 ms [User: 295.8 ms, System: 32.2 ms]
Range (min … max): 324.9 ms … 331.2 ms 20 runs
Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
Time (mean ± σ): 2.688 s ± 0.029 s [User: 2.365 s, System: 0.238 s]
Range (min … max): 2.642 s … 2.742 s 20 runs
after:
Benchmark 1: nix search --no-eval-cache --offline ../nixpkgs hello
Time (mean ± σ): 6.902 s ± 0.039 s [User: 5.844 s, System: 0.783 s]
Range (min … max): 6.820 s … 6.956 s 20 runs
Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
Time (mean ± σ): 330.7 ms ± 2.2 ms [User: 300.6 ms, System: 30.0 ms]
Range (min … max): 327.5 ms … 334.5 ms 20 runs
Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
Time (mean ± σ): 2.330 s ± 0.027 s [User: 2.040 s, System: 0.234 s]
Range (min … max): 2.272 s … 2.383 s 20 runs
this slightly increases the amount of memory used for any given symbol, but this
increase is more than made up for if the symbol is referenced more than once in
the EvalState that holds it. on average every symbol should be referenced at
least twice (once to introduce a binding, once to use it), so we expect no
increase in memory on average.
symbol tables are limited to 2³² entries like position tables, and similar
arguments apply to why overflow is not likely: 2³² symbols would require as many
string instances (at 24 bytes each) and map entries (at 24 bytes or more each,
assuming that the map holds on average at most one item per bucket as the docs
say). a full symbol table would require at least 192GB of memory just for
symbols, which is well out of reach. (an ofborg eval of nixpks today creates
less than a million symbols!)
PosTable deduplicates origin information, so using symbols for paths is no
longer necessary. moving away from path Symbols also reduces the usage of
symbols for things that are not keys in attribute sets, which will become
important in the future when we turn symbols into indices as well.
Pos objects are somewhat wasteful as they duplicate the origin file name and
input type for each object. on files that produce more than one Pos when parsed
this a sizeable waste of memory (one pointer per Pos). the same goes for
ptr<Pos> on 64 bit machines: parsing enough source to require 8 bytes to locate
a position would need at least 8GB of input and 64GB of expression memory. it's
not likely that we'll hit that any time soon, so we can use a uint32_t index to
locate positions instead.
when we introduce position and symbol tables we'll need to do lookups to turn
indices into those tables into actual positions/symbols. having the error
functions as members of EvalState will avoid a lot of churn for adding lookups
into the tables for each caller.
only file and line of the returned position were ever used, it wasn't actually
used a position. as such we may as well use a path+int pair for only those two
values and remove a use of Pos that would not work well with a position table.
a future commit will remove the ability to convert the symbol type used in
bindings to strings. since we only have two users we can inline the error check.
the only use of this function is to determine whether a lambda has a non-set
formal, but this use is arguably better served by Symbol::set and using a
non-Symbol instead of an empty symbol in the parser when no such formal is present.
we don't *need* symbols here. the only advantage they have over strings is
making call-counting slightly faster, but that's a diagnostic feature and thus
needn't be optimized.
this also fixes a move bug that previously didn't show up: PrimOp structs were
accessed after being moved from, which technically invalidates them. previously
the names remained valid because Symbol copies on move, but strings are
invalidated. we now copy the entire primop struct instead of moving since primop
registration happen once and are not performance-sensitive.
In particular, this means that 'nix eval` (which uses toValue()) no
longer auto-calls functions or functors (because
AttrCursor::findAlongAttrPath() doesn't).
Fixes#6152.
Also use ref<> in a few places, and don't return attrpaths from
getCursor() because cursors already have a getAttrPath() method.
Impure derivations are derivations that can produce a different result
every time they're built. Example:
stdenv.mkDerivation {
name = "impure";
__impure = true; # marks this derivation as impure
outputHashAlgo = "sha256";
outputHashMode = "recursive";
buildCommand = "date > $out";
};
Some important characteristics:
* This requires the 'impure-derivations' experimental feature.
* Impure derivations are not "cached". Thus, running "nix-build" on
the example above multiple times will cause a rebuild every time.
* They are implemented similar to CA derivations, i.e. the output is
moved to a content-addressed path in the store. The difference is
that we don't register a realisation in the Nix database.
* Pure derivations are not allowed to depend on impure derivations. In
the future fixed-output derivations will be allowed to depend on
impure derivations, thus forming an "impurity barrier" in the
dependency graph.
* When sandboxing is enabled, impure derivations can access the
network in the same way as fixed-output derivations. In relaxed
sandboxing mode, they can access the local filesystem.
Rather than having four different but very similar types of hashes, make
only one, with a tag indicating whether it corresponds to a regular of
deferred derivation.
This implies a slight logical change: The original Nix+multiple-outputs
model assumed only one hash-modulo per derivation. Adding
multiple-outputs CA derivations changed this as these have one
hash-modulo per output. This change is now treating each derivation as
having one hash modulo per output.
This obviously means that we internally loose the guaranty that
all the outputs of input-addressed derivations have the same hash
modulo. But it turns out that it doesn’t matter because there’s nothing
in the code taking advantage of that fact (and it probably shouldn’t
anyways).
The upside is that it is now much easier to work with these hashes, and
we can get rid of a lot of useless `std::visit{ overloaded`.
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
This allows closures to be imported at evaluation time, without
requiring the user to configure substituters. E.g.
builtins.fetchClosure {
storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2;
from = "https://cache.ngi0.nixos.org";
}
Before the change lexter errors did not report the location:
$ nix build -f. mc
error: path has a trailing slash
(use '--show-trace' to show detailed location information)
Note that it's not clear what file generates the error.
After the change location is reported:
$ src/nix/nix --extra-experimental-features nix-command build -f ~/nm mc
error: path has a trailing slash
at .../pkgs/development/libraries/glib/default.nix:54:18:
53| };
54| src = /tmp/foo/;
| ^
55|
(use '--show-trace' to show detailed location information)
Here we see both problematic file and the string itself.
1. `DerivationOutput` now as the `std::variant` as a base class. And the
variants are given hierarchical names under `DerivationOutput`.
In 8e0d0689be @matthewbauer and I
didn't know a better idiom, and so we made it a field. But this sort
of "newtype" is anoying for literals downstream.
Since then we leaned the base class, inherit the constructors trick,
e.g. used in `DerivedPath`. Switching to use that makes this more
ergonomic, and consistent.
2. `store-api.hh` and `derivations.hh` are now independent.
In bcde5456cc I swapped the dependency,
but I now know it is better to just keep on using incomplete types as
much as possible for faster compilation and good separation of
concerns.
The current `--out-path` flag has two disadvantages when one is only
concerned with querying the names of outputs:
- it requires evaluating every output's `outPath`, which takes
significantly more resources and runs into more failures
- it destroys the information of the order of outputs so we can't tell
which one is the main output
This patch makes the output names always present (replacing paths with
`null` in JSON if `--out-path` isn't given), and adds an `outputName`
field.
This changes was taken from dynamic derivation (#4628). It` somewhat
undoes the refactors I first did for floating CA derivations, as the
benefit of hindsight + requirements of dynamic derivations made me
reconsider some things.
They aren't to consequential, but I figured they might be good to land
first, before the more profound changes @thufschmitt has in the works.
reduces peak hep memory use on eval of our test system from 264.4MB to 242.3MB,
possibly also a slight performance boost.
theoretically memory use could be cut down by another eight bytes per Pos on
average by turning it into a tuple containing an index into a global base
position table with row and column offsets, but that doesn't seem worth the
effort at this point.
```console
$ nix eval --expr '({ foo ? 1 }: foo) { fob = 2; }'
error: anonymous function at (string):1:2 called with unexpected argument 'fob'
at «string»:1:1:
1| ({ foo ? 1 }: foo) { fob = 2; }
| ^
Did you mean foo?
```
Not that because Nix will first check for _missing_ arguments before
checking for extra arguments, `({ foo }: foo) { fob = 1; }` will
complain about the missing `foo` argument (rather than extra `fob`) and
so won’t display a suggestion.
Make the evaluator show some suggestions when trying to access an
invalid field from an attrset.
```console
$ nix eval --expr '{ foo = 1; }.foa'
error: attribute 'foa' missing
at «string»:1:1:
1| { foo = 1; }.foa
| ^
Did you mean foo?
```
No real need for keeping a separate header for such a simple class.
This requires changing a bit `OrSuggestions<T>::operator*` to not throw
an `Error` to prevent a cyclic dependency. But since this error is only
thrown on programmer error, we can replace the whole method by a direct
call to `std::get` which will raise its own assertion if needs be.
Refactor the `size == 0` logic into a new helper function that
replaces dupStringWithLen.
The name had to change, because unlike a `dup`-function, it does
not always allocate a new string.
We now memoize on Bindings / list element vectors rather than Values,
so that e.g. two Values that point to the same Bindings will be
printed only once.
This is useful whenever we want to evaluate something to a store path
(e.g. in get-drvs.cc).
Extracted from the lazy-trees branch (where we can require that a
store path must come from a store source tree accessor).
This was introduced in #6174. However fetch{url,Tarball} are legacy
and we shouldn't have an undocumented attribute that does the same
thing as one that already exists ('sha256').
Starting work on #5638
The exact boundary between `FetchSettings` and `EvalSettings` is not
clear to me, but that's fine. First lets clean out `libstore`, and then
worry about what, if anything, should be the separation between those
two.
previously :a would override old bindings of a name with new values if the added
set contained names that were already bound. in nix 2.6 this doesn't happen any
more, which is potentially confusing.
fixes#6041
we'll retain the old coerceToString interface that returns a string, but callers
that don't need the returned value to outlive the Value it came from can save
copies by using the new interface instead. for values that weren't stringy we'll
pass a new buffer argument that'll be used for storage and shouldn't be
inspected.
It’s totally valid to have entries in `NIX_PATH` that aren’t valid paths
(they can even be arbitrary urls or `channel:<channel-name>`).
Fix#5998 and #5980
