Commit graph

7731 commits

Author SHA1 Message Date
Eelco Dolstra
2016b7142a Fix compilation, style fixes 2022-04-19 13:41:32 +02:00
Eelco Dolstra
b529a41814 Merge branch 'make-flake-show-more-lenient-on-apps' of https://github.com/flox/nix 2022-04-19 13:41:24 +02:00
Eelco Dolstra
1b43d64648
Merge pull request #6415 from aakropotkin/doc.connect-timeout.default
doc: document nix.conf connect-timeout default
2022-04-19 12:27:30 +02:00
Eelco Dolstra
018db1c420
Merge pull request #6404 from edolstra/unify-flake-attr-eval
Make InstallableFlake::toValue() and toDerivation() behave consistently
2022-04-19 11:53:27 +02:00
John Ericson
75b62e5260 Avoid fmt when constructor already does it
There is a correctnes issue here, but #3724 will fix that. This is just
a cleanup for brevity's sake.
2022-04-19 01:44:11 +00:00
Robert Helgesson
8b659eacce
Add .tgz as tarball extension in documentation
Support for the `tgz` shorthand was added in
52f5fa948a.
2022-04-18 17:14:15 +02:00
Alex Ameen
e5c934cd48
doc: rephrase connect-timeout help message
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2022-04-17 18:17:37 -05:00
Alex Ameen
25c85f5a0e
doc: document nix.conf connect-timeout default 2022-04-17 17:14:38 -05:00
Ben Burdette
93b8d31508 Merge branch 'master' into debug-exploratory-PR 2022-04-15 14:49:08 -06:00
Eelco Dolstra
b135de2b5f
Merge pull request #6323 from erikarvstedt/eval-read-only
`nix eval`: Add option `--read-only`
2022-04-15 10:31:05 +02:00
Tom Bereknyei
9b41239d8f fix: ensure apps are apps and packages are packages 2022-04-14 23:57:52 -04:00
Eelco Dolstra
d6effddd3b
Merge pull request #6387 from Uthar/fix
assert hash types for Git and Mercurial
2022-04-14 14:55:27 +02:00
Eelco Dolstra
d89840b103 Make InstallableFlake::toValue() and toDerivation() behave consistently
In particular, this means that 'nix eval` (which uses toValue()) no
longer auto-calls functions or functors (because
AttrCursor::findAlongAttrPath() doesn't).

Fixes #6152.

Also use ref<> in a few places, and don't return attrpaths from
getCursor() because cursors already have a getAttrPath() method.
2022-04-14 14:07:04 +02:00
Kasper Gałkowski
2769e43f61 assert hash types for Git and Mercurial 2022-04-12 21:13:14 +02:00
Eelco Dolstra
f7276bc948
Merge pull request #6392 from danpls/fix-actualUrl-mercurial
libfetchers: Fix assertion (Mercurial)
2022-04-11 11:16:04 +02:00
Eelco Dolstra
092f6d2e7a
Merge pull request #6380 from thufschmitt/fix-double-slahsh-in-uri
Allow empty path segments in urls
2022-04-11 11:15:14 +02:00
Eelco Dolstra
2311868aaa
Merge pull request #6391 from danpls/replace-regex
libfetchers: Replace regex to clarify intent
2022-04-11 11:14:10 +02:00
Sebastian Blunt
63d9a81819 Log builder args and environment variables
Previously it only logged the builder's path, this changes it to log the
arguments at the same log level, and the environment variables at the
vomit level.

This helped me debug https://github.com/svanderburg/node2nix/issues/75
2022-04-10 21:10:37 -07:00
Ben Burdette
8b197c492e remove comma 2022-04-09 21:54:41 -06:00
Daniel Pauls
d6b7529579 libfetchers: Fix assertion (Mercurial)
See commit 1e1cd6e7a for more information.
2022-04-09 19:10:23 +02:00
Daniel Pauls
770f7371f3 libfetchers: Replace regex to clarify intent 2022-04-09 17:00:14 +02:00
Ben Burdette
a61841ac41 don't use std::map merge 2022-04-09 07:45:23 -06:00
Ben Burdette
f5757a0804 revise command help 2022-04-08 16:34:20 -06:00
Ben Burdette
3aaf02839f trace stack, not call stack 2022-04-08 16:22:27 -06:00
Ben Burdette
31bcd55626 clean up makefiles 2022-04-08 15:53:24 -06:00
Ben Burdette
27d45f9eb3 minor cleanup 2022-04-08 15:46:12 -06:00
Ben Burdette
a86c2a8481 remove 'debugError', dead code 2022-04-08 13:30:18 -06:00
Ben Burdette
b8b8ec7101 move throw to preverve Error type; turn off debugger for tryEval 2022-04-08 12:34:27 -06:00
Théophane Hufschmitt
646af7325d
Merge pull request #6376 from Uthar/master
don't assume that rev is a SHA1 hash
2022-04-08 17:56:27 +02:00
Théophane Hufschmitt
f3d3587ab3 Allow empty path segments in urls
Valid per https://datatracker.ietf.org/doc/html/rfc3986#section-3.3 (and
also somewhat frequently happening for local paths)
2022-04-08 16:09:49 +02:00
Eelco Dolstra
c68963eaea Remove duplicate "error:" 2022-04-08 11:48:30 +02:00
Eelco Dolstra
8bd9ebf52c Error: Remove unused sname() method 2022-04-08 11:31:51 +02:00
Eelco Dolstra
168ef9f3ab Remove unused Error.name field 2022-04-08 11:31:51 +02:00
Ben Burdette
1a93ac8133 Merge remote-tracking branch 'upstream/master' into upstream-merge 2022-04-07 13:42:01 -06:00
Ben Burdette
d2ec9b4e15 in debugger mode, print the current error when another repl returns. 2022-04-07 12:09:47 -06:00
Ben Burdette
50b52d5110 remove debug code 2022-04-07 12:03:18 -06:00
Kasper Gałkowski
2c2fd4946f don't assume that rev is a SHA1 hash
This was a problem when writing a fetcher that uses e.g. sha256 hashes
for revisions. This doesn't actually do anything new, but allows for
creating such fetchers in the future (perhaps when support for Git's
SHA256 object format gains more popularity).
2022-04-07 19:49:47 +02:00
Ben Burdette
d29af88d58 newline before env 2022-04-07 11:17:57 -06:00
Ben Burdette
f37562187f free valmap on exit 2022-04-07 11:17:31 -06:00
Eelco Dolstra
8b1e328d5d
Merge pull request #6348 from cole-h/fix-restoring-mount-namespace
libutil: Fix restoring mount namespace
2022-04-07 18:15:33 +02:00
Théophane Hufschmitt
b53e0a6aa0
Merge pull request #6374 from danpls/fix-actualUrl
libfetchers: Fix assertion
2022-04-07 17:57:24 +02:00
Ben Burdette
5cfd038bd8 show expr pos if DebugTrace one is noPos 2022-04-06 19:08:29 -06:00
Rehno Lindeque
b9c969a866 nix flake check: Warn about deprecated nixosModule output 2022-04-06 12:20:39 -04:00
Daniel Pauls
1e1cd6e7a9 libfetchers: Fix assertion
The filter expects all paths to have a prefix of the raw `actualUrl`, but
`Store::addToStore(...)` provides absolute canonicalized paths.
To fix this create an absolute and canonicalized path from the `actualUrl` and
use it instead.

Fixes #6195.
2022-04-06 17:33:23 +02:00
Eelco Dolstra
318936366d Fix empty 'nix copy' error message
This was caused by SubstitutionGoal not setting the errorMsg field in
its BuildResult. We now get a more descriptive message than in 2.7.0, e.g.

  error: path '/nix/store/13mh...' is required, but there is no substituter that can build it

instead of the misleading (since there was no build)

  error: build of '/nix/store/13mh...' failed

Fixes #6295.
2022-04-06 12:43:53 +02:00
Eelco Dolstra
589f6f267b fetchClosure: Don't allow URL query parameters
Allowing this is a potential security hole, since it allows the user
to specify parameters like 'local-nar-cache'.
2022-04-06 11:52:51 +02:00
Eelco Dolstra
c0ad86f681
Merge pull request #6366 from danpls/base64-reserve
libutil: Reserve memory when en/decoding base64
2022-04-05 23:20:33 +02:00
Daniel Pauls
513652d594 tokenizeString: Fix semantic mistake
`string_view::find_first_not_of(...)` and
`string_view::find_first_of(...)` return `string_view::npos` on error
not `string::npos`.
2022-04-05 22:33:03 +02:00
Daniel Pauls
1fa0393479 libutil: Reserve memory when en/decoding base64
The size of the output when encoding to and decoding from base64 is
(roughly) known so we can allocate it in advance to prevent
reallocation.
2022-04-05 21:30:50 +02:00
Eelco Dolstra
27b952a8a1
Merge pull request #6362 from thufschmitt/verbose-doctor
doctor: Always show the output
2022-04-05 17:33:10 +02:00
Théophane Hufschmitt
9a640afc1e doctor: Always show the output
Fix https://github.com/NixOS/nix/issues/6342
2022-04-05 14:04:01 +02:00
Théophane Hufschmitt
5abe3f4aa6 Allow welcomeText when checking a flake template
Fix https://github.com/NixOS/nix/issues/6321
2022-04-05 11:03:43 +02:00
Cole Helbling
56009b2639 libutil: don't save cwd fd, use path instead
Saving the cwd fd didn't actually work well -- prior to this commit, the
following would happen:

    : ~/w/vc/nix ; doas outputs/out/bin/nix --experimental-features 'nix-command flakes' run nixpkgs#coreutils -- --coreutils-prog=pwd
    pwd: couldn't find directory entry in ‘../../../..’ with matching i-node
    : ~/w/vc/nix ; doas outputs/out/bin/nix --experimental-features 'nix-command flakes' develop -c pwd
    pwd: couldn't find directory entry in ‘../../../..’ with matching i-node
2022-04-04 10:28:08 -07:00
Cole Helbling
10b9c1b2b2 libutil: save cwd fd in restoreMountNamespace
This doesn't work very well (maybe I'm misunderstanding the desired
implementation):

    : ~/w/vc/nix ; doas outputs/out/bin/nix --experimental-features 'nix-command flakes' develop -c pwd
    pwd: couldn't find directory entry in ‘../../../..’ with matching i-node
2022-04-04 10:28:00 -07:00
Cole Helbling
f89b0f7846 libutil: try restoring the cwd from fdSavedCwd 2022-04-04 08:33:59 -07:00
Cole Helbling
e135d223f6 libutil: save fd to cwd instead of cwd itself 2022-04-04 08:32:45 -07:00
Cole Helbling
e5b70d47aa libutil: cleanup savedCwd logic
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2022-04-04 08:20:11 -07:00
Eelco Dolstra
a4a1de69dc Add missing #include 2022-04-04 16:49:39 +02:00
Tom Bereknyei
c1e2ce4515 fix(run): set applyNixConfig lockFlag 2022-04-01 23:41:34 -04:00
Cole Helbling
2a45cf54e4 libutil: Properly guard self-allocating getcwd on GNU
It's a GNU extension, as pointed out by pennae.
2022-04-01 12:20:34 -07:00
Cole Helbling
7f5caaa7c0 libutil: Don't use std::filesystem
Just in case making libutil depend on std::filesystem is unacceptable,
here is the non-filesystem approach.
2022-04-01 10:24:31 -07:00
aszlig
435848cef1 libutil: Fix restoring mount namespace
I regularly pass around simple scripts by using nix-shell as the script
interpreter, eg. like this:

    #!/usr/bin/env nix-shell
    #!nix-shell -p dd_rescue coreutils bash -i bash

While this works most of the time, I recently had one occasion where it
would not and the above would result in the following:

    $ sudo ./myscript.sh
    bash: ./myscript.sh: No such file or directory

Note the "sudo" here, because this error only occurs if we're root.

The reason for the latter is because running Nix as root means that we
can directly access the store, which makes sure we use a filesystem
namespace to make the store writable. XXX - REWORD!

So when stracing the process, I stumbled on the following sequence:

    openat(AT_FDCWD, "/proc/self/ns/mnt", O_RDONLY) = 3
    unshare(CLONE_NEWNS)                            = 0
    ... later ...
    getcwd("/the/real/cwd", 4096)                   = 14
    setns(3, CLONE_NEWNS)                           = 0
    getcwd("/", 4096)                               = 2

In the whole strace output there are no calls to chdir() whatsoever, so
I decided to look into the kernel source to see what else could change
directories and found this[1]:

    /* Update the pwd and root */
    set_fs_pwd(fs, &root);
    set_fs_root(fs, &root);

The set_fs_pwd() call is roughly equivalent to a chdir() syscall and
this is called when the setns() syscall is invoked[2].

[1]: b14ffae378/fs/namespace.c (L4659)
[2]: b14ffae378/kernel/nsproxy.c (L346)
2022-04-01 09:30:52 -07:00
Eelco Dolstra
bf4895961d
Merge pull request #6344 from flox/profile_url_uri
profile!: consistent use of url/uri. create new version
2022-04-01 14:38:32 +02:00
Eelco Dolstra
fdfe737867 Fix handling of outputHash when outputHashAlgo is not specified
https://hydra.nixos.org/build/171351131
2022-04-01 12:40:49 +02:00
Eelco Dolstra
c9a29d0d92
Merge pull request #6227 from NixOS/impure-derivations-ng
Impure derivations
2022-03-31 19:58:35 +02:00
Ben Burdette
f41c18b221 comments 2022-03-31 09:39:18 -06:00
Ben Burdette
1096d17b65 show 'with' bindings as well as static 2022-03-31 09:37:36 -06:00
Eelco Dolstra
7537097284 Provide default values for outputHashAlgo and outputHashMode 2022-03-31 16:56:44 +02:00
Eelco Dolstra
a99af85a77 Fix macOS build 2022-03-31 16:39:18 +02:00
Eelco Dolstra
6051cc954b Rename 'pure' -> 'sandboxed' for consistency 2022-03-31 16:12:25 +02:00
Eelco Dolstra
e279fbb16a needsNetworkAccess() -> isSandboxed() 2022-03-31 16:06:40 +02:00
Eelco Dolstra
4e043c2f32 Document isPure() 2022-03-31 16:01:50 +02:00
Eelco Dolstra
d7fc33c842 Fix macOS build 2022-03-31 15:59:14 +02:00
Eelco Dolstra
18935e8b9f Support fixed-output derivations depending on impure derivations 2022-03-31 13:43:20 +02:00
Eelco Dolstra
5cd72598fe Add support for impure derivations
Impure derivations are derivations that can produce a different result
every time they're built. Example:

  stdenv.mkDerivation {
    name = "impure";
    __impure = true; # marks this derivation as impure
    outputHashAlgo = "sha256";
    outputHashMode = "recursive";
    buildCommand = "date > $out";
  };

Some important characteristics:

* This requires the 'impure-derivations' experimental feature.

* Impure derivations are not "cached". Thus, running "nix-build" on
  the example above multiple times will cause a rebuild every time.

* They are implemented similar to CA derivations, i.e. the output is
  moved to a content-addressed path in the store. The difference is
  that we don't register a realisation in the Nix database.

* Pure derivations are not allowed to depend on impure derivations. In
  the future fixed-output derivations will be allowed to depend on
  impure derivations, thus forming an "impurity barrier" in the
  dependency graph.

* When sandboxing is enabled, impure derivations can access the
  network in the same way as fixed-output derivations. In relaxed
  sandboxing mode, they can access the local filesystem.
2022-03-31 13:43:20 +02:00
Eelco Dolstra
0fe8849914
Merge pull request #6337 from danpls/fix-to-json-repl
libexpr: Throw the correct error in toJSON
2022-03-31 11:33:16 +02:00
Eelco Dolstra
28309352d9 replaceEnv(): Pass newEnv by reference 2022-03-31 10:39:53 +02:00
Tom Bereknyei
50f9f335c9 profile!: consistent use of url/uri. create new version 2022-03-30 16:35:26 -04:00
Tom Bereknyei
d77823b502 bundler: update default bundler to support new bundler API 2022-03-30 16:10:42 -04:00
Daniel Pauls
629edd43ba libutil: Change return value of addTrace to void
The return value of BaseError::addTrace(...) is never used and
error-prone as subclasses calling it will return a BaseError instead of
the subclass.
This commit changes its return value to be void.
2022-03-30 18:37:32 +02:00
Daniel Pauls
fa83b865a2 libexpr: Throw the correct error in toJSON
BaseError::addTrace(...) returns a BaseError, but we want to
throw a TypeError instead.

Fixes #6336.
2022-03-30 15:50:13 +02:00
Ben Burdette
c0a567e196 remove const_cast 2022-03-29 16:44:47 -06:00
Théophane Hufschmitt
390269ed87 Simplify the handling of the hash modulo
Rather than having four different but very similar types of hashes, make
only one, with a tag indicating whether it corresponds to a regular of
deferred derivation.

This implies a slight logical change: The original Nix+multiple-outputs
model assumed only one hash-modulo per derivation. Adding
multiple-outputs CA derivations changed this as these have one
hash-modulo per output. This change is now treating each derivation as
having one hash modulo per output.
This obviously means that we internally loose the guaranty that
all the outputs of input-addressed derivations have the same hash
modulo. But it turns out that it doesn’t matter because there’s nothing
in the code taking advantage of that fact (and it probably shouldn’t
anyways).

The upside is that it is now much easier to work with these hashes, and
we can get rid of a lot of useless `std::visit{ overloaded`.

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2022-03-29 18:17:35 +02:00
Ben Burdette
5ab7bdf0b1 load debug trace staticenv on 'show' 2022-03-28 15:28:59 -06:00
Ben Burdette
14f515544b debugTraceIndex 2022-03-28 12:09:21 -06:00
Théophane Hufschmitt
2d572a250f
Merge pull request #6330 from edolstra/run-remote-store
nix {run,shell}: Print a better error message if the store is not local
2022-03-28 17:57:37 +02:00
Eelco Dolstra
b266fd53dd nix {run,shell}: Print a better error message if the store is not local
Closes #6317
2022-03-28 14:58:38 +02:00
Eelco Dolstra
057f9ee190 nix profile install: Don't use queryDerivationOutputMap()
Instead get the outputs from Installable::build(). This will also
allow 'nix profile install' to support impure derivations.

Fixes #6286.
2022-03-28 14:23:39 +02:00
Erik Arvstedt
16860a0328 nix eval: Add option read-only 2022-03-26 11:32:38 +01:00
Erik Arvstedt
679b3b32c9 Minor comment fix 2022-03-26 11:32:37 +01:00
Ben Burdette
1bec3fb337 add DebugTrace for error 2022-03-25 18:15:31 -06:00
Eelco Dolstra
1844172dd1
Merge pull request #6314 from edolstra/experimental-primop
Only provide builtins is the corresponding experimental feature is enabled
2022-03-25 16:16:31 +01:00
Eelco Dolstra
16cf1e6089
Merge pull request #6311 from edolstra/return-wanted-paths
Make buildPathsWithResults() only return info on wanted outputs
2022-03-25 15:44:39 +01:00
Eelco Dolstra
fc35b11a7c Fix mismatched tag warning on clang 2022-03-25 15:22:22 +01:00
Eelco Dolstra
8c363eb3eb Document getFlake
Fixes #5523.
2022-03-25 14:19:55 +01:00
Eelco Dolstra
86b05ccd54 Only provide builtin.{getFlake,fetchClosure} is the corresponding experimental feature is enabled
This allows writing fallback code like

  if builtins ? fetchClosure then
    builtins.fetchClose { ... }
  else
    builtins.storePath ...
2022-03-25 14:04:18 +01:00
Eelco Dolstra
55bc524019
Merge pull request #6051 from Ma27/fix-empty-nix-log
`nix log` should also work if the log didn't provide any output
2022-03-25 10:35:10 +01:00
Eelco Dolstra
50c229ad9a
Use wantOutput
Co-authored-by: John Ericson <git@JohnEricson.me>
2022-03-25 08:02:49 +01:00
polykernel
cbcb69a39c nix: allow whitespace characters before command in repl
Before this change, processLine always uses the first character
as the start of the line. This cause whitespaces to matter at the
beginning of the line whereas it does not matter anywhere else.

This commit trims leading white spaces of the string line so that
subsequent operations can be performed on the string without explicitly
tracking starting and ending indices of the string.
2022-03-24 21:33:29 -04:00
Eelco Dolstra
540d7e33d8 Retry substitution after an incomplete closure only once
This avoids an infinite loop in the final test in
tests/binary-cache.sh. I think this was only not triggered previously
by accident (because we were clearing wantedOutputs in between).
2022-03-24 23:25:12 +01:00
Eelco Dolstra
fe5509df9a Only return wanted outputs 2022-03-24 23:24:48 +01:00
Eelco Dolstra
09796c0263 Random cleanup 2022-03-24 23:24:10 +01:00
Eelco Dolstra
175c78591b Random cleanup 2022-03-24 23:09:43 +01:00
Maximilian Bosch
c85467a1b6
Revert "TarArchive: Small refactoring"
This reverts commit 50a35860ee.

With this change Nix fails to open bzip2 logfiles that were created from
builds with no stdout/stderr.
2022-03-24 22:30:46 +01:00
Eelco Dolstra
f902f3c2cb Add experimental feature 'fetch-closure' 2022-03-24 21:33:33 +01:00
Eelco Dolstra
e5f7029ba4 nix store make-content-addressed: Support --from / --to 2022-03-24 21:33:33 +01:00
Eelco Dolstra
98658ae9d2 Document fetchClosure 2022-03-24 21:33:33 +01:00
Eelco Dolstra
28186b7044 Add a test for fetchClosure and 'nix store make-content-addressed' 2022-03-24 21:33:33 +01:00
Eelco Dolstra
4120930ac1 fetchClosure: Only allow some "safe" store types 2022-03-24 21:33:33 +01:00
Eelco Dolstra
7ffda0af6e fetchClosure: Skip makeContentAddressed() if toPath is already valid 2022-03-24 21:33:33 +01:00
Eelco Dolstra
5acaf13d35 Rename 'nix store make-content-addressable' to 'nix store make-content-addressed' 2022-03-24 21:33:33 +01:00
Eelco Dolstra
f18607549c Fix makeContentAddressed() on self-references
LocalStore::addToStore() since
79ae9e4558 expects a regular NAR hash,
rather than a NAR hash modulo self-references. Fixes #6300.

Also, makeContentAddressed() now rewrites the entire closure (so 'nix
store make-content-addressable' no longer needs '-r'). See #6301.
2022-03-24 21:33:33 +01:00
Eelco Dolstra
545c2d0d8c fetchClosure: Allow a path to be rewritten to CA on the fly
The advantage is that the resulting closure doesn't need to be signed,
so you don't need to configure any binary cache keys on the client.
2022-03-24 21:33:33 +01:00
Eelco Dolstra
7f6fe8ca1d Rename 2022-03-24 21:33:33 +01:00
Eelco Dolstra
41659418cf fetchClosure: Require a CA path in pure mode 2022-03-24 21:33:33 +01:00
Eelco Dolstra
f4bafc412f Add builtins.fetchClosure
This allows closures to be imported at evaluation time, without
requiring the user to configure substituters. E.g.

  builtins.fetchClosure {
    storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2;
    from = "https://cache.ngi0.nixos.org";
  }
2022-03-24 21:33:33 +01:00
Eelco Dolstra
d67fe90375
Merge pull request #6305 from flox/genericClosure_doc
docs: genericClosure
2022-03-24 14:02:58 +01:00
Tom Bereknyei
0736f3651d docs: genericClosure 2022-03-24 08:03:59 -04:00
Rok Garbas
4546a007a4
Fix flake profile use of originalUrl vs. originalUri
Fixes #5872
2022-03-24 12:28:38 +01:00
Sergei Trofimovich
9174d884d7 lexer: add error location to lexer errors
Before the change lexter errors did not report the location:

    $ nix build -f. mc
    error: path has a trailing slash
    (use '--show-trace' to show detailed location information)

Note that it's not clear what file generates the error.

After the change location is reported:

    $ src/nix/nix --extra-experimental-features nix-command build -f ~/nm mc
    error: path has a trailing slash

           at .../pkgs/development/libraries/glib/default.nix:54:18:

               53|   };
               54|   src = /tmp/foo/;
                 |                  ^
               55|
    (use '--show-trace' to show detailed location information)

Here we see both problematic file and the string itself.
2022-03-24 08:16:14 +00:00
Hideaki Kawai
3b776cb0a7 nix edit: support kakoune 2022-03-22 23:18:02 +09:00
Eelco Dolstra
a0259a21a4 Don't hide repeated values while generating manifest.nix
Fixes #6243.
2022-03-22 13:18:56 +01:00
Eelco Dolstra
732296ddc0 printValue(): <REPEAT> -> «repeated»
This ensures that it doesn't get parsed as a valid Nix expression.
2022-03-22 13:00:27 +01:00
Eelco Dolstra
0b42afe027 buildProfile(): Ignore manifest.{nix,json}
If a package installs a file named manifest.json, it caused nix-env to
consider the profile a new-style profile created by 'nix
profile'. Fixes #6032.
2022-03-22 10:48:02 +01:00
Eelco Dolstra
e4ff430866
Merge pull request #6237 from obsidiansystems/store-path-string-context
Decode string context straight to using StorePaths
2022-03-22 10:29:46 +01:00
Gabriel Fontes
9720797f69
Don't partial match sourcehut refs 2022-03-19 11:04:04 -03:00
Gabriel Fontes
345a8ee0cb
Fix sourcehut tag ref resolving 2022-03-19 10:56:13 -03:00
John Ericson
4d6a3806d2 Decode string context straight to using StorePaths
I gather decoding happens on demand, so I hope don't think this should
have any perf implications one way or the other.
2022-03-18 15:36:11 +00:00
John Ericson
a544ed7684 Generalize DerivationType in preparation for impure derivations 2022-03-18 14:59:56 +00:00
John Ericson
049fae155a Avoid some pointless copying of drvs 2022-03-18 14:59:56 +00:00
John Ericson
8496be7def Use Deferred when building an input-addressed drv
Easier than using dummy path with input addressed.
2022-03-18 14:59:56 +00:00
John Ericson
197feed51d Clean up DerivationOutput, and headers
1. `DerivationOutput` now as the `std::variant` as a base class. And the
   variants are given hierarchical names under `DerivationOutput`.

   In 8e0d0689be @matthewbauer and I
   didn't know a better idiom, and so we made it a field. But this sort
   of "newtype" is anoying for literals downstream.

   Since then we leaned the base class, inherit the constructors trick,
   e.g. used in `DerivedPath`. Switching to use that makes this more
   ergonomic, and consistent.

2. `store-api.hh` and `derivations.hh` are now independent.

   In bcde5456cc I swapped the dependency,
   but I now know it is better to just keep on using incomplete types as
   much as possible for faster compilation and good separation of
   concerns.
2022-03-17 22:35:53 +00:00
Eelco Dolstra
6afc361798
Merge pull request #6277 from thufschmitt/ca/nix-build-dry-run
Fix `nix build --dry-run` with CA derivations
2022-03-17 22:23:15 +01:00
Eelco Dolstra
d8f69b9a4c Merge branch 'avoid-quadratic-gc' of https://github.com/trofi/nix 2022-03-17 21:35:42 +01:00
Sergei Trofimovich
d58453f72e gc: don't visit implicit referrers on garbage collection
Before the change garbage collector was not considering
`.drv` and outputs as alive even if configuration says otherwise.
As a result `nix store gc --dry-run` could visit (and parse)
`.drv` files multiple times (worst case it's quadratic).

It happens because `alive` set was populating only runtime closure
without regard for actual configuration. The change fixes it.

Benchmark: my system has about 139MB, 40K `.drv` files.

Performance before the change:

    $ time nix store gc --dry-run
    real    4m22,148s

Performance after the change:

    $ time nix store gc --dry-run
    real    0m14,178s
2022-03-17 18:47:29 +00:00
Théophane Hufschmitt
7117053457
Merge pull request #6270 from Artturin/stdineval
nix: allow using --file - to read from stdin
2022-03-17 14:35:01 +01:00
Théophane Hufschmitt
3fc4c612fb Fix nix build --dry-run with CA derivations
Don’t try and assume that we know the output paths when we’ve just built
with `--dry-run`. Instead make `--dry-run` follow a different code path
that won’t assume the knowledge of the output paths at all.

Fix #6275
2022-03-17 11:55:19 +01:00
Théophane Hufschmitt
a0b517de57
Merge pull request #6242 from ncfavier/print-output-names
nix-env: always print output names in JSON and XML
2022-03-17 10:55:22 +01:00
Naïm Favier
8dcecc0738
nix-env: print a final newline after JSON 2022-03-16 21:26:19 +01:00
Naïm Favier
5736661922
nix-env: always print output names in JSON and XML
The current `--out-path` flag has two disadvantages when one is only
concerned with querying the names of outputs:
- it requires evaluating every output's `outPath`, which takes
  significantly more resources and runs into more failures
- it destroys the information of the order of outputs so we can't tell
  which one is the main output

This patch makes the output names always present (replacing paths with
`null` in JSON if `--out-path` isn't given), and adds an `outputName`
field.
2022-03-16 21:26:19 +01:00
Artturin
a5c969db49 nix: allow using --file - to read from stdin 2022-03-16 21:01:51 +02:00
Ben Burdette
88a54108eb formatting 2022-03-16 12:09:47 -06:00
Eelco Dolstra
a53c1dc96d
Merge pull request #6245 from trofi/fix-nix-store-gc-limit-auto-optimise-store
nix store gc: account for auto-optimised store
2022-03-16 09:19:57 +01:00
Théophane Hufschmitt
516a7ac4de
Merge pull request #6229 from obsidiansystems/refactor-hash-modulo
Overhaul derivation hash modulo somewhat
2022-03-15 21:23:44 +01:00
Maximilian Bosch
244baff2c7
libfetchers: remove obsolete filesystem #include 2022-03-15 12:32:11 +01:00
Maximilian Bosch
34e20c164c
libfetchers/path: set lastModified to path's mtime
When importing e.g. a local `nixpkgs` in a flake to test a change like

    {
      inputs.nixpkgs.url = path:/home/ma27/Projects/nixpkgs;
      outputs = /* ... */
    }

then the input is missing a `lastModified`-field that's e.g. used in
`nixpkgs.lib.nixosSystem`. Due to the missing `lastMoified`-field, the
mtime is set to 19700101:

    result -> /nix/store/b7dg1lmmsill2rsgyv2w7b6cnmixkvc1-nixos-system-nixos-22.05.19700101.dirty

With this change, the `path`-fetcher now sets a `lastModified` attribute
to the `mtime` just like it's the case in the `tarball`-fetcher already.
When building NixOS systems with `nixpkgs` being a `path`-input and this
patch, the output-path now looks like this:

    result -> /nix/store/ld2qf9c1s98dxmiwcaq5vn9k5ylzrm1s-nixos-system-nixos-22.05.20220217.dirty
2022-03-15 12:32:11 +01:00
Ben Burdette
3dfab6e534 have only one debuggerHook declaration 2022-03-14 11:58:11 -06:00
Ben Burdette
eaecaaa00b more debug_throw coverage of EvalErrors 2022-03-14 11:39:53 -06:00
Eelco Dolstra
a61809722f
Merge pull request #6244 from Artturin/nixenvbettermessage
nix-env: Add a suggestion for when there's a name collision in channels
2022-03-14 14:17:50 +01:00
Théophane Hufschmitt
0e86ebf461
Merge pull request #6220 from obsidiansystems/log-store
Factor out a `LogStore` interface
2022-03-14 10:51:01 +01:00
Sergei Trofimovich
6b1872312f nix store gc: account for auto-optimised store
Before the change on a system with `auto-optimise-store = true`:

    $ nix store gc --verbose --max 1

deleted all the paths instead of one path (we requested 1 byte limit).

It happens because every file in `auto-optimise-store = true` has at
least 2 links: file itself and a link in /nix/store/.links/ directory.

The change conservatively assumes that any file that has one (as before)
or two links (assume auto-potimise mode) will free space.

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
2022-03-13 07:24:48 +00:00
Artturin
cb1a76112e nix-env: Add a suggestion for when there's a name collision in channels
help new users find a solution to their problem

./result/bin/nix-env -qa hello
warning: name collision in input Nix expressions, skipping '/home/artturin/.nix-defexpr/channels_root/master'
suggestion: remove 'master' from either the root channels or the user channels
hello-2.12
hello-2.12
2022-03-13 00:24:46 +02:00
John Ericson
91adfb8894 Create some type aliases for string Contexts 2022-03-11 22:30:10 +00:00
John Ericson
0948b8e94d Reduce variants for derivation hash modulo
This changes was taken from dynamic derivation (#4628). It` somewhat
undoes the refactors I first did for floating CA derivations, as the
benefit of hindsight + requirements of dynamic derivations made me
reconsider some things.

They aren't to consequential, but I figured they might be good to land
first, before the more profound changes @thufschmitt has in the works.
2022-03-11 21:20:37 +00:00
John Ericson
015d2ad507 Desugar StorePathWithOutputs in nix-build implementation
`DerivedPath` has replaced `StorePathWithOutputs` internally, so
shrinking the usage of `StorePathWithOutputs` to just the boundary is
good.
2022-03-11 20:50:52 +00:00
Kevin Amado
2191dab657
nix-fmt: add command 2022-03-11 10:00:19 -05:00
John Ericson
a03b1fd7f6 Deduplicate the Store downcasting with a template 2022-03-11 13:32:33 +00:00
John Ericson
678d1c2aa0 Factor out a LogStore interface
Continue progress on #5729.

Just as I hoped, this uncovered an issue: the daemon protocol is missing
a way to query build logs. This doesn't effect `unix://`, but does
effect `ssh://`. A FIXME is left for this, so we come back to it later.
2022-03-11 13:32:16 +00:00
John Ericson
89effe9d4a GcStore::resolve should print the URI 2022-03-11 13:27:38 +00:00
John Ericson
073e134de6 Rename requireGcStore to GcStore::require
I should have done this to begin with. This will be nicer once more
Store sub-interfaces exist too, to illustrate the pattern.
2022-03-11 13:27:38 +00:00
Eelco Dolstra
aee56e0f89 Merge remote-tracking branch 'origin/eval-suggestions' 2022-03-11 12:02:26 +01:00
Eelco Dolstra
31a392dfe2 Merge pull request #5865 from pennae/memory-friendliness
be more memory friendly
2022-03-11 11:52:39 +01:00
Eelco Dolstra
167766b65c Style 2022-03-11 11:19:21 +01:00
Théophane Hufschmitt
7ebd6f1093
Merge pull request #5758 from mschwaig/fix-git-workspace-dirty-detection
git fetcher: fix detection of dirty git workspaces
2022-03-11 09:35:01 +01:00
Eelco Dolstra
4d98143914 BuildResult: Remove unused drvPath field 2022-03-09 20:31:50 +01:00
Eelco Dolstra
761242afa0 BuildResult: Use DerivedPath 2022-03-09 12:25:35 +01:00
pennae
4d629c4f7a add HAVE_BOEHMGC guards to batched allocation functions 2022-03-09 00:18:50 +01:00
pennae
8e2eaaaf69 make Finally more local
no need for function<> with c++17 deduction. this saves allocations and virtual
calls, but has the same semantics otherwise. not going through function has the
side effect of giving compilers more insight into the cleanup code, so we need a
few local warning disables.
2022-03-09 00:16:50 +01:00
pennae
47baa9d43c make Pos smaller
reduces peak hep memory use on eval of our test system from 264.4MB to 242.3MB,
possibly also a slight performance boost.

theoretically memory use could be cut down by another eight bytes per Pos on
average by turning it into a tuple containing an index into a global base
position table with row and column offsets, but that doesn't seem worth the
effort at this point.
2022-03-08 23:30:18 +01:00
pennae
c96460f352 force-inline a few much-used functions
these functions are called a whole lot, and they're all comparatively small.
always inlining them gives ~0.7% performance boost on eval.

before:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.935 s ±  0.052 s    [User: 5.852 s, System: 0.853 s]
    Range (min … max):    6.808 s …  7.026 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     329.8 ms ±   2.7 ms    [User: 299.0 ms, System: 30.8 ms]
    Range (min … max):   326.6 ms … 336.5 ms    20 runs

  Benchmark 3: nix flakes eval --raw --impure --file expr.nix
    Time (mean ± σ):      2.655 s ±  0.038 s    [User: 2.364 s, System: 0.220 s]
    Range (min … max):    2.574 s …  2.737 s    20 runs

after:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.912 s ±  0.036 s    [User: 5.823 s, System: 0.856 s]
    Range (min … max):    6.849 s …  6.980 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     325.1 ms ±   2.5 ms    [User: 293.2 ms, System: 31.8 ms]
    Range (min … max):   322.2 ms … 332.8 ms    20 runs

  Benchmark 3: nix flakes eval --raw --impure --file expr.nix
    Time (mean ± σ):      2.636 s ±  0.024 s    [User: 2.352 s, System: 0.226 s]
    Range (min … max):    2.574 s …  2.681 s    20 runs
2022-03-08 23:30:18 +01:00
pennae
60ed4e908a cache singleton Envs just like Values
vast majority of envs is this size.

before:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.946 s ±  0.041 s    [User: 5.875 s, System: 0.835 s]
    Range (min … max):    6.834 s …  7.005 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     330.3 ms ±   2.5 ms    [User: 299.2 ms, System: 30.9 ms]
    Range (min … max):   327.5 ms … 337.7 ms    20 runs

  Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.671 s ±  0.035 s    [User: 2.370 s, System: 0.232 s]
    Range (min … max):    2.597 s …  2.749 s    20 runs

after:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.935 s ±  0.052 s    [User: 5.852 s, System: 0.853 s]
    Range (min … max):    6.808 s …  7.026 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     329.8 ms ±   2.7 ms    [User: 299.0 ms, System: 30.8 ms]
    Range (min … max):   326.6 ms … 336.5 ms    20 runs

  Benchmark 3: nix flakes eval --raw --impure --file expr.nix
    Time (mean ± σ):      2.655 s ±  0.038 s    [User: 2.364 s, System: 0.220 s]
    Range (min … max):    2.574 s …  2.737 s    20 runs
2022-03-08 23:30:18 +01:00
pennae
4b2b0d3a55 remove GC_PTR_STORE_AND_DIRTY
turns out it's only necessary for MANUAL_VDB, which nix doesn't use. omitting
them gives a slight performance improvement on eval.

before:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.988 s ±  0.061 s    [User: 5.935 s, System: 0.845 s]
    Range (min … max):    6.865 s …  7.075 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     332.6 ms ±   3.9 ms    [User: 299.6 ms, System: 32.9 ms]
    Range (min … max):   328.1 ms … 339.1 ms    20 runs

  Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.681 s ±  0.049 s    [User: 2.382 s, System: 0.228 s]
    Range (min … max):    2.607 s …  2.776 s    20 runs

after:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.946 s ±  0.041 s    [User: 5.875 s, System: 0.835 s]
    Range (min … max):    6.834 s …  7.005 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     330.3 ms ±   2.5 ms    [User: 299.2 ms, System: 30.9 ms]
    Range (min … max):   327.5 ms … 337.7 ms    20 runs

  Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.671 s ±  0.035 s    [User: 2.370 s, System: 0.232 s]
    Range (min … max):    2.597 s …  2.749 s    20 runs
2022-03-08 23:30:18 +01:00
Eelco Dolstra
a4604f1928 Add Store::buildPathsWithResults()
This function is like buildPaths(), except that it returns a vector of
BuildResults containing the exact statuses and output paths of each
derivation / substitution. This is convenient for functions like
Installable::build(), because they then don't need to do another
series of calls to get the outputs of CA derivations. It's also a
precondition to impure derivations, where we *can't* query the output
of those derivations since they're not stored in the Nix database.

Note that PathSubstitutionGoal can now also return a BuildStatus.
2022-03-08 19:56:34 +01:00
regnat
f6078e474d Also display some suggestions for invalid formal arguments
```console
$ nix eval --expr '({ foo ? 1 }: foo) { fob = 2; }'
error: anonymous function at (string):1:2 called with unexpected argument 'fob'

       at «string»:1:1:

            1| ({ foo ? 1 }: foo) { fob = 2; }
             | ^
       Did you mean foo?
```

Not that because Nix will first check for _missing_ arguments before
checking for extra arguments, `({ foo }: foo) { fob = 1; }` will
complain about the missing `foo` argument (rather than extra `fob`) and
so won’t display a suggestion.
2022-03-08 16:40:22 +01:00
regnat
33b7514035 Try and make the darwin build happy 2022-03-08 16:07:17 +01:00
regnat
0c6e46e349 Add some suggestions to the evaluator
Make the evaluator show some suggestions when trying to access an
invalid field from an attrset.

```console
$ nix eval --expr '{ foo = 1; }.foa'
error: attribute 'foa' missing

       at «string»:1:1:

            1| { foo = 1; }.foa
             | ^
       Did you mean foo?
```
2022-03-08 06:21:45 +01:00
Eelco Dolstra
504e3b2a8f Style 2022-03-07 20:07:43 +01:00
Eelco Dolstra
fa614fac7f Merge branch 'GuillaumeDesforges/issue6192' of https://github.com/GuillaumeDesforges/nix 2022-03-07 19:58:06 +01:00
Eelco Dolstra
30ddd37873 Merge branch 'cli-suggestions' of https://github.com/thufschmitt/nix 2022-03-07 19:47:45 +01:00
Guillaume Desforges
436c6e900f Explicit error in flake init/new when not in store 2022-03-07 17:57:52 +01:00
regnat
dcf4f77fac Merge or-suggestions.hh into suggestions.hh
No real need for keeping a separate header for such a simple class.

This requires changing a bit `OrSuggestions<T>::operator*` to not throw
an `Error` to prevent a cyclic dependency. But since this error is only
thrown on programmer error, we can replace the whole method by a direct
call to `std::get` which will raise its own assertion if needs be.
2022-03-07 17:49:02 +01:00
Robert Hensing
da260f579d dupStringWithLen -> makeImmutableString
Refactor the `size == 0` logic into a new helper function that
replaces dupStringWithLen.
The name had to change, because unlike a `dup`-function, it does
not always allocate a new string.
2022-03-07 16:09:12 +01:00
Robert Hensing
bbf55383e7 Value::mkPath: Avoid potential crash from null string_view 2022-03-07 16:09:12 +01:00
Robert Hensing
1b978596b5 Value::mkString: Avoid crash from null string_view 2022-03-07 16:09:12 +01:00
Naïm Favier
da7d8daa77
Add shell completion for --override-flake
Requires moving the MixEvalArgs class from libexpr to libcmd because
that's where completeFlakeRef is.
2022-03-07 12:01:54 +01:00
Naïm Favier
55c6906701
Perform tilde expansion when completing flake fragments
Allows completing `nix build ~/flake#<Tab>`.
We can implement expansion for `~user` later if needed.
Not using wordexp(3) since that expands way too much.
2022-03-07 12:01:54 +01:00
Naïm Favier
5461ff532d
Make completeDir follow symlinks
Allows completing `nix why-depends /run/cur<Tab>` to /run/current-system
2022-03-07 12:01:54 +01:00
Naïm Favier
a6d7cd4183
Ensure the completion marker is not processed beyond completion
I was surprised to see an error mentioning ___COMPLETE___ when trying to
complete a flag argument that had no completer implemented
2022-03-07 12:01:54 +01:00
Naïm Favier
5f06a91bf7
Fix completion of nested attributes in completeInstallable
Without this, completing `nix eval -f file.nix foo.<Tab>` suggests `bar`
instead of `foo.bar`, which messes up the command
2022-03-07 12:01:54 +01:00
Naïm Favier
7ddcb39206
Add shell completion for --override-input 2022-03-07 12:01:54 +01:00
Naïm Favier
7f5cf87d56
Accept and discard fragments in getFlakeRefForCompletion
Otherwise trying to complete `nix build foo#bar --update-input <Tab>`
fails with "unexpected fragment"
2022-03-07 12:01:54 +01:00
regnat
313bbc07a8 Implement operator<< for Suggestions
That way there’s no need to explicitely convert it to a string when
printing it
2022-03-07 10:09:10 +01:00
regnat
fd45d85b41 Move OrSuggestions to its own header
Prevents a recursive inclusion
2022-03-07 10:09:10 +01:00
regnat
98e361ad4c Also display suggestions for the commands using the eval cache
Make `nix build .#nix-armv8l-linux` work for example
2022-03-07 10:09:10 +01:00
regnat
2405bbbb5e Add some tests for the suggestions 2022-03-07 10:09:10 +01:00
regnat
c0792b1546 Implement a suggestions mechanism
Each `Error` class now includes a set of suggestions, and these are printed by
the top-level handler.
2022-03-07 10:09:09 +01:00
Théophane Hufschmitt
a2ace54fe4
Merge pull request #6029 from Ma27/nix-log-ssh-ng
ssh-ng: also store build logs to make them accessible by `nix log`
2022-03-07 09:51:40 +01:00
Nicholas Sielicki
314852a10e
Point to new github oauth docs url
Previous URL was 404'ing.
2022-03-06 17:01:14 -06:00
John Ericson
6636202356 Factor out a GcStore interface
Starts progress on #5729.

The idea is that we should not have these default methods throwing
"unimplemented". This is a small step in that direction.

I kept `addTempRoot` because it is a no-op, rather than failure. Also,
as a practical matter, it is called all over the place, while doing
other tasks, so the downcasting would be annoying.

Maybe in the future I could move the "real" `addTempRoot` to `GcStore`,
and the existing usecases use a `tryAddTempRoot` wrapper to downcast or
do nothing, but I wasn't sure whether that was a good idea so with a
bias to less churn I didn't do it yet.
2022-03-03 19:01:25 +00:00
Eelco Dolstra
e9c04c3351 Be more aggressive in hiding repeated values
We now memoize on Bindings / list element vectors rather than Values,
so that e.g. two Values that point to the same Bindings will be
printed only once.
2022-03-03 13:33:34 +01:00