Commit graph

7024 commits

Author SHA1 Message Date
Tom Bereknyei
30496af598 Adds an accept-flake-config flag 2021-11-12 09:50:07 -05:00
Eelco Dolstra
bceda30498 Typo 2021-11-12 13:41:15 +01:00
Eelco Dolstra
44aed58538
Merge pull request #5540 from samueldr/fix/key-subcommand
nix key: Fix error message and don't require flakes
2021-11-12 10:38:00 +01:00
Samuel Dionne-Riel
30e5c5c55f nix registry: Mark experimental
This is part of the flakes feature. Mark it as such.
2021-11-12 03:18:31 -05:00
Samuel Dionne-Riel
c1dea92dd6 nix key: Fix error message and don't require flakes 2021-11-11 21:31:26 -05:00
Alexander Bantyev
07bffe7998
Flakes: refetch the input when a follows disappears
When an input follows disappears, we can't just reuse the old lock
file entries since we may be missing some required ones. Refetch the
input when this happens.

Closes https://github.com/NixOS/nix/issues/5289
2021-11-11 12:05:18 +03:00
Eelco Dolstra
a218cfd6c7
Merge pull request #5528 from matthewbauer/recognize-nixosModule
Recognize singular "nixosModule" in nix flake show
2021-11-10 12:17:11 +01:00
Ben Burdette
7e2a3db4eb cleanup 2021-11-09 13:14:49 -07:00
Ben Burdette
885f819922 remove dead code 2021-11-09 11:20:14 -07:00
Matthew Bauer
ceeecf2f9e Recognize singular "nixosModule" in nix flake show
This makes nixosModule appears as a "NixOS Module" like nixosModules
does.
2021-11-09 11:14:15 -06:00
Eelco Dolstra
67179472df
Merge pull request #5494 from tweag/balsoft/allow-references-in-addPath
Allow references in addPath
2021-11-09 15:57:39 +01:00
Alexander Bantyev
0b005bc9d6
addToStore, addToStoreFromDump: refactor: pass refs by const reference
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
2021-11-09 12:24:49 +03:00
Eelco Dolstra
ff2af4d64e Unshare mount namespace in main()
Doing it as a side-effect of calling LocalStore::makeStoreWritable()
is very ugly.

Also, make sure that stopping the progress bar joins the update
thread, otherwise that thread should be unshared as well.
2021-11-08 22:00:45 +01:00
Eelco Dolstra
7a71621b7c Merge branch 'fix-writable-shell' of https://github.com/yorickvP/nix 2021-11-08 21:12:51 +01:00
Eelco Dolstra
6c2af1f201
Merge pull request #5434 from timothyklim/git-url-submodules
Support building flakes from a Git repo url with submodules
2021-11-08 14:13:42 +01:00
Eelco Dolstra
736d6ab721
Merge pull request #5504 from NixOS/flake-options-and-daemon
Make the flake options work when using the daemon
2021-11-08 13:54:55 +01:00
Andreas Rammhold
8e7359db64
Remove unused "<let-body>" symbol
The requirement for the symbol has been removed since at least 7d47498.
2021-11-07 18:26:43 +01:00
Alexander Bantyev
9d4dcff37a
addPath: allow paths with references
Since 4806f2f6b0, we can't have paths with
references passed to builtins.{path,filterSource}. This prevents many cases
of those functions called on IFD outputs from working. Resolve this by
passing the references found in the original path to the added path.
2021-11-05 22:41:30 +03:00
Théophane Hufschmitt
3d6ee223d6
Merge pull request #5476 from NixOS/ca/fix-build-in-nix-repl
Fix `nix repl`’s building of CA derivations
2021-11-05 17:35:18 +01:00
regnat
1f3c3a3785 Make the flake options work when using the daemon
When setting flake-local options (with the `nixConfig` field), forward
these options to the daemon in case we’re using one.

This is necessary in particular for options like `binary-caches` or
`post-build-hook` to make sense.

Fix <343239fc8a (r44356843)>
2021-11-05 16:19:16 +01:00
regnat
1a4c9ba50b Fix nix repl’s building of CA derivations
When running a `:b` command in the repl, after building the derivations
query the store for its outputs rather than just assuming that they are
known in the derivation itself (which isn’t true for CA derivations)

Fix #5328
2021-11-05 11:04:14 +01:00
Eelco Dolstra
40925337a9 Remove maxPrimOpArity 2021-11-04 15:04:07 +01:00
Eelco Dolstra
acd6bddec7 Fix derivation primop 2021-11-04 15:04:00 +01:00
Eelco Dolstra
cbfbf71e08 Use callFunction() with an array for some calls with arity > 1 2021-11-04 15:03:57 +01:00
Eelco Dolstra
bcf4780006 Add level / displacement types 2021-11-04 15:03:45 +01:00
Eelco Dolstra
81e7c40264 Optimize primop calls
We now parse function applications as a vector of arguments rather
than as a chain of binary applications, e.g. 'substring 1 2 "foo"' is
parsed as

  ExprCall { .fun = <substring>, .args = [ <1>, <2>, <"foo"> ] }

rather than

  ExprApp (ExprApp (ExprApp <substring> <1>) <2>) <"foo">

This allows primops to be called immediately (if enough arguments are
supplied) without having to allocate intermediate tPrimOpApp values.

On

  $ nix-instantiate --dry-run '<nixpkgs/nixos/release-combined.nix>' -A nixos.tests.simple.x86_64-linux

this gives a substantial performance improvement:

  user CPU time:      median =      0.9209  mean =      0.9218  stddev =      0.0073  min =      0.9086  max =      0.9340  [rejected, p=0.00000, Δ=-0.21433±0.00677]
  elapsed time:       median =      1.0585  mean =      1.0584  stddev =      0.0024  min =      1.0523  max =      1.0623  [rejected, p=0.00000, Δ=-0.20594±0.00236]

because it reduces the number of tPrimOpApp allocations from 551990 to
42534 (i.e. only small minority of primop calls are partially
applied) which in turn reduces time spent in the garbage collector.
2021-11-04 15:03:40 +01:00
Eelco Dolstra
ab35cbd675 StaticEnv: Use std::vector instead of std::map 2021-11-04 15:03:34 +01:00
Eelco Dolstra
c4bd6a15c2 Add helper function to check whether a function arg is 'X' or '_X'
Also allow '_'.
2021-11-04 14:52:35 +01:00
Eelco Dolstra
1e7c796e66
Merge pull request #5475 from doronbehar/SQLiteWAL-vfs
libstore: Use unix-dotfile vfs if useSQLiteWAL is false
2021-11-03 21:20:27 +01:00
Christopher League
3f070cc417 In checkOverlay, accept underscored names for final/prev args.
Resolves #4416.
2021-11-03 09:25:27 -04:00
Doron Behar
14fcf17277 libstore: Use unix-dotfile vfs if useSQLiteWAL is false 2021-11-03 14:19:11 +02:00
regnat
f4c869977c Make the DrvOutputSubstitutionGoal more async 2021-11-03 06:51:34 +01:00
regnat
fbc70034b3 Make the realisation fetching from binary caches async
That way we can fetch several realisations from the same cache in
parallel
2021-11-03 06:51:34 +01:00
regnat
96670ed216 Expose an async interface for queryRealisation
Doesn’t change much so far because everything is still using it
synchronously, but should allow the binary cache to fetch stuff in
parallel
2021-11-03 06:51:34 +01:00
regnat
5b2aa61f1b Don’t require ca-derivations when __contentAddressed = false
If we explicitely opt-out of it, there’s no need to require the
experimental feature
2021-11-03 06:51:32 +01:00
Sebastian Ullrich
b459a3e856 git: extend cache dir lock over all mutating operations 2021-11-01 09:14:56 +01:00
Eelco Dolstra
e6795c4350 Style 2021-10-29 14:45:13 +02:00
Eelco Dolstra
6e30d9b69f Merge branch 'master' of https://github.com/alekswn/nix 2021-10-29 14:42:26 +02:00
Eelco Dolstra
33d04e8a8d Use nix::connect() to connect to the garbage collector 2021-10-28 22:51:11 +02:00
Eelco Dolstra
22c35ea5b8 Remove unused variable 2021-10-28 22:51:11 +02:00
Eelco Dolstra
a7d4f3411e Merge remote-tracking branch 'origin/master' into non-blocking-gc 2021-10-28 14:56:55 +02:00
Shay Bergmann
a50c027ece
toJSON: improve pos accuracy, add trace 2021-10-27 19:48:48 +00:00
Shay Bergmann
465a167c43
nix-instantiate: pass pos in the --eval --json code path 2021-10-27 19:01:32 +00:00
Eelco Dolstra
5a160171d0 Remove redundant 'warning:' 2021-10-27 18:14:12 +02:00
Eelco Dolstra
1254e8753c build-remote: Implicitly add the 'builtin' system type to all machines
This makes 'nix-env -i --max-jobs 0' work with remote builders.
2021-10-27 14:25:13 +02:00
Eelco Dolstra
f2280749b1 If max-jobs == 0, do preferLocalBuild on remote builders 2021-10-27 14:21:31 +02:00
Eelco Dolstra
6e684d1b87 daemon: Accept 'repeat' setting from untrusted users
Fixes #5352.
2021-10-27 13:09:13 +02:00
Eelco Dolstra
13a7a24ba5 Style 2021-10-27 13:02:37 +02:00
Shay Bergmann
769de259f0
toJSON: pass pos in case of a list as well 2021-10-26 14:43:15 +00:00
Timothy
8919b81dad Support building flakes from a Git repo url with submodules query parameter 2021-10-26 20:02:37 +07:00
Eelco Dolstra
0d9e050ba7 parseExperimentalFeature(): Initialize atomically 2021-10-26 14:29:48 +02:00
Eelco Dolstra
9ce84c64c5 Tweak fetchTree docs 2021-10-26 14:21:24 +02:00
regnat
af99941279 Make experimental-features a proper type
Rather than having them plain strings scattered through the whole
codebase, create an enum containing all the known experimental features.

This means that
- Nix can now `warn` when an unkwown experimental feature is passed
  (making it much nicer to spot typos and spot deprecated features)
- It’s now easy to remove a feature altogether (once the feature isn’t
  experimental anymore or is dropped) by just removing the field for the
  enum and letting the compiler point us to all the now invalid usages
  of it.
2021-10-26 07:02:31 +02:00
Shay Bergmann
ba81e871b2
toJSON: report error position for fancier output
Given flake:

```nix
{ description = "nix json error provenance";
  inputs = {};
  outputs = { self }: {
    jsonFunction = _: "function";
    json = builtins.toJSON (_: "function");
  };
}

```
- Before:

```console
❯ nix eval --json .#jsonFunction
error: cannot convert a function to JSON
```

- After:

```console
❯ nix eval --json .#jsonFunction
error: cannot convert a function to JSON

       at /nix/store/b7imf1c2j4jnkg3ys7fsfbj02s5j0i4f-source/testflake/flake.nix:4:5:

            3|   outputs = { self }: {
            4|     jsonFunction = _: "function";
             |     ^
            5|     json = builtins.toJSON (_: "function");
```
2021-10-25 21:17:52 +00:00
Alexander Bantyev
b9234142f5
addToStore, addToStoreFromDump: add references argument
Allow to pass a set of references to be added as info to the added paths.
2021-10-23 21:30:51 +03:00
Ben Burdette
fb8377547b more code cleanup 2021-10-22 14:49:58 -06:00
Ben Burdette
71da988d47 more debug removal 2021-10-22 14:34:50 -06:00
Ben Burdette
e54f17eb46 remove more debug code 2021-10-22 14:27:04 -06:00
Ben Burdette
cbc2f0fe31 remove dead code 2021-10-22 14:02:47 -06:00
Ben Burdette
383ab600ee show expr on error 2021-10-22 13:41:04 -06:00
Kevin Amado
823dce945a
fetch: nicer infinite recursion errors
- This change applies to builtins.fetchurl and builtins.fetchTarball
- PoC: `let x = builtins.fetchurl x; in x`
- Before:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered
  ```
- After:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered

       at /data/github/kamadorueda/nix/test.nix:1:9:

            1| let x = builtins.fetchurl x; in x
             |         ^
  ```

Mentions: #3505
2021-10-17 12:54:53 -05:00
Alexey Novikov
e989c83b44 Add error reporting to machine spec paser
Currently machine specification (`/etc/nix/machine`) parser fails
with a vague exception if the file had incorrect format.
This commit adds verbose exceptions and unit-tests for the parser.
2021-10-17 12:45:56 +04:00
Alexey Novikov
64a3b045c1 Fix error detection in 'base64Decode()'
Fixed a bug in initialization of 'base64DecodeChars' variable.
Currently decoder do not fail on invalid Base64 strings.
Added test-case to verify the fix.

Also have made 'base64DecodeChars' to be computed at compile time.
And added a test case to encode/decode string with non-printable charactes.
2021-10-17 12:45:26 +04:00
Kevin Amado
e5a27a3b4e
fetchTree: add pos to EvalState::forceValue
- This way we improve error messages
  on infinite recursion
- Demo:
  ```nix
  let x = builtins.fetchTree {
    type = "git";
    inherit x;
  };
  in x
  ```
- Before:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered
  ```
- After:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered

       at /data/github/kamadorueda/nix/test.nix:3:10:

            2|   type = "git";
            3|   inherit x;
             |          ^
            4| };
  ```

Mentions: #3505
2021-10-15 19:25:19 -05:00
Kevin Amado
18e3d63341
fetchTree: add pos to EvalState::forceValue
- This way we improve error messages
  on infinite recursion
- Demo:
  ```nix
  let x = builtins.fetchTree x;
  in x
  ```
- Before:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered
  ```
- After:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered

       at /data/github/kamadorueda/nix/test.nix:1:9:

            1| let x = builtins.fetchTree x;
             |         ^
            2| in x
  ```

Mentions: #3505
2021-10-15 19:25:19 -05:00
Eelco Dolstra
10f9a8e77d Add a test for the non-blocking GC 2021-10-15 16:52:37 +02:00
Eelco Dolstra
ac54c6faa6 Fix main GC thread exiting 2021-10-15 16:36:48 +02:00
Yorick van Pelt
fcb8af550f
Restore parent mount namespace in restoreProcessContext
This ensures any started processes can't write to /nix/store (except
during builds). This partially reverts 01d07b1e, which happened because
of #2646.

The problem was only happening after nix downloads anything, causing
me to suspect the download thread. The problem turns out to be:
"A  process  can't  join a new mount namespace if it is sharing
filesystem-related attributes with another process", in this case this
process is the curl thread.

Ideally, we might kill it before spawning the shell process, but it's
inside a static variable in the getFileTransfer() function. So
instead, stop it from sharing FS state using unshare(). A strategy
such as the one from #5057 (single-threaded chroot helper binary) is
also very much on the table.

Fixes #4337.
2021-10-15 16:25:49 +02:00
Eelco Dolstra
e0936ae38f Fix crash when a GC client disconnects
The client thread can't just delete its own thread object from
connections, it has to detach it.
2021-10-15 16:12:21 +02:00
Eelco Dolstra
304180d0de Memoize queryReferrers() 2021-10-15 12:20:23 +02:00
Eelco Dolstra
c0951299b3
Merge pull request #5383 from kvtb/patch-7
fix build with gcc11
2021-10-15 09:03:12 +02:00
Eelco Dolstra
c574ab3907
Merge pull request #5388 from yvt/fix-oahd-path
Add another path where a Rosetta 2 daemon plist file is possibly located
2021-10-15 09:02:16 +02:00
Kevin Amado
1bdeef8395
add pos to EvalState::forceValue
- This way we improve error messages
  on infinite recursion
- Demo:
  ```nix
  let
    x = builtins.fetchMercurial x;
  in
  x
  ```
- Before:
  ```bash
  $ nix-instantiate --show-trace --strict
  error: infinite recursion encountered
  ```
- After:
  ```bash
  nix-instantiate --show-trace --strict
  error: infinite recursion encountered

       at /data/github/kamadorueda/test/default.nix:2:7:

            1| let
            2|   x = builtins.fetchMercurial x;
             |       ^
            3| in
  ```

Mentions: #3505
2021-10-14 23:23:05 -05:00
yvt
a9d9e55551 Add another path where a Rosetta 2 configuration file is possibly located 2021-10-15 09:48:15 +09:00
Eelco Dolstra
17e6ebcc90 Speed up GC by marking entire closures as live 2021-10-14 14:13:57 +02:00
Eelco Dolstra
0154fa30cf Remove GCState 2021-10-14 13:52:49 +02:00
Eelco Dolstra
0317ffdad3 Move deleteFromStore() 2021-10-14 13:34:48 +02:00
Eelco Dolstra
0be8cc1466 pathInfoCache: Use the entire base name as the cache key
This fixes a bug in the garbage collector where if a path
/nix/store/abcd-foo is valid, but we do a
isValidPath("/nix/store/abcd-foo.lock") first, then a negative entry
for /nix/store/abcd is added to pathInfoCache, so /nix/store/abcd-foo
is subsequently considered invalid and deleted.
2021-10-14 13:28:22 +02:00
Eelco Dolstra
eab934cb2a Make the canReachRoots() traversal non-recursive 2021-10-14 12:34:32 +02:00
Eelco Dolstra
09b14ea97a Cleanup 2021-10-14 10:04:13 +02:00
kvtb
eae29b0385
fix build with gcc11 2021-10-13 18:03:33 +00:00
Eelco Dolstra
35c98a59c5 Fix GC when there are cycles in the referrers graph
(where "referrers" includes the reverse of derivation outputs and
derivers). Now we do a full traversal to look if we can reach any
root. If not, all paths reached can be deleted.
2021-10-13 12:12:44 +02:00
Eelco Dolstra
1785ba2980 Simplify 2021-10-13 12:12:44 +02:00
Eelco Dolstra
dced45f146 strcpy -> memcpy
Co-authored-by: Jörg Thalheim <Mic92@users.noreply.github.com>
2021-10-13 12:12:44 +02:00
Eelco Dolstra
262520fcfe Use a thread per connection 2021-10-13 12:12:44 +02:00
Eelco Dolstra
ff453b06f9 Fix auto-gc 2021-10-13 12:12:44 +02:00
Eelco Dolstra
8614cf1334 Non-blocking garbage collector
The garbage collector no longer blocks other processes from
adding/building store paths or adding GC roots. To prevent the
collector from deleting store paths just added by another process,
processes need to connect to the garbage collector via a Unix domain
socket to register new temporary roots.
2021-10-13 12:12:44 +02:00
Eelco Dolstra
9947f1646a Remove syncWithGC() 2021-10-13 12:12:44 +02:00
Eelco Dolstra
8eac7dfad4 Remove trash directory 2021-10-13 12:12:44 +02:00
Eelco Dolstra
624dfde3df
Merge pull request #5362 from Artturin/nixunpack
nix develop: add --unpack
2021-10-13 11:39:08 +02:00
Eelco Dolstra
0fac86fd6f Style tweaks 2021-10-13 11:00:10 +02:00
Eelco Dolstra
abd685d373 Merge branch 'feature/comp-level' of https://github.com/tomberek/nix 2021-10-13 10:45:44 +02:00
Eelco Dolstra
3e0c6aac9a
Merge pull request #5375 from edolstra/repl-ctrl-c
nix repl: Fix plugin-files warning
2021-10-12 17:11:39 +02:00
Eelco Dolstra
5176b072ed Cleanup 2021-10-12 16:43:00 +02:00
Eelco Dolstra
3a778ea8a0 Merge branch 'nix-repl-download-interruption' of https://github.com/Ma27/nix 2021-10-12 16:34:38 +02:00
Eelco Dolstra
f6cdae5181 nix repl: Don't write to std::cout directly
Writing to std::cout doesn't play nice with ProgressBar.
2021-10-12 15:36:45 +02:00
Eelco Dolstra
9ebe02a81e nix repl: Don't build in a child process
Fixes #5356. This is a bit risky due to interrupts, but we have to
deal with those anyway (#5353).
2021-10-12 15:27:02 +02:00
Tom Bereknyei
03bb8f84e0 Add compression level for NARs
Based off on @dtzWill's #2276
2021-10-12 02:14:36 -04:00
Ben Burdette
427fb8d158 comment out debugs 2021-10-11 16:48:10 -06:00
Ben Burdette
98eb13691a print staticenv bindings 2021-10-11 16:32:43 -06:00