Commit graph

7152 commits

Author SHA1 Message Date
Farid Zakaria
bc14465e08 Fix stack buffer overflow
Fix a stack buffer overflow found by running MemorySanitizer.
2021-11-18 04:05:25 +00:00
Jan Tojnar
e96faadcd6 Fix XDG_CONFIG_DIRS fallback
According to XDG Base Directory Specification,
it should fall back to /etc/xdg when the env var is not present.
2021-11-17 14:31:15 +01:00
Kalle Jepsen
46d2a5a10b Simplify fix by disallowing / in front of @ to match scp style 2021-11-17 13:49:10 +01:00
Eelco Dolstra
d03e89e5d1 Parse '(f x) y' the same as 'f x y'
(cherry picked from commit 5253cb4b68ad248f37b27849c0ebf3614e4f2777)
2021-11-17 09:53:57 +01:00
Kalle Jepsen
6d46b5b609 Fix detection of scp-style URIs to support non-standard SSH ports for git 2021-11-17 08:41:26 +01:00
Eelco Dolstra
d7bae52b9d Call functors with both arguments at once
This is not really useful on its own, but it does recover the
'infinite recursion' error message for '{ __functor = x: x; } 1', and
is more efficient in conjunction with #3718.

Fixes #5515.
2021-11-16 22:34:17 +01:00
Eelco Dolstra
e41cf8511f Don't hang when calling an attrset
Fixes #5565.
2021-11-16 17:44:19 +01:00
Eelco Dolstra
8c93a481af Ignore errors unsharing/restoring the mount namespace
This prevents Nix from barfing when run in a container where it
doesn't have the appropriate privileges.
2021-11-16 14:48:44 +01:00
Eelco Dolstra
ec608e3def
Merge pull request #5452 from Kha/git-extend-lock
git: extend cache dir lock over all mutating operations
2021-11-16 12:55:12 +01:00
Eelco Dolstra
6463eaca14
Merge pull request #5472 from NixOS/async-realisation-substitution
async realisation substitution
2021-11-16 12:54:20 +01:00
Eelco Dolstra
6d0aa8d175
Merge pull request #5559 from Kloenk/libfetchers-gitlab-headers
libfetchers: set free gitlab headers
2021-11-16 12:47:49 +01:00
Eelco Dolstra
9fe0343bfd Merge branch 'master' of github.com:NixOS/nix 2021-11-15 18:45:20 +01:00
Eelco Dolstra
671817a858 Simplify lockFlake() a bit 2021-11-15 18:44:27 +01:00
Eelco Dolstra
d1d223838b Merge branch 'balsoft/fix-subinputs-not-getting-updated' of https://github.com/tweag/nix 2021-11-15 18:37:13 +01:00
Domen Kožar
164179983e
Merge pull request #5428 from kreisys/add-pos-to-json-type-error
toJSON: report error position for fancier output
2021-11-15 07:57:46 -06:00
Finn Behrens
79d07d0980 libfetchers: set free gitlab headers 2021-11-14 12:23:46 +01:00
Kevin Amado
d0e9e18489
toXML: display errors position
- This change applies to builtins.toXML and inner workings
- Proof of concept:
  ```nix
  let e = builtins.toXML e; in e
  ```
- Before:
  ```
  $ nix-instantiate --eval poc.nix
  error: infinite recursion encountered
  ```
- After:
  ```
  $ nix-instantiate --eval poc.nix
  error: infinite recursion encountered

       at /data/github/kamadorueda/nix/poc.nix:1:9:

            1| let e = builtins.toXML e; in e
             |
  ```
2021-11-13 20:33:34 -05:00
Tom Bereknyei
30496af598 Adds an accept-flake-config flag 2021-11-12 09:50:07 -05:00
Eelco Dolstra
bceda30498 Typo 2021-11-12 13:41:15 +01:00
Eelco Dolstra
44aed58538
Merge pull request #5540 from samueldr/fix/key-subcommand
nix key: Fix error message and don't require flakes
2021-11-12 10:38:00 +01:00
Samuel Dionne-Riel
30e5c5c55f nix registry: Mark experimental
This is part of the flakes feature. Mark it as such.
2021-11-12 03:18:31 -05:00
Samuel Dionne-Riel
c1dea92dd6 nix key: Fix error message and don't require flakes 2021-11-11 21:31:26 -05:00
Alexander Bantyev
07bffe7998
Flakes: refetch the input when a follows disappears
When an input follows disappears, we can't just reuse the old lock
file entries since we may be missing some required ones. Refetch the
input when this happens.

Closes https://github.com/NixOS/nix/issues/5289
2021-11-11 12:05:18 +03:00
Eelco Dolstra
a218cfd6c7
Merge pull request #5528 from matthewbauer/recognize-nixosModule
Recognize singular "nixosModule" in nix flake show
2021-11-10 12:17:11 +01:00
Sergei Trofimovich
edfc5b2f12 ca-specific-schema.sql: add index on RealisationsRefs(referrer) and (outputPath)
For a typical desktop system (~2K packages) we can easily get 100K
entries in RealisationsRefs. Without indices query for RealisationsRefs
requires linear scan.

RealisationsRefs(referrer)
--------------------------

Inefficiency is seen as a 100% CPU load of nix-daemon for the following
scenario:

    $ nix edit -f . bash # add unused environment variable, like FOO="1"
    # populate RealisationsRefs, build fresh system
    $ nix build -f nixos system --arg config '{ contentAddressedByDefault = true; }'
    $ nix edit -f . bash # add unused environment variable, like FOO="2"
    $ time nix build -f nixos system --arg config '{ contentAddressedByDefault = true; }'

In this case `bash `will be rebuilt a few times and then rest of CPU
time is spent on scanning RealisationsRefs table (about 5 CPU-minutes
on my machine).

Before the change:

    $ time nix build -f nixos system ... # step 4 above
    real    34m3,613s
    user    0m5,232s
    sys     0m0,758s

Of all this time about 29.5 minutes are taken by nix-daemon's CPU time.

After the change:

    $ time nix build -f nixos system ... # step 4 above
    real    4m50,061s
    user    0m5,038s
    sys     0m0,677s

Of all this time about 1 minute is taken by nix-daemon's CPU time.
Most of the time is spent polling for non-existent realisations on
cache-nixos.org.

Realisations(outputPath)
------------------------

After running CA system for two weeks I got ~1M entries in Realisations
table. `nix-collect-garbage` became very slow (seemingly 100 path deletions
per second). It happens due to a slow cascading delete from Realisations
triggered by deletion from ValidPaths.

The fix is to add an index on primary key from ValidPaths(id) that
triggers cascading deletions.

Before the change:
    $ time nix-collect-garbage -d --max-freed 100G
    <interrupted before finish, took too long>
    real    23m32.411s
    user    17m49.679s
    sys     4m50.609s

Most of time was spent in re-scanning Realisations table on each path deletion.

After the change:
    $ time nix-collect-garbage -d --max-freed 100G

    real    8m43.226s
    user    6m16.317s
    sys     1m40.188s

Time is spent scanning sqlite indices and in kernel when unlinking directories.
2021-11-10 08:32:05 +00:00
Ben Burdette
7e2a3db4eb cleanup 2021-11-09 13:14:49 -07:00
Ben Burdette
885f819922 remove dead code 2021-11-09 11:20:14 -07:00
Matthew Bauer
ceeecf2f9e Recognize singular "nixosModule" in nix flake show
This makes nixosModule appears as a "NixOS Module" like nixosModules
does.
2021-11-09 11:14:15 -06:00
Eelco Dolstra
67179472df
Merge pull request #5494 from tweag/balsoft/allow-references-in-addPath
Allow references in addPath
2021-11-09 15:57:39 +01:00
Alexander Bantyev
0b005bc9d6
addToStore, addToStoreFromDump: refactor: pass refs by const reference
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
2021-11-09 12:24:49 +03:00
Eelco Dolstra
ff2af4d64e Unshare mount namespace in main()
Doing it as a side-effect of calling LocalStore::makeStoreWritable()
is very ugly.

Also, make sure that stopping the progress bar joins the update
thread, otherwise that thread should be unshared as well.
2021-11-08 22:00:45 +01:00
Eelco Dolstra
7a71621b7c Merge branch 'fix-writable-shell' of https://github.com/yorickvP/nix 2021-11-08 21:12:51 +01:00
Eelco Dolstra
6c2af1f201
Merge pull request #5434 from timothyklim/git-url-submodules
Support building flakes from a Git repo url with submodules
2021-11-08 14:13:42 +01:00
Eelco Dolstra
736d6ab721
Merge pull request #5504 from NixOS/flake-options-and-daemon
Make the flake options work when using the daemon
2021-11-08 13:54:55 +01:00
Andreas Rammhold
8e7359db64
Remove unused "<let-body>" symbol
The requirement for the symbol has been removed since at least 7d47498.
2021-11-07 18:26:43 +01:00
Alexander Bantyev
9d4dcff37a
addPath: allow paths with references
Since 4806f2f6b0, we can't have paths with
references passed to builtins.{path,filterSource}. This prevents many cases
of those functions called on IFD outputs from working. Resolve this by
passing the references found in the original path to the added path.
2021-11-05 22:41:30 +03:00
Théophane Hufschmitt
3d6ee223d6
Merge pull request #5476 from NixOS/ca/fix-build-in-nix-repl
Fix `nix repl`’s building of CA derivations
2021-11-05 17:35:18 +01:00
regnat
1f3c3a3785 Make the flake options work when using the daemon
When setting flake-local options (with the `nixConfig` field), forward
these options to the daemon in case we’re using one.

This is necessary in particular for options like `binary-caches` or
`post-build-hook` to make sense.

Fix <343239fc8a (r44356843)>
2021-11-05 16:19:16 +01:00
regnat
1a4c9ba50b Fix nix repl’s building of CA derivations
When running a `:b` command in the repl, after building the derivations
query the store for its outputs rather than just assuming that they are
known in the derivation itself (which isn’t true for CA derivations)

Fix #5328
2021-11-05 11:04:14 +01:00
Eelco Dolstra
40925337a9 Remove maxPrimOpArity 2021-11-04 15:04:07 +01:00
Eelco Dolstra
acd6bddec7 Fix derivation primop 2021-11-04 15:04:00 +01:00
Eelco Dolstra
cbfbf71e08 Use callFunction() with an array for some calls with arity > 1 2021-11-04 15:03:57 +01:00
Eelco Dolstra
bcf4780006 Add level / displacement types 2021-11-04 15:03:45 +01:00
Eelco Dolstra
81e7c40264 Optimize primop calls
We now parse function applications as a vector of arguments rather
than as a chain of binary applications, e.g. 'substring 1 2 "foo"' is
parsed as

  ExprCall { .fun = <substring>, .args = [ <1>, <2>, <"foo"> ] }

rather than

  ExprApp (ExprApp (ExprApp <substring> <1>) <2>) <"foo">

This allows primops to be called immediately (if enough arguments are
supplied) without having to allocate intermediate tPrimOpApp values.

On

  $ nix-instantiate --dry-run '<nixpkgs/nixos/release-combined.nix>' -A nixos.tests.simple.x86_64-linux

this gives a substantial performance improvement:

  user CPU time:      median =      0.9209  mean =      0.9218  stddev =      0.0073  min =      0.9086  max =      0.9340  [rejected, p=0.00000, Δ=-0.21433±0.00677]
  elapsed time:       median =      1.0585  mean =      1.0584  stddev =      0.0024  min =      1.0523  max =      1.0623  [rejected, p=0.00000, Δ=-0.20594±0.00236]

because it reduces the number of tPrimOpApp allocations from 551990 to
42534 (i.e. only small minority of primop calls are partially
applied) which in turn reduces time spent in the garbage collector.
2021-11-04 15:03:40 +01:00
Eelco Dolstra
ab35cbd675 StaticEnv: Use std::vector instead of std::map 2021-11-04 15:03:34 +01:00
Eelco Dolstra
c4bd6a15c2 Add helper function to check whether a function arg is 'X' or '_X'
Also allow '_'.
2021-11-04 14:52:35 +01:00
Eelco Dolstra
1e7c796e66
Merge pull request #5475 from doronbehar/SQLiteWAL-vfs
libstore: Use unix-dotfile vfs if useSQLiteWAL is false
2021-11-03 21:20:27 +01:00
Christopher League
3f070cc417 In checkOverlay, accept underscored names for final/prev args.
Resolves #4416.
2021-11-03 09:25:27 -04:00
Doron Behar
14fcf17277 libstore: Use unix-dotfile vfs if useSQLiteWAL is false 2021-11-03 14:19:11 +02:00
regnat
f4c869977c Make the DrvOutputSubstitutionGoal more async 2021-11-03 06:51:34 +01:00
regnat
fbc70034b3 Make the realisation fetching from binary caches async
That way we can fetch several realisations from the same cache in
parallel
2021-11-03 06:51:34 +01:00
regnat
96670ed216 Expose an async interface for queryRealisation
Doesn’t change much so far because everything is still using it
synchronously, but should allow the binary cache to fetch stuff in
parallel
2021-11-03 06:51:34 +01:00
regnat
5b2aa61f1b Don’t require ca-derivations when __contentAddressed = false
If we explicitely opt-out of it, there’s no need to require the
experimental feature
2021-11-03 06:51:32 +01:00
Sebastian Ullrich
b459a3e856 git: extend cache dir lock over all mutating operations 2021-11-01 09:14:56 +01:00
Eelco Dolstra
e6795c4350 Style 2021-10-29 14:45:13 +02:00
Eelco Dolstra
6e30d9b69f Merge branch 'master' of https://github.com/alekswn/nix 2021-10-29 14:42:26 +02:00
Eelco Dolstra
33d04e8a8d Use nix::connect() to connect to the garbage collector 2021-10-28 22:51:11 +02:00
Eelco Dolstra
22c35ea5b8 Remove unused variable 2021-10-28 22:51:11 +02:00
Eelco Dolstra
a7d4f3411e Merge remote-tracking branch 'origin/master' into non-blocking-gc 2021-10-28 14:56:55 +02:00
Shay Bergmann
a50c027ece
toJSON: improve pos accuracy, add trace 2021-10-27 19:48:48 +00:00
Shay Bergmann
465a167c43
nix-instantiate: pass pos in the --eval --json code path 2021-10-27 19:01:32 +00:00
figsoda
a2473823d7 run: use pname as a fallback for main program 2021-10-27 14:49:24 -04:00
Eelco Dolstra
5a160171d0 Remove redundant 'warning:' 2021-10-27 18:14:12 +02:00
Eelco Dolstra
1254e8753c build-remote: Implicitly add the 'builtin' system type to all machines
This makes 'nix-env -i --max-jobs 0' work with remote builders.
2021-10-27 14:25:13 +02:00
Eelco Dolstra
f2280749b1 If max-jobs == 0, do preferLocalBuild on remote builders 2021-10-27 14:21:31 +02:00
Eelco Dolstra
6e684d1b87 daemon: Accept 'repeat' setting from untrusted users
Fixes #5352.
2021-10-27 13:09:13 +02:00
Eelco Dolstra
13a7a24ba5 Style 2021-10-27 13:02:37 +02:00
Shay Bergmann
769de259f0
toJSON: pass pos in case of a list as well 2021-10-26 14:43:15 +00:00
Timothy
8919b81dad Support building flakes from a Git repo url with submodules query parameter 2021-10-26 20:02:37 +07:00
Eelco Dolstra
0d9e050ba7 parseExperimentalFeature(): Initialize atomically 2021-10-26 14:29:48 +02:00
Eelco Dolstra
9ce84c64c5 Tweak fetchTree docs 2021-10-26 14:21:24 +02:00
regnat
af99941279 Make experimental-features a proper type
Rather than having them plain strings scattered through the whole
codebase, create an enum containing all the known experimental features.

This means that
- Nix can now `warn` when an unkwown experimental feature is passed
  (making it much nicer to spot typos and spot deprecated features)
- It’s now easy to remove a feature altogether (once the feature isn’t
  experimental anymore or is dropped) by just removing the field for the
  enum and letting the compiler point us to all the now invalid usages
  of it.
2021-10-26 07:02:31 +02:00
Shay Bergmann
ba81e871b2
toJSON: report error position for fancier output
Given flake:

```nix
{ description = "nix json error provenance";
  inputs = {};
  outputs = { self }: {
    jsonFunction = _: "function";
    json = builtins.toJSON (_: "function");
  };
}

```
- Before:

```console
❯ nix eval --json .#jsonFunction
error: cannot convert a function to JSON
```

- After:

```console
❯ nix eval --json .#jsonFunction
error: cannot convert a function to JSON

       at /nix/store/b7imf1c2j4jnkg3ys7fsfbj02s5j0i4f-source/testflake/flake.nix:4:5:

            3|   outputs = { self }: {
            4|     jsonFunction = _: "function";
             |     ^
            5|     json = builtins.toJSON (_: "function");
```
2021-10-25 21:17:52 +00:00
Alexander Bantyev
b9234142f5
addToStore, addToStoreFromDump: add references argument
Allow to pass a set of references to be added as info to the added paths.
2021-10-23 21:30:51 +03:00
Ben Burdette
fb8377547b more code cleanup 2021-10-22 14:49:58 -06:00
Ben Burdette
71da988d47 more debug removal 2021-10-22 14:34:50 -06:00
Ben Burdette
e54f17eb46 remove more debug code 2021-10-22 14:27:04 -06:00
Ben Burdette
cbc2f0fe31 remove dead code 2021-10-22 14:02:47 -06:00
Ben Burdette
383ab600ee show expr on error 2021-10-22 13:41:04 -06:00
Kevin Amado
823dce945a
fetch: nicer infinite recursion errors
- This change applies to builtins.fetchurl and builtins.fetchTarball
- PoC: `let x = builtins.fetchurl x; in x`
- Before:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered
  ```
- After:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered

       at /data/github/kamadorueda/nix/test.nix:1:9:

            1| let x = builtins.fetchurl x; in x
             |         ^
  ```

Mentions: #3505
2021-10-17 12:54:53 -05:00
Alexey Novikov
e989c83b44 Add error reporting to machine spec paser
Currently machine specification (`/etc/nix/machine`) parser fails
with a vague exception if the file had incorrect format.
This commit adds verbose exceptions and unit-tests for the parser.
2021-10-17 12:45:56 +04:00
Alexey Novikov
64a3b045c1 Fix error detection in 'base64Decode()'
Fixed a bug in initialization of 'base64DecodeChars' variable.
Currently decoder do not fail on invalid Base64 strings.
Added test-case to verify the fix.

Also have made 'base64DecodeChars' to be computed at compile time.
And added a test case to encode/decode string with non-printable charactes.
2021-10-17 12:45:26 +04:00
Arthur Gautier
fa4abe46e2 preloadNSS: document the preload mechanism
Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-10-16 02:55:25 +00:00
Kevin Amado
e5a27a3b4e
fetchTree: add pos to EvalState::forceValue
- This way we improve error messages
  on infinite recursion
- Demo:
  ```nix
  let x = builtins.fetchTree {
    type = "git";
    inherit x;
  };
  in x
  ```
- Before:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered
  ```
- After:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered

       at /data/github/kamadorueda/nix/test.nix:3:10:

            2|   type = "git";
            3|   inherit x;
             |          ^
            4| };
  ```

Mentions: #3505
2021-10-15 19:25:19 -05:00
Kevin Amado
18e3d63341
fetchTree: add pos to EvalState::forceValue
- This way we improve error messages
  on infinite recursion
- Demo:
  ```nix
  let x = builtins.fetchTree x;
  in x
  ```
- Before:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered
  ```
- After:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered

       at /data/github/kamadorueda/nix/test.nix:1:9:

            1| let x = builtins.fetchTree x;
             |         ^
            2| in x
  ```

Mentions: #3505
2021-10-15 19:25:19 -05:00
Eelco Dolstra
10f9a8e77d Add a test for the non-blocking GC 2021-10-15 16:52:37 +02:00
Eelco Dolstra
ac54c6faa6 Fix main GC thread exiting 2021-10-15 16:36:48 +02:00
Yorick van Pelt
fcb8af550f
Restore parent mount namespace in restoreProcessContext
This ensures any started processes can't write to /nix/store (except
during builds). This partially reverts 01d07b1e, which happened because
of #2646.

The problem was only happening after nix downloads anything, causing
me to suspect the download thread. The problem turns out to be:
"A  process  can't  join a new mount namespace if it is sharing
filesystem-related attributes with another process", in this case this
process is the curl thread.

Ideally, we might kill it before spawning the shell process, but it's
inside a static variable in the getFileTransfer() function. So
instead, stop it from sharing FS state using unshare(). A strategy
such as the one from #5057 (single-threaded chroot helper binary) is
also very much on the table.

Fixes #4337.
2021-10-15 16:25:49 +02:00
Eelco Dolstra
e0936ae38f Fix crash when a GC client disconnects
The client thread can't just delete its own thread object from
connections, it has to detach it.
2021-10-15 16:12:21 +02:00
Eelco Dolstra
304180d0de Memoize queryReferrers() 2021-10-15 12:20:23 +02:00
Arthur Gautier
c345a4a1e8 fixup macos build
Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-10-15 07:34:01 +00:00
Eelco Dolstra
c0951299b3
Merge pull request #5383 from kvtb/patch-7
fix build with gcc11
2021-10-15 09:03:12 +02:00
Eelco Dolstra
c574ab3907
Merge pull request #5388 from yvt/fix-oahd-path
Add another path where a Rosetta 2 daemon plist file is possibly located
2021-10-15 09:02:16 +02:00
Kevin Amado
1bdeef8395
add pos to EvalState::forceValue
- This way we improve error messages
  on infinite recursion
- Demo:
  ```nix
  let
    x = builtins.fetchMercurial x;
  in
  x
  ```
- Before:
  ```bash
  $ nix-instantiate --show-trace --strict
  error: infinite recursion encountered
  ```
- After:
  ```bash
  nix-instantiate --show-trace --strict
  error: infinite recursion encountered

       at /data/github/kamadorueda/test/default.nix:2:7:

            1| let
            2|   x = builtins.fetchMercurial x;
             |       ^
            3| in
  ```

Mentions: #3505
2021-10-14 23:23:05 -05:00
Arthur Gautier
ca8989daf3 preloadNSS: warn if unable to open nss backend
Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-10-15 02:59:00 +00:00
Arthur Gautier
85717eff15 preloadNSS: detect glibc
Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-10-15 02:59:00 +00:00
Arthur Gautier
d1da45855c preloadNSS: Drop the dns query workaround
We can actually just load nss ourselves and call in nss to configure it
and we don't need to run a dummy query entirely to have nss load nss_dns
as a side-effect.

Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-10-15 02:58:18 +00:00
yvt
a9d9e55551 Add another path where a Rosetta 2 configuration file is possibly located 2021-10-15 09:48:15 +09:00
Eelco Dolstra
17e6ebcc90 Speed up GC by marking entire closures as live 2021-10-14 14:13:57 +02:00
Eelco Dolstra
0154fa30cf Remove GCState 2021-10-14 13:52:49 +02:00
Eelco Dolstra
0317ffdad3 Move deleteFromStore() 2021-10-14 13:34:48 +02:00
Eelco Dolstra
0be8cc1466 pathInfoCache: Use the entire base name as the cache key
This fixes a bug in the garbage collector where if a path
/nix/store/abcd-foo is valid, but we do a
isValidPath("/nix/store/abcd-foo.lock") first, then a negative entry
for /nix/store/abcd is added to pathInfoCache, so /nix/store/abcd-foo
is subsequently considered invalid and deleted.
2021-10-14 13:28:22 +02:00
Eelco Dolstra
eab934cb2a Make the canReachRoots() traversal non-recursive 2021-10-14 12:34:32 +02:00
Eelco Dolstra
09b14ea97a Cleanup 2021-10-14 10:04:13 +02:00
kvtb
eae29b0385
fix build with gcc11 2021-10-13 18:03:33 +00:00
Eelco Dolstra
35c98a59c5 Fix GC when there are cycles in the referrers graph
(where "referrers" includes the reverse of derivation outputs and
derivers). Now we do a full traversal to look if we can reach any
root. If not, all paths reached can be deleted.
2021-10-13 12:12:44 +02:00
Eelco Dolstra
1785ba2980 Simplify 2021-10-13 12:12:44 +02:00
Eelco Dolstra
dced45f146 strcpy -> memcpy
Co-authored-by: Jörg Thalheim <Mic92@users.noreply.github.com>
2021-10-13 12:12:44 +02:00
Eelco Dolstra
262520fcfe Use a thread per connection 2021-10-13 12:12:44 +02:00
Eelco Dolstra
ff453b06f9 Fix auto-gc 2021-10-13 12:12:44 +02:00
Eelco Dolstra
8614cf1334 Non-blocking garbage collector
The garbage collector no longer blocks other processes from
adding/building store paths or adding GC roots. To prevent the
collector from deleting store paths just added by another process,
processes need to connect to the garbage collector via a Unix domain
socket to register new temporary roots.
2021-10-13 12:12:44 +02:00
Eelco Dolstra
9947f1646a Remove syncWithGC() 2021-10-13 12:12:44 +02:00
Eelco Dolstra
8eac7dfad4 Remove trash directory 2021-10-13 12:12:44 +02:00
Eelco Dolstra
624dfde3df
Merge pull request #5362 from Artturin/nixunpack
nix develop: add --unpack
2021-10-13 11:39:08 +02:00
Eelco Dolstra
0fac86fd6f Style tweaks 2021-10-13 11:00:10 +02:00
Eelco Dolstra
abd685d373 Merge branch 'feature/comp-level' of https://github.com/tomberek/nix 2021-10-13 10:45:44 +02:00
Eelco Dolstra
3e0c6aac9a
Merge pull request #5375 from edolstra/repl-ctrl-c
nix repl: Fix plugin-files warning
2021-10-12 17:11:39 +02:00
Eelco Dolstra
5176b072ed Cleanup 2021-10-12 16:43:00 +02:00
Eelco Dolstra
3a778ea8a0 Merge branch 'nix-repl-download-interruption' of https://github.com/Ma27/nix 2021-10-12 16:34:38 +02:00
Eelco Dolstra
f6cdae5181 nix repl: Don't write to std::cout directly
Writing to std::cout doesn't play nice with ProgressBar.
2021-10-12 15:36:45 +02:00
Eelco Dolstra
9ebe02a81e nix repl: Don't build in a child process
Fixes #5356. This is a bit risky due to interrupts, but we have to
deal with those anyway (#5353).
2021-10-12 15:27:02 +02:00
Tom Bereknyei
03bb8f84e0 Add compression level for NARs
Based off on @dtzWill's #2276
2021-10-12 02:14:36 -04:00
Ben Burdette
427fb8d158 comment out debugs 2021-10-11 16:48:10 -06:00
Ben Burdette
98eb13691a print staticenv bindings 2021-10-11 16:32:43 -06:00
Ben Burdette
2ee1fa4afd add nullable Expr argument 2021-10-11 14:42:29 -06:00
regnat
7466048d39 (partially) Revert "Don't copy in rethrow"
This reverts some parts of commit
8430a8f086 which was trying to rethrow
some exceptions while we weren’t in the context of a `catch` block,
causing some weird “terminate called without an active exception”
errors.

Fix #5368
2021-10-11 10:51:22 +02:00
Artturin
e399c6ab7f nix develop: add --unpack 2021-10-09 01:19:50 +03:00
Sergei Trofimovich
d7d6fe44d6 libstore-tests: add libutil dependency (fix static link failure)
In https://github.com/NixOS/nix/pull/5350 we noticed link failures
pkgsStatic.nixUnstable. Adding explicit dependency on libutil fixes
libstore-tests linking.
2021-10-08 07:16:12 +01:00
Maximilian Bosch
0872659002
nix repl: properly deal with interruptions
When I stop a download with Ctrl-C in a `nix repl` of a flake, the REPL
refuses to do any other downloads:

    nix-repl> builtins.getFlake "nix-serve"
    [0.0 MiB DL] downloading 'https://api.github.com/repos/edolstra/nix-serve/tarball/e9828a9e01a14297d15ca41 error: download of 'e9828a9e01' was interrupted
    [0.0 MiB DL]
    nix-repl> builtins.getFlake "nix-serve"
    error: interrupted by the user
    [0.0 MiB DL]

To fix this issue, two changes were necessary:

* Reset the global `_isInterrupted` variable: only because a single
  operation was aborted, it should still be possible to continue the
  session.
* Recreate a `fileTransfer`-instance if the current one was shut down by
  an abort.
2021-10-07 23:58:02 +02:00
Eelco Dolstra
d39692e6b3 Make builtins.{path,filterSource} work with chroot stores 2021-10-07 14:22:39 +02:00
Eelco Dolstra
c4dcf3cf25 Add a trace to all errors in addPath() 2021-10-07 13:47:15 +02:00
Eelco Dolstra
4806f2f6b0 Allow builtins.{path,filterSource} on paths with a context
We now build the context (so this has the side-effect of making
builtins.{path,filterSource} work on derivations outputs, if IFD is
enabled) and then check that the path has no references (which is what
we really care about).
2021-10-07 13:43:17 +02:00
Eelco Dolstra
66c4b20d8b Typo 2021-10-07 13:34:04 +02:00
Eelco Dolstra
7c50568788 Remove unnecessary call to queryMissing()
Worker::run() already does this.
2021-10-07 13:15:01 +02:00
Eelco Dolstra
972405edf5 Allow access to path copied to the store
Fixes https://github.com/NixOS/nix/pull/5163#issuecomment-931733912.
2021-10-07 12:15:22 +02:00
Eelco Dolstra
cfaad7168e Refactoring: Add allowPath() method 2021-10-07 12:11:00 +02:00
Eelco Dolstra
c9ee634f75
Merge pull request #5341 from andir/libexpr-formals
libexpr: remove matchAttrs boolean from ExprLambda
2021-10-07 11:58:56 +02:00
Eelco Dolstra
53e4794289
Merge pull request #5286 from ilkecan/add-a-warning-to-filterSource
Warn about the usage of filterSource with Nix store paths
2021-10-06 21:02:39 +02:00
ilkecan
a4a6ef4fb2 Add a warning to filterSource
Warn about the usage of `filterSource` with Nix store paths
2021-10-06 19:25:33 +03:00
Eelco Dolstra
0dc8172458 Remove no-op call to realiseContext() 2021-10-06 18:08:37 +02:00
Eelco Dolstra
c497fce011 Merge branch 'flakes_filterSource' of https://github.com/tomberek/nix 2021-10-06 18:08:18 +02:00
Eelco Dolstra
57a8eb4c01 fetchTree(): Parse type attribute first
The 'url' attribute depends on whether type == 'git', so this is needed for

  builtins.fetchTree {url = "git@github.com:NixOS/nix.git"; type = "git";}
2021-10-06 17:39:02 +02:00
Eelco Dolstra
83d86cc1b0 Cleanup 2021-10-06 17:30:10 +02:00
Andreas Rammhold
cae41eebff libexpr: remove matchAttrs boolean from ExprLambda
The boolean is only used to determine if the formals are set to a
non-null pointer in all our cases. We can get rid of that allocation and
instead just compare the pointer value with NULL. Saving up to
sizeof(bool) + platform specific alignment per ExprLambda instace.
Probably not a lot of memory but perhaps a few kilobyte with nixpkgs?

This also gets rid of a potential issue with dereferencing formals based on
the value of the boolean that didn't have to be aligned with the formals
pointer but was in all our cases.
2021-10-06 17:24:06 +02:00
Eelco Dolstra
46753b5e9c Merge branch 'considerate/scp-like-urls' of https://github.com/considerate/nix 2021-10-06 17:19:27 +02:00
Eelco Dolstra
c6718a9d95 Don't reset the logger in a vfork
9c766a40cb broke logging from the
daemon, because commonChildInit is called when starting the build hook
in a vfork, so it ends up resetting the parent's logger. So don't
vfork.

It might be best to get rid of vfork altogether, but that may cause
problems, e.g. when we call an external program like git from the
evaluator.
2021-10-06 13:54:59 +02:00
Eelco Dolstra
08aa7daee0 Remove links to .md files in help output
Fixes #5337.
2021-10-06 13:01:18 +02:00
Eelco Dolstra
fd57e7074f
Merge pull request #5335 from edolstra/socket-paths
Support arbitrary-length socket paths
2021-10-05 17:36:07 +02:00
Eelco Dolstra
b14bc06955 Don't ignore SIGCHLD in createUnixDomainSocket() 2021-10-05 13:23:16 +02:00
Eelco Dolstra
e4a5d64a81 Show failing PID 2021-10-05 13:19:55 +02:00
Eelco Dolstra
b299560872 Typo 2021-10-05 11:24:09 +02:00
Eelco Dolstra
3b7f4c7d9d Add FIXME about ptsname 2021-10-05 11:04:46 +02:00
Eelco Dolstra
43d4d75e22 Connect/bind Unix domain sockets in a child process
In the child process, we can do a chdir() and avoid the problem of the
path not fitting into sockaddr_un.
2021-10-05 10:44:59 +02:00
Eelco Dolstra
77ebbc9f54 Add a test for RefScanSink and clean up the code
Issue #5322.
2021-10-04 14:29:42 +02:00
Eelco Dolstra
ef34fd0656 scanForReferences(): Use a StorePathSet 2021-10-04 13:47:38 +02:00
Eelco Dolstra
6fadb5df05
Merge pull request #5310 from trofi/drop-spammy-rewrite
local-derivation-goal.cc: drop spammy "warning: rewriting hashes in..…
2021-10-03 19:53:06 +02:00
kvtb
638c73776a
fix creation of NAR files >4GB on 32-bit platforms
`size_t` is 32-bit on 32-bit platforms, while the file size can be bigger
2021-10-02 21:04:01 +00:00
Ben Burdette
aad27143c6 storing staticenv bindings 2021-10-02 13:47:36 -06:00
Aleksey Kladov
3e884aa002
Fix typo 2021-10-02 10:17:45 +03:00
Sergei Trofimovich
621aa65325 local-derivation-goal.cc: downgrade "warning: rewriting hashes in..." down to debug
Before the changes when building the whole system with
`contentAddressedByDefault = true;` we get many noninformative messages:

    $ nix build -f nixos system --keep-going
    ...
    warning: rewriting hashes in '/nix/store/...-clang-11.1.0.drv.chroot/nix/store/...-11.1.0'; cross fingers
    warning: rewriting hashes in '/nix/store/...-clang-11.1.0.drv.chroot/nix/store/...-11.1.0-dev'; cross fingers
    warning: rewriting hashes in '/nix/store/...-clang-11.1.0.drv.chroot/nix/store/...-11.1.0-python'; cross fingers
    error: 2 dependencies of derivation '/nix/store/...-hub-2.14.2.drv' failed to build
    warning: rewriting hashes in '/nix/store/...-subversion-1.14.1.drv.chroot/nix/store/...-subversion-1.14.1-dev'; cross fingers
    warning: rewriting hashes in '/nix/store/...-subversion-1.14.1.drv.chroot/nix/store/...-subversion-1.14.1-man'; cross fingers
    ...

Let's downgrade these messages down to debug().
2021-10-01 17:41:44 +01:00
Eelco Dolstra
7cc220825d
Merge pull request #5167 from Ma27/keep-failed-on-ssh-remote-build
nix-store --serve: pass on `settings.keepFailed` from SSH store
2021-10-01 16:35:02 +02:00
John Ericson
242f9bf3dc std::visit by reference
I had started the trend of doing `std::visit` by value (because a type
error once mislead me into thinking that was the only form that
existed). While the optomizer in principle should be able to deal with
extra coppying or extra indirection once the lambdas inlined, sticking
with by reference is the conventional default. I hope this might even
improve performance.
2021-09-30 21:35:09 +00:00
Taeer Bar-Yam
f14660d5e2 reset yylloc when yyless(0) is called 2021-09-29 19:47:01 -04:00
Eelco Dolstra
fd01c48d34
Merge pull request #5301 from Ma27/builtins-missing-feature-error
libexpr: throw a more helpful eval-error if a builtin is not available due to a missing feature-flag
2021-09-29 12:53:29 +02:00
Maximilian Bosch
2b02ce0e48
libexpr: throw a more helpful eval-error if a builtin is not available due to a missing feature-flag
I found it somewhat confusing to have an error like

    error: attribute 'getFlake' missing

if the required experimental-feature (`flakes`) is not enabled. Instead,
I'd expect Nix to throw an error just like it's the case when using e.g. `nix
flake` without `flakes` being enabled.

With this change, the error looks like this:

    $ nix-instantiate -E 'builtins.getFlake "nixpkgs"'
    error: Cannot call 'builtins.getFlake' because experimental Nix feature 'flakes' is disabled. You can enable it via '--extra-experimental-features flakes'.

           at «string»:1:1:

                1| builtins.getFlake "nixpkgs"
                 | ^

I didn't use `settings.requireExperimentalFeature` here on purpose
because this doesn't contain a position. Also, it doesn't seem as if we
need to catch the error and check for the missing feature here since
this already happens at evaluation time.
2021-09-29 11:57:15 +02:00
Eelco Dolstra
6b5efeafd1
Merge pull request #5295 from Ma27/bmcheck-fod-mismatch
build: also throw hash-mismatch errors if `buildMode == bmCheck`
2021-09-29 10:13:45 +02:00
Maximilian Bosch
bb1a851bcf
build: also throw hash-mismatch errors if buildMode == bmCheck
This actually bit me quite recently in `nixpkgs` because I assumed that
`nix-build --check` would also error out if hashes don't match anymore[1]
and so I wrongly assumed that I couldn't reproduce the mismatch error.

The fix is rather simple, during the output registration a so-called
`delayedException` is instantiated e.g. if a FOD hash-mismatch occurs.
However, in case of `nix-build --check` (or `--rebuild` in case of `nix
build`), the code-path where this exception is thrown will never be
reached.

By adding that check to the if-clause that causes an early exit in case
of `bmCheck`, the issue is gone. Also added a (previously failing)
test-case to demonstrate the problem.

[1] https://github.com/NixOS/nixpkgs/pull/139238, the underlying issue
    was that `nix-prefetch-git` returns different hashes than `fetchgit`
    because the latter one fetches submodules by default.
2021-09-27 15:44:39 +02:00
Eelco Dolstra
9c766a40cb Fix 'error: reading a line: Input/output error' in startBuilder()
With -vvvv, the ProgressBar was polluting the stderr of the child,
messing up its \2 message to the parent.
2021-09-27 14:44:21 +02:00
Eelco Dolstra
8430a8f086 Don't copy in rethrow 2021-09-27 14:38:10 +02:00
Eelco Dolstra
4b2b151131 nix path-info -r: Don't duplicate the root paths
This fixes

  $ nix path-info -r $(type -P ls)
  /nix/store/vfilzcp8a467w3p0mp54ybq6bdzb8w49-coreutils-8.32
  /nix/store/5d821pjgzb90lw4zbg6xwxs7llm335wr-libunistring-0.9.10
  ...
  /nix/store/mrv4y369nw6hg4pw8d9p9bfdxj9pjw0x-acl-2.3.0
  /nix/store/vfilzcp8a467w3p0mp54ybq6bdzb8w49-coreutils-8.32

Also, output the paths in topologically sorted order like we used to.
2021-09-27 12:47:39 +02:00
Eelco Dolstra
a15e65eef0 run(): Move 2021-09-27 11:12:06 +02:00
Eelco Dolstra
283e7da470
Merge pull request #5292 from edolstra/update-lowdown
Update lowdown
2021-09-27 11:10:05 +02:00
oxalica
81b8e910a0
Explicitly set initial branch name for git 2021-09-24 22:09:49 +08:00
Eelco Dolstra
aedbc7b683 Use latest lowdown
This improves list rendering
(https://github.com/kristapsdz/lowdown/issues/73).
2021-09-24 15:22:51 +02:00
Eelco Dolstra
362d8f925e
Merge pull request #5253 from edolstra/flake-ifd
Don't allow IFD in flakes by default
2021-09-24 10:48:23 +02:00
Ben Burdette
b9d08b98da ok was unconditoinally throwing on any With var 2021-09-23 13:02:39 -06:00
Eelco Dolstra
ea9df6fe51 Shut down write side before draining the read side
This is important if the remote side *does* execute
nix-store/nix-daemon successfully, but stdout is polluted
(e.g. because the remote user's bashrc script prints something to
stdout). In that case we have to shutdown the write side to force the
remote nix process to exit.
2021-09-23 18:01:04 +02:00
Eelco Dolstra
994348e9e0 SSHStore / LegacySSHStore: Show a better error message if the remote is "nologin"
Instead of

  error: serialised integer 7161674624452356180 is too large for type 'j'

we now get

  error: 'nix-store --serve' protocol mismatch from 'sshtest@localhost', got 'This account is currently not available.'

Fixes https://github.com/NixOS/nixpkgs/issues/37287.
2021-09-23 17:50:29 +02:00
Eelco Dolstra
60642aa5e2 Remove risky char * 2021-09-23 12:07:50 +02:00
Eelco Dolstra
198fa786a1 Add some more instrumentation 2021-09-23 11:42:33 +02:00
Geoff Reedy
9d67332e4b Better eval error locations for interpolation and +
Previously, type or coercion errors for string interpolation, path
interpolation, and plus expressions were always reported at the
beginning of the outer expression. This leads to confusing evaluation
error messages making it hard to accurately diagnose and then fix the
error.

For example, errors were reported as follows.

```
cannot coerce an integer to a string
1| let foo = 7; in "bar" + foo
 |                 ^

cannot add a string to an integer
1| let foo = "bar"; in 4 + foo
 |                     ^

cannot coerce an integer to a string
1| let foo = 7; in "x${foo}"
 |                  ^
```

This commit changes the ExprConcatStrings expression vector to store a
sequence of expressions *and* their expansion locations so that error
locations can be reported accurately. For interpolation, the error is
reported at the beginning of the entire `${foo}`, not at the beginning
of `foo` because I thought this was slightly clearer. The previous
errors are now reported as:

```
cannot coerce an integer to a string
1| let foo = 7; in "bar" + foo
 |                         ^

cannot add a string to an integer
1| let foo = "bar"; in 4 + foo
 |                         ^

cannot coerce an integer to a string
1| let foo = 7; in "x${foo}"
 |                   ^
```

The error is reported at this kind of precise location even for
multi-line indented strings.

This probably helps with at least some of the cases mentioned in #561
2021-09-22 20:57:34 -06:00
Ben Burdette
c07edb1932 staticenv should be With 2021-09-22 18:14:57 -06:00
Ben Burdette
c7e3d830c1 more debug stuff 2021-09-22 16:22:53 -06:00
Eelco Dolstra
d1bf7431bb Revert "Merge pull request #4922 from nrdxp/default-submodules"
This reverts commit 6678e98411, reversing
changes made to 90b2dd570c.
2021-09-22 17:25:25 +02:00
Eelco Dolstra
8623a5b595 Disable IFD selectively
It's now disabled by default for the following:

* 'nix search' (this was already implied by read-only mode)
* 'nix flake show'
* 'nix flake check', but only on the hydraJobs output
2021-09-22 17:15:07 +02:00
Eelco Dolstra
d8c10028d9 Make setDefault() typed 2021-09-22 14:15:35 +02:00
Eelco Dolstra
ff28fffce2 Don't cache realiseContext() errors
Errors that depend on the configuration (such as whether
allow-import-from-derivation is set) should not be cached.
2021-09-22 14:00:56 +02:00
Eelco Dolstra
bcd73ebf60 Add missing #include
Fixes #5282.
2021-09-22 12:14:50 +02:00
Eelco Dolstra
60cc975d22 Set input parent at construction time 2021-09-21 14:07:16 +02:00
Eelco Dolstra
06557299b3 Allow relative paths anywhere into the parent's store path 2021-09-21 13:45:11 +02:00
Eelco Dolstra
5cbb9c5406 path fetcher: Fix relative path check 2021-09-21 13:19:26 +02:00
Michael Bishop
374fe49ff7 set the PER_LINUX32 personality flag, when building for armv6l-linux or armv7l-linux
this prevents 32bit builds from detecting a 64bit kernel and picking the wrong target
2021-09-19 23:07:10 -03:00
Gregor Kleen
fd67a0f927 Fix scheme argument to s3 cache 2021-09-17 10:45:19 +02:00
Geoff Reedy
cbe9ddfd1a Include subpath in flake fingerprint
Without this, flakes within the same tree and same lock data will have
the same fingerprint and the eval cache for one flake will be
incorrectly used for another.
2021-09-16 15:58:21 -06:00
Ben Burdette
cd8c232b55 add cout debugging 2021-09-15 16:16:53 -06:00
Eelco Dolstra
1ec4efa6c8
Merge pull request #5257 from edolstra/dirty-lock-file
If we can't write a lock file, pretend the top-level flake is dirty
2021-09-15 20:18:23 +02:00
Eelco Dolstra
c17f3c5e69 Merge branch 'mh/fix-chroot-eval' of https://github.com/obsidiansystems/nix 2021-09-15 18:37:58 +02:00
Eelco Dolstra
027344ce7e If we can't write a lock file, pretend the top-level flake is dirty
Alternative to #4639. You can still read flake.lock, but at least in
reproducible workflows like NixOS configurations where you require a
non-dirty tree, evaluation will fail because there is no rev.
2021-09-15 18:31:42 +02:00
Eelco Dolstra
991cc53386 Revert "Disallow reading flake.lock"
This reverts commit e5596113f7.
2021-09-15 18:30:37 +02:00
Emily
56025ad3b1 sandbox: allow Rosetta 2 on Darwin
This allows sandboxed x86_64-darwin builds on aarch64-darwin.
2021-09-15 02:02:03 +01:00