Commit graph

1889 commits

Author SHA1 Message Date
Domen Kožar
2d2769f68c
Merge pull request #2338 from bobvanderlinden/pr-cannot-delete-alive-why
mention `nix-store --query --roots` when a path cannot be deleted
2019-09-30 14:06:52 +02:00
Eelco Dolstra
5038e1bec4
Merge pull request #3103 from bhipple/fix/spelling
Fix spelling in comment
2019-09-23 10:46:40 +02:00
Eelco Dolstra
02b4632e77
Merge pull request #3104 from zimbatm/no-show-trace-forwarding
libstore: don't forward --show-trace
2019-09-23 10:44:42 +02:00
Eelco Dolstra
3a022d4599 Shut up some warnings
(cherry picked from commit 99e8e58f2d)
2019-09-22 21:57:05 +02:00
Eelco Dolstra
bd79c1f6f6 Don't catch exceptions by value
(cherry picked from commit 893be6f5e3)
2019-09-22 21:56:56 +02:00
zimbatm
e63c9e73e3
libstore: don't forward --show-trace 2019-09-22 10:57:20 +00:00
Benjamin Hipple
c6a542f22a Fix spelling in comment 2019-09-21 18:53:15 -04:00
Eelco Dolstra
6b83174fff std::uncaught_exception() -> std::uncaught_exceptions()
The former is deprecated in C++17. Fixes a clang warning.
2019-09-13 20:05:44 +02:00
Eelco Dolstra
5dafde28db
BinaryCacheStore: Add index-debug-info option
This integrates the functionality of the index-debuginfo program in
nixos-channel-scripts to maintain an index of DWARF debuginfo files in
a format usable by dwarffs. Thus the debug info index is updated by
Hydra rather than by the channel mirroring script.

Example usage:

  $ nix copy --to 'file:///tmp/binary-cache?index-debug-info=true' /nix/store/vr9mhcch3fljzzkjld3kvkggvpq38cva-nix-2.2.2-debug

  $ cat /tmp/binary-cache/debuginfo/036b210b03bad75ab2d8fc80b7a146f98e7f1ecf.debug
  {"archive":"../nar/0313h2kdhk4v73xna9ysiksp2v8xrsk5xsw79mmwr3rg7byb4ka8.nar.xz","member":"lib/debug/.build-id/03/6b210b03bad75ab2d8fc80b7a146f98e7f1ecf.debug"}

Fixes #3083.
2019-09-04 19:28:26 +02:00
Eelco Dolstra
e07ec8d27e
Support allowSubstitutes attribute in structured attribute derivations
Hopefully fixes #3081 (didn't test).
2019-09-03 16:03:49 +02:00
Eelco Dolstra
f186000367
Add some noexcepts
This is to assert that callback functions should never throw (since
the context in which they're called may not be able to handle the
exception).
2019-09-03 13:45:35 +02:00
Eelco Dolstra
7348653ff4
Ensure that Callback is called only once
Also, make Callback movable but uncopyable.
2019-09-03 13:45:35 +02:00
Eelco Dolstra
8c4ea7a451
Downloader: Remove a possible double call to Callback 2019-09-03 13:45:32 +02:00
Eelco Dolstra
84de821004
Merge pull request #3069 from matthewbauer/max-name
Set maximum name length in Nix
2019-08-29 15:22:36 +02:00
Eelco Dolstra
a2c4fcd5e9 Don't rely on st_blocks
It doesn't seem very reliable on ZFS.
2019-08-29 14:49:58 +02:00
Eelco Dolstra
f27e53f77e Cleanup 2019-08-29 12:09:58 +02:00
Matthew Bauer
693e68e09c Set maximum name length in Nix
Previously we allowed any length of name for Nix derivations. This is
bad because different file systems have different max lengths. To make
things predictable, I have picked a max. This was done by trying to
build this derivation:

  derivation {
    name = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
    builder = "/no-such-path";
    system = "x86_64-linux";
  }

Take off one a and it will not lead to file name too long. That ends
up being 212 a’s. An even smaller max could be picked if we want to
support more file systems.

Working backwards, this is why:

aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-${name}.drv.chroot

> 255 - 32 - 1 - 4 - 7 = 211
2019-08-28 12:32:54 -04:00
Eelco Dolstra
7298a38a07
Don't send certain setting overrides to the daemon
These are already handled separately. This fixes warnings like

  warning: ignoring the user-specified setting 'max-jobs', because it is a restricted setting and you are not a trusted user

when using the -j flag.
2019-08-28 16:29:44 +02:00
Eelco Dolstra
e5b397b2c7 Merge branch 'test-sandboxing' of https://github.com/matthewbauer/nix 2019-08-27 20:58:47 +02:00
Matthew Bauer
5c06a8d328 Reset tmpDirInSandbox for unsandboxed 2019-08-23 20:24:39 -04:00
Eelco Dolstra
35ebae198f
Merge pull request #3031 from grahamc/low-speed-limit
conf: stalled-download-timeout: make tunable
2019-08-08 22:06:26 +02:00
Graham Christensen
a02457db71
conf: stalled-download-timeout: make tunable
Make curl's low speed limit configurable via stalled-download-timeout.
Before, this limit was five minutes without receiving a single byte.
This is much too long as if the remote end may not have even
acknowledged the HTTP request.
2019-08-08 10:22:13 -04:00
Eelco Dolstra
f9021c4c6c
Merge pull request #3030 from dtzWill/fix/missing-include-ocloexec
pathlocks: add include to fcntl.h for O_CLOEXEC
2019-08-07 22:03:09 +02:00
Eelco Dolstra
56df30cd3f
Merge pull request #2995 from tweag/post-build-hook
Add a post build hook
2019-08-07 15:02:29 +02:00
Will Dietz
c3fefd1a6e
pathlocks: add include to fcntl.h for O_CLOEXEC 2019-08-07 07:41:22 -05:00
Eelco Dolstra
399b6f3c46
nix-store --verify: Don't repair while holding the GC lock 2019-08-02 18:48:26 +02:00
Eelco Dolstra
a2597d5f27
Simplify
With BSD locks we don't have to guard against reading our own
temproots.
2019-08-02 18:39:16 +02:00
Eelco Dolstra
e349f2c0a3
Use BSD instead of POSIX file locks
POSIX file locks are essentially incompatible with multithreading. BSD
locks have much saner semantics. We need this now that there can be
multiple concurrent LocalStore::buildPaths() invocations.
2019-08-02 18:39:16 +02:00
Eelco Dolstra
ec415d7166
Add a test for auto-GC
This currently fails because we're using POSIX file locks. So when the
garbage collector opens and closes its own temproots file, it causes
the lock to be released and then deleted by another GC instance.
2019-08-02 18:39:16 +02:00
regnat
7c5596734f
Add a post-build-hook
Passing `--post-build-hook /foo/bar` to a nix-* command will cause
`/foo/bar` to be executed after each build with the following
environment variables set:

    DRV_PATH=/nix/store/drv-that-has-been-built.drv
    OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev

This can be useful in particular to upload all the builded artifacts to
the cache (including the ones that don't appear in the runtime closure
of the final derivation or are built because of IFD).

This new feature prints the stderr/stdout output to the `nix-build`
and `nix build` client, and the output is printed in a Nix 2
compatible format:

    [nix]$ ./inst/bin/nix-build ./test.nix
    these derivations will be built:
      /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv
    building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'...
    hello!
    bye!
    running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'...
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + printf 'very important stuff'
    /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation

    [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix
    my-example-derivation> hello!
    my-example-derivation> bye!
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + printf 'very important stuff'
    [1 built, 0.0 MiB DL]

Co-authored-by: Graham Christensen <graham@grahamc.com>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-08-02 10:48:15 -04:00
Matthew Bauer
9a0855bbb6 Don’t rely on EPERM
startProcess does not appear to send the exit code to the helper
correctly. Not sure why this is, but it is probably safe to just
fallback on all sandbox errors.
2019-07-30 17:53:37 -04:00
Bas van Dijk
ee1e3132ca Disable findRuntimeRoots on darwin when running tests because lsof is slow
See: https://github.com/NixOS/nix/issues/3011
2019-07-30 11:29:03 +02:00
Matthew Bauer
11d8534629 Use sandbox fallback when cloning fails in builder
When sandbox-fallback = true (the default), the Nix builder will fall
back to disabled sandbox mode when the kernel doesn’t allow users to
set it up. This prevents hard errors from occuring in tricky places,
especially the initial installer. To restore the previous behavior,
users can set:

  sandbox-fallback = false

in their /etc/nix/nix.conf configuration.
2019-07-25 14:42:30 -04:00
Matthew Bauer
d171090530 Disable CLONE_NEWUSER when it’s unavailable
Some kernels disable "unpriveleged user namespaces". This is
unfortunate, but we can still use mount namespaces. Anyway, since each
builder has its own nixbld user, we already have most of the benefits
of user namespaces.
2019-07-25 14:42:25 -04:00
Eelco Dolstra
2f853b20df
Merge pull request #2975 from matthewbauer/fix-nsswitch-issue
Don’t use entire /etc/nsswitch.conf file
2019-07-13 17:08:02 +02:00
Eelco Dolstra
53247d6b11
Resume NAR downloads
This is a much simpler fix to the 'error 9 while decompressing xz
file' problem than 78fa47a7f0. We just
do a ranged HTTP request starting after the data that we previously
wrote into the sink.

Fixes #2952, #379.
2019-07-10 23:12:17 +02:00
Eelco Dolstra
00f6fafad6
HttpBinaryCacheStore: Use default number of retries for NARs 2019-07-10 23:05:04 +02:00
Eelco Dolstra
f76b2a7fdd
Downloader: Use warn() 2019-07-10 22:27:50 +02:00
Eelco Dolstra
03f09e1d18
Revert "Fix 'error 9 while decompressing xz file'"
This reverts commit 78fa47a7f0.
2019-07-10 19:46:15 +02:00
Eelco Dolstra
7e1c85c5fb
Merge pull request #2779 from LnL7/build-exit-codes
build: add exit code for hash and check mismatches
2019-07-02 17:37:49 +02:00
Eelco Dolstra
db700f730e
Merge pull request #2974 from grahamc/invalid-name
checkStoreName: give more precise/verbose error information
2019-07-02 16:12:01 +02:00
Graham Christensen
17d3ec3405
checkStoreName: give more precise/verbose error information
$ sudo ./inst/bin/nix-instantiate -E '"${./.git}"'
error: The path name '.git' is invalid: it is illegal to start the
name with a period. Path names are alphanumeric and can include the
symbols +-._?= and must not begin with a period. Note: If '.git' is a
source file and you cannot rename it on disk,
builtins.path { name = ... } can be used to give it an alternative
name.
2019-07-02 08:41:53 -04:00
Daiderd Jordan
a52c331edb
build: replace 100 offset for build exit codes 2019-07-02 00:12:38 +02:00
Daiderd Jordan
cbf84bcce7
build: use binary mask for build status flags
If multiple builds with fail with different errors it will be reflected
in the status code.

eg.

	103 => timeout + hash mismatch
	105 => timeout + check mismatch
	106 => hash mismatch + check mismatch
	107 => timeout + hash mismatch + check mismatch
2019-07-02 00:12:38 +02:00
Daiderd Jordan
97baf32fbc
build: add exit code for hash and check mismatches
Makes it easier to identify the failure reason in other tooling, eg.
differentiate between a non-deterministic --check vs a failed build.

	$ nix-build '<nix/fetchurl.nix>' --argstr url http://example.org --argstr sha256 0000000000000000000000000000000000000000000000000000
	hash mismatch in fixed-output derivation '/nix/store/nzi9ck45rwlxzcwr25is7qlf3hs5xl83-example.org':
	  wanted: sha256:0000000000000000000000000000000000000000000000000000
	  got:    sha256:08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm
	$ echo $?
	102

	$ nix-build -E 'with import <nixpkgs> {}; runCommand "foo" {} "date +%s > $out"' --check
	warning: rewriting hashes in '/nix/store/g3k47g0399fvjmbm0p0mnad74k4w8vkz-foo'; cross fingers
	error: derivation '/nix/store/mggc8dz13ackb49qca6m23zq4fpq132q-foo.drv' may not be deterministic: output '/nix/store/g3k47g0399fvjmbm0p0mnad74k4w8vkz-foo' differs
	$ echo $?
	104
2019-07-02 00:12:34 +02:00
Aniket Deshpande
ec58ba38c5
Fix http2 = false having no effect. Fixes #2971.
Setting `http2 = false` in nix config (e.g. /etc/nix/nix.conf)
had no effect, and `nix-env -vvvvv -i hello` still downloaded .nar
packages using HTTP/2.

In `src/libstore/download.cc`, the `CURL_HTTP_VERSION_2TLS` option was
being explicitly set when `downloadSettings.enableHttp2` was `true`,
but, `CURL_HTTP_VERSION_1_1` option was not being explicitly set when
`downloadSettings.enableHttp2` was `false`.

This may be because `https://curl.haxx.se/libcurl/c/libcurl-env.html` states:
"You have to set this option if you want to use libcurl's HTTP/2 support."
but, also, in the changelog, states:
"DEFAULT
Since curl 7.62.0: CURL_HTTP_VERSION_2TLS
Before that: CURL_HTTP_VERSION_1_1"

So, the default setting for `libcurl` is HTTP/2 for version >= 7.62.0.

In this commit, option `CURLOPT_HTTP_VERSION` is explicitly set to
`CURL_HTTP_VERSION_1_1` when `downloadSettings.enableHttp2` nix config
setting is `false`.

This can be tested by running `nix-env -vvvvv -i hello | grep HTTP`
2019-06-28 20:44:46 +05:30
Matthew Bauer
ec0087df0a Don’t use entire /etc/nsswitch.conf file
The default nsswitch.conf(5) file in most distros can handle many
different things including host name, user names, groups, etc. In Nix,
we want to limit the amount of impurities that come from these things.
As a result, we should only allow nss to be used for gethostbyname(3)
and getservent(3).

/cc @Ericson2314
2019-06-27 14:31:29 -04:00
Eelco Dolstra
88571219d9
nix-channel: Don't fetch binary-cache-url
This has been ignored since the Perl->C++ rewrite.
2019-06-25 13:27:16 +02:00
Eelco Dolstra
64ec087f58
Fix 32-bit overflow with --no-net
--no-net causes tarballTtl to be set to the largest 32-bit integer,
which causes comparison like 'time + tarballTtl < other_time' to
fail on 32-bit systems. So cast them to 64-bit first.

https://hydra.nixos.org/build/95076624
(cherry picked from commit 29ccb2e969)
2019-06-24 22:16:43 +02:00
Eelco Dolstra
f8b30338ac
Refactor downloadCached() interface
(cherry picked from commit df3f5a78d5)
2019-06-24 22:12:26 +02:00