Commit graph

201 commits

Author SHA1 Message Date
Eelco Dolstra
7348653ff4
Ensure that Callback is called only once
Also, make Callback movable but uncopyable.
2019-09-03 13:45:35 +02:00
regnat
7c5596734f
Add a post-build-hook
Passing `--post-build-hook /foo/bar` to a nix-* command will cause
`/foo/bar` to be executed after each build with the following
environment variables set:

    DRV_PATH=/nix/store/drv-that-has-been-built.drv
    OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev

This can be useful in particular to upload all the builded artifacts to
the cache (including the ones that don't appear in the runtime closure
of the final derivation or are built because of IFD).

This new feature prints the stderr/stdout output to the `nix-build`
and `nix build` client, and the output is printed in a Nix 2
compatible format:

    [nix]$ ./inst/bin/nix-build ./test.nix
    these derivations will be built:
      /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv
    building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'...
    hello!
    bye!
    running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'...
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + printf 'very important stuff'
    /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation

    [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix
    my-example-derivation> hello!
    my-example-derivation> bye!
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + printf 'very important stuff'
    [1 built, 0.0 MiB DL]

Co-authored-by: Graham Christensen <graham@grahamc.com>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-08-02 10:48:15 -04:00
Eelco Dolstra
8f6c72faee
Merge pull request #2810 from NixOS/print-build-logs
nix: Add --print-build-logs flag
2019-05-15 20:38:38 +02:00
Eelco Dolstra
66b8a62101 nix: Add --print-build-logs flag
This causes 'nix' to print build log output to stderr rather than
showing the last log line in the progress bar. Log lines are prefixed
by the name of the derivation (minus the version string), e.g.

  binutils> make[1]: Leaving directory '/build/binutils-2.31.1'
  binutils-wrapper> unpacking sources
  binutils-wrapper> patching sources
  ...
  binutils-wrapper> Using dynamic linker: '/nix/store/kr51dlsj9v5cr4n8700jliyz8v5b2q7q-bootstrap-stage0-glibc/lib/ld-linux-x86-64.so.2'
  bootstrap-stage2-gcc-wrapper> unpacking sources
  ...
  linux-headers> unpacking sources
  linux-headers> unpacking source archive /nix/store/8javli69jhj3bkql2c35gsj5vl91p382-linux-4.19.16.tar.xz
2019-05-15 17:33:56 +02:00
Graham Christensen
b4a05edbfe
runProgram: support gid, uid, chdir 2019-05-12 13:17:27 -04:00
Graham Christensen
6df61db060
diff hook: execute as the build user, and pass the temp dir 2019-05-12 13:17:26 -04:00
Eelco Dolstra
f9a2ea4486
Fix "Bad system call" running i686-linux binaries on x86_64-linux
To determine which seccomp filters to install, we were incorrectly
using settings.thisSystem, which doesn't denote the actual system when
--system is used.

Fixes #2791.
2019-05-03 10:48:33 +02:00
Eelco Dolstra
ef52ccf035
experimental/optional -> optional 2019-03-14 14:10:52 +01:00
Eelco Dolstra
01d07b1e92
Revert "Restore parent mount namespace before executing a child process"
This reverts commit a0ef21262f. This
doesn't work in 'nix run' and nix-shell because setns() fails in
multithreaded programs, and Boehm GC mark threads are uncancellable.

Fixes #2646.
2019-02-05 10:49:19 +01:00
Eelco Dolstra
a32ff2573b
Fix 'Read-only file system' when building a derivation 2018-11-15 13:20:39 +01:00
Eelco Dolstra
a0ef21262f
Restore parent mount namespace before executing a child process
This ensures that they can't write to /nix/store. Fixes #2535.
2018-11-13 16:15:30 +01:00
Linus Heckemann
f3b8173a93 config: use all of XDG_CONFIG_DIRS
Previously, config would only be read from XDG_CONFIG_HOME. This change
allows reading config from additional directories, which enables e.g.
per-project binary caches or chroot stores with the help of direnv.
2018-10-31 09:24:43 +01:00
Eelco Dolstra
7d21863bb3
Make <nix/fetchurl.nix> run in constant memory
E.g.

  nix-build --store ~/my-nix/ -E 'import <nix/fetchurl.nix> { url = https://cache.nixos.org/nar/0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv.nar.xz; sha256 = "0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv"; }'

now runs in 17 MiB (was 70 MiB), while

  nix-build --store ~/my-nix/ -E 'import <nix/fetchurl.nix> { url = https://cache.nixos.org/nar/0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv.nar.xz; sha256 = "0d2fxljdih3nc5dqx41hjzic3141ajil94m8kdbpryq569dpsbvb"; unpack = true; }'

runs in 17 MiB (was 346 MiB).
2018-05-30 13:42:51 +02:00
Eelco Dolstra
08ec757726
Make LocalBinaryCacheStore::narFromPath() run in constant memory
This reduces memory consumption of

  nix copy --from file://... --to ~/my-nix /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79

from 514 MiB to 18 MiB for an uncompressed binary cache, and from 192
MiB to 53 MiB for a bzipped binary cache. It may also be faster
because fetching can happen concurrently with decompression/writing.

Continuation of 48662d151b.

Issue https://github.com/NixOS/nix/issues/1681.
2018-05-30 13:34:57 +02:00
Eelco Dolstra
81ea8bd5ce
Simplify the callback mechanism 2018-05-30 13:34:37 +02:00
Eelco Dolstra
1aca195e52
Hack to get SSH error messages from build-remote
E.g.

  cannot build on 'ssh://mac1': cannot connect to 'mac1': bash: nix-store: command not found
  cannot build on 'ssh://mac2': cannot connect to 'mac2': Host key verification failed.
  cannot build on 'ssh://mac3': cannot connect to 'mac3': Received disconnect from 213... port 6001:2: Too many authentication failures
  Authentication failed.
2018-03-20 15:25:26 +01:00
Will Dietz
9b33201e72 util: rename stdout/stdin members to avoid conflicts w/standard macro
(cherry picked from commit c389a7fb617ed7bcd617efa68c6a48c00405310d)
2018-03-19 11:18:56 -05:00
Eelco Dolstra
3e6b194d78
decompress(): Use a Source and Sink
This allows decompression to happen in O(1) memory.
2018-03-16 20:35:59 +01:00
Eelco Dolstra
4868721506
Filter ANSI colors when not writing to a terminal
Fixes https://github.com/NixOS/nixpkgs/issues/37114.
2018-03-15 16:08:07 +01:00
Eelco Dolstra
24ec750003
nix run: Fix segfault on macOS
Note that clearenv() is not available on macOS.

Fixes #1907.
2018-02-26 18:29:40 +01:00
Eelco Dolstra
84989d3af2
Improve filtering of ANSI escape sequences in build logs
All ANSI sequences except color setting are now filtered out. In
particular, terminal resets (such as from NixOS VM tests) are filtered
out.

Also, fix the completely broken tab character handling.
2018-02-07 15:23:57 +01:00
Eelco Dolstra
d4dcffd643
Add pure evaluation mode
In this mode, the following restrictions apply:

* The builtins currentTime, currentSystem and storePath throw an
  error.

* $NIX_PATH and -I are ignored.

* fetchGit and fetchMercurial require a revision hash.

* fetchurl and fetchTarball require a sha256 attribute.

* No file system access is allowed outside of the paths returned by
  fetch{Git,Mercurial,url,Tarball}. Thus 'nix build -f ./foo.nix' is
  not allowed.

Thus, the evaluation result is completely reproducible from the
command line arguments. E.g.

  nix build --pure-eval '(
    let
      nix = fetchGit { url = https://github.com/NixOS/nixpkgs.git; rev = "9c927de4b179a6dd210dd88d34bda8af4b575680"; };
      nixpkgs = fetchGit { url = https://github.com/NixOS/nixpkgs.git; ref = "release-17.09"; rev = "66b4de79e3841530e6d9c6baf98702aa1f7124e4"; };
    in (import (nix + "/release.nix") { inherit nix nixpkgs; }).build.x86_64-linux
  )'

The goal is to enable completely reproducible and traceable
evaluation. For example, a NixOS configuration could be fully
described by a single Git commit hash. 'nixos-rebuild' would do
something like

  nix build --pure-eval '(
    (import (fetchGit { url = file:///my-nixos-config; rev = "..."; })).system
  ')

where the Git repository /my-nixos-config would use further fetchGit
calls or Git externals to fetch Nixpkgs and whatever other
dependencies it has. Either way, the commit hash would uniquely
identify the NixOS configuration and allow it to reproduced.
2018-01-16 19:23:18 +01:00
Eelco Dolstra
e026bc3b05
fetchMercurial: Don't fetch hashes we already have 2017-11-01 18:43:11 +01:00
Eelco Dolstra
72cd52c3cd
builtins.fetchgit: Support importing a working tree
For example, you can write

  src = fetchgit ./.;

and if ./. refers to an unclean working tree, that tree will be copied
to the Nix store. This removes the need for "cleanSource".
2017-10-30 19:59:25 +01:00
Eelco Dolstra
2d5b1b24bf
Pass lists/attrsets to bash as (associative) arrays 2017-10-25 13:01:50 +02:00
Eelco Dolstra
b7376edf06
ThreadPool: On exception, interrupt the other worker threads 2017-09-08 15:31:24 +02:00
Eelco Dolstra
db1d45037c
Handle SIGWINCH 2017-08-25 15:59:03 +02:00
Andy Wingo
6bb4e3e8fe Remove unused decodeOctalEscaped
Besides being unused, this function has a bug that it will incorrectly
decode the path component Ubuntu\04016.04.2\040LTS\040amd64 as
"Ubuntu.04.2 LTS amd64" instead of "Ubuntu 16.04.2 LTS amd64".
2017-08-18 11:07:00 +02:00
Eelco Dolstra
bf1f123b09
Progress indicator: Show number of active items 2017-08-16 20:56:03 +02:00
Eelco Dolstra
af765a8eab
Use /proc/self/fd to efficiently close all FDs on Linux
Issue #1506.
2017-08-09 16:22:05 +02:00
Eelco Dolstra
b01d62285c
Improve progress indicator 2017-05-16 16:09:57 +02:00
Eelco Dolstra
465cb68244
Figure out the user's home directory if $HOME is not set 2017-05-05 17:08:23 +02:00
Eelco Dolstra
5bd8795e1f
nix repl: Use $XDG_DATA_HOME for the readline history 2017-04-25 18:56:29 +02:00
David McFarland
804ac52489 add helper function to set 'interruptThrown'
this fixes a linker failure on cygwin 64 due to some bad
interaction between tls and shared libraries.

see: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64697
2017-04-21 11:28:14 -03:00
Eelco Dolstra
f05d5f89ff
Read per-user settings from ~/.config/nix/nix.conf 2017-04-20 14:58:16 +02:00
Eelco Dolstra
f8b84a3b8c
Move istringstream_nocopy to a separate file 2017-03-21 14:43:03 +01:00
Eelco Dolstra
20df50d8e1
Merge branch 'darwin-s3-binary-cache-store' of https://github.com/shlevy/nix 2017-03-20 17:46:35 +01:00
Eelco Dolstra
0ec7f47b00
Remove "killing process <pid>" messages
They convey no useful information.
2017-03-16 10:52:28 +01:00
Eelco Dolstra
25dff2b7db
runProgram(): Distinguish between empty input and no input
For example, if we call brotli with an empty input, it shouldn't read
from the caller's stdin.
2017-03-15 16:50:19 +01:00
Adrien Devresse
d853877ce9 Add missing header <sstream> 2017-03-08 22:24:10 +01:00
Shea Levy
4fc30922cf istringstream_nocopy: Implement in a standards-compliant way.
Fixes the problem mentioned in e6a61b8da7

See #1135
2017-03-06 13:03:02 -05:00
Eelco Dolstra
302386f775
Support netrc in <nix/fetchurl.nix>
This allows <nix/fetchurl.nix> to fetch private Git/Mercurial
repositories, e.g.

  import <nix/fetchurl.nix> {
    url = https://edolstra@bitbucket.org/edolstra/my-private-repo/get/80a14018daed.tar.bz2;
    sha256 = "1mgqzn7biqkq3hf2697b0jc4wabkqhmzq2srdymjfa6sb9zb6qs7";
  }

where /etc/nix/netrc contains:

  machine bitbucket.org
  login edolstra
  password blabla...

This works even when sandboxing is enabled.

To do: add unpacking support (i.e. fetchzip functionality).
2017-02-16 15:51:50 +01:00
Eelco Dolstra
7a65b2470e
Restore default signal handling in child processes
In particular, this fixes Ctrl-C in nix-shell sessions.
2017-02-01 13:00:21 +01:00
Eelco Dolstra
83ae6503e8
Fix interrupt handling 2017-01-26 20:40:33 +01:00
Eelco Dolstra
8af062f372 Merge pull request #981 from shlevy/build-remote-c++
build-remote: Implement in C++
2017-01-19 18:21:55 +01:00
Eelco Dolstra
21948deed9
Kill builds when we get EOF on the log FD
This closes a long-time bug that allowed builds to hang Nix
indefinitely (regardless of timeouts) simply by doing

  exec > /dev/null 2>&1; while true; do true; done

Now, on EOF, we just send SIGKILL to the child to make sure it's
really gone.
2017-01-19 17:16:14 +01:00
Eelco Dolstra
63e10b4d28
Cleanup 2017-01-19 17:06:04 +01:00
Eelco Dolstra
cc3b93c991
Handle SIGINT etc. via a sigwait() signal handler thread
This allows other threads to install callbacks that run in a regular,
non-signal context. In particular, we can use this to signal the
downloader thread to quit.

Closes #1183.
2017-01-17 18:21:02 +01:00
Eelco Dolstra
8079ab87a2 AutoCloseDir: Use std::unique_ptr 2017-01-16 22:39:27 +01:00
Eelco Dolstra
2b9d0a99cb AutoDeleteArray -> std::unique_ptr
Also, switch to C++14 for std::make_unique.
2017-01-16 22:24:49 +01:00