Commit graph

2947 commits

Author SHA1 Message Date
Eelco Dolstra
dd77f7d593 Store::computeFSClosure(): Support a set of paths
This way, callers can exploits the parallelism of computeFSClosure()
when they have multiple paths that they need the (combined) closure of.
2016-11-10 17:45:04 +01:00
Shea Levy
bff3ad767e build-remote: replace strtoull with stoull to take advantage of C++ error handling 2016-11-10 11:09:15 -05:00
Shea Levy
167d12b02c build-remote: Implement in C++ 2016-11-10 11:09:15 -05:00
Eelco Dolstra
e4655b166c copyClosure() / copyStorePath(): Expose dontCheckSigs
Needed by Hydra.
2016-11-09 19:08:20 +01:00
Eelco Dolstra
4b8f1b0ec0 Merge branch 'ssh-store' of https://github.com/shlevy/nix 2016-11-09 18:57:22 +01:00
Eelco Dolstra
21c55ab3b5 Implement backwards-compatible RemoteStore::addToStore()
The SSHStore PR adds this functionality to the daemon, but we have to
handle the case where the Nix daemon is 1.11.

Also, don't require signatures for trusted users. This restores 1.11
behaviour.

Fixes https://github.com/NixOS/hydra/issues/398.
2016-11-09 18:45:06 +01:00
Eelco Dolstra
a83b10f84c Typo 2016-11-08 20:19:02 +01:00
Eelco Dolstra
b3ba762dbf Add missing #include 2016-11-07 14:35:47 +01:00
Vladimír Čunát
818aad3ec4
Detect and disallow base32 hash overflow
Example (before this commit):
$ nix-hash --type sha256 --to-base16 4n0igfxbd3kqvvj2k2xgysrp63l4v2gd110fwkk4apfpm0hvzwh0 \
    | xargs nix-hash --type sha256 --to-base32
0n0igfxbd3kqvvj2k2xgysrp63l4v2gd110fwkk4apfpm0hvzwh0

It's a real-life example:
https://github.com/NixOS/nixpkgs/pull/20208/files#r86695567
2016-11-06 22:17:37 +01:00
Eelco Dolstra
18b7363a69 Support optional sandbox paths
For example, you can now set

  build-sandbox-paths = /dev/nvidiactl?

to specify that /dev/nvidiactl should only be mounted in the sandbox
if it exists in the host filesystem. This is useful e.g. for EC2
images that should support both CUDA and non-CUDA instances.
2016-10-31 17:09:52 +01:00
Shea Levy
c4969aebaf Add nix.conf options for -k and -K
Fixes #1084
2016-10-27 12:49:34 -04:00
Tuomas Tynkkynen
f0fc3dd88b Fix SIGFPE from integer overflow during division
On some architectures (like x86_64 or i686, but not ARM for example)
overflow during integer division causes a crash due to SIGFPE.
Reproduces on a 64-bit system with:

    nix-instantiate --eval -E '(-9223372036854775807 - 1) / -1'

The only way this can happen is when the smallest possible integer is
divided by -1, so just special-case that.
2016-10-26 18:41:59 +03:00
Eelco Dolstra
5c0bd51d49 Fix build log output in nix-store --serve 2016-10-26 17:13:38 +02:00
Eelco Dolstra
d2c58ba605 Restore the CachedFailure status code
The removal of CachedFailure caused the value of TimedOut to change,
which broke timed-out handling in Hydra (so timed-out builds would
show up as "aborted" and would be retried, e.g. at
http://hydra.nixos.org/build/42537427).
2016-10-26 15:02:38 +02:00
Eelco Dolstra
fdbbcc4492 Remove addPathToAccessor 2016-10-21 18:09:30 +02:00
Eelco Dolstra
542ae5c8f8 BinaryCacheStore: Optionally write a NAR listing
The store parameter "write-nar-listing=1" will cause BinaryCacheStore
to write a file ‘<store-hash>.ls.xz’ for each ‘<store-hash>.narinfo’
added to the binary cache. This file contains an XZ-compressed JSON
file describing the contents of the NAR, excluding the contents of
regular files.

E.g.

  {
    "version": 1,
    "root": {
      "type": "directory",
      "entries": {
        "lib": {
          "type": "directory",
          "entries": {
            "Mcrt1.o": {
              "type": "regular",
              "size": 1288
            },
            "Scrt1.o": {
              "type": "regular",
              "size": 3920
            },
          }
        }
      }
      ...
    }
  }

(The actual file has no indentation.)

This is intended to speed up the NixOS channels programs index
generator [1], since fetching gazillions of large NARs from
cache.nixos.org is currently a bottleneck for updating the regular
(non-small) channel.

[1] https://github.com/NixOS/nixos-channel-scripts/blob/master/generate-programs-index.cc
2016-10-21 16:50:28 +02:00
Eelco Dolstra
307cc8c33d Merge branch 'priorityqueue' of https://github.com/groxxda/nix 2016-10-19 16:37:04 +02:00
Eelco Dolstra
795d9b8668 Merge branch 'regex' of https://github.com/groxxda/nix 2016-10-19 16:32:24 +02:00
Alexander Ried
c935e8eeaf fixup! replace own regex class with std::regex 2016-10-19 16:03:53 +02:00
Eelco Dolstra
19c278de89 Fix Darwin build
Done slightly differently from https://github.com/NixOS/nix/pull/1093.
2016-10-19 15:21:18 +02:00
Eelco Dolstra
29727ff944 Shut up clang warning 2016-10-19 15:02:48 +02:00
Eelco Dolstra
e5346c54df Fix uninitialised variable 2016-10-19 15:02:38 +02:00
Alexander Ried
b05b98df75 replace own regex class with std::regex 2016-10-18 20:22:25 +02:00
Alexander Ried
efb938468c downloader: use priority_queue 2016-10-18 15:45:51 +02:00
Eelco Dolstra
ae8884b949 Fix /dev/ptmx in sandboxes
This was broken since ff0c0b645c. Since
I can't figure out how to mount a devpts instance in the sandbox,
let's just bind-mount the host devpts.
2016-10-14 15:42:13 +02:00
Eelco Dolstra
2f9d60ed7f CURL_AT_LEAST_VERSION -> LIBCURL_VERSION_NUM
http://hydra.nixos.org/build/42025230
2016-10-14 14:44:15 +02:00
Eelco Dolstra
5e61b422c5 Fix HTTP/2 support
Commit 86e8c67efc broke it, because
CURL_* are not actually #defines.
2016-10-13 17:25:42 +02:00
Eelco Dolstra
fb2dd32100 SSL_CERT_FILE -> NIX_SSL_CERT_FILE
This prevents collisions with the "native" OpenSSL, in particular on
OS X.

Fixes #921.
2016-10-13 17:09:10 +02:00
Eelco Dolstra
816d3e5724 Shut up some warnings 2016-10-12 15:53:38 +02:00
Eelco Dolstra
82e2a070e0 Add some functions needed by hydra 2016-10-12 15:49:37 +02:00
Eelco Dolstra
edf9eb8181 querySubstitutablePaths(): Don't query paths for which we already have a substituter 2016-10-07 19:57:47 +02:00
Eelco Dolstra
0c85ef7090 LocalStore::querySubstitutablePaths(): Implement using queryValidPaths() 2016-10-07 19:46:56 +02:00
Eelco Dolstra
844219f364 Store::queryValidPaths(): Use async queryPathInfo()
This allows the binary cache substituter to pipeline requests.
2016-10-07 19:43:36 +02:00
Eelco Dolstra
c663b84573 Implement generic Store::queryValidPaths() 2016-10-07 19:20:47 +02:00
Eelco Dolstra
35db4f65a0 Add copyClosure utility function for Hydra 2016-10-07 19:15:25 +02:00
Eelco Dolstra
629ab80022 importPaths(): Fix accessor support for Hydra 2016-10-07 18:13:40 +02:00
Eelco Dolstra
b0f7f9c98f toJSON(): Support some more types 2016-10-06 17:00:52 +02:00
Eelco Dolstra
ae522f930d Fix getS3Stats() 2016-10-06 17:00:52 +02:00
Eelco Dolstra
cd128f4bad Merge pull request #998 from veprbl/rx_chmod_fix
override rx directory permissions in deletePath()
2016-10-06 11:08:21 +02:00
Eelco Dolstra
c6a929986a Use std::random_device 2016-10-05 21:29:18 +02:00
Eelco Dolstra
86e8c67efc Fix build with older versions of libcurl 2016-10-04 14:43:23 +02:00
Eelco Dolstra
4bd51d74af Handle the case where signed-binary-caches consists of whitespace 2016-09-22 15:59:25 +02:00
Eelco Dolstra
818ab58cc6 Add sandbox-dev-shm-size option
Fixes #1069.
2016-09-21 16:54:53 +02:00
Eelco Dolstra
4546be1b3e nix-build, nix-shell: Don't print error message if nix-store/nix-instantiate fails 2016-09-21 16:54:53 +02:00
Eelco Dolstra
c55bf085eb printMsg(lvlError, ...) -> printError(...) etc. 2016-09-21 16:54:53 +02:00
Eelco Dolstra
4036185cb4 Some notational convenience for formatting strings
We can now write

  throw Error("file '%s' not found", path);

instead of

  throw Error(format("file '%s' not found") % path);

and similarly

  printError("file '%s' not found", path);

instead of

  printMsg(lvlError, format("file '%s' not found") % path);
2016-09-21 16:54:53 +02:00
Eelco Dolstra
beaefdf706 Tweak 2016-09-20 17:49:31 +02:00
Eelco Dolstra
f12d56b27b Improve robustness 2016-09-20 17:25:44 +02:00
Eelco Dolstra
794e4ab475 Fix "Promise already satisfied" error 2016-09-20 17:25:12 +02:00
Eelco Dolstra
4de0639105 nix-shell: Fix $PATH handling in the impure case
We were passing "p=$PATH" rather than "p=$PATH;", resulting in some
invalid shell code.

Also, construct a separate environment for the child rather than
overwriting the parent's.
2016-09-20 15:41:41 +02:00
Eelco Dolstra
9fc4cb2ae9 nix-shell: Restore CPU affinity
Otherwise the shell and its children will be bound to one CPU core...
2016-09-20 15:41:41 +02:00
Eric Litak
0d38b4c792 armv5tel can be built by armv6l and armv7l (#1063) 2016-09-16 22:03:34 +02:00
Eelco Dolstra
75989bdca7 Make computeFSClosure() single-threaded again
The fact that queryPathInfo() is synchronous meant that we needed a
thread for every concurrent binary cache lookup, even though they end
up being handled by the same download thread. Requiring hundreds of
threads is not a good idea. So now there is an asynchronous version of
queryPathInfo() that takes a callback function to process the
result. Similarly, enqueueDownload() now takes a callback rather than
returning a future.

Thus, a command like

  nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5

that returns 4941 paths now takes 1.87s using only 2 threads (the main
thread and the downloader thread). (This is with a prewarmed
CloudFront.)
2016-09-16 18:54:14 +02:00
Eelco Dolstra
054be50257 printMsg(): Don't check for interrupts
Having the logger function potentially throw exceptions is
Heisenbuggy.
2016-09-16 18:52:42 +02:00
Eelco Dolstra
2e1493037b nix path-info: Add some more examples 2016-09-14 18:20:11 +02:00
Eelco Dolstra
6656ef7b5b Revive binary-caches-parallel-connections
It's a slight misnomer now because it actually limits *all* downloads,
not just binary cache lookups.

Also add a "enable-http2" option to allow disabling use of HTTP/2
(enabled by default).
2016-09-14 16:38:26 +02:00
Eelco Dolstra
90ad02bf62 Enable HTTP/2 support
The binary cache store can now use HTTP/2 to do lookups. This is much
more efficient than HTTP/1.1 due to multiplexing: we can issue many
requests in parallel over a single TCP connection. Thus it's no longer
necessary to use a bunch of concurrent TCP connections (25 by
default).

For example, downloading 802 .narinfo files from
https://cache.nixos.org/, using a single TCP connection, takes 11.8s
with HTTP/1.1, but only 0.61s with HTTP/2.

This did require a fairly substantial rewrite of the Downloader class
to use the curl multi interface, because otherwise curl wouldn't be
able to do multiplexing for us. As a bonus, we get connection reuse
even with HTTP/1.1. All downloads are now handled by a single worker
thread. Clients call Downloader::enqueueDownload() to tell the worker
thread to start the download, getting a std::future to the result.
2016-09-14 16:36:02 +02:00
Eelco Dolstra
a75d11a7e6 Add a toLower utility function 2016-09-14 14:58:49 +02:00
Eelco Dolstra
e07c0dcf5c Move some .drv parsing functions out of util 2016-09-14 10:54:57 +02:00
Shea Levy
b99c6e0e29 nix-daemon: Fix error message 2016-09-12 08:09:41 -04:00
Shea Levy
196815f700 ssh-store: Start master on-demand 2016-09-12 08:07:50 -04:00
Shea Levy
ab31f9986c Inline ssh-store.hh into ssh-store.cc 2016-09-12 08:03:29 -04:00
Eelco Dolstra
e5949b5ce8 Fix build on GCC 4.9
GCC 4.9 doesn't like reassigning a std::stringstream.

http://hydra.nixos.org/build/40371644
2016-09-12 13:22:23 +02:00
Eelco Dolstra
5039d3b9de Write "T x" instead of "auto x = T"
That's just silly. Hopefully this also fixes the Debian build failure:

http://hydra.nixos.org/build/40371644
2016-09-12 12:06:13 +02:00
Eelco Dolstra
ff0c0b645c Don't run builds as root in the user namespace
This largely reverts c68e5913c7. Running
builds as root breaks "cp -p", since when running as root, "cp -p"
assumes that it can succesfully chown() files. But that's not actually
the case since the user namespace doesn't provide a complete uid
mapping. So it barfs with a fatal error message ("cp: failed to
preserve ownership for 'foo': Invalid argument").
2016-09-08 18:16:23 +02:00
Eelco Dolstra
e4bdd49022 Give a better error message if a path in build-sandbox-paths is invalid 2016-09-08 17:30:25 +02:00
Eelco Dolstra
0babc4ff31 Fix assertion failure after early build setup failure 2016-09-08 17:29:50 +02:00
Shea Levy
a5f2750ebb Fix early removal of rc-file for nix-shell
BASH_ENV causes all non-interactive shells called via eg. /etc/bashrc to
remove the rc-file before the main shell gets to run it. Completion
scripts will often do this. Fixes #976.

Adapted from and fixes #1034.
2016-09-08 09:09:52 -04:00
Shea Levy
584f8a62de Implement nar-based addToStore for remote-store 2016-09-02 14:33:58 -04:00
Shea Levy
ecba88de93 Add ssh store implementation 2016-09-02 14:31:38 -04:00
Shea Levy
b4b5e9ce2f Add narFromPath op to nix daemon 2016-09-02 14:26:02 -04:00
Shea Levy
a705e8ce0a Factor a general remote FS accessor out of BinaryCacheStore 2016-09-02 14:24:34 -04:00
Shea Levy
0f39633290 Factor out the unix domain socket-specific code from RemoteStore 2016-09-02 14:15:04 -04:00
Shea Levy
7d4ccd9b17 nix-daemon: add --stdio flag for handling connections over stdin/stdout 2016-09-02 13:30:28 -04:00
Shea Levy
53b27ddce2 Factor a function to get the store type from a URI out of the main RegisterStoreImplementation 2016-09-02 06:39:29 -04:00
Shea Levy
a91954f0c6 Merge openStore and openStoreAt with default arguments 2016-09-02 06:35:48 -04:00
Shea Levy
87b189c2b3 Merge branch 'nix-build-c++' 2016-08-31 12:10:21 -04:00
Shea Levy
821380c77b nix-build: Clean up a bit 2016-08-31 10:08:00 -04:00
Shea Levy
dfe0938614 download.hh: Fix conflicts from nix-channel-c++ merge 2016-08-31 09:57:56 -04:00
Shea Levy
572aba284a Merge branch 'nix-channel-c++' 2016-08-31 09:49:24 -04:00
Eelco Dolstra
aa1ea0d1e4 Merge pull request #1030 from pikajude/rsd-cc
Implement resolve-system-dependencies in C++
2016-08-31 15:19:10 +02:00
Eelco Dolstra
d74c8a3f4e Fix 32-bit build 2016-08-30 17:38:09 +02:00
Eelco Dolstra
6631a6e1a1 Increase the sleep time between download retries 2016-08-30 15:48:24 +02:00
Eelco Dolstra
97b1af1cbe Fix --fallback
This fixes an assertion failure in "assert(goal);" in
Worker::waitForInput() after a substitution goal is cancelled by the
termination of another goal. The problem was the line

  //worker.childTerminated(shared_from_this()); // FIXME

in the SubstitutionGoal destructor. This was disabled because
shared_from_this() obviously doesn't work from a destructor. So we now
use a real pointer for object identity.
2016-08-30 15:45:39 +02:00
Eelco Dolstra
67364a5795 Fix Fedora build 2016-08-30 13:56:22 +02:00
Eelco Dolstra
92d917b71a Fix GC build 2016-08-30 13:12:12 +02:00
Eelco Dolstra
c42d1acfeb forceBool(): Show position info 2016-08-29 19:37:19 +02:00
Eelco Dolstra
26d92017d3 Add builtin function "partition"
The implementation of "partition" in Nixpkgs is O(n^2) (because of the
use of ++), and for some reason was causing stack overflows in
multi-threaded evaluation (not sure why).

This reduces "nix-env -qa --drv-path" runtime by 0.197s and memory
usage by 298 MiB (in non-Boehm mode).
2016-08-29 19:36:54 +02:00
Eelco Dolstra
c0a7b84748 nix path-info: Add --json flag
Also, factor out JSON generation from value-to-json.{cc,hh}, and
support producing indented JSON.
2016-08-29 17:29:24 +02:00
Shea Levy
9fa21765e7 callFunction: Copy functors to the heap
Normally it's impossible to take a reference to the function passed to
callFunction, so some callers (e.g. ExprApp::eval) allocate that value
on the stack. For functors, a reference to the functor itself may be
kept, so we need to have it on the heap.

Fixes #1045
2016-08-29 07:36:28 -04:00
Eelco Dolstra
a9d4f2b303 Fix queryPathFromHashPart()
The inner lambda was returning a SQLite-internal char * rather than a
std::string, leading to Hydra errors liks

  Caught exception in Hydra::Controller::Root->narinfo "path ‘ø˜£â€™ is not in the Nix store at /nix/store/6mvvyb8fgwj23miyal5mdr8ik4ixk15w-hydra-0.1.1234.abcdef/libexec/hydra/lib/Hydra/Controller/Root.pm line 352."
2016-08-24 14:53:33 +02:00
Eelco Dolstra
d74236d1f2 nix build: Use Nix search path
That is, unless --file is specified, the Nix search path is
synthesized into an attribute set. Thus you can say

  $ nix build nixpkgs.hello

assuming $NIX_PATH contains an entry of the form "nixpkgs=...". This
is more verbose than

  $ nix build hello

but is less ambiguous.
2016-08-23 17:11:19 +02:00
Jude Taylor
50c3b5df32 iterate through fat_headers correctly 2016-08-17 10:24:11 -07:00
Eelco Dolstra
22d6e31fc6 Add a mechanism for derivation attributes to reference the derivation's outputs
For example, you can now say:

  configureFlags = "--prefix=${placeholder "out"} --includedir=${placeholder "dev"}";

The strings returned by the ‘placeholder’ builtin are replaced at
build time by the actual store paths corresponding to the specified
outputs.

Previously, you had to work around the inability to self-reference by doing stuff like:

  preConfigure = ''
    configureFlags+=" --prefix $out --includedir=$dev"
  '';

or rely on ad-hoc variable interpolation semantics in Autoconf or Make
(e.g. --prefix=\$(out)), which doesn't always work.
2016-08-17 17:19:32 +02:00
Jude Taylor
ca0bce2851 remove double casting 2016-08-16 14:09:57 -07:00
Jude Taylor
cfb77d6e5b account for unknown magic numbers 2016-08-16 12:52:49 -07:00
Jude Taylor
90516c5a7b switch to mmap strategy 2016-08-16 12:45:55 -07:00
Eelco Dolstra
ac841a4679 Merge pull request #1031 from abbradar/replacestrings-context
Allow contexted strings in replaceStrings
2016-08-16 12:47:20 +02:00
Nikolay Amiantov
f1b8dd43be Allow contexted strings in replaceStrings 2016-08-16 13:20:56 +03:00
Jude Taylor
7b006122ae ensure presence of directories 2016-08-15 10:46:21 -07:00
Jude Taylor
adf0216d98 code review comments 2016-08-15 10:43:14 -07:00
Shea Levy
d52d391164 builtins.fetch{url,tarball}: Allow name attribute 2016-08-15 07:37:11 -04:00
Jude Taylor
60f4b25d7d make inclusion conditional 2016-08-14 19:10:38 -07:00
Jude Taylor
f37b6fd07e add a missing load command 2016-08-14 18:54:40 -07:00
Jude Taylor
596e4a5693 remove old traces of resolve-system-dependencies 2016-08-13 15:27:49 -07:00
Jude Taylor
2df9a972fc resolve-system-dependencies: implement in C++ 2016-08-13 11:36:22 -07:00
Shea Levy
59124228b3 nix-channel: implement in c++ 2016-08-11 11:34:43 -04:00
Vladimír Čunát
1bffd83e1a nix-build: extend the meaning of $IN_NIX_SHELL
An equivalent was originally filed against the perl version:
https://github.com/NixOS/nix/pull/933
2016-08-11 17:14:09 +02:00
Eelco Dolstra
b0ebad88cd Merge branch 'buildenv-c++' of https://github.com/shlevy/nix 2016-08-11 17:11:17 +02:00
Shea Levy
91978e3b9a buildenv: perl -> c++ 2016-08-11 07:58:33 -04:00
Eelco Dolstra
66adbdfd97 HttpBinaryCacheStore: Retry on transient HTTP errors
This makes us more robust against 500 errors from CloudFront or S3
(assuming the 500 error isn't cached by CloudFront...).
2016-08-10 18:08:23 +02:00
Eelco Dolstra
9204ea7294 Show HTTP error codes 2016-08-10 18:05:35 +02:00
Eelco Dolstra
f294623d1d SQLite:: Add some convenience 2016-08-10 18:05:35 +02:00
Eelco Dolstra
6cb4bdf152 FSAccessor: Throw InvalidPath 2016-08-10 18:05:35 +02:00
Eelco Dolstra
f43823f676 NarAccessor: Fix handling of non-executable files 2016-08-10 18:05:35 +02:00
Eelco Dolstra
d961c29c9c Mark content-addressed paths in the Nix database and in .narinfo
This allows such paths to be imported without signatures.
2016-08-10 18:05:35 +02:00
Eelco Dolstra
36a51ecab3 LocalFSStore::narFromPath(): Handle the diverted store case 2016-08-10 18:05:35 +02:00
Eelco Dolstra
3eb6217508 Add a "root" parameter to local stores
This makes it easier to create a diverted store, i.e.

  NIX_REMOTE="local?root=/tmp/root"

instead of

  NIX_REMOTE="local?real=/tmp/root/nix/store&state=/tmp/root/nix/var/nix" NIX_LOG_DIR=/tmp/root/nix/var/log
2016-08-10 18:05:35 +02:00
Eelco Dolstra
2fad86f361 Remove $NIX_DB_DIR
This variable has no reason to exist, given $NIX_STATE_DIR.
2016-08-10 18:05:35 +02:00
Eelco Dolstra
be64fbb501 RemoteStore / nix-daemon: Drop support for Nix < 1.0 2016-08-10 18:05:35 +02:00
Shea Levy
80ebc553ec nix-build: Port to c++
This was a dumb line-for-line rewrite, because nix build/nix run/etc.
will replace it.
2016-08-09 07:42:20 -04:00
Eelco Dolstra
ee22a91ab8 makeFixedOutputPath(): Drop superfluous HashType argument 2016-07-26 21:25:52 +02:00
Eelco Dolstra
06bbfb6004 builtins.{fetchurl,fetchTarball}: Support a sha256 attribute
Also, allow builtins.{fetchurl,fetchTarball} in restricted mode if a
hash is specified.
2016-07-26 21:16:52 +02:00
Dmitry Kalinkin
f91748ba73 override rx directory permissions in deletePath()
This fixes instantiation of pythonPackages.pytest that produces a
directory with less permissions during one of it's tests that leads to
a nix error like:

error: opening directory ‘/tmp/nix-build-python2.7-pytest-2.9.2.drv-0/pytest-of-user/pytest-0/testdir/test_cache_failure_warns0/.cache’: Permission denied
2016-07-25 18:11:46 -04:00
Shea Levy
ee3032e4de Merge branch 'find-runtime-roots-c++' 2016-07-24 07:33:49 -04:00
Shea Levy
18b0808475 Respect --keep-going when a substituter fails.
Fixes #977
2016-07-23 09:16:08 -04:00
Eelco Dolstra
d3cd0f5856 Fix assertion failure 2016-07-21 18:39:32 +02:00
Eelco Dolstra
e19c90fc6b Fix random failures caused by CurlDownloader modifying a shared string 2016-07-21 18:35:31 +02:00
Eelco Dolstra
ea8e8df6c7 NarInfo::NarInfo(): Ensure that we get a NAR size/hash 2016-07-21 18:35:12 +02:00
Eelco Dolstra
e682a8e138 Fix assertion failure in ThreadPool::enqueue() 2016-07-21 18:14:16 +02:00
Eelco Dolstra
d57981bac4 Store::queryMissing(): Use a thread pool
For one particular NixOS configuration, this cut the runtime of
"nix-store -r --dry-run" from 6m51s to 3.4s. It also fixes a bug in
the size calculation that was causing certain paths to be counted
twice, e.g. before:

  these paths will be fetched (1249.98 MiB download, 2995.74 MiB unpacked):

and after:

  these paths will be fetched (1219.56 MiB download, 2862.17 MiB unpacked):
2016-07-21 17:50:39 +02:00
Eelco Dolstra
a728780fbd Store::computeFSClosure(): Use thread pool
This speeds up queries against the binary cache.
2016-07-21 14:13:35 +02:00
Shea Levy
3c46fe62b8 find-runtime-roots: fold into gc.cc 2016-07-21 07:04:41 -04:00
Shea Levy
06068b353d FdSink: Restore move constructor/assignment 2016-07-13 06:27:41 -04:00
Shea Levy
b33e85229d Make Buffered{Source,Sink} move-safe 2016-07-13 06:03:37 -04:00
Shea Levy
cb5e7254b6 Modernize AutoCloseFD 2016-07-11 15:44:44 -04:00
Eelco Dolstra
74dd603495 Re-implement negative binary cache lookup caching 2016-06-20 17:39:05 +02:00
Eelco Dolstra
c68e5913c7 Run builds in a user namespace
This way, all builds appear to have a uid/gid of 0 inside the
chroot. In the future, this may allow using programs like
systemd-nspawn inside builds, but that will require assigning a larger
UID/GID map to the build.

Issue #625.
2016-06-09 18:30:39 +02:00
Eelco Dolstra
202683a4fc Use O_CLOEXEC in most places 2016-06-09 16:37:08 +02:00
Eelco Dolstra
9bdd949cfd Fix "creating statement: table ValidPaths has no column named ultimate" 2016-06-03 16:06:42 +02:00
Eelco Dolstra
5e51ffb1c2 Support sandbox builds by non-root users
This allows an unprivileged user to perform builds on a diverted store
(i.e. where the physical store location differs from the logical
location).

Example:

  $ NIX_LOG_DIR=/tmp/log NIX_REMOTE="local?real=/tmp/store&state=/tmp/var" nix-build -E \
    'with import <nixpkgs> {}; runCommand "foo" { buildInputs = [procps nettools]; } "id; ps; ifconfig; echo $out > $out"'

will do a build in the Nix store physically in /tmp/store but
logically in /nix/store (and thus using substituters for the latter).
2016-06-03 15:59:28 +02:00
Eelco Dolstra
2f8b0e557b Consistent quotes 2016-06-03 14:03:09 +02:00
Eelco Dolstra
080d607975 Don't call shared_from_this() from a destructor 2016-06-03 14:02:44 +02:00
Eelco Dolstra
30c8c3ba2d Fix some more references to storeDir 2016-06-03 14:02:44 +02:00
Eelco Dolstra
a8dfdc52b8 Merge pull request #925 from abbradar/master
Show both cycle ends
2016-06-03 13:58:51 +02:00
Nikolay Amiantov
c87a56f4d0 Show both cycle ends 2016-06-03 14:25:37 +03:00
Eelco Dolstra
75361b6dce Fix narrowing conversion from int64_t to bool
http://hydra.nixos.org/build/36613774
2016-06-03 13:20:46 +02:00
Eelco Dolstra
f8a8b4d8f8 nix run: Set a reasonable uid/gid 2016-06-02 19:04:09 +02:00
Eelco Dolstra
d64e0c1b64 Make derivationFromPath work on diverted stores 2016-06-02 18:43:36 +02:00
Eelco Dolstra
eda2aaae92 nix run: Handle the case where the /nix/store mount point doesn't exist 2016-06-02 18:24:51 +02:00
Eelco Dolstra
a424ab0444 createTempDir(): Don't do a chown on Linux
It's not needed and can cause problems in a user namespace.
2016-06-02 18:17:30 +02:00
Eelco Dolstra
a24f2c9b84 nix run: Mount the Nix store in a private namespace
This is a convenience command to allow users who are not privileged to
create /nix/store to use Nix with regular binary caches. For example,

  $ NIX_REMOTE="local?state=$HOME/nix/var&real=/$HOME/nix/store" nix run firefox bashInteractive

will download Firefox and bash from cache.nixos.org, then start a
shell in which $HOME/nix/store is mounted on /nix/store.
2016-06-02 16:51:43 +02:00
Eelco Dolstra
6f2d51287c Add basic "nix run" command 2016-06-02 16:29:49 +02:00
Eelco Dolstra
4494000e04 LocalStore: Allow the physical and logical store directories to differ
This is primarily to subsume the functionality of the
copy-from-other-stores substituter. For example, in the NixOS
installer, we can now do (assuming we're in the target chroot, and the
Nix store of the installation CD is bind-mounted on /tmp/nix):

  $ nix-build ... --option substituters 'local?state=/tmp/nix/var&real=/tmp/nix/store'

However, unlike copy-from-other-stores, this also allows write access
to such a store. One application might be fetching substitutes for
/nix/store in a situation where the user doesn't have sufficient
privileges to create /nix, e.g.:

  $ NIX_REMOTE="local?state=/home/alice/nix/var&real=/home/alice/nix/store" nix-build ...
2016-06-02 16:02:48 +02:00
Eelco Dolstra
064816ab98 Respect build-use-substitutes 2016-06-02 16:02:48 +02:00
Eelco Dolstra
812c0dfbe2 Allow setting the state directory as a store parameter
E.g. "local?store=/tmp/store&state=/tmp/var".
2016-06-02 16:02:48 +02:00
Eelco Dolstra
f2682e6e18 <nix/fetchurl.nix>: Use tarballs.nixos.org
This ensures that the disappearance of upstream bootstrap tarballs
(e.g. https://github.com/NixOS/nixpkgs-channels/pull/1) doesn't break
stdenv rebuilds.
2016-06-01 17:11:51 +02:00
Eelco Dolstra
dead8189ac Typo 2016-06-01 16:56:22 +02:00
Eelco Dolstra
e24e2caaaf Skip substituters with an incompatible store directory 2016-06-01 16:40:49 +02:00
Eelco Dolstra
cf198952d0 HttpBinaryCacheStore: Fix caching of WantMassQuery
Also, test HttpBinaryCacheStore in addition to LocalBinaryCacheStore.
2016-06-01 16:24:17 +02:00
Eelco Dolstra
7850d3d279 Make the store directory a member variable of Store 2016-06-01 16:24:17 +02:00
Eelco Dolstra
a9fa5e050a Shut up some clang warnings 2016-05-31 13:31:04 +02:00
Eelco Dolstra
10f3a2e5f2 Fix clang build failure
Apparently opinion is divided on whether [[noreturn]] is allowed on a
lambda: http://stackoverflow.com/questions/26888805/how-to-declare-a-lambdas-operator-as-noreturn

http://hydra.nixos.org/build/36462100
2016-05-31 13:23:54 +02:00
Eelco Dolstra
c2d27d30cf nix-copy-closure / build-remote.pl: Disable signature checking
This restores the Nix 1.11 behaviour.
2016-05-31 11:48:05 +02:00
Dmitry Kalinkin
78b00bbd8a use $(LIBLZMA_LIBS) instead of -llzma
This is needed in case of non-standard lzma installation path that will
be specified in pkgconfig manifest as extra -L option for LDFLAGS.
2016-05-31 03:20:11 -04:00
Eelco Dolstra
cd35f0280c Fix build 2016-05-30 16:12:50 +02:00
Eelco Dolstra
57d33013ce Check signatures before downloading the substitute 2016-05-30 15:18:12 +02:00
Eelco Dolstra
bac123ddd9 Test the NAR info cache 2016-05-30 15:18:12 +02:00
Eelco Dolstra
3593c8285d Re-implement binary cache signature checking
This is now done in LocalStore::addToStore(), rather than in the
binary cache substituter (which no longer exists).
2016-05-30 15:18:12 +02:00
Eelco Dolstra
12ddbad458 LocalStore::addToStore: Verify hash of the imported path 2016-05-30 15:18:12 +02:00
Eelco Dolstra
e222484401 Re-implement the WantMassQuery property of binary caches 2016-05-30 15:18:12 +02:00
Eelco Dolstra
b66ab6cdbc Fix repair during substitution 2016-05-30 15:18:12 +02:00
Eelco Dolstra
3be2e71ab3 BinaryCacheStore: Remove buildPaths() / ensurePath() 2016-05-30 15:18:12 +02:00
Eelco Dolstra
75d2492f20 Make the aws-cpp-sdk dependency optional 2016-05-04 17:16:48 +02:00
Eelco Dolstra
6c75cf69c3 Cleanup: Remove singleton() 2016-05-04 16:16:53 +02:00
Eelco Dolstra
0d4a10e910 Do compression in a sink 2016-05-04 16:16:53 +02:00
Eelco Dolstra
c6a21aed07 More GC fixes 2016-05-04 16:16:53 +02:00
Eelco Dolstra
538a64e8c3 Add a Store::addToStore() variant that accepts a NAR
As a side effect, this ensures that signatures are propagated when
copying paths between stores.

Also refactored import/export to make use of this.
2016-05-04 16:15:54 +02:00
Eelco Dolstra
b6c768fb6a nix-daemon: Fix queryPathInfo() backward compatibility 2016-05-04 12:53:24 +02:00
Eelco Dolstra
33de2bc080 Fix segfault in xz compression/decompression 2016-05-04 11:50:12 +02:00
Eelco Dolstra
f435f82475 Remove OpenSSL-based signing 2016-05-04 11:01:48 +02:00
Eelco Dolstra
dfebfc835f Add a copyStorePath() utility function 2016-05-04 11:01:48 +02:00
Eelco Dolstra
d8bf0d4859 Support Git repos in the Nix path
E.g.

  $ nix-build -I nixpkgs=git://github.com/NixOS/nixpkgs '<nixpkgs>' -A hello

This is not extremely useful yet because you can't specify a
branch/revision.
2016-04-29 21:07:47 +02:00
Eelco Dolstra
38539b943a Add fetchgit builtin
The function builtins.fetchgit fetches Git repositories at evaluation
time, similar to builtins.fetchTarball. (Perhaps the name should be
changed, being confusing with respect to Nixpkgs's fetchgit function,
with works at build time.)

Example:

  (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello

or

  (import (builtins.fetchgit {
    url = git://github.com/NixOS/nixpkgs-channels;
    rev = "nixos-16.03";
  }) {}).hello

Note that the result does not contain a .git directory.
2016-04-29 20:47:36 +02:00
Eelco Dolstra
83258225e6 nix-store -l: Simplify implementation 2016-04-29 17:47:03 +02:00
Eelco Dolstra
4dde0b0562 BinaryCacheStore: Support bzip2 compression 2016-04-29 17:43:37 +02:00
Eelco Dolstra
6963de2091 nix verify --all: Support local binary caches 2016-04-29 17:34:31 +02:00
Eelco Dolstra
0dd988d2e3 Better error message 2016-04-29 17:33:22 +02:00
Eelco Dolstra
5acb691402 BinaryCacheStore: Support "none" compression method 2016-04-29 17:02:57 +02:00
Eelco Dolstra
8e065c6b3e BinaryCacheStore: Make the signing key a parameter 2016-04-29 16:47:20 +02:00
Eelco Dolstra
f6aee2f477 BinaryCacheStore::queryPathInfo: Don't check signatures
Other stores don't do this either. It's up to the caller to check
signatures.
2016-04-29 16:28:57 +02:00
Eelco Dolstra
95d20dfde9 Allow parameters in store URIs
This is to allow store-specific configuration,
e.g. s3://my-cache?compression=bzip2&secret-key=/path/to/key.
2016-04-29 16:26:16 +02:00
Eelco Dolstra
aa3bc3d5dc Eliminate the substituter mechanism
Substitution is now simply a Store -> Store copy operation, most
typically from BinaryCacheStore to LocalStore.
2016-04-29 13:57:08 +02:00
Eelco Dolstra
21e9d183cc Really handle carriage return 2016-04-28 14:27:00 +02:00
Eelco Dolstra
ce5776758d Fix error message 2016-04-28 14:12:10 +02:00
Eelco Dolstra
cc804d0dc6 Handle carriage return 2016-04-25 19:18:45 +02:00
Eelco Dolstra
ca9de88a51 nix: Disable verbose builds 2016-04-25 19:18:45 +02:00
Eelco Dolstra
5761827d5b Show the log tail when a build fails
If --no-build-output is given (which will become the default for the
"nix" command at least), show the last 10 lines of the build output if
the build fails.
2016-04-25 19:18:45 +02:00
Eelco Dolstra
6e1b099279 Remove --print-build-trace
This was added to support Hydra, but Hydra no longer uses it.
2016-04-25 19:18:45 +02:00
Eelco Dolstra
9eba2c3945 Fix "path is not in the Nix store" during GC 2016-04-25 19:18:45 +02:00