Commit graph

4095 commits

Author SHA1 Message Date
volth
841747b0e6
prim_concatMap: allocate intermediate list on stack 2018-07-05 12:37:37 +00:00
volth
ee218f99ca
primops.cc: fix comment 2018-07-05 11:58:15 +00:00
volth
403a76a18f lib.concatMap and lib.mapAttrs to be builtins 2018-07-05 02:54:09 +00:00
Eelco Dolstra
e388739098
nix-daemon: Respect --store
For example, this allows you to do run nix-daemon as a non-privileged
user:

  eelco$ NIX_STATE_DIR=~/my-nix/nix/var nix-daemon --store ~/my-nix/

The NIX_STATE_DIR is still needed because settings.nixDaemonSocketFile
is not derived from settings.storeUri (and we can't derive it from the
store's state directory because we don't want to open the store in the
parent process).
2018-07-03 19:52:32 +02:00
Eelco Dolstra
ea3c9dab5f
Include cpptoml for build simplicity 2018-07-03 18:39:36 +02:00
Eelco Dolstra
3b1f54cf06
Add a fromTOML primop
This is primarily useful for processing Cargo.lock files.
2018-07-03 18:39:36 +02:00
Eelco Dolstra
a92ed973e5
Store floating point numbers in double precision
Even on 32-bit systems, Value has enough space to hold a double.
2018-07-03 18:39:32 +02:00
Maximilian Bosch
8a98ffc276
nix why-depends: render output into $PAGER 2018-07-03 12:49:12 +02:00
Eelco Dolstra
dd98683e65
Merge pull request #2158 from Ma27/improve-search-ux
search.cc: improve UX for `nix search`
2018-07-02 11:16:21 +02:00
Maximilian Bosch
6b74fdac27
search.cc: improve UX for nix search
As proposed in #1634 the `nix search` command could use some
improvements. Initially 0413aeb35d added
some basic sorting behavior using `std::map`, a next step would be an
improvement of the output.

This patch includes the following changes:

* Use `$PAGER` for outputs with `RunPager` from `shared.hh`:
  The same behavior is defined for `nix-env --query`, furthermore it
  makes searching huge results way easier.

* Simplified result blocks:
  The new output is heavily inspired by the output from `nox`, the first
  line shows the attribute path and the derivaiton name
  (`attribute path (derivation name)`) and the description in the second
  line.
2018-07-02 11:05:59 +02:00
Will Dietz
6a24e49ba8 repl: use nix build for building instead of nix-store -r
progress bar!
2018-06-23 07:56:26 -05:00
Will Dietz
44de71a396 progress-bar: re-draw last update if nothing new for 1sec.
Slightly nicer behavior when updates are somewhat far apart
(during a long linking step, perhaps) ensuring things
don't appear unresponsive.

If we wait the maximum amount for the update,
don't bother waiting another 50ms (for rate-limiting purposes)
and just check if we should quit.

This also ensures we'll notice the request to quit within 1s
if quit is signalled but there is not an udpate.
(I'm not sure if this happens or not)
2018-06-18 17:54:09 -05:00
Will Dietz
f601bc0492 sync.hh: return cv_status instead of dropping it, in case useful. 2018-06-18 17:31:00 -05:00
Eelco Dolstra
b920b90857
Detect redirect loops
Fixes #2225.
2018-06-18 10:36:19 +02:00
Eelco Dolstra
2e244fb68f
Merge pull request #2234 from symphorien/esrch2
libstore/gc.cc: ignore ESRCH when reading symlinks in /proc
2018-06-18 09:01:55 +02:00
Symphorien Gibol
ef8122064b libstore/gc.cc: ignore ESRCH when reading symlinks in /proc
readlink is also affected by the problem fixed for regular files in
c567afe355df
2018-06-16 18:04:09 +02:00
Matthew Justin Bauer
6d09e4400c
Fix optimise-store.cc to skip more paths on macOS
I hate to make this such a large check but the lack of documentation means we really have no idea what's allowed. All of them reported so far have been within ".app/Contents" directories. That appears to be a safe starting point. However, I would not be surprised to also find more paths that are disallowed for instance in .framework or .bundle directories.

Fixes #2031
Fixes #2229
2018-06-14 12:34:55 -04:00
Eelco Dolstra
85530bcc0b
nix: Remove special handling of .drv files
This makes 'nix copy' and 'nix path-info' work on .drv store
paths. Removing special treatment of .drv files seems the most
future-proof approach given the possible removal of .drv files in the
future.

Note that 'nix build' will still build (rather than substitute) .drv
paths due to the unfortunate overloading in Store::buildPaths().
2018-06-13 18:04:22 +02:00
Eelco Dolstra
7902fccf6f
hashDerivationModulo(): Handle chroot stores
This makes 'nix copy --to /mnt /nix/store/bla.drv' work.
2018-06-13 18:00:54 +02:00
Eelco Dolstra
3c5d9f478d
std::random_shuffle -> std::shuffle
The former is removed in C++17.
2018-06-13 16:56:19 +02:00
Eelco Dolstra
0629601da1
Move EvalState from the stack to the heap
EvalState contains a few counters (e.g. nrValues) that increase
quickly enough that they end up being interpreted as pointers by the
garbage collector. Moving it to the heap makes them invisible to the
garbage collector.

This reduces the max RSS doing 100 evaluations of
nixos.tests.firefox.x86_64-linux.drvPath from 455 MiB to 292 MiB.

Note: ideally, allocations would be much further up in the 64-bit
address space to reduce the odds of an integer being misinterpreted as
a pointer. Maybe we can use some linker magic to move the .bss segment
to a higher address.
2018-06-12 17:49:55 +02:00
Eelco Dolstra
c905d8b0a8
GC_malloc -> GC_MALLOC
This makes it possible to build with -DGC_DEBUG.
2018-06-12 17:49:55 +02:00
Eelco Dolstra
455d1f01d0
Don't scan for roots in dynamic libraries
This reduces the risk of object liveness misdetection. For example,
Glibc has an internal variable "mp_" that often points to a Boehm
object, keeping it alive unnecessarily. Since we don't store any
actual roots in global variables, we can just disable data segment
scanning.

With this, the max RSS doing 100 evaluations of
nixos.tests.firefox.x86_64-linux.drvPath went from 718 MiB to 455 MiB.
2018-06-12 17:49:55 +02:00
Eelco Dolstra
30964103dc
Add temporary stats 2018-06-12 17:49:55 +02:00
Eelco Dolstra
24c6806994
Cache parse trees
This prevents EvalState::resetFileCache() from parsing everything all
over again.
2018-06-12 17:49:55 +02:00
Eelco Dolstra
6ad0a2f749
Remove duplicate definition of allocBytes() 2018-06-12 17:49:51 +02:00
Yorick van Pelt
72a78beb34
Fix #2162: use getaddrinfo instead of curl to preload NSS 2018-06-12 13:05:14 +02:00
Symphorien Gibol
8c567afe35 libstore/gc.cc: ignore ESRCH when reading /proc
If a process disappears between the time /proc/[pid]/maps is opened and
the time it is read, the read() syscall will return ESRCH. This should be ignored.
2018-06-11 16:29:47 +02:00
Will Dietz
6bf7dc1baf download: if there are active requests, never sleep for 10s
In some versions/configurations libcurl doesn't handle timeouts
(especially DNS timeouts) in a way that wakes curl_multi_wait.

This doesn't appear to be a problem if using c-ares, FWIW.
2018-06-08 21:12:43 -05:00
AmineChikhaoui
170165ee5a
use debug instead of printTalkative to avoid spamming hydra queue
runner logs.
2018-06-08 11:43:32 +01:00
Eelco Dolstra
4ac4f675df
Don't require --fallback to recover from disappeared binary cache NARs 2018-06-05 16:04:41 +02:00
Eelco Dolstra
691b7582c7
Handle HTTP 410 Gone errors 2018-06-05 16:03:32 +02:00
Eelco Dolstra
ca06a9cea7
Improve binary cache fallback test 2018-06-05 15:26:12 +02:00
Eelco Dolstra
a2ec7a3bfd
Further improve upload messages 2018-06-05 14:37:26 +02:00
Will Dietz
54b1c59643 download.cc: fix readCallback return type (int -> size_t)
I'm not sure if curl ever asks for enough data at once
for truncation to occur but better safe than sorry.
2018-06-01 10:59:26 -05:00
Eelco Dolstra
73357500ac
Improve binary cache upload messages
Don't say "download" when we mean "upload".
2018-06-01 14:14:22 +02:00
Peter Simons
93aa3bea2e
Merge pull request #767 from mogorman/garbage_collect_keep_last_few
Implement --delete-generations + flag for keeping last N number of gens
2018-05-31 10:00:21 +02:00
Eelco Dolstra
5d4a9d5677
Fix static assertion failure on 32-bit systems 2018-05-30 17:47:30 +02:00
Eelco Dolstra
3c2de9830d
RemoteStore::addToStore(): Ensure capacity is decreased on exceptions 2018-05-30 13:48:31 +02:00
Eelco Dolstra
a4c1618876
Fix deadlock in RemoteStore::queryPathInfoUncached() 2018-05-30 13:46:14 +02:00
Eelco Dolstra
7d21863bb3
Make <nix/fetchurl.nix> run in constant memory
E.g.

  nix-build --store ~/my-nix/ -E 'import <nix/fetchurl.nix> { url = https://cache.nixos.org/nar/0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv.nar.xz; sha256 = "0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv"; }'

now runs in 17 MiB (was 70 MiB), while

  nix-build --store ~/my-nix/ -E 'import <nix/fetchurl.nix> { url = https://cache.nixos.org/nar/0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv.nar.xz; sha256 = "0d2fxljdih3nc5dqx41hjzic3141ajil94m8kdbpryq569dpsbvb"; unpack = true; }'

runs in 17 MiB (was 346 MiB).
2018-05-30 13:42:51 +02:00
Eelco Dolstra
5a654fd7dd
getDownloader(): Simplify 2018-05-30 13:42:30 +02:00
Eelco Dolstra
e87e4a60d6
Make HttpBinaryCacheStore::narFromPath() run in constant memory
This reduces memory consumption of

  nix copy --from https://cache.nixos.org --to ~/my-nix /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79

from 176 MiB to 82 MiB. (The remaining memory is probably due to xz
decompression overhead.)

Issue https://github.com/NixOS/nix/issues/1681.
Issue https://github.com/NixOS/nix/issues/1969.
2018-05-30 13:42:29 +02:00
Eelco Dolstra
08ec757726
Make LocalBinaryCacheStore::narFromPath() run in constant memory
This reduces memory consumption of

  nix copy --from file://... --to ~/my-nix /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79

from 514 MiB to 18 MiB for an uncompressed binary cache, and from 192
MiB to 53 MiB for a bzipped binary cache. It may also be faster
because fetching can happen concurrently with decompression/writing.

Continuation of 48662d151b.

Issue https://github.com/NixOS/nix/issues/1681.
2018-05-30 13:34:57 +02:00
Eelco Dolstra
81ea8bd5ce
Simplify the callback mechanism 2018-05-30 13:34:37 +02:00
Eelco Dolstra
1672bcd230
Move evaluator-specific settings out of libstore 2018-05-30 13:29:50 +02:00
Eelco Dolstra
c1d445ecec
Move some Download-specific settings to download.cc 2018-05-30 13:29:50 +02:00
Eelco Dolstra
737ed88f35
Modularize config settings
Allow global config settings to be defined in multiple Config
classes. For example, this means that libutil can have settings and
evaluator settings can be moved out of libstore. The Config classes
are registered in a new GlobalConfig class to which config files
etc. are applied.

Relevant to https://github.com/NixOS/nix/issues/2009 in that it
removes the need for ad hoc handling of useCaseHack, which was the
underlying cause of that issue.
2018-05-30 13:28:01 +02:00
Eelco Dolstra
e606cd412f
Fix assertion failure in storePathToHash()
Fixes https://github.com/NixOS/nix/issues/2015.
2018-05-30 13:27:19 +02:00
Eelco Dolstra
6185d25e52
Make 'nix copy --to daemon' run in constant memory (daemon side)
Continuation of 97002b684c. This makes
the daemon use constant memory. For example, it reduces the daemon's
maximum RSS on

  $ nix copy --from ~/my-nix --to daemon /nix/store/1n7x0yv8vq6zi90hfmian84vdhd04bgp-blender-2.79a

from 264 MiB to 7 MiB.

We now use a TunnelSource to prevent the connection from ending up in
an undefined state if an exception is thrown while the NAR is being
sent.

Issue https://github.com/NixOS/nix/issues/1681.
2018-05-30 13:26:48 +02:00
Eelco Dolstra
23d6bb583a
LocalStore::addToStore: Throw an exception if no hash is given 2018-05-30 13:25:40 +02:00
Eelco Dolstra
3cab639e21
Use extended SQLite error codes 2018-05-30 13:25:03 +02:00
Eelco Dolstra
bd56b5fe3f
Merge pull request #2187 from bgamari/stoi-exceptions
json-to-value: Use strtol instead of strtoi
2018-05-28 14:54:49 +02:00
Ben Gamari
7f560b81ea json-to-value: Use strtol instead of strtoi
NixInt is long, so strtoi is too restrictive.
2018-05-26 18:43:46 -04:00
Will Dietz
93ae90de0f eval.cc: add message to static_assert, message can be omitted w/c++17 2018-05-26 14:04:41 -05:00
Will Dietz
14c464b6c8 nix-daemon: remove unused "pendingMsgs" variable 2018-05-26 13:09:55 -05:00
Will Dietz
ca9c6cb95d libstore/download: remove unused function "getTime()" 2018-05-25 19:49:45 -05:00
Eelco Dolstra
743359bc8a
Merge pull request #2157 from volth/bitwise
add builtins: __bitAnd,  __bitOr,  __bitXor
2018-05-24 15:00:39 +02:00
Eelco Dolstra
4bb8741b98
Make Env self-describing
If the Env denotes a 'with', then values[0] may be an Expr* cast to a
Value*. For code that generically traverses Values/Envs, it's useful
to know this.
2018-05-22 16:02:32 +02:00
Eelco Dolstra
9fd7cf98db
Memoise checkSourcePath()
This prevents hydra-eval-jobs from statting the same files over and
over again.
2018-05-22 13:02:14 +02:00
Will Dietz
b08923b4a0 serialise: fix buffer size used, hide method for internal use only
Fixes #2169.
2018-05-21 17:30:38 -05:00
volth
6cc28c0589
add `mod' and bitwise builtins: camel-case function names 2018-05-16 10:52:19 +00:00
volth
f3c090f91c
add mod' and bitwise builtins: remove mod' and shifts 2018-05-16 06:57:11 +00:00
volth
49b7cf1813
add `mod' and bitwise builtins: remove infix functions 2018-05-16 06:55:24 +00:00
Eelco Dolstra
966407bcf1
Merge pull request #2145 from serokell/uncached-channel
make sure not to use cached channels for nix-channel --update
2018-05-15 14:57:18 +02:00
volth
8a6a14e1f5 add `mod' and bitwise builtins 2018-05-12 16:50:39 +00:00
Eelco Dolstra
1ad19232c4
Don't return negative numbers from the flex tokenizer
Fixes #1374.
Closes #2129.
2018-05-11 12:05:12 +02:00
Eelco Dolstra
f3c85f9eb3
Revert "Throw a specific error for incomplete parse errors."
This reverts commit 6498adb002. We don't
actually use IncompleteParseError in 'nix repl'.
2018-05-11 11:40:50 +02:00
Eelco Dolstra
2b3c1b3a88
Remove mention of default since we already know the default 2018-05-10 12:06:28 +02:00
Eelco Dolstra
38def17627
Merge branch 's3-multipart-uploads' of https://github.com/AmineChikhaoui/nix 2018-05-10 12:06:02 +02:00
Yorick van Pelt
b9289e4855
make sure not to use cached channels for nix-channel --update
fixes #1964
2018-05-09 16:18:20 +02:00
Eelco Dolstra
a91c4ca01f
In restricted eval mode, allow access to the closure of store paths
E.g. this makes

  nix eval --restrict-eval -I /nix/store/foo '(builtins.readFile "/nix/store/foo/symlink/bla")'

(where /nix/store/foo/symlink is a symlink to another path in the
closure of /nix/store/foo) succeed.

This fixes a regression in Hydra compared to Nix 1.x (where there were
no restrictions at all on access to the Nix store).
2018-05-09 15:45:05 +02:00
AmineChikhaoui
854c0860f4
share the executor between multiple copy threads. 2018-05-07 15:07:00 +01:00
AmineChikhaoui
591e75cd01
add a FIXME note to find a way to abort the multipart uploads in
case the nix command is interrupted.
2018-05-07 14:27:53 +01:00
AmineChikhaoui
129394fb95
Support multi-part uploads for large NARs that exceed the size of
5Gb.
2018-05-07 14:23:51 +01:00
Eelco Dolstra
a1adcdf087
nix-shell: Only download requested outputs
Fixes #2116.
2018-05-07 14:58:25 +02:00
Eelco Dolstra
c717d8e3bd
Don't set a CA assertion for paths with references
Really fixes #2133.
2018-05-03 14:49:56 +02:00
Eelco Dolstra
80a7b16593
LocalStore::addValidPath(): Throw error when the CA assertion is wrong
Closes #2133.
2018-05-03 11:02:40 +02:00
Eelco Dolstra
53ec5ac69f
Fix some random -Wconversion warnings 2018-05-02 13:56:34 +02:00
Eelco Dolstra
4a2c948943
Fix bzip2 compression of files > 4 GiB
Bzip2's 'avail_in' parameter is declared as an unsigned int, so
assigning a size_t length to it led to silent truncation.

Fixes #2111.
2018-05-02 12:54:30 +02:00
Will Dietz
8e7d77d494 globals.hh: don't use '==' to compare string literals
Saw this in logs, also reported here:

8e6108ff71 (r28707288)
2018-04-23 20:57:05 -05:00
Shea Levy
8e6108ff71
Merge branch 'aarch64-armv7' of git://github.com/lheckemann/nix
Support extra compatible architectures (#1916)
2018-04-23 08:48:22 -04:00
Shea Levy
e2b028353b
Merge branch 'pos-crash-fix' of git://github.com/dezgeg/nix 2018-04-23 08:43:54 -04:00
Shea Levy
58f9ae0dcd
Merge branch 'hash-mismatch' of git://github.com/LnL7/nix 2018-04-23 08:40:34 -04:00
Andrew Dunham
3a918014b2 Fix library ordering in Makefiles
The existing ordering linked `libutil` before `libstore`, which causes
link failures when building statically. This is due to `libstore` using
functions from `libutil`, and the fact that symbol resolution works
"forward" - i.e. if you pass `-lfoo -lbar -lbaz`, any symbols that
`libbar` uses from `libbaz` will be resolved, but symbols from `libfoo`
will not since it comes first in the command line.

All this to say: this commit reorders the libraries which fixes the link
errors.
2018-04-21 21:10:52 -07:00
Peter Simons
040acdcee2
Merge pull request #1707 from dtzWill/fix/issue-1692
ignore "interrupted" exception in progress callback
2018-04-21 16:51:19 +02:00
Daiderd Jordan
13d4d1c0a2
download: make hash mismatch error consistent with fetchurl 2018-04-20 21:07:32 +02:00
Will Dietz
6d9129014d build.cc: fix bind-mount of /dev/{pts,ptmx} fallback
Don't bind-mount these to themselves,
mount them into the chroot directory.

Fixes pty issues when using sandbox on CentOS 7.4.
(build of perlPackages.IOTty fails before this change)
2018-04-20 12:24:23 -05:00
Daniel Poelzleithner
b712d4674b Allow multiple search experssions in nix search
The common use case is to search for packages containing multiple words
like a "git" "frontend". Having only one expressions makes this simple regular
use case very complicated. Instead, search accepts multiple regular epressions
which all need to match.

nix search git 'gui|frontend'

returns a list of all git uis for example
2018-04-18 21:08:35 +02:00
Félix Baylac-Jacqué
00584bb091
parser: Allow mixed nested and top-level attrpaths
Fixes #2077.
2018-04-18 18:44:38 +02:00
Shea Levy
b37f5ae31d
isFunction: True on primops.
Fixes #2073
2018-04-17 14:33:12 -04:00
Eelco Dolstra
a4aac7f88c
Handle arguments in $EDITOR
Fixes #2079.
2018-04-17 12:16:04 +02:00
Eelco Dolstra
c74f838620
exportReferences: Check whether the path is in the input closure
Fixes #2082.
2018-04-17 12:03:27 +02:00
Félix Baylac-Jacqué
a91fb422fe
nix repl: Fix multiline SIGINT handling.
Fixes #2076
2018-04-16 16:09:30 +02:00
Eelco Dolstra
d34fa2bcc3
Fix #1921 2018-04-13 15:42:35 +02:00
Shea Levy
dc0a542c9f
initPlugins: Fix dlopen error message. 2018-04-11 21:02:50 -04:00
Eelco Dolstra
1839a5542a
Fix #2057 2018-04-11 11:42:17 +02:00
Eelco Dolstra
d6aaa35478
Fix perl-bindings
https://hydra.nixos.org/build/72636781
2018-04-10 13:11:43 +02:00
Shea Levy
a38fe5c1a8
Export required C++ version in pkgconfig. 2018-04-09 11:32:43 -04:00
Shea Levy
7459388448
Merge branch 'fix/ruby-shebang' of git://github.com/samueldr/nix 2018-04-09 10:54:39 -04:00
Shea Levy
fd98fca7bb
Make prim_exec and prim_importNative available to plugins 2018-04-09 10:26:50 -04:00
Shea Levy
346c0ac361
nix-daemon: Exit successfully when interrupted.
Fixes #2058.
2018-04-09 08:05:54 -04:00
Samuel Dionne-Riel
399f43c3d5 nix-shell: Fixes use with ruby shebangs.
The ported code in 80ebc553ec was incorrectly ported.

```
-            $envCommand = "exec $execArgs $interpreter -e 'load(\"$script\")' -- ${\(join ' ', (map shellEscape, @savedArgs))}";
...
+                    envCommand = (format("exec %1% %2% -e 'load(\"%3%\") -- %4%") % execArgs % interpreter % script % joined.str()).str();
```

The single-quote finishing the small ruby snippet was lost in
translation.
2018-04-07 22:38:45 -04:00
AmineChikhaoui
62d75ad3e1
rename the options to mention it's a narinfo TTL as disk cache is used all over the place for other operations 2018-04-06 11:05:15 +01:00
AmineChikhaoui
86930ed414
add documentation for the local disk cache TTL config 2018-04-06 10:51:18 +01:00
AmineChikhaoui
2855c3d965
Make the TTL for disk cache configurable, we can now completely disable
disk cache lookup for example by doing:
  nix copy --from <binary-cahe> <store-path> --option \
  positive-disk-cache-ttl 0

Issues: #1885 #2035
2018-04-06 10:51:18 +01:00
Eelco Dolstra
1d9742b95f
<nix/buildenv.nix>: Ignore dangling symlinks
Fixes

  error: getting status of '/nix/store/j8p0vv89k1pf0cn7kmfsdcs7bshwga1i-firefox-52.7.2esr/share/icons/hicolor/48x48/apps/firefox.png': No such file or directory

https://github.com/NixOS/nix/issues/1934

Also improve error message on directory/non-directory collisions.
2018-04-05 20:04:42 +02:00
Eelco Dolstra
5c904e10eb
<nix/buildenv.nix>: Skip non-directory top-level paths
Fixes https://github.com/NixOS/nix/issues/1934 (at least the "error:
opening directory '/nix/store/...-linux-config-4.4.14': Not a
directory" issue).
2018-04-05 19:34:56 +02:00
Tuomas Tynkkynen
af86132e1a libexpr: Make unsafeGetAttrPos not crash on noPos
Currently e.g. `builtins.unsafeGetAttrPos "abort" builtins` will
eventually segfault because pos->file is an unset Symbol.

Found by afl-fuzz.
2018-04-03 15:54:42 +03:00
zimbatm
865ca2402f nix copy: add an example with S3
I couldn't find a good example how to use it with non-us-east-1 buckets.
2018-03-30 02:31:14 +01:00
Eelco Dolstra
446bb88f13
Remove reference to non-existent manpages 2018-03-29 12:29:10 +02:00
Eelco Dolstra
2b61c74922
Process --option use-case-hack properly
Fixes https://github.com/NixOS/nix/issues/2009.
2018-03-29 12:28:29 +02:00
Eelco Dolstra
9d1220a01d
ValidPathInfo::isContentAddressed(): Ensure there are no references 2018-03-29 12:27:42 +02:00
Will Dietz
74da813912 download: improve error for hash mismatch ("store mismatch")
Fixes #1905
2018-03-22 15:22:06 -05:00
Will Dietz
59b32403f2 download.cc: fix path for cached files 2018-03-22 15:05:46 -05:00
Eelco Dolstra
28eb9621cc
Fix build 2018-03-22 13:46:19 +01:00
Eelco Dolstra
dce8af59a6
Tweak 2018-03-22 13:32:18 +01:00
Eelco Dolstra
f87e286e82
Merge branch 'fix/avoid-large-stack-buffers' of https://github.com/dtzWill/nix 2018-03-22 13:19:25 +01:00
Shea Levy
c24e0439b2
Merge branch 'fix/execl-null-sentinel' of git://github.com/dtzWill/nix 2018-03-21 23:25:53 -04:00
Eelco Dolstra
ec91840e70 Also make the backwards compatible case of RemoteStore::addToStore run in constant memory
Just because we can.
2018-03-21 23:46:03 +01:00
Eelco Dolstra
97002b684c Make 'nix copy --to daemon' run in constant memory 2018-03-21 23:42:52 +01:00
Will Dietz
f66ae22288 shared.cc: style: prefer "nullptr" over '(char *)NULL' 2018-03-21 17:17:37 -05:00
Eelco Dolstra
11898d6a37 Make 'nix copy --to ssh://...' run in constant memory 2018-03-21 23:13:06 +01:00
Eelco Dolstra
47f7e5585b Make 'nix copy --from ssh://...' run in constant memory
For instance, this reduced the memory consumption of

  $ nix copy --from ssh://localhost --to ~/my-nix /nix/store/1n7x0yv8vq6zi90hfmian84vdhd04bgp-blender-2.79a

from 632 MiB to 16 MiB.
2018-03-21 23:10:40 +01:00
Eelco Dolstra
8615cfb130
LegacySSHStore: Allow overriding the path to nix-store 2018-03-21 15:10:46 +01:00
Will Dietz
02d36fdab7 execl: cast NULL sentinel to (char *), per man page and compiler warning
From exec(3):

> The list of arguments must be terminated by a null pointer, and, since these
> are variadic functions, this pointer must be cast (char *) NULL
2018-03-20 13:31:58 -05:00
Eelco Dolstra
03d8136b02
Remove unused channel-cache directory 2018-03-20 18:22:49 +01:00
Eelco Dolstra
e0c1597910
Style fix 2018-03-20 17:33:45 +01:00
Eelco Dolstra
668ac3ea2c
Make <nix/buildenv.nix> a builtin builder
This avoids sandbox annoyances.
2018-03-20 17:28:09 +01:00
Eelco Dolstra
9d40787938
Move builtinFetchurl to its own file 2018-03-20 17:01:36 +01:00
Eelco Dolstra
5df427f35b
Slight simplification 2018-03-20 16:32:59 +01:00
Eelco Dolstra
21ebf6a160
SSHMaster: Optionally pass -v to ssh 2018-03-20 15:30:55 +01:00
Eelco Dolstra
1aca195e52
Hack to get SSH error messages from build-remote
E.g.

  cannot build on 'ssh://mac1': cannot connect to 'mac1': bash: nix-store: command not found
  cannot build on 'ssh://mac2': cannot connect to 'mac2': Host key verification failed.
  cannot build on 'ssh://mac3': cannot connect to 'mac3': Received disconnect from 213... port 6001:2: Too many authentication failures
  Authentication failed.
2018-03-20 15:25:26 +01:00
Will Dietz
f5a2136a40 serialise.cc: remove pessimising move
from clang6:

src/libutil/serialise.cc:189:23: warning: moving a temporary object prevents copy elision [-Wpessimizing-move]
2018-03-19 19:18:47 -05:00
Will Dietz
9b33201e72 util: rename stdout/stdin members to avoid conflicts w/standard macro
(cherry picked from commit c389a7fb617ed7bcd617efa68c6a48c00405310d)
2018-03-19 11:18:56 -05:00
Eelco Dolstra
138af2e554
Shut up signedness warning 2018-03-19 11:57:15 +01:00
Linus Heckemann
639c166647 build-remote: take extra-platforms into account 2018-03-16 22:50:57 +00:00
Linus Heckemann
637701b604 rename build-extra-platforms -> extra-platforms
also document it
2018-03-16 22:50:27 +00:00
Linus Heckemann
12913ccf45 nix-shell: allow symlinks to .drvs
This makes persistent shell environments easier to use.
2018-03-16 22:15:24 +00:00
Eelco Dolstra
48662d151b
Reduce substitution memory consumption
copyStorePath() now pipes the output of srcStore->narFromPath()
directly into dstStore->addToStore(). The sink used by the former is
converted into a source usable by the latter using
boost::coroutine2. This is based on [1].

This reduces the maximum resident size of

  $ nix build --store ~/my-nix/ /nix/store/b0zlxla7dmy1iwc3g459rjznx59797xy-binutils-2.28.1 --substituters file:///tmp/binary-cache-xz/ --no-require-sigs

from 418592 KiB to 53416 KiB. (The previous commit also reduced the
runtime from ~4.2s to ~3.4s, not sure why.) A further improvement will
be to download files into a Sink.

[1] https://github.com/NixOS/nix/compare/master...Mathnerd314:dump-fix-coroutine#diff-dcbcac55a634031f9cc73707da6e4b18

Issue #1969.
2018-03-16 20:35:59 +01:00
Eelco Dolstra
3e6b194d78
decompress(): Use a Source and Sink
This allows decompression to happen in O(1) memory.
2018-03-16 20:35:59 +01:00
Eelco Dolstra
64441f0551
Merge pull request #1939 from dezgeg/lexer-fix
libexpr: Recognize newline in more places in lexer
2018-03-16 12:14:58 +01:00
Eelco Dolstra
eb75bc5afb
Don't retry on CURLE_SSL_CACERT_BADFILE
The certificates won't get any better if we retry.
2018-03-16 12:03:18 +01:00
Eelco Dolstra
4868721506
Filter ANSI colors when not writing to a terminal
Fixes https://github.com/NixOS/nixpkgs/issues/37114.
2018-03-15 16:08:07 +01:00
Eelco Dolstra
7b8914825a
Merge pull request #1965 from masaeedu/master
Wrap thread local in function for Cygwin
2018-03-15 14:56:21 +01:00
Shea Levy
cfdbfa6b2c
Catch more possible instances of passing NULL to memcpy.
Actually fixes #1976.
2018-03-14 23:53:43 -04:00
Shea Levy
d25d9f7cec
Merge branch 'fix/errno-sighandler' of git://github.com/dtzWill/nix 2018-03-14 23:45:35 -04:00
Shea Levy
e2088febf3
concatLists: Don't pass NULL pointers to memcpy.
This is UB, even if the size is 0. See #1976.

Fixes #1976.
2018-03-14 23:44:02 -04:00
Will Dietz
d98755b0c2 nix-daemon: preserve errno in signal handler (thanks tsan) 2018-03-14 21:39:46 -05:00
Eelco Dolstra
55aa622fb1
Fix compatibility with latest boost::format 2018-03-14 19:25:09 +01:00
Eelco Dolstra
ca14b14200
Use boost::format from the boost package
Note that this only requires headers from boost so it doesn't add a
runtime dependency.

Also, use Nixpkgs 18.03.
2018-03-14 19:24:04 +01:00
Shea Levy
56f2ed0081
Merge pull request #1906 from dtzWill/fix/nix-search
nix search: tests and fix #1893 and part of #1892
2018-03-13 18:43:47 -06:00
Eelco Dolstra
7afdc8d4a1
build-remote: Don't substitute the build result 2018-03-13 17:56:51 +01:00
Eelco Dolstra
6e60141a80
Typo 2018-03-13 17:48:09 +01:00
Guillaume Maudoux
80735c4cc9
fetchGit: Fix debug message 2018-03-13 10:28:23 +01:00
Asad Saeeduddin
be54f4a0b6 Wrap thread local in function for Cygwin
Fixes #1826. See #1352 for a previous instance of a similar change.
2018-03-12 00:56:41 -04:00
Eelco Dolstra
24b739817f
Fix double free in Store::queryPathInfo()
It was holding on to a Value* (i.e. a std::shared_ptr<ValidPathInfo>*)
outside of the pathInfoCache lock, so the std::shared_ptr could be
destroyed between the release of the lock and the decrement of the
std::shared_ptr refcount. This can happen if more than
'path-info-cache-size' paths are added in the meantime, *or* if
clearPathInfoCache() is called. The hydra-queue-runner queue monitor
thread periodically calls the later, so is likely to trigger a crash.

Fixes https://github.com/NixOS/hydra/issues/542.
2018-03-09 14:19:51 +01:00
Tim Engler
383afab9ad Modified MakeBinOp to no longer produce its name using concatenation and "##".
Doing so prevents emacs tags from working, as well as makes the code extremely
confusing for a newbie.

In the prior state, if someone wants to find the definition of "ExprApp" for
example, a grep through the code reveals nothing. Since the definition could be
hiding in numerous ".h" files, it's really difficult to find. This personally
took me several hours to figure out.
2018-03-09 03:16:33 +00:00
Eelco Dolstra
16551f54c9
Fix #1957 2018-03-07 11:40:50 +01:00
Eelco Dolstra
4452f6e855
Merge pull request #1955 from dtzWill/fix/rand-random
rand() -> random(), since we use srandom().
2018-03-07 11:00:08 +01:00
Will Dietz
56253bb08f rand() -> random(), since we use srandom().
rand() requires we call srand() instead,
but might as well use random().
2018-03-06 17:44:25 -06:00
Will Dietz
e917c05628 logging: Don't throw from Activity destructor
Move definition of destructor to logging.cc for
access to util.hh's ignoreException.
2018-03-06 10:42:58 -06:00
Shea Levy
088ef81759
ssh-ng: Don't forward options to the daemon.
This can be iterated on and currently leaves out settings we know we
want to forward, but it fixes #1713 and fixes #1935 and isn't
fundamentally broken like the status quo. Future changes are suggested
in a comment.
2018-03-05 07:42:15 -05:00
Will Dietz
6b9a03f5d8 hoist vector out of loop just in case 2018-03-02 10:52:05 -06:00
Will Dietz
c89a3d5368 don't allocate large buffers on the stack 2018-03-02 10:52:04 -06:00
Shea Levy
3748a0ca1e
Merge branch 'improve-search-algorithm' of git://github.com/Ma27/nix 2018-03-02 11:41:08 -05:00
Shea Levy
aa8bbbf69d
Merge branch 'write-failure-fixes' of git://github.com/lheckemann/nix 2018-03-02 10:59:59 -05:00
Tuomas Tynkkynen
a0e38c16bc libexpr: Recognize newline in more places in lexer
Flex's regexes have an annoying feature: the dot matches everything
except a newline. This causes problems for expressions like:

"${0}\
"

where the backslash-newline combination matches this rule instead of the
intended one mentioned in the comment:

    <STRING>\$|\\|\$\\ {
                    /* This can only occur when we reach EOF, otherwise the above
                    (...|\$[^\{\"\\]|\\.|\$\\.)+ would have triggered.
                    This is technically invalid, but we leave the problem to the
                    parser who fails with exact location. */
                    return STR;
                }
However, the parser actually accepts the resulting token sequence
('"' DOLLAR_CURLY 0 '}' STR '"'), which is a problem because the lexer
rule didn't assign anything to yylval. Ultimately this leads to a crash
when dereferencing a NULL pointer in ExprConcatStrings::bindVars().

The fix does change the syntax of the language in some corner cases
but I think it's only turning previously invalid (or crashing) syntax
to valid syntax. E.g.

"a\
b"

and

''a''\
b''

were previously syntax errors but now both result in "a\nb".

Found by afl-fuzz.
2018-03-02 17:30:48 +02:00
Eelco Dolstra
939cf4cceb
Fix error message 2018-03-02 14:32:00 +01:00
Matthew O'Gorman
467fdd8ca4 only delete things older than current gen and update logic in doc as
well
2018-03-01 22:59:00 -05:00
Matthew O'Gorman
3c16044cb0 remove unused variable and make sure to check that the current
generation is not the one we are deleting
2018-03-01 22:22:02 -05:00
Matt O'Gorman
12fe2249e1 Update nix-env.cc
missing comma
2018-03-01 21:47:57 -05:00
Matthew O'Gorman
0312d30315 this updates issues that were addressed by people in pr 2018-03-01 21:47:57 -05:00
Matthew O'Gorman
429154b74c Implement --delete-generations + flag for keeping last N number of generations 2018-03-01 21:47:57 -05:00
Will Dietz
e89d02bf03 fetchGit: use "HEAD" as default ref 2018-02-28 16:34:34 -06:00
Shea Levy
14ca85688c
Actually fix nixDataDir in non-canonical path 2018-02-28 06:19:40 -05:00
Linus Heckemann
919c3c20b3 seccomp: add 32-bit ARM on aarch64-linux
This allows building armv[67]l-linux derivations on compatible aarch64
machines. Failure to add the architecture may result from missing
hardware support, in which case we can't run 32-bit binaries and don't
need to restrict them with seccomp anyway,
2018-02-27 00:32:15 +00:00
Linus Heckemann
eb03a296c1 Add build-extra-platforms setting
This allows specifying additional systems that a machine is able to
build for. This may apply on some armv7-capable aarch64 processors, or
on systems using qemu-user with binfmt-misc to support transparent
execution of foreign-arch programs.

This removes the previous hard-coded assumptions about which systems are
ABI-compatible with which other systems, and instead relies on the user
to specify any additional platforms that they have ensured compatibility
for and wish to build for locally.

NixOS should probably add i686-linux on x86_64-linux systems for this
setting by default.
2018-02-27 00:27:58 +00:00
Tuomas Tynkkynen
77e9e1ed91 libexpr: Fix prim_replaceStrings() to work on an empty source string
Otherwise, running e.g.

nix-instantiate --eval -E --strict 'builtins.replaceStrings [""] ["X"] "abc"'

would just hang in an infinite loop.

Found by afl-fuzz.

First attempt of this was reverted in e2d71bd186 because it caused
another infinite loop, which is fixed now and a test added.
2018-02-26 19:49:13 +02:00
Eelco Dolstra
24ec750003
nix run: Fix segfault on macOS
Note that clearenv() is not available on macOS.

Fixes #1907.
2018-02-26 18:29:40 +01:00
Will Dietz
009cf9cd23 nix search: explicitly handle empty search string, fixes #1893
This is important since this is given as an example.
Other patterns containing "empty search string" will still
be handled differently on different platforms ("asdf|")
but that's less of an issue.
2018-02-25 16:40:56 -06:00
Will Dietz
3cac8734ac nix search: fix bug where we wrote to cache when shouldn't, breaking
This is exposed by the tests added previously,
and resolves the error reported in #1892:
"expected JSON value".
2018-02-25 16:40:05 -06:00
Eelco Dolstra
64e486ab63
Fix downloadCached() with a chroot store
E.g.

  nix run --store ~/my-nix -f channel:nixos-17.03 hello -c hello

This problem was mentioned in #1897.
2018-02-23 13:00:42 +01:00
Eelco Dolstra
179b896acb
Merge branch 'data-dir-non-canon' of https://github.com/shlevy/nix 2018-02-22 14:20:43 +01:00
Eelco Dolstra
e8d53bfdc9
Revert "Enable sandbox builds on Linux by default"
This reverts commit ddc58e7896.

https://hydra.nixos.org/eval/1435322
2018-02-22 14:20:07 +01:00
Shea Levy
ddbcd01c83
Fix restricted mode when installing in non-canonical data dir 2018-02-22 07:18:14 -05:00
Eelco Dolstra
ddc58e7896
Enable sandbox builds on Linux by default
The overhead of sandbox builds is a problem on NixOS (since building a
NixOS configuration involves a lot of small derivations) but not for
typical non-NixOS use cases. So outside of NixOS we can enable it.

Issue #179.
2018-02-22 12:27:25 +01:00
Eelco Dolstra
de4c03d201
Merge branch 'fix/dry-run-partially' of https://github.com/dtzWill/nix 2018-02-22 12:18:20 +01:00
Eelco Dolstra
0d54671b7b
Manual: Update chapter on remote builds
Alos add a command "nix ping-store" to make it easier to see if Nix
can connect to a remote builder (e.g. 'nix ping-store --store
ssh://mac').
2018-02-21 16:24:26 +01:00
Eelco Dolstra
e2d71bd186
Revert "libexpr: Fix prim_replaceStrings() to work on an empty source string"
This reverts commit 4ea9707591.

It causes an infinite loop in Nixpkgs evaluation,
e.g. "nix-instantiate -A hello" hung.

PR #1886.
2018-02-21 15:35:28 +01:00
Shea Levy
a6c497f526
Merge branch 'nix-copy' of git://github.com/Mic92/nix-1 2018-02-20 21:05:17 -05:00
Jörg Thalheim
fa7fd76c5e nix-copy: fix examples
maybe a left-over from nix-store -r ?
2018-02-21 01:13:46 +00:00
Tuomas Tynkkynen
546f98dace libutil: Fix invalid assert on decoding base64 hashes
The assertion is broken because there is no one-to-one mapping from
length of a base64 string to the length of the output.

E.g.

"1q69lz7Empb06nzfkj651413n9icx0njmyr3xzq1j9q=" results in a 32-byte output.
"1q69lz7Empb06nzfkj651413n9icx0njmyr3xzq1j9qy" results in a 33-byte output.

To reproduce, evaluate:

builtins.derivationStrict {
    name = "0";
    builder = "0";
    system = "0";
    outputHashAlgo = "sha256";
    outputHash = "1q69lz7Empb06nzfkj651413n9icx0njmyr3xzq1j9qy";
}

Found by afl-fuzz.
2018-02-19 23:20:26 +02:00
Tuomas Tynkkynen
4ea9707591 libexpr: Fix prim_replaceStrings() to work on an empty source string
Otherwise, running e.g.

nix-instantiate --eval -E --strict 'builtins.replaceStrings [""] ["X"] "abc"'

would just hang in an infinite loop.

Found by afl-fuzz.
2018-02-19 23:20:26 +02:00
Tuomas Tynkkynen
1d0e42879f libutil: Fix infinite loop in filterANSIEscapes on '\r'
E.g. nix-instantiate --eval -E 'abort "\r"' hangs.

Found by afl-fuzz.
2018-02-19 23:20:26 +02:00
Tuomas Tynkkynen
056d28a601 libexpr: Don't create lots of temporary strings in Bindings::lexicographicOrder
Avoids ~180,000 string temporaries created when evaluating a headless
NixOS system.
2018-02-19 22:47:25 +02:00
Eelco Dolstra
d4e93532e2
Fix incorrect (and unnecessary) format string
https://hydra.nixos.org/eval/1434547#tabs-now-fail
2018-02-19 20:46:39 +01:00
Eelco Dolstra
70eb64147e
Update release notes
Also add some examples to nix --help.
2018-02-19 20:38:53 +01:00
Eelco Dolstra
623fcb071e
Merge pull request #1882 from shlevy/no-seccomp-no-filterSyscalls
Don't silently succeed seccomp setup when !HAVE_SECCOMP.
2018-02-19 17:39:46 +01:00
Shea Levy
e1eb63a586
Merge branch 'perf-fixes' of git://github.com/dezgeg/nix 2018-02-19 10:11:52 -05:00
Shea Levy
e59a8a63e1
Don't silently succeed seccomp setup when !HAVE_SECCOMP.
Running Nix with build users without seccomp on Linux is dangerous,
and administrators should very explicitly opt-in to it.
2018-02-19 09:56:24 -05:00
Eelco Dolstra
ed73d40c3b
Config::handleUnknownSettings(): Remove unused 'fatal' argument 2018-02-19 14:00:34 +01:00
Eelco Dolstra
75a1d96cfd
Merge branch 'register-settings' of https://github.com/shlevy/nix 2018-02-19 13:58:34 +01:00
Eelco Dolstra
7fe5910bf8
Merge pull request #1857 from dtzWill/fix/check-for-lzma-mt
configure.ac: check if lzma has MT support, fix deb build/etc.
2018-02-19 13:06:14 +01:00
Shea Levy
690ac7c90b
configure: Add a flag to disable seccomp.
This is needed for new arches where libseccomp support doesn't exist
yet.

Fixes #1878.
2018-02-18 02:35:01 -05:00
Tuomas Tynkkynen
37264ed0ad libexpr: Avoid an unnecessary string copy in prim_derivationStrict 2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen
66eeff3345 libexpr: Remove unnecessary drvName assignment in prim_derivationStrict
drvName is already assigned to the same value right at the start of the
function.
2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen
7e0360504d libexpr: Optimize prim_derivationStrict by using more symbol comparisons 2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen
0845cdf944 libexpr: Rely on Boehm returning zeroed memory in EvalState::allocEnv()
Boehm guarantees that memory returned by GC_malloc() is zeroed, so take
advantage of that.
2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen
b8bed7da14 libexpr: Optimize prim_attrNames a bit
Instead of having lexicographicOrder() create a temporary sorted array
of Attr*:s and copying attr names from that, copy the attr names
first and then sort that.
2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen
f67a7007a2 libexpr: Pre-reserve space in string in unescapeStr()
Avoids some malloc() traffic.
2018-02-16 04:39:43 +02:00
Eelco Dolstra
3a5a241b32
Merge pull request #1873 from matthewbauer/fix-manpath
Set MANPATH in case man path isn’t set correctly.
2018-02-15 17:37:21 +01:00
Matthew Bauer
8f186722a9 Set backup MANPATH in case man path isn’t set correctly.
Previously, this would fail at startup for non-NixOS installs:

nix-env --help

The fix for this is to just use "nixManDir" as the value for MANPATH
when spawning "man".

To test this, I’m using the following:

$ nix-build release.nix -A build
$ MANPATH= ./result/bin/nix-env --help

Fixes #1627
2018-02-15 10:20:02 -06:00
Shea Levy
ac973a6d3c
monitor-fds: Fix on macOS.
Fixes #1871.
2018-02-14 18:26:37 -05:00
Shea Levy
b095c06139
Add splitVersion primop.
Fixes #1868.
2018-02-14 09:55:43 -05:00
Shea Levy
de4934ab3b
Allow plugins to define new settings. 2018-02-13 14:43:32 -05:00
Eelco Dolstra
52c777a793
Merge pull request #1863 from shlevy/conf-includes
Allow includes from nix.conf
2018-02-13 17:33:07 +01:00
Shea Levy
b8739f2fb3
Enable specifying directories in plugin-files. 2018-02-13 11:25:01 -05:00
Linus Heckemann
78ac3eb4eb Fix #1762
nix-store --export, nix-store --dump, and nix dump-path would previously
fail silently if writing the data out failed, because
 a) FdSink::write ignored exceptions, and
 b) the commands relied on FdSink's destructor, which ignores
    exceptions, to flush the data out.

This could cause rather opaque issues with installing nixos, because
nix-store --export would happily proceed even if it couldn't write its
data out (e.g. if nix-store --import on the other side of the pipe
failed).

This commit adds tests that expose these issues in the nix-store
commands, and fixes them for all three.
2018-02-13 14:26:19 +00:00
Shea Levy
6eb1040e90
Allow includes from nix.conf 2018-02-13 08:16:32 -05:00
Eelco Dolstra
7828dca9e8
Merge branch 'register-constant' of https://github.com/shlevy/nix 2018-02-13 12:24:48 +01:00
Eelco Dolstra
1c10a74c73
Merge branch 'plugins' of https://github.com/shlevy/nix 2018-02-13 12:15:27 +01:00
Eelco Dolstra
c5cc57e962
Merge pull request #1859 from FRidh/flatten
Nix stats: flatten statistics
2018-02-13 11:11:12 +01:00
Eelco Dolstra
9bcb4d2dd9 Fix hang in build-remote 2018-02-12 22:48:55 +01:00
Eelco Dolstra
4f09ce7940
Fix 'deadlock: trying to re-acquire self-held lock'
This was caused by derivations with 'allowSubstitutes = false'. Such
derivations will be built locally. However, if there is another
SubstitionGoal that has the output of the first derivation in its
closure, then the path will be simultaneously built and substituted.

There was a check to catch this situation (via pathIsLockedByMe()),
but it no longer worked reliably because substitutions are now done in
another thread. (Thus the comment 'It can't happen between here and
the lockPaths() call below because we're not allowing multi-threading'
was no longer valid.)

The fix is to handle the path already being locked in both
SubstitutionGoal and DerivationGoal.
2018-02-12 17:06:06 +01:00
Eelco Dolstra
35fd31770c
toBuildables -> build 2018-02-12 17:06:06 +01:00
Will Dietz
c6209030c4 compression: make parallel sink separate class 2018-02-11 13:50:28 -06:00
Will Dietz
a0bdc96726 compression: print warning if parallel requested but not supported 2018-02-11 13:03:47 -06:00
Frederik Rietdijk
60eca58533 Nix stats: flatten statistics
Flattens the list of statistics as suggested in
https://github.com/NixOS/ofborg/issues/67. This makes it easier to work
with.
2018-02-11 14:37:50 +01:00
Will Dietz
5a082ad15a configure.ac: check if lzma has MT support, fix deb build/etc. 2018-02-09 21:02:25 -06:00
Eelco Dolstra
960e9c560e
nix: Ensure that the user sees errors from substituters 2018-02-09 15:06:47 +01:00
Eelco Dolstra
aa02cdc33c
getDefaultSubstituters(): Skip broken substituters
Fixes #1340.
2018-02-09 15:06:47 +01:00
Eelco Dolstra
3d2d207aad
Merge pull request #1848 from AmineChikhaoui/parallel-xz
support multi threaded xz encoder
2018-02-09 15:03:25 +01:00
Shea Levy
081f14a169
Allow using RegisterPrimop to define constants.
This enables plugins to add new constants, as well as new primops.
2018-02-08 14:35:50 -05:00
Shea Levy
88cd2d41ac
Add plugins to make Nix more extensible.
All plugins in plugin-files will be dlopened, allowing them to
statically construct instances of the various Register* types Nix
supports.
2018-02-08 12:44:37 -05:00
Eelco Dolstra
ad97a21834
nix-env: Fix parsing of --system
https://hydra.nixos.org/build/68827814
2018-02-08 15:25:03 +01:00
AmineChikhaoui
0685a6480a
Merge branch 'master' of https://github.com/NixOS/nix into parallel-xz 2018-02-08 12:24:48 +01:00
Will Dietz
98031b6050 nix build: Don't create output links with --dry-run.
Fixes #1849.
2018-02-07 15:20:54 -06:00
Will Dietz
f201b7733e More completely fix recursive nix, unbreak tests
See:
88b5d0c8e8 (commitcomment-27406365)
2018-02-07 15:19:28 -06:00
AmineChikhaoui
a56637205a
Merge branch 'master' of https://github.com/NixOS/nix into parallel-xz 2018-02-07 21:09:04 +01:00
AmineChikhaoui
47ad88099b
move the parallel-compression setting to binary-cache-store, the setting
can be done now from the url e.g s3://nix-cache?parallel-compression=1
instead of nix.conf.
2018-02-07 21:06:11 +01:00
Eelco Dolstra
88b5d0c8e8
Prevent accidental recursive Nix 2018-02-07 19:07:38 +01:00
AmineChikhaoui
55ecdfe2a8
make multi threaded compression configurable and use single threaded
by default.
2018-02-07 17:54:08 +01:00
AmineChikhaoui
163e39547a
Merge branch 'master' of https://github.com/NixOS/nix into parallel-xz 2018-02-07 17:53:50 +01:00
Eelco Dolstra
84989d3af2
Improve filtering of ANSI escape sequences in build logs
All ANSI sequences except color setting are now filtered out. In
particular, terminal resets (such as from NixOS VM tests) are filtered
out.

Also, fix the completely broken tab character handling.
2018-02-07 15:23:57 +01:00
Eelco Dolstra
abe6be578b
Merge pull request #1816 from shlevy/add-path
Add path primop.
2018-02-07 13:32:35 +01:00
AmineChikhaoui
9d1e22f743
set block size to 0 to let the lzma lib choose the right one, add
some comments about possible improvements wrt memory usage/threading.
2018-02-07 11:18:55 +01:00
Shea Levy
69d82e5c58
Add path primop.
builtins.path allows specifying the name of a path (which makes paths
with store-illegal names now addable), allows adding paths with flat
instead of recursive hashes, allows specifying a filter (so is a
generalization of filterSource), and allows specifying an expected
hash (enabling safe path adding in pure mode).
2018-02-06 16:48:08 -05:00
AmineChikhaoui
bc7e3a4dd6
support multi threaded xz encoder, this might be particularly useful in
the case of hydra where the overhead of single threaded encoding is more
noticeable e.g most of the time spent in "Sending inputs"/"Receiving outputs"
is due to compression while the actual upload to the binary cache seems
to be negligible.
2018-02-06 22:42:02 +01:00
Eelco Dolstra
6f6bfc8205 Update the progress bar at most 20 times per second
Fixes #1834.
2018-02-06 20:51:37 +01:00
Eelco Dolstra
43f8ef73c6
realiseContext(): Add derivation outputs to the allowed paths
This makes import-from-derivation work in restricted mode again.
2018-02-06 15:38:45 +01:00
Eelco Dolstra
f24e726ba5
checkURI(): Check file URIs against allowedPaths
This makes e.g. 'fetchGit ./.' work (assuming that ./. is an allowed
path).
2018-02-06 14:35:33 +01:00
Eelco Dolstra
55012ec0b9
Tweak progress bar message
Say "copying" instead of "fetching" when copying from another local
store. Nice for nixos-install.
2018-02-05 18:32:23 +01:00
Eelco Dolstra
bb1d046f5c
Allow substituters to be marked as trusted
This is needed by nixos-install, which uses the Nix store on the
installation CD as a substituter. We don't want to disable signature
checking entirely because substitutes from cache.nixos.org should
still be checked. So now we can pas "local?trusted=1" to mark only the
Nix store in /nix as not requiring signatures.

Fixes #1819.
2018-02-05 18:08:30 +01:00
Eelco Dolstra
2175eee9fe
Fix segfault using non-binary cache stores as substituters 2018-02-05 17:46:43 +01:00
Eelco Dolstra
84722d67d2
Remove nix-build --hash
Instead, if a fixed-output derivation produces has an incorrect output
hash, we now unconditionally move the outputs to the path
corresponding with the actual hash and register it as valid. Thus,
after correcting the hash in the Nix expression (e.g. in a fetchurl
call), the fixed-output derivation doesn't have to be built again.

It would still be good to have a command for reporting the actual hash
of a fixed-output derivation (instead of throwing an error), but
"nix-build --hash" didn't do that.
2018-02-03 10:08:05 +01:00
Shea Levy
de96daf54f
Merge branch 'master' of git://github.com/catern/nix 2018-02-01 13:21:45 -05:00
Eelco Dolstra
19477e8815
nix-build: Ignore --indirect
Note that nix-build always creates indirect roots.

Fixes #1830.
2018-02-01 17:09:56 +01:00
Eelco Dolstra
855699855f
Remove obsolete references to manifests
Closes #323.
2018-02-01 10:39:16 +01:00
Spencer Baugh
e5432574e2 document ability to set NIX_REMOTE=unix://path/to/socket 2018-01-31 22:47:16 +00:00
Eelco Dolstra
c129fc6ee8
Merge pull request #1801 from catern/master
remote_store: register for NIX_REMOTE=unix://path
2018-01-31 23:12:39 +01:00
Eelco Dolstra
88f4f0231b Fix building without aws-sdk-cpp 2018-01-31 22:34:51 +01:00
Eelco Dolstra
c287d73121
Rename 1.12 -> 2.0
Following discussion with Shea and Graham. It's a big enough change
from the last release. Also, from a semver perspective, 2.0 makes more
sense because we did remove some interfaces (like nix-pull/nix-push).
2018-01-31 18:58:45 +01:00
Eelco Dolstra
6fa690291a
Add 'nix upgrade-nix' command
This command upgrades Nix to the latest stable version by installing a
store path obtained from

  https://github.com/NixOS/nixpkgs/raw/master/nixos/modules/installer/tools/nix-fallback-paths.nix

which is the same store path that the installer at
https://nixos.org/nix/install.sh uses.

The upgrade fails if Nix is not installed in a profile (e.g. on NixOS,
or when installed outside of the Nix store).
2018-01-31 16:24:43 +01:00
Eelco Dolstra
30370f168f
Cleanup 2018-01-31 15:14:03 +01:00
Eelco Dolstra
6270b2e50f
Merge branch 'http-binary-cache-put-upsert' of https://github.com/adelbertc/nix 2018-01-31 15:10:12 +01:00
Eelco Dolstra
478e3e4649
Indent properly 2018-01-31 13:46:38 +01:00
Spencer Baugh
746f8aed86 remote_store: register for NIX_REMOTE=unix://path
This allows overriding the socket path so the daemon may be listening at
an arbitrary Unix domain socket location.

Fixes #1800
2018-01-26 22:05:16 +00:00
Shea Levy
1d5d277ac7
HttpBinaryCacheStore: Support upsertFile with PUT.
Some servers, such as Artifactory, allow uploading with PUT and BASIC
auth. This allows nix copy to work to upload binaries to those
servers.

Worked on together with @adelbertc
2018-01-26 11:12:30 -08:00
Eelco Dolstra
e09161d05c
Remove signed-binary-caches as the default for require-sigs
This was for backward compatibility. However, with security-related
configuration settings, it's best not to have any confusion.

Issue #495.
2018-01-26 17:12:15 +01:00
Eelco Dolstra
479692a068
Merge pull request #1811 from dtzWill/update/json-3.0.1
nlohmann-json: 2.1.1 -> 3.0.1
2018-01-24 17:11:25 +01:00
Dan Peebles
d43a8b25f0 Fix obscure corner case in name resolution for builtin:fetchurl in sandboxed environments 2018-01-23 14:45:50 -05:00
Will Dietz
f7c26365eb nlohmann-json: 2.1.1 -> 3.0.1 2018-01-22 12:19:50 -06:00
Eelco Dolstra
89a2a11d9f
Don't use [[noreturn]] 2018-01-19 15:00:38 +01:00
Eelco Dolstra
3c4c30eadd
Rewrite builtin derivation environment
Also add a test.

Fixes #1803.
Closes #1805.
2018-01-19 15:00:38 +01:00
Eelco Dolstra
2896bb6826
Don't retry CURLE_URL_MALFORMAT 2018-01-19 14:05:08 +01:00
Eelco Dolstra
0c95776c3e
Don't define builtins.{currentSystem,currentTime} in pure mode
This makes it easier to provide a default, e.g.

  system = builtins.currentSystem or "x86_64-linux";
2018-01-18 16:38:48 +01:00
Eelco Dolstra
27b510af5c
nix eval: Stop progress bar before printing the result 2018-01-17 12:04:44 +01:00
Eelco Dolstra
16e0287556
nix eval: Take only one argument
Thus --json no longer produces a list.
2018-01-17 12:03:06 +01:00
Eelco Dolstra
d8b4cfad82
Typo 2018-01-17 11:53:08 +01:00
Eelco Dolstra
75b9670df6
Make show-trace a config setting 2018-01-16 19:24:23 +01:00
Eelco Dolstra
d4dcffd643
Add pure evaluation mode
In this mode, the following restrictions apply:

* The builtins currentTime, currentSystem and storePath throw an
  error.

* $NIX_PATH and -I are ignored.

* fetchGit and fetchMercurial require a revision hash.

* fetchurl and fetchTarball require a sha256 attribute.

* No file system access is allowed outside of the paths returned by
  fetch{Git,Mercurial,url,Tarball}. Thus 'nix build -f ./foo.nix' is
  not allowed.

Thus, the evaluation result is completely reproducible from the
command line arguments. E.g.

  nix build --pure-eval '(
    let
      nix = fetchGit { url = https://github.com/NixOS/nixpkgs.git; rev = "9c927de4b179a6dd210dd88d34bda8af4b575680"; };
      nixpkgs = fetchGit { url = https://github.com/NixOS/nixpkgs.git; ref = "release-17.09"; rev = "66b4de79e3841530e6d9c6baf98702aa1f7124e4"; };
    in (import (nix + "/release.nix") { inherit nix nixpkgs; }).build.x86_64-linux
  )'

The goal is to enable completely reproducible and traceable
evaluation. For example, a NixOS configuration could be fully
described by a single Git commit hash. 'nixos-rebuild' would do
something like

  nix build --pure-eval '(
    (import (fetchGit { url = file:///my-nixos-config; rev = "..."; })).system
  ')

where the Git repository /my-nixos-config would use further fetchGit
calls or Git externals to fetch Nixpkgs and whatever other
dependencies it has. Either way, the commit hash would uniquely
identify the NixOS configuration and allow it to reproduced.
2018-01-16 19:23:18 +01:00
Eelco Dolstra
23fa7e3606
parseExprFromFile -> evalFile
parseExprFromFile() should be avoided since it doesn't cache anything.
2018-01-16 17:11:58 +01:00
Eelco Dolstra
ba75c69e00
Barf when using a diverted store on macOS
Fixes #1792.
2018-01-15 12:14:43 +01:00
Will Dietz
6454ca393a nix log: use pager 2018-01-12 15:05:26 -06:00
Eelco Dolstra
74f75c8558
import, builtins.readFile: Handle diverted stores
Fixes #1791
2018-01-12 17:31:08 +01:00
Renzo Carbonara
b0328c244d nix.conf: builders-use-substitutes
Fixes #937
2018-01-09 22:40:07 +01:00
Will Dietz
428680b307 fetchGit: fix creation of uninitialized cache dir, let git create it
fetchGit test (as modified in previous commit) now passes.
2018-01-09 09:05:18 -06:00
Eelco Dolstra
7b9583680e
Improve error message with --repair for untrusted users 2018-01-08 19:13:48 +01:00
Eelco Dolstra
44272d8719
Rename "use-substitutes" to "substitute"
Commit c2154d4c84 renamed
"build-use-substitutes" to "use-substitutes", but that broke
"nix-copy-closure --use-substitutes".
2018-01-04 16:58:39 +01:00
Shea Levy
689b2783fc
Add hasContext primop 2018-01-02 12:25:14 -05:00
Will Dietz
9dd2b8ac7b use libbrotli directly when available
* Look for both 'brotli' and 'bro' as external command,
  since upstream has renamed it in newer versions.
  If neither are found, current runtime behavior
  is preserved: try to find 'bro' on PATH.
* Limit amount handed to BrotliEncoderCompressStream
  to ensure interrupts are processed in a timely manner.
  Testing shows negligible performance impact.
  (Other compression sinks don't seem to require this)
2017-12-30 20:26:33 -06:00
Shea Levy
6a0dd63508
Merge branch 'fix/issue-1757' of git://github.com/dtzWill/nix 2017-12-27 18:51:05 -05:00
Will Dietz
bd17ccf1d8 nix repl: use linenoiseKeyType to differentiate ^C and ^D
Fixes #1757.
2017-12-26 19:25:50 -06:00
Will Dietz
2e6f06c37e fetchGit: Fix handling of local repo when not using 'master' branch
Add tests checking this behavior.
2017-12-22 15:29:52 -06:00
Eelco Dolstra
aa43cbb764
Check aws-sdk-cpp version 2017-12-22 12:05:13 +01:00
Will Dietz
71987b18d4 linenoise.cpp: allow completions from empty input
Fixes #1742.
2017-12-19 12:52:10 -06:00
Eelco Dolstra
8efe937a35 Fix build on gcc 7
Fixes #1738.
2017-12-16 00:49:31 +01:00
Ben Gamari
626a94d70e edit: Catch stoi exceptions from line number parsing 2017-12-14 19:11:56 -05:00
Ben Gamari
f9bcbddef2 json-to-value: Throw sensible error message on invalid numbers 2017-12-14 19:09:45 -05:00
Eelco Dolstra
5d5b931fb1 Fix build against current aws-sdk-cpp 2017-12-14 16:45:40 +01:00
Maximilian Bosch
0413aeb35d
search.cc: sort attribute names with std::map 2017-12-12 15:13:36 +01:00
Eelco Dolstra
1ca8e95178
Merge pull request #1722 from bhipple/fix-for-older-libcurl
Fix for builds with system libcurl < 7.30
2017-12-12 11:48:08 +01:00
Will Dietz
00e0c416ff Mark comparison call operator as const 2017-12-11 12:08:40 -06:00
Will Dietz
e4d43f1f94 logging.cc: add missing 'override' 2017-12-11 12:08:40 -06:00
Will Dietz
c800f47395 Fix forward-decl of Bindings as "class", match definition.
(appease clang -Wmismatched-tags warning)
2017-12-11 11:51:55 -06:00
Shea Levy
5ba5ca7888
Merge branch 'feature/linenoise-ng' of git://github.com/dtzWill/nix 2017-12-11 07:52:20 -05:00
Benjamin Hipple
d76c4fe770 Fix for builds with system libcurl < 7.30
CentOS 7.4 and RHEL 7.4 ship with libcurl-devel-7.29.0-42.el7.x86_64; this flag
was added in 7.30.0
https://curl.haxx.se/libcurl/c/CURLMOPT_MAX_TOTAL_CONNECTIONS.html
2017-12-09 20:02:21 -05:00
Eelco Dolstra
2df9cbeb47 Provide random access to cached NARs
E.g.

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nars \
    /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79/share/icons/hicolor/scalable/apps/blender.svg > /dev/null
  real    0m4.139s

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nars \
    /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79/share/icons/hicolor/scalable/apps/blender.svg > /dev/null
  real    0m0.024s

(Before, the second call took ~0.220s.)

This will use a NAR listing in
/tmp/nars/b0w2hafndl09h64fhb86kw6bmhbmnpm1.ls containing all metadata,
including the offsets of regular files inside the NAR. Thus, we don't
need to read the entire NAR. (We do read the entire listing, but
that's generally pretty small. We could use a SQLite DB by borrowing
some more code from nixos-channel-scripts/file-cache.hh.)

This is primarily useful when Hydra is serving files from an S3 binary
cache, in particular when you have giant NARs. E.g. we had some 12 GiB
NARs, so accessing individuals files was pretty slow.
2017-12-07 01:07:07 +01:00
Eelco Dolstra
338f29dbd4 nix ls-{nar,store}: Return offset of files in the NAR if known
E.g.

  $ nix ls-store --json --recursive --store https://cache.nixos.org /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79 \
    | jq .entries.bin.entries.blender.narOffset
  400
2017-12-07 01:07:07 +01:00
Daiderd Jordan
2cb8aaa610
ssh-store: fix length when writing nar
This fixes nix copy and other things that use copyStorePath.
2017-12-06 11:41:08 +01:00
Shea Levy
11a7f8ce14
Merge branch 'fetchGit-fast-revision-update' 2017-12-05 11:16:39 -05:00
Eelco Dolstra
7f2c324ed1
Simplify build by including nlohmann/json.hpp 2017-12-04 17:11:36 +01:00
Eelco Dolstra
f1efb97075
Fix #1635. 2017-12-04 13:19:16 +01:00
Will Dietz
79f5c296c0 ignore "interrupted" exception in progress callback
Context/discusson:

https://github.com/NixOS/nix/issues/1692#issuecomment-348282301
2017-11-30 12:45:54 -06:00
Linus Heckemann
f986a44980 Fix "Unexpected EOF reading a line" error
propagated-user-env-packages files in nixpkgs aren't all terminated by
newlines, as buildenv expected. Now it does not require a terminating
newline; note that this introduces a behaviour change: propagated user
env packages may now be spread across multiple lines. However, nix
1.11.x still expects them to be on a single line so this shouldn't be
used in nixpkgs for now.
2017-11-29 11:56:15 +00:00
Eelco Dolstra
b52846ab5b
Show log tail when a remote build fails 2017-11-28 15:36:28 +01:00
Will Dietz
964349e44d no "linenoiseFree" in linenoise-ng 2017-11-27 18:30:05 -06:00
Will Dietz
21f515724c replace vendored linenoise with linenoise-ng 2017-11-27 18:27:28 -06:00
aszlig
6567ab95a0
build-remote: Fix missing log output
The storeUri variable in the build-remote hook is declared very much to
the start of the main function and a bunch of lines later, the same
variable gets checked via hasPrefix() but it gets assigned *after* that
check when the most suitable machine for the build was choosen.

So I guess this was just a typo in d16fd24973
and what we really want is to either checkd the prefix *after* assigning
storeUri or use bestMachine->storeUri directly.

I choose the latter, because the former could introduce even more
regressions if the try block where the variable gets assigned terminates
early.

Nevertheless, the reason why the log output didn't work is because
hasPrefix() checked for "ssh://" in front of storeUri, but if the
storeUri isn't set correctly (or at all), we don't get the log file
descriptor set up properly, leading to no log output.

I've adjusted the remote-builds test to include a regression test for
this, so that we can make sure we get a build output when using remote
builds.

In addition to that I've tested this with two of my build farms and the
build logs are emitted correctly again.

Signed-off-by: aszlig <aszlig@nix.build>
2017-11-25 01:34:18 +01:00
Eelco Dolstra
549c3706a5
nix path-info: Show URL of NARs in binary caches 2017-11-24 18:08:50 +01:00
Eelco Dolstra
90948a4e3a
nix-shell/nix-build: Support .drv files again
Fixes #1663.

Also handle '!<output-name>' (#1694).
2017-11-24 18:08:35 +01:00
Shea Levy
eedbc4e06c
fetchGit: Ignore tarballTtl if rev is set and not in the repo.
Fixes #1697.
2017-11-24 06:09:24 -05:00
Eelco Dolstra
d7da6c9ea9
fetchGit/fetchMercurial: Fix directory inclusion check
E.g. the existence of .gitignore would cause .git to be included.
2017-11-21 19:34:46 +01:00
Eelco Dolstra
6cdaa858d0
Propagate flags like --sandbox to the daemon properly 2017-11-21 18:50:56 +01:00
Eelco Dolstra
f5f29dc2b7
nix eval --raw: Use coerceToString()
Thus,

  $ nix eval --raw '("foo")'
  foo

  $ nix eval --raw nixpkgs.hello
  /nix/store/1y6ckg6khrdsvll54s5spcmf3w6ka9k4-hello-2.10

  $ nix eval --raw '(/etc/resolv.conf)'
  /nix/store/vml92ama92i8mz013nny461mlvg8mvap-resolv.conf
2017-11-20 18:27:29 +01:00
Eelco Dolstra
1ff01187e2
nix run: Fix "flag '--command' requires 2 argument(s)" 2017-11-20 18:07:58 +01:00
Eelco Dolstra
4eb9e20028
nix run: Fix accidental removal of /nix/store existence check
Parenthetical to #1686, we don't need to create a new root if we can
just bind-mount on top of the existing /nix/store.
2017-11-20 17:59:32 +01:00
Eelco Dolstra
a3aa850f0f
nix run: Ignore non-directories while setting up the chroot
Fixes #1686.
2017-11-20 17:58:47 +01:00
Eelco Dolstra
91a1987607
signed-binary-caches -> require-sigs
Unlike signed-binary-caches (which could only be '*' or ''),
require-sigs is a proper Boolean option. The default is true.
2017-11-20 17:44:07 +01:00
Eelco Dolstra
d0b88db441
Cleanup 2017-11-20 17:32:58 +01:00
Eelco Dolstra
7a2b64e55c
binary-cache-public-keys -> trusted-public-keys
The name had become a misnomer since it's not only for substitution
from binary caches, but when adding/copying any
(non-content-addressed) path to a store.
2017-11-20 17:32:34 +01:00
Eelco Dolstra
72804dc0bd
Merge pull request #1645 from twhitehead/stat-race
Fix (highly unlikely) race condition in readLink
2017-11-20 17:32:27 +01:00
Eelco Dolstra
7474ac871b
nix copy: Abbreviate "daemon" 2017-11-20 15:17:11 +01:00
Eelco Dolstra
8956ae1987
Add a "profile" option to S3BinaryCacheStore
This allows specifying the AWS configuration profile to use. E.g.

  nix copy --from s3://my-cache?profile=aws-dev-account /nix/store/cf3isrlqavvd5w7rpky1fa8j9lcnlggm-...
2017-11-15 14:18:20 +01:00
Will Dietz
9d7ce0bf45 anchor nix::Exit exception 2017-11-14 12:39:30 -06:00
Eelco Dolstra
ec5b04862b
nix sign-paths: Support binary caches 2017-11-14 18:44:05 +01:00
Eelco Dolstra
2c39e4eca0
Revert "Don't parse "x:x" as a URI"
This reverts commit f90f660b24.

This broke Hydra's release.nix, which contained

  preCheck = ''export LOGNAME=${LOGNAME:-foo}'';
2017-11-14 15:10:52 +01:00
Eelco Dolstra
5773d667ee
nix ls-{nar,store}: Don't abort on missing files 2017-11-14 14:49:06 +01:00
Eelco Dolstra
4db0a9555e
nix ls-{nar,store} --json: Respect -R 2017-11-14 14:31:38 +01:00
Eelco Dolstra
c8155e9f5f
Don't indent JSON output 2017-11-14 14:28:03 +01:00
Eelco Dolstra
bac8055652
nix ls-{store,nar}: Add --json flag 2017-11-14 14:23:53 +01:00
Eelco Dolstra
5ef5d70b5e
Make config options available to legacy commands 2017-11-14 14:04:09 +01:00
Eelco Dolstra
9d87d03331
Remove extraneous comment 2017-11-08 16:17:32 +01:00
Eelco Dolstra
513b143cd8
Merge pull request #1650 from copumpkin/darwin-sandbox-unix-socket
Always allow builds to use unix domain sockets in Darwin sandbox
2017-11-08 16:16:42 +01:00
Eelco Dolstra
dc30856141
Merge pull request #1632 from AmineChikhaoui/sigint-copy
run query paths in parallel during nix copy and handle SIGINT
2017-11-06 13:36:28 +01:00
Eelco Dolstra
7a4d9574d9
fetchgit.cc -> fetchGit.cc 2017-11-03 13:55:31 +01:00
Eelco Dolstra
6cf7c6a6b0
Remove git:// support in NIX_PATH
This didn't support specifying a revision/branch, and was restricted
to git:// URIs (since https:// or ssh:// would be ambiguous).
2017-11-03 13:55:31 +01:00
Eelco Dolstra
ee6ac38848
fetchGit/fetchMercurial: Filter out directories with untracked files 2017-11-03 13:55:31 +01:00
Eelco Dolstra
4dee01da7c
fetchGit: Add a test 2017-11-03 13:55:30 +01:00
Eelco Dolstra
0e77aa3982
fetchGit: Don't do a remote fetch if we already have the rev 2017-11-03 13:55:30 +01:00
Eelco Dolstra
9ccea31dc2
Merge pull request #1651 from LnL7/darwin-sandbox-getpwuid
Allow getpwuid in the darwin sandbox
2017-11-03 10:55:31 +01:00
Daiderd Jordan
453f675810
Allow getpwuid in the darwin sandbox. 2017-11-03 10:50:49 +01:00
Eelco Dolstra
e104842f8b
Merge pull request #1660 from 4z3/patch-1
fetchMercurial: fix error message
2017-11-03 10:49:57 +01:00
Eelco Dolstra
4070264613
Merge pull request #1655 from copumpkin/patch-1
Don't freak out if we get a 403 from S3
2017-11-03 10:49:38 +01:00
Daniel Peebles
3105679226 Don't freak out if we get a 403 from S3
As far as we're concerned, not being able to access a file just means
the file is missing. Plus, AWS explicitly goes out of its way to
return a 403 if the file is missing and the requester doesn't have
permission to list the bucket.

Also getting rid of an old hack that Eelco said was only relevant
to an older AWS SDK.
2017-11-03 01:31:42 +01:00
tv
5ab37f0e44
fetchMercurial: fix error message 2017-11-02 23:37:42 +01:00
Eelco Dolstra
212e72c609 Fix build
https://hydra.nixos.org/build/63172338
2017-11-01 21:32:30 +01:00
Eelco Dolstra
e026bc3b05
fetchMercurial: Don't fetch hashes we already have 2017-11-01 18:43:11 +01:00
Eelco Dolstra
1969f357b7
Add fetchMercurial primop
E.g.

  $ nix eval '(fetchMercurial https://www.mercurial-scm.org/repo/hello)'
  { branch = "default"; outPath = "/nix/store/alvb9y1kfz42bjishqmyy3pphnrh1pfa-source"; rev = "82e55d328c8ca4ee16520036c0aaace03a5beb65"; revCount = 1; shortRev = "82e55d328c8c"; }

  $ nix eval '(fetchMercurial { url = https://www.mercurial-scm.org/repo/hello; rev = "0a04b987be5ae354b710cefeba0e2d9de7ad41a9"; })'
  { branch = "default"; outPath = "/nix/store/alvb9y1kfz42bjishqmyy3pphnrh1pfa-source"; rev = "0a04b987be5ae354b710cefeba0e2d9de7ad41a9"; revCount = 0; shortRev = "0a04b987be5a"; }

  $ nix eval '(fetchMercurial /tmp/unclean-hg-tree)'
  { branch = "default"; outPath = "/nix/store/cm750cdw1x8wfpm3jq7mz09r30l9r024-source"; rev = "0000000000000000000000000000000000000000"; revCount = 0; shortRev = "000000000000"; }
2017-11-01 17:45:32 +01:00
Eelco Dolstra
cd532a9251
Fix filterSource 2017-10-31 16:18:32 +01:00
Dan Peebles
bc6b3f7e8f Always allow builds to use unix domain sockets in Darwin sandbox 2017-10-31 15:33:57 +01:00
Eelco Dolstra
72cd52c3cd
builtins.fetchgit: Support importing a working tree
For example, you can write

  src = fetchgit ./.;

and if ./. refers to an unclean working tree, that tree will be copied
to the Nix store. This removes the need for "cleanSource".
2017-10-30 19:59:25 +01:00
Eelco Dolstra
197922ea4e
Merge pull request #1646 from copumpkin/optional-sandbox-local-network
Allow optional localhost network access to sandboxed derivations
2017-10-30 18:54:40 +01:00
Dan Peebles
4a4a009f78 Allow optional localhost network access to sandboxed derivations
This will allow bind and connect to 127.0.0.1, which can reduce purity/
security (if you're running a vulnerable service on localhost) but is
also needed for a ton of test suites, so I'm leaving it turned off by
default but allowing certain derivations to turn it on as needed.

It also allows DNS resolution of arbitrary hostnames but I haven't found
a way to avoid that. In principle I'd just want to allow resolving
localhost but that doesn't seem to be possible.

I don't think this belongs under `build-use-sandbox = relaxed` because we
want it on Hydra and I don't think it's the end of the world.
2017-10-30 17:59:12 +01:00
Eelco Dolstra
f90f660b24
Don't parse "x:x" as a URI
URIs now have to contain "://" or start with "channel:".
2017-10-30 17:58:01 +01:00
Tyson Whitehead
07d2c6d213
Fix (highly unlikely) race condition in readLink
Used to determine symlink size with stat and value with readlink.
This could technically result in garbage if symlink changed between
calls.  Also gets around the broken stat implementation in our
network filesystem (returns size + 1 giving a byte of garbage).
2017-10-30 11:49:55 -04:00
Eelco Dolstra
12991152be
nix-build: Fix --hash 2017-10-30 13:31:05 +01:00
Eelco Dolstra
63c80ae26f
Make "fetchGit /path" work 2017-10-30 13:18:28 +01:00
Eelco Dolstra
a5c392a80e
fetchGit: Fix broken assertion
Different URIs can map to the same cache entry if they have the same
revision.
2017-10-30 12:55:46 +01:00
Eelco Dolstra
812e027e1d
Add option allowed-uris
This allows network access in restricted eval mode.
2017-10-30 12:41:49 +01:00
Eelco Dolstra
e38382895d
builtins.fetchGit: Return an attrset with revision info
This adds rev, shortRev and revCount attributes, equal to what Hydra
provides. E.g.

  $ nix eval '(fetchGit https://github.com/NixOS/patchelf.git)'
  { outPath = "/nix/store/ghigrkw02l440g8vfxa9wj4c3zpfmw99-source"; rev = "29c085fd9d3fc972f75b3961905d6b4ecce7eb2b"; revCount = 303; shortRev = "29c085f"; }
2017-10-30 11:49:03 +01:00
Eelco Dolstra
f9686885be
enable-http2 -> http2 2017-10-30 11:00:59 +01:00
Eelco Dolstra
049322702b
fetchgit -> fetchGit
Almost all other primops are camelCase so no reason not to use that
here.
2017-10-30 10:25:08 +01:00
Eelco Dolstra
23ce4b3393
fetchTarball: Use "source" as the default name
This ensures that it produces the same output as fetchgit:

  $ nix eval --raw '(builtins.fetchgit https://github.com/NixOS/patchelf.git)'
  /nix/store/ghigrkw02l440g8vfxa9wj4c3zpfmw99-source

  $ nix eval --raw '(fetchTarball https://github.com/NixOS/patchelf/archive/master.tar.gz)'
  /nix/store/ghigrkw02l440g8vfxa9wj4c3zpfmw99-source
2017-10-30 10:22:58 +01:00
Eelco Dolstra
66ddbef754
fetchurl/fetchTarball: Respect name changes
The computation of urlHash didn't take the name into account, so
subsequent fetchurl calls with the same URL but a different name would
resolve to the same cached store path.
2017-10-30 10:22:58 +01:00
Eelco Dolstra
f25791c196
builtins.fetchgit: Don't mess up the progress indicator 2017-10-30 10:22:58 +01:00
Eelco Dolstra
65b5f177b5
builtins.fetchgit: Support a "name" attribute
The "name" attribute defaults to "source", which we should use for all
similar functions (e.g. fetchTarball and in Hydra) to ensure that we
get a consistent store path regardless of how the tree is fetched.

"source" is not necessarily a correct label, but using an empty name
is problematic: you get an ugly store path ending in a dash, and it's
impossible to have a fixed-output derivation that produces that path
because ".drv" is not a valid store name.

Fixes #904.
2017-10-30 10:22:07 +01:00
Eelco Dolstra
c1ae18941a
nix repl :t: Indicate strings that have a context 2017-10-29 17:11:22 +01:00
AmineChikhaoui
0f9a7225ab respect SIGINT in nix copy during the paths queries #1629 2017-10-25 16:51:45 +01:00
AmineChikhaoui
54a2cd9ce4 Merge branch 'master' of github.com:NixOS/nix into sigint-copy 2017-10-25 16:50:52 +01:00
Eelco Dolstra
9971d875a4
Fix building on clang
https://hydra.nixos.org/build/62945761
2017-10-25 17:20:47 +02:00
AmineChikhaoui
9f01a3f0a8 attempt to fix #1630: make the queries of store paths run in parallel using a thread pool 2017-10-25 16:13:49 +01:00
Eelco Dolstra
82327e3cc4
exportReferencesGraph: Allow exporting a list of store paths 2017-10-25 15:18:49 +02:00
Eelco Dolstra
8191992c83
Bump language version 2017-10-25 15:18:37 +02:00
Eelco Dolstra
3395e3bbc4
Fix exportReferencesGraph in the structured attrs case 2017-10-25 14:08:29 +02:00
Eelco Dolstra
2d5b1b24bf
Pass lists/attrsets to bash as (associative) arrays 2017-10-25 13:01:50 +02:00
Eelco Dolstra
86aab6e9a0
Add --store argument to legacy commands 2017-10-24 15:41:11 +02:00
Eelco Dolstra
d16fd24973
Allow shorter syntax for chroot stores
You can now say '--store /tmp/nix' instead of '--store local?root=/tmp/nix'.
2017-10-24 15:32:38 +02:00
Eelco Dolstra
3460e4cf00
More progress indicator improvements
In particular, don't show superfluous "fetching path" and "building
path(s)" messages, and show the current round (with --repeat).
2017-10-24 15:32:38 +02:00
Eelco Dolstra
96051dd057
More progress indicator improvements
Fixes #1599.
2017-10-24 14:47:23 +02:00
Eelco Dolstra
be220702a7
Progress indicator: Show on what machine we're building
E.g.

  $ nix build nixpkgs.hello --builders 'root@wendy'
  [1/0/1 built] building hello-2.10 on ssh://root@wendy: checking for minix/config.h... no
2017-10-24 14:24:57 +02:00
Eelco Dolstra
f8624762ac
build-remote: Work properly on a chrooted store 2017-10-24 14:14:28 +02:00
Eelco Dolstra
fe9d2f974d
Handle log messages from build-remote
This makes the progress indicator show statuses like "connecting to
'root@machine'".
2017-10-24 14:05:22 +02:00
Eelco Dolstra
0d59f1ca49
nix: Respect -I, --arg, --argstr
Also, random cleanup to argument handling.
2017-10-24 12:58:34 +02:00
Eelco Dolstra
25f32625e2
Remove the remote-builds option
This is superfluous since you can now just set "builders" to empty,
e.g. "--builders ''".
2017-10-24 11:00:16 +02:00
Eelco Dolstra
af241ae7d3
Remove the builder-files option
You can now include files via the "builders" option, using the syntax
"@<filename>". Having only one option makes it easier to override
builders completely.

For backward compatibility, the default is "@/etc/nix/machines", or
"@<filename>" for each file name in NIX_REMOTE_SYSTEMS.
2017-10-24 10:54:43 +02:00
Eelco Dolstra
37fbfffd8e
Pass all settings to build-remote
This ensures that command line flags such as --builders get passed
correctly.
2017-10-23 20:50:28 +02:00
Eelco Dolstra
f32cdc4fab
Turn $NIX_REMOTE into a configuration option 2017-10-23 20:50:28 +02:00
Eelco Dolstra
863a45f1f3
build-remote: Put current load under the store state directory
Fixes the error

  error: opening lock file '/nix/var/nix/current-load/main-lock': Permission denied

when using a chroot store.
2017-10-23 20:50:24 +02:00
Eelco Dolstra
f31c66d392 Merge pull request #1616 from copumpkin/separate-darwin-sandbox
Shift Darwin sandbox to separate installed files
2017-10-20 12:00:27 +02:00
Shea Levy
c7af84ce84
nix-shell: Fix file name resolution in shebangs 2017-10-19 15:02:40 -04:00
Eelco Dolstra
ef1d0142a0
Revert "Let's not populate the NAR cache from hydra-queue-runner for now"
This reverts commit 908590dc6c. Since
hydra-server can have a different store URI from hydra-queue-runner
now, we don't really need this.
2017-10-19 12:11:07 +02:00
Eelco Dolstra
908590dc6c
Let's not populate the NAR cache from hydra-queue-runner for now 2017-10-18 15:45:39 +02:00
Eelco Dolstra
75cd75b1ae
Errors writing to the NAR cache should not be fatal 2017-10-18 15:41:14 +02:00
Eelco Dolstra
d8306148e0
Suppress "copying 0 paths" message 2017-10-18 15:02:58 +02:00
Dan Peebles
6e5165b773 Shift Darwin sandbox to separate installed files
This makes it slightly more manageable to see at a glance what in a
build's sandbox profile is unique to the build and what is standard. Also
a first step to factoring more of our Darwin logic into scheme functions
that will allow us a bit more flexibility. And of course less of that
nasty codegen in C++! 😀
2017-10-17 19:15:51 -04:00
Eelco Dolstra
b24b8ef77c BinaryCacheStore::addToStore(): Add NARs to the local cache 2017-10-17 21:39:48 +02:00
Eelco Dolstra
ca580bec35 BinaryCacheStore: Support local caching of NARs
This speeds up commands like "nix cat-store". For example:

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nar-cache /nix/store/i60yncmq6w9dyv37zd2k454g0fkl3arl-systemd-234/etc/udev/udev.conf
  real    0m4.336s

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nar-cache /nix/store/i60yncmq6w9dyv37zd2k454g0fkl3arl-systemd-234/etc/udev/udev.conf
  real    0m0.045s

The primary motivation is to allow hydra-server to serve files from S3
binary caches. Previously Hydra had a hack to do "nix-store -r
<path>", but that fetches the entire closure so is prohibitively
expensive.

There is no garbage collection of the NAR cache yet. Also, the entire
NAR is read when accessing a single member file. We could generate the
NAR listing to provide random access.

Note: the NAR cache is indexed by the store path hash, not the content
hash, so NAR caches should not be shared between binary caches, unless
you're sure that all your builds are binary-reproducible.
2017-10-17 21:15:33 +02:00
Eelco Dolstra
11ba4302e3 Remove BinaryCacheStoreAccessor
Probably as a result of a bad merge in
4b8f1b0ec0, we had both a
BinaryCacheStoreAccessor and a
RemoteFSAccessor. BinaryCacheStore::getFSAccessor() returned the
latter, but BinaryCacheStore::addToStore() checked for the
former. This probably caused hydra-queue-runner to download paths that
it just uploaded.
2017-10-17 20:51:42 +02:00
Shea Levy
cbc216911d
fetchgit: Remove incomplete/unneeded ref check.
This check fails for tags and branches, and is made redundant by the
checks git itself will do when fetching the repo.
2017-10-17 12:30:54 -04:00
Shea Levy
4e58294ae6
fetchgit: Remove incomplete/unneeded isURI check.
This check spuriously fails for e.g. git@github.com:NixOS/nixpkgs.git,
and even for ssh://git@github.com/NixOS/nixpkgs.git, and is made
redundant by the checks git itself will do when fetching the repo. We
instead pass a -- before passing the URI to git to avoid injection.
2017-10-16 14:35:28 -04:00
Eelco Dolstra
1dd29d7aeb
Add option to disable the seccomp filter
I needed this to test ACL/xattr removal in
canonicalisePathMetaData(). Might also be useful if you need to build
old Nixpkgs that doesn't have the required patches to remove
setuid/setgid creation.
2017-10-12 18:21:55 +02:00
Eelco Dolstra
97307811ee Merge pull request #1571 from teto/improve_priority_msg
Improve error message for conflicting priorities
2017-10-12 13:10:44 +02:00
Eelco Dolstra
177aee07c6 Merge pull request #1580 from copumpkin/retry-after-timeout
Retry in all error cases but a few
2017-10-12 13:05:45 +02:00
Eelco Dolstra
fda7b95cb0
Fix a hang in ThreadPool
The worker threads could exit prematurely if they finished processing
all items while the main thread was still adding items. In particular,
this caused hanging nix-store --serve processes in the build farm.

Also, process items from the main thread.
2017-10-09 15:41:09 +02:00
Dan Peebles
476493dbf5 Reverse retry logic to retry in all but a few cases
It was getting too much like whac-a-mole listing all the retriable error
conditions, so we now retry by default and list the cases where retrying
is almost certainly hopeless.
2017-10-02 23:22:02 -04:00
Eelco Dolstra
f3e0d46821
nix run: Restore signals
Otherwise Ctrl-C doesn't work.
2017-09-28 17:58:59 +02:00
Eelco Dolstra
5421ad243e
nix copy: make recursive by default 2017-09-27 18:28:54 +02:00
Eelco Dolstra
b8875213dc
Tweak message 2017-09-27 13:21:25 +02:00
Eelco Dolstra
bfef3ec855 nix build: Fix realising a store path 2017-09-27 13:14:45 +02:00
Eelco Dolstra
d1c6e0fe8c
nix why-depends: Write to stdout 2017-09-25 16:59:16 +02:00
Eelco Dolstra
e8d6ee7c1b
Add "nix show-derivation"
This debug command prints a store derivation in JSON format. For
example:

  $ nix show-derivation nixpkgs.hello
  {
    "/nix/store/ayjwpwwiyy04nh9z71rsdgd3q7bra7ch-hello-2.10.drv": {
      "outputs": {
        "out": {
          "path": "/nix/store/w5w4v29ql0qwqhczkdxs94ix2lh7ibgs-hello-2.10"
        }
      },
      "inputSrcs": [
        "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
      ],
      "inputDrvs": {
        "/nix/store/13839aqdf6x4k3b785rw5f2l7857l6y3-bash-4.4-p12.drv": [
          "out"
        ],
        "/nix/store/vgdx7fdc7d4iirmnwj2py1nrvr5qwzj7-hello-2.10.tar.gz.drv": [
          "out"
        ],
        "/nix/store/x3kkd0vsqfflbvwf1055l9mr39bg0ms0-stdenv.drv": [
          "out"
        ]
      },
      "platform": "x86_64-linux",
      "builder": "/nix/store/qp5fw57d38bd1n07ss4zxh88zg67c3vg-bash-4.4-p12/bin/bash",
      "args": [
        "-e",
        "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
      ],
      "env": {
        "buildInputs": "",
        "builder": "/nix/store/qp5fw57d38bd1n07ss4zxh88zg67c3vg-bash-4.4-p12/bin/bash",
        "configureFlags": "",
        "doCheck": "1",
        "name": "hello-2.10",
        "nativeBuildInputs": "",
        "out": "/nix/store/w5w4v29ql0qwqhczkdxs94ix2lh7ibgs-hello-2.10",
        "propagatedBuildInputs": "",
        "propagatedNativeBuildInputs": "",
        "src": "/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz",
        "stdenv": "/nix/store/6zngq1rdh0ans9qyckqimqibgnlvlfrm-stdenv",
        "system": "x86_64-linux"
      }
    }
  }

This removes the need for pp-aterm.
2017-09-25 13:43:35 +02:00
Eelco Dolstra
e350671737
Doh 2017-09-25 13:25:55 +02:00
Matthieu Coudron
6920c23701 Improve error message for conflicting priorities
I find the error message 'nix-env --set-flag priority NUMBER PKGNAME'
not as helpful as it could be :
- doesn't share the current priorities
- doesn't say that the command must be run on the already installed
PKGNAME (which is confusing the first time)
- the doc needs careful reading:
"If there are multiple derivations matching a name in args that have the same name (e.g., gcc-3.3.6 and gcc-4.1.1), then the derivation with the highest priority is used."
if one stops reading there, he is screwed. Salvation comes with reading "A derivation can define a priority by declaring the meta.priority attribute. This attribute should be a number, with a higher value denoting a lower priority. The default priority is 0."
To sum it up, lower number wins. I tried to convey this idea in the
message too.
2017-09-21 10:25:20 +09:00
Eelco Dolstra
84f112b1c8
nix-shell: Ignore readFile() errors
Fixes #1563.
2017-09-18 13:28:00 +02:00
Eelco Dolstra
aca4f7dff0
Don't remove Content-Encoding in fetchurl / nix-prefetch-url
Fixes #1568.
2017-09-18 11:07:28 +02:00
Eelco Dolstra
89dc62c174
RemoteStore: Add option to drop old connections from the pool
This is a hack to make hydra-queue-runner free its temproots
periodically, thereby ensuring that garbage collection of the
corresponding paths is not blocked until the queue runner is
restarted.

It would be better if temproots could be released earlier than at
process exit. I started working on a RAII object returned by functions
like addToStore() that releases temproots. However, this would be a
pretty massive change so I gave up on it for now.
2017-09-14 18:10:38 +02:00
Eelco Dolstra
308ecf6361
Show the pid of temporary roots 2017-09-14 15:02:52 +02:00
Eelco Dolstra
da1e4fdfb5
nix-store -q --roots / --gc --print-roots: Print temporary / in-memory roots
For example,

  $ nix-store -q --roots /nix/store/7phd2sav7068nivgvmj2vpm3v47fd27l-patchelf-0.8pre845_0315148
  {temp:1}

denotes that the path is only being kept alive by a temporary root
(i.e. /nix/var/nix/temproots/). Similarly,

  $ nix-store --gc --print-roots
  ...
  {memory:9} -> /nix/store/094gpjn9f15ip17wzxhma4r51nvsj17p-curl-7.53.1

shows that curl is being used by some process.
2017-09-14 14:38:36 +02:00
Eelco Dolstra
970366266b
Add "nix add-to-store" command 2017-09-14 13:29:33 +02:00
Eelco Dolstra
1c58ad2ffa
nix why-depends: Fix showing self-references 2017-09-14 11:17:32 +02:00
Eelco Dolstra
fc0ded3408
nix why-depends: Add option to show all edges causing a dependency
For example, without --all:

  $ nix why-depends nixpkgs.nixUnstable nixpkgs.libssh2
  /nix/store/s9n5gvj2l49b4n19nz6xl832654nf7n7-nix-1.12pre5511_c94f3d55
  └───lib/libnixstore.so: …/lib:/nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0/lib…
      => /nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0
      └───lib/libcurl.la: …ib -L/nix/store/4mbayl1y5hpjbjzkx8ndyhkv98kqw1wi-libssh2-1.8.0/l…
          => /nix/store/4mbayl1y5hpjbjzkx8ndyhkv98kqw1wi-libssh2-1.8.0

but with --all:

  $ nix why-depends -a nixpkgs.nixUnstable nixpkgs.libssh2
  /nix/store/s9n5gvj2l49b4n19nz6xl832654nf7n7-nix-1.12pre5511_c94f3d55
  ├───lib/libnixstore.so: …/lib:/nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0/lib…
  │   => /nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0
  │   └───lib/libcurl.la: …ib -L/nix/store/4mbayl1y5hpjbjzkx8ndyhkv98kqw1wi-libssh2-1.8.0/l…
  │       lib/libcurl.so.4.4.0: …/lib:/nix/store/4mbayl1y5hpjbjzkx8ndyhkv98kqw1wi-libssh2-1.8.0/l…
  │       => /nix/store/4mbayl1y5hpjbjzkx8ndyhkv98kqw1wi-libssh2-1.8.0
  └───lib/libnixstore.so: …/lib:/nix/store/bx2i9vi76lps6w9rr73fxf6my31s4dg5-aws-sdk-cpp-1.0…
      => /nix/store/bx2i9vi76lps6w9rr73fxf6my31s4dg5-aws-sdk-cpp-1.0.153
      └───lib/libaws-cpp-sdk-core.so: …e.so./nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0/lib…
          lib/libaws-cpp-sdk-s3.so: …/lib:/nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0/lib…
          => /nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0
2017-09-11 16:21:27 +02:00
Eelco Dolstra
d41c5eb13f Add command "nix why-depends"
This command shows why a package has another package in its runtime
closure. For example, to see why VLC has libdrm.dev in its closure:

  $ nix why-depends nixpkgs.vlc nixpkgs.libdrm.dev
  /nix/store/g901z9pcj0n5yy5n6ykxk3qm4ina1d6z-vlc-2.2.5.1:
    lib/libvlccore.so.8.0.0: …nfig:/nix/store/405lmx6jl8lp0ad1vrr6j498chrqhz8g-libdrm-2.4.75-d…

  /nix/store/s3nm7kd8hlcg0facn2q1ff2n7wrwdi2l-mesa-noglu-17.0.7-dev:
    nix-support/propagated-native-build-inputs: …-dev /nix/store/405lmx6jl8lp0ad1vrr6j498chrqhz8g-libdrm-2.4.75-d…

Thus, VLC's lib/libvlccore.so.8.0.0 as well as mesa-noglu's
nix-support/propagated-native-build-inputs cause the dependency.
2017-09-10 22:40:33 +02:00
Eelco Dolstra
8af704eef1 Typo 2017-09-10 22:40:33 +02:00
Eelco Dolstra
ad228d84e5 nix build: Only download the requested derivation outputs
Also some refactoring.
2017-09-10 22:40:33 +02:00
Eelco Dolstra
3ed8290e53
LegacySSHStore: Include signatures etc. 2017-09-08 16:55:27 +02:00
Eelco Dolstra
e02edb1483
nix copy: Add --substitute flag 2017-09-08 15:32:07 +02:00
Eelco Dolstra
b7376edf06
ThreadPool: On exception, interrupt the other worker threads 2017-09-08 15:31:24 +02:00
Eelco Dolstra
6a888ec29a
copyStorePath(): Fill in missing narHash regardless of checkSigs
I don't remember what the reasoning was here, but security is provided
by the signatures, not by whether the hash is provided by the other
store.
2017-09-08 14:48:08 +02:00
Eelco Dolstra
3cf1705583
nix copy: Don't open the --from store twice 2017-09-08 14:46:55 +02:00
Eelco Dolstra
8f6b347abd
ThreadPool: Improve exception handling
In particular, process() won't return as long as there are active
items. This prevents work item lambdas from referring to stack frames
that no longer exist.
2017-09-08 14:42:09 +02:00
Eelco Dolstra
a2740c9ca2 nix copy: Add examples 2017-09-08 11:33:46 +02:00
Eelco Dolstra
88e6bb76de Communicate the structured log FD to builders
Since we may use a dedicated file descriptor in the future, this
allows us to change it. So builders can do

  if [[ -n $NIX_LOG_FD ]]; then
    echo "@nix { message... }" >&$NIX_LOG_FD
  fi
2017-09-08 11:27:10 +02:00
Eelco Dolstra
24e23a1a73 nix search: Add examples 2017-09-07 20:42:11 +02:00
Eelco Dolstra
4967f0509a nix eval: Add examples 2017-09-07 20:37:46 +02:00
Eelco Dolstra
2ebeffcfd4 nix log: Add examples 2017-09-07 20:18:29 +02:00
Eelco Dolstra
17afc42895 nix build: Add examples 2017-09-07 20:14:04 +02:00
Eelco Dolstra
2e9b7c4cb2 nix run: Add some examples 2017-09-07 20:09:04 +02:00
Eelco Dolstra
dff440aab3
nix build: Add --out-link and --no-link options 2017-09-06 16:20:34 +02:00