Commit graph

3278 commits

Author SHA1 Message Date
Eelco Dolstra
5cc8609e30
nix run: Allow passing a command to execute
E.g.

  nix run nixpkgs.hello -c hello --greeting Hallo

Note that unlike "nix-shell --command", no quoting of arguments is
necessary.

"-c" (short for "--command") cannot be combined with "--" because they
both consume all remaining arguments. But since installables shouldn't
start with a dash, this is unlikely to cause problems.
2017-08-29 14:42:48 +02:00
Eelco Dolstra
93a5ef0516
nix run: Fix chroot execution
Running "nix run" with a diverted store, e.g.

  $ nix run --store local?root=/tmp/nix nixpkgs.hello

stopped working when Nix became multithreaded, because
unshare(CLONE_NEWUSER) doesn't work in multithreaded processes. The
obvious solution is to terminate all other threads first, but 1) there
is no way to terminate Boehm GC marker threads; and 2) it appears that
the kernel has a race where unshare(CLONE_NEWUSER) will still fail for
some indeterminate amount of time after joining other threads.

So instead, "nix run" will now exec() a single-threaded helper ("nix
__run_in_chroot") that performs the actual unshare()/chroot()/exec().
2017-08-29 13:21:07 +02:00
Eelco Dolstra
1c58e13bee
Hide commands that don't have a description
These are assumed to be internal.
2017-08-29 11:52:55 +02:00
Eelco Dolstra
05c45f301d
readLink(): Handle symlinks in /proc
Symlinks like /proc/self/exe report a stat() size of 0, so use a
buffer of at least PATH_MAX instead.
2017-08-29 11:52:34 +02:00
Eelco Dolstra
2cc345b95f
Give activities a verbosity level again
And print them (separately from the progress bar) given sufficient -v
flags.
2017-08-28 19:13:24 +02:00
Eelco Dolstra
cfc8132391
Don't send progress messages to older clients 2017-08-28 18:54:23 +02:00
Eelco Dolstra
fe34b91289
Tunnel progress messages from the daemon to the client
This makes the progress bar work for non-root users.
2017-08-28 18:49:42 +02:00
Eelco Dolstra
e681b1f064
Simplify 2017-08-28 14:30:35 +02:00
Eelco Dolstra
8fff3e7bb5
Make TunnelLogger thread-safe
Now that we use threads in lots of places, it's possible for
TunnelLogger::log() to be called asynchronously from other threads
than the main loop. So we need to ensure that STDERR_NEXT messages
don't clobber other messages.
2017-08-28 14:17:07 +02:00
Eelco Dolstra
94a0548dc4
Simplify 2017-08-25 21:26:37 +02:00
Eelco Dolstra
9b845e6936
Doh 2017-08-25 20:52:34 +02:00
Eelco Dolstra
0ac35b67b8
Allow derivations to update the build phase
So the progress bar can show

  [1/0/1 built, 0.0 MiB DL] building hello-2.10 (configuring): checking whether pread is declared without a macro... yes
2017-08-25 18:04:05 +02:00
Eelco Dolstra
c137c0a5eb
Allow activities to be nested
In particular, this allows more relevant activities ("substituting X")
to supersede inferior ones ("downloading X").
2017-08-25 17:49:40 +02:00
Eelco Dolstra
f194629f96
Fix Debian build
https://hydra.nixos.org/build/59390148
2017-08-25 16:11:18 +02:00
Eelco Dolstra
db1d45037c
Handle SIGWINCH 2017-08-25 15:59:03 +02:00
Eelco Dolstra
ec9e0c03c3
When truncating the progress bar, take ANSI escape sequences into account 2017-08-25 15:59:03 +02:00
Eelco Dolstra
a3015db6c3
Typo 2017-08-25 15:59:03 +02:00
Eelco Dolstra
0e9ddcc306
Restore activity metadata
This allows the progress bar to display "building perl-5.22.3" instead
of "building /nix/store/<hash>-perl-5.22.3.drv".
2017-08-25 15:58:35 +02:00
Eelco Dolstra
1f56235438
Clean up JSON construction 2017-08-21 12:18:46 +02:00
Eelco Dolstra
4af2611bd1
Allow builders to create activities
Actually, currently they can only create download activities. Thus,
downloads by builtins.fetchurl show up in the progress bar.
2017-08-21 12:18:46 +02:00
Eelco Dolstra
4c6a26539c
Remove debug line 2017-08-21 12:18:46 +02:00
Eelco Dolstra
9400cb36b7
Disallow accidental copy construction 2017-08-21 12:18:42 +02:00
Andy Wingo
6bb4e3e8fe Remove unused decodeOctalEscaped
Besides being unused, this function has a bug that it will incorrectly
decode the path component Ubuntu\04016.04.2\040LTS\040amd64 as
"Ubuntu.04.2 LTS amd64" instead of "Ubuntu 16.04.2 LTS amd64".
2017-08-18 11:07:00 +02:00
Eelco Dolstra
2ee1b9359b Merge branch 'tokenize' of https://github.com/nbp/nix 2017-08-16 21:21:36 +02:00
Eelco Dolstra
c2cab20732
nix verify: Restore the progress indicator 2017-08-16 20:56:03 +02:00
Eelco Dolstra
b4ed97e3a3
nix optimise-store: Show how much space has been freed 2017-08-16 20:56:03 +02:00
Eelco Dolstra
23b8b7e096
nix optimise-store: Add
This replaces "nix-store --optimise". Main difference is that it has a
progress indicator.
2017-08-16 20:56:03 +02:00
Eelco Dolstra
40bffe0a43
Progress indicator: Cleanup 2017-08-16 20:56:03 +02:00
Eelco Dolstra
dff12b38f9
Progress indicator: More improvements 2017-08-16 20:56:03 +02:00
Eelco Dolstra
bf1f123b09
Progress indicator: Show number of active items 2017-08-16 20:56:03 +02:00
Eelco Dolstra
0e0dcf2c7e
Progress indicator: Unify "copying" and "substituting"
They're the same thing after all.

Example:

  $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped
  [0/1 built, 49/98 copied, 16.3/92.8 MiB DL, 55.8/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/0pl9li1jigcj2dany47hpmn0r3r48wc4nz48v5mqhh426lgz3bz6.nar.xz'
2017-08-16 20:56:03 +02:00
Eelco Dolstra
c36467ad2e
Improve substitution progress indicator
E.g.

  $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped
  [0/1 built, 1/97/98 fetched, 65.8/92.8 MiB DL, 203.2/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/1czm9fk0svacy4h6a3fzkpafi4f7a9gml36kk8cq1igaghbspg3k.nar.xz'
2017-08-16 20:56:02 +02:00
Eelco Dolstra
b29b6feaba
nix copy: Improve progress indicator
It now shows the amount of data copied:

  [8/1038 copied, 160.4/1590.9 MiB copied] copying path '...'
2017-08-16 20:56:02 +02:00
Eelco Dolstra
c5e4404580
nix copy: Revive progress bar 2017-08-16 20:56:02 +02:00
Eelco Dolstra
dffc3fe43b
nix copy: Add --no-check-sigs flag 2017-08-16 20:56:02 +02:00
Eelco Dolstra
e4bd42f98f
Disallow SSH host names starting with a dash 2017-08-16 20:55:58 +02:00
Nicolas B. Pierron
b8867a0239 Add builtins.string function.
The function 'builtins.split' takes a POSIX extended regular expression
and an arbitrary string. It returns a list of non-matching substring
interleaved by lists of matched groups of the regular expression.

```nix
with builtins;
assert split "(a)b" "abc"      == [ "" [ "a" ] "c" ];
assert split "([ac])" "abc"    == [ "" [ "a" ] "b" [ "c" ] "" ];
assert split "(a)|(c)" "abc"   == [ "" [ "a" null ] "b" [ null "c" ] "" ];
assert split "([[:upper:]]+)" "  FOO   "
                               == [ "  " [ "FOO" ] "   " ];
```
2017-08-15 20:04:11 +00:00
Eelco Dolstra
b6ee5e5bf0
Style fix 2017-08-10 13:51:07 +02:00
Eelco Dolstra
af765a8eab
Use /proc/self/fd to efficiently close all FDs on Linux
Issue #1506.
2017-08-09 16:22:05 +02:00
Eelco Dolstra
c6184dec6c
nix repl: Support printing floating-point numbers 2017-08-09 15:17:29 +02:00
Brian McKenna
b39cc4fc81 Include missing <cstdlib> for abort()
This is needed to get Nix compiled using Android NDK.
2017-08-03 07:03:22 +10:00
Eelco Dolstra
a2778988f2
Merge branch 'macOS' of https://github.com/davidak/nix 2017-07-31 10:31:51 +02:00
Jörg Thalheim
2fd8f8bb99 Replace Unicode quotes in user-facing strings by ASCII
Relevant RFC: NixOS/rfcs#4

$ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 12:32:45 +01:00
davidak
fcb8d6a7a0 replace "OS X" with "macOS" 2017-07-30 12:28:50 +02:00
davidak
92bcb61127 replace "Mac OS X" with "macOS"
except in older release notes where the name was actually Mac OS X.
2017-07-30 12:26:17 +02:00
Eelco Dolstra
c7654bc491
nix-build: Fix regression causing all outputs to be built 2017-07-28 15:17:52 +02:00
Eelco Dolstra
af4689f9e9
nix-prefetch-url: Fix regression in hash printing 2017-07-28 14:56:39 +02:00
Eelco Dolstra
7480f4f9a4
builtins.fetchgit: Support specifying commit hashes
This adds an argument "rev" specififying the Git commit hash. The
existing argument "rev" is renamed to "ref". The default value for
"ref" is "master". When specifying a hash, it's necessary to specify a
ref since we're not cloning the entire repository but only fetching a
specific ref.

Example usage:

  builtins.fetchgit {
    url = https://github.com/NixOS/nixpkgs.git;
    ref = "release-16.03";
    rev = "c1c0484041ab6f9c6858c8ade80a8477c9ae4442";
  };
2017-07-27 18:08:23 +02:00
Eelco Dolstra
9f64cb89cb
builtins.fetchgit: Respect tarball-ttl
I.e. if the local ref is more recent than tarball-ttl seconds, then
don't check the remote.
2017-07-27 17:23:29 +02:00
Eelco Dolstra
69deca194e
builtins.fetchgit: Use proper refs locally 2017-07-27 17:02:25 +02:00
Eelco Dolstra
6d7de7f3de
builtins.fetchgit: Cache hash -> store path mappings
This prevents an expensive call to addToStore() in the cached case.
2017-07-27 16:16:08 +02:00
Eelco Dolstra
57b9505731
nix search: Add a cache
The package list is now cached in
~/.cache/nix/package-search.json. This gives a substantial speedup to
"nix search" queries. For example (on an SSD):

First run: (no package search cache, cold page cache)

  $ time nix search blender
  Attribute name: nixpkgs.blender
  Package name: blender
  Version: 2.78c
  Description: 3D Creation/Animation/Publishing System

  real    0m6.516s

Second run: (package search cache populated)

  $ time nix search blender
  Attribute name: nixpkgs.blender
  Package name: blender
  Version: 2.78c
  Description: 3D Creation/Animation/Publishing System

  real    0m0.143s
2017-07-26 17:29:10 +02:00
Eelco Dolstra
4c9ff89c26
nix-build/nix-shell: Eliminate call to nix-instantiate / nix-store
Note that this removes the need for a derivation symlink, so the
--drv-path and --add-drv-link flags now do nothing.
2017-07-26 17:29:09 +02:00
Eelco Dolstra
c94f3d5575
nix-shell: Use bashInteractive from <nixpkgs>
This adds about 0.1s to nix-shell runtime in the case where
bashInteractive already exists.

See discussion at https://github.com/NixOS/nixpkgs/issues/27493.
2017-07-20 13:50:25 +02:00
Eelco Dolstra
57a30e101b
nix search: Ignore top-level eval errors
$NIX_PATH may contain elements that don't evaluate to an attrset (like
"nixos-config"), so ignore those.
2017-07-20 13:33:13 +02:00
Eelco Dolstra
fc3568e263
FD_SETSIZE check: BuildError -> Error
BuildError denotes a permanent build failure, which is not the case
here.
2017-07-20 13:33:13 +02:00
Eelco Dolstra
b144c4d617
nix search: Add --json flag 2017-07-20 13:33:13 +02:00
Eelco Dolstra
90825dea51
Add "nix search" command 2017-07-20 13:33:13 +02:00
Dmitry Kalinkin
d5e1bffd2a
Do not try to fill fd_set with fd>=FD_SETSIZE
This is UB and causes buffer overflow and crash on linux.
2017-07-18 17:51:50 -04:00
Eelco Dolstra
bf6792c0df
Always use base-16 for hashed mirror lookups
In particular, don't use base-64, which we don't support. (We do have
base-32 redirects for hysterical reasons.)

Also, add a test for the hashed mirror feature.
2017-07-17 13:13:18 +02:00
Eelco Dolstra
49304bae81
Make the hashes mirrors used by builtins.fetchurl configurable
In particular, this allows it to be disabled in our tests.
2017-07-17 13:07:08 +02:00
Eelco Dolstra
4ec6eb1fdf
Fix accidental printError 2017-07-17 11:38:15 +02:00
Eelco Dolstra
766ad5db3b
nix path-info: Show download sizes for binary cache stores
E.g.

  $ nix path-info --json --store https://cache.nixos.org nixpkgs.thunderbird -S
  ...
      "downloadHash": "sha256:1jlixpzi225wwa0f4xdrwrqgi47ip1qpj9p06fyxxg07sfmyi4q0",
      "downloadSize": 43047620,
      "closureDownloadSize": 84745960
    }
  ]
2017-07-14 18:29:10 +02:00
Eelco Dolstra
fdc9da034f
Avoid a call to derivationFromPath()
This doesn't work in read-only mode, ensuring that operations like

  nix path-info --store https://cache.nixos.org -S nixpkgs.hello

(asking for the closure size of nixpkgs.hello in cache.nixos.org) work
when nixpkgs.hello doesn't exist in the local store.
2017-07-14 18:29:10 +02:00
Eelco Dolstra
3908d3929c
nix path-info: Don't barf on invalid paths
Now you get

  [
    {
      "path": "/nix/store/fzvliz4j5xzvnd0w5zgw2l0ksqh578yk-bla",
      "valid": false
    }
  ]
2017-07-14 18:29:10 +02:00
Eelco Dolstra
6438ba22af
StorePathsCommand: Don't build installables
On second though this was annoying. E.g. "nix log nixpkgs.hello" would
build/download Hello first, even though the log can be fetched
directly from the binary cache.

May need to revisit this.
2017-07-14 18:29:07 +02:00
Eelco Dolstra
112ff7833d
nix: Show help when no arguments are given
Fixes #1464.
2017-07-14 13:44:45 +02:00
Eelco Dolstra
0681f8c907
Shut up a memory leak warning 2017-07-14 11:40:57 +02:00
Eelco Dolstra
2965d40612 replaceSymlink(): Handle the case where the temporary file already exists
Not really necessary anymore for #849, but still nice to have.
2017-07-11 23:21:40 +02:00
Eelco Dolstra
8e8caf7f3e fetchTarball: Prevent concurrent downloads of the same file
Fixes #849.
2017-07-11 23:21:24 +02:00
Eelco Dolstra
9c00fa4179 Merge pull request #1422 from nh2/fix-potential-hash-comparison-crash
Fix potential crash/wrong result two hashes of unequal length are compared
2017-07-10 18:09:49 +02:00
Eelco Dolstra
1762b9616c Merge pull request #1428 from rimmington/clearer-regex-space-error
Clearer error message when regex exceeds space limit
2017-07-10 11:45:05 +02:00
Rhys
17bb00d378 Clearer error message when regex exceeds space limit 2017-07-10 09:35:53 +10:00
Eelco Dolstra
d3713716b6 Merge pull request #1445 from matthewbauer/macos-skip-hardlink
Don’t hardlink disallowed paths in OS X.
2017-07-07 11:05:21 +02:00
Matthew Bauer
72e80c59b5 Don’t hardlink disallowed paths in OS X.
Fixes #1443
2017-07-06 19:30:19 -07:00
Eelco Dolstra
a3dc1e65ab
Add X32 to the seccomp filter
Fixes #1432.
2017-07-04 19:00:51 +02:00
Eelco Dolstra
42c5774e78
Sort substituters by priority
Fixes #1438.
2017-07-04 16:34:53 +02:00
Eelco Dolstra
b7203e853e
getDefaultSubstituters(): Simplify initialisation
As shlevy pointed out, static variables in C++11 have thread-safe
initialisation.
2017-07-04 16:26:48 +02:00
Eelco Dolstra
6cf23c3e8f
Add allow-new-privileges option
This allows builds to call setuid binaries. This was previously
possible until we started using seccomp. Turns out that seccomp by
default disallows processes from acquiring new privileges. Generally,
any use of setuid binaries (except those created by the builder
itself) is by definition impure, but some people were relying on this
ability for certain tests.

Example:

  $ nix build '(with import <nixpkgs> {}; runCommand "foo" {} "/run/wrappers/bin/ping -c 1 8.8.8.8; exit 1")' --no-allow-new-privileges
  builder for ‘/nix/store/j0nd8kv85hd6r4kxgnwzvr0k65ykf6fv-foo.drv’ failed with exit code 1; last 2 log lines:
    cannot raise the capability into the Ambient set
    : Operation not permitted

  $ nix build '(with import <nixpkgs> {}; runCommand "foo" {} "/run/wrappers/bin/ping -c 1 8.8.8.8; exit 1")' --allow-new-privileges
  builder for ‘/nix/store/j0nd8kv85hd6r4kxgnwzvr0k65ykf6fv-foo.drv’ failed with exit code 1; last 6 log lines:
    PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
    64 bytes from 8.8.8.8: icmp_seq=1 ttl=46 time=15.2 ms

Fixes #1429.
2017-07-04 15:48:25 +02:00
Eelco Dolstra
ad8b96f1f2
Fix handling of expression installables with a / in them 2017-07-04 15:38:23 +02:00
Eelco Dolstra
c0015e87af
Support base-64 hashes
Also simplify the Hash API.

Fixes #1437.
2017-07-04 15:07:41 +02:00
Eelco Dolstra
0a5a867758
nix-shell: Respect --dry-run
Fixes #824.
2017-07-03 11:54:30 +02:00
Eelco Dolstra
fcca702a96
Replace a few bool flags with enums
Functions like copyClosure() had 3 bool arguments, which creates a
severe risk of mixing up arguments.

Also, implement copyClosure() using copyPaths().
2017-07-03 11:38:08 +02:00
Eelco Dolstra
90da34e421
processGraph(): Call getEdges in parallel 2017-07-03 11:38:08 +02:00
Niklas Hambüchen
b591536e93 Fix potential crash/wrong result two hashes of unequal length are compared 2017-06-24 02:17:45 +02:00
David McFarland
596b0e0a04 Call SetDllDirectory("") after sqlite3 init on cygwin
Cygwin sqlite3 is patched to call SetDllDirectory("/usr/bin") on init, which
affects the current process and is inherited by child processes.  It causes
DLLs to be loaded from /usr/bin/ before $PATH, which breaks all sorts of
things.  A typical failures would be header/lib version mismatches (e.g.
openssl when running checkPhase on openssh).  We'll just set it back to the
default value.

Note that this is a problem with the cygwin version of sqlite3 (currently
3.18.0).  nixpkgs doesn't have the problematic patch.
2017-06-20 10:59:13 -03:00
Eelco Dolstra
c7346a275c
Restore thunks on any exception
There's no reason to restrict this to Error exceptions. This shouldn't
matter to #1407 since the repl doesn't catch non-Error exceptions
anyway, but you never know...
2017-06-20 12:13:17 +02:00
Eelco Dolstra
a1355917ec
Disable use of virtual hosting in aws-sdk-cpp
Recently aws-sdk-cpp quietly switched to using S3 virtual host URIs
(https://github.com/aws/aws-sdk-cpp/commit/69d9c53882), i.e. it sends
requests to http://<bucket>.<region>.s3.amazonaws.com rather than
http://<region>.s3.amazonaws.com/<bucket>. However this interacts
badly with curl connection reuse. For example, if we do the following:

1) Check whether a bucket exists using GetBucketLocation.
2) If it doesn't, create it using CreateBucket.
3) Do operations on the bucket.

then 3) will fail for a minute or so with a NoSuchBucket exception,
presumably because the server being hit is a fallback for cases when
buckets don't exist.

Disabling the use of virtual hosts ensures that 3) succeeds
immediately. (I don't know what S3's consistency guarantees are for
bucket creation, but in practice buckets appear to be available
immediately.)
2017-06-19 18:51:33 +02:00
Eelco Dolstra
82a0d614cf
Support creating S3 caches in other regions than us-east-1 2017-06-19 18:51:33 +02:00
Eelco Dolstra
b33621d425
Handle S3Errors::RESOURCE_NOT_FOUND from aws-sdk-cpp
This is returned by recent versions. Also handle NO_SUCH_KEY even
though the library doesn't actually return that at the moment.
2017-06-19 18:51:32 +02:00
Eelco Dolstra
1c969611ba
Suppress "will retry in N ms" for non-retriable errors
Newer versions of aws-sdk-cpp call CalculateDelayBeforeNextRetry()
even for non-retriable errors (like NoSuchKey) whih causes log spam in
hydra-queue-runner.
2017-06-19 18:51:32 +02:00
Eelco Dolstra
00aa7c6705
Show aws-sdk-cpp log messages 2017-06-19 18:51:32 +02:00
Eelco Dolstra
1888f7889b
macOS: Ugly hack to make the tests succeed
Sandboxes cannot be nested, so if Nix's build runs inside a sandbox,
it cannot use a sandbox itself. I don't see a clean way to detect
whether we're in a sandbox, so use a test-specific hack.

https://github.com/NixOS/nix/issues/1413
2017-06-19 14:26:05 +02:00
Eelco Dolstra
b5bdfdef73
macOS: Remove flags
In particular, UF_IMMUTABLE (uchg) needs to be cleared to allow the
path to be garbage-collected or optimised.

See https://github.com/NixOS/nixpkgs/issues/25819.
+       the file from being garbage-collected.
2017-06-19 14:19:21 +02:00
Eelco Dolstra
38b7d55af1
Remove redundant debug line 2017-06-14 13:45:38 +02:00
Eelco Dolstra
88b291ffc4
canonicalisePathMetaData(): Ignore security.selinux attribute
Untested, hopefully fixes #1406.
2017-06-14 11:41:03 +02:00
Eelco Dolstra
177f3996e2
Suppress spurious "killing process N: Operation not permitted" on macOS 2017-06-12 18:34:48 +02:00
Eelco Dolstra
25230a17a9
On macOS, don't use /var/folders for TMPDIR
This broke "nix-store --serve".
2017-06-12 17:43:19 +02:00
Eelco Dolstra
847f19a5f7
Provide a builtin default for $NIX_SSL_CERT_FILE
This is mostly to ensure that when Nix is started on macOS via a
launchd service or sshd (for a remote build), it gets a certificate
bundle.
2017-06-12 16:44:43 +02:00
Eelco Dolstra
7f5b750b40
Don't run pre-build-hook if we don't have a derivation
This fixes a build failure on OS X when using Hydra or Nix 1.12's
build-remote (since they don't copy the derivation to the build
machine).
2017-06-12 16:07:34 +02:00