Commit graph

5639 commits

Author SHA1 Message Date
Eelco Dolstra
fda7b95cb0
Fix a hang in ThreadPool
The worker threads could exit prematurely if they finished processing
all items while the main thread was still adding items. In particular,
this caused hanging nix-store --serve processes in the build farm.

Also, process items from the main thread.
2017-10-09 15:41:09 +02:00
Eelco Dolstra
838509d1a0
Whitespace 2017-10-09 15:41:09 +02:00
Eelco Dolstra
1893f78f8e Merge pull request #1594 from dtzWill/fix/busybox-builtins
release-common: Fix busybox builtins (busybox >= 1.27)
2017-10-09 12:09:33 +02:00
Will Dietz
2f1a1c5a49 release-common: Fix busybox builtins (busybox >= 1.27)
See https://github.com/NixOS/nixpkgs/pull/28261
2017-10-07 07:43:55 -05:00
Eelco Dolstra
b2e00fd5b7 Merge pull request #1593 from Mic92/bashism
fixing bashisms in test code
2017-10-06 14:58:13 +02:00
Jörg Thalheim
e94fc238cf fixing bashisms in test code
This fixed the build on ubuntu/debian, where dash is the sh.
2017-10-06 06:12:33 -05:00
Eelco Dolstra
c6676ea253
Merge branch 'elegant-tests' of https://github.com/copumpkin/nix 2017-10-03 11:16:34 +02:00
Eelco Dolstra
fce24b7d6c Merge pull request #1587 from copumpkin/fix-tests
Fix tests
2017-10-03 11:06:03 +02:00
Dan Peebles
60ecbd7934 More elegant test output
I got sick of trying to find the failures in the sea of debug output, so
we now:
- Hide test output unless it fails
- Sprinkle in some simple color
- Pad results for a more tabular look

If Nix is getting a more friendly user interface, we might as well get
a friendlier developer interface, right? :)
2017-10-03 00:59:32 -04:00
Dan Peebles
3139cad9cd Fix tests
`nix copy` no longer accepts a `--recursive` argument
2017-10-02 23:45:27 -04:00
Eelco Dolstra
f3e0d46821
nix run: Restore signals
Otherwise Ctrl-C doesn't work.
2017-09-28 17:58:59 +02:00
Eelco Dolstra
5421ad243e
nix copy: make recursive by default 2017-09-27 18:28:54 +02:00
Eelco Dolstra
b8875213dc
Tweak message 2017-09-27 13:21:25 +02:00
Eelco Dolstra
bfef3ec855 nix build: Fix realising a store path 2017-09-27 13:14:45 +02:00
Eelco Dolstra
d1c6e0fe8c
nix why-depends: Write to stdout 2017-09-25 16:59:16 +02:00
Eelco Dolstra
e8d6ee7c1b
Add "nix show-derivation"
This debug command prints a store derivation in JSON format. For
example:

  $ nix show-derivation nixpkgs.hello
  {
    "/nix/store/ayjwpwwiyy04nh9z71rsdgd3q7bra7ch-hello-2.10.drv": {
      "outputs": {
        "out": {
          "path": "/nix/store/w5w4v29ql0qwqhczkdxs94ix2lh7ibgs-hello-2.10"
        }
      },
      "inputSrcs": [
        "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
      ],
      "inputDrvs": {
        "/nix/store/13839aqdf6x4k3b785rw5f2l7857l6y3-bash-4.4-p12.drv": [
          "out"
        ],
        "/nix/store/vgdx7fdc7d4iirmnwj2py1nrvr5qwzj7-hello-2.10.tar.gz.drv": [
          "out"
        ],
        "/nix/store/x3kkd0vsqfflbvwf1055l9mr39bg0ms0-stdenv.drv": [
          "out"
        ]
      },
      "platform": "x86_64-linux",
      "builder": "/nix/store/qp5fw57d38bd1n07ss4zxh88zg67c3vg-bash-4.4-p12/bin/bash",
      "args": [
        "-e",
        "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
      ],
      "env": {
        "buildInputs": "",
        "builder": "/nix/store/qp5fw57d38bd1n07ss4zxh88zg67c3vg-bash-4.4-p12/bin/bash",
        "configureFlags": "",
        "doCheck": "1",
        "name": "hello-2.10",
        "nativeBuildInputs": "",
        "out": "/nix/store/w5w4v29ql0qwqhczkdxs94ix2lh7ibgs-hello-2.10",
        "propagatedBuildInputs": "",
        "propagatedNativeBuildInputs": "",
        "src": "/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz",
        "stdenv": "/nix/store/6zngq1rdh0ans9qyckqimqibgnlvlfrm-stdenv",
        "system": "x86_64-linux"
      }
    }
  }

This removes the need for pp-aterm.
2017-09-25 13:43:35 +02:00
Eelco Dolstra
e350671737
Doh 2017-09-25 13:25:55 +02:00
Eelco Dolstra
84f112b1c8
nix-shell: Ignore readFile() errors
Fixes #1563.
2017-09-18 13:28:00 +02:00
Eelco Dolstra
aca4f7dff0
Don't remove Content-Encoding in fetchurl / nix-prefetch-url
Fixes #1568.
2017-09-18 11:07:28 +02:00
Eelco Dolstra
5324bb9399
Update release notes 2017-09-18 11:07:17 +02:00
Eelco Dolstra
346aeee1cb
Remove Debian 8 and Ubuntu 14.10
These have a GCC (4.9) that is too old.

https://hydra.nixos.org/eval/1391740
2017-09-14 18:56:33 +02:00
Eelco Dolstra
89dc62c174
RemoteStore: Add option to drop old connections from the pool
This is a hack to make hydra-queue-runner free its temproots
periodically, thereby ensuring that garbage collection of the
corresponding paths is not blocked until the queue runner is
restarted.

It would be better if temproots could be released earlier than at
process exit. I started working on a RAII object returned by functions
like addToStore() that releases temproots. However, this would be a
pretty massive change so I gave up on it for now.
2017-09-14 18:10:38 +02:00
Eelco Dolstra
308ecf6361
Show the pid of temporary roots 2017-09-14 15:02:52 +02:00
Eelco Dolstra
da1e4fdfb5
nix-store -q --roots / --gc --print-roots: Print temporary / in-memory roots
For example,

  $ nix-store -q --roots /nix/store/7phd2sav7068nivgvmj2vpm3v47fd27l-patchelf-0.8pre845_0315148
  {temp:1}

denotes that the path is only being kept alive by a temporary root
(i.e. /nix/var/nix/temproots/). Similarly,

  $ nix-store --gc --print-roots
  ...
  {memory:9} -> /nix/store/094gpjn9f15ip17wzxhma4r51nvsj17p-curl-7.53.1

shows that curl is being used by some process.
2017-09-14 14:38:36 +02:00
Eelco Dolstra
970366266b
Add "nix add-to-store" command 2017-09-14 13:29:33 +02:00
Eelco Dolstra
1c58ad2ffa
nix why-depends: Fix showing self-references 2017-09-14 11:17:32 +02:00
Eelco Dolstra
359ede1d72 Merge pull request #1562 from peti/master
docker: update to Nix 1.11.14
2017-09-11 16:44:16 +02:00
Eelco Dolstra
fc0ded3408
nix why-depends: Add option to show all edges causing a dependency
For example, without --all:

  $ nix why-depends nixpkgs.nixUnstable nixpkgs.libssh2
  /nix/store/s9n5gvj2l49b4n19nz6xl832654nf7n7-nix-1.12pre5511_c94f3d55
  └───lib/libnixstore.so: …/lib:/nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0/lib…
      => /nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0
      └───lib/libcurl.la: …ib -L/nix/store/4mbayl1y5hpjbjzkx8ndyhkv98kqw1wi-libssh2-1.8.0/l…
          => /nix/store/4mbayl1y5hpjbjzkx8ndyhkv98kqw1wi-libssh2-1.8.0

but with --all:

  $ nix why-depends -a nixpkgs.nixUnstable nixpkgs.libssh2
  /nix/store/s9n5gvj2l49b4n19nz6xl832654nf7n7-nix-1.12pre5511_c94f3d55
  ├───lib/libnixstore.so: …/lib:/nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0/lib…
  │   => /nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0
  │   └───lib/libcurl.la: …ib -L/nix/store/4mbayl1y5hpjbjzkx8ndyhkv98kqw1wi-libssh2-1.8.0/l…
  │       lib/libcurl.so.4.4.0: …/lib:/nix/store/4mbayl1y5hpjbjzkx8ndyhkv98kqw1wi-libssh2-1.8.0/l…
  │       => /nix/store/4mbayl1y5hpjbjzkx8ndyhkv98kqw1wi-libssh2-1.8.0
  └───lib/libnixstore.so: …/lib:/nix/store/bx2i9vi76lps6w9rr73fxf6my31s4dg5-aws-sdk-cpp-1.0…
      => /nix/store/bx2i9vi76lps6w9rr73fxf6my31s4dg5-aws-sdk-cpp-1.0.153
      └───lib/libaws-cpp-sdk-core.so: …e.so./nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0/lib…
          lib/libaws-cpp-sdk-s3.so: …/lib:/nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0/lib…
          => /nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0
2017-09-11 16:21:27 +02:00
Peter Simons
1511f01351 docker: update to Nix 1.11.14
- Use the latest Nix version 1.11.14.

- Attempts to download the Nix installation tarball from http://nixos.org
  redirect to https these days, which wget doesn't support unless OpenSSL is
  available.

- Use addgroup and adduser commands to create the Nix build users.

- Link the Nix profile script into /etc/profile.d, where it's run
  automatically.

- Dropped installation of bash and tar. Neither tool is essential for running
  Nix.

Use the command "docker build -t nix . && docker run -it --rm nix sh -"
to build and run the Nix docker container.
2017-09-11 14:26:14 +02:00
Eelco Dolstra
d41c5eb13f Add command "nix why-depends"
This command shows why a package has another package in its runtime
closure. For example, to see why VLC has libdrm.dev in its closure:

  $ nix why-depends nixpkgs.vlc nixpkgs.libdrm.dev
  /nix/store/g901z9pcj0n5yy5n6ykxk3qm4ina1d6z-vlc-2.2.5.1:
    lib/libvlccore.so.8.0.0: …nfig:/nix/store/405lmx6jl8lp0ad1vrr6j498chrqhz8g-libdrm-2.4.75-d…

  /nix/store/s3nm7kd8hlcg0facn2q1ff2n7wrwdi2l-mesa-noglu-17.0.7-dev:
    nix-support/propagated-native-build-inputs: …-dev /nix/store/405lmx6jl8lp0ad1vrr6j498chrqhz8g-libdrm-2.4.75-d…

Thus, VLC's lib/libvlccore.so.8.0.0 as well as mesa-noglu's
nix-support/propagated-native-build-inputs cause the dependency.
2017-09-10 22:40:33 +02:00
Eelco Dolstra
8af704eef1 Typo 2017-09-10 22:40:33 +02:00
Eelco Dolstra
ad228d84e5 nix build: Only download the requested derivation outputs
Also some refactoring.
2017-09-10 22:40:33 +02:00
Eelco Dolstra
3ed8290e53
LegacySSHStore: Include signatures etc. 2017-09-08 16:55:27 +02:00
Eelco Dolstra
e02edb1483
nix copy: Add --substitute flag 2017-09-08 15:32:07 +02:00
Eelco Dolstra
b7376edf06
ThreadPool: On exception, interrupt the other worker threads 2017-09-08 15:31:24 +02:00
Eelco Dolstra
6a888ec29a
copyStorePath(): Fill in missing narHash regardless of checkSigs
I don't remember what the reasoning was here, but security is provided
by the signatures, not by whether the hash is provided by the other
store.
2017-09-08 14:48:08 +02:00
Eelco Dolstra
3cf1705583
nix copy: Don't open the --from store twice 2017-09-08 14:46:55 +02:00
Eelco Dolstra
8f6b347abd
ThreadPool: Improve exception handling
In particular, process() won't return as long as there are active
items. This prevents work item lambdas from referring to stack frames
that no longer exist.
2017-09-08 14:42:09 +02:00
Eelco Dolstra
a2740c9ca2 nix copy: Add examples 2017-09-08 11:33:46 +02:00
Eelco Dolstra
88e6bb76de Communicate the structured log FD to builders
Since we may use a dedicated file descriptor in the future, this
allows us to change it. So builders can do

  if [[ -n $NIX_LOG_FD ]]; then
    echo "@nix { message... }" >&$NIX_LOG_FD
  fi
2017-09-08 11:27:10 +02:00
Eelco Dolstra
24e23a1a73 nix search: Add examples 2017-09-07 20:42:11 +02:00
Eelco Dolstra
4967f0509a nix eval: Add examples 2017-09-07 20:37:46 +02:00
Eelco Dolstra
2ebeffcfd4 nix log: Add examples 2017-09-07 20:18:29 +02:00
Eelco Dolstra
17afc42895 nix build: Add examples 2017-09-07 20:14:04 +02:00
Eelco Dolstra
2e9b7c4cb2 nix run: Add some examples 2017-09-07 20:09:04 +02:00
Eelco Dolstra
dff440aab3
nix build: Add --out-link and --no-link options 2017-09-06 16:20:34 +02:00
Eelco Dolstra
df4342bc17
nix build: Create result symlinks 2017-09-06 16:03:22 +02:00
Eelco Dolstra
1277aab219
Fix abort when the GC thread gets an exception 2017-09-06 11:37:58 +02:00
Eelco Dolstra
0b606aad46
Add automatic garbage collection
Nix can now automatically run the garbage collector during builds or
while adding paths to the store. The option "min-free = <bytes>"
specifies that Nix should run the garbage collector whenever free
space in the Nix store drops below <bytes>. It will then delete
garbage until "max-free" bytes are available.

Garbage collection during builds is asynchronous; running builds are
not paused and new builds are not blocked. However, there also is a
synchronous GC run prior to the first build/substitution.

Currently, no old GC roots are deleted (as in "nix-collect-garbage
-d").
2017-09-05 20:43:42 +02:00
Eelco Dolstra
b932ea58ec
GC: Don't delete own temproots file
Since file locks are per-process rather than per-file-descriptor, the
garbage collector would always acquire a lock on its own temproots
file and conclude that it's stale.
2017-09-05 20:39:57 +02:00