Commit graph

3008 commits

Author SHA1 Message Date
Eelco Dolstra
1c718f80d3
Verify content-addressability assertions at registration time 2017-02-24 16:38:14 +01:00
Eelco Dolstra
668fef2e4f
nix-shell: Overwrite environment variables
Need to remember that std::map::insert() and emplace() don't overwrite
existing entries...

This fixes a regression relative to 1.11 that in particular triggers
in nested nix-shells.

Before:

  $ nativeBuildInputs=/foo nix-shell -p hello --run 'hello'
  build input /foo does not exist

After:

  $ nativeBuildInputs=/foo nix-shell -p hello --run 'hello'
  Hello, world!
2017-02-24 13:31:46 +01:00
Dan Peebles
e7cb2847ab Explicitly model all settings and fail on unrecognized ones
Previously, the Settings class allowed other code to query for string
properties, which led to a proliferation of code all over the place making
up new options without any sort of central registry of valid options. This
commit pulls all those options back into the central Settings class and
removes the public get() methods, to discourage future abuses like that.

Furthermore, because we know the full set of options ahead of time, we
now fail loudly if someone enters an unrecognized option, thus preventing
subtle typos. With some template fun, we could probably also dump the full
set of options (with documentation, defaults, etc.) to the command line,
but I'm not doing that yet here.
2017-02-22 20:19:25 -05:00
Eelco Dolstra
f023f64f40
RemoteStore::addToStore(): Pass content-addressability assertion
... and use this in Downloader::downloadCached(). This fixes

  $ nix-build https://nixos.org/channels/nixos-16.09-small/nixexprs.tar.xz -A hello
  error: cannot import path ‘/nix/store/csfbp1s60dkgmk9f8g0zk0mwb7hzgabd-nixexprs.tar.xz’ because it lacks a valid signature
2017-02-22 16:58:00 +01:00
Eelco Dolstra
8b1b5f9a12
Handle CURLE_RECV_ERROR as a transient error
This fixes

  unable to download ‘https://cache.nixos.org/nar/077h8ji74y9b0qx7rjk71xd80vjqp6q5gy137r553jlvdlxdcdlk.nar.xz’: HTTP error 200 (curl error: Failure when receiving data from the peer)
2017-02-21 16:04:47 +01:00
Eelco Dolstra
99bbddedb1
Fix building without S3 support
http://hydra.nixos.org/build/49031196/nixlog/2/raw
2017-02-21 13:15:07 +01:00
Eelco Dolstra
df66d346df
Log AWS retries 2017-02-21 11:50:31 +01:00
Eelco Dolstra
302386f775
Support netrc in <nix/fetchurl.nix>
This allows <nix/fetchurl.nix> to fetch private Git/Mercurial
repositories, e.g.

  import <nix/fetchurl.nix> {
    url = https://edolstra@bitbucket.org/edolstra/my-private-repo/get/80a14018daed.tar.bz2;
    sha256 = "1mgqzn7biqkq3hf2697b0jc4wabkqhmzq2srdymjfa6sb9zb6qs7";
  }

where /etc/nix/netrc contains:

  machine bitbucket.org
  login edolstra
  password blabla...

This works even when sandboxing is enabled.

To do: add unpacking support (i.e. fetchzip functionality).
2017-02-16 15:51:50 +01:00
Eelco Dolstra
cde4b60919
Move netrcFile to Settings
Also get rid of Settings::processEnvironment(), it appears to be
useless.
2017-02-16 14:50:41 +01:00
Eelco Dolstra
40f0e3b366
Include "curl" in the User-Agent header
Some sites (e.g. BitBucket) give a helpful 401 error when trying to
download a private archive if the User-Agent contains "curl", but give
a redirect to a login page otherwise (so for instance
"nix-prefetch-url" will succeed but produce useless output).
2017-02-16 13:55:43 +01:00
Eelco Dolstra
d1139ff36b Merge pull request #1215 from k0001/netrc-1
Add netrc-file support
2017-02-16 12:45:44 +01:00
Eelco Dolstra
9ff9c3f2f8
Add support for s3:// URIs
This adds support for s3:// URIs in all places where Nix allows URIs,
e.g. in builtins.fetchurl, builtins.fetchTarball, <nix/fetchurl.nix>
and NIX_PATH. It allows fetching resources from private S3 buckets,
using credentials obtained from the standard places (i.e. AWS_*
environment variables, ~/.aws/credentials and the EC2 metadata
server). This may not be super-useful in general, but since we already
depend on aws-sdk-cpp, it's a cheap feature to add.
2017-02-14 14:20:00 +01:00
Eelco Dolstra
62ff5ad424 Merge pull request #1224 from dezgeg/configh
Unbreak 32-bit builds by always implicitly including config.h
2017-02-13 14:44:58 +01:00
Tuomas Tynkkynen
649a81bcd6 nix-daemon: Don't splice with len=SIZE_MAX
Currently, 'nix-daemon --stdio' is always failing for me, due to the
splice call always failing with (on a 32-bit host):

splice(0, NULL, 3, NULL, 4294967295, SPLICE_F_MOVE) = -1 EINVAL (Invalid argument)

With a bit of ftracing (and luck) the problem seems to be that splice()
always fails with EINVAL if the len cast as ssize_t is negative:
http://lxr.free-electrons.com/source/fs/read_write.c?v=4.4#L384

So use SSIZE_MAX instead of SIZE_MAX.
2017-02-13 15:14:44 +02:00
Renzo Carbonara
e2257d4eeb Documentation. 2017-02-09 18:16:09 +01:00
Tuomas Tynkkynen
2cd468874f Include config.h implicitly with '-include config.h' in CFLAGS
Because config.h can #define things like _FILE_OFFSET_BITS=64 and not
every compilation unit includes config.h, we currently compile half of
Nix with _FILE_OFFSET_BITS=64 and other half with _FILE_OFFSET_BITS
unset. This causes major havoc with the Settings class on e.g. 32-bit ARM,
where different compilation units disagree with the struct layout.

E.g.:

diff --git a/src/libstore/globals.cc b/src/libstore/globals.cc
@@ -166,6 +166,8 @@ void Settings::update()
     _get(useSubstitutes, "build-use-substitutes");
+    fprintf(stderr, "at Settings::update(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes);
     _get(buildUsersGroup, "build-users-group");
diff --git a/src/libstore/remote-store.cc b/src/libstore/remote-store.cc
+++ b/src/libstore/remote-store.cc
@@ -138,6 +138,8 @@ void RemoteStore::initConnection(Connection & conn)
 void RemoteStore::setOptions(Connection & conn)
 {
+    fprintf(stderr, "at RemoteStore::setOptions(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes);
     conn.to << wopSetOptions

Gave me:

at Settings::update(): &useSubstitutes = 0xb6e5c5cb
at RemoteStore::setOptions(): &useSubstitutes = 0xb6e5c5c7

That was not a fun one to debug!
2017-02-08 21:51:02 +02:00
Shea Levy
418a837897 Remove perl dependency.
Fixes #341
2017-02-07 15:56:32 -05:00
Eelco Dolstra
4724903c78
nix-copy-closure: Use computeFSClosure() and LegacySSHStore 2017-02-07 20:55:47 +01:00
Eelco Dolstra
6f4682ad36
Merge branch 'nix-copy-closure-c++' of https://github.com/shlevy/nix 2017-02-07 20:47:45 +01:00
Eelco Dolstra
caa5793b4a
Add a LegacySSHStore that uses nix-store --serve
This is useful for nix-copy-closure.
2017-02-07 19:29:21 +01:00
Eelco Dolstra
f38224e924
copyStorePath(): Don't require signatures for "trusted" stores
For example, SSH stores could be trusted.
2017-02-07 19:29:21 +01:00
Eelco Dolstra
fa07558a06
Provide default implementations for a couple of Store methods 2017-02-07 19:29:21 +01:00
Eelco Dolstra
ddb5577f2e
Move SavingSourceAdapter to serialise.hh 2017-02-07 19:29:20 +01:00
Eelco Dolstra
ce4d8e3ef8
Remove unnecessary call to topoSortPaths()
exportPaths() already does this.
2017-02-07 19:20:41 +01:00
Eelco Dolstra
7a58ad0ef5
SSHStore: uri -> host 2017-02-07 19:20:15 +01:00
Eelco Dolstra
612aeb2df5
Cleanup 2017-02-07 19:16:25 +01:00
Eelco Dolstra
c54814b175
Remove download-via-ssh
Replaced by SSHStore.
2017-02-07 18:54:33 +01:00
Renzo Carbonara
e6e74f987f Add netrc-file support 2017-02-02 13:24:20 +01:00
Eelco Dolstra
1351b0df87
exportReferencesGraph: Only export in JSON format when in structured mode
This prevents breaking compatibility with builders that read
"closure.*", since they would accidentally pick up the new JSON files.
2017-02-02 12:20:28 +01:00
Eelco Dolstra
7a65b2470e
Restore default signal handling in child processes
In particular, this fixes Ctrl-C in nix-shell sessions.
2017-02-01 13:00:21 +01:00
Eelco Dolstra
cb1951e746
Periodically purge binary-cache.sqlite 2017-01-27 15:19:33 +01:00
Eelco Dolstra
211bc7f0e6
Implement TTL for binary cache lookups 2017-01-27 13:17:08 +01:00
Eelco Dolstra
f57a38b109
Remove unused NARExistence table 2017-01-27 12:57:49 +01:00
Eelco Dolstra
c2b0d8749f
exportReferencesGraph: Export more complete info in JSON format
This writes info about every path in the closure in the same format as
‘nix path-info --json’. Thus it also includes NAR hashes and sizes.

Example:

  [
    {
      "path": "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10",
      "narHash": "sha256:0ckdc4z20kkmpqdilx0wl6cricxv90lh85xpv2qljppcmz6vzcxl",
      "narSize": 197648,
      "references": [
        "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10",
        "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24"
      ],
      "closureSize": 20939776
    },
    {
      "path": "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24",
      "narHash": "sha256:1nfn3m3p98y1c0kd0brp80dn9n5mycwgrk183j17rajya0h7gax3",
      "narSize": 20742128,
      "references": [
        "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24"
      ],
      "closureSize": 20742128
    }
  ]

Fixes #1134.
2017-01-26 20:41:08 +01:00
Eelco Dolstra
6de33a9c67
Add support for passing structured data to builders
Previously, all derivation attributes had to be coerced into strings
so that they could be passed via the environment. This is lossy
(e.g. lists get flattened, necessitating configureFlags
vs. configureFlagsArray, of which the latter cannot be specified as an
attribute), doesn't support attribute sets at all, and has size
limitations (necessitating hacks like passAsFile).

This patch adds a new mode for passing attributes to builders, namely
encoded as a JSON file ".attrs.json" in the current directory of the
builder. This mode is activated via the special attribute

  __structuredAttrs = true;

(The idea is that one day we can set this in stdenv.mkDerivation.)

For example,

  stdenv.mkDerivation {
    __structuredAttrs = true;
    name = "foo";
    buildInputs = [ pkgs.hello pkgs.cowsay ];
    doCheck = true;
    hardening.format = false;
  }

results in a ".attrs.json" file containing (sans the indentation):

  {
    "buildInputs": [],
    "builder": "/nix/store/ygl61ycpr2vjqrx775l1r2mw1g2rb754-bash-4.3-p48/bin/bash",
    "configureFlags": [
      "--with-foo",
      "--with-bar=1 2"
    ],
    "doCheck": true,
    "hardening": {
      "format": false
    },
    "name": "foo",
    "nativeBuildInputs": [
      "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10",
      "/nix/store/4jnvjin0r6wp6cv1hdm5jbkx3vinlcvk-cowsay-3.03"
    ],
    "propagatedBuildInputs": [],
    "propagatedNativeBuildInputs": [],
    "stdenv": "/nix/store/f3hw3p8armnzy6xhd4h8s7anfjrs15n2-stdenv",
    "system": "x86_64-linux"
  }

"passAsFile" is ignored in this mode because it's not needed - large
strings are included directly in the JSON representation.

It is up to the builder to do something with the JSON
representation. For example, in bash-based builders, lists/attrsets of
string values could be mapped to bash (associative) arrays.
2017-01-26 20:40:33 +01:00
Eelco Dolstra
54801ed6ad
Bindings: Add a method for iterating in lexicographically sorted order 2017-01-26 20:40:33 +01:00
Eelco Dolstra
b1f001538e
Fix assertion failure when a path is locked
Fixes:

  nix-store: src/libstore/build.cc:3649: void nix::Worker::run(const Goals&): Assertion `!awake.empty()' failed.
2017-01-26 20:40:33 +01:00
Eelco Dolstra
83ae6503e8
Fix interrupt handling 2017-01-26 20:40:33 +01:00
Eelco Dolstra
951357e5fb
UserLock: Fix multi-threaded access to a global variable 2017-01-26 20:40:33 +01:00
Eelco Dolstra
a55f589720
openLockFile: Return an AutoCloseFD 2017-01-26 20:40:33 +01:00
Eelco Dolstra
c0f2f4eeef
UserLock: Make more RAII-ish 2017-01-26 20:40:33 +01:00
Eelco Dolstra
a529c740d2
Moving more code out of DerivationGoal::startBuilder() 2017-01-26 20:40:33 +01:00
Eelco Dolstra
e8c43abd9a
On HTTP errors, also show the curl error
This is a hopefully temporary measure to diagnose the intermittent
"HTTP error 200" failures.
2017-01-26 20:40:32 +01:00
Eelco Dolstra
4425a5c547
Move exportReferencesGraph into a separate method
startBuilder() is getting rather obese.
2017-01-26 20:40:32 +01:00
Eelco Dolstra
4e6a2fbc56
Revert "Propagate path context via builtins.readFile"
This reverts commit f7f0116dd7.

Issue #1174.
2017-01-24 15:31:54 +01:00
Eelco Dolstra
fa738e50bc
Revert "builtins.readFile: Put the references of the file, not those needed to realize the file, into the context"
Reverting commit 451c223dee for now
because it breaks http://hydra.nixos.org/build/46805136, not clear
why.
2017-01-24 15:29:46 +01:00
Eelco Dolstra
4b6d3c5a28
Hopefully fix build on older GCC
http://hydra.nixos.org/build/46805140
2017-01-24 15:28:50 +01:00
Eelco Dolstra
19ce732a13
Fix typo 2017-01-24 13:57:26 +01:00
Eelco Dolstra
e3bf228c92
Enable verbose curl output
Closes #1182.
2017-01-24 13:57:01 +01:00
Shea Levy
3b4a15bd48 build-remote: Use futimes instead of futimens on APPLE 2017-01-24 06:22:02 -05:00
Eelco Dolstra
e5641dfe1e
Work around a bug in clang and older versions of gcc
http://hydra.nixos.org/build/46597440

https://llvm.org/bugs/show_bug.cgi?id=28096
2017-01-24 10:57:24 +01:00
Shea Levy
bfa41eb671 nix-copy-closure: Implement in C++.
Tests fail currently because the database is not given proper hashes in the VM
2017-01-20 09:47:58 -05:00
Eelco Dolstra
8af062f372 Merge pull request #981 from shlevy/build-remote-c++
build-remote: Implement in C++
2017-01-19 18:21:55 +01:00
Eelco Dolstra
21948deed9
Kill builds when we get EOF on the log FD
This closes a long-time bug that allowed builds to hang Nix
indefinitely (regardless of timeouts) simply by doing

  exec > /dev/null 2>&1; while true; do true; done

Now, on EOF, we just send SIGKILL to the child to make sure it's
really gone.
2017-01-19 17:16:14 +01:00
Eelco Dolstra
63e10b4d28
Cleanup 2017-01-19 17:06:04 +01:00
Eelco Dolstra
2579e32c2b
Use std::unique_ptr for HookInstance 2017-01-19 17:06:04 +01:00
Eelco Dolstra
cc3b93c991
Handle SIGINT etc. via a sigwait() signal handler thread
This allows other threads to install callbacks that run in a regular,
non-signal context. In particular, we can use this to signal the
downloader thread to quit.

Closes #1183.
2017-01-17 18:21:02 +01:00
Jude Taylor
c0d55f9183 assign environ to a temp variable to ensure liveness 2017-01-16 23:52:44 +01:00
Eelco Dolstra
8079ab87a2 AutoCloseDir: Use std::unique_ptr 2017-01-16 22:39:27 +01:00
Eelco Dolstra
2b9d0a99cb AutoDeleteArray -> std::unique_ptr
Also, switch to C++14 for std::make_unique.
2017-01-16 22:24:49 +01:00
Eelco Dolstra
40dfac968a nix-build: Use showManPage() and printVersion() 2017-01-16 22:11:17 +01:00
Shea Levy
451c223dee builtins.readFile: Put the references of the file, not those needed to realize the file, into the context 2017-01-10 12:22:22 -05:00
Shea Levy
28db297862 build-remote: Don't use C++ streams to read the conf file 2017-01-10 10:36:26 -05:00
Shea Levy
d771c28613 build-remote: Use std::set for feature sets 2017-01-10 10:29:06 -05:00
Eelco Dolstra
7adb986e35 Merge pull request #1139 from Mic92/master
Simplify remouting with MS_PRIVATE in sandbox build
2017-01-09 16:46:59 +01:00
Tuomas Tynkkynen
3890de049d nix-shell: Fix 'nix-shell --command' doing nothing without TTY
Regression from a5f2750e ("Fix early removal of rc-file for nix-shell").
The removal of BASH_ENV causes nothing to be executed by bash if it
detects itself in a non-interactive context. Instead, just
use the same condition used by bash to launch bash differently.

According to bash sources, the condition (stdin and stder both
must be TTYs) is specified by POSIX so this should be pretty
safe to rely on.

Fixes #1171 on master, needs a backport to the Perl code in 1.11.
2017-01-07 19:08:28 +02:00
Eelco Dolstra
c287e797a8
nix-shell: In #! mode, pass the last argument
"i < argc - 1" should be "i < argc".
2017-01-03 11:40:51 +01:00
Tuomas Tynkkynen
018f884ffd nix-shell: Don't act interactive in shebangs
I had observed that 'bash --rcfile' would do nothing in a
non-interactive context and cause nothing to be executed if a script
using nix-shell shebangs were run in a non-interactive context.
2017-01-03 10:11:09 +02:00
Tuomas Tynkkynen
c698ec1f22 nix-shell: Fix 'nix-shell -i'
The 'args' variable here is shadowing one in the outer scope and its
contents end up unused. This causes any '#! nix-shell' lines to
effectively be ignored. The intention here was to clear the args vector,
as far as I can tell (and it seems to work).
2017-01-03 10:11:09 +02:00
Eelco Dolstra
b6b142b4b1
Provide /var/run/nscd/socket in the sandbox
Otherwise sandbox builds can fail, e.g.

  $ NIX_REMOTE=local?root=/tmp/nix nix-build '<nixpkgs>' -A hello --option build-use-substitutes false
  ...
  downloading ‘http://ftpmirror.gnu.org/bash/bash-4.3-patches/bash43-047’...
  error: unable to download ‘http://ftpmirror.gnu.org/bash/bash-4.3-patches/bash43-047’: Couldn't resolve host name (6)
2017-01-02 14:46:37 +01:00
Eelco Dolstra
d0a2db17d9
Call Aws::InitAPI
This is required now.
2016-12-22 17:39:49 +01:00
Eelco Dolstra
786ee585b8
Add comment 2016-12-22 17:39:49 +01:00
Eelco Dolstra
3a4bd320c2
Revert "Merge branch 'seccomp' of https://github.com/aszlig/nix"
This reverts commit 9f3f2e21ed, reversing
changes made to 47f587700d.
2016-12-19 11:52:57 +01:00
Eelco Dolstra
05862209de
Revert "Give root a valid home directory"
This reverts commit ec7d498b72.
2016-12-19 11:49:03 +01:00
Eelco Dolstra
ec7d498b72
Give root a valid home directory
Some programs barf if the current user has a non-writable home
directory, e.g. http://hydra.nixos.org/build/44818144.
2016-12-15 15:56:08 +01:00
Eelco Dolstra
5278bb7c16
Merge branch 'master' of github.com:NixOS/nix 2016-12-15 12:31:52 +01:00
Eelco Dolstra
9f3f2e21ed
Merge branch 'seccomp' of https://github.com/aszlig/nix 2016-12-15 12:04:45 +01:00
Shea Levy
f867f090ed Merge branch 'base32-overflow' of git://github.com/vcunat/nix 2016-12-13 09:41:02 -05:00
Eelco Dolstra
47f587700d
Probably fix a segfault in PathLocks 2016-12-09 13:26:43 +01:00
Eelco Dolstra
b30d1e7ada
Don't delete .check directories of running builds
We need to keep them around for diffoscope.
2016-12-08 21:38:58 +01:00
Eelco Dolstra
88ef77226e
Fix warning on 32-bit systems
http://hydra.nixos.org/build/44628517
2016-12-08 20:37:58 +01:00
Eelco Dolstra
e629a17cc1
Fix build
http://hydra.nixos.org/build/44628517
2016-12-08 20:36:14 +01:00
Eelco Dolstra
fe1162a805
S3BinaryCacheStore: Ensure it only builds on Linux 2016-12-08 15:35:46 +01:00
Eelco Dolstra
e6a61b8da7
Fix S3BinaryCacheStore
It failed with

   AWS error uploading ‘6gaxphsyhg66mz0a00qghf9nqf7majs2.ls.xz’: Unable to parse ExceptionName: MissingContentLength Message: You must provide the Content-Length HTTP header.

possibly because the istringstream_nocopy introduced in
0d2ebb4373 doesn't supply the seek
method that the AWS library expects. So bring back the old version,
but only for S3BinaryCacheStore.
2016-12-08 15:31:27 +01:00
Eelco Dolstra
9a313469a4
Add a hook to run diffoscope when non-determinism is detected 2016-12-07 17:57:35 +01:00
Eelco Dolstra
b07060688a
Keep track of the exact build start/stop times 2016-12-07 16:09:38 +01:00
Eelco Dolstra
dadfddfa7c
Bail out early when non-determinism is detected 2016-12-07 15:31:18 +01:00
Eelco Dolstra
cea4e2c618
Expose enforce-determinism and the result to Hydra 2016-12-07 13:43:40 +01:00
Eelco Dolstra
8bdf83f936
Add an option to make non-determinism non-fatal
That is, when build-repeat > 0, and the output of two rounds differ,
then print a warning rather than fail the build. This is primarily to
let Hydra check reproducibility of all packages.
2016-12-07 13:16:06 +01:00
Eelco Dolstra
ceeedb58d2
Use a steady clock for timeouts
Fixes #1146.
2016-12-06 21:58:04 +01:00
Eelco Dolstra
7a3e7d0e61
nix-store --serve: Suppress log output on stderr when repeating a build 2016-12-06 17:43:39 +01:00
Eelco Dolstra
6069b946ad
nix-store --serve: Support setting build-repeat
This allows Hydra to test whether builds are reproducible.
2016-12-06 17:19:22 +01:00
Eelco Dolstra
603f08506e
Tweak error message 2016-12-06 17:18:40 +01:00
Guillaume Maudoux
e4b82af387 Improve error message on trailing path slashes 2016-11-27 17:48:46 +01:00
Jörg Thalheim
855abd85d8
Simplify remouting with MS_PRIVATE in sandbox build
also fix race condition if mounts are added after mountinfo is read.
2016-11-25 00:15:39 +01:00
Eelco Dolstra
7ee43df862
nix-channel: Fix --update <CHANNELS>
This unbreaks "nixos-rebuild --upgrade".
2016-11-21 15:54:19 +01:00
Eelco Dolstra
349e988903
Typo 2016-11-17 17:55:41 +01:00
Shea Levy
00b8bce4d0 Fix binary-cache-store build 2016-11-17 11:48:10 -05:00
Shea Levy
0d2ebb4373 istringstream_nocopy: Implement in a standards-compliant way
Fixes #1135.
2016-11-17 10:10:32 -05:00
Shea Levy
3f4d3f8a1a nix-daemon: Fix splice faking on non-Linux 2016-11-17 08:10:12 -05:00
Shea Levy
8bf378e999 Update darwin build for optional sandbox paths
Fixes #1132
2016-11-17 08:06:32 -05:00
aszlig
4e1a2cd537
seccomp: Forge return values for *chown32
These syscalls are only available in 32bit architectures, but libseccomp
should handle them correctly even if we're on native architectures that
do not have these syscalls.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 17:29:36 +01:00
aszlig
ed64976cec
seccomp: Forge return codes for POSIX ACL syscalls
Commands such as "cp -p" also use fsetxattr() in addition to fchown(),
so we need to make sure these syscalls always return successful as well
in order to avoid nasty "Invalid value" errors.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 17:29:14 +01:00
aszlig
b90a435332
libstore/build: Forge chown() to return success
What we basically want is a seccomp mode 2 BPF program like this but for
every architecture:

  BPF_STMT(BPF_LD+BPF_W+BPF_ABS, offsetof(struct seccomp_data, nr)),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_chown, 4, 0),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_fchown, 3, 0),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_fchownat, 2, 0),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_lchown, 1, 0),
  BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_ALLOW),
  BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_ERRNO)

However, on 32 bit architectures we do have chown32, lchown32 and
fchown32, so we'd need to add all the architecture blurb which
libseccomp handles for us.

So we only need to make sure that we add the 32bit seccomp arch while
we're on x86_64 and otherwise we just stay at the native architecture
which was set during seccomp_init(), which more or less replicates
setting 32bit personality during runChild().

The FORCE_SUCCESS() macro here could be a bit less ugly but I think
repeating the seccomp_rule_add() all over the place is way uglier.

Another way would have been to create a vector of syscalls to iterate
over, but that would make error messages uglier because we can either
only print the (libseccomp-internal) syscall number or use
seccomp_syscall_resolve_num_arch() to get the name or even make the
vector a pair number/name, essentially duplicating everything again.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 16:48:30 +01:00
aszlig
1c52e344c4
Add build dependency for libseccomp
We're going to use libseccomp instead of creating the raw BPF program,
because we have different syscall numbers on different architectures.

Although our initial seccomp rules will be quite small it really doesn't
make sense to generate the raw BPF program because we need to duplicate
it and/or make branches on every single architecture we want to suuport.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 16:48:26 +01:00
aszlig
e8838713df
Run builds as root in user namespace again
This reverts commit ff0c0b645c.

We're going to use seccomp to allow "cp -p" and force chown-related
syscalls to always return 0.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 16:48:08 +01:00
Eelco Dolstra
4be4f6de56
S3BinaryCacheStore:: Eliminate a string copy while uploading
This cuts hydra-queue-runner's peak memory usage by about a third.
2016-11-16 16:21:30 +01:00
Eelco Dolstra
10ae8fabf1 buildPaths(): Handle ecIncompleteClosure
buildPaths() on a non-derivation would incorrectly not throw an error
if the path didn't have a substitute.
2016-11-14 15:00:17 +01:00
Eelco Dolstra
b77fb8acb5 Don't rely on %m 2016-11-14 13:37:16 +01:00
Ludovic Courtès
ccb1022022 daemon: Do not error out when deduplication fails due to ENOSPC.
This solves a problem whereby if /gnu/store/.links had enough entries,
ext4's directory index would be full, leading to link(2) returning
ENOSPC.

* nix/libstore/optimise-store.cc (LocalStore::optimisePath_): Upon
ENOSPC from link(2), print a message and return instead of throwing a
'SysError'.
2016-11-14 13:35:03 +01:00
Guillaume Maudoux
a5e761dddb Fix comments parsing
Fixed the parsing of multiline strings ending with an even number of
stars, like /** this **/.
Added test cases for comments.
2016-11-13 17:20:34 +01:00
Eelco Dolstra
dd77f7d593 Store::computeFSClosure(): Support a set of paths
This way, callers can exploits the parallelism of computeFSClosure()
when they have multiple paths that they need the (combined) closure of.
2016-11-10 17:45:04 +01:00
Shea Levy
bff3ad767e build-remote: replace strtoull with stoull to take advantage of C++ error handling 2016-11-10 11:09:15 -05:00
Shea Levy
167d12b02c build-remote: Implement in C++ 2016-11-10 11:09:15 -05:00
Eelco Dolstra
e4655b166c copyClosure() / copyStorePath(): Expose dontCheckSigs
Needed by Hydra.
2016-11-09 19:08:20 +01:00
Eelco Dolstra
4b8f1b0ec0 Merge branch 'ssh-store' of https://github.com/shlevy/nix 2016-11-09 18:57:22 +01:00
Eelco Dolstra
21c55ab3b5 Implement backwards-compatible RemoteStore::addToStore()
The SSHStore PR adds this functionality to the daemon, but we have to
handle the case where the Nix daemon is 1.11.

Also, don't require signatures for trusted users. This restores 1.11
behaviour.

Fixes https://github.com/NixOS/hydra/issues/398.
2016-11-09 18:45:06 +01:00
Eelco Dolstra
a83b10f84c Typo 2016-11-08 20:19:02 +01:00
Eelco Dolstra
b3ba762dbf Add missing #include 2016-11-07 14:35:47 +01:00
Vladimír Čunát
818aad3ec4
Detect and disallow base32 hash overflow
Example (before this commit):
$ nix-hash --type sha256 --to-base16 4n0igfxbd3kqvvj2k2xgysrp63l4v2gd110fwkk4apfpm0hvzwh0 \
    | xargs nix-hash --type sha256 --to-base32
0n0igfxbd3kqvvj2k2xgysrp63l4v2gd110fwkk4apfpm0hvzwh0

It's a real-life example:
https://github.com/NixOS/nixpkgs/pull/20208/files#r86695567
2016-11-06 22:17:37 +01:00
Eelco Dolstra
18b7363a69 Support optional sandbox paths
For example, you can now set

  build-sandbox-paths = /dev/nvidiactl?

to specify that /dev/nvidiactl should only be mounted in the sandbox
if it exists in the host filesystem. This is useful e.g. for EC2
images that should support both CUDA and non-CUDA instances.
2016-10-31 17:09:52 +01:00
Shea Levy
c4969aebaf Add nix.conf options for -k and -K
Fixes #1084
2016-10-27 12:49:34 -04:00
Tuomas Tynkkynen
f0fc3dd88b Fix SIGFPE from integer overflow during division
On some architectures (like x86_64 or i686, but not ARM for example)
overflow during integer division causes a crash due to SIGFPE.
Reproduces on a 64-bit system with:

    nix-instantiate --eval -E '(-9223372036854775807 - 1) / -1'

The only way this can happen is when the smallest possible integer is
divided by -1, so just special-case that.
2016-10-26 18:41:59 +03:00
Eelco Dolstra
5c0bd51d49 Fix build log output in nix-store --serve 2016-10-26 17:13:38 +02:00
Eelco Dolstra
d2c58ba605 Restore the CachedFailure status code
The removal of CachedFailure caused the value of TimedOut to change,
which broke timed-out handling in Hydra (so timed-out builds would
show up as "aborted" and would be retried, e.g. at
http://hydra.nixos.org/build/42537427).
2016-10-26 15:02:38 +02:00
Eelco Dolstra
fdbbcc4492 Remove addPathToAccessor 2016-10-21 18:09:30 +02:00
Eelco Dolstra
542ae5c8f8 BinaryCacheStore: Optionally write a NAR listing
The store parameter "write-nar-listing=1" will cause BinaryCacheStore
to write a file ‘<store-hash>.ls.xz’ for each ‘<store-hash>.narinfo’
added to the binary cache. This file contains an XZ-compressed JSON
file describing the contents of the NAR, excluding the contents of
regular files.

E.g.

  {
    "version": 1,
    "root": {
      "type": "directory",
      "entries": {
        "lib": {
          "type": "directory",
          "entries": {
            "Mcrt1.o": {
              "type": "regular",
              "size": 1288
            },
            "Scrt1.o": {
              "type": "regular",
              "size": 3920
            },
          }
        }
      }
      ...
    }
  }

(The actual file has no indentation.)

This is intended to speed up the NixOS channels programs index
generator [1], since fetching gazillions of large NARs from
cache.nixos.org is currently a bottleneck for updating the regular
(non-small) channel.

[1] https://github.com/NixOS/nixos-channel-scripts/blob/master/generate-programs-index.cc
2016-10-21 16:50:28 +02:00
Eelco Dolstra
307cc8c33d Merge branch 'priorityqueue' of https://github.com/groxxda/nix 2016-10-19 16:37:04 +02:00
Eelco Dolstra
795d9b8668 Merge branch 'regex' of https://github.com/groxxda/nix 2016-10-19 16:32:24 +02:00
Alexander Ried
c935e8eeaf fixup! replace own regex class with std::regex 2016-10-19 16:03:53 +02:00
Eelco Dolstra
19c278de89 Fix Darwin build
Done slightly differently from https://github.com/NixOS/nix/pull/1093.
2016-10-19 15:21:18 +02:00
Eelco Dolstra
29727ff944 Shut up clang warning 2016-10-19 15:02:48 +02:00
Eelco Dolstra
e5346c54df Fix uninitialised variable 2016-10-19 15:02:38 +02:00
Alexander Ried
b05b98df75 replace own regex class with std::regex 2016-10-18 20:22:25 +02:00
Alexander Ried
efb938468c downloader: use priority_queue 2016-10-18 15:45:51 +02:00
Eelco Dolstra
ae8884b949 Fix /dev/ptmx in sandboxes
This was broken since ff0c0b645c. Since
I can't figure out how to mount a devpts instance in the sandbox,
let's just bind-mount the host devpts.
2016-10-14 15:42:13 +02:00
Eelco Dolstra
2f9d60ed7f CURL_AT_LEAST_VERSION -> LIBCURL_VERSION_NUM
http://hydra.nixos.org/build/42025230
2016-10-14 14:44:15 +02:00
Eelco Dolstra
5e61b422c5 Fix HTTP/2 support
Commit 86e8c67efc broke it, because
CURL_* are not actually #defines.
2016-10-13 17:25:42 +02:00
Eelco Dolstra
fb2dd32100 SSL_CERT_FILE -> NIX_SSL_CERT_FILE
This prevents collisions with the "native" OpenSSL, in particular on
OS X.

Fixes #921.
2016-10-13 17:09:10 +02:00
Eelco Dolstra
816d3e5724 Shut up some warnings 2016-10-12 15:53:38 +02:00
Eelco Dolstra
82e2a070e0 Add some functions needed by hydra 2016-10-12 15:49:37 +02:00
Eelco Dolstra
edf9eb8181 querySubstitutablePaths(): Don't query paths for which we already have a substituter 2016-10-07 19:57:47 +02:00
Eelco Dolstra
0c85ef7090 LocalStore::querySubstitutablePaths(): Implement using queryValidPaths() 2016-10-07 19:46:56 +02:00
Eelco Dolstra
844219f364 Store::queryValidPaths(): Use async queryPathInfo()
This allows the binary cache substituter to pipeline requests.
2016-10-07 19:43:36 +02:00
Eelco Dolstra
c663b84573 Implement generic Store::queryValidPaths() 2016-10-07 19:20:47 +02:00
Eelco Dolstra
35db4f65a0 Add copyClosure utility function for Hydra 2016-10-07 19:15:25 +02:00
Eelco Dolstra
629ab80022 importPaths(): Fix accessor support for Hydra 2016-10-07 18:13:40 +02:00
Eelco Dolstra
b0f7f9c98f toJSON(): Support some more types 2016-10-06 17:00:52 +02:00
Eelco Dolstra
ae522f930d Fix getS3Stats() 2016-10-06 17:00:52 +02:00
Eelco Dolstra
cd128f4bad Merge pull request #998 from veprbl/rx_chmod_fix
override rx directory permissions in deletePath()
2016-10-06 11:08:21 +02:00