Commit graph

5015 commits

Author SHA1 Message Date
Eelco Dolstra
9a313469a4
Add a hook to run diffoscope when non-determinism is detected 2016-12-07 17:57:35 +01:00
Eelco Dolstra
b07060688a
Keep track of the exact build start/stop times 2016-12-07 16:09:38 +01:00
Eelco Dolstra
dadfddfa7c
Bail out early when non-determinism is detected 2016-12-07 15:31:18 +01:00
Eelco Dolstra
cea4e2c618
Expose enforce-determinism and the result to Hydra 2016-12-07 13:43:40 +01:00
Eelco Dolstra
8bdf83f936
Add an option to make non-determinism non-fatal
That is, when build-repeat > 0, and the output of two rounds differ,
then print a warning rather than fail the build. This is primarily to
let Hydra check reproducibility of all packages.
2016-12-07 13:16:06 +01:00
Eelco Dolstra
ceeedb58d2
Use a steady clock for timeouts
Fixes #1146.
2016-12-06 21:58:04 +01:00
Eelco Dolstra
7a3e7d0e61
nix-store --serve: Suppress log output on stderr when repeating a build 2016-12-06 17:43:39 +01:00
Eelco Dolstra
6069b946ad
nix-store --serve: Support setting build-repeat
This allows Hydra to test whether builds are reproducible.
2016-12-06 17:19:22 +01:00
Eelco Dolstra
603f08506e
Tweak error message 2016-12-06 17:18:40 +01:00
Eelco Dolstra
09191caea8
Add shell.nix 2016-12-06 17:17:29 +01:00
Eelco Dolstra
d1da6967b8
Drop unused WWW::Curl dependency 2016-12-06 17:17:29 +01:00
Eelco Dolstra
ae71895f55 Merge pull request #947 from layus/fix-path-slash
Improve error message on trailing path slashes
2016-12-06 15:21:17 +01:00
Guillaume Maudoux
e4b82af387 Improve error message on trailing path slashes 2016-11-27 17:48:46 +01:00
Eelco Dolstra
215b70f51e
Revert "Get rid of unicode quotes (#1140)"
This reverts commit f78126bfd6. There
really is no need for such a massive change...
2016-11-26 00:38:01 +01:00
Guillaume Maudoux
f78126bfd6 Get rid of unicode quotes (#1140) 2016-11-25 15:48:27 +01:00
Jörg Thalheim
855abd85d8
Simplify remouting with MS_PRIVATE in sandbox build
also fix race condition if mounts are added after mountinfo is read.
2016-11-25 00:15:39 +01:00
Eelco Dolstra
7ee43df862
nix-channel: Fix --update <CHANNELS>
This unbreaks "nixos-rebuild --upgrade".
2016-11-21 15:54:19 +01:00
Eelco Dolstra
349e988903
Typo 2016-11-17 17:55:41 +01:00
Shea Levy
00b8bce4d0 Fix binary-cache-store build 2016-11-17 11:48:10 -05:00
Shea Levy
0d2ebb4373 istringstream_nocopy: Implement in a standards-compliant way
Fixes #1135.
2016-11-17 10:10:32 -05:00
Shea Levy
3f4d3f8a1a nix-daemon: Fix splice faking on non-Linux 2016-11-17 08:10:12 -05:00
Shea Levy
8bf378e999 Update darwin build for optional sandbox paths
Fixes #1132
2016-11-17 08:06:32 -05:00
aszlig
4e1a2cd537
seccomp: Forge return values for *chown32
These syscalls are only available in 32bit architectures, but libseccomp
should handle them correctly even if we're on native architectures that
do not have these syscalls.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 17:29:36 +01:00
aszlig
ed64976cec
seccomp: Forge return codes for POSIX ACL syscalls
Commands such as "cp -p" also use fsetxattr() in addition to fchown(),
so we need to make sure these syscalls always return successful as well
in order to avoid nasty "Invalid value" errors.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 17:29:14 +01:00
aszlig
651a18dd24
release.nix: Add a test for sandboxing
Right now it only tests whether seccomp correctly forges the return
value of chown, but the long-term goal is to test the full sandboxing
functionality at some point in the future.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 16:48:34 +01:00
aszlig
b90a435332
libstore/build: Forge chown() to return success
What we basically want is a seccomp mode 2 BPF program like this but for
every architecture:

  BPF_STMT(BPF_LD+BPF_W+BPF_ABS, offsetof(struct seccomp_data, nr)),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_chown, 4, 0),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_fchown, 3, 0),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_fchownat, 2, 0),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_lchown, 1, 0),
  BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_ALLOW),
  BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_ERRNO)

However, on 32 bit architectures we do have chown32, lchown32 and
fchown32, so we'd need to add all the architecture blurb which
libseccomp handles for us.

So we only need to make sure that we add the 32bit seccomp arch while
we're on x86_64 and otherwise we just stay at the native architecture
which was set during seccomp_init(), which more or less replicates
setting 32bit personality during runChild().

The FORCE_SUCCESS() macro here could be a bit less ugly but I think
repeating the seccomp_rule_add() all over the place is way uglier.

Another way would have been to create a vector of syscalls to iterate
over, but that would make error messages uglier because we can either
only print the (libseccomp-internal) syscall number or use
seccomp_syscall_resolve_num_arch() to get the name or even make the
vector a pair number/name, essentially duplicating everything again.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 16:48:30 +01:00
aszlig
1c52e344c4
Add build dependency for libseccomp
We're going to use libseccomp instead of creating the raw BPF program,
because we have different syscall numbers on different architectures.

Although our initial seccomp rules will be quite small it really doesn't
make sense to generate the raw BPF program because we need to duplicate
it and/or make branches on every single architecture we want to suuport.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 16:48:26 +01:00
aszlig
e8838713df
Run builds as root in user namespace again
This reverts commit ff0c0b645c.

We're going to use seccomp to allow "cp -p" and force chown-related
syscalls to always return 0.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 16:48:08 +01:00
Eelco Dolstra
4be4f6de56
S3BinaryCacheStore:: Eliminate a string copy while uploading
This cuts hydra-queue-runner's peak memory usage by about a third.
2016-11-16 16:21:30 +01:00
Eelco Dolstra
10ae8fabf1 buildPaths(): Handle ecIncompleteClosure
buildPaths() on a non-derivation would incorrectly not throw an error
if the path didn't have a substitute.
2016-11-14 15:00:17 +01:00
Eelco Dolstra
b77fb8acb5 Don't rely on %m 2016-11-14 13:37:16 +01:00
Ludovic Courtès
ccb1022022 daemon: Do not error out when deduplication fails due to ENOSPC.
This solves a problem whereby if /gnu/store/.links had enough entries,
ext4's directory index would be full, leading to link(2) returning
ENOSPC.

* nix/libstore/optimise-store.cc (LocalStore::optimisePath_): Upon
ENOSPC from link(2), print a message and return instead of throwing a
'SysError'.
2016-11-14 13:35:03 +01:00
Eelco Dolstra
b8d9616af1 Merge pull request #1129 from layus/fix-comments
Fix comments parsing
2016-11-14 11:06:30 +01:00
Guillaume Maudoux
a5e761dddb Fix comments parsing
Fixed the parsing of multiline strings ending with an even number of
stars, like /** this **/.
Added test cases for comments.
2016-11-13 17:20:34 +01:00
Eelco Dolstra
dd77f7d593 Store::computeFSClosure(): Support a set of paths
This way, callers can exploits the parallelism of computeFSClosure()
when they have multiple paths that they need the (combined) closure of.
2016-11-10 17:45:04 +01:00
Shea Levy
bff3ad767e build-remote: replace strtoull with stoull to take advantage of C++ error handling 2016-11-10 11:09:15 -05:00
Shea Levy
167d12b02c build-remote: Implement in C++ 2016-11-10 11:09:15 -05:00
Eelco Dolstra
2af5d35fdc Merge branch 'master' of github.com:NixOS/nix 2016-11-09 19:09:44 +01:00
Eelco Dolstra
e4655b166c copyClosure() / copyStorePath(): Expose dontCheckSigs
Needed by Hydra.
2016-11-09 19:08:20 +01:00
Eelco Dolstra
4b8f1b0ec0 Merge branch 'ssh-store' of https://github.com/shlevy/nix 2016-11-09 18:57:22 +01:00
Eelco Dolstra
21c55ab3b5 Implement backwards-compatible RemoteStore::addToStore()
The SSHStore PR adds this functionality to the daemon, but we have to
handle the case where the Nix daemon is 1.11.

Also, don't require signatures for trusted users. This restores 1.11
behaviour.

Fixes https://github.com/NixOS/hydra/issues/398.
2016-11-09 18:45:06 +01:00
Eelco Dolstra
a83b10f84c Typo 2016-11-08 20:19:02 +01:00
Eelco Dolstra
b3ba762dbf Add missing #include 2016-11-07 14:35:47 +01:00
Vladimír Čunát
818aad3ec4
Detect and disallow base32 hash overflow
Example (before this commit):
$ nix-hash --type sha256 --to-base16 4n0igfxbd3kqvvj2k2xgysrp63l4v2gd110fwkk4apfpm0hvzwh0 \
    | xargs nix-hash --type sha256 --to-base32
0n0igfxbd3kqvvj2k2xgysrp63l4v2gd110fwkk4apfpm0hvzwh0

It's a real-life example:
https://github.com/NixOS/nixpkgs/pull/20208/files#r86695567
2016-11-06 22:17:37 +01:00
Manav Rathi
eec5409a69 installation: allow profile modification to be skipped (#1072)
The current behaviour modifies the first writeable file from amongst
.bash_profile, .bash_login and .profile.  So .bash_profile (if it is
writable) would be modified even if a user has already sourced nix.sh
in, say, .profile.

This commit introduces a new environment variable,
NIX_INSTALLER_NO_MODIFY_PROFILE.  If this is set during installation,
then the modifications are unconditionally skipped.

This is useful for users who have a manually curated set of dotfiles
that they are porting to a new machine. In such scenarios, nix.sh is
already sourced at a place where the user prefers.  Without this
change, the nix installer would insist on modifying .bash_profile if
it exists.

This commit also add documentations for both the current behaviour and
the new override.
2016-11-03 18:02:29 +01:00
Eelco Dolstra
18b7363a69 Support optional sandbox paths
For example, you can now set

  build-sandbox-paths = /dev/nvidiactl?

to specify that /dev/nvidiactl should only be mounted in the sandbox
if it exists in the host filesystem. This is useful e.g. for EC2
images that should support both CUDA and non-CUDA instances.
2016-10-31 17:09:52 +01:00
Shea Levy
c4969aebaf Add nix.conf options for -k and -K
Fixes #1084
2016-10-27 12:49:34 -04:00
Eelco Dolstra
e02a1352c1 Merge pull request #1108 from dezgeg/fix-sigfpe
Fix SIGFPE from integer overflow during division
2016-10-26 17:56:14 +02:00
Tuomas Tynkkynen
f0fc3dd88b Fix SIGFPE from integer overflow during division
On some architectures (like x86_64 or i686, but not ARM for example)
overflow during integer division causes a crash due to SIGFPE.
Reproduces on a 64-bit system with:

    nix-instantiate --eval -E '(-9223372036854775807 - 1) / -1'

The only way this can happen is when the smallest possible integer is
divided by -1, so just special-case that.
2016-10-26 18:41:59 +03:00
Eelco Dolstra
5c0bd51d49 Fix build log output in nix-store --serve 2016-10-26 17:13:38 +02:00