This patch adds the configuration file variable "build-cores" and the
command line argument "--cores". These settings specify the number of
CPU cores to utilize for parallel building within a job, i.e. by passing
an appropriate "-j" flag to GNU Make. The default value is 1, which
means that parallel building is *disabled*. If the number of build cores
is specified as 0 (synonymously: "guess" or "auto"), then the actual
value is supposed to be auto-detected by builders at run-time, i.e by
calling the nproc(1) utility from coreutils.
The environment variable $NIX_BUILD_CORES is available to builders, but
the contents of that variable does *not* influence the hash that goes
into the $out store path, i.e. the number of build cores to be utilized
can be changed at will without requiring any re-builds.
_FILE_OFFSET_BITS=64. Without it, functions like stat() fail on
large file sizes. This happened with a Nix store on squashfs:
$ nix-store --dump /tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds > /dev/null
error: getting attributes of path `/tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds': Value too large for defined data type
$ stat /tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds
File: `/tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds'
Size: 0 Blocks: 36028797018963968 IO Block: 1024 regular empty file
(This is a bug in squashfs or mksquashfs, but it shouldn't cause Nix
to fail.)
(that is, call the build hook with a certain interval until it
accepts the build).
* build-remote.pl was totally broken: for all system types other than
the local system type, it would send all builds to the *first*
machine of the appropriate type.
poll for it (i.e. if we can't acquire the lock, then let the main
select() loop wait for at most a few seconds and then try again).
This improves parallelism: if two nix-store processes are both
trying to build a path at the same time, the second one shouldn't
block; it should first see if it can build other goals. Also, it
prevents the deadlocks that have been occuring in Hydra lately,
where a process waits for a lock held by another process that's
waiting for a lock held by the first.
The downside is that polling isn't really elegant, but POSIX doesn't
provide a way to wait for locks in a select() loop. The only
solution would be to spawn a thread for each lock to do a blocking
fcntl() and then signal the main thread, but that would require
pthreads.
sure that it works as expected when you pass it a derivation. That
is, we have to make sure that all build-time dependencies are built,
and that they are all in the input closure (otherwise remote builds
might fail, for example). This is ensured at instantiation time by
adding all derivations and their sources to inputDrvs and inputSrcs.
hook. This fixes a problem with log files being partially or
completely filled with 0's because another nix-store process
truncates the log file. It should also be more efficient.
the DerivationGoal runs. Otherwise, if a goal is a top-level goal,
then the lock won't be released until nix-store finishes. With
--keep-going and lots of top-level goals, it's possible to run out
of file descriptors (this happened sometimes in the build farm for
Nixpkgs). Also, for failed derivation, it won't be possible to
build it again until the lock is released.
* Idem for locks on build users: these weren't released in a timely
manner for failed top-level derivation goals. So if there were more
than (say) 10 such failed builds, you would get an error about
having run out of build users.
scan for runtime dependencies (i.e. the local machine shouldn't do a
scan that the remote machine has already done). Also pipe directly
into `nix-store --import': don't use a temporary file.
(e.g. an SSH connection problem) and permanent failures (i.e. the
builder failed). This matters to Hydra (it wants to know whether it
makes sense to retry a build).
closure of the inputs. This really enforces that there can't be any
undeclared dependencies on paths in the store. This is done by
creating a fake Nix store and creating bind-mounts or hard-links in
the fake store for all paths in the closure. After the build, the
build output is moved from the fake store to the real store. TODO:
the chroot has to be on the same filesystem as the Nix store for
this to work, but this isn't enforced yet. (I.e. it only works
currently if /tmp is on the same FS as /nix/store.)
bind-mounts we do are only visible to the builder process and its
children. So accidentally doing "rm -rf" on the chroot directory
won't wipe out /nix/store and other bind-mounted directories
anymore. Also, the bind-mounts in the private namespace disappear
automatically when the builder exits.
necessary that at least one build hook doesn't return "postpone",
otherwise nix-store will barf ("waiting for a build slot, yet there
are no running children"). So inform the build hook when this is
the case, so that it can start a build even when that would exceed
the maximum load on a machine.
subtle and often hard-to-reproduce bugs where programs in pipes
either barf with a "Broken pipe" message or not, depending on the
exact timing conditions. This particularly happened in GNU M4 (and
Bison, which uses M4).
disasters involving `rm -rf' on bind mounts. Will try the
definitive fix (per-process mounts, apparently possible via the
CLONE_NEWNS flag in clone()) some other time.
particular, dietlibc cannot figure out the cwd because the inode of
the current directory doesn't appear in .. (because getdents returns
the inode of the mount point).
need /etc in the chroot (in particular, /etc/resolv.conf for
fetchurl). Not having /etc/resolv.conf in the chroot is a good
thing, since we don't want normal derivations to download files.
again. (After the previous substituter mechanism refactoring I
didn't update the code that obtains the references of substitutable
paths.) This required some refactoring: the substituter programs
are now kept running and receive/respond to info requests via
stdin/stdout.
/tmp/nix-<pid>-<counter> for temporary build directories. This
increases purity a bit: many packages store the temporary build path
in their output, causing (generally unimportant) binary differences.
executed in a chroot that contains just the Nix store, the temporary
build directory, and a configurable set of additional directories
(/dev and /proc by default). This allows a bit more purity
enforcement: hidden build-time dependencies on directories such as
/usr or /nix/var/nix/profiles are no longer possible. As an added
benefit, accidental network downloads (cf. NIXPKGS-52) are prevented
as well (because files such as /etc/resolv.conf are not available in
the chroot).
However the usefulness of chroots is diminished by the fact that
many builders depend on /bin/sh, so you need /bin in the list of
additional directories. (And then on non-NixOS you need /lib as
well...)
fixed-output derivations or substitutions try to build the same
store path at the same time. Locking generally catches this, but
not between multiple goals in the same process. This happened
especially often (actually, only) in the build farm with fetchurl
downloads of the same file being executed on multiple machines and
then copied back to the main machine where they would clobber each
other (NIXBF-13).
Solution: if a goal notices that the output path is already locked,
then go to sleep until another goal finishes (hopefully the one
locking the path) and try again.
need any info on substitutable paths, we just call the substituters
(such as download-using-manifests.pl) directly. This means that
it's no longer necessary for nix-pull to register substitutes or for
nix-channel to clear them, which makes those operations much faster
(NIX-95). Also, we don't have to worry about keeping nix-pull
manifests (in /nix/var/nix/manifests) and the database in sync with
each other.
The downside is that there is some overhead in calling an external
program to get the substitutes info. For instance, "nix-env -qas"
takes a bit longer.
Abolishing the substitutes table also makes the logic in
local-store.cc simpler, as we don't need to store info for invalid
paths. On the downside, you cannot do things like "nix-store -qR"
on a substitutable but invalid path (but nobody did that anyway).
* Never catch interrupts (the Interrupted exception).
from a source directory. All files for which a predicate function
returns true are copied to the store. Typical example is to leave
out the .svn directory:
stdenv.mkDerivation {
...
src = builtins.filterSource
(path: baseNameOf (toString path) != ".svn")
./source-dir;
# as opposed to
# src = ./source-dir;
}
This is important because the .svn directory influences the hash in
a rather unpredictable and variable way.
* Throw more exceptions as BuildErrors instead of Errors. This
matters when --keep-going is turned on. (A BuildError is caught
and terminates the goal in question, an Error terminates the
program.)
seconds without producing output on stdout or stderr (NIX-65). This
timeout can be specified using the `--max-silent-time' option or the
`build-max-silent-time' configuration setting. The default is
infinity (0).
* Fix a tricky race condition: if we kill the build user before the
child has done its setuid() to the build user uid, then it won't be
killed, and we'll potentially lock up in pid.wait(). So also send a
conventional kill to the child.
that have to be done as root: running builders under different uids,
changing ownership of build results, and deleting paths in the store
with the wrong ownership).
`nix-store --delete'. But unprivileged users are not allowed to
ignore liveness.
* `nix-store --delete --ignore-liveness': ignore the runtime roots as
well.
* Added `build-users-group', the group under which builds are to be
performed.
* Check that /nix/store has 1775 permission and is owner by the
build-users-group.
containing functions that operate on the Nix store. One
implementation is LocalStore, which operates on the Nix store
directly. The next step, to enable secure multi-user Nix, is to
create a different implementation RemoteStore that talks to a
privileged daemon process that uses LocalStore to perform the actual
operations.
graph to be passed to a builder. This attribute should be a list of
pairs [name1 path1 name2 path2 ...]. The references graph of each
`pathN' will be stored in a text file `nameN' in the temporary build
directory. The text files have the format used by `nix-store
--register-validity'. However, the deriver fields are left empty.
`exportReferencesGraph' is useful for builders that want to do
something with the closure of a store path. Examples: the builders
that make initrds and ISO images for NixOS.
`exportReferencesGraph' is entirely pure. It's necessary because
otherwise the only way for a builder to get this information would
be to call `nix-store' directly, which is not allowed (though
unfortunately possible).
available. For instance,
$ nix-store -l $(which svn) | less
lets you read the build log of the Subversion instance in your
profile.
* `nix-store -qb': if applied to a non-derivation, take the deriver.
check that the references of the output of a derivation are in the
specified set. For instance,
allowedReferences = [];
specifies that the output cannot have any references. (This is
useful, for instance, for the generation of bootstrap binaries for
stdenv-linux, which must not have any references for purity). It
could also be used to guard against undesired runtime dependencies,
e.g.,
{gcc, dynlib}: derivation {
...
allowedReferences = [dynlib];
}
says that the output can refer to the path of `dynlib' but not
`gcc'. A `forbiddedReferences' attribute would be more useful for
this, though.
writable. File permissions on Cygwin are rather complex, and in this
case this check introduced a problem with build jobs invoke from
outside of Cygwin (MSYS). It seemed almost impossible to fix the
permissions of the directory, so for now this safety check is disabled
on Cygwin.
creates a new process group but also a new session. New sessions
have no controlling tty, so child processes like ssh cannot open
/dev/tty (which is bad).
intended). This ensures that any ssh child processes to remote
machines are also killed, and thus the Nix process on the remote
machine also exits. Without this, the remote Nix process will
continue until it exists or until its stdout buffer gets full and it
locks up. (Partially fixes NIX-35.)
builder. Instead, require that the Nix store has sticky permission
(S_ISVTX); everyone can created files in the Nix store, but they
cannot delete, rename or modify files created by others.
root (or setuid root), then builds will be performed under one of
the users listed in the `build-users' configuration variables. This
is to make it impossible to influence build results externally,
allowing locally built derivations to be shared safely between
users (see ASE-2005 paper).
To do: only one builder should be active per build user.
continue building when one fails unless `--keep-going' is
specified.
* When `--keep-going' is specified, print out the set of failing
derivations at the end (otherwise it can be hard to find out which
failed).
multiple times is also a top-level goal, then the second and later
instantiations would never be created because there would be a
stable pointer to the first one that would keep it alive in the
WeakGoalMap.
* Some tracing code for debugging this kind of problem.
* Removed some dead code (successor stuff) from nix-push.
* Updated terminology in the tests (store expr -> drv path).
* Check that the deriver is set properly in the tests.
roots to a per-process temporary file in /nix/var/nix/temproots
while holding a write lock on that file. The garbage collector
acquires read locks on all those files, thus blocking further
progress in other Nix processes, and reads the sets of temporary
roots.
though). In particular it's now much easier to register a GC root.
Just place a symlink to whatever store path it is that you want to
keep in /nix/var/nix/gcroots.
This simplifies garbage collection and `nix-store --query
--requisites' since we no longer need to treat derivations
specially.
* Better maintaining of the invariants, e.g., setReferences() can only
be called on a valid/substitutable path.
promise :-) This allows derivations to specify on *what* output
paths of input derivations they are dependent. This helps to
prevent unnecessary downloads. For instance, a build might be
dependent on the `devel' and `lib' outputs of some library
component, but not the `docs' output.
`derivations.cc', etc.
* Store the SHA-256 content hash of store paths in the database after
they have been built/added. This is so that we can check whether
the store has been messed with (a la `rpm --verify').
* When registering path validity, verify that the closure property
holds.
