the hook every time we want to ask whether we can run a remote build
(which can be very often), we now reuse a hook process for answering
those queries until it accepts a build. So if there are N
derivations to be built, at most N hooks will be started.
faster than the old mode when fsyncs are enabled, because it only
performs an fsync() when doing a checkpoint, rather than at every
commit. Some timings for doing a "nix-instantiate /etc/nixos/nixos
-A system" after modifying the stdenv setup script:
42.5s - SQLite 3.6.23 with truncate mode and fsync
3.4s - SQLite 3.6.23 with truncate mode and no fsync
32.1s - SQLite 3.7.0 with truncate mode and fsync
16.8s - SQLite 3.7.0 with WAL mode and fsync, auto-checkpoint
every 1000 pages
8.3s - SQLite 3.7.0 with WAL mode and fsync, auto-checkpoint
every 8192 pages
1.7s - SQLite 3.7.0 with WAL mode and no fsync
The default is now to use WAL mode with fsyncs. Because WAL doesn't
work on remote filesystems such as NFS (as it uses shared memory),
truncate mode can be re-enabled by setting the "use-sqlite-wal"
option to false.
using the build hook mechanism, by setting the derivation attribute
"preferLocalBuild" to true. This has a few use cases:
- The user environment builder. Since it just creates a bunch of
symlinks without much computation, there is no reason to do it
remotely. In fact, doing it remotely requires the entire closure
of the user environment to be copied to the remote machine, which
is extremely wasteful.
- `fetchurl'. Performing the download on a remote machine and then
copying it to the local machine involves twice as much network
traffic as performing the download locally, and doesn't save any
CPU cycles on the local machine.
An "using namespace std" was added locally in those functions that refer to
names from <cstring>. That is not pretty, but it's a very portable solution,
because strcpy() and friends will be found in both the 'std' and in the global
namespace.
This patch adds the configuration file variable "build-cores" and the
command line argument "--cores". These settings specify the number of
CPU cores to utilize for parallel building within a job, i.e. by passing
an appropriate "-j" flag to GNU Make. The default value is 1, which
means that parallel building is *disabled*. If the number of build cores
is specified as 0 (synonymously: "guess" or "auto"), then the actual
value is supposed to be auto-detected by builders at run-time, i.e by
calling the nproc(1) utility from coreutils.
The environment variable $NIX_BUILD_CORES is available to builders, but
the contents of that variable does *not* influence the hash that goes
into the $out store path, i.e. the number of build cores to be utilized
can be changed at will without requiring any re-builds.
doesn't work because the garbage collector doesn't actually look at
locks. So r22253 was stupid. Use addTempRoot() instead. Also,
locking the temporary directory in exportPath() was silly because it
isn't even in the store.
changed. This prevents corrupt paths from spreading to other
machines. Note that checking the hash is cheap because we're
hashing anyway (because of the --sign feature).
to make the Refs table more space-efficient. For instance, this
reduces the size of the database on my laptop from 93 MiB to 18
MiB. (It was 72 MiB with the old schema on an ext3 disk with a 1
KiB block size.)
This prevents remote builders from being killed by the
`max-silent-time' inactivity monitor while they are waiting for a
long garbage collection to finish. This happens fairly often in the
Hydra build farm.
_FILE_OFFSET_BITS=64. Without it, functions like stat() fail on
large file sizes. This happened with a Nix store on squashfs:
$ nix-store --dump /tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds > /dev/null
error: getting attributes of path `/tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds': Value too large for defined data type
$ stat /tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds
File: `/tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds'
Size: 0 Blocks: 36028797018963968 IO Block: 1024 regular empty file
(This is a bug in squashfs or mksquashfs, but it shouldn't cause Nix
to fail.)
complete set of live and dead paths before starting the actual
deletion, but determines liveness on demand. I.e. for any path in
the store, it first tries to delete all the referrers, and then the
path itself. This means that the collector can start deleting paths
almost immediately.
(Linux) machines no longer maintain the atime because it's too
expensive, and on the machines where --use-atime is useful (like the
buildfarm), reading the atimes on the entire Nix store takes way too
much time to make it practical.
UTC) rather than 0 (00:00:00). 1 is a better choice because some
programs use 0 as a special value. For instance, the Template
Toolkit uses a timestamp of 0 to denote the non-existence of a file,
so it barfs on files in the Nix store (see
template-toolkit-nix-store.patch in Nixpkgs). Similarly, Maya 2008
fails to load script directories with a timestamp of 0 and can't be
patched because it's closed source.
This will also shut up those "implausibly old time stamp" GNU tar
warnings.
(that is, call the build hook with a certain interval until it
accepts the build).
* build-remote.pl was totally broken: for all system types other than
the local system type, it would send all builds to the *first*
machine of the appropriate type.
poll for it (i.e. if we can't acquire the lock, then let the main
select() loop wait for at most a few seconds and then try again).
This improves parallelism: if two nix-store processes are both
trying to build a path at the same time, the second one shouldn't
block; it should first see if it can build other goals. Also, it
prevents the deadlocks that have been occuring in Hydra lately,
where a process waits for a lock held by another process that's
waiting for a lock held by the first.
The downside is that polling isn't really elegant, but POSIX doesn't
provide a way to wait for locks in a select() loop. The only
solution would be to spawn a thread for each lock to do a blocking
fcntl() and then signal the main thread, but that would require
pthreads.
would just silently store only (fileSize % 2^32) bytes.
* Use posix_fallocate if available when unpacking archives.
* Provide a better error message when trying to unpack something that
isn't a NAR archive.
sure that it works as expected when you pass it a derivation. That
is, we have to make sure that all build-time dependencies are built,
and that they are all in the input closure (otherwise remote builds
might fail, for example). This is ensured at instantiation time by
adding all derivations and their sources to inputDrvs and inputSrcs.
hook. This fixes a problem with log files being partially or
completely filled with 0's because another nix-store process
truncates the log file. It should also be more efficient.
the DerivationGoal runs. Otherwise, if a goal is a top-level goal,
then the lock won't be released until nix-store finishes. With
--keep-going and lots of top-level goals, it's possible to run out
of file descriptors (this happened sometimes in the build farm for
Nixpkgs). Also, for failed derivation, it won't be possible to
build it again until the lock is released.
* Idem for locks on build users: these weren't released in a timely
manner for failed top-level derivation goals. So if there were more
than (say) 10 such failed builds, you would get an error about
having run out of build users.
scan for runtime dependencies (i.e. the local machine shouldn't do a
scan that the remote machine has already done). Also pipe directly
into `nix-store --import': don't use a temporary file.
(e.g. an SSH connection problem) and permanent failures (i.e. the
builder failed). This matters to Hydra (it wants to know whether it
makes sense to retry a build).
closure of the inputs. This really enforces that there can't be any
undeclared dependencies on paths in the store. This is done by
creating a fake Nix store and creating bind-mounts or hard-links in
the fake store for all paths in the closure. After the build, the
build output is moved from the fake store to the real store. TODO:
the chroot has to be on the same filesystem as the Nix store for
this to work, but this isn't enforced yet. (I.e. it only works
currently if /tmp is on the same FS as /nix/store.)
bind-mounts we do are only visible to the builder process and its
children. So accidentally doing "rm -rf" on the chroot directory
won't wipe out /nix/store and other bind-mounted directories
anymore. Also, the bind-mounts in the private namespace disappear
automatically when the builder exits.
necessary that at least one build hook doesn't return "postpone",
otherwise nix-store will barf ("waiting for a build slot, yet there
are no running children"). So inform the build hook when this is
the case, so that it can start a build even when that would exceed
the maximum load on a machine.
SHA-256 outputs of fixed-output derivations. I.e. they now produce
the same store path:
$ nix-store --add x
/nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x
$ nix-store --add-fixed --recursive sha256 x
/nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x
the latter being the same as the path that a derivation
derivation {
name = "x";
outputHashAlgo = "sha256";
outputHashMode = "recursive";
outputHash = "...";
...
};
produces.
This does change the output path for such fixed-output derivations.
Fortunately they are quite rare. The most common use is fetchsvn
calls with SHA-256 hashes. (There are a handful of those is
Nixpkgs, mostly unstable development packages.)
* Documented the computation of store paths (in store-api.cc).
subtle and often hard-to-reproduce bugs where programs in pipes
either barf with a "Broken pipe" message or not, depending on the
exact timing conditions. This particularly happened in GNU M4 (and
Bison, which uses M4).
disasters involving `rm -rf' on bind mounts. Will try the
definitive fix (per-process mounts, apparently possible via the
CLONE_NEWNS flag in clone()) some other time.
accessed time of paths that may be deleted. Anything more recently
used won't be deleted. The time is specified in time_t,
e.g. seconds since 1970-01-01 00:00:00 UTC; use `date +%s' to
convert to time_t from the command line.
Example: to delete everything that hasn't been used in the last two
months:
$ nix-store --gc -v --max-atime $(date +%s -d "2 months ago")
order of ascending last access time. This is useful in conjunction
with --max-freed or --max-links to prefer deleting non-recently used
garbage, which is good (especially in the build farm) since garbage
may become live again.
The code could easily be modified to accept other criteria for
ordering garbage by changing the comparison operator used by the
priority queue in collectGarbage().
particular, dietlibc cannot figure out the cwd because the inode of
the current directory doesn't appear in .. (because getdents returns
the inode of the mount point).
need /etc in the chroot (in particular, /etc/resolv.conf for
fetchurl). Not having /etc/resolv.conf in the chroot is a good
thing, since we don't want normal derivations to download files.
~/.nix-defexpr, otherwise the attribute cannot be selected with the
`-A' option. Useful if you want to stick a Nix expression directly
in ~/.nix-defexpr.
again. (After the previous substituter mechanism refactoring I
didn't update the code that obtains the references of substitutable
paths.) This required some refactoring: the substituter programs
are now kept running and receive/respond to info requests via
stdin/stdout.
bytes have been freed, `--max-links' to stop when the Nix store
directory has fewer than N hard links (the latter being important
for very large Nix stores on filesystems with a 32000 subdirectories
limit).
store under the reference relation, since that means that the
garbage collector will need a long time to start deleting paths.
Instead just delete the referrers of a path first.
This isn't usually a problem, except that it causes tests to fail
when performed in a directory with a very long path name. So chdir
to the socket directory and use a relative path name.
/tmp/nix-<pid>-<counter> for temporary build directories. This
increases purity a bit: many packages store the temporary build path
in their output, causing (generally unimportant) binary differences.
$ nix-env -e $(which firefox)
or
$ nix-env -e /nix/store/nywzlygrkfcgz7dfmhm5xixlx1l0m60v-pan-0.132
* nix-env -i: if an argument contains a slash anywhere, treat it as a
path and follow it through symlinks into the Nix store. This allows
things like
$ nix-build -A firefox
$ nix-env -i ./result
* nix-env -q/-i/-e: don't complain when the `*' selector doesn't match
anything. In particular, `nix-env -q \*' doesn't fail anymore on an
empty profile.
executed in a chroot that contains just the Nix store, the temporary
build directory, and a configurable set of additional directories
(/dev and /proc by default). This allows a bit more purity
enforcement: hidden build-time dependencies on directories such as
/usr or /nix/var/nix/profiles are no longer possible. As an added
benefit, accidental network downloads (cf. NIXPKGS-52) are prevented
as well (because files such as /etc/resolv.conf are not available in
the chroot).
However the usefulness of chroots is diminished by the fact that
many builders depend on /bin/sh, so you need /bin in the list of
additional directories. (And then on non-NixOS you need /lib as
well...)
usage by finding identical files in the store and hard-linking them
to each other. It typically reduces the size of the store by
something like 25-35%. This is what the optimise-store.pl script
did, but the new command is faster and more correct (it's safe wrt
garbage collection and concurrent builds).
(/nix/var/nix/daemon-socket). This allows access to the Nix daemon
to be restricted by setting the mode/ownership on that directory as
desired, e.g.
$ chmod 770 /nix/var/nix/daemon-socket
$ chown root.wheel /nix/var/nix/daemon-socket
to allow only users in the wheel group to use Nix.
Setting the ownership on a socket is much trickier, since the socket
must be deleted and recreated every time the daemon is started
(which would require additional Nix configuration file directives to
specify the mode/ownership, and wouldn't support arbitrary ACLs),
some BSD variants appear to ignore permissions on sockets, and it's
not clear whether the umask is respected on every platform when
creating sockets.
fixed-output derivations or substitutions try to build the same
store path at the same time. Locking generally catches this, but
not between multiple goals in the same process. This happened
especially often (actually, only) in the build farm with fetchurl
downloads of the same file being executed on multiple machines and
then copied back to the main machine where they would clobber each
other (NIXBF-13).
Solution: if a goal notices that the output path is already locked,
then go to sleep until another goal finishes (hopefully the one
locking the path) and try again.
need any info on substitutable paths, we just call the substituters
(such as download-using-manifests.pl) directly. This means that
it's no longer necessary for nix-pull to register substitutes or for
nix-channel to clear them, which makes those operations much faster
(NIX-95). Also, we don't have to worry about keeping nix-pull
manifests (in /nix/var/nix/manifests) and the database in sync with
each other.
The downside is that there is some overhead in calling an external
program to get the substitutes info. For instance, "nix-env -qas"
takes a bit longer.
Abolishing the substitutes table also makes the logic in
local-store.cc simpler, as we don't need to store info for invalid
paths. On the downside, you cannot do things like "nix-store -qR"
on a substitutable but invalid path (but nobody did that anyway).
* Never catch interrupts (the Interrupted exception).
--export' into the Nix store, and optionally check the cryptographic
signatures against /nix/etc/nix/signing-key.pub. (TODO: verify
against a set of public keys.)
path. This is like `nix-store --dump', only it also dumps the
meta-information of the store path (references, deriver). Will add
a `--sign' flag later to add a cryptographic signature, which we
will use for exchanging store paths between build farm machines in a
secure manner.
computing the store path (NIX-77). This is an important security
property in multi-user Nix stores.
Note that this changes the store paths of derivations (since the
derivation aterms are added using addTextToStore), but not most
outputs (unless they use builtins.toFile).
important to get garbage collection to work if there is any
inconsistency in the database (because the referrer table is used to
determine whether it is safe to delete a path).
* `nix-store --verify': show some progress.
from a source directory. All files for which a predicate function
returns true are copied to the store. Typical example is to leave
out the .svn directory:
stdenv.mkDerivation {
...
src = builtins.filterSource
(path: baseNameOf (toString path) != ".svn")
./source-dir;
# as opposed to
# src = ./source-dir;
}
This is important because the .svn directory influences the hash in
a rather unpredictable and variable way.
matters when running as root, since then we don't use the setuid
helper (which already used lchown()).
* Also check for an obscure security problem on platforms that don't
have lchown. Then we can't change the ownership of symlinks, which
doesn't matter *except* when the containing directory is writable by
the owner (which is the case with the top-level Nix store directory).
* Throw more exceptions as BuildErrors instead of Errors. This
matters when --keep-going is turned on. (A BuildError is caught
and terminates the goal in question, an Error terminates the
program.)
seconds without producing output on stdout or stderr (NIX-65). This
timeout can be specified using the `--max-silent-time' option or the
`build-max-silent-time' configuration setting. The default is
infinity (0).
* Fix a tricky race condition: if we kill the build user before the
child has done its setuid() to the build user uid, then it won't be
killed, and we'll potentially lock up in pid.wait(). So also send a
conventional kill to the child.
that have to be done as root: running builders under different uids,
changing ownership of build results, and deleting paths in the store
with the wrong ownership).
`nix-store --delete'. But unprivileged users are not allowed to
ignore liveness.
* `nix-store --delete --ignore-liveness': ignore the runtime roots as
well.
process, so forward the operation.
* Spam the user about GC misconfigurations (NIX-71).
* findRoots: skip all roots that are unreadable - the warnings with
which we spam the user should be enough.
processes can register indirect roots. Of course, there is still
the problem that the garbage collector can only read the targets of
the indirect roots when it's running as root...
* SIGIO -> SIGPOLL (POSIX calls it that).
* Use sigaction instead of signal to register the SIGPOLL handler.
Sigaction is better defined, and a handler registered with signal
appears not to interrupt fcntl(..., F_SETLKW, ...), which is bad.
via the Unix domain socket in /nix/var/nix/daemon.socket. The
server forks a worker process per connection.
* readString(): use the heap, not the stack.
* Some protocol fixes.
* Added `build-users-group', the group under which builds are to be
performed.
* Check that /nix/store has 1775 permission and is owner by the
build-users-group.
mode. Presumably nix-worker would be setuid to the Nix store user.
The worker performs all operations on the Nix store and database, so
the caller can be completely unprivileged.
This is already much more secure than the old setuid scheme, since
the worker doesn't need to do Nix expression evaluation and so on.
Most importantly, this means that it doesn't need to access any user
files, with all resulting security risks; it only performs pure
store operations.
Once this works, it is easy to move to a daemon model that forks off
a worker for connections established through a Unix domain socket.
That would be even more secure.
* Some refactoring: put the NAR archive integer/string serialisation
code in a separate file so it can be reused by the worker protocol
implementation.
containing functions that operate on the Nix store. One
implementation is LocalStore, which operates on the Nix store
directly. The next step, to enable secure multi-user Nix, is to
create a different implementation RemoteStore that talks to a
privileged daemon process that uses LocalStore to perform the actual
operations.
graph to be passed to a builder. This attribute should be a list of
pairs [name1 path1 name2 path2 ...]. The references graph of each
`pathN' will be stored in a text file `nameN' in the temporary build
directory. The text files have the format used by `nix-store
--register-validity'. However, the deriver fields are left empty.
`exportReferencesGraph' is useful for builders that want to do
something with the closure of a store path. Examples: the builders
that make initrds and ISO images for NixOS.
`exportReferencesGraph' is entirely pure. It's necessary because
otherwise the only way for a builder to get this information would
be to call `nix-store' directly, which is not allowed (though
unfortunately possible).
available. For instance,
$ nix-store -l $(which svn) | less
lets you read the build log of the Subversion instance in your
profile.
* `nix-store -qb': if applied to a non-derivation, take the deriver.
check that the references of the output of a derivation are in the
specified set. For instance,
allowedReferences = [];
specifies that the output cannot have any references. (This is
useful, for instance, for the generation of bootstrap binaries for
stdenv-linux, which must not have any references for purity). It
could also be used to guard against undesired runtime dependencies,
e.g.,
{gcc, dynlib}: derivation {
...
allowedReferences = [dynlib];
}
says that the output can refer to the path of `dynlib' but not
`gcc'. A `forbiddedReferences' attribute would be more useful for
this, though.
concatenation and string coercion. This was a big mess (see
e.g. NIX-67). Contexts are now folded into strings, so that they
don't cause evaluation errors when they're not expected. The
semantics of paths has been clarified (see nixexpr-ast.def).
toString() and coerceToString() have been merged.
Semantic change: paths are now copied to the store when they're in a
concatenation (and in most other situations - that's the
formalisation of the meaning of a path). So
"foo " + ./bla
evaluates to "foo /nix/store/hash...-bla", not "foo
/path/to/current-dir/bla". This prevents accidental impurities, and
is more consistent with the treatment of derivation outputs, e.g.,
`"foo " + bla' where `bla' is a derivation. (Here `bla' would be
replaced by the output path of `bla'.)
Nix-env failed to call addPermRoot(), which is necessary to safely
add a new root. So if nix-env started after and finished before the
garbage collector, the user environment (plus all other new stuff)
it built might be garbage collected, leading to a dangling symlink
chain in ~/.nix-profile...
* Be more explicit if we block on the GC lock ("waiting for the big
garbage collector lock...").
* Don't loop trying to create a new generation. It's not necessary
anymore since profiles are locked nowadays.
writable. File permissions on Cygwin are rather complex, and in this
case this check introduced a problem with build jobs invoke from
outside of Cygwin (MSYS). It seemed almost impossible to fix the
permissions of the directory, so for now this safety check is disabled
on Cygwin.
the disk is full (because to delete something from the Nix store, we
need a Berkeley DB transaction, which takes up disk space). Under
normal operation, we make sure that there exists a file
/nix/var/nix/db/reserved of 1 MB. When running the garbage
collector, we delete that file before we open the Berkeley DB
environment.
creates a new process group but also a new session. New sessions
have no controlling tty, so child processes like ssh cannot open
/dev/tty (which is bad).
intended). This ensures that any ssh child processes to remote
machines are also killed, and thus the Nix process on the remote
machine also exits. Without this, the remote Nix process will
continue until it exists or until its stdout buffer gets full and it
locks up. (Partially fixes NIX-35.)
deletes a path even if it is reachable from a root. However, it
won't delete a path that still has referrers (since that would
violate store invariants).
Don't try this at home. It's a useful hack for recovering from
certain situations in a somewhat clean way (e.g., holes in closures
due to disk corruption).
nix-store query options `--referer' and `--referer-closure' have
been changed to `--referrer' and `--referrer-closure' (but the old
ones are still accepted for compatibility).
mapping. The referer table is replaced by a referrer table (note
spelling fix) that stores each referrer separately. That is,
instead of having
referer[P] = {Q_1, Q_2, Q_3, ...}
we store
referer[(P, Q_1)] = ""
referer[(P, Q_2)] = ""
referer[(P, Q_3)] = ""
...
To find the referrers of P, we enumerate over the keys with a value
lexicographically greater than P. This requires the referrer table
to be stored as a B-Tree rather than a hash table.
(The tuples (P, Q) are stored as P + null-byte + Q.)
Old Nix databases are upgraded automatically to the new schema.
builder. Instead, require that the Nix store has sticky permission
(S_ISVTX); everyone can created files in the Nix store, but they
cannot delete, rename or modify files created by others.
root (or setuid root), then builds will be performed under one of
the users listed in the `build-users' configuration variables. This
is to make it impossible to influence build results externally,
allowing locally built derivations to be shared safely between
users (see ASE-2005 paper).
To do: only one builder should be active per build user.
derivations. This is mostly to simplify the implementation of
nix-prefetch-{url, svn}, which now work properly in setuid
installations.
* Enforce valid store names in `nix-store --add / --add-fixed'.
continue building when one fails unless `--keep-going' is
specified.
* When `--keep-going' is specified, print out the set of failing
derivations at the end (otherwise it can be hard to find out which
failed).
multiple times is also a top-level goal, then the second and later
instantiations would never be created because there would be a
stable pointer to the first one that would keep it alive in the
WeakGoalMap.
* Some tracing code for debugging this kind of problem.
user environment, e.g.,
$ nix-env -i /nix/store/z58v41v21xd3ywrqk1vmvdwlagjx7f10-aterm-2.3.1.drv
or
$ nix-env -i /nix/store/hsyj5pbn0d9iz7q0aj0fga7cpaadvp1l-aterm-2.3.1
This is useful because it allows Nix expressions to be bypassed
entirely. For instance, if only a nix-pull manifest is provided,
plus the top-level path of some component, it can be installed
without having to supply the Nix expression (e.g., for obfuscation,
or to be independent of Nix expression language changes or context
dependencies).
to derivations in user environments. Nice for developers (since it
prevents build-time-only dependencies from being GC'ed, in
conjunction with `gc-keep-outputs'). Turned off by default.
* Removed some dead code (successor stuff) from nix-push.
* Updated terminology in the tests (store expr -> drv path).
* Check that the deriver is set properly in the tests.
for finding build-time dependencies (possibly after a build). E.g.,
$ nix-store -qb aterm $(nix-store -qd $(which strc))
/nix/store/jw7c7s65n1gwhxpn35j9rgcci6ilzxym-aterm-2.3.1
* Arguments to nix-store can be files within store objects, e.g.,
/nix/store/jw7c...-aterm-2.3.1/bin/baffle.
* Idem for garbage collector roots.
This was necessary becase root finding must be done after
acquisition of the global GC lock.
This makes `nix-collect-garbage' obsolete; it is now just a wrapper
around `nix-store --gc'.
* Automatically remove stale GC roots (i.e., indirect GC roots that
point to non-existent paths).
get rid of GC roots. Nix-build places a symlink `result' in the
current directory. Previously, removing that symlink would not
remove the store path being linked to as a GC root. Now, the GC
root created by nix-build is actually a symlink in
`/nix/var/nix/gcroots/auto' to `result'. So if that symlink is
removed the GC root automatically becomes invalid (since it can no
longer be resolved). The root itself is not automatically removed -
the garbage collector should delete dangling roots.
immediately add the result as a permanent GC root. This is the only
way to prevent a race with the garbage collector. For instance, the
old style
ln -s $(nix-store -r $(nix-instantiate foo.nix)) \
/nix/var/nix/gcroots/result
has two time windows in which the garbage collector can interfere
(by GC'ing the derivation and the output, respectively). On the
other hand,
nix-store --add-root /nix/var/nix/gcroots/result -r \
$(nix-instantiate --add-root /nix/var/nix/gcroots/drv \
foo.nix)
is safe.
* nix-build: use `--add-root' to prevent GC races.
being created after the garbage collector has read the temproots
directory. This blocks the creation of new processes, but the
garbage collector could periodically release the GC lock to allow
them to run.
that they are deleted in an order that maintains the closure
invariant.
* Presence of a path in a temporary roots file does not imply that all
paths in its closure are also present, so add the closure.
roots to a per-process temporary file in /nix/var/nix/temproots
while holding a write lock on that file. The garbage collector
acquires read locks on all those files, thus blocking further
progress in other Nix processes, and reads the sets of temporary
roots.
though). In particular it's now much easier to register a GC root.
Just place a symlink to whatever store path it is that you want to
keep in /nix/var/nix/gcroots.
This simplifies garbage collection and `nix-store --query
--requisites' since we no longer need to treat derivations
specially.
* Better maintaining of the invariants, e.g., setReferences() can only
be called on a valid/substitutable path.
closure of the referers relation rather than the references
relation, i.e., the set of all paths that directly or indirectly
refer to the given path. Note that contrary to the references
closure this set is not fixed; it can change as paths are added to
or removed from the store.
promise :-) This allows derivations to specify on *what* output
paths of input derivations they are dependent. This helps to
prevent unnecessary downloads. For instance, a build might be
dependent on the `devel' and `lib' outputs of some library
component, but not the `docs' output.
graph. That is, `nix-store --query --references PATH' shows the set
of paths referenced by PATH, and `nix-store --query --referers PATH'
shows the set of paths referencing PATH.
`derivations.cc', etc.
* Store the SHA-256 content hash of store paths in the database after
they have been built/added. This is so that we can check whether
the store has been messed with (a la `rpm --verify').
* When registering path validity, verify that the closure property
holds.
representation of closures as ATerms in the Nix store. Instead, the
file system pointer graph is now stored in the Nix database. This
has many advantages:
- It greatly simplifies the implementation (we can drop the notion
of `successors', and so on).
- It makes registering roots for the garbage collector much easier.
Instead of specifying the closure expression as a root, you can
simply specify the store path that must be retained as a root.
This could not be done previously, since there was no way to find
the closure store expression containing a given store path.
- Better traceability: it is now possible to query what paths are
referenced by a path, and what paths refer to a path.
* Formalise the notion of fixed-output derivations, i.e., derivations
for which a cryptographic hash of the output is known in advance.
Changes to such derivations should not propagate upwards through the
dependency graph. Previously this was done by specifying the hash
component of the output path through the `id' attribute, but this is
insecure since you can lie about it (i.e., you can specify any hash
and then produce a completely different output). Now the
responsibility for checking the output is moved from the builder to
Nix itself.
A fixed-output derivation can be created by specifying the
`outputHash' and `outputHashAlgo' attributes, the latter taking
values `md5', `sha1', and `sha256', and the former specifying the
actual hash in hexadecimal or in base-32 (auto-detected by looking
at the length of the attribute value). MD5 is included for
compatibility but should be considered deprecated.
* Removed the `drvPath' pseudo-attribute in derivation results. It's
no longer necessary.
* Cleaned up the support for multiple output paths in derivation store
expressions. Each output now has a unique identifier (e.g., `out',
`devel', `docs'). Previously there was no way to tell output paths
apart at the store expression level.
* `nix-hash' now has a flag `--base32' to specify that the hash should
be printed in base-32 notation.
* `fetchurl' accepts parameters `sha256' and `sha1' in addition to
`md5'.
* `nix-prefetch-url' now prints out a SHA-1 hash in base-32. (TODO: a
flag to specify the hash.)
bits, then encode them in a radix-32 representation (using digits
and letters except e, o, u, and t). This produces store paths like
/nix/store/4i0zb0z7f88mwghjirkz702a71dcfivn-aterm-2.3.1. The nice
thing about this is that the hash part of the file name is still 32
characters, as before with MD5.
(Of course, shortening SHA-256 to 160 bits makes it no better than
SHA-160 in theory, but hopefully it's a bit more resistant to
attacks; it's certainly a lot slower.)
* Start cleaning up unique store path generation (they weren't always
unique; in particular the suffix ("-aterm-2.2", "-builder.sh") was
not part of the hash, therefore changes to the suffix would cause
multiple store objects with the same hash).
- Drop the store expression. So now a substitute is just a
command-line invocation (a program name + arguments). If you
register a substitute you are responsible for registering the
expression that built it (if any) as a root of the garbage
collector.
- Drop the substitutes-rev DB table.
permission to the Nix store or database. E.g., `nix-env -qa' will
work, but `nix-env -qas' won't (the latter needs DB access). The
option `--readonly-mode' forces this mode; otherwise, it's only
activated when the database cannot be opened.
derivation, since NormalisationGoal would first run a
NormalisationGoal on the subderivation (a no-op, since in a
situation where we need fallback the successor is known), and then
runs a RealisationGoal on the normal form, which then cannot do a
fallback because it doesn't know the derivation expression for which
it is a normal form.
Tossed out the 2-phase normalisation/realisation in
NormalisationGoal and SubstitutionGoal since it's no longer needed -
a RealisationGoal will run a NormalisationGoal if necessary.
Previously there was the problem that all files read by nix-env
etc. should be reachable and readable by the Nix user. So for
instance building a Nix expression in your home directory meant that
the home directory should have at least g+x or o+x permission so
that the Nix user could reach the Nix expression. Now we just
switch back to the original user just prior to reading sources and
the like. The places where this happens are somewhat arbitrary,
however. Any scope that has a live SwitchToOriginalUser object in
it is executed as the original user.
* Back out r1385. setreuid() sets the saved uid to the new
real/effective uid, which prevents us from switching back to the
original uid. setresuid() doesn't have this problem (although the
manpage has a bug: specifying -1 for the saved uid doesn't leave it
unchanged; an explicit value must be specified).
unreachable paths that haven't been used for N hours. For instance,
`nix-collect-garbage --min-age 168' only deletes paths that haven't
been accessed in the last week.
This is useful for instance in the build farm where many derivations
can be shared between consecutive builds, and we wouldn't want a
garbage collect to throw them all away. We could of course register
them as roots, but then we'd to unregister them at some point, which
would be a pain to manage. The `--min-age' flag gives us a sort of
MRU caching scheme.
BUG: this really shouldn't be in gc.cc since that violates
mechanism/policy separation.
suboperations `--print-live', `--print-dead', and `--delete'. The
roots are not determined by nix-store; they are read from standard
input. This is to make it easy to customise what the roots are.
The collector now no longer fails when store expressions are missing
(which legally happens when using substitutes). It never tries to
fetch paths through substitutes.
TODO: acquire a global lock on the store while garbage collecting.
* Removed `nix-store --delete'.
* Builder output is written to standard error by default.
* The option `-B' is gone.
* The option `-Q' suppresses builder output.
The result of this is that most Nix invocations shouldn't need any
flags w.r.t. logging.
derivation disables scanning for dependencies. Use at your own
risk. This is a quick hack to speed up UML image generation (image
are very big, say 1 GB).
It would be better if the scanner were faster, and didn't read the
whole file into memory.
Whenever Nix attempts to realise a derivation for which a closure is
already known, but this closure cannot be realised, fall back on
normalising the derivation.
The most common scenario in which this is useful is when we have
registered substitutes in order to perform binary distribution from,
say, a network repository. If the repository is down, the
realisation of the derivation will fail. When this option is
specified, Nix will build the derivation instead. Thus, binary
installation falls back on a source installation. This option is
not the default since it is generally not desirable for a transient
failure in obtaining the substitutes to lead to a full build from
source (with the related consumption of resources).
much as possible. (This is similar to GNU Make's `-k' flag.)
* Refactoring to implement this: previously we just bombed out when
a build failed, but now we have to clean up. In particular this
means that goals must be freed quickly --- they shouldn't hang
around until the worker exits. So the worker now maintains weak
pointers in order not to prevent garbage collection.
* Documented the `-k' and `-j' flags.
improve throughput.
* Don't build the `substitute-rev' table for now, since it caused
Theta(N^2) time and log file consumption when adding N substitutes.
Maybe we can do without it.
* A better substitute mechanism.
Instead of generating a store expression for each store path for
which we have a substitute, we can have a single store expression
that builds a generic program that is invoked to build the desired
store path, which is passed as an argument.
This means that operations like `nix-pull' only produce O(1) files
instead of O(N) files in the store when registering N substitutes.
(It consumes O(N) database storage, of course, but that's not a
performance problem).
* Added a test for the substitute mechanism.
* `nix-store --substitute' reads the substitutes from standard input,
instead of from the command line. This prevents us from running
into the kernel's limit on command line length.
* When a fast build wakes up a goal, try to start that goal in the
same iteration of the startBuild() loop of run(). Otherwise no job
might be started until the next job terminates.
in parallel. Hooks are more efficient: locks on output paths are
only acquired when the hook says that it is willing to accept a
build job. Hooks now work in two phases. First, they should first
tell Nix whether they are willing to accept a job. Nix guarantuees
that no two hooks will ever be in the first phase at the same time
(this simplifies the implementation of hooks, since they don't have
to perform locking (?)). Second, if they accept a job, they are
then responsible for building it (on the remote system), and copying
the result back. These can be run in parallel with other hooks and
locally executed jobs.
The implementation is a bit messy right now, though.
* The directory `distributed' shows a (hacky) example of a hook that
distributes build jobs over a set of machines listed in a
configuration file.
distributing a build action to another machine. In particular, the
paths in the input closures, the output paths, and successor mapping
for sub-derivations.
parallel as possible (similar to GNU Make's `-j' switch). This is
useful on SMP systems, but it is especially useful for doing builds
on multiple machines. The idea is that a large derivation is
initiated on one master machine, which then distributes
sub-derivations to any number of slave machines. This should not
happen synchronously or in lock-step, so the master must be capable
of dealing with multiple parallel build jobs. We now have the
infrastructure to support this.
TODO: substitutes are currently broken.
builders to point to the store and the temporary build directory,
respectively. Useful for purity checking.
* Also set TEMPDIR, TMPDIR, TEMP, and TEMP to NIX_BUILD_TOP to make
sure that tools in the builder store temporary files in the right
location.
chroot() environment.
* A operation `--validpath' to register path validity. Useful for
bootstrapping in a pure Nix environment.
* Safety checks: ensure that files involved in store operations are in
the store.
Nix. This is to prevent Berkeley DB from becoming wedged.
Unfortunately it is not possible to throw C++ exceptions from a
signal handler. In fact, you can't do much of anything except
change variables of type `volatile sig_atomic_t'. So we set an
interrupt flag in the signal handler and check it at various
strategic locations in the code (by calling checkInterrupt()).
Since this is unlikely to cover all cases (e.g., (semi-)infinite
loops), sometimes SIGTERM may now be required to kill Nix.
it automatically removes log files when they are no longer needed.
*** IMPORTANT ***
If you have an existing Nix installation, you must checkpoint the
Nix database to prevent recent transactions from being undone. Do
the following:
- optional: make a backup of $prefix/var/nix/db.
- run `db_checkpoint' from Berkeley DB 4.1:
$ db_checkpoint -h $prefix/var/nix/db -1
- optional (?): run `db_recover' from Berkeley DB 4.1:
$ db_recover -h $prefix/var/nix/db
- remove $prefix/var/nix/db/log* and $prefix/var/nix/db/__db*
path of the Nix expression to be used with the import, upgrade, and
query commands. For instance,
$ nix-env -I ~/nixpkgs/pkgs/system/i686-linux.nix
$ nix-env --query --available [aka -qa]
sylpheed-0.9.7
bison-1.875
pango-1.2.5
subversion-0.35.1
...
$ nix-env -i sylpheed
$ nix-env -u subversion
There can be only one default at a time.
* If the path to a Nix expression is a symlink, follow the symlink
prior to resolving relative path references in the expression.
