the hook every time we want to ask whether we can run a remote build
(which can be very often), we now reuse a hook process for answering
those queries until it accepts a build. So if there are N
derivations to be built, at most N hooks will be started.
faster than the old mode when fsyncs are enabled, because it only
performs an fsync() when doing a checkpoint, rather than at every
commit. Some timings for doing a "nix-instantiate /etc/nixos/nixos
-A system" after modifying the stdenv setup script:
42.5s - SQLite 3.6.23 with truncate mode and fsync
3.4s - SQLite 3.6.23 with truncate mode and no fsync
32.1s - SQLite 3.7.0 with truncate mode and fsync
16.8s - SQLite 3.7.0 with WAL mode and fsync, auto-checkpoint
every 1000 pages
8.3s - SQLite 3.7.0 with WAL mode and fsync, auto-checkpoint
every 8192 pages
1.7s - SQLite 3.7.0 with WAL mode and no fsync
The default is now to use WAL mode with fsyncs. Because WAL doesn't
work on remote filesystems such as NFS (as it uses shared memory),
truncate mode can be re-enabled by setting the "use-sqlite-wal"
option to false.
using the build hook mechanism, by setting the derivation attribute
"preferLocalBuild" to true. This has a few use cases:
- The user environment builder. Since it just creates a bunch of
symlinks without much computation, there is no reason to do it
remotely. In fact, doing it remotely requires the entire closure
of the user environment to be copied to the remote machine, which
is extremely wasteful.
- `fetchurl'. Performing the download on a remote machine and then
copying it to the local machine involves twice as much network
traffic as performing the download locally, and doesn't save any
CPU cycles on the local machine.
instance) "nix-env -i wine" works on x86_64-linux, even though Wine
is built on i686-linux. In the event that there are multiple
matching derivations, prefer those built for the current system.
An "using namespace std" was added locally in those functions that refer to
names from <cstring>. That is not pretty, but it's a very portable solution,
because strcpy() and friends will be found in both the 'std' and in the global
namespace.
This patch adds the configuration file variable "build-cores" and the
command line argument "--cores". These settings specify the number of
CPU cores to utilize for parallel building within a job, i.e. by passing
an appropriate "-j" flag to GNU Make. The default value is 1, which
means that parallel building is *disabled*. If the number of build cores
is specified as 0 (synonymously: "guess" or "auto"), then the actual
value is supposed to be auto-detected by builders at run-time, i.e by
calling the nproc(1) utility from coreutils.
The environment variable $NIX_BUILD_CORES is available to builders, but
the contents of that variable does *not* influence the hash that goes
into the $out store path, i.e. the number of build cores to be utilized
can be changed at will without requiring any re-builds.
doesn't work because the garbage collector doesn't actually look at
locks. So r22253 was stupid. Use addTempRoot() instead. Also,
locking the temporary directory in exportPath() was silly because it
isn't even in the store.
errors with position info.
* For all positions, use the position of the first character of the
first token, rather than the last character of the first token plus
one.
check' now succeeds :-)
* An attribute set such as `{ foo = { enable = true; };
foo.port = 23; }' now parses. It was previously rejected, but I'm
too lazy to implement the check. (The only reason to reject it is
that the reverse, `{ foo.port = 23; foo = { enable = true; }; }', is
rejected, which is kind of ugly.)
values. This improves sharing and gives another speed up.
Evaluation of the NixOS system attribute is now almost 7 times
faster than the old evaluator.
use site, allowing environments to be stores as vectors of values
rather than maps. This should speed up evaluation and reduce the
number of allocations.
precedence, i.e. `with {x=1;}; with {x=2;}; x' evaluates to 2'.
This has a simpler implementation and seems more natural. There
doesn't seem to be any code in Nixpkgs or NixOS that relies on the
old behaviour.
efficiently. The symbol table ensures that there is only one copy
of each symbol, thus allowing symbols to be compared efficiently
using a pointer equality test.
This fixes a regression introduced in r20882 ("Add source location
information to the XML output.").
* src/libexpr/expr-to-xml.cc (nix::printTermAsXML): Dereference the
attribute RHS from "drvPath" and "outPath".
then the blackhole has to be removed to ensure that repeated
evaluation of the same value gives an assertion failure again rather
than an "infinite recursion" error.
that there are some places in Nixpkgs (php_configurable /
composableDerivation, it seems) that call `derivation' with
incorrect arguments (namely, the `name' attribute missing) but get
away with it because of laziness.
* Removed exprToString and stringToExpr because there is no ATerm
representation to work on anymore (and exposing the internals of the
evaluator like this is not a good idea anyway).
* src/libexpr/expr-to-xml.cc (nix::showAttrs): Add `location'
parameter. Provide location XML attributes when it's true. Update
callers.
(nix::printTermAsXML): Likewise.
* src/libexpr/expr-to-xml.hh (nix::printTermAsXML): Update prototype;
have `location' default to `false'.
* src/nix-instantiate/nix-instantiate.cc (printResult, processExpr): Add
`location' parameter; update callers.
(run): Add support for `--no-location'.
* src/nix-instantiate/help.txt: Update accordingly.
* tests/lang.sh: Invoke `nix-instantiate' with `--no-location' for the
XML tests.
* tests/lang/eval-okay-toxml.exp, tests/lang/eval-okay-to-xml.nix: New
files.
* src/libexpr/expr-to-xml.cc (nix::showAttrs): Dereference the attribute
RHS. Add "path", "line", and "column" XML attributes to the node when
source location information is available.
(nix::printTermAsXML): Likewise for functions.
allowed. So `name1@name2', `{attrs1}@{attrs2}' and so on are now no
longer legal. This is no big loss because they were not useful
anyway.
This also changes the output of builtins.toXML for @-patterns
slightly.
changed. This prevents corrupt paths from spreading to other
machines. Note that checking the hash is cheap because we're
hashing anyway (because of the --sign feature).
to make the Refs table more space-efficient. For instance, this
reduces the size of the database on my laptop from 93 MiB to 18
MiB. (It was 72 MiB with the old schema on an ext3 disk with a 1
KiB block size.)
This prevents remote builders from being killed by the
`max-silent-time' inactivity monitor while they are waiting for a
long garbage collection to finish. This happens fairly often in the
Hydra build farm.
_FILE_OFFSET_BITS=64. Without it, functions like stat() fail on
large file sizes. This happened with a Nix store on squashfs:
$ nix-store --dump /tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds > /dev/null
error: getting attributes of path `/tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds': Value too large for defined data type
$ stat /tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds
File: `/tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds'
Size: 0 Blocks: 36028797018963968 IO Block: 1024 regular empty file
(This is a bug in squashfs or mksquashfs, but it shouldn't cause Nix
to fail.)
complete set of live and dead paths before starting the actual
deletion, but determines liveness on demand. I.e. for any path in
the store, it first tries to delete all the referrers, and then the
path itself. This means that the collector can start deleting paths
almost immediately.
(Linux) machines no longer maintain the atime because it's too
expensive, and on the machines where --use-atime is useful (like the
buildfarm), reading the atimes on the entire Nix store takes way too
much time to make it practical.
intersectAttrs returns the (right-biased) intersection between two
attribute sets, e.g. every attribute from the second set that also
exists in the first. functionArgs returns the set of attributes
expected by a function.
The main goal of these is to allow the elimination of most of
all-packages.nix. Most package instantiations in all-packages.nix
have this form:
foo = import ./foo.nix {
inherit a b c;
};
With intersectAttrs and functionArgs, this can be written as:
foo = callPackage (import ./foo.nix) { };
where
callPackage = f: args:
f ((builtins.intersectAttrs (builtins.functionArgs f) pkgs) // args);
I.e., foo.nix is called with all attributes from "pkgs" that it
actually needs (e.g., pkgs.a, pkgs.b and pkgs.c). (callPackage can
do any other generic package-level stuff we might want, such as
applying makeOverridable.) Of course, the automatically supplied
arguments can be overriden if needed, e.g.
foo = callPackage (import ./foo.nix) {
c = c_version_2;
};
but for the vast majority of packages, this won't be needed.
The advantages are to reduce the amount of typing needed to add a
dependency (from three sites to two), and to reduce the number of
trivial commits to all-packages.nix. For the former, there have
been two previous attempts:
- Use "args: with args;" in the package's function definition.
This however obscures the actual expected arguments of a
function, which is very bad.
- Use "{ arg1, arg2, ... }:" in the package's function definition
(i.e. use the ellipis "..." to allow arbitrary additional
arguments), and then call the function with all of "pkgs" as an
argument. But this inhibits error detection if you call it with
an misspelled (or obsolete) argument.
NixOS evaluation errors in particular look intimidating and
generally aren't very useful. Ideally the builtins.throw messages
should be self-contained.
UTC) rather than 0 (00:00:00). 1 is a better choice because some
programs use 0 as a special value. For instance, the Template
Toolkit uses a timestamp of 0 to denote the non-existence of a file,
so it barfs on files in the Nix store (see
template-toolkit-nix-store.patch in Nixpkgs). Similarly, Maya 2008
fails to load script directories with a timestamp of 0 and can't be
patched because it's closed source.
This will also shut up those "implausibly old time stamp" GNU tar
warnings.
attributes of the rec are in scope of `e'. This is useful in
expressions such as
rec {
lib = import ./lib;
inherit (lib) concatStrings;
}
It does change the semantics of expressions such as
let x = {y = 1;}; in rec { x = {y = 2;}; inherit (x) y; }.y
This now returns 2 instead of 1. However, no code in Nixpkgs or
NixOS seems to rely on the old behaviour.
shorthand for {x = {y = {z = ...;};};}. This is especially useful
for NixOS configuration files, e.g.
{
services = {
sshd = {
enable = true;
port = 2022;
};
};
}
can now be written as
{
services.sshd.enable = true;
services.sshd.port = 2022;
}
However, it is currently not permitted to write
{
services.sshd = {enable = true;};
services.sshd.port = 2022;
}
as this is considered a duplicate definition of `services.sshd'.
broken, but now the evaluator checks for it to prevent Nix
expressions from relying on undefined behaviour. Equality tests are
implemented using a shallow pointer equality test between ATerms.
However, because attribute sets are lazy and contain position
information, this can give false positives. For instance,
previously
let y = {x = 1;}; in y == y
evaluated to true, while the equivalent expression
{x = 1;} == {x = 1;}
evaluated to false. So disallow these tests for now. (Eventually
we may want to implement deep equality tests for attribute sets,
like lib.eqStrict.)
* Idem: disallow comparisons between functions.
* Implemented deep comparisons of lists. This had the same problem as
attribute sets - the elements in the list weren't evaluated. For
instance,
["xy"] == [("x" + "y")]
evaluated to false. Now it works properly.
(that is, call the build hook with a certain interval until it
accepts the build).
* build-remote.pl was totally broken: for all system types other than
the local system type, it would send all builds to the *first*
machine of the appropriate type.
poll for it (i.e. if we can't acquire the lock, then let the main
select() loop wait for at most a few seconds and then try again).
This improves parallelism: if two nix-store processes are both
trying to build a path at the same time, the second one shouldn't
block; it should first see if it can build other goals. Also, it
prevents the deadlocks that have been occuring in Hydra lately,
where a process waits for a lock held by another process that's
waiting for a lock held by the first.
The downside is that polling isn't really elegant, but POSIX doesn't
provide a way to wait for locks in a select() loop. The only
solution would be to spawn a thread for each lock to do a blocking
fcntl() and then signal the main thread, but that would require
pthreads.
would just silently store only (fileSize % 2^32) bytes.
* Use posix_fallocate if available when unpacking archives.
* Provide a better error message when trying to unpack something that
isn't a NAR archive.
sure that it works as expected when you pass it a derivation. That
is, we have to make sure that all build-time dependencies are built,
and that they are all in the input closure (otherwise remote builds
might fail, for example). This is ensured at instantiation time by
adding all derivations and their sources to inputDrvs and inputSrcs.
hook. This fixes a problem with log files being partially or
completely filled with 0's because another nix-store process
truncates the log file. It should also be more efficient.
the DerivationGoal runs. Otherwise, if a goal is a top-level goal,
then the lock won't be released until nix-store finishes. With
--keep-going and lots of top-level goals, it's possible to run out
of file descriptors (this happened sometimes in the build farm for
Nixpkgs). Also, for failed derivation, it won't be possible to
build it again until the lock is released.
* Idem for locks on build users: these weren't released in a timely
manner for failed top-level derivation goals. So if there were more
than (say) 10 such failed builds, you would get an error about
having run out of build users.
scan for runtime dependencies (i.e. the local machine shouldn't do a
scan that the remote machine has already done). Also pipe directly
into `nix-store --import': don't use a temporary file.
(e.g. an SSH connection problem) and permanent failures (i.e. the
builder failed). This matters to Hydra (it wants to know whether it
makes sense to retry a build).
allocate memory, which is verboten in signal handlers. This caused
random failures in the test suite on Mac OS X (triggered by the spurious
SIGPOLL signals on Mac OS X, which should also be fixed).
closure of the inputs. This really enforces that there can't be any
undeclared dependencies on paths in the store. This is done by
creating a fake Nix store and creating bind-mounts or hard-links in
the fake store for all paths in the closure. After the build, the
build output is moved from the fake store to the real store. TODO:
the chroot has to be on the same filesystem as the Nix store for
this to work, but this isn't enforced yet. (I.e. it only works
currently if /tmp is on the same FS as /nix/store.)
bind-mounts we do are only visible to the builder process and its
children. So accidentally doing "rm -rf" on the chroot directory
won't wipe out /nix/store and other bind-mounted directories
anymore. Also, the bind-mounts in the private namespace disappear
automatically when the builder exits.
necessary that at least one build hook doesn't return "postpone",
otherwise nix-store will barf ("waiting for a build slot, yet there
are no running children"). So inform the build hook when this is
the case, so that it can start a build even when that would exceed
the maximum load on a machine.
nix-store -r (or some other operation) is started via ssh, it will
at least have a chance of terminating quickly when the connection is
killed. Right now it just runs to completion, because it never
notices that stderr is no longer connected to anything. Of course
it would be better if sshd would just send a SIGHUP, but it doesn't
(https://bugzilla.mindrot.org/show_bug.cgi?id=396).
derivation should be a source rather than a derivation dependency of
the call to the NAR derivation. Otherwise the derivation (and all
its dependencies) will be built as a side-effect, which may not even
succeed.
SHA-256 outputs of fixed-output derivations. I.e. they now produce
the same store path:
$ nix-store --add x
/nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x
$ nix-store --add-fixed --recursive sha256 x
/nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x
the latter being the same as the path that a derivation
derivation {
name = "x";
outputHashAlgo = "sha256";
outputHashMode = "recursive";
outputHash = "...";
...
};
produces.
This does change the output path for such fixed-output derivations.
Fortunately they are quite rare. The most common use is fetchsvn
calls with SHA-256 hashes. (There are a handful of those is
Nixpkgs, mostly unstable development packages.)
* Documented the computation of store paths (in store-api.cc).
dependency. `storePath /nix/store/bla' gives exactly the same
result as `toPath /nix/store/bla', except that the former includes
/nix/store/bla in the dependency context of the string.
Useful in some generated Nix expressions like nix-push, which now
finally does the right thing wrt distributed builds. (Previously
the path to be packed wasn't an explicit dependency, so it wouldn't
be copied to the remote machine.)
subtle and often hard-to-reproduce bugs where programs in pipes
either barf with a "Broken pipe" message or not, depending on the
exact timing conditions. This particularly happened in GNU M4 (and
Bison, which uses M4).
disasters involving `rm -rf' on bind mounts. Will try the
definitive fix (per-process mounts, apparently possible via the
CLONE_NEWNS flag in clone()) some other time.
accessed time of paths that may be deleted. Anything more recently
used won't be deleted. The time is specified in time_t,
e.g. seconds since 1970-01-01 00:00:00 UTC; use `date +%s' to
convert to time_t from the command line.
Example: to delete everything that hasn't been used in the last two
months:
$ nix-store --gc -v --max-atime $(date +%s -d "2 months ago")
order of ascending last access time. This is useful in conjunction
with --max-freed or --max-links to prefer deleting non-recently used
garbage, which is good (especially in the build farm) since garbage
may become live again.
The code could easily be modified to accept other criteria for
ordering garbage by changing the comparison operator used by the
priority queue in collectGarbage().
attributes from the meta attribute. Not doing so caused nix-env to
barf on the "psi" package, which has a meta.function attribute,
the textual serialisation of which causes a gigantic string to be
produced --- so big that it causes nix-env to run out of memory.
Note however that "meta" really only should contain strings.
meta.function should be passthru.function.
particular, dietlibc cannot figure out the cwd because the inode of
the current directory doesn't appear in .. (because getdents returns
the inode of the mount point).
(which means it can only be defined via "inherit"), otherwise we get
scoping bugs, since __overrides can't be recursive (or at least, it
would be hard).
need /etc in the chroot (in particular, /etc/resolv.conf for
fetchurl). Not having /etc/resolv.conf in the chroot is a good
thing, since we don't want normal derivations to download files.
~/.nix-defexpr, otherwise the attribute cannot be selected with the
`-A' option. Useful if you want to stick a Nix expression directly
in ~/.nix-defexpr.
a rec. This will be very useful to allow end-user customisation of
all-packages.nix, for instance globally overriding GCC or some other
dependency. The // operator doesn't cut it: you could replace the
"gcc" attribute, but all other attributes would continue to
reference the original value due to the substitution semantics of
rec.
The syntax is a bit hacky but this is to allow backwards
compatibility.
in attribute set pattern matches. This allows defining a function
that takes *at least* the listed attributes, while ignoring
additional attributes. For instance,
{stdenv, fetchurl, fuse, ...}:
stdenv.mkDerivation {
...
};
defines a function that requires an attribute set that contains the
specified attributes but ignores others. The main advantage is that
we can then write in all-packages.nix
aefs = import ../bla/aefs pkgs;
instead of
aefs = import ../bla/aefs {
inherit stdenv fetchurl fuse;
};
This saves a lot of typing (not to mention not having to update
all-packages.nix with purely mechanical changes). It saves as much
typing as the "args: with args;" style, but has the advantage that
the function arguments are properly declared (not implicit in what
the body of the "with" uses).
functions that take a single argument (plain lambdas) into one AST
node (Function) that contains a Pattern node describing the
arguments. Current patterns are single lazy arguments (VarPat) and
matching against an attribute set (AttrsPat).
This refactoring allows other kinds of patterns to be added easily,
such as Haskell-style @-patterns, or list pattern matching.
`-u'. Instead of acquiring an exclusive lock on the profile for the
entire duration of the operation, we just perform the operation
optimistically (without an exclusive lock), and check at the end
whether the profile changed while we were busy (i.e., the symlink
target changed). If so, the operation is restarted. Restarting is
generally cheap, since the build results are still in the Nix store.
Most of the time, only the user environment has to be rebuilt.
again. (After the previous substituter mechanism refactoring I
didn't update the code that obtains the references of substitutable
paths.) This required some refactoring: the substituter programs
are now kept running and receive/respond to info requests via
stdin/stdout.
it only does something if $NIX_OTHER_STORES (not really a good
name...) is set.
* Do globbing on the elements of $NIX_OTHER_STORES. E.g. you could
set it to /mnts/*/nix or something.
* Install substituters in libexec/nix/substituters.
logic through the `parseDrvName' and `compareVersions' primops.
This will allow expressions to easily check whether some dependency
is a specific needed version or falls in some version range. See
tests/lang/eval-okay-versions.nix for examples.
bytes have been freed, `--max-links' to stop when the Nix store
directory has fewer than N hard links (the latter being important
for very large Nix stores on filesystems with a 32000 subdirectories
limit).
store under the reference relation, since that means that the
garbage collector will need a long time to start deleting paths.
Instead just delete the referrers of a path first.
This isn't usually a problem, except that it causes tests to fail
when performed in a directory with a very long path name. So chdir
to the socket directory and use a relative path name.
/tmp/nix-<pid>-<counter> for temporary build directories. This
increases purity a bit: many packages store the temporary build path
in their output, causing (generally unimportant) binary differences.
undefined variables by definition. This matters for the
implementation of "with", which does a call to checkVarDefs to see
if the body of the with has no undefined variables. (It can't be
checked at parse time because you don't know which variables are in
the "with" attribute set.) If we check closed terms, then we check
not just the with body but also the substituted terms, which are
typically very large. This is the cause of the poor nix-env
performance on Nixpkgs lately. It didn't happen earlier because
"with" wasn't used very often in the past.
This fix improves nix-env performance roughly 60x on current Nixpkgs.
nix-env -qa is down from 29.3s to 0.5s on my laptop, and nix-env -qa
--out-path is down from 229s to 3.39s. Not bad for a 1-line fix :-)
single quotes. Example (from NixOS):
job = ''
start on network-interfaces
start script
rm -f /var/run/opengl-driver
${if videoDriver == "nvidia"
then "ln -sf ${nvidiaDrivers} /var/run/opengl-driver"
else if cfg.driSupport
then "ln -sf ${mesa} /var/run/opengl-driver"
else ""
}
rm -f /var/log/slim.log
end script
'';
This style has two big advantages:
- \, ' and " aren't special, only '' and ${. So you get a lot less
escaping in shell scripts / configuration files in Nixpkgs/NixOS.
The delimiter '' is rare in scripts (and can usually be written as
""). ${ is also fairly rare.
Other delimiters such as <<...>>, {{...}} and <|...|> were also
considered but this one appears to have the fewest drawbacks
(thanks Martin).
- Indentation is intelligently stripped so that multi-line strings
can follow the nesting structure of the containing Nix
expression. E.g. in the example above 6 spaces are stripped from
the start of each line. This prevents unnecessary indentation in
generated files (which sometimes even breaks things).
See tests/lang/eval-okay-ind-string.nix for some examples.
$ nix-env -e $(which firefox)
or
$ nix-env -e /nix/store/nywzlygrkfcgz7dfmhm5xixlx1l0m60v-pan-0.132
* nix-env -i: if an argument contains a slash anywhere, treat it as a
path and follow it through symlinks into the Nix store. This allows
things like
$ nix-build -A firefox
$ nix-env -i ./result
* nix-env -q/-i/-e: don't complain when the `*' selector doesn't match
anything. In particular, `nix-env -q \*' doesn't fail anymore on an
empty profile.
