evaluator. This was important because the NixOS expressions started
to hit 2 MB default stack size on Linux.
GCC is really dumb about stack space: it just adds up all the local
variables and temporaries of every scope into one huge stack frame.
This is really bad for deeply recursive functions. For instance,
every `throw Error(format("error message"))' causes a format object
of a few hundred bytes to be allocated on the stack. As a result,
every recursive call to evalExpr2() consumed 4680 bytes. By
splitting evalExpr2() and by moving the exception-throwing code out
of the main functions, evalExpr2() now only consumes 40 bytes.
Similar for evalExpr().
which paths specified on the command line are invalid (i.e., don't
barf when encountering an invalid path, just print it). This is
useful for build-remote.pl to figure out which paths need to be
copied to a remote machine. (Currently we use rsync, but that's
rather inefficient.)
--export' into the Nix store, and optionally check the cryptographic
signatures against /nix/etc/nix/signing-key.pub. (TODO: verify
against a set of public keys.)
path. This is like `nix-store --dump', only it also dumps the
meta-information of the store path (references, deriver). Will add
a `--sign' flag later to add a cryptographic signature, which we
will use for exchanging store paths between build farm machines in a
secure manner.
attribute) about installed packages in user environments. Thus, an
operation like `nix-env -q --description' shows useful information
not only on available packages but also on installed packages.
* nix-env now passes the entire manifest as an argument to the Nix
expression of the user environment builder (not just a list of
paths), so that in particular the user environment builder has
access to the meta attributes.
* New operation `--set-flag' in nix-env to change meta info of
installed packages. This will be useful to pass per-package
policies to the user environment builder (e.g., how to resolve
collision or whether to disable a package (NIX-80)) or upgrade
policies in nix-env (e.g., that a package should be "masked", that
is, left untouched by upgrade actions). Example:
$ nix-env --set-flag enabled false ghc-6.4
computing the store path (NIX-77). This is an important security
property in multi-user Nix stores.
Note that this changes the store paths of derivations (since the
derivation aterms are added using addTextToStore), but not most
outputs (unless they use builtins.toFile).
* `sub' to subtract two numbers.
* `stringLength' to get the length of a string.
* `substring' to get a substring of a string. These should be enough
to allow most string operations to be expressed.
programs, so if a builder uses TMPDIR, then it will fail when
executed through nix-setuid-helper. In fact Glibc clears a whole
bunch of variables (see sysdeps/generic/unsecvars.h in the Glibc
sources), but only TMPDIR should matter in practice. As a
workaround, we reinitialise TMPDIR from NIX_BUILD_TOP.
important to get garbage collection to work if there is any
inconsistency in the database (because the referrer table is used to
determine whether it is safe to delete a path).
* `nix-store --verify': show some progress.
* nix-unpack-closure: extract the top-level paths from the closure and
print them on stdout. This allows them to be installed, e.g.,
"nix-env -i $(nix-unpack-closure)". (NIX-64)
<derivation outPath=... drvPath=...> attrs </derivation>. Only emit
the attributes of any specific derivation only. This prevents
exponententially large XML output due to the absense of sharing.
from a source directory. All files for which a predicate function
returns true are copied to the store. Typical example is to leave
out the .svn directory:
stdenv.mkDerivation {
...
src = builtins.filterSource
(path: baseNameOf (toString path) != ".svn")
./source-dir;
# as opposed to
# src = ./source-dir;
}
This is important because the .svn directory influences the hash in
a rather unpredictable and variable way.
single derivation specified by the argument. This is useful when we
want to have a profile for a single derivation, such as a server
configuration. Then we can just say (e.g.)
$ nix-env -p /.../server-profile -f server.nix --set -A server
We can't do queries or upgrades on such a profile, but we can do
rollbacks. The advantage over -i is that we don't have to worry
about other packages having been installed in the profile
previously; --set gets rid of them.
matters when running as root, since then we don't use the setuid
helper (which already used lchown()).
* Also check for an obscure security problem on platforms that don't
have lchown. Then we can't change the ownership of symlinks, which
doesn't matter *except* when the containing directory is writable by
the owner (which is the case with the top-level Nix store directory).
* Throw more exceptions as BuildErrors instead of Errors. This
matters when --keep-going is turned on. (A BuildError is caught
and terminates the goal in question, an Error terminates the
program.)
seconds without producing output on stdout or stderr (NIX-65). This
timeout can be specified using the `--max-silent-time' option or the
`build-max-silent-time' configuration setting. The default is
infinity (0).
* Fix a tricky race condition: if we kill the build user before the
child has done its setuid() to the build user uid, then it won't be
killed, and we'll potentially lock up in pid.wait(). So also send a
conventional kill to the child.
that have to be done as root: running builders under different uids,
changing ownership of build results, and deleting paths in the store
with the wrong ownership).
`nix-store --delete'. But unprivileged users are not allowed to
ignore liveness.
* `nix-store --delete --ignore-liveness': ignore the runtime roots as
well.
process, so forward the operation.
* Spam the user about GC misconfigurations (NIX-71).
* findRoots: skip all roots that are unreadable - the warnings with
which we spam the user should be enough.
processes can register indirect roots. Of course, there is still
the problem that the garbage collector can only read the targets of
the indirect roots when it's running as root...
* SIGIO -> SIGPOLL (POSIX calls it that).
* Use sigaction instead of signal to register the SIGPOLL handler.
Sigaction is better defined, and a handler registered with signal
appears not to interrupt fcntl(..., F_SETLKW, ...), which is bad.
via the Unix domain socket in /nix/var/nix/daemon.socket. The
server forks a worker process per connection.
* readString(): use the heap, not the stack.
* Some protocol fixes.
* Added `build-users-group', the group under which builds are to be
performed.
* Check that /nix/store has 1775 permission and is owner by the
build-users-group.
The problem is that when we kill the client while the worker is
building, and the builder is not writing anything to stderr, then
the worker never notice that the socket is closed on the other side,
so it just continues indefinitely. The solution is to catch SIGIO,
which is sent when the far side of the socket closes, and simulate
an normal interruption. Of course, SIGIO is also sent every time
the client sends data over the socket, so we only enable the signal
handler when we're not expecting any data...
mode. Presumably nix-worker would be setuid to the Nix store user.
The worker performs all operations on the Nix store and database, so
the caller can be completely unprivileged.
This is already much more secure than the old setuid scheme, since
the worker doesn't need to do Nix expression evaluation and so on.
Most importantly, this means that it doesn't need to access any user
files, with all resulting security risks; it only performs pure
store operations.
Once this works, it is easy to move to a daemon model that forks off
a worker for connections established through a Unix domain socket.
That would be even more secure.
* Some refactoring: put the NAR archive integer/string serialisation
code in a separate file so it can be reused by the worker protocol
implementation.
containing functions that operate on the Nix store. One
implementation is LocalStore, which operates on the Nix store
directly. The next step, to enable secure multi-user Nix, is to
create a different implementation RemoteStore that talks to a
privileged daemon process that uses LocalStore to perform the actual
operations.
Rather, setuid support is now always compiled in (at least on
platforms that have the setresuid system call, e.g., Linux and
FreeBSD), but it must enabled by chowning/chmodding the Nix
binaries.
gives a huge speedup in operations that read or write from standard
input/output. (So libstdc++'s I/O isn't that bad, you just have to
call std::ios::sync_with_stdio(false).) For instance, `nix-store
--register-substitutes' went from 1.4 seconds to 0.1 seconds on a
certain input. Another victory for Valgrind.
graph to be passed to a builder. This attribute should be a list of
pairs [name1 path1 name2 path2 ...]. The references graph of each
`pathN' will be stored in a text file `nameN' in the temporary build
directory. The text files have the format used by `nix-store
--register-validity'. However, the deriver fields are left empty.
`exportReferencesGraph' is useful for builders that want to do
something with the closure of a store path. Examples: the builders
that make initrds and ISO images for NixOS.
`exportReferencesGraph' is entirely pure. It's necessary because
otherwise the only way for a builder to get this information would
be to call `nix-store' directly, which is not allowed (though
unfortunately possible).
available. For instance,
$ nix-store -l $(which svn) | less
lets you read the build log of the Subversion instance in your
profile.
* `nix-store -qb': if applied to a non-derivation, take the deriver.
check that the references of the output of a derivation are in the
specified set. For instance,
allowedReferences = [];
specifies that the output cannot have any references. (This is
useful, for instance, for the generation of bootstrap binaries for
stdenv-linux, which must not have any references for purity). It
could also be used to guard against undesired runtime dependencies,
e.g.,
{gcc, dynlib}: derivation {
...
allowedReferences = [dynlib];
}
says that the output can refer to the path of `dynlib' but not
`gcc'. A `forbiddedReferences' attribute would be more useful for
this, though.
concatenation and string coercion. This was a big mess (see
e.g. NIX-67). Contexts are now folded into strings, so that they
don't cause evaluation errors when they're not expected. The
semantics of paths has been clarified (see nixexpr-ast.def).
toString() and coerceToString() have been merged.
Semantic change: paths are now copied to the store when they're in a
concatenation (and in most other situations - that's the
formalisation of the meaning of a path). So
"foo " + ./bla
evaluates to "foo /nix/store/hash...-bla", not "foo
/path/to/current-dir/bla". This prevents accidental impurities, and
is more consistent with the treatment of derivation outputs, e.g.,
`"foo " + bla' where `bla' is a derivation. (Here `bla' would be
replaced by the output path of `bla'.)
side should be a path, I guess.
* Handle paths that are in the store but not direct children of the
store directory.
* Ugh, hack to prevent double context wrapping.
attribute existence and to return an attribute from an attribute
set, respectively. Example: `hasAttr "foo" {foo = 1;}'. They
differ from the `?' and `.' operators in that the attribute name is
an arbitrary expression. (NIX-61)
Nix-env failed to call addPermRoot(), which is necessary to safely
add a new root. So if nix-env started after and finished before the
garbage collector, the user environment (plus all other new stuff)
it built might be garbage collected, leading to a dangling symlink
chain in ~/.nix-profile...
* Be more explicit if we block on the GC lock ("waiting for the big
garbage collector lock...").
* Don't loop trying to create a new generation. It's not necessary
anymore since profiles are locked nowadays.
and returns its path. This can be used to (for instance) write
builders inside a Nix expression, e.g.,
stdenv.mkDerivation {
builder = "
source $stdenv/setup
...
";
...
}
derivation attributes to flatten them into strings. This is
possible since string can nowadays be wrapped in contexts that
describe the derivations/sources referenced by the evaluation of the
string.
all the primops. This allows Nix expressions to test for new
primops and take appropriate action if they're not available. For
instance, rather than calling a primop `foo' directly, they could
say `if builtins ? foo then builtins.foo ... else ...'.
writable. File permissions on Cygwin are rather complex, and in this
case this check introduced a problem with build jobs invoke from
outside of Cygwin (MSYS). It seemed almost impossible to fix the
permissions of the directory, so for now this safety check is disabled
on Cygwin.
https://svn.cs.uu.nl:12443/repos/trace/buildfarm-control/trunk/ext/nix/,
with some modifications. This allows `nix-env -qa' to show the
attribute path that can be used to unambiguously install a package
using `nix-env -i -A'. Example:
$ nix-env -f top-level/all-packages.nix -qaA subversion xorg-server
subversionWithJava subversion-1.2.3
subversion subversion-1.3.2
subversion14 subversion-1.4.0pre-rc1
xorg.xorgserver xorg-server-1.1.0
e.g.,
$ nix-env -i -A subversion xorg.xorgserver
The main advantage over using symbolic names is that using attribute
names is unambiguous and much, much faster.
argument has a valid value, i.e., is in a certain domain. E.g.,
{ foo : [true false]
, bar : ["a" "b" "c"]
}: ...
This previously could be done using assertions, but domain checks
will allow the buildfarm to automatically extract the configuration
space from functions.
"--with-freetype2-library=" + freetype + "/lib"
can now be written as
"--with-freetype2-library=${freetype}/lib"
An arbitrary expression can be enclosed within ${...}, not just
identifiers.
* Escaping in string literals: \n, \r, \t interpreted as in C, any
other character following \ is interpreted as-is.
* Newlines are now allowed in string literals.
packages (provided that they have a `meta.description' attribute).
E.g.,
$ ./src/nix-env/nix-env -qa --description gcc
gcc-4.0.2 GNU Compiler Collection, 4.0.x (cross-compiler for sparc-linux)
gcc-4.0.2 GNU Compiler Collection, 4.0.x (cross-compiler for mips-linux)
gcc-4.0.2 GNU Compiler Collection, 4.0.x (cross-compiler for arm-linux)
gcc-4.0.2 GNU Compiler Collection, 4.0.x
to be queried, e.g., `nix-env -qa firefox'. This does require the
argument '*' to be passed if one wants information about all
derivations, so the old `nix-env -qa' now is `nix-env -qa "*"'.
instantiation, e.g. "nix-env -i" and "nix-env -qas" (but not
"nix-env -qa"). It turns out that many redundant calls to
addToStore(path) were made, which reads and hashes the entire path.
For instance, the bash bootstrap binary in Nixpkgs would be read and
hashed many times. As a result nix-env would spend around 92% of
its time in the function sha256_block (according to callgrind).
Some simple memoization fixes this.
expressions that cause an assertion failure (like `assert system ==
"i686-linux"'). This allows all-packages.nix in Nixpkgs to be used
on all platforms, even if some Nix expressions don't work on all
platforms.
Not sure if this is a good idea; it's a bit hacky. In particular,
due to laziness some derivations might appear in `nix-env -qa' but
disappear in `nix-env -qas' or `nix-env -i'.
Commit 5000!
with the same name *and* version number, and pick the first one
(this means that the order in which channels appear in
~/.nix-channels matters). E.g.:
$ nix-env ii aterm
warning: there are multiple derivations named `aterm-2.4.2'; using the first one
installing `aterm-2.4.2'
the disk is full (because to delete something from the Nix store, we
need a Berkeley DB transaction, which takes up disk space). Under
normal operation, we make sure that there exists a file
/nix/var/nix/db/reserved of 1 MB. When running the garbage
collector, we delete that file before we open the Berkeley DB
environment.
implementations of MD5, SHA-1 and SHA-256. The main benefit is that
we get assembler-optimised implementations of MD5 and SHA-1 (though
not SHA-256 (at least on x86), unfortunately). OpenSSL's SHA-1
implementation on Intel is twice as fast as ours.
derivation(s) we're interested, e.g.,
$ nix-instantiate ./all-packages.nix --attr xlibs.libX11
List elements can also be selected:
$ nix-instantiate ./build-for-release.nix --attr 0.subversion
This allows a non-ambiguous specification of a derivation. Of
course, this should also be added to nix-env and nix-build.
creates a new process group but also a new session. New sessions
have no controlling tty, so child processes like ssh cannot open
/dev/tty (which is bad).
intended). This ensures that any ssh child processes to remote
machines are also killed, and thus the Nix process on the remote
machine also exits. Without this, the remote Nix process will
continue until it exists or until its stdout buffer gets full and it
locks up. (Partially fixes NIX-35.)
deletes a path even if it is reachable from a root. However, it
won't delete a path that still has referrers (since that would
violate store invariants).
Don't try this at home. It's a useful hack for recovering from
certain situations in a somewhat clean way (e.g., holes in closures
due to disk corruption).
nix-store query options `--referer' and `--referer-closure' have
been changed to `--referrer' and `--referrer-closure' (but the old
ones are still accepted for compatibility).
mapping. The referer table is replaced by a referrer table (note
spelling fix) that stores each referrer separately. That is,
instead of having
referer[P] = {Q_1, Q_2, Q_3, ...}
we store
referer[(P, Q_1)] = ""
referer[(P, Q_2)] = ""
referer[(P, Q_3)] = ""
...
To find the referrers of P, we enumerate over the keys with a value
lexicographically greater than P. This requires the referrer table
to be stored as a B-Tree rather than a hash table.
(The tuples (P, Q) are stored as P + null-byte + Q.)
Old Nix databases are upgraded automatically to the new schema.
Nix is properly shut down when it receives those signals. In
particular this ensures that killing the garbage collector doesn't
cause a subsequent database recovery.
builder. Instead, require that the Nix store has sticky permission
(S_ISVTX); everyone can created files in the Nix store, but they
cannot delete, rename or modify files created by others.
root (or setuid root), then builds will be performed under one of
the users listed in the `build-users' configuration variables. This
is to make it impossible to influence build results externally,
allowing locally built derivations to be shared safely between
users (see ASE-2005 paper).
To do: only one builder should be active per build user.
versions to available versions, or vice versa.
For example, the following compares installed versions to available
versions:
$ nix-env -qc
autoconf-2.59 = 2.59
automake-1.9.4 < 1.9.6
f-spot-0.0.10 - ?
firefox-1.0.4 < 1.0.7
...
I.e., there are newer versions available (in the current default Nix
expression) for Automake and Firefox, but not for Autoconf, and
F-Spot is missing altogether.
Conversely, the available versions can be compared to the installed
versions:
$ nix-env -qac
autoconf-2.59 = 2.59
automake-1.9.6 > 1.9.4
bash-3.0 - ?
firefox-1.0.7 > 1.0.4
...
Note that bash is available but no version of it is installed.
If multiple versions are available for comparison, then the highest
is used. E.g., if Subversion 1.2.0 is installed, and Subversion
1.1.4 and 1.2.3 are available, then `nix-env -qc' will print `<
1.2.3', not `> 1.1.4'.
If higher versions are available, the version column is printed in
red (using ANSI escape codes).
dependencyClosure { ... searchPath = [ ../foo ../bar ]; ... }
* Primop `dirOf' to return the directory part of a path (e.g., dirOf
/a/b/c == /a/b).
* Primop `relativise' (according to Webster that's a real word!) that
given paths A and B returns a string representing path B relative
path to A; e.g., relativise /a/b/c a/b/x/y => "../x/y".
determination (e.g., finding the header files dependencies of a C
file) in Nix low-level builds automatically.
For instance, in the function `compileC' in make/lib/default.nix, we
find the header file dependencies of C file `main' as follows:
localIncludes =
dependencyClosure {
scanner = file:
import (findIncludes {
inherit file;
});
startSet = [main];
};
The function works by "growing" the set of dependencies, starting
with the set `startSet', and calling the function `scanner' for each
file to get its dependencies (which should yield a list of strings
representing relative paths). For instance, when `scanner' is
called on a file `foo.c' that includes the line
#include "../bar/fnord.h"
then `scanner' should yield ["../bar/fnord.h"]. This list of
dependencies is absolutised relative to the including file and added
to the set of dependencies. The process continues until no more
dependencies are found (hence its a closure).
`dependencyClosure' yields a list that contains in alternation a
dependency, and its relative path to the directory of the start
file, e.g.,
[ /bla/bla/foo.c
"foo.c"
/bla/bar/fnord.h
"../bar/fnord.h"
]
These relative paths are necessary for the builder that compiles
foo.c to reconstruct the relative directory structure expected by
foo.c.
The advantage of `dependencyClosure' over the old approach (using
the impure `__currentTime') is that it's completely pure, and more
efficient because it only rescans for dependencies (i.e., by
building the derivations yielded by `scanner') if sources have
actually changed. The old approach rescanned every time.
(closed(closed(closed(...)))) since this reduces performance by
producing bigger terms and killing caching (which incidentally also
prevents useful infinite recursion detection).
with default values automatically. I.e., e -> e {}.
This feature makes convenience expressions such as
pkgs/system/i686-linux.nix in Nixpkgs obsolete, since we can just do
$ nix-instantiate ./pkgs/system/all-packages.nix
since all-packages.nix takes a single argument (system) that has a
default value (__thisSystem).
`removeAttrs attrs ["x", "y"]' returns the set `attrs' with the
attributes named `x' and `y' removed. It is not an error for the
named attributes to be missing from the input set.
* Make the `derivation' primitive much more lazy. The expression
`derivation attrs' now evaluates to (essentially)
attrs // {
type = "derivation";
outPath = derivation! attrs;
drvPath = derivation! attrs;
}
where `derivation!' is a primop that does the actual derivation
instantiation (i.e., it does what `derivation' used to do). The
advantage is that it allows commands such as `nix-env -qa' and
`nix-env -i' to be much faster since they no longer need to
instantiate all derivations, just the `name' attribute. (However,
`nix-env' doesn't yet take advantage of this since it still always
evaluates the `outPath' and `drvPath' attributes).
Also, this allows derivations to cyclically reference each other,
for example,
webServer = derivation {
...
hostName = "svn.cs.uu.nl";
services = [svnService];
};
svnService = derivation {
...
hostName = webServer.hostName;
};
Previously, this would yield a black hole (infinite recursion).
derivations. This is mostly to simplify the implementation of
nix-prefetch-{url, svn}, which now work properly in setuid
installations.
* Enforce valid store names in `nix-store --add / --add-fixed'.
continue building when one fails unless `--keep-going' is
specified.
* When `--keep-going' is specified, print out the set of failing
derivations at the end (otherwise it can be hard to find out which
failed).
multiple times is also a top-level goal, then the second and later
instantiations would never be created because there would be a
stable pointer to the first one that would keep it alive in the
WeakGoalMap.
* Some tracing code for debugging this kind of problem.
of the given derivation. Useful for getting a quick overview of how
something was built. E.g., to find out how the `baffle' program in
your user environment was built, you can do
$ nix-store -q --tree $(nix-store -qd $(which baffle))
Tree nesting depth is minimised (?) by topologically sorting paths
under the relation A < B iff A \in closure(B).
environment elements from one user environment to another, e.g.,
$ nix-env -i --from-profile /nix/var/nix/profiles/other-profile aterm
copies the `aterm' component installed in the `other-profile' to the
user's current profile.
user environment, e.g.,
$ nix-env -i /nix/store/z58v41v21xd3ywrqk1vmvdwlagjx7f10-aterm-2.3.1.drv
or
$ nix-env -i /nix/store/hsyj5pbn0d9iz7q0aj0fga7cpaadvp1l-aterm-2.3.1
This is useful because it allows Nix expressions to be bypassed
entirely. For instance, if only a nix-pull manifest is provided,
plus the top-level path of some component, it can be installed
without having to supply the Nix expression (e.g., for obfuscation,
or to be independent of Nix expression language changes or context
dependencies).
install derivations from a Nix expression specified on the command
line. This is particularly useful for disambiguation if there are
multiple derivations with the same name. For instance, in Nixpkgs,
to install the Firefox wrapper rather than the plain Firefox
component:
$ nix-env -f .../i686-linux.nix -i -E 'x: x.firefoxWrapper'
The Nix expressions should be functions to which the default Nix
expression (in this case, `i686-linux.nix') is passed, hence `x:
...'.
This might also be a nice way to deal with high-level (user-level)
variability, e.g.,
$ nix-env -f ./server.nix -i -E 'x: x {port = 8080; ssl = false;}'
to derivations in user environments. Nice for developers (since it
prevents build-time-only dependencies from being GC'ed, in
conjunction with `gc-keep-outputs'). Turned off by default.
* Set the references for the user environment manifest properly.
* Don't copy the manifest (this was accidental).
* Don't store derivation paths in the manifest (maybe this should be
made optional). This cleans up the semantics of nix-env, which were
weird.
* Hash on the output paths of activated components, not on derivation
paths. This is because we don't know the derivation path of already
installed components anymore, and it allows the installation of
components by store path (skipping Nix expressions entirely).
* Query options `--out-path' and `--drv-path' to show the output and
derivation paths of components, respectively (the latter replaces
the `--expr' query).
* Removed some dead code (successor stuff) from nix-push.
* Updated terminology in the tests (store expr -> drv path).
* Check that the deriver is set properly in the tests.
for finding build-time dependencies (possibly after a build). E.g.,
$ nix-store -qb aterm $(nix-store -qd $(which strc))
/nix/store/jw7c7s65n1gwhxpn35j9rgcci6ilzxym-aterm-2.3.1
* Arguments to nix-store can be files within store objects, e.g.,
/nix/store/jw7c...-aterm-2.3.1/bin/baffle.
* Idem for garbage collector roots.
This was necessary becase root finding must be done after
acquisition of the global GC lock.
This makes `nix-collect-garbage' obsolete; it is now just a wrapper
around `nix-store --gc'.
* Automatically remove stale GC roots (i.e., indirect GC roots that
point to non-existent paths).
get rid of GC roots. Nix-build places a symlink `result' in the
current directory. Previously, removing that symlink would not
remove the store path being linked to as a GC root. Now, the GC
root created by nix-build is actually a symlink in
`/nix/var/nix/gcroots/auto' to `result'. So if that symlink is
removed the GC root automatically becomes invalid (since it can no
longer be resolved). The root itself is not automatically removed -
the garbage collector should delete dangling roots.
