improve throughput.
* Don't build the `substitute-rev' table for now, since it caused
Theta(N^2) time and log file consumption when adding N substitutes.
Maybe we can do without it.
* A better substitute mechanism.
Instead of generating a store expression for each store path for
which we have a substitute, we can have a single store expression
that builds a generic program that is invoked to build the desired
store path, which is passed as an argument.
This means that operations like `nix-pull' only produce O(1) files
instead of O(N) files in the store when registering N substitutes.
(It consumes O(N) database storage, of course, but that's not a
performance problem).
* Added a test for the substitute mechanism.
* `nix-store --substitute' reads the substitutes from standard input,
instead of from the command line. This prevents us from running
into the kernel's limit on command line length.
* When a fast build wakes up a goal, try to start that goal in the
same iteration of the startBuild() loop of run(). Otherwise no job
might be started until the next job terminates.
in parallel. Hooks are more efficient: locks on output paths are
only acquired when the hook says that it is willing to accept a
build job. Hooks now work in two phases. First, they should first
tell Nix whether they are willing to accept a job. Nix guarantuees
that no two hooks will ever be in the first phase at the same time
(this simplifies the implementation of hooks, since they don't have
to perform locking (?)). Second, if they accept a job, they are
then responsible for building it (on the remote system), and copying
the result back. These can be run in parallel with other hooks and
locally executed jobs.
The implementation is a bit messy right now, though.
* The directory `distributed' shows a (hacky) example of a hook that
distributes build jobs over a set of machines listed in a
configuration file.
distributing a build action to another machine. In particular, the
paths in the input closures, the output paths, and successor mapping
for sub-derivations.
parallel as possible (similar to GNU Make's `-j' switch). This is
useful on SMP systems, but it is especially useful for doing builds
on multiple machines. The idea is that a large derivation is
initiated on one master machine, which then distributes
sub-derivations to any number of slave machines. This should not
happen synchronously or in lock-step, so the master must be capable
of dealing with multiple parallel build jobs. We now have the
infrastructure to support this.
TODO: substitutes are currently broken.
print a nice backtrace of the stack, rather than vomiting a gigantic
(and useless) aterm on the screen. Example:
error: while evaluating file `.../pkgs/system/test.nix':
while evaluating attribute `subversion' at `.../pkgs/system/all-packages-generic.nix', line 533:
while evaluating function at `.../pkgs/applications/version-management/subversion/default.nix', line 1:
assertion failed at `.../pkgs/applications/version-management/subversion/default.nix', line 13
Since the Nix expression language is lazy, the trace may be
misleading. The purpose is to provide a hint as to the location of
the problem.
instead of `derivation' triggered a huge slowdown in the Nix
expression evaluator. Total execution time of `nix-env -qa' went up
by a factor of 60 or so.
This scalability problem was caused by expressions such as
(x: y: ... x ...) a b
where `a' is a large term (say, the one in
`all-packages-generic.nix'). Then the first beta-reduction would
produce
(y: ... a ...) b
by substituting `a' for `x'. The second beta-reduction would then
substitute `b' for `y' into the body `... a ...', which is a large
term due to `a', and thus causes a large traversal to be performed
by substitute() in the second reduction. This is however entirely
redundant, since `a' cannot contain free variables (since we never
substitute below a weak head normal form).
The solution is to wrap substituted terms into a `Closed'
constructor, i.e.,
subst(subs, Var(x)) = Closed(e) iff subs[x] = e
have substitution not descent into closed terms,
subst(subs, Closed(x)) = Closed(x)
and otherwise ignore them for evaluation,
eval(Closed(x)) = eval(x).
* Fix a typo that caused incorrect substitutions to be performed in
simple lambdas, e.g., `(x: x: x) a' would reduce to `(x: a)'.
`bla:' is now no longer parsed as a URL.
* Re-enabled support for the `args' attribute in derivations to
specify command line arguments to the builder, e.g.,
...
builder = /usr/bin/python;
args = ["-c" ./builder.py];
...
This is because the contents of these symlinks are not incorporated
into the hashes of derivations, and could therefore cause a mismatch
between the build system and the target system. E.g., if
`/nix/store' is a symlink to `/data/nix/store', then a builder could
expand this path and store the result. If on the target system
`/nix/store' is not a symlink, or is a symlink that points somewhere
else, we have a dangling pointer.
The trigger for this change is that gcc 3.3.3 does exactly that (it
applies realpath() to some files, such as libraries, which causes
our impurity checker to bail out.)
An annoying side-effect of this change is that it makes it harder to
move the Nix store to a different file system. On Linux, bind
mounts can be used instead of symlink for this purpose (e.g., `mount
-o bind /data/nix/store /nix/store').
writes to stderr:
- `pretty': the old nested style (default)
- `escapes': uses escape codes to indicate nesting and message
level; can be processed using `log2xml'
- `flat': just plain text, no nesting
These can be set using `--log-type TYPE' or the NIX_LOG_TYPE
environment variable.
unimportant messages, it is collapsed by the default.
* Also added an optional integer argument to the escape code for opening a nesting
level to indicate lack of importance. If set, the tree is collapsed by default.
build logs. The program `log2xml' converts a Nix build log (read
from standard input) into XML file that can then be converted to
XHTML by the `log2html.xsl' stylesheet. The CSS stylesheet
`logfile.css' is necessary to make it look good.
This is primarily useful if the log file has a *tree structure*,
i.e., that sub-tasks such as the various phases of a build (unpack,
configure, make, etc.) or recursive invocations of Make are
represented as such. While a log file is in principle an
unstructured plain text file, builders can communicate this tree
structure to `log2xml' by using escape sequences:
- "\e[p" starts a new nesting level; the first line following the
escape code is the header;
- "\e[q" ends the current nesting level.
The generic builder in nixpkgs (not yet committed) uses this. It
shouldn't be to hard to patch GNU Make to speak this protocol.
Further improvements to the generated HTML pages are to allow
collapsing/expanding of subtrees, and to abbreviate store paths (but
to show the full path by hovering the mouse over it).
builders to point to the store and the temporary build directory,
respectively. Useful for purity checking.
* Also set TEMPDIR, TMPDIR, TEMP, and TEMP to NIX_BUILD_TOP to make
sure that tools in the builder store temporary files in the right
location.
* Do not create stuff in localstatedir when doing `make install'
(since we may not have write access). In general, installation of
constant code/data should be separate from the initialisation of
mutable state.
chroot() environment.
* A operation `--validpath' to register path validity. Useful for
bootstrapping in a pure Nix environment.
* Safety checks: ensure that files involved in store operations are in
the store.
derivation (i.e., the closure store expression) a root of the
garbage collector. This ensures that running `nix-collect-garbage
--no-successors' is safe.
whether we want to upgrade if the current version is less than the
available version (default), when it is less or equal, or always.
* Added a flag `--dry-run' to show what would happen in `--install',
`--uninstall', and `--upgrade', without actually performing the
operation.
of the current profile, e.g.,
$ nix-env --list-generations
...
39 2004-02-02 17:53:53
40 2004-02-02 17:55:18
41 2004-02-02 17:55:41
42 2004-02-02 17:55:50 (current)
$ nix-env --switch-generation 39
$ ls -l /nix/var/nix/profiles/default
... default -> default-39-link
* Also a command `--rollback' which is just a convenience operation to
rollback to the oldest generation younger than the current one.
Note that generations properly form a tree. E.g., if after
switching to generation 39, we perform an installation action,
a generation 43 is created which is a descendant of 39, not 42. So
a rollback from 43 ought to go back to 39. This is not currently
implemented; generations form a linear sequence.
default -> default-94-link
default-82-link -> /nix/store/cc4480...
default-83-link -> /nix/store/caeec8...
...
default-94-link -> /nix/store/2896ca...
experimental -> experimental-2-link
experimental-1-link -> /nix/store/cc4480...
experimental-2-link -> /nix/store/a3148f...
* `--profile' / `-p' -> `--switch-profile' / `-S'
* `--link' / `-l' -> `--profile' / `-p'
* The default profile is stored in $prefix/var/nix/profiles.
$prefix/var/nix/links is gone. Profiles can be stored anywhere.
* The current profile is now referenced from ~/.nix-profile, not
~/.nix-userenv.
* The roots to the garbage collector now have extension `.gcroot', not
`.id'.
other attribute sets, rather than the current scope. E.g.,
{inherit (pkgs) gcc binutils;}
is equivalent to
{gcc = pkgs.gcc; binutils = pkgs.binutils;}
I am not so happy about the syntax.
parser (roughly 80x faster).
The absolutely latest version of Bison (1.875c) is required for
reentrant GLR support, as well as a recent version of Flex (say,
2.5.31). Note that most Unix distributions ship with the
prehistoric Flex 2.5.4, which doesn't support reentrancy.
Nix. This is to prevent Berkeley DB from becoming wedged.
Unfortunately it is not possible to throw C++ exceptions from a
signal handler. In fact, you can't do much of anything except
change variables of type `volatile sig_atomic_t'. So we set an
interrupt flag in the signal handler and check it at various
strategic locations in the code (by calling checkInterrupt()).
Since this is unlikely to cover all cases (e.g., (semi-)infinite
loops), sometimes SIGTERM may now be required to kill Nix.
the output path of a derivation, not the path of its store
expression. This ensures that changes that affect the path of the
store expression but not the output path, do not affect the
`installed' state of a derivation.
it automatically removes log files when they are no longer needed.
*** IMPORTANT ***
If you have an existing Nix installation, you must checkpoint the
Nix database to prevent recent transactions from being undone. Do
the following:
- optional: make a backup of $prefix/var/nix/db.
- run `db_checkpoint' from Berkeley DB 4.1:
$ db_checkpoint -h $prefix/var/nix/db -1
- optional (?): run `db_recover' from Berkeley DB 4.1:
$ db_recover -h $prefix/var/nix/db
- remove $prefix/var/nix/db/log* and $prefix/var/nix/db/__db*
path of the Nix expression to be used with the import, upgrade, and
query commands. For instance,
$ nix-env -I ~/nixpkgs/pkgs/system/i686-linux.nix
$ nix-env --query --available [aka -qa]
sylpheed-0.9.7
bison-1.875
pango-1.2.5
subversion-0.35.1
...
$ nix-env -i sylpheed
$ nix-env -u subversion
There can be only one default at a time.
* If the path to a Nix expression is a symlink, follow the symlink
prior to resolving relative path references in the expression.
the symlink ~/.nix-userenv to the given argument (which defaults to
.../links/current). /etc/profile.d/nix-profile creates this symlink
if it doesn't exist yet. Example use:
$ nix-env -l my_profile -i foo.nix subversion quake
$ nix-env -p my_profile
I don't like the term "profile". Let's deprecate it :-)
nix-env -u foo.nix strategoxt
to replace the installed `strategoxt' derivation with the one from `foo.nix', if
the latter has a higher version number. This is a no-op if `strategoxt' is not
installed. Wildcards are also accepted, so
nix-env -u foo.nix '*'
will replace any installed derivation with newer versions from `foo.nix', if
available.
The notion of "version number" is somewhat ad hoc, but should be useful in most
cases, as evidenced by the following unit tests for the version comparator:
TEST("1.0", "2.3", -1);
TEST("2.1", "2.3", -1);
TEST("2.3", "2.3", 0);
TEST("2.5", "2.3", 1);
TEST("3.1", "2.3", 1);
TEST("2.3.1", "2.3", 1);
TEST("2.3.1", "2.3a", 1);
TEST("2.3pre1", "2.3", -1);
TEST("2.3pre3", "2.3pre12", -1);
TEST("2.3a", "2.3c", -1);
TEST("2.3pre1", "2.3c", -1);
TEST("2.3pre1", "2.3q", -1);
(-1 = less, 0 = equal, 1 = greater)
* A new verbosity level `lvlInfo', between `lvlError' and `lvlTalkative'. This is
the default for `nix-env', so without any `-v' flags users should get useful
output, e.g.,
$ nix-env -u foo.nix strategoxt
upgrading `strategoxt-0.9.2' to `strategoxt-0.9.3'
turned out to be a huge performance bottleneck (the text to printed
would always be evaluated, even when it was above the verbosity
level). This reduces fix-ng execution time by over 50%.
gprof(1) is very useful. :-)
The ATerm library doesn't search the heap for pointers to ATerms
when garbage collecting. As a result, C++ containers such as
`map<ATerm, ATerm>' will cause pointer to be hidden from the garbage
collector, causing crashes. Instead, we now use ATermTables.
writes stdout/stderr of the builder to ${prefix}/var/log/nix/x,
where x is the file name of the derivation expression, e.g.,
/nix/var/log/nix/54256391624be04fcb426048ae3ea0a4-d-pan-0.14.2.nix
Note that consecutive builds of the same expression overwrite,
rather than append to, existing log files.
* Fixed a segfault caused by the buffering of stderr.
* Fix now allows the specification of the full output path. This
should be used with great care, since it by-passes the normal hash
generation.
* Incremented the version number to 0.4 (prerelease).
log on commit. This means that there is a small change that some
transactions may be rolled back in case of a system crash, but this
should not be a problem (it merely might cause some expression
realisations to be rolled back), and it vastly improves performance.
* Upgraded to ATerm 2.0.5 (which also includes Armijn's 64-bit
patches).
* Point $HOME to a non-existing path when building to prevent certain tools (such as
wget) from falling back on /etc/passwd to locate the home directory (which we
don't want them to look at since it's not declared as an input).
keys to reference slice elements, e.g.,
Slice(["1ef7..."], [("/nix/store/1ef7...-foo", "1ef7", ["8c99..."]), ...])
This was wrong, since ids represent contents, not locations. Therefore we
now have:
Slice(["/nix/store/1ef7..."], [("/nix/store/1ef7...-foo", "1ef7", ["/nix/store/8c99-..."]), ...])
* Fix a bug in the computation of slice closures that could cause slice
elements to be duplicated.
("srcs", [Relative("foo/bar.c"), Relative("foo/baz.h")])
The result is an environment variable that contains the path names of the
inputs separated by spaces (so this is not safe for values containing
spaces).
builder using the `args' binding:
("args", ["bla", True, IncludeFix("aterm/aterm.fix")])
Note that packages can also be declared as inputs by specifying them
in the argument list.
process is already holding a lock on a path, it may acquire the lock
again without blocking or failing). (This might be dangerous, not
sure). Necessary for fast builds to work.
normal form in a single transaction to ensure that if we crash,
either everything is registered or nothing is. This is for
recoverability: unregistered paths in the store can be deleted
arbitrarily, while registered paths can only be deleted by running
the garbage collector.
* Open all database tables (Db objects) at initialisation time, not
every time they are used. This is necessary because tables have to
outlive all transactions that refer to them.
Renamed `fstateRefs' to `fstateRequisites'. The semantics of this
function is that it returns a list of all paths necessary to realise
a given expression. For a derive expression, this is the union of
requisites of the inputs; for a slice expression, it is the path of
each element in the slice. Also included are the paths of the
expressions themselves. Optionally, one can also include the
requisites of successor expressions (to recycle intermediate
results).
* `nix-switch' now distinguishes between an expression and its normal
form. Usually, only the normal form is registered as a root of the
garbage collector. With the `--source-root' flag, it will also
register the original expression as a root.
* `nix-collect-garbage' now has a flag `--keep-successors' which
causes successors not to be included in the list of garbage paths.
* `nix-collect-garbage' now has a flag `--invert' which will print all
paths that should *not* be garbage collected.
up to the given verbosity levels. These currently are:
lvlError = 0,
lvlNormal = 5,
lvlDebug = 10,
lvlDebugMore = 15
although only lvlError and lvlDebug are actually used right now.
substituting for (obvious, really).
* For greater efficiency, nix-pull/unnar will place the output in a
path that is probably the same as what is actually needed, thus
preventing a path copy.
* Even if a output id is given in a Fix package expression, ensure
that the resulting Nix derive expression has a different id. This
is because Nix expressions that are semantically equivalent (i.e.,
build the same result) might be different w.r.t. efficiency or
divergence. It is absolutely vital for the substitute mechanism
that such expressions are not used interchangeably.
value; this potentially dangerous feature enables better
sharing for those paths for which the content is known in
advance (e.g., because a content hash is given).
* Fast builds: if we can expand all output paths of a derive
expression, we don't have to build.
* A function to find all Nix expressions whose output ids are
completely contained in some set. Useful for uploading relevant Nix
expressions to a shared cache.
number of bytes, e.g., in case of a signal like SIGSTOP.
This caused `nix --dump' to fail sometimes.
Note that this bug went unnoticed because the call to `nix
--dump' is in a pipeline, and the shell ignores non-zero
exit codes from all but the last element in the pipeline.
Is there any way to check the result of the initial elements
in the pipeline? (In other words, is it at all possible to
write reliable shell scripts?)
input path is referenced in an output paths, we also have to
add all ids referenced by that input path.
* Better debug assertions to catch these sorts of errors.
This is not entirely trivial since this introduces the possibility
of mutual recursion.
* Made normal forms self-contained.
* Use unique ids, not content hashes, for content referencing.
Unifying substitutes and successors isn't very feasible for now,
since substitutes are only used when no path with a certain is
known. Therefore, a normal form of some expression stored as a
substitute would not be used unless the expression itself was
missing.
hash for which no local expansion is available, Nix can execute a
`substitute' which should produce a path with such a hash.
This is policy-free since Nix does not in any way specify how the
substitute should work, i.e., it's an arbitrary (unnormalised)
fstate expression. For example, `nix-pull' registers substitutes
that fetch Nix archives from the network (through `wget') and unpack
them, but any other method is possible as well. This is an
improvement over the old Nix sharing scheme, which had a policy
(fetching through `wget') built in.
The sharing scheme doesn't work completely yet because successors
from fstate rewriting have to be registered on the receiving side.
Probably the whole successor stuff can be folded up into the
substitute mechanism; this would be a nice simplification.
archives (using the package in corepkgs/nar).
* queryPathByHash -> expandHash, and it takes an argument specifying
the target path (which may be empty).
* Install the core Fix packages in $prefix/share/fix. TODO: bootstrap
Nix and install Nix as a Fix package.
a mapping from the hash to a url has been registered through `nix
regurl'.
* Bug fix in nix: don't pollute stdout when running tar, it made
nix-switch barf.
* Bug fix in nix-push-prebuilts: don't create a subdirectory on the
target when rsync'ing.
sharing package directories (i.e., the result of building a Nix
descriptor).
`nix-pull-prebuilts' obtains a list of all known prebuilts by
consulting the paths and URLs specified in
$prefix/etc/nix/prebuilts.conf. The mappings ($pkghash,
$prebuilthash) and ($prebuilthash, $location) are registered with
Nix so that it can use the prebuilt with hash $prebuilthash when
installing a package with hash $pkghash by downloading and unpacking
$location.
`nix-push-prebuilts' creates prebuilts for all packages for which no
prebuilt is known to exist. It can then optionally upload these
to the network through rsync.
`nix-[pull|push]-prebuilts' just provide a policy. Nix provides the
mechanism through the `nix [export|regprebuilt|regurl]' commands.
* Conditionals and variables in Fix expressions. This allows, e.g.,
Descr(
[ Bind("pkgId", "subversion-0.21.0")
, Bind("httpsClient", Bool(True))
, Bind("httpServer", Bool(True))
, Bind("ssl", If(Var("httpsClient"), Fix("./openssl-0.9.7b.fix"), ""))
, Bind("httpd", If(Var("httpServer"), Fix("./httpd-2.0.45.fix"), ""))
...
])
which introduces domain feature variables httpsClient and httpServer
(i.e., whether Subversion is built with https client and webdav
server support); the values of the variables influences package
dependencies and the build scripts.
The next step is to allow that packages can express constraints on
each other. E.g., StrategoXT is dependent on an ATerm library with
the "gcc" variant enabled. In fact, this may cause several
Nix instantiations to be created from a single Fix descriptor. If
possible, Fix should try to find the least set of instantiations
that obeys the constraints.
descriptors generated out of Fix descriptors specified on the
command line. This allows us to say:
nix-switch $(fix -i ./test/fixdescriptors/system.fix)
build action for `system' packages (like system.fix) that have
dependencies on all packages we want to activate.
So the command sequence to switch to a new activation configuration
of the system would be:
$ fix -i .../fixdescriptors/system.fix
...
system.fix -> 89cf4713b37cc66989304abeb9ea189f
$ nix-switch 89cf4713b37cc66989304abeb9ea189f
* A nix-profile.sh script that can be included in .bashrc.
packages (i.e., the packages that should appear in the user's $PATH,
and so on). Based on this list, the script nix-populate creates a
hierarchy of symlinks to the relevant files in those packages (e.g.,
for pkg/bin and pkg/lib).
A nice property of nix-populate is that on each run it creates a
*new* tree, rather than updating the old one. It then atomically
switches over to the new tree. This allows atomic upgrades or
rollbacks on the set of activated packages.
