Note that this will only work if the client has a very recent Nix
version (post 15e1b2c223), otherwise the
--option flag will just be ignored.
Fixes#50.
This handles the chroot and build hook cases, which are easy.
Supporting the non-chroot-build case will require more work (hash
rewriting!).
Issue #21.
Output names are now appended to resulting GC symlinks, e.g. by
nix-build. For backwards compatibility, if the output is named "out",
nothing is appended. E.g. doing "nix-build -A foo" on a derivation
that produces outputs "out", "bin" and "dev" will produce symlinks
"./result", "./result-bin" and "./result-dev", respectively.
Channels can now advertise a binary cache by creating a file
<channel-url>/binary-cache-url. The channel unpacker puts these in
its "binary-caches" subdirectory. Thus, the URLS of the binary caches
for the channels added by root appear in
/nix/var/nix/profiles/per-user/eelco/channels/binary-caches/*. The
binary cache substituter reads these and adds them to the list of
binary caches.
The .nixpkg file format is extended to optionally include the URL of a
binary cache, which will be used in preference to the manifest URL
(which can be set to a non-existent value).
Querying all substitutable paths via "nix-env -qas" is potentially
hard on a server, since it involves sending thousands of HEAD
requests. So a binary cache must now have a meta-info file named
"nix-cache-info" that specifies whether the server wants this. It
also specifies the store prefix so that we don't send useless queries
to a binary cache for a different store prefix.
Since SubstitutionGoal::finished() in build.cc computes the hash
anyway, we can prevent the inefficiency of computing the hash twice by
letting the substituter tell Nix about the expected hash, which can
then verify it.
Commit 6a214f3e06 reused the NixOS
environment initialisation for nix-profile.sh, but this is
inappropriate on systems that don't have multi-user support enabled.
In "nix-env -qas", we don't need the substitute info, we just need to
know if it exists. This can be done using a HTTP HEAD request, which
saves bandwidth.
Note however that curl currently has a bug that prevents it from
reusing HTTP connections if HEAD requests return a 404:
https://sourceforge.net/tracker/?func=detail&aid=3542731&group_id=976&atid=100976
Without the patch attached to the issue, using HEAD is actually quite
a bit slower than GET.
Getting substitute information using the binary cache substituter has
non-trivial latency overhead. A package or NixOS system configuration
can have hundreds of dependencies, and in the worst case (when the
local info cache is empty) we have to do a separate HTTP request for
each of these. If the ping time to the server is t, getting N info
files will take tN seconds; e.g., with a ping time of 0.1s to
nixos.org, sequentially downloading 1000 info files (a typical NixOS
config) will take at least 100 seconds.
To fix this problem, the binary cache substituter can now perform
requests in parallel. This required changing the substituter
interface to support a function querySubstitutablePathInfos() that
queries multiple paths at the same time, and rewriting queryMissing()
to take advantage of parallelism. (Due to local caching,
parallelising queryMissing() is sufficient for most use cases, since
it's almost always called before building a derivation and thus fills
the local info cache.)
For example, parallelism speeds up querying all 1056 paths in a
particular NixOS system configuration from 116s to 2.6s. It works so
well because the eccentricity of the top-level derivation in the
dependency graph is only 9. So we only need 10 round-trips (when
using an unlimited number of parallel connections) to get everything.
Currently we do a maximum of 150 parallel connections to the server.
Thus it's important that the binary cache server (e.g. nixos.org) has
a high connection limit. Alternatively we could use HTTP pipelining,
but WWW::Curl doesn't support it and libcurl has a hard-coded limit of
5 requests per pipeline.
Using WWW::Curl rather than running an external curl process for every
NAR info file halves the time it takes to get info thanks to libcurl's
support for persistent HTTP connections. (We save a roundtrip per
file.) But the real gain will come from using parallel and/or
pipelined requests.
XZ compresses significantly better than bzip2. Here are the
compression ratios and execution times (using 4 cores in parallel) on
my /var/run/current-system (3.1 GiB):
bzip2: total compressed size 849.56 MiB, 30.8% [2m08]
xz -6: total compressed size 641.84 MiB, 23.4% [6m53]
xz -7: total compressed size 621.82 MiB, 22.6% [7m19]
xz -8: total compressed size 599.33 MiB, 21.8% [7m18]
xz -9: total compressed size 588.18 MiB, 21.4% [7m40]
Note that compression takes much longer. More importantly, however,
decompression is much faster:
bzip2: 1m47.274s
xz -6: 0m55.446s
xz -7: 0m54.119s
xz -8: 0m52.388s
xz -9: 0m51.842s
The only downside to using -9 is that decompression takes a fair
amount (~65 MB) of memory.
Manifests are a huge pain, since users need to run nix-pull directly
or indirectly to obtain them. They tend to be large and lag behind
the available binaries; also, the downloaded manifests in
/nix/var/nix/manifest need to be in sync with the Nixpkgs sources. So
we want to get rid of them.
The idea of manifest-free operation works as follows. Nix is
configured with a set of URIs of binary caches, e.g.
http://nixos.org/binary-cache
Whenever Nix needs a store path X, it checks each binary cache for the
existence of a file <CACHE-URI>/<SHA-256 hash of X>.narinfo, e.g.
http://nixos.org/binary-cache/bi1gh9...ia17.narinfo
The .narinfo file contains the necessary information about the store
path that was formerly kept in the manifest, i.e., (relative) URI of
the compressed NAR, references, size, hash, etc. For example:
StorePath: /nix/store/xqp4l88cr9bxv01jinkz861mnc9p7qfi-neon-0.29.6
URL: 1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd.nar.bz2
CompressedHash: sha256:1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd
CompressedSize: 202542
NarHash: sha256:1af26536781e6134ab84201b33408759fc59b36cc5530f57c0663f67b588e15f
NarSize: 700440
References: 043zrsanirjh8nbc5vqpjn93hhrf107f-bash-4.2-p24 cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13 ...
Deriver: 4idz1bgi58h3pazxr3akrw4fsr6zrf3r-neon-0.29.6.drv
System: x86_64-linux
Nix then knows that it needs to download
http://nixos.org/binary-cache/1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd.nar.bz2
to substitute the store path.
Note that the store directory is omitted from the References and
Deriver fields to save space, and that the URL can be relative to the
binary cache prefix.
This patch just makes nix-push create binary caches in this format.
The next step is to make a substituter that supports them.
For several platforms we don't currently have "native" Nix packages
(e.g. Mac OS X and FreeBSD). This provides the next best thing: a
tarball containing the closure of Nix, plus a simple script
"nix-finish-install" that initialises the Nix database, registers the
paths in the closure as valid, and runs "nix-env -i /path/to/nix" to
initialise the user profile.
The tarball must be unpacked in the root directory. It creates
/nix/store/... and /usr/bin/nix-finish-install. Typical installation
is as follows:
$ cd /
$ tar xvf /path/to/nix-1.1pre1234_abcdef-x86_64-linux.tar.bz2
$ nix-finish-install
(if necessary add ~/.nix-profile/etc/profile.d/nix.sh to the shell
login scripts)
After this, /usr/bin/nix-finish-install can be deleted, if desired.
The downside to the binary tarball is that it's pretty big (~55 MiB
for x86_64-linux).
Mandatory features are features that MUST be present in a derivation's
requiredSystemFeatures attribute. One application is performance
testing, where we have a dedicated machine to run performance tests
(and nothing else). Then we would add the label "perf" to the
machine's mandatory features and to the performance testing
derivations.
"nix-channel --add" now accepts a second argument: the channel name.
This allows channels to have a nicer name than (say) nixpkgs_unstable.
If no name is given, it defaults to the last component of the URL
(with "-unstable" or "-stable" removed).
Also, channels are now stored in a profile
(/nix/var/nix/profiles/per-user/$USER/channels). One advantage of
this is that it allows rollbacks (e.g. if "nix-channel --update" gives
an undesirable update).
Sometimes when doing "nix-build --run-env" you don't want all
dependencies to be built. For instance, if we want to do "--run-env"
on the "build" attribute in Hydra's release.nix (to get Hydra's build
environment), we don't want its "tarball" dependency to be built. So
we can do:
$ nix-build --run-env release.nix -A build --exclude 'hydra-tarball'
This will skip the dependency whose name matches the "hydra-tarball"
regular expression. The "--exclude" option can be repeated any number
of times.
This command builds or fetches all dependencies of the given
derivation, then starts a shell with the environment variables from
the derivation. This shell also sources $stdenv/setup to initialise
the environment further.
The current directory is not changed. Thus this is a convenient way
to reproduce a build environment in an existing working tree.
Existing environment variables are left untouched (unless the
derivation overrides them). As a special hack, the original value of
$PATH is appended to the $PATH produced by $stdenv/setup.
Example session:
$ nix-build --run-env '<nixpkgs>' -A xterm
(the dependencies of xterm are built/fetched...)
$ tar xf $src
$ ./configure
$ make
$ emacs
(... hack source ...)
$ make
$ ./xterm
directory. Previously in this situation we did add the Nix
expressions from the channel to allow installation from source, but
this doesn't work for binary-only channels and leads to confusing
error messages.
scripts.
* Include the version and architecture in the -I flag so that there is
at least a chance that a Nix binary built for one Perl version will
run on another version.
other simplifications.
* Use <nix/...> to locate the corepkgs. This allows them to be
overriden through $NIX_PATH.
* Use bash's pipefail option in the NAR builder so that we don't need
to create a temporary file.
closure to a given machine at the same time. This prevents the case
where multiple instances try to copy the same missing store path to
the target machine, which is very wasteful.
‘nix-store --export’.
* Add a Perl module that provides the functionality of
‘nix-copy-closure --to’. This is used by build-remote.pl so it no
longer needs to start a separate nix-copy-closure process. Also, it
uses the Perl API to do the export, so it doesn't need to start a
separate nix-store process either. As a result, nix-copy-closure
and build-remote.pl should no longer fail on very large closures due
to an "Argument list too long" error. (Note that having very many
dependencies in a single derivation can still fail because the
environment can become too large. Can't be helped though.)
read the manifest just to check the version and print the number of
paths. This makes nix-pull very fast for the cached cache (speeding
up nixos-rebuild without the ‘--no-pull’ or ‘--fast’ options).
brackets, e.g.
import <nixpkgs/pkgs/lib>
are resolved by looking them up relative to the elements listed in
the search path. This allows us to get rid of hacks like
import "${builtins.getEnv "NIXPKGS_ALL"}/pkgs/lib"
The search path can be specified through the ‘-I’ command-line flag
and through the colon-separated ‘NIX_PATH’ environment variable,
e.g.,
$ nix-build -I /etc/nixos ...
If a file is not found in the search path, an error message is
lazily thrown.
SQLite manifest cache. The DBI AutoCommit feature caused every
process to have an active transaction at all times, which could
indefinitely block processes wanting to update the manifest cache.
* Disable fsync() in the manifest cache because we don't need
integrity (the cache can always be recreated if it gets corrupted).
them into memory. This brings memory use down to (more or less)
O(1). For instance, on my test case, the maximum resident size of
download-using-manifests while filling the DB went from 142 MiB to
11 MiB.
This significantly speeds up the download-using-manifests
substituter, especially if manifests are very large. For instance,
one "nix-build -A geeqie" operation that updated four packages using
binary patches went from 18.5s to 1.6s. It also significantly
reduces memory use.
The cache is kept in /nix/var/nix/manifests/cache.sqlite. It's
updated automatically when manifests are added to or removed from
/nix/var/nix/manifests. It might be interesting to have nix-pull
store manifests directly in the DB, rather than storing them as
separate flat files, but then we would need a command line interface
to delete manifests from the DB.
`+' and `?' in filenames. This is very slow if /nix/store is very
large. (This is a quick hack - a cleaner solution would be to
bypass the shell entirely.)
`nix-store -q --hash' to get the hash of the base path rather than
`nix-hash'. However, only do this for estimating the size of a
download, not for the actual substitution, because sometimes the
contents of store paths are modified (which they shouldn't, of
course).
hook script proper, and the stdout/stderr of the builder. Only the
latter should be saved in /nix/var/log/nix/drvs.
* Allow the verbosity to be set through an option.
* Added a flag --quiet to lower the verbosity level.
it requires a certain feature on the build machine, e.g.
requiredSystemFeatures = [ "kvm" ];
We need this in Hydra to make sure that builds that require KVM
support are forwarded to machines that have KVM support. Probably
this should also be enforced for local builds.
the hook every time we want to ask whether we can run a remote build
(which can be very often), we now reuse a hook process for answering
those queries until it accepts a build. So if there are N
derivations to be built, at most N hooks will be started.
load on the Hydra build farm (where it's unnecessary anyway because
it has a fast connection to the build machines). In any case,
compression can be enabled by using the `-C' option to ssh.
doesn't exist. The Debian packages don't include the manifests
directory, so nix-channel would silently skip doing a nix-pull,
resulting in everything being built from source. Thanks to Juan
Pedro Bolívar Puente.
giving jobs to the first machine until it hits its job limit, then
the second machine and so on. This should improve utilisation of
the Hydra build farm a lot. Also take an optional speed factor
into account to cause fast machines to be preferred over slower
machines with a similar load.
(that is, call the build hook with a certain interval until it
accepts the build).
* build-remote.pl was totally broken: for all system types other than
the local system type, it would send all builds to the *first*
machine of the appropriate type.
sure that it works as expected when you pass it a derivation. That
is, we have to make sure that all build-time dependencies are built,
and that they are all in the input closure (otherwise remote builds
might fail, for example). This is ensured at instantiation time by
adding all derivations and their sources to inputDrvs and inputSrcs.
downloaded files; rather, check the hash of the unpacked store
path.
When the server produces bzipped NAR archives on demand (like Hydra
does), the hash of the file is not known in advance; it's streamed
from the server. Thus the manifest doesn't contain a hash for the
bzipped NAR archive. However, the server does know the hash of the
*uncompressed* NAR archive (the "NarHash" field), since it's stored
in the Nix database (nix-store -q --hash /nix/store/bla). So we use
that instead for checking the integrity of the download.
scan for runtime dependencies (i.e. the local machine shouldn't do a
scan that the remote machine has already done). Also pipe directly
into `nix-store --import': don't use a temporary file.
(e.g. an SSH connection problem) and permanent failures (i.e. the
builder failed). This matters to Hydra (it wants to know whether it
makes sense to retry a build).
makes more sense for the build farm, otherwise every nix-store
invocation will lead to at least one local build. Will come up with
a better solution later...
necessary that at least one build hook doesn't return "postpone",
otherwise nix-store will barf ("waiting for a build slot, yet there
are no running children"). So inform the build hook when this is
the case, so that it can start a build even when that would exceed
the maximum load on a machine.
list like
root@example.org x86_64-linux /root/.ssh/id_buildfarm 1
root@example.org i686-darwin /root/.ssh/id_buildfarm 1
This is possible when the Nix installation on example.org itself has
remote builds enabled.
derivation should be a source rather than a derivation dependency of
the call to the NAR derivation. Otherwise the derivation (and all
its dependencies) will be built as a side-effect, which may not even
succeed.
dependency. `storePath /nix/store/bla' gives exactly the same
result as `toPath /nix/store/bla', except that the former includes
/nix/store/bla in the dependency context of the string.
Useful in some generated Nix expressions like nix-push, which now
finally does the right thing wrt distributed builds. (Previously
the path to be packed wasn't an explicit dependency, so it wouldn't
be copied to the remote machine.)
Common code in local build package sources refactored out in a function; before building the real set of derivations needed is found (slightly slower for only one build strategy, but less garbage on output and better performance for multiple build strategies).
Now you have full choice of best-effort build regardless of method (substituters or actual build), using substituters, building only fixed derivations (should get you all the downloads) and local build without even trying substituters.
Some minor fix in the help text about behavior with no package sources.
again. (After the previous substituter mechanism refactoring I
didn't update the code that obtains the references of substitutable
paths.) This required some refactoring: the substituter programs
are now kept running and receive/respond to info requests via
stdin/stdout.
it only does something if $NIX_OTHER_STORES (not really a good
name...) is set.
* Do globbing on the elements of $NIX_OTHER_STORES. E.g. you could
set it to /mnts/*/nix or something.
* Install substituters in libexec/nix/substituters.
stores (typically remote Nix stores mounted via e.g. NFS, or the Nix
store on the NixOS installation CD). Example use:
$ sshfs foo@example.org:/ /mnt
$ NIX_OTHER_STORES=/mnt/nix \
NIX_SUBSTITUTERS=.../copy-from-other-stores.pl \
nix-env -i foo
This will be especially useful for the installation CD since it
doesn't require a manifest for the CD contents.
usage by finding identical files in the store and hard-linking them
to each other. It typically reduces the size of the store by
something like 25-35%. This is what the optimise-store.pl script
did, but the new command is faster and more correct (it's safe wrt
garbage collection and concurrent builds).
Nix expressions in that directory are combined into an attribute set
{file1 = import file1; file2 = import file2; ...}, i.e. each Nix
expression is an attribute with the file name as the attribute
name. Also recurses into directories.
* nix-env: removed the "--import" (-I) option which set the
~/.nix-defexpr symlink.
* nix-channel: don't use "nix-env --import", instead symlink
~/.nix-defexpr/channels. So finally nix-channel --update doesn't
override any default Nix expressions but combines with them.
This means that you can have (say) a local Nixpkgs SVN tree and use
it as a default for nix-env:
$ ln -s .../path-to-nixpkgs-tree ~/.nix-defexpr/nixpkgs_svn
and be subscribed to channels (including Nixpkgs) at the same time.
(If there is any ambiguity, the -A flag can be used to
disambiguate, e.g. "nix-env -i -A nixpkgs_svn.pan".)
manifests directory. In that case, we don't do a nix-pull, so the
user gets pure source deployment.
The directory /nix/var/nix/gcroots/per-user/$USER should be
writable. (It's created automatically if
/nix/var/nix/gcroots/per-user is writable, e.g. if it has 1777
permission.)
need any info on substitutable paths, we just call the substituters
(such as download-using-manifests.pl) directly. This means that
it's no longer necessary for nix-pull to register substitutes or for
nix-channel to clear them, which makes those operations much faster
(NIX-95). Also, we don't have to worry about keeping nix-pull
manifests (in /nix/var/nix/manifests) and the database in sync with
each other.
The downside is that there is some overhead in calling an external
program to get the substitutes info. For instance, "nix-env -qas"
takes a bit longer.
Abolishing the substitutes table also makes the logic in
local-store.cc simpler, as we don't need to store info for invalid
paths. On the downside, you cannot do things like "nix-store -qR"
on a substitutable but invalid path (but nobody did that anyway).
* Never catch interrupts (the Interrupted exception).
NIX_DOWNLOAD_CACHE is set, then nix-prefetch-url will store the hash
and timestamp of downloaded files in the directory
$NIX_DOWNLOAD_CACHE. This allows it to figure out if the file is
still in the Nix store.
get the basename of the channel URL (e.g., nixpkgs-unstable). The
top-level Nix expression of the channel is now an attribute set, the
attributes of which are the individual channels (e.g.,
{nixpkgs_unstable = ...; strategoxt_unstable = ...}). This makes
attribute paths ("nix-env -qaA" and "nix-env -iA") more sensible,
e.g., "nix-env -iA nixpkgs_unstable.subversion".
another machine through ssh. E.g.,
$ nix-copy-closure xyzzy $(which svn)
copies the closure of Subversion to machine `xyzzy'. This is like
`nix-pack-closure $(which svn) | ssh xyzzy', but it's much more
efficient since it only copies those paths that are missing on the
target machine.
* nix-unpack-closure: extract the top-level paths from the closure and
print them on stdout. This allows them to be installed, e.g.,
"nix-env -i $(nix-unpack-closure)". (NIX-64)
seconds without producing output on stdout or stderr (NIX-65). This
timeout can be specified using the `--max-silent-time' option or the
`build-max-silent-time' configuration setting. The default is
infinity (0).
* Fix a tricky race condition: if we kill the build user before the
child has done its setuid() to the build user uid, then it won't be
killed, and we'll potentially lock up in pid.wait(). So also send a
conventional kill to the child.
a different location than the user's. This makes channels usable as
a source deployment mechanism for people who install Nix under
non-standard prefixes. (NIX-57)
* `nix-install-package --help' (NIX-9).
* `nix-install-package --non-interactive': don't prompt or pause.
* Tests for nix-install-package.
* Security fixes: filter the values obtained from the nixpkg.
package duplication present in (e.g.) a profile. It shows the
number of instances of each package in a closure, along with the
size in bytes of each instance as well as the "waste" (the
difference between the sum of the sizes of all instances and the
average size).
$ ./show-duplication.pl /nix/var/nix/profiles/default
gcc 11
3.3.6 19293318
3.4.4 21425257
...
average 14942970, waste 149429707
coreutils 6
...
average package duplication 1.87628865979381, total size 3486330471, total waste 1335324237, 38.3017114443825% wasted
This utility is useful for measuring the cost in terms of disk space
of the Nix approach.
old generations of *all* profiles in /nix/var/nix/profiles, then
runs the garbage collector. Quick way to get rid of all old stuff.
Of course, one cannot roll back to earlier points in time after
this.
such as open files, current directories, mmaped files, etc. This is
inherently unportable, but it's easy to adapt this script to other
platforms. Currently we call `lsof' and try to read various bits in
/proc/NNN.
The goal is to prevent the garbage collector from removing store
paths that are no longer reachable from a permanent root but that
are still in use (for instance, after the user has done "nix-env -e"
on a running program).
target no longer applies to any available release. This is a
partial fix for NIX-34 (when producing linear patch sequences
between releases, the number of patches grows without bound).
useful way to transfer the closure of a store path to another
machine.
These commands provide functionality previously possible through
`nix-push --copy'. However, they are much more convenient in many
situations (though possibly less efficient).
Example:
$ nix-pack-closure /nix/store/hj232g1r...-subversion-1.3.0 > svn.closure
(on another machine:)
$ nix-unpack-closure < svn.closure
Note that Subversion is added to the store, but not installed into a
user environment. One should do `nix-env -i
/nix/store/hj232g1r...-subversion-1.3.0' for that.
Another example: copy the application Azureus to the machine
`scratchy' through ssh:
$ nix-pack-closure $(which azureus) | ssh scratchy nix-unpack-closure
nix-store query options `--referer' and `--referer-closure' have
been changed to `--referrer' and `--referrer-closure' (but the old
ones are still accepted for compatibility).
the parent runs before the child, it closes some pipe file
descriptors which causes the child to fail due to a bad file
descriptor. So we just use the normal open() function instead.
This fixes NIX-14 (intermittent nix-pull failures).
handle it. It crashed on the 234 MB tetex archive. Probably we
will never be able to handle archives of that size on 32-bit
machines (because bsdiff does everything in memory requiring
max(17*n,9*n+m)+O(1) bytes, so the address space simply isn't
there).
derivations. This is mostly to simplify the implementation of
nix-prefetch-{url, svn}, which now work properly in setuid
installations.
* Enforce valid store names in `nix-store --add / --add-fixed'.
occurances of a component. If the shortest path distance between a
component P and Q in the referers graph is D, then the contribution
of Q to the use of P is 1 / R^D, where R >= 1, typically 2. This
expresses that distant indirect uses are less important than nearby
uses.
For instance, this can disambiguate between the bootstrap GCC in
Nixpkgs and the GCC of the final stdenv (the former has more uses,
but they are further away), and between the GCC of the final stdenv
and the GCC+G77 build (the latter has very few uses).
by the build farm. See e.g.,
http://catamaran.labs.cs.uu.nl/dist/nixpkgs-0.8/nixpkgs-0.7pre2302/;
the user can click on packages, and they will be installed (assuming
the `application/nix-package' MIME type has been associated with
`nix-install-package').
Nix expressions are no longer involved: a "package" is just a
pointer to a manifest, and the top-level store derivation to be
added to the user environment. This makes these packages
independent from Nix expression evolution.
Note that we install the store derivation ($drvPath), not the
resulting output path ($outPath). This is equivalent, except that
installing the derivation maintains the back-link from the output
path to the derivation that built it. This is useful for
maintenance.
* Automatically re-exec in an xterm so that the user sees something
when `nix-install-package' is run from a browser.
too.
* Change the default hash for nix-prefetch-url back to md5, since
that's what we use in Nixpkgs (for now; a birthday attack is rather
unlikely there).
* Removed some dead code (successor stuff) from nix-push.
* Updated terminology in the tests (store expr -> drv path).
* Check that the deriver is set properly in the tests.
This was necessary becase root finding must be done after
acquisition of the global GC lock.
This makes `nix-collect-garbage' obsolete; it is now just a wrapper
around `nix-store --gc'.
* Automatically remove stale GC roots (i.e., indirect GC roots that
point to non-existent paths).
get rid of GC roots. Nix-build places a symlink `result' in the
current directory. Previously, removing that symlink would not
remove the store path being linked to as a GC root. Now, the GC
root created by nix-build is actually a symlink in
`/nix/var/nix/gcroots/auto' to `result'. So if that symlink is
removed the GC root automatically becomes invalid (since it can no
longer be resolved). The root itself is not automatically removed -
the garbage collector should delete dangling roots.
immediately add the result as a permanent GC root. This is the only
way to prevent a race with the garbage collector. For instance, the
old style
ln -s $(nix-store -r $(nix-instantiate foo.nix)) \
/nix/var/nix/gcroots/result
has two time windows in which the garbage collector can interfere
(by GC'ing the derivation and the output, respectively). On the
other hand,
nix-store --add-root /nix/var/nix/gcroots/result -r \
$(nix-instantiate --add-root /nix/var/nix/gcroots/drv \
foo.nix)
is safe.
* nix-build: use `--add-root' to prevent GC races.
roots to a per-process temporary file in /nix/var/nix/temproots
while holding a write lock on that file. The garbage collector
acquires read locks on all those files, thus blocking further
progress in other Nix processes, and reads the sets of temporary
roots.
though). In particular it's now much easier to register a GC root.
Just place a symlink to whatever store path it is that you want to
keep in /nix/var/nix/gcroots.
* Formalise the notion of fixed-output derivations, i.e., derivations
for which a cryptographic hash of the output is known in advance.
Changes to such derivations should not propagate upwards through the
dependency graph. Previously this was done by specifying the hash
component of the output path through the `id' attribute, but this is
insecure since you can lie about it (i.e., you can specify any hash
and then produce a completely different output). Now the
responsibility for checking the output is moved from the builder to
Nix itself.
A fixed-output derivation can be created by specifying the
`outputHash' and `outputHashAlgo' attributes, the latter taking
values `md5', `sha1', and `sha256', and the former specifying the
actual hash in hexadecimal or in base-32 (auto-detected by looking
at the length of the attribute value). MD5 is included for
compatibility but should be considered deprecated.
* Removed the `drvPath' pseudo-attribute in derivation results. It's
no longer necessary.
* Cleaned up the support for multiple output paths in derivation store
expressions. Each output now has a unique identifier (e.g., `out',
`devel', `docs'). Previously there was no way to tell output paths
apart at the store expression level.
* `nix-hash' now has a flag `--base32' to specify that the hash should
be printed in base-32 notation.
* `fetchurl' accepts parameters `sha256' and `sha1' in addition to
`md5'.
* `nix-prefetch-url' now prints out a SHA-1 hash in base-32. (TODO: a
flag to specify the hash.)
files and hard-links them to each other to save disk space.
Currently it doesn't actually do the hard-linking, it just reports
the amount of space saved if it did.
substitute mechanism) creates a store path by downloading full NAR
archives and/or patches specified in the available manifests.
Any combination of present paths, full downloads, and patches can be
used to construct the target path. In particular, patches can be
chained in sequence; and full NAR archives of the target path can be
omitted (i.e., patch-only deployment is possible). A shortest path
algorithm is used to find the smallest set of files to be downloaded
(the edge weights are currently file sizes, but one can imagine
taking the network speed to the various source into account).
Patches are binary deltas between two store paths. To be precise,
they are the output of the `bsdiff' program applied to the NAR
archives obtained by dumping (`nix-store --dump') the two store
paths. The advantage of diff'ing NAR archives (and not, say, doing
file-by-file diffs) is that file renames/moves are handled
automatically. The disadvantage is that we cannot optimise creation
of unchanged files (by hard-linking).
unreachable paths that haven't been used for N hours. For instance,
`nix-collect-garbage --min-age 168' only deletes paths that haven't
been accessed in the last week.
This is useful for instance in the build farm where many derivations
can be shared between consecutive builds, and we wouldn't want a
garbage collect to throw them all away. We could of course register
them as roots, but then we'd to unregister them at some point, which
would be a pain to manage. The `--min-age' flag gives us a sort of
MRU caching scheme.
BUG: this really shouldn't be in gc.cc since that violates
mechanism/policy separation.
doesn't just print the set of paths that should be deleted. So
there is no more need to pipe the result into `nix-store --delete'
(which doesn't even exist anymore).
users.
If the configure flag `--enable-setuid' is used, the Nix programs
nix-env, nix-store, etc. are installed with the setuid bit turned on
so that they are executed as the user and group specified by
`--with-nix-user=USER' and `--with-nix-group=GROUP', respectively
(with defaults `nix' and `nix').
The setuid programs drop all special privileges if they are executed
by a user who is not a member of the Nix group.
The setuid feature is a quick hack to enable sharing of a Nix
installation between users who trust each other. It is not
generally secure, since any user in the Nix group can modify (by
building an appropriate derivation) any object in the store, and for
instance inject trojans into binaries used by other users.
The setuid programs are owned by root, not the Nix user. This is
because on Unix normal users cannot change the real uid, only the
effective uid. Many programs don't work properly when the real uid
differs from the effective uid. For instance, Perl will turn on
taint mode. However, the setuid programs drop all root privileges
immediately, changing all uids and gids to the Nix user and group.
Nix expressions.
To subscribe to a channel (needs to be done only once):
nix-channel --add \
http://catamaran.labs.cs.uu.nl/dist/nix/channels/nixpkgs-unstable
This just adds the given URL to ~/.nix-channels (which can also be
edited manually).
To update from all channels:
nix-channel --update
This fetches the latest expressions and pulls cache manifests. The
default Nix expression (~/.nix-defexpr) is made to point to the
conjunction of the expressions downloaded from all channels.
So to update all installed derivations in the current user
environment:
nix-channel --update
nix-env --upgrade '*'
If you are really courageous, you can put this in a cronjob or
something.
You can subscribe to multiple channels. It is not entirely clear
what happens when there are name clashes between derivations from
different channels. From nix-env/main.cc it appears that the one
with the lowest (highest?) hash will be used, which is pretty
meaningless.
environment variable. This is useful for passing authentication
information (it won't show up in `ps'). Hacky - nix-push should
abstract over the use of Curl.
default -> default-94-link
default-82-link -> /nix/store/cc4480...
default-83-link -> /nix/store/caeec8...
...
default-94-link -> /nix/store/2896ca...
experimental -> experimental-2-link
experimental-1-link -> /nix/store/cc4480...
experimental-2-link -> /nix/store/a3148f...
* `--profile' / `-p' -> `--switch-profile' / `-S'
* `--link' / `-l' -> `--profile' / `-p'
* The default profile is stored in $prefix/var/nix/profiles.
$prefix/var/nix/links is gone. Profiles can be stored anywhere.
* The current profile is now referenced from ~/.nix-profile, not
~/.nix-userenv.
* The roots to the garbage collector now have extension `.gcroot', not
`.id'.
the symlink ~/.nix-userenv to the given argument (which defaults to
.../links/current). /etc/profile.d/nix-profile creates this symlink
if it doesn't exist yet. Example use:
$ nix-env -l my_profile -i foo.nix subversion quake
$ nix-env -p my_profile
I don't like the term "profile". Let's deprecate it :-)
The script nix-install-package takes a `Nix package file' (which
contains one or more derivations, along with URLs of Nix caches),
unpacks it, pulls the caches, and installs the derivations in the
user's environment.
For best results, associate the command `xterm -e
/nix/bin/nix-install-package' with the MIME type
`application/x-nix-package' and visit
http://losser.st-lab.cs.uu.nl/~eelco/test/.
* Fixed a segfault caused by the buffering of stderr.
* Fix now allows the specification of the full output path. This
should be used with great care, since it by-passes the normal hash
generation.
* Incremented the version number to 0.4 (prerelease).
NAME'. E.g., on the losser Subversion server, I do `nix-switch --name
svn $(fix ...)' to atomically upgrade the server (the SVN server
uses the Apache and Subversion installations in /nix/var/nix/links/svn).
Renamed `fstateRefs' to `fstateRequisites'. The semantics of this
function is that it returns a list of all paths necessary to realise
a given expression. For a derive expression, this is the union of
requisites of the inputs; for a slice expression, it is the path of
each element in the slice. Also included are the paths of the
expressions themselves. Optionally, one can also include the
requisites of successor expressions (to recycle intermediate
results).
* `nix-switch' now distinguishes between an expression and its normal
form. Usually, only the normal form is registered as a root of the
garbage collector. With the `--source-root' flag, it will also
register the original expression as a root.
* `nix-collect-garbage' now has a flag `--keep-successors' which
causes successors not to be included in the list of garbage paths.
* `nix-collect-garbage' now has a flag `--invert' which will print all
paths that should *not* be garbage collected.
dynamically links against libdb4 (?!), due to LD_LIBRARY_PATH it picks
up our libdb4 instead of SuSE's libdb4, but our libdb4 uses another
glibc so loading barfs.
Instead, all packages should use rpaths to store library locations in
executables/libraries. The disadvantage is that overriding rpaths is
harder. (It is possible by invoking the dynamic linker directly, e.g.,
`/lib/ld-linux.so.2 --ignore-path LIST program args...' to ignore the
rpath for the libraries in LIST). It would be better to use DT_RUNPATH,
which is consulted by the dynamic linker *after* LD_LIBRARY_PATH but
*before* ld.so.cache and the system directories.
substituting for (obvious, really).
* For greater efficiency, nix-pull/unnar will place the output in a
path that is probably the same as what is actually needed, thus
preventing a path copy.
* Even if a output id is given in a Fix package expression, ensure
that the resulting Nix derive expression has a different id. This
is because Nix expressions that are semantically equivalent (i.e.,
build the same result) might be different w.r.t. efficiency or
divergence. It is absolutely vital for the substitute mechanism
that such expressions are not used interchangeably.
number of bytes, e.g., in case of a signal like SIGSTOP.
This caused `nix --dump' to fail sometimes.
Note that this bug went unnoticed because the call to `nix
--dump' is in a pipeline, and the shell ignores non-zero
exit codes from all but the last element in the pipeline.
Is there any way to check the result of the initial elements
in the pipeline? (In other words, is it at all possible to
write reliable shell scripts?)
hash for which no local expansion is available, Nix can execute a
`substitute' which should produce a path with such a hash.
This is policy-free since Nix does not in any way specify how the
substitute should work, i.e., it's an arbitrary (unnormalised)
fstate expression. For example, `nix-pull' registers substitutes
that fetch Nix archives from the network (through `wget') and unpack
them, but any other method is possible as well. This is an
improvement over the old Nix sharing scheme, which had a policy
(fetching through `wget') built in.
The sharing scheme doesn't work completely yet because successors
from fstate rewriting have to be registered on the receiving side.
Probably the whole successor stuff can be folded up into the
substitute mechanism; this would be a nice simplification.
archives (using the package in corepkgs/nar).
* queryPathByHash -> expandHash, and it takes an argument specifying
the target path (which may be empty).
* Install the core Fix packages in $prefix/share/fix. TODO: bootstrap
Nix and install Nix as a Fix package.
a mapping from the hash to a url has been registered through `nix
regurl'.
* Bug fix in nix: don't pollute stdout when running tar, it made
nix-switch barf.
* Bug fix in nix-push-prebuilts: don't create a subdirectory on the
target when rsync'ing.
sharing package directories (i.e., the result of building a Nix
descriptor).
`nix-pull-prebuilts' obtains a list of all known prebuilts by
consulting the paths and URLs specified in
$prefix/etc/nix/prebuilts.conf. The mappings ($pkghash,
$prebuilthash) and ($prebuilthash, $location) are registered with
Nix so that it can use the prebuilt with hash $prebuilthash when
installing a package with hash $pkghash by downloading and unpacking
$location.
`nix-push-prebuilts' creates prebuilts for all packages for which no
prebuilt is known to exist. It can then optionally upload these
to the network through rsync.
`nix-[pull|push]-prebuilts' just provide a policy. Nix provides the
mechanism through the `nix [export|regprebuilt|regurl]' commands.
doesn't actually delete any packages, it just prints their
descriptor hashes. So we can do
nix info $(nix-collect-garbage)
to print out the ids of the packages that would be deleted, and
nix delpkg $(nix-collect-garbage)
to actually delete them.
This allows us to find out all `live' packages on the system by
doing
nix closure $(cat /nix/var/nix/links/*.hash)
which will print out the activated configurations and all packages
referenced by them. We could then garbage collect unused packages
by deleting the difference between `nix listinst' and the set
returned by `nix closure ...'.
build action for `system' packages (like system.fix) that have
dependencies on all packages we want to activate.
So the command sequence to switch to a new activation configuration
of the system would be:
$ fix -i .../fixdescriptors/system.fix
...
system.fix -> 89cf4713b37cc66989304abeb9ea189f
$ nix-switch 89cf4713b37cc66989304abeb9ea189f
* A nix-profile.sh script that can be included in .bashrc.
