Commit graph

708 commits

Author SHA1 Message Date
Eelco Dolstra
108e14bb18 Fix race condition when two processes create the same link in /nix/store/.links 2012-08-05 18:17:55 -04:00
Eelco Dolstra
6763084ae5 Count bytes freed deleting unused links 2012-08-01 22:43:03 -04:00
Eelco Dolstra
01d56c1eec Drop the block count in the garbage collector 2012-08-01 22:34:46 -04:00
Eelco Dolstra
967d066d8e nix-store --gc: Make ‘--max-freed 0’ do the right thing
That is, delete almost nothing (it will still remove unused links from
/nix/store/.links).
2012-08-01 19:14:58 -04:00
Eelco Dolstra
1df702d347 removeUnusedLinks(): Print stats on disk space saved by hard linking 2012-08-01 19:01:50 -04:00
Eelco Dolstra
9c2decaa19 Merge branch 'master' into no-manifests 2012-08-01 18:03:49 -04:00
Eelco Dolstra
234ce610e0 Doh 2012-08-01 16:09:47 -04:00
Eelco Dolstra
7b10562370 Make ‘nix-store --optimise’ interruptible 2012-08-01 16:06:49 -04:00
Eelco Dolstra
c770a2422a Report substituter errors to clients of the Nix daemon 2012-08-01 11:19:24 -04:00
Eelco Dolstra
eb7849e3a2 Prevent an injection attack in passing untrusted options to substituters 2012-07-31 18:50:32 -04:00
Eelco Dolstra
90d9c58d4d Pass all --option flags to the daemon 2012-07-31 18:19:44 -04:00
Eelco Dolstra
89a8207029 Add an option ‘build-fallback’ (equivalent to the --fallback flag) 2012-07-31 17:56:02 -04:00
Eelco Dolstra
97421eb5ec Refactor settings processing
Put all Nix configuration flags in a Settings object.
2012-07-30 19:55:41 -04:00
Eelco Dolstra
d50d7a2874 Whitespace 2012-07-30 17:13:25 -04:00
Eelco Dolstra
d059bf48e4 Pass configuration settings to the substituters
Previously substituters could read nix.conf themselves, but this
didn't take --option flags into account.
2012-07-30 16:09:54 -04:00
Eelco Dolstra
f9613da180 Remove unused variables 2012-07-30 15:43:37 -04:00
Eelco Dolstra
6183cf2f19 Fix whitespace 2012-07-30 15:42:18 -04:00
Eelco Dolstra
73acb8b836 Let build.cc verify the expected hash of a substituter's output
Since SubstitutionGoal::finished() in build.cc computes the hash
anyway, we can prevent the inefficiency of computing the hash twice by
letting the substituter tell Nix about the expected hash, which can
then verify it.
2012-07-27 12:16:02 -04:00
Eelco Dolstra
fbf59d95f6 Remove more tabs 2012-07-27 10:56:33 -04:00
Eelco Dolstra
b4ea83249b Remove trailing whitespace / tabs 2012-07-27 09:59:18 -04:00
Eelco Dolstra
50395b71a9 Fix the substituter tests 2012-07-26 17:36:07 -04:00
Eelco Dolstra
8c79100839 Merge branch 'master' into no-manifests 2012-07-26 15:14:33 -04:00
Eelco Dolstra
3a4623afbb Set permissions on temporary build directories to 0700
Fixes #39.
2012-07-26 15:04:40 -04:00
Eelco Dolstra
e98c029717 Handle platforms that don't support linking to a symlink
E.g. Darwin doesn't allow this.
2012-07-23 18:42:18 -04:00
Eelco Dolstra
fd63c8bfcd Unlink the right file 2012-07-23 18:06:37 -04:00
Eelco Dolstra
680ab6f83d Garbage collect unused links in /nix/store/.links
Incremental optimisation requires creating links in /nix/store/.links
to all files in the store.  However, this means that if we delete a
store path, no files are actually deleted because links in
/nix/store/.links still exists.  So we need to check /nix/store/.links
for files with a link count of 1 and delete them.
2012-07-23 17:14:16 -04:00
Eelco Dolstra
6193105710 Automatically optimise the Nix store when a new path is added
Auto-optimisation is enabled by default.  It can be turned off by
setting auto-optimise-store to false in nix.conf.
2012-07-23 17:14:15 -04:00
Eelco Dolstra
564fb7d9fa optimiseStore(): Use a content-addressed file store in /nix/store/.links
optimiseStore() now creates persistent, content-addressed hard links
in /nix/store/.links.  For instance, if it encounters a file P with
hash H, it will create a hard link

  P' = /nix/store/.link/<H>

to P if P' doesn't already exist; if P' exist, then P is replaced by a
hard link to P'.  This is better than the previous in-memory map,
because it had the tendency to unnecessarily replace hard links with a
hard link to whatever happened to be the first file with a given hash
it encountered.  It also allows on-the-fly, incremental optimisation.
2012-07-23 17:14:15 -04:00
Eelco Dolstra
6852289c46 Use lutimes() if available to canonicalise the timestamp of symlinks
Also use utimes() instead of utime() if lutimes() is not available.
2012-07-23 16:52:25 -04:00
Eelco Dolstra
b7fd2c2822 Use "#pragma once" to prevent repeated header file inclusion 2012-07-18 14:59:03 -04:00
Eelco Dolstra
fe241ece29 Merge branch 'master' into no-manifests 2012-07-18 10:47:59 -04:00
Eelco Dolstra
ccc52adfb2 Add function queryPathFromHashPart()
To implement binary caches efficiently, Hydra needs to be able to map
the hash part of a store path (e.g. "gbg...zr7") to the full store
path (e.g. "/nix/store/gbg...kzr7-subversion-1.7.5").  (The binary
cache mechanism uses hash parts as a key for looking up store paths to
ensure privacy.)  However, doing a search in the Nix store for
/nix/store/<hash>* is expensive since it requires reading the entire
directory.  queryPathFromHashPart() prevents this by doing a cheap
database lookup.
2012-07-17 18:55:39 -04:00
Eelco Dolstra
3a9fdf2747 Return an exit code of 100 for cached failed builds
Exit code 100 should be returned for all permanent failures.  This
includes cached failures.

Fixes #34.
2012-07-17 15:55:30 -04:00
Eelco Dolstra
6c01fb4d68 Update Nix 1.1 release notes 2012-07-17 10:06:20 -04:00
Eelco Dolstra
53b24f3518 Allow disabling log compression 2012-07-17 09:40:12 -04:00
Eelco Dolstra
f2bdc87595 Update the other substituters 2012-07-11 18:52:09 -04:00
Eelco Dolstra
09a6321aeb Replace hasSubstitutes() with querySubstitutablePaths()
querySubstitutablePaths() takes a set of paths, so this greatly
reduces daemon <-> client latency.
2012-07-11 17:52:18 -04:00
Eelco Dolstra
58ef4d9a95 Add a function queryValidPaths()
queryValidPaths() combines multiple calls to isValidPath() in one.
This matters when using the Nix daemon because it reduces latency.
For instance, on "nix-env -qas \*" it reduces execution time from 5.7s
to 4.7s (which is indistinguishable from the non-daemon case).
2012-07-11 11:08:47 -04:00
Eelco Dolstra
667d5f1936 Rename queryValidPaths() to queryAllValidPaths() 2012-07-11 10:49:04 -04:00
Eelco Dolstra
eb3036da87 Implement querySubstitutablePathInfos() in the daemon
Also removed querySubstitutablePathInfo().
2012-07-11 10:43:24 -04:00
Eelco Dolstra
425cc612ad build.cc: Don't use hasSubstitute()
Instead make a single call to querySubstitutablePathInfo() per
derivation output.  This is faster and prevents having to implement
the "have" function in the binary cache substituter.
2012-07-08 18:39:24 -04:00
Eelco Dolstra
11800e6198 download-from-binary-cache: parallelise fetching of NAR info files
Getting substitute information using the binary cache substituter has
non-trivial latency overhead.  A package or NixOS system configuration
can have hundreds of dependencies, and in the worst case (when the
local info cache is empty) we have to do a separate HTTP request for
each of these.  If the ping time to the server is t, getting N info
files will take tN seconds; e.g., with a ping time of 0.1s to
nixos.org, sequentially downloading 1000 info files (a typical NixOS
config) will take at least 100 seconds.

To fix this problem, the binary cache substituter can now perform
requests in parallel.  This required changing the substituter
interface to support a function querySubstitutablePathInfos() that
queries multiple paths at the same time, and rewriting queryMissing()
to take advantage of parallelism.  (Due to local caching,
parallelising queryMissing() is sufficient for most use cases, since
it's almost always called before building a derivation and thus fills
the local info cache.)

For example, parallelism speeds up querying all 1056 paths in a
particular NixOS system configuration from 116s to 2.6s.  It works so
well because the eccentricity of the top-level derivation in the
dependency graph is only 9.  So we only need 10 round-trips (when
using an unlimited number of parallel connections) to get everything.

Currently we do a maximum of 150 parallel connections to the server.
Thus it's important that the binary cache server (e.g. nixos.org) has
a high connection limit.  Alternatively we could use HTTP pipelining,
but WWW::Curl doesn't support it and libcurl has a hard-coded limit of
5 requests per pipeline.
2012-07-06 19:08:20 -04:00
Eelco Dolstra
1aba0bf0fa nix-store -r: do substitutions in parallel
I.e. when multiple non-derivation arguments are passed to ‘nix-store
-r’ to be substituted, do them in parallel.
2012-06-27 16:58:15 -04:00
Eelco Dolstra
42f5a2fc29 Mount an empty /dev/shm tmpfs in the chroot
This ensures that whatever the builder writes in /dev/shm is
automatically cleaned up.
2012-06-27 09:52:27 -04:00
Eelco Dolstra
3ee208516f Check the return code of the clone() call 2012-06-27 09:52:06 -04:00
Eelco Dolstra
1db38ae81b When using chroots, use a private PID namespace
In a private PID namespace, processes have PIDs that are separate from
the rest of the system.  The initial child gets PID 1.  Processes in
the chroot cannot see processes outside of the chroot.  This improves
isolation between builds.  However, processes on the outside can see
processes in the chroot and send signals to them (if they have
appropriate rights).

Since the builder gets PID 1, it serves as the reaper for zombies in
the chroot.  This might turn out to be a problem.  In that case we'll
need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 15:45:16 -04:00
Eelco Dolstra
5489086456 Use a private UTS namespace to provide a deterministic host/domain name to builders
In chroot builds, set the host name to "localhost" and the domain name
to "(none)" (the latter being the kernel's default).  This improves
determinism a bit further.

P.S. I have to idea what UTS stands for.
2012-06-25 14:12:17 -04:00
Eelco Dolstra
723a68c826 Improve error message 2012-06-23 00:57:14 -04:00
Eelco Dolstra
7ffa523fd1 In chroot builds, use a private SysV IPC namespace
This improves isolation a bit further, and it's just one extra flag in
the unshare() call.

P.S. It would be very cool to use CLONE_NEWPID (to put the builder in
a private PID namespace) as well, but that's slightly more risky since
having a builder start as PID 1 may cause problems.
2012-06-23 00:51:40 -04:00
Eelco Dolstra
df716c98d2 In chroot builds, use a private network namespace
On Linux it's possible to run a process in its own network namespace,
meaning that it gets its own set of network interfaces, disjunct from
the rest of the system.  We use this to completely remove network
access to chroot builds, except that they get a private loopback
interface.  This means that:

- Builders cannot connect to the outside network or to other processes
  on the same machine, except processes within the same build.

- Vice versa, other processes cannot connect to processes in a chroot
  build, and open ports/connections do not show up in "netstat".

- If two concurrent builders try to listen on the same port (e.g. as
  part of a test), they no longer conflict with each other.

This was inspired by the "PrivateNetwork" flag in systemd.
2012-06-23 00:28:35 -04:00
Eelco Dolstra
f5398d374b Compress build logs on the fly using bzip2 2012-05-30 10:12:29 -04:00
Eelco Dolstra
4bc4da331a Reserve some disk space for the garbage collector
We can't open a SQLite database if the disk is full.  Since this
prevents the garbage collector from running when it's most needed, we
reserve some dummy space that we can free just before doing a garbage
collection.  This actually revives some old code from the Berkeley DB
days.

Fixes #27.
2012-05-29 22:59:12 -04:00
Eelco Dolstra
2c26985835 Add option ‘build-keep-log’ to enable/disable writing of build logs
Fixes #26.
2012-05-29 16:42:05 -04:00
Eelco Dolstra
db5b86ef13 * Add an option ‘build-use-substitutes’, which can be set to ‘false’
to disable use of substitutes; i.e., force building from source.
  Fixes Nix/221.
2012-04-30 19:15:34 -04:00
Eelco Dolstra
46cdc6ad51 Handle EPERM when creating a hard link for the chroot
There is a race condition when doing parallel builds with chroots and
the immutable bit enabled.  One process may call makeImmutable()
before the other has called link(), in which case link() will fail
with EPERM.  We could retry or wrap the operation in a lock, but since
this condition is rare and I'm lazy, we just use the existing copy
fallback.

Fixes #9.
2012-04-30 10:58:04 -04:00
Eelco Dolstra
6387830c3b Merge pull request #2 from viric/master
Again, adding the sync option
2012-04-23 03:29:03 -07:00
Eelco Dolstra
1132dd27ea Fix obscure race condition in GC root creation
This should fix rare Hydra errors of the form:

error: symlinking `/nix/var/nix/gcroots/per-user/hydra/hydra-roots/7sfhs5fdmjxm8sqgcpd0pgcsmz1kq0l0-nixos-iso-0.1pre33785-33795' to `/nix/store/7sfhs5fdmjxm8sqgcpd0pgcsmz1kq0l0-nixos-iso-0.1pre33785-33795': File exists
2012-04-16 18:47:01 +02:00
Eelco Dolstra
31e34fcf93 Close almost all file descriptors in the builder
This regression was accidentally introduced in
35355fc1fc.
2012-04-15 00:20:32 +02:00
Eelco Dolstra
f2467eb6a0 On Linux, pretend we're building on Linux 2.6
Setting the UNAME26 personality causes "uname" to return "2.6.x",
regardless of the kernel version.  This improves determinism in
a few misbehaved packages.
2012-04-05 13:03:19 +02:00
Eelco Dolstra
446b827bae Mac OS X fix 2012-03-26 23:18:36 +02:00
Eelco Dolstra
480dda0e42 Delete non-directory valid paths right away
It's unlikely that rename() is faster than unlink() on a regular file
or symlink, so don't bother.
2012-03-26 20:56:30 +02:00
Eelco Dolstra
117670be57 Make the garbage collector more concurrent
Make the garbage collector more concurrent by deleting valid paths
outside the region where we're holding the global GC lock.  This
should greatly reduce the time during which new builds are blocked,
since the deletion accounts for the vast majority of the time spent in
the GC.

To ensure that this is safe, the valid paths are invalidated and
renamed to some arbitrary path while we're holding the lock.  This
ensures that we when we finally delete the path, it's not a (newly)
valid or locked path.
2012-03-26 20:43:33 +02:00
Eelco Dolstra
8be1979f1a Remove the --max-links GC option
We don't need this anymore now that current filesystems support more
than 32,000 files in a directory.
2012-03-26 20:00:02 +02:00
Lluís Batlle i Rossell
85799bf89e Fixing the default of sync-before-registering
Setting 'false' as default, as suggested by Eelco.

I also added a comment about the setting in the code.
2012-03-22 20:05:54 +01:00
Lluís Batlle i Rossell
25de80e2b6 Adding a nix option to sync before registering a path, for non-ext*
filesystems.
2012-03-22 19:57:42 +01:00
Eelco Dolstra
19d9762ad5 nix-store --clear-failed-paths: Clear derivation outputs
If the argument to ‘nix-store --clear-failed-paths’ is a derivation,
then clear the failed state of its outputs.
2012-03-20 18:23:26 +01:00
Eelco Dolstra
b006a31d52 Drop the externals directory
Nix now requires SQLite and bzip2 to be pre-installed.  SQLite is
detected using pkg-config.  We required DBD::SQLite anyway, so
depending on SQLite is not a big problem.

The --with-bzip2, --with-openssl and --with-sqlite flags are gone.
2012-03-18 23:54:57 +01:00
Eelco Dolstra
bd50c01972 Ensure that Perl processes delete their entry in the temproots directory
By moving the destructor object to libstore.so, it's also run when
download-using-manifests and nix-prefetch-url exit.  This prevents
them from cluttering /nix/var/nix/temproots with stale files.
2012-03-13 17:07:49 +01:00
Eelco Dolstra
35355fc1fc Set the close-on-exec flag on file descriptors 2012-03-05 20:29:00 +01:00
Eelco Dolstra
7b22bec252 Don't leak a file descriptor in commonChildInit() 2012-03-05 20:28:09 +01:00
Eelco Dolstra
2b4964f319 Restore progress indication during nix-copy-closure 2012-03-05 18:13:44 +01:00
Eelco Dolstra
8afd28a922 Remove dependency on sqlite3_table_column_metadata
Not all SQLite builds have the function sqlite3_table_column_metadata.
We were only using it in a schema upgrade check for compatibility with
databases that were probably never seen in the wild.  So remove it.
2012-03-01 16:06:49 +01:00
Eelco Dolstra
1c94524458 Fix an uninitialised variable
The variable ‘useChroot’ was not initialised properly.  This caused
random failures if using the build hook.  Seen on Mac OS X 10.7 with Clang.
Thanks to KolibriFX for finding this :-)
2012-03-01 16:06:49 +01:00
Eelco Dolstra
6fe13e6aba Fix chroots builds
Chroots are initialised by hard-linking inputs from the Nix store to
the chroot.  This doesn't work if the input has its immutable bit set,
because it's forbidden to create hard links to immutable files.  So
temporarily clear the immutable bit when creating and destroying the
chroot.

Note that making regular files in the Nix store immutable isn't very
reliable, since the bit can easily become cleared: for instance, if we
run the garbage collector after running ‘nix-store --optimise’.  So
maybe we should only make directories immutable.
2012-02-18 01:23:52 +01:00
Eelco Dolstra
bd013b6f98 On Linux, make the Nix store really read-only by using the immutable bit
I was bitten one time too many by Python modifying the Nix store by
creating *.pyc files when run as root.  On Linux, we can prevent this
by setting the immutable bit on files and directories (as in ‘chattr
+i’).  This isn't supported by all filesystems, so it's not an error
if setting the bit fails.  The immutable bit is cleared by the garbage
collector before deleting a path.  The only tricky aspect is in
optimiseStore(), since it's forbidden to create hard links to an
immutable file.  Thus optimiseStore() temporarily clears the immutable
bit before creating the link.
2012-02-15 01:31:56 +01:00
Eelco Dolstra
5e57047d87 Fix a broken guard around utime()
Because of an outdated check for a timestamp of 0, we were calling
utime() even when it wasn't necessary.
2012-02-15 01:00:30 +01:00
Eelco Dolstra
58ac7a17a4 Don't use deletePath() to delete a single file 2012-02-15 00:28:01 +01:00
Eelco Dolstra
d5a5a83ad4 Use data() instead of c_str() where appropriate 2012-02-09 18:27:45 +01:00
Eelco Dolstra
a71d02440b * Oops. 2011-12-30 17:13:25 +00:00
Eelco Dolstra
6f5e3326ce * Move topoSortPaths() out of gc.cc. 2011-12-30 15:02:50 +00:00
Eelco Dolstra
b1004f40f7 * Reject a build if there is a cycle among the outputs. This is
necessary because existing code assumes that the references graph is
  acyclic.
2011-12-30 14:47:14 +00:00
Eelco Dolstra
254b3399ba * Sync with the trunk. 2011-12-30 13:08:14 +00:00
Eelco Dolstra
8c42a8c8ff * Make sure that lock files are cleaned up properly when building
through the build hook.
2011-12-25 16:38:37 +00:00
Eelco Dolstra
524fa8a4f1 * Oops. 2011-12-22 16:27:03 +00:00
Eelco Dolstra
b33da599c5 * In the garbage collector, delete invalid paths before deleting
unreachable paths.  This matters when using --max-freed etc.:
  unreachable paths could become reachable again, so it's nicer to
  keep them if there is "real" garbage to be deleted.  Also, don't use
  readDirectory() but read the Nix store and delete invalid paths in
  parallel.  This reduces GC latency on very large Nix stores.
2011-12-22 15:55:53 +00:00
Eelco Dolstra
58d974336c * Drop unnecessary call to canonPath() (nixStore is already canonical). 2011-12-22 14:33:34 +00:00
Eelco Dolstra
2aac7cd021 * Another case of lock file permissions being too liberal. 2011-12-21 19:17:45 +00:00
Shea Levy
b4cee3f816 Revert previous commit
It doesn't detect indirect references
2011-12-21 17:31:34 +00:00
Shea Levy
f3c88f297d Detect and reject mutually-referential outputs
There is probably a more efficient way to do this.
2011-12-21 17:14:28 +00:00
Eelco Dolstra
b5363810bb * Fix the build. 2011-12-20 16:37:01 +00:00
Eelco Dolstra
194d21f9f6 * Sync with the trunk. 2011-12-16 23:33:01 +00:00
Eelco Dolstra
273b288a7e * importPath() -> importPaths(). Because of buffering of the input
stream it's now necessary for the daemon to process the entire
  sequence of exported paths, rather than letting the client do it.
2011-12-16 22:31:25 +00:00
Eelco Dolstra
e0bd307802 * Make the import operation through the daemon much more efficient
(way fewer roundtrips) by allowing the client to send data in bigger
  chunks.
* Some refactoring.
2011-12-16 19:44:13 +00:00
Eelco Dolstra
5a1b9ed0aa * Refactoring: move sink/source buffering into separate classes.
* Buffer the HashSink.  This speeds up hashing a bit because it
  prevents lots of calls to the hash update functions (e.g. nix-hash
  went from 9.3s to 8.7s of user time on the closure of my
  /var/run/current-system).
2011-12-15 16:19:53 +00:00
Eelco Dolstra
3a48282b06 * Buffer writes in FdSink. This significantly reduces the number of
system calls / context switches when dumping a NAR and in the worker
  protocol.
2011-12-14 23:30:06 +00:00
Eelco Dolstra
c8c0380744 * Remove unnecessary quotes. showPaths() already adds quotes. 2011-12-05 21:04:20 +00:00
Eelco Dolstra
000160f5b9 * In ‘nix-store --verify --check-contents’, repair missing hashes
rather than complain about them.
2011-12-02 17:52:18 +00:00
Eelco Dolstra
24f863d86b * When doing "nix-store --add-fixed" without "--recursive" via the Nix
daemon (which is an error), print a nicer error message than
  "Connection reset by peer" or "broken pipe".
* In the daemon, log errors that occur during request parameter
  processing.
2011-12-01 13:48:48 +00:00
Eelco Dolstra
b1eb8f4249 * Get rid of some superfluous error messages if a substituter fails.
* Say "fetch" instead of "substitute".
2011-11-29 13:00:41 +00:00
Eelco Dolstra
5bbd693cae * Add an API function exportPaths() that provides the functionality of
‘nix-store --export’.
* Add a Perl module that provides the functionality of
  ‘nix-copy-closure --to’.  This is used by build-remote.pl so it no
  longer needs to start a separate nix-copy-closure process.  Also, it
  uses the Perl API to do the export, so it doesn't need to start a
  separate nix-store process either.  As a result, nix-copy-closure
  and build-remote.pl should no longer fail on very large closures due
  to an "Argument list too long" error.  (Note that having very many
  dependencies in a single derivation can still fail because the
  environment can become too large.  Can't be helped though.)
2011-11-23 15:13:37 +00:00