Commit graph

613 commits

Author SHA1 Message Date
Eelco Dolstra
1db38ae81b When using chroots, use a private PID namespace
In a private PID namespace, processes have PIDs that are separate from
the rest of the system.  The initial child gets PID 1.  Processes in
the chroot cannot see processes outside of the chroot.  This improves
isolation between builds.  However, processes on the outside can see
processes in the chroot and send signals to them (if they have
appropriate rights).

Since the builder gets PID 1, it serves as the reaper for zombies in
the chroot.  This might turn out to be a problem.  In that case we'll
need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 15:45:16 -04:00
Eelco Dolstra
5489086456 Use a private UTS namespace to provide a deterministic host/domain name to builders
In chroot builds, set the host name to "localhost" and the domain name
to "(none)" (the latter being the kernel's default).  This improves
determinism a bit further.

P.S. I have to idea what UTS stands for.
2012-06-25 14:12:17 -04:00
Eelco Dolstra
723a68c826 Improve error message 2012-06-23 00:57:14 -04:00
Eelco Dolstra
7ffa523fd1 In chroot builds, use a private SysV IPC namespace
This improves isolation a bit further, and it's just one extra flag in
the unshare() call.

P.S. It would be very cool to use CLONE_NEWPID (to put the builder in
a private PID namespace) as well, but that's slightly more risky since
having a builder start as PID 1 may cause problems.
2012-06-23 00:51:40 -04:00
Eelco Dolstra
df716c98d2 In chroot builds, use a private network namespace
On Linux it's possible to run a process in its own network namespace,
meaning that it gets its own set of network interfaces, disjunct from
the rest of the system.  We use this to completely remove network
access to chroot builds, except that they get a private loopback
interface.  This means that:

- Builders cannot connect to the outside network or to other processes
  on the same machine, except processes within the same build.

- Vice versa, other processes cannot connect to processes in a chroot
  build, and open ports/connections do not show up in "netstat".

- If two concurrent builders try to listen on the same port (e.g. as
  part of a test), they no longer conflict with each other.

This was inspired by the "PrivateNetwork" flag in systemd.
2012-06-23 00:28:35 -04:00
Eelco Dolstra
f5398d374b Compress build logs on the fly using bzip2 2012-05-30 10:12:29 -04:00
Eelco Dolstra
4bc4da331a Reserve some disk space for the garbage collector
We can't open a SQLite database if the disk is full.  Since this
prevents the garbage collector from running when it's most needed, we
reserve some dummy space that we can free just before doing a garbage
collection.  This actually revives some old code from the Berkeley DB
days.

Fixes #27.
2012-05-29 22:59:12 -04:00
Eelco Dolstra
2c26985835 Add option ‘build-keep-log’ to enable/disable writing of build logs
Fixes #26.
2012-05-29 16:42:05 -04:00
Eelco Dolstra
db5b86ef13 * Add an option ‘build-use-substitutes’, which can be set to ‘false’
to disable use of substitutes; i.e., force building from source.
  Fixes Nix/221.
2012-04-30 19:15:34 -04:00
Eelco Dolstra
46cdc6ad51 Handle EPERM when creating a hard link for the chroot
There is a race condition when doing parallel builds with chroots and
the immutable bit enabled.  One process may call makeImmutable()
before the other has called link(), in which case link() will fail
with EPERM.  We could retry or wrap the operation in a lock, but since
this condition is rare and I'm lazy, we just use the existing copy
fallback.

Fixes #9.
2012-04-30 10:58:04 -04:00
Eelco Dolstra
6387830c3b Merge pull request #2 from viric/master
Again, adding the sync option
2012-04-23 03:29:03 -07:00
Eelco Dolstra
1132dd27ea Fix obscure race condition in GC root creation
This should fix rare Hydra errors of the form:

error: symlinking `/nix/var/nix/gcroots/per-user/hydra/hydra-roots/7sfhs5fdmjxm8sqgcpd0pgcsmz1kq0l0-nixos-iso-0.1pre33785-33795' to `/nix/store/7sfhs5fdmjxm8sqgcpd0pgcsmz1kq0l0-nixos-iso-0.1pre33785-33795': File exists
2012-04-16 18:47:01 +02:00
Eelco Dolstra
31e34fcf93 Close almost all file descriptors in the builder
This regression was accidentally introduced in
35355fc1fc.
2012-04-15 00:20:32 +02:00
Eelco Dolstra
f2467eb6a0 On Linux, pretend we're building on Linux 2.6
Setting the UNAME26 personality causes "uname" to return "2.6.x",
regardless of the kernel version.  This improves determinism in
a few misbehaved packages.
2012-04-05 13:03:19 +02:00
Eelco Dolstra
446b827bae Mac OS X fix 2012-03-26 23:18:36 +02:00
Eelco Dolstra
480dda0e42 Delete non-directory valid paths right away
It's unlikely that rename() is faster than unlink() on a regular file
or symlink, so don't bother.
2012-03-26 20:56:30 +02:00
Eelco Dolstra
117670be57 Make the garbage collector more concurrent
Make the garbage collector more concurrent by deleting valid paths
outside the region where we're holding the global GC lock.  This
should greatly reduce the time during which new builds are blocked,
since the deletion accounts for the vast majority of the time spent in
the GC.

To ensure that this is safe, the valid paths are invalidated and
renamed to some arbitrary path while we're holding the lock.  This
ensures that we when we finally delete the path, it's not a (newly)
valid or locked path.
2012-03-26 20:43:33 +02:00
Eelco Dolstra
8be1979f1a Remove the --max-links GC option
We don't need this anymore now that current filesystems support more
than 32,000 files in a directory.
2012-03-26 20:00:02 +02:00
Lluís Batlle i Rossell
85799bf89e Fixing the default of sync-before-registering
Setting 'false' as default, as suggested by Eelco.

I also added a comment about the setting in the code.
2012-03-22 20:05:54 +01:00
Lluís Batlle i Rossell
25de80e2b6 Adding a nix option to sync before registering a path, for non-ext*
filesystems.
2012-03-22 19:57:42 +01:00
Eelco Dolstra
19d9762ad5 nix-store --clear-failed-paths: Clear derivation outputs
If the argument to ‘nix-store --clear-failed-paths’ is a derivation,
then clear the failed state of its outputs.
2012-03-20 18:23:26 +01:00
Eelco Dolstra
b006a31d52 Drop the externals directory
Nix now requires SQLite and bzip2 to be pre-installed.  SQLite is
detected using pkg-config.  We required DBD::SQLite anyway, so
depending on SQLite is not a big problem.

The --with-bzip2, --with-openssl and --with-sqlite flags are gone.
2012-03-18 23:54:57 +01:00
Eelco Dolstra
bd50c01972 Ensure that Perl processes delete their entry in the temproots directory
By moving the destructor object to libstore.so, it's also run when
download-using-manifests and nix-prefetch-url exit.  This prevents
them from cluttering /nix/var/nix/temproots with stale files.
2012-03-13 17:07:49 +01:00
Eelco Dolstra
35355fc1fc Set the close-on-exec flag on file descriptors 2012-03-05 20:29:00 +01:00
Eelco Dolstra
7b22bec252 Don't leak a file descriptor in commonChildInit() 2012-03-05 20:28:09 +01:00
Eelco Dolstra
2b4964f319 Restore progress indication during nix-copy-closure 2012-03-05 18:13:44 +01:00
Eelco Dolstra
8afd28a922 Remove dependency on sqlite3_table_column_metadata
Not all SQLite builds have the function sqlite3_table_column_metadata.
We were only using it in a schema upgrade check for compatibility with
databases that were probably never seen in the wild.  So remove it.
2012-03-01 16:06:49 +01:00
Eelco Dolstra
1c94524458 Fix an uninitialised variable
The variable ‘useChroot’ was not initialised properly.  This caused
random failures if using the build hook.  Seen on Mac OS X 10.7 with Clang.
Thanks to KolibriFX for finding this :-)
2012-03-01 16:06:49 +01:00
Eelco Dolstra
6fe13e6aba Fix chroots builds
Chroots are initialised by hard-linking inputs from the Nix store to
the chroot.  This doesn't work if the input has its immutable bit set,
because it's forbidden to create hard links to immutable files.  So
temporarily clear the immutable bit when creating and destroying the
chroot.

Note that making regular files in the Nix store immutable isn't very
reliable, since the bit can easily become cleared: for instance, if we
run the garbage collector after running ‘nix-store --optimise’.  So
maybe we should only make directories immutable.
2012-02-18 01:23:52 +01:00
Eelco Dolstra
bd013b6f98 On Linux, make the Nix store really read-only by using the immutable bit
I was bitten one time too many by Python modifying the Nix store by
creating *.pyc files when run as root.  On Linux, we can prevent this
by setting the immutable bit on files and directories (as in ‘chattr
+i’).  This isn't supported by all filesystems, so it's not an error
if setting the bit fails.  The immutable bit is cleared by the garbage
collector before deleting a path.  The only tricky aspect is in
optimiseStore(), since it's forbidden to create hard links to an
immutable file.  Thus optimiseStore() temporarily clears the immutable
bit before creating the link.
2012-02-15 01:31:56 +01:00
Eelco Dolstra
5e57047d87 Fix a broken guard around utime()
Because of an outdated check for a timestamp of 0, we were calling
utime() even when it wasn't necessary.
2012-02-15 01:00:30 +01:00
Eelco Dolstra
58ac7a17a4 Don't use deletePath() to delete a single file 2012-02-15 00:28:01 +01:00
Eelco Dolstra
d5a5a83ad4 Use data() instead of c_str() where appropriate 2012-02-09 18:27:45 +01:00
Eelco Dolstra
a71d02440b * Oops. 2011-12-30 17:13:25 +00:00
Eelco Dolstra
6f5e3326ce * Move topoSortPaths() out of gc.cc. 2011-12-30 15:02:50 +00:00
Eelco Dolstra
b1004f40f7 * Reject a build if there is a cycle among the outputs. This is
necessary because existing code assumes that the references graph is
  acyclic.
2011-12-30 14:47:14 +00:00
Eelco Dolstra
254b3399ba * Sync with the trunk. 2011-12-30 13:08:14 +00:00
Eelco Dolstra
8c42a8c8ff * Make sure that lock files are cleaned up properly when building
through the build hook.
2011-12-25 16:38:37 +00:00
Eelco Dolstra
524fa8a4f1 * Oops. 2011-12-22 16:27:03 +00:00
Eelco Dolstra
b33da599c5 * In the garbage collector, delete invalid paths before deleting
unreachable paths.  This matters when using --max-freed etc.:
  unreachable paths could become reachable again, so it's nicer to
  keep them if there is "real" garbage to be deleted.  Also, don't use
  readDirectory() but read the Nix store and delete invalid paths in
  parallel.  This reduces GC latency on very large Nix stores.
2011-12-22 15:55:53 +00:00
Eelco Dolstra
58d974336c * Drop unnecessary call to canonPath() (nixStore is already canonical). 2011-12-22 14:33:34 +00:00
Eelco Dolstra
2aac7cd021 * Another case of lock file permissions being too liberal. 2011-12-21 19:17:45 +00:00
Shea Levy
b4cee3f816 Revert previous commit
It doesn't detect indirect references
2011-12-21 17:31:34 +00:00
Shea Levy
f3c88f297d Detect and reject mutually-referential outputs
There is probably a more efficient way to do this.
2011-12-21 17:14:28 +00:00
Eelco Dolstra
b5363810bb * Fix the build. 2011-12-20 16:37:01 +00:00
Eelco Dolstra
194d21f9f6 * Sync with the trunk. 2011-12-16 23:33:01 +00:00
Eelco Dolstra
273b288a7e * importPath() -> importPaths(). Because of buffering of the input
stream it's now necessary for the daemon to process the entire
  sequence of exported paths, rather than letting the client do it.
2011-12-16 22:31:25 +00:00
Eelco Dolstra
e0bd307802 * Make the import operation through the daemon much more efficient
(way fewer roundtrips) by allowing the client to send data in bigger
  chunks.
* Some refactoring.
2011-12-16 19:44:13 +00:00
Eelco Dolstra
5a1b9ed0aa * Refactoring: move sink/source buffering into separate classes.
* Buffer the HashSink.  This speeds up hashing a bit because it
  prevents lots of calls to the hash update functions (e.g. nix-hash
  went from 9.3s to 8.7s of user time on the closure of my
  /var/run/current-system).
2011-12-15 16:19:53 +00:00
Eelco Dolstra
3a48282b06 * Buffer writes in FdSink. This significantly reduces the number of
system calls / context switches when dumping a NAR and in the worker
  protocol.
2011-12-14 23:30:06 +00:00