Commit graph

62 commits

Author SHA1 Message Date
Shea Levy
4d652875bd Add the pre-build hook.
This hook can be used to set system-specific per-derivation build
settings that don't fit into the derivation model and are too complex or
volatile to be hard-coded into nix. Currently, the pre-build hook can
only add chroot dirs/files through the interface, but it also has full
access to the chroot root.

The specific use case for this is systems where the operating system ABI
is more complex than just the kernel-support system calls. For example,
on OS X there is a set of system-provided frameworks that can reliably
be accessed by any program linked to them, no matter the version the
program is running on. Unfortunately, those frameworks do not
necessarily live in the same locations on each version of OS X, nor do
their dependencies, and thus nix needs to know the specific version of
OS X currently running in order to make those frameworks available. The
pre-build hook is a perfect mechanism for doing just that.
2015-04-18 16:56:02 -04:00
Shea Levy
fd6774e285 Revert "Add the pre-build hook."
Going to reimplement differently.

This reverts commit 1e4a4a2e9f.
2015-04-18 14:59:58 -04:00
Eelco Dolstra
035aeb9547 Fix using restricted mode with chroots 2015-04-16 18:46:17 +02:00
Shea Levy
1e4a4a2e9f Add the pre-build hook.
This hook can be used to set system specific per-derivation build
settings that don't fit into the derivation model and are too complex or
volatile to be hard-coded into nix. Currently, the pre-build hook can
only add chroot dirs/files.

The specific use case for this is systems where the operating system ABI
is more complex than just the kernel-supported system calls. For
example, on OS X there is a set of system-provided frameworks that can
reliably be accessed by any program linked to them, no matter the
version the program is running on. Unfortunately, those frameworks do
not necessarily live in the same locations on each version of OS X, nor
do their dependencies, and thus nix needs to know the specific version
of OS X currently running in order to make those frameworks available.
The pre-build hook is a perfect mechanism for doing just that.
2015-04-12 12:56:38 -04:00
Eelco Dolstra
c1f04fae35 Implement a TTL on cached fetchurl/fetchTarball results
This is because we don't want to do HTTP requests on every evaluation,
even though we can prevent a full redownload via the cached ETag. The
default is one hour.
2015-04-09 11:55:36 +02:00
Eelco Dolstra
99897f6979 Use chroots for all derivations
If ‘build-use-chroot’ is set to ‘true’, fixed-output derivations are
now also chrooted. However, unlike normal derivations, they don't get
a private network namespace, so they can still access the
network. Also, the use of the ‘__noChroot’ derivation attribute is
no longer allowed.

Setting ‘build-use-chroot’ to ‘relaxed’ gives the old behaviour.
2015-02-23 15:54:31 +01:00
Eelco Dolstra
d37d012774 Settings: Add bool get() 2014-09-17 15:18:13 +02:00
Eelco Dolstra
f02c52e3d4 Doh 2014-08-05 10:19:57 +02:00
Eelco Dolstra
988bf59421 Move some options out of globals 2014-08-04 18:13:14 +02:00
Eelco Dolstra
daccd68999 Refactor 2014-08-04 18:02:29 +02:00
Eelco Dolstra
d5a076c36f Add option ‘build-extra-chroot-dirs’
This is useful for extending (rather than overriding) the default set
of chroot paths.
2014-08-04 18:00:00 +02:00
Eelco Dolstra
67937907ca nix-daemon: Pass on the user's $SSH_AUTH_SOCK to the SSH substituter 2014-07-25 18:02:48 +02:00
Eelco Dolstra
049c0eb49c nix-daemon: Add trusted-users and allowed-users options
‘trusted-users’ is a list of users and groups that have elevated
rights, such as the ability to specify binary caches. It defaults to
‘root’. A typical value would be ‘@wheel’ to specify all users in the
wheel group.

‘allowed-users’ is a list of users and groups that are allowed to
connect to the daemon. It defaults to ‘*’. A typical value would be
‘@users’ to specify the ‘users’ group.
2014-07-17 16:57:07 +02:00
Shea Levy
d62f46e500 Only add the importNative primop if the allow-arbitrary-code-during-evaluation option is true (default false) 2014-06-24 10:50:03 -04:00
Eelco Dolstra
9f9080e2c0 nix-store -l: Fetch build logs from the Internet
If a build log is not available locally, then ‘nix-store -l’ will now
try to download it from the servers listed in the ‘log-servers’ option
in nix.conf. For instance, if you have:

  log-servers = http://hydra.nixos.org/log

then it will try to get logs from http://hydra.nixos.org/log/<base
name of the store path>. So you can do things like:

  $ nix-store -l $(which xterm)

and get a log even if xterm wasn't built locally.
2014-05-21 17:19:36 +02:00
Shea Levy
df5de9dfd7 Add use-ssh-substituter setting.
It defaults to false and can be overridden by RemoteStore.

Untested currently, just quickly put this together
2014-02-26 13:32:47 +01:00
Eelco Dolstra
dba33d4018 Minor style fixes 2014-02-14 11:48:42 +01:00
Shea Levy
1614603165 Pass in params by const ref
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-10 06:49:37 -05:00
Shea Levy
64e23d0a38 Add download-via-ssh substituter
This substituter connects to a remote host, runs nix-store --serve
there, and then forwards substituter commands on to the remote host and
sends their results to the calling program. The ssh-substituter-hosts
option can be specified as a list of hosts to try.

This is an initial implementation and, while it works, it has some
limitations:

* Only the first host is used
* There is no caching of query results (all queries are sent to the
  remote machine)
* There is no informative output (such as progress bars)
* Some failure modes may cause unhelpful error messages
* There is no concept of trusted-ssh-substituter-hosts

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:33 -05:00
Eelco Dolstra
89e6781cc5 Make function calls show up in stack traces again
Note that adding --show-trace prevents functions calls from being
tail-recursive, so an expression that evaluates without --show-trace
may fail with a stack overflow if --show-trace is given.
2013-11-12 12:51:59 +01:00
Eelco Dolstra
936f9d45ba Don't apply the CPU affinity hack to nix-shell (and other Perl programs)
As discovered by Todd Veldhuizen, the shell started by nix-shell has
its affinity set to a single CPU.  This is because nix-shell connects
to the Nix daemon, which causes the affinity hack to be applied.  So
we turn this off for Perl programs.
2013-09-06 16:36:56 +02:00
Eelco Dolstra
efe4289464 Add an option to limit the log output of builders
This is mostly useful for Hydra to deal with builders that get stuck
in an infinite loop writing data to stdout/stderr.
2013-09-02 11:58:18 +02:00
Eelco Dolstra
bdd4646338 Revert "Prevent config.h from being clobbered"
This reverts commit 28bba8c44f.
2013-03-08 01:24:59 +01:00
Eelco Dolstra
28bba8c44f Prevent config.h from being clobbered 2013-03-07 23:55:55 +01:00
Ludovic Courtès
3e067ac11c Add `Settings::nixDaemonSocketFile'. 2013-02-19 10:19:18 +01:00
Eelco Dolstra
b7629778ef Allow mounting a path in a different location in the chroot
Fixes #24.
2012-12-29 23:04:02 +01:00
Eelco Dolstra
8b8ee53bc7 Add builtin constants ‘langVersion’ and ‘nixVersion’
The integer constant ‘langVersion’ denotes the current language
version.  It gets increased every time a language feature is
added/changed/removed.  It's currently 1.

The string constant ‘nixVersion’ contains the current Nix version,
e.g. "1.2pre2980_9de6bc5".
2012-11-27 13:29:55 +01:00
Eelco Dolstra
28bf183d2d Include <sys/types.h> for off_t
Reported by "gio" on IRC.
2012-09-25 13:00:19 -04:00
Eelco Dolstra
90d9c58d4d Pass all --option flags to the daemon 2012-07-31 18:19:44 -04:00
Eelco Dolstra
97421eb5ec Refactor settings processing
Put all Nix configuration flags in a Settings object.
2012-07-30 19:55:41 -04:00
Eelco Dolstra
d059bf48e4 Pass configuration settings to the substituters
Previously substituters could read nix.conf themselves, but this
didn't take --option flags into account.
2012-07-30 16:09:54 -04:00
Eelco Dolstra
b7fd2c2822 Use "#pragma once" to prevent repeated header file inclusion 2012-07-18 14:59:03 -04:00
Eelco Dolstra
993fa94fb4 * Move initialisation of variables like nixConfDir from libmain to
libstore so that the Perl bindings can use it as well.  It's vital
  that the Perl bindings use the configuration file, because otherwise
  nix-copy-closure will fail with a ‘database locked’ message if the
  value of ‘use-sqlite-wal’ is changed from the default.
2011-11-22 17:28:41 +00:00
Ludovic Courtès
5c9e9f732d Add support for the build-timeout' and --timeout' options. 2011-06-30 15:19:13 +00:00
Peter Simons
8b7f8b56f1 Added support for passing an (impure) NIX_BUILD_CORES variable to build expressions.
This patch adds the configuration file variable "build-cores" and the
command line argument "--cores". These settings specify the number of
CPU cores to utilize for parallel building within a job, i.e. by passing
an appropriate "-j" flag to GNU Make. The default value is 1, which
means that parallel building is *disabled*. If the number of build cores
is specified as 0 (synonymously: "guess" or "auto"), then the actual
value is supposed to be auto-detected by builders at run-time, i.e by
calling the nproc(1) utility from coreutils.

The environment variable $NIX_BUILD_CORES is available to builders, but
the contents of that variable does *not* influence the hash that goes
into the $out store path, i.e. the number of build cores to be utilized
can be changed at will without requiring any re-builds.
2010-06-23 14:34:08 +00:00
Eelco Dolstra
cacff1be88 * No longer block while waiting for a lock on a store path. Instead
poll for it (i.e. if we can't acquire the lock, then let the main
  select() loop wait for at most a few seconds and then try again).
  This improves parallelism: if two nix-store processes are both
  trying to build a path at the same time, the second one shouldn't
  block; it should first see if it can build other goals.  Also, it
  prevents the deadlocks that have been occuring in Hydra lately,
  where a process waits for a lock held by another process that's
  waiting for a lock held by the first.

  The downside is that polling isn't really elegant, but POSIX doesn't
  provide a way to wait for locks in a select() loop.  The only
  solution would be to spawn a thread for each lock to do a blocking
  fcntl() and then signal the main thread, but that would require
  pthreads.
2009-03-23 01:05:54 +00:00
Eelco Dolstra
60cb7de336 * Allow options from the Nix config file to be overriden from the
command line (e.g. "--option build-use-chroot true").
2009-02-27 11:04:41 +00:00
Eelco Dolstra
652817046b * Revert r13150: now that we use private namespaces for the chroot, we
don't have to put the chroot in /nix/var/nix/chroots anymore.
  They're back in /tmp now.
2008-12-11 17:52:34 +00:00
Eelco Dolstra
a55113411f * Nix daemon: reload the configuration file after forking (NIX-100). 2008-11-20 12:25:11 +00:00
Eelco Dolstra
a519bb0635 * Some somewhat ad hoc mechanism to allow the build farm to monitor
build progress.
2008-11-12 11:08:27 +00:00
Eelco Dolstra
709b55ee02 * Put the chroots under /nix/var/nix/chroots to reduce the risk of
disasters involving `rm -rf' on bind mounts.  Will try the
  definitive fix (per-process mounts, apparently possible via the
  CLONE_NEWNS flag in clone()) some other time.
2008-10-29 15:34:48 +00:00
Eelco Dolstra
c370755583 * Flag `--no-build-hook' to disable distributed builds.
* queryDeriver in daemon mode: don't barf if the other side returns an
  empty string (which means there is no deriver).
2007-11-16 16:15:26 +00:00
Eelco Dolstra
9e975458b4 * Get rid of the substitutes database table (NIX-47). Instead, if we
need any info on substitutable paths, we just call the substituters
  (such as download-using-manifests.pl) directly.  This means that
  it's no longer necessary for nix-pull to register substitutes or for
  nix-channel to clear them, which makes those operations much faster
  (NIX-95).  Also, we don't have to worry about keeping nix-pull
  manifests (in /nix/var/nix/manifests) and the database in sync with
  each other.

  The downside is that there is some overhead in calling an external
  program to get the substitutes info.  For instance, "nix-env -qas"
  takes a bit longer.

  Abolishing the substitutes table also makes the logic in
  local-store.cc simpler, as we don't need to store info for invalid
  paths.  On the downside, you cannot do things like "nix-store -qR"
  on a substitutable but invalid path (but nobody did that anyway).

* Never catch interrupts (the Interrupted exception).
2007-08-12 00:29:28 +00:00
Eelco Dolstra
9dbfe242e3 * Kill a build if it has gone for more than a certain number of
seconds without producing output on stdout or stderr (NIX-65).  This
  timeout can be specified using the `--max-silent-time' option or the
  `build-max-silent-time' configuration setting.  The default is
  infinity (0).

* Fix a tricky race condition: if we kill the build user before the
  child has done its setuid() to the build user uid, then it won't be
  killed, and we'll potentially lock up in pid.wait().  So also send a
  conventional kill to the child.
2006-12-08 15:44:00 +00:00
Eelco Dolstra
1e16d20655 * Install the worker in bindir, not libexecdir.
* Allow the worker path to be overriden through the NIX_WORKER
  environment variable.
2006-12-04 13:09:16 +00:00
Eelco Dolstra
6ecb840fd1 * Put building in the store API. 2006-11-30 18:02:04 +00:00
Eelco Dolstra
75068e7d75 * Use a proper namespace.
* Optimise header file usage a bit.
* Compile the parser as C++.
2006-09-04 21:06:23 +00:00
Eelco Dolstra
a4273156c4 * Use $(libexecdir) to find find-runtime-roots.pl. 2006-07-20 13:21:37 +00:00
Eelco Dolstra
d51aede4af * Allow the canonical system name to be specified at runtime in the
Nix config file.
2006-07-06 15:30:37 +00:00
Eelco Dolstra
345a95afe9 * Allow the size of the GC reserved file to be specified in nix.conf
through the new `gc-reserved-space' option.
2006-02-16 13:58:10 +00:00