Commit graph

46 commits

Author SHA1 Message Date
Eelco Dolstra
59a26360c7 Support mandatory system features in the build hook
Mandatory features are features that MUST be present in a derivation's
requiredSystemFeatures attribute.  One application is performance
testing, where we have a dedicated machine to run performance tests
(and nothing else).  Then we would add the label "perf" to the
machine's mandatory features and to the performance testing
derivations.
2012-04-30 17:22:45 -04:00
Eelco Dolstra
6de5d53416 Fix a warning in the build hook about $progressViewer 2012-04-24 12:56:30 +02:00
Eelco Dolstra
56042a120a build-remote.pl: don't wait forever for the upload lock
In the build hook, don't wait forever to get the upload lock.  This
ensures progress if another process gets stuck while holding the
upload lock.
2012-03-05 17:58:09 +01:00
Eelco Dolstra
4d728bc3e6 * Security: make sure the lock files used by build-remote.pl are not
readable to other users.  Otherwise, any user can open the lock file
  for reading and lock it, thus DoSing the remote build mechanism.
2011-12-21 19:11:58 +00:00
Eelco Dolstra
69d6f0936a * Use a lock to ensure that only one build-remote instance can copy a
closure to a given machine at the same time.  This prevents the case
  where multiple instances try to copy the same missing store path to
  the target machine, which is very wasteful.
2011-12-21 18:59:25 +00:00
Eelco Dolstra
1df120cb05 * Get rid of the shell in ssh calls. 2011-11-29 10:51:22 +00:00
Eelco Dolstra
5bbd693cae * Add an API function exportPaths() that provides the functionality of
‘nix-store --export’.
* Add a Perl module that provides the functionality of
  ‘nix-copy-closure --to’.  This is used by build-remote.pl so it no
  longer needs to start a separate nix-copy-closure process.  Also, it
  uses the Perl API to do the export, so it doesn't need to start a
  separate nix-store process either.  As a result, nix-copy-closure
  and build-remote.pl should no longer fail on very large closures due
  to an "Argument list too long" error.  (Note that having very many
  dependencies in a single derivation can still fail because the
  environment can become too large.  Can't be helped though.)
2011-11-23 15:13:37 +00:00
Eelco Dolstra
ab20af3e6f * build-remote.pl: drop a hard-coded reference to /nix/etc/nix. 2011-11-23 12:21:35 +00:00
Eelco Dolstra
c362e4d718 * Move SSH.pm. 2011-10-11 11:45:36 +00:00
Eelco Dolstra
d787285af9 * nix-instantiate: return exit status 100 to denote a permanent build
failure.  The build hook can use this to distinguish between
  transient and permanent failures on the remote side.
2010-12-13 16:53:23 +00:00
Eelco Dolstra
365f3028dd * Use CamelCase for the Perl modules. 2010-12-05 17:50:29 +00:00
Eelco Dolstra
bfa6ee7d91 * Don't use SSH's `-tt' flag because it doesn't seem to work
on OpenSolaris when using connection sharing.  Instead have
  the remote side check for disconnection and kill the process 
  group when that happens.
2010-10-04 12:30:46 +00:00
Eelco Dolstra
bf0dde9597 * Always print hook output on stderr, even if --no-build-output is
set.
* In the build hook, print a trace message to allow Hydra to pick up
  the name of the remote machine used for the build.
2010-08-31 12:36:24 +00:00
Eelco Dolstra
80e722278c * When using the build hook, distinguish between the stderr of the
hook script proper, and the stdout/stderr of the builder.  Only the
  latter should be saved in /nix/var/log/nix/drvs.
* Allow the verbosity to be set through an option.
* Added a flag --quiet to lower the verbosity level.
2010-08-30 14:53:03 +00:00
Eelco Dolstra
766f708418 * Experimental feature: allow a derivation to tell the build hook that
it requires a certain feature on the build machine, e.g.

    requiredSystemFeatures = [ "kvm" ];

  We need this in Hydra to make sure that builds that require KVM
  support are forwarded to machines that have KVM support.  Probably
  this should also be enforced for local builds.
2010-08-27 13:18:13 +00:00
Eelco Dolstra
e437b08250 * Made the build hook mechanism more efficient. Rather than starting
the hook every time we want to ask whether we can run a remote build
  (which can be very often), we now reuse a hook process for answering
  those queries until it accepts a build.  So if there are N
  derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
Eelco Dolstra
1a396f3789 * Don't call "cat". 2010-08-25 12:19:30 +00:00
Eelco Dolstra
95deba581d * In the build hook, temporarily register the derivation and its
output as GC roots.  This prevents a race if the garbage collector
  is running during the build.
2010-08-25 11:54:11 +00:00
Eelco Dolstra
819548d92f * Pass `--fallback' to the remote build to ignore failing
substituters.
2010-06-22 14:41:22 +00:00
Eelco Dolstra
7ec5a65925 * Doh! The scope of $slotLock should extend to the end of the script,
because otherwise the lock will be released at the end of the while
  loop.
2010-02-04 02:18:29 +00:00
Eelco Dolstra
c9b2d80bcd * Typo. 2010-02-04 02:05:34 +00:00
Eelco Dolstra
408913bbaf * Revert r19796 for now. 2010-02-04 00:12:57 +00:00
Eelco Dolstra
c45de33c67 * Respect @sysconfdir@. 2010-02-03 21:13:37 +00:00
Eelco Dolstra
d0c32dc135 * In the build hook, if connecting to a machine fails, try the other
machines of the right type (if available).  This makes the build
  farm more robust to failures.
2010-02-03 20:35:37 +00:00
Eelco Dolstra
f56a039775 * Use SSH connection sharing in the remote build script.
* Removed the Cygwin password hack since the problem is apparently
  fixed in Visual Studio.
2010-02-03 20:12:18 +00:00
Eelco Dolstra
50e34891f0 * Disable gzip compression in build-remote.pl because it puts too much
load on the Hydra build farm (where it's unnecessary anyway because
  it has a fast connection to the build machines).  In any case,
  compression can be enabled by using the `-C' option to ssh.
2010-01-25 16:14:45 +00:00
Eelco Dolstra
997db91e07 * Don't pass -K. It should really inherit the setting of the calling
Nix though.
2009-11-17 16:22:39 +00:00
Eelco Dolstra
e1df4ef73c 2009-09-17 17:02:14 +00:00
Eelco Dolstra
86408b3f47 * build-remote.pl: Pick machines in a round-robin order, rather than
giving jobs to the first machine until it hits its job limit, then
  the second machine and so on.  This should improve utilisation of
  the Hydra build farm a lot.  Also take an optional speed factor
  into account to cause fast machines to be preferred over slower
  machines with a similar load.
2009-09-17 15:48:17 +00:00
Eelco Dolstra
57e0d73c77 * build-remote.pl: allow the system type to be a comma-separated list
of system types.  Don't treat the x86_64-linux system type
  specially.
2009-09-17 13:51:04 +00:00
Eelco Dolstra
c710fe540e * Machines of type x86_64-linux can do i686-linux builds. 2009-04-21 12:42:47 +00:00
Eelco Dolstra
096affb55b * Update the mtime on the lock file to make it easy to see when a slot
was last used.
2009-03-29 18:40:44 +00:00
Eelco Dolstra
737423a89c * Use polling to wait for a remote build slot when using a build hook
(that is, call the build hook with a certain interval until it
  accepts the build).
* build-remote.pl was totally broken: for all system types other than
  the local system type, it would send all builds to the *first*
  machine of the appropriate type.
2009-03-29 18:00:45 +00:00
Eelco Dolstra
47706e3924 * nix-copy-closure: compute the closure in one call to nix-store,
which is much faster.
2009-03-28 21:10:29 +00:00
Eelco Dolstra
3a2bbe7f8a * Simplify communication with the hook a bit (don't use file
descriptors 3/4, just use stdin/stderr).
2009-03-28 19:29:55 +00:00
Eelco Dolstra
7fb548aa26 * Clean up the output a bit. 2009-03-28 16:12:46 +00:00
Eelco Dolstra
1bb0f1e84b * Build hook: compress the transferred data. 2009-02-03 10:34:15 +00:00
Eelco Dolstra
b682fae9d9 * Build hooks: use nix-store --import. This prevents a redundant
scan for runtime dependencies (i.e. the local machine shouldn't do a
  scan that the remote machine has already done).  Also pipe directly
  into `nix-store --import': don't use a temporary file.
2009-02-02 17:24:10 +00:00
Eelco Dolstra
019176137f * When using a build hook, distinguish between transient failures
(e.g. an SSH connection problem) and permanent failures (i.e. the
  builder failed).  This matters to Hydra (it wants to know whether it
  makes sense to retry a build).
2009-01-13 11:39:09 +00:00
Eelco Dolstra
d91dc086bb 2008-12-04 16:51:36 +00:00
Eelco Dolstra
9ac3f5df9c * Propagate --max-silent-time to remote machines. 2008-12-04 16:51:16 +00:00
Eelco Dolstra
bcfe98acff * Prefer building on a remote machine over a local machine. This
makes more sense for the build farm, otherwise every nix-store
  invocation will lead to at least one local build.  Will come up with
  a better solution later...
2008-12-04 16:35:47 +00:00
Eelco Dolstra
909fbb9de1 * When using build hooks, for any nix-store -r build operation, it is
necessary that at least one build hook doesn't return "postpone",
  otherwise nix-store will barf ("waiting for a build slot, yet there
  are no running children").  So inform the build hook when this is
  the case, so that it can start a build even when that would exceed
  the maximum load on a machine.
2008-12-04 14:29:41 +00:00
Eelco Dolstra
5dfba0b4db * Force allocation of a pseudo-terminal to clean up the remote
nix-store process when the connection is interrupted.
2008-12-04 13:36:52 +00:00
Eelco Dolstra
9fd9c4c635 * Support multiple system types per remote machine, e.g. a machine
list like

    root@example.org x86_64-linux /root/.ssh/id_buildfarm 1
    root@example.org i686-darwin  /root/.ssh/id_buildfarm 1

  This is possible when the Nix installation on example.org itself has
  remote builds enabled.
2008-12-04 12:20:06 +00:00
Eelco Dolstra
bfec245efa * Add build-remote.pl to the Nix distribution. 2007-11-15 14:28:08 +00:00