nix-super/doc/manual/advanced-topics/distributed-builds.xml

<chapter xmlns="http://docbook.org/ns/docbook"
      xmlns:xlink="http://www.w3.org/1999/xlink"
      xmlns:xi="http://www.w3.org/2001/XInclude"
      version="5.0"
      xml:id='chap-distributed-builds'>

<title>Remote Builds</title>

<para>Nix supports remote builds, where a local Nix installation can
forward Nix builds to other machines.  This allows multiple builds to
be performed in parallel and allows Nix to perform multi-platform
builds in a semi-transparent way.  For instance, if you perform a
build for a <literal>x86_64-darwin</literal> on an
<literal>i686-linux</literal> machine, Nix can automatically forward
the build to a <literal>x86_64-darwin</literal> machine, if
available.</para>

<para>To forward a build to a remote machine, it’s required that the
remote machine is accessible via SSH and that it has Nix
installed. You can test whether connecting to the remote Nix instance
works, e.g.

<screen>
$ nix ping-store --store ssh://mac
</screen>

will try to connect to the machine named <literal>mac</literal>. It is
possible to specify an SSH identity file as part of the remote store
URI, e.g.

<screen>
$ nix ping-store --store ssh://mac?ssh-key=/home/alice/my-key
</screen>

Since builds should be non-interactive, the key should not have a
passphrase. Alternatively, you can load identities ahead of time into
<command>ssh-agent</command> or <command>gpg-agent</command>.</para>

<para>If you get the error

<screen>
bash: nix-store: command not found
error: cannot connect to 'mac'
</screen>

then you need to ensure that the <literal>PATH</literal> of
non-interactive login shells contains Nix.</para>

<warning><para>If you are building via the Nix daemon, it is the Nix
daemon user account (that is, <literal>root</literal>) that should
have SSH access to the remote machine. If you can’t or don’t want to
configure <literal>root</literal> to be able to access to remote
machine, you can use a private Nix store instead by passing
e.g. <literal>--store ~/my-nix</literal>.</para></warning>

<para>The list of remote machines can be specified on the command line
or in the Nix configuration file. The former is convenient for
testing. For example, the following command allows you to build a
derivation for <literal>x86_64-darwin</literal> on a Linux machine:

<screen>
$ uname
Linux

$ nix build \
  '(with import &lt;nixpkgs> { system = "x86_64-darwin"; }; runCommand "foo" {} "uname > $out")' \
  --builders 'ssh://mac x86_64-darwin'
[1/0/1 built, 0.0 MiB DL] building foo on ssh://mac

$ cat ./result
Darwin
</screen>

It is possible to specify multiple builders separated by a semicolon
or a newline, e.g.

<screen>
  --builders 'ssh://mac x86_64-darwin ; ssh://beastie x86_64-freebsd'
</screen>
</para>

<para>Each machine specification consists of the following elements,
separated by spaces. Only the first element is required.
To leave a field at its default, set it to <literal>-</literal>.

<orderedlist>

  <listitem><para>The URI of the remote store in the format
  <literal>ssh://[<emphasis>username</emphasis>@]<emphasis>hostname</emphasis></literal>,
  e.g. <literal>ssh://nix@mac</literal> or
  <literal>ssh://mac</literal>. For backward compatibility,
  <literal>ssh://</literal> may be omitted. The hostname may be an
  alias defined in your
  <filename>~/.ssh/config</filename>.</para></listitem>

  <listitem><para>A comma-separated list of Nix platform type
  identifiers, such as <literal>x86_64-darwin</literal>.  It is
  possible for a machine to support multiple platform types, e.g.,
  <literal>i686-linux,x86_64-linux</literal>. If omitted, this
  defaults to the local platform type.</para></listitem>

  <listitem><para>The SSH identity file to be used to log in to the
  remote machine. If omitted, SSH will use its regular
  identities.</para></listitem>

  <listitem><para>The maximum number of builds that Nix will execute
  in parallel on the machine.  Typically this should be equal to the
  number of CPU cores.  For instance, the machine
  <literal>itchy</literal> in the example will execute up to 8 builds
  in parallel.</para></listitem>

  <listitem><para>The “speed factor”, indicating the relative speed of
  the machine.  If there are multiple machines of the right type, Nix
  will prefer the fastest, taking load into account.</para></listitem>

  <listitem><para>A comma-separated list of <emphasis>supported
  features</emphasis>.  If a derivation has the
  <varname>requiredSystemFeatures</varname> attribute, then Nix will
  only perform the derivation on a machine that has the specified
  features.  For instance, the attribute

<programlisting>
requiredSystemFeatures = [ "kvm" ];
</programlisting>

  will cause the build to be performed on a machine that has the
  <literal>kvm</literal> feature.</para></listitem>

  <listitem><para>A comma-separated list of <emphasis>mandatory
  features</emphasis>.  A machine will only be used to build a
  derivation if all of the machine’s mandatory features appear in the
  derivation’s <varname>requiredSystemFeatures</varname>
  attribute..</para></listitem>

</orderedlist>

For example, the machine specification

<programlisting>
nix@scratchy.labs.cs.uu.nl  i686-linux      /home/nix/.ssh/id_scratchy_auto        8 1 kvm
nix@itchy.labs.cs.uu.nl     i686-linux      /home/nix/.ssh/id_scratchy_auto        8 2
nix@poochie.labs.cs.uu.nl   i686-linux      /home/nix/.ssh/id_scratchy_auto        1 2 kvm benchmark
</programlisting>

specifies several machines that can perform
<literal>i686-linux</literal> builds. However,
<literal>poochie</literal> will only do builds that have the attribute

<programlisting>
requiredSystemFeatures = [ "benchmark" ];
</programlisting>

or

<programlisting>
requiredSystemFeatures = [ "benchmark" "kvm" ];
</programlisting>

<literal>itchy</literal> cannot do builds that require
<literal>kvm</literal>, but <literal>scratchy</literal> does support
such builds. For regular builds, <literal>itchy</literal> will be
preferred over <literal>scratchy</literal> because it has a higher
speed factor.</para>

<para>Remote builders can also be configured in
<filename>nix.conf</filename>, e.g.

<programlisting>
builders = ssh://mac x86_64-darwin ; ssh://beastie x86_64-freebsd
</programlisting>

Finally, remote builders can be configured in a separate configuration
file included in <option>builders</option> via the syntax
<literal>@<emphasis>file</emphasis></literal>. For example,

<programlisting>
builders = @/etc/nix/machines
</programlisting>

causes the list of machines in <filename>/etc/nix/machines</filename>
to be included. (This is the default.)</para>

<para>If you want the builders to use caches, you likely want to set
the option <link linkend='conf-builders-use-substitutes'><literal>builders-use-substitutes</literal></link>
in your local <filename>nix.conf</filename>.</para>

<para>To build only on remote builders and disable building on the local machine,
you can use the option <option>--max-jobs 0</option>.</para>

</chapter>