nix-super/doc/manual/src/language/advanced-attributes.md

362 lines
16 KiB
Markdown
Raw Normal View History

2020-07-23 00:17:48 +03:00
# Advanced Attributes
Derivations can declare some infrequently used optional attributes.
- [`allowedReferences`]{#adv-attr-allowedReferences}\
2020-07-23 00:17:48 +03:00
The optional attribute `allowedReferences` specifies a list of legal
references (dependencies) of the output of the builder. For example,
2020-07-31 16:43:25 +03:00
```nix
allowedReferences = [];
```
2020-07-23 00:17:48 +03:00
enforces that the output of a derivation cannot have any runtime
dependencies on its inputs. To allow an output to have a runtime
dependency on itself, use `"out"` as a list item. This is used in
NixOS to check that generated files such as initial ramdisks for
booting Linux dont have accidental dependencies on other paths in
the Nix store.
- [`allowedRequisites`]{#adv-attr-allowedRequisites}\
2020-07-23 00:17:48 +03:00
This attribute is similar to `allowedReferences`, but it specifies
the legal requisites of the whole closure, so all the dependencies
recursively. For example,
2020-07-31 16:43:25 +03:00
```nix
allowedRequisites = [ foobar ];
```
2020-07-23 00:17:48 +03:00
enforces that the output of a derivation cannot have any other
runtime dependency than `foobar`, and in addition it enforces that
`foobar` itself doesn't introduce any other dependency itself.
- [`disallowedReferences`]{#adv-attr-disallowedReferences}\
2020-07-23 00:17:48 +03:00
The optional attribute `disallowedReferences` specifies a list of
illegal references (dependencies) of the output of the builder. For
example,
2020-07-31 16:43:25 +03:00
```nix
disallowedReferences = [ foo ];
```
2020-07-23 00:17:48 +03:00
enforces that the output of a derivation cannot have a direct
runtime dependencies on the derivation `foo`.
- [`disallowedRequisites`]{#adv-attr-disallowedRequisites}\
2020-07-23 00:17:48 +03:00
This attribute is similar to `disallowedReferences`, but it
specifies illegal requisites for the whole closure, so all the
dependencies recursively. For example,
2020-07-31 16:43:25 +03:00
```nix
disallowedRequisites = [ foobar ];
```
2020-07-23 00:17:48 +03:00
enforces that the output of a derivation cannot have any runtime
dependency on `foobar` or any other derivation depending recursively
on `foobar`.
- [`exportReferencesGraph`]{#adv-attr-exportReferencesGraph}\
2020-07-23 00:17:48 +03:00
This attribute allows builders access to the references graph of
their inputs. The attribute is a list of inputs in the Nix store
2020-07-31 16:43:25 +03:00
whose references graph the builder needs to know. The value of
this attribute should be a list of pairs `[ name1 path1 name2
path2 ... ]`. The references graph of each *pathN* will be stored
in a text file *nameN* in the temporary build directory. The text
files have the format used by `nix-store --register-validity`
(with the deriver fields left empty). For example, when the
following derivation is built:
2020-07-31 16:43:25 +03:00
```nix
derivation {
...
exportReferencesGraph = [ "libfoo-graph" libfoo ];
};
```
2020-07-23 00:17:48 +03:00
the references graph of `libfoo` is placed in the file
`libfoo-graph` in the temporary build directory.
2020-07-23 00:17:48 +03:00
`exportReferencesGraph` is useful for builders that want to do
something with the closure of a store path. Examples include the
builders in NixOS that generate the initial ramdisk for booting
Linux (a `cpio` archive containing the closure of the boot script)
and the ISO-9660 image for the installation CD (which is populated
with a Nix store containing the closure of a bootable NixOS
configuration).
- [`impureEnvVars`]{#adv-attr-impureEnvVars}\
2020-07-23 00:17:48 +03:00
This attribute allows you to specify a list of environment variables
that should be passed from the environment of the calling user to
the builder. Usually, the environment is cleared completely when the
builder is executed, but with this attribute you can allow specific
environment variables to be passed unmodified. For example,
`fetchurl` in Nixpkgs has the line
2020-07-31 16:43:25 +03:00
```nix
impureEnvVars = [ "http_proxy" "https_proxy" ... ];
```
2020-07-23 00:17:48 +03:00
to make it use the proxy server configuration specified by the user
2020-07-23 11:44:54 +03:00
in the environment variables `http_proxy` and friends.
2020-07-24 16:46:16 +03:00
This attribute is only allowed in *fixed-output derivations* (see
below), where impurities such as these are okay since (the hash
of) the output is known in advance. It is ignored for all other
derivations.
2020-07-23 00:17:48 +03:00
> **Warning**
>
2020-07-23 00:17:48 +03:00
> `impureEnvVars` implementation takes environment variables from
> the current builder process. When a daemon is building its
> environmental variables are used. Without the daemon, the
> environmental variables come from the environment of the
> `nix-build`.
If the [`configurable-impure-env` experimental
feature](@docroot@/contributing/experimental-features.md#xp-feature-configurable-impure-env)
is enabled, these environment variables can also be controlled
through the
[`impure-env`](@docroot@/command-ref/conf-file.md#conf-impure-env)
configuration setting.
- [`outputHash`]{#adv-attr-outputHash}; [`outputHashAlgo`]{#adv-attr-outputHashAlgo}; [`outputHashMode`]{#adv-attr-outputHashMode}\
2020-07-23 00:17:48 +03:00
These attributes declare that the derivation is a so-called
*fixed-output derivation*, which means that a cryptographic hash of
the output is already known in advance. When the build of a
fixed-output derivation finishes, Nix computes the cryptographic
hash of the output and compares it to the hash declared with these
attributes. If there is a mismatch, the build fails.
2020-07-23 00:17:48 +03:00
The rationale for fixed-output derivations is derivations such as
those produced by the `fetchurl` function. This function downloads a
file from a given URL. To ensure that the downloaded file has not
been modified, the caller must also specify a cryptographic hash of
the file. For example,
2020-07-31 16:43:25 +03:00
```nix
fetchurl {
url = "http://ftp.gnu.org/pub/gnu/hello/hello-2.1.1.tar.gz";
sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
}
```
2020-07-23 00:17:48 +03:00
It sometimes happens that the URL of the file changes, e.g., because
servers are reorganised or no longer available. We then must update
the call to `fetchurl`, e.g.,
2020-07-31 16:43:25 +03:00
```nix
fetchurl {
url = "ftp://ftp.nluug.nl/pub/gnu/hello/hello-2.1.1.tar.gz";
sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
}
```
2020-07-23 00:17:48 +03:00
If a `fetchurl` derivation was treated like a normal derivation, the
output paths of the derivation and *all derivations depending on it*
would change. For instance, if we were to change the URL of the
Glibc source distribution in Nixpkgs (a package on which almost all
other packages depend) massive rebuilds would be needed. This is
unfortunate for a change which we know cannot have a real effect as
it propagates upwards through the dependency graph.
2020-07-23 00:17:48 +03:00
For fixed-output derivations, on the other hand, the name of the
output path only depends on the `outputHash*` and `name` attributes,
while all other attributes are ignored for the purpose of computing
the output path. (The `name` attribute is included because it is
part of the path.)
2020-07-23 00:17:48 +03:00
As an example, here is the (simplified) Nix expression for
`fetchurl`:
2020-07-31 16:43:25 +03:00
```nix
{ stdenv, curl }: # The curl program is used for downloading.
{ url, sha256 }:
stdenv.mkDerivation {
name = baseNameOf (toString url);
builder = ./builder.sh;
buildInputs = [ curl ];
# This is a fixed-output derivation; the output must be a regular
# file with SHA256 hash sha256.
outputHashMode = "flat";
outputHashAlgo = "sha256";
outputHash = sha256;
inherit url;
}
```
The `outputHash` attribute must be a string containing the hash in either hexadecimal or "nix32" encoding, or following the format for integrity metadata as defined by [SRI](https://www.w3.org/TR/SRI/).
The "nix32" encoding is an adaptation of base-32 encoding.
The [`convertHash`](@docroot@/language/builtins.md#builtins-convertHash) function shows how to convert between different encodings, and the [`nix-hash` command](../command-ref/nix-hash.md) has information about obtaining the hash for some contents, as well as converting to and from encodings.
The `outputHashAlgo` attribute specifies the hash algorithm used to compute the hash.
It can currently be `"sha1"`, `"sha256"`, `"sha512"`, or `null`.
`outputHashAlgo` can only be `null` when `outputHash` follows the SRI format.
2020-07-23 00:17:48 +03:00
The `outputHashMode` attribute determines how the hash is computed.
It must be one of the following two values:
<!-- FIXME link to store object content-addressing not file system object content addressing once we have the page for that. -->
- `"flat"`
The output must be a non-executable regular file; if it isnt, the build fails.
The hash is
[simply computed over the contents of that file](@docroot@/store/file-system-object/content-address.md#serial-flat)
(so its equal to what Unix commands like `sha256sum` or `sha1sum` produce).
2020-07-23 00:17:48 +03:00
This is the default.
- `"recursive"` or `"nar"`
The hash is computed over the
[Nix Archive (NAR)](@docroot@/store/file-system-object/content-address.md#serial-nix-archive)
dump of the output (i.e., the result of [`nix-store --dump`](@docroot@/command-ref/nix-store/dump.md)).
In this case, the output is allowed to be any [file system object], including directories and more.
`"recursive"` is the traditional way of indicating this,
and is supported since 2005 (virtually the entire history of Nix).
`"nar"` is more clear, and consistent with other parts of Nix (such as the CLI),
however support for it is only added in Nix version 2.21.
- [`__contentAddressed`]{#adv-attr-__contentAddressed}
> **Warning**
> This attribute is part of an [experimental feature](@docroot@/contributing/experimental-features.md).
>
> To use this attribute, you must enable the
> [`ca-derivations`](@docroot@/contributing/experimental-features.md#xp-feature-ca-derivations) experimental feature.
> For example, in [nix.conf](../command-ref/conf-file.md) you could add:
>
> ```
> extra-experimental-features = ca-derivations
> ```
If this attribute is set to `true`, then the derivation
outputs will be stored in a content-addressed location rather than the
traditional input-addressed one.
Setting this attribute also requires setting
[`outputHashMode`](#adv-attr-outputHashMode)
and
[`outputHashAlgo`](#adv-attr-outputHashAlgo)
like for *fixed-output derivations* (see above).
2020-07-23 00:17:48 +03:00
2023-07-19 14:59:18 +03:00
It also implicitly requires that the machine to build the derivation must have the `ca-derivations` [system feature](@docroot@/command-ref/conf-file.md#conf-system-features).
- [`passAsFile`]{#adv-attr-passAsFile}\
2020-07-23 00:17:48 +03:00
A list of names of attributes that should be passed via files rather
than environment variables. For example, if you have
2020-07-31 16:43:25 +03:00
```nix
2020-07-23 00:17:48 +03:00
passAsFile = ["big"];
big = "a very long string";
```
2020-07-31 16:43:25 +03:00
then when the builder runs, the environment variable `bigPath`
will contain the absolute path to a temporary file containing `a
very long string`. That is, for any attribute *x* listed in
`passAsFile`, Nix will pass an environment variable `xPath`
holding the path of the file containing the value of attribute
*x*. This is useful when you need to pass large strings to a
builder, since most operating systems impose a limit on the size
of the environment (typically, a few hundred kilobyte).
2020-07-23 00:17:48 +03:00
- [`preferLocalBuild`]{#adv-attr-preferLocalBuild}\
If this attribute is set to `true` and [distributed building is enabled](@docroot@/command-ref/conf-file.md#conf-builders), then, if possible, the derivation will be built locally instead of being forwarded to a remote machine.
This is useful for derivations that are cheapest to build locally.
2020-07-23 00:17:48 +03:00
- [`allowSubstitutes`]{#adv-attr-allowSubstitutes}\
If this attribute is set to `false`, then Nix will always build this derivation (locally or remotely); it will not try to substitute its outputs.
This is useful for derivations that are cheaper to build than to substitute.
This attribute can be ignored by setting [`always-allow-substitutes`](@docroot@/command-ref/conf-file.md#conf-always-allow-substitutes) to `true`.
2020-07-23 00:17:48 +03:00
> **Note**
>
> If set to `false`, the [`builder`](./derivations.md#attr-builder) should be able to run on the system type specified in the [`system` attribute](./derivations.md#attr-system), since the derivation cannot be substituted.
- [`__structuredAttrs`]{#adv-attr-structuredAttrs}\
If the special attribute `__structuredAttrs` is set to `true`, the other derivation
structured attrs: improve support / usage of NIX_ATTRS_{SH,JSON}_FILE In #4770 I implemented proper `nix-shell(1)` support for derivations using `__structuredAttrs = true;`. Back then we decided to introduce two new environment variables, `NIX_ATTRS_SH_FILE` for `.attrs.sh` and `NIX_ATTRS_JSON_FILE` for `.attrs.json`. This was to avoid having to copy these files to `$NIX_BUILD_TOP` in a `nix-shell(1)` session which effectively meant copying these files to the project dir without cleaning up afterwords[1]. On last NixCon I resumed hacking on `__structuredAttrs = true;` by default for `nixpkgs` with a few other folks and getting back to it, I identified a few problems with the how it's used in `nixpkgs`: * A lot of builders in `nixpkgs` don't care about the env vars and assume that `.attrs.sh` and `.attrs.json` are in `$NIX_BUILD_TOP`. The sole reason why this works is that `nix-shell(1)` sources the contents of `.attrs.sh` and then sources `$stdenv/setup` if it exists. This may not be pretty, but it mostly works. One notable difference when using nixpkgs' stdenv as of now is however that `$__structuredAttrs` is set to `1` on regular builds, but set to an empty string in a shell session. Also, `.attrs.json` cannot be used in shell sessions because it can only be accessed by `$NIX_ATTRS_JSON_FILE` and not by `$NIX_BUILD_TOP/.attrs.json`. I considered changing Nix to be compatible with what nixpkgs effectively does, but then we'd have to either move $NIX_BUILD_TOP for shell sessions to a temporary location (and thus breaking a lot of assumptions) or we'd reintroduce all the problems we solved back then by using these two env vars. This is partly because I didn't document these variables back then (mea culpa), so I decided to drop all mentions of `.attrs.{json,sh}` in the manual and only refer to `$NIX_ATTRS_SH_FILE` and `$NIX_ATTRS_JSON_FILE`. The same applies to all our integration tests. Theoretically we could deprecated using `"$NIX_BUILD_TOP"/.attrs.sh` in the future now. * `nix develop` and `nix print-dev-env` don't support this environment variable at all even though they're supposed to be part of the replacement for `nix-shell` - for the drv debugging part to be precise. This isn't a big deal for the vast majority of derivations, i.e. derivations relying on nixpkgs' `stdenv` wiring things together properly. This is because `nix develop` effectively "clones" the derivation and replaces the builder with a script that dumps all of the environment, shell variables, functions etc, so the state of structured attrs being "sourced" is transmitted into the dev shell and most of the time you don't need to worry about `.attrs.sh` not existing because the shell is correctly configured and the if [ -e .attrs.sh ]; then source .attrs.sh; fi is simply omitted. However, this will break when having a derivation that reads e.g. from `.attrs.json` like with import <nixpkgs> {}; runCommand "foo" { __structuredAttrs = true; foo.bar = 23; } '' cat $NIX_ATTRS_JSON_FILE # doesn't work because it points to /build/.attrs.json '' To work around this I employed a similar approach as it exists for `nix-shell`: the `NIX_ATTRS_{JSON,SH}_FILE` vars are replaced with temporary locations. The contents of `.attrs.sh` and `.attrs.json` are now written into the JSON by `get-env.sh`, the builder that `nix develop` injects into the derivation it's debugging. So finally the exact file contents are present and exported by `nix develop`. I also made `.attrs.json` a JSON string in the JSON printed by `get-env.sh` on purpose because then it's not necessary to serialize the object structure again. `nix develop` only needs the JSON as string because it's only written into the temporary file. I'm not entirely sure if it makes sense to also use a temporary location for `nix print-dev-env` (rather than just skipping the rewrite in there), but this would probably break certain cases where it's relied upon `$NIX_ATTRS_SH_FILE` to exist (prime example are the `nix print-dev-env` test-cases I wrote in this patch using `tests/shell.nix`, these would fail because the env var exists, but it cannot read from it). [1] https://github.com/NixOS/nix/pull/4770#issuecomment-836799719
2023-09-24 14:19:04 +03:00
attributes are serialised into a file in JSON format. The environment variable
`NIX_ATTRS_JSON_FILE` points to the exact location of that file both in a build
and a [`nix-shell`](../command-ref/nix-shell.md). This obviates the need for
[`passAsFile`](#adv-attr-passAsFile) since JSON files have no size restrictions,
unlike process environments.
It also makes it possible to tweak derivation settings in a structured way; see
[`outputChecks`](#adv-attr-outputChecks) for example.
As a convenience to Bash builders,
structured attrs: improve support / usage of NIX_ATTRS_{SH,JSON}_FILE In #4770 I implemented proper `nix-shell(1)` support for derivations using `__structuredAttrs = true;`. Back then we decided to introduce two new environment variables, `NIX_ATTRS_SH_FILE` for `.attrs.sh` and `NIX_ATTRS_JSON_FILE` for `.attrs.json`. This was to avoid having to copy these files to `$NIX_BUILD_TOP` in a `nix-shell(1)` session which effectively meant copying these files to the project dir without cleaning up afterwords[1]. On last NixCon I resumed hacking on `__structuredAttrs = true;` by default for `nixpkgs` with a few other folks and getting back to it, I identified a few problems with the how it's used in `nixpkgs`: * A lot of builders in `nixpkgs` don't care about the env vars and assume that `.attrs.sh` and `.attrs.json` are in `$NIX_BUILD_TOP`. The sole reason why this works is that `nix-shell(1)` sources the contents of `.attrs.sh` and then sources `$stdenv/setup` if it exists. This may not be pretty, but it mostly works. One notable difference when using nixpkgs' stdenv as of now is however that `$__structuredAttrs` is set to `1` on regular builds, but set to an empty string in a shell session. Also, `.attrs.json` cannot be used in shell sessions because it can only be accessed by `$NIX_ATTRS_JSON_FILE` and not by `$NIX_BUILD_TOP/.attrs.json`. I considered changing Nix to be compatible with what nixpkgs effectively does, but then we'd have to either move $NIX_BUILD_TOP for shell sessions to a temporary location (and thus breaking a lot of assumptions) or we'd reintroduce all the problems we solved back then by using these two env vars. This is partly because I didn't document these variables back then (mea culpa), so I decided to drop all mentions of `.attrs.{json,sh}` in the manual and only refer to `$NIX_ATTRS_SH_FILE` and `$NIX_ATTRS_JSON_FILE`. The same applies to all our integration tests. Theoretically we could deprecated using `"$NIX_BUILD_TOP"/.attrs.sh` in the future now. * `nix develop` and `nix print-dev-env` don't support this environment variable at all even though they're supposed to be part of the replacement for `nix-shell` - for the drv debugging part to be precise. This isn't a big deal for the vast majority of derivations, i.e. derivations relying on nixpkgs' `stdenv` wiring things together properly. This is because `nix develop` effectively "clones" the derivation and replaces the builder with a script that dumps all of the environment, shell variables, functions etc, so the state of structured attrs being "sourced" is transmitted into the dev shell and most of the time you don't need to worry about `.attrs.sh` not existing because the shell is correctly configured and the if [ -e .attrs.sh ]; then source .attrs.sh; fi is simply omitted. However, this will break when having a derivation that reads e.g. from `.attrs.json` like with import <nixpkgs> {}; runCommand "foo" { __structuredAttrs = true; foo.bar = 23; } '' cat $NIX_ATTRS_JSON_FILE # doesn't work because it points to /build/.attrs.json '' To work around this I employed a similar approach as it exists for `nix-shell`: the `NIX_ATTRS_{JSON,SH}_FILE` vars are replaced with temporary locations. The contents of `.attrs.sh` and `.attrs.json` are now written into the JSON by `get-env.sh`, the builder that `nix develop` injects into the derivation it's debugging. So finally the exact file contents are present and exported by `nix develop`. I also made `.attrs.json` a JSON string in the JSON printed by `get-env.sh` on purpose because then it's not necessary to serialize the object structure again. `nix develop` only needs the JSON as string because it's only written into the temporary file. I'm not entirely sure if it makes sense to also use a temporary location for `nix print-dev-env` (rather than just skipping the rewrite in there), but this would probably break certain cases where it's relied upon `$NIX_ATTRS_SH_FILE` to exist (prime example are the `nix print-dev-env` test-cases I wrote in this patch using `tests/shell.nix`, these would fail because the env var exists, but it cannot read from it). [1] https://github.com/NixOS/nix/pull/4770#issuecomment-836799719
2023-09-24 14:19:04 +03:00
Nix writes a script that initialises shell variables
corresponding to all attributes that are representable in Bash. The
environment variable `NIX_ATTRS_SH_FILE` points to the exact
location of the script, both in a build and a
[`nix-shell`](../command-ref/nix-shell.md). This includes non-nested
(associative) arrays. For example, the attribute `hardening.format = true`
ends up as the Bash associative array element `${hardening[format]}`.
- [`outputChecks`]{#adv-attr-outputChecks}\
When using [structured attributes](#adv-attr-structuredAttrs), the `outputChecks`
attribute allows defining checks per-output.
In addition to
[`allowedReferences`](#adv-attr-allowedReferences), [`allowedRequisites`](#adv-attr-allowedRequisites),
[`disallowedReferences`](#adv-attr-disallowedReferences) and [`disallowedRequisites`](#adv-attr-disallowedRequisites),
the following attributes are available:
- `maxSize` defines the maximum size of the resulting [store object](@docroot@/store/store-object.md).
- `maxClosureSize` defines the maximum size of the output's closure.
- `ignoreSelfRefs` controls whether self-references should be considered when
checking for allowed references/requisites.
2023-02-07 12:38:09 +02:00
Example:
```nix
__structuredAttrs = true;
outputChecks.out = {
# The closure of 'out' must not be larger than 256 MiB.
maxClosureSize = 256 * 1024 * 1024;
# It must not refer to the C compiler or to the 'dev' output.
disallowedRequisites = [ stdenv.cc "dev" ];
};
outputChecks.dev = {
# The 'dev' output must not be larger than 128 KiB.
maxSize = 128 * 1024;
};
```
- [`unsafeDiscardReferences`]{#adv-attr-unsafeDiscardReferences}\
2023-02-07 12:38:09 +02:00
When using [structured attributes](#adv-attr-structuredAttrs), the
attribute `unsafeDiscardReferences` is an attribute set with a boolean value for each output name.
If set to `true`, it disables scanning the output for runtime dependencies.
Example:
```nix
__structuredAttrs = true;
unsafeDiscardReferences.out = true;
```
This is useful, for example, when generating self-contained filesystem images with
their own embedded Nix store: hashes found inside such an image refer
to the embedded store and not to the host's Nix store.
- [`requiredSystemFeatures`]{#adv-attr-requiredSystemFeatures}\
If a derivation has the `requiredSystemFeatures` attribute, then Nix will only build it on a machine that has the corresponding features set in its [`system-features` configuration](@docroot@/command-ref/conf-file.md#conf-system-features).
For example, setting
```nix
requiredSystemFeatures = [ "kvm" ];
```
ensures that the derivation can only be built on a machine with the `kvm` feature.