This looks a lot better (and is also more semantically meaningful).
Since this list is generated in a Nix expression, I don't think using
HTML here is going to be the thing that puts people off modifying this
part of the documentation!
They are equivalent according to
<https://spec.commonmark.org/0.29/#hard-line-breaks>,
and the trailing spaces tend to be a pain (because the make git
complain, editors tend to want to remove them − the `.editorconfig`
actually specifies that − etc..).
This is already used by Hydra, and is very useful when materializing
a remote builder list from service discovery. This allows the service
discovery tool to only sync one file instead of two.
This is technically a breaking change, since attempting to set plugin
files after the first non-flag argument will now throw an error. This
is acceptable given the relative lack of stability in a plugin
interface and the need to tie the knot somewhere once plugins can
actually define new subcommands.
http://nixos.org redirects to https://nixos.org and apparently the HTTP library doesn't follow the redirect, so the output is empty.
When defining https in the request it crashes because the library doesn't seem to support https.
So this switches the example to a different http library.
As we are working towards Nix 3.0 we want to make sure that we make a
huge step forward in Nix's user experience. And once 3.0 is out of the
door we need to make sure that all future commands and features keep up
the standard of user experience.
This PR adds a CLI guideline document to the Nix documentation. Consider
this document a good starting point and a checklist when somebody will
be (re)implementing commands.
Clearly this guideline does nothing to improve user experience on its
own and can only be useful as long as it is going to be read and
cared for. But it is a first step into that direction.
Since c4c3c15c19 (#4251) building Nix for
macOS with sandboxing fails:
```
getting status of /nix/var/nix/profiles/per-user/root/channels/nixpkgs: Operation not permitted
```
This happens, because `EvalSettings::getDefaultNixPath` tries to access
paths outside the sandbox. Since the state-dir is not required for
doc generation, it is set to the dummy folder. This needs to be done
for all nix invocations during doc generation, as
`EvalSettings::getDefaultNixPath` is called unconditionally.
This removes the extra-substituters and extra-sandbox-paths settings
and instead makes every array setting extensible by setting
"extra-<name> = <value>" in the configuration file or passing
"--<name> <value>" on the command line.
pkgs.fetchurl supports an executable argument, which is especially nice
when downloading a large executable. This patch adds the same option to
nix-prefetch-url.
I have tested this to work on the simple case of prefetching a little
executable:
1. nix-prefetch-url --executable https://my/little/script
2. Paste the hash into a pkgs.fetchurl-based package, script-pkg.nix
3. Delete the output from the store to avoid any misidentified artifacts
4. Realise the package script-pkg.nix
5. Run the executable
I repeated the above while using --name, as well.
I suspect --executable would have no meaningful effect if combined with
--unpack, but I have not tried it.
Otherwise, the steps advertised in this document won't actually work
(e.g. `make install` will fail, trying to access /usr, and
`./inst/bin/nix` won't exist).
Some users have their own hashed-mirrors setup, that is used to mirror
things in addition to what’s available on tarballs.nixos.org. Although
this should be feasable to do with a Binary Cache, it’s not always
easy, since you have to remember what "name" each of the tarballs has.
Continuing to support hashed-mirrors is cheap, so it’s best to leave
support in Nix. Note that NIX_HASHED_MIRRORS is also supported in
Nixpkgs through fetchurl.nix.
Note that this excludes tarballs.nixos.org from the default, as in
\#3689. All of these are available on cache.nixos.org.
The new error-format is pretty nice from a UX point-of-view, however
it's fairly hard to parse the output e.g. for editor plugins such as
vim-ale[1] that use `nix-instantiate --parse` to determine syntax errors in
Nix expression files.
This patch extends the `internal-json` logger by adding the fields
`line`, `column` and `file` to easily locate an error in a file and the
field `raw_msg` which contains the error-message itself without
code-lines and additional helpers.
An exemplary output may look like this:
```
[nix-shell]$ ./inst/bin/nix-instantiate ~/test.nix --log-format minimal
{"action":"msg","column":1,"file":"/home/ma27/test.nix","level":0,"line":4,"raw_msg":"syntax error, unexpected IF, expecting $end","msg":"<full error-msg with code-lines etc>"}
```
[1] https://github.com/dense-analysis/ale
Nix installation now requires following redirects using `curl -L`. This
is currently represented on the [Nix download page][] but not in the
manual. This change updates the manual to reflect this.
Using `curl` without the `-L` flag results in an empty body, making
installation a no-op.
[Nix download page]: https://nixos.org/download.html
This should handle installation scenarios we can handle with
anything resembling confidence. Goal is approximating the existing
setup--not enforcing a best-practice...
Approaches (+ installer-handled, - manual) and configs each covers:
+ no change needed; /nix OK on boot volume:
All pre-Catalina (regardless of T2 or FileVault use)
+ create new unencrypted volume:
Catalina, pre-T2, no FileVault
+ create new encrypted-at-rest volume:
Catalina, pre-T2, FileVault
Catalina, T2, no FileVault
- require user to pre-create encrypted volume
Catalina, T2, FileVault
By default Nix/NixOS already set a reasonable default `max-jobs = auto`
so we don't need to mention it in this tutorial.
The option is still documented in other parts of the documentation
if users ever stumble over this.
Fixes https://github.com/NixOS/nix/issues/2531
This PR proposes two changes to the "Upgrading Nix" documentation:
* Besides updating `nixpkgs.nix`, we also update `nixpkgs.cacert`, so that the certificates are up-to-date as well.
* Add the instructions for multi-user mode on Linux.
Motivation: maintain project-level configuration files.
Document the whole situation a bit better so that it corresponds to the
implementation, and add NIX_USER_CONF_FILES that allows overriding
which user files Nix will load during startup.
This line has been this way since it was written, in 9e08f5efe
in 2006.
I think it was just a small mistake then; Eelco's thesis earlier
that year says the permission on each file is set to 0444 or 0555
in a derivation's output as part of the build process. In any
case I'm pretty sure that's the behavior now.
Running `nix-store --gc --delete` will, as of Nix 2.3.3, simply fail
because the --delete option conflicts with the --delete operation.
$ nix-store --gc --delete
error: only one operation may be specified
Try 'nix-store --help' for more information.
Furthermore, it has been broken since at least Nix 0.16 (which was
released sometime in 2010), which means that any scripts which depend
on it should have been broken at least nine years ago. This commit
simply formally removes the option. There should be no actual difference
in behaviour as far as the user is concerned: it errors with the exact
same error message. The manual has been edited to remove any references
to the (now gone) --delete option.
Other information:
* Path for Nix 0.16 used:
/nix/store/rp3sgmskn0p0pj1ia2qwd5al6f6pinz4-nix-0.16
This file is licensed under the GPL. Originally, Nix was also
GPL-licensed so that was fine. However, we later changed the license
to the LGPL but missed the fact that style.css has an incompatible
license.
Since the Nix manual at nixos.org uses its own styling, we can remove
this file.
Fixes#3392.
The flag is `--pure-eval`, which can be found by looking at the test suite; it
should be in the notes describing the feature as well, since otherwise users may
assume this is referencing something like `nix-shell --pure`.
- At the top of the release notes, we announce sandboxing is now enabled by default,
then at the bottom it says it's now disabled when missing kernel support. These
can be merged into one point for clarity.
- The point about `max-jobs` defaulting to 1 appears unrelated to sandboxing.
This documents the outcome of the change in
https://github.com/NixOS/nix/issues/454:
> We can also automatically add parentheses in the generated
> `buildInputs`, so you can type `nix-shell -p "expr"`
> instead of `"(expr").
The `post-build-hook` text currently appears in the index, but not on the actual title line of the section, this follows the pattern used in a previous section to get a reference into a title.
With the merge of #2582, the syntax "tags/1.9" for refs does not work
anymore.
However, the new syntax "refs/tags/1.9" seems to support annotated tags,
such as "refs/tags/2.0".
Closes#2385.
With this patch, and this file I called `log.py`:
#!/usr/bin/env nix-shell
#!nix-shell -i python3 -p python3 --pure
import sys
from pprint import pprint
stack = []
timestack = []
for line in open(sys.argv[1]):
components = line.strip().split(" ", 2)
if components[0] != "function-trace":
continue
direction = components[1]
components = components[2].rsplit(" ", 2)
loc = components[0]
_at = components[1]
time = int(components[2])
if direction == "entered":
stack.append(loc)
timestack.append(time)
elif direction == "exited":
dur = time - timestack.pop()
vst = ";".join(stack)
print(f"{vst} {dur}")
stack.pop()
and:
nix-instantiate --trace-function-calls -vvvv ../nixpkgs/pkgs/top-level/release.nix -A unstable > log.matthewbauer 2>&1
./log.py ./log.matthewbauer > log.matthewbauer.folded
flamegraph.pl --title matthewbauer-post-pr log.matthewbauer.folded > log.matthewbauer.folded.svg
I can make flame graphs like: http://gsc.io/log.matthewbauer.folded.svg
---
Includes test cases around function call failures and tryEval. Uses
RAII so the finish is always called at the end of the function.
Make curl's low speed limit configurable via stalled-download-timeout.
Before, this limit was five minutes without receiving a single byte.
This is much too long as if the remote end may not have even
acknowledged the HTTP request.
Passing `--post-build-hook /foo/bar` to a nix-* command will cause
`/foo/bar` to be executed after each build with the following
environment variables set:
DRV_PATH=/nix/store/drv-that-has-been-built.drv
OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev
This can be useful in particular to upload all the builded artifacts to
the cache (including the ones that don't appear in the runtime closure
of the final derivation or are built because of IFD).
This new feature prints the stderr/stdout output to the `nix-build`
and `nix build` client, and the output is printed in a Nix 2
compatible format:
[nix]$ ./inst/bin/nix-build ./test.nix
these derivations will be built:
/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv
building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'...
hello!
bye!
running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'...
post-build-hook: + sleep 1
post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: + sleep 1
post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: + sleep 1
post-build-hook: + printf 'very important stuff'
/nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
[nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix
my-example-derivation> hello!
my-example-derivation> bye!
my-example-derivation (post)> + sleep 1
my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> + sleep 1
my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> + sleep 1
my-example-derivation (post)> + printf 'very important stuff'
[1 built, 0.0 MiB DL]
Co-authored-by: Graham Christensen <graham@grahamc.com>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
And probably many other distributions.
Until now, ./configure would fail silently printing a warning
./configure: line 4621: AX_CXX_COMPILE_STDCXX_17: command not found
and then continuing, later failing with a C++ #error saying that some C++11
feature isn't supported (it didn't even get to the C++17 features).
This is because older distributions don't come with the
`AX_CXX_COMPILE_STDCXX_17` m4 macro.
This commit vendors that macro accordingly.
Now ./configure complains correctly:
configure: error: *** A compiler with support for C++17 language features is required.
On Ubuntu 16.04, ./configure completes if a newer compiler is used, e.g. with
gcc-7 from https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test
using:
./bootstrap.sh
./configure CXX=g++-7 --disable-doc-gen --with-boost=$(nix-build --no-link '<nixpkgs>' -A boost.dev)
And probably other Linux distributions with long-term support releases.
Also update manual stating what version is needed;
I checked that 1.14 is the oldest version with which current nix compiles,
and added autoconf feature checks for some functions added in that release
that nix uses.
In `args@{ a ? 1 }: /* ... */` the value `a` won't be a part of `args`
unless it's specified when calling the function, the default value will
be ignored in this case.
My personal point of view is that this behavior is a matter of taste, at
least I was pretty sure that unmatched arguments will be a part of
`args@` while debugging some Nix code last week.
I decided to add a warning to the docs which hopefully reduces the
confusion of further Nix developers who thought the same about `args@`.
For text files it is possible to do it like so:
`builtins.hashString "sha256" (builtins.readFile /tmp/a)`
but that doesn't work for binary files.
With builtins.hashFile any kind of file can be conveniently hashed.
Inside a derivation, exportReferencesGraph already provides a way to
dump the Nix database for a specific closure. On the command line,
--dump-db gave us the same information, but only for the entire Nix
database at once.
With this change, one can now pass a list of paths to --dump-db to get
the Nix database dumped for just those paths. (The user is responsible
for ensuring this is a closure, like for --export).
Among other things, this is useful for deploying a closure to a new
host without using --import/--export; one can use tar to transfer the
store paths, and --dump-db/--load-db to transfer the validity
information. This is useful if the new host doesn't actually have Nix
yet, and the closure that is being deployed itself contains Nix.
- The instructions for using nix-shell as an interpreter has a Haskell script
example that doesn't work on more recent versions of Nix. Update the
instructions with a working command
This prints the references graph of the store paths in the graphML
format [1]. The graphML format is supported by several graph tools
such as the Python Networkx library or the Apache Thinkerpop project.
[1] http://graphml.graphdrawing.org
`fetchurl` will now throw if given an `md5`, and the hashes have generally
been upgraded to avoid it and use `sha256` as a default. This updates the
documentation examples in the manual accordingly.
This removes confusing documentation. It's better to remove doc than add implementation, because Nix 1.12 will surely have new GC interface anyway.
Fixes https://github.com/NixOS/nix/issues/641
While trying to understand garbage collection it was not immediately
clear that only the runtime dependency closure of output paths
would be kept (instead of the build-time dependency closure).
This commit attempts to clarify this by expanding some of the
glossary definitions and extending the Garbage Collection
section.
The overhead of sandbox builds is a problem on NixOS (since building a
NixOS configuration involves a lot of small derivations) but not for
typical non-NixOS use cases. So outside of NixOS we can enable it.
Issue #179.
builtins.path allows specifying the name of a path (which makes paths
with store-illegal names now addable), allows adding paths with flat
instead of recursive hashes, allows specifying a filter (so is a
generalization of filterSource), and allows specifying an expected
hash (enabling safe path adding in pure mode).
Instead, if a fixed-output derivation produces has an incorrect output
hash, we now unconditionally move the outputs to the path
corresponding with the actual hash and register it as valid. Thus,
after correcting the hash in the Nix expression (e.g. in a fetchurl
call), the fixed-output derivation doesn't have to be built again.
It would still be good to have a command for reporting the actual hash
of a fixed-output derivation (instead of throwing an error), but
"nix-build --hash" didn't do that.
Following discussion with Shea and Graham. It's a big enough change
from the last release. Also, from a semver perspective, 2.0 makes more
sense because we did remove some interfaces (like nix-pull/nix-push).
The name had become a misnomer since it's not only for substitution
from binary caches, but when adding/copying any
(non-content-addressed) path to a store.
Nix can now automatically run the garbage collector during builds or
while adding paths to the store. The option "min-free = <bytes>"
specifies that Nix should run the garbage collector whenever free
space in the Nix store drops below <bytes>. It will then delete
garbage until "max-free" bytes are available.
Garbage collection during builds is asynchronous; running builds are
not paused and new builds are not blocked. However, there also is a
synchronous GC run prior to the first build/substitution.
Currently, no old GC roots are deleted (as in "nix-collect-garbage
-d").
In particular, drop the "build-" and "gc-" prefixes which are
pointless. So now you can say
nix build --no-sandbox
instead of
nix build --no-build-use-sandbox
This removes the file nix-mode.el from Nix. The file is now available within the
repository https://github.com/NixOS/nix-mode.
Fixes#662Fixes#1040Fixes#1054Fixes#1055Closes#1119Fixes#1419
NOTE: all of the above should be fixed within NixOS/nix-mode. If one of those
hasn’t please reopen within NixOS/nix-mode and not within NixOS/nix.
This allows builds to call setuid binaries. This was previously
possible until we started using seccomp. Turns out that seccomp by
default disallows processes from acquiring new privileges. Generally,
any use of setuid binaries (except those created by the builder
itself) is by definition impure, but some people were relying on this
ability for certain tests.
Example:
$ nix build '(with import <nixpkgs> {}; runCommand "foo" {} "/run/wrappers/bin/ping -c 1 8.8.8.8; exit 1")' --no-allow-new-privileges
builder for ‘/nix/store/j0nd8kv85hd6r4kxgnwzvr0k65ykf6fv-foo.drv’ failed with exit code 1; last 2 log lines:
cannot raise the capability into the Ambient set
: Operation not permitted
$ nix build '(with import <nixpkgs> {}; runCommand "foo" {} "/run/wrappers/bin/ping -c 1 8.8.8.8; exit 1")' --allow-new-privileges
builder for ‘/nix/store/j0nd8kv85hd6r4kxgnwzvr0k65ykf6fv-foo.drv’ failed with exit code 1; last 6 log lines:
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=46 time=15.2 ms
Fixes#1429.
This is to simplify remote build configuration. These environment
variables predate nix.conf.
The build hook now has a sensible default (namely build-remote).
The current load is kept in the Nix state directory now.
This portion of the quick start guide may lead to confusion for
newcomers to Nix. This change clarifies the example to one that can be
copied in its entirety.
This allows various Store implementations to provide different ways to
get build logs. For example, BinaryCacheStore can get the build logs
from the binary cache.
Also, remove the log-servers option since we can use substituters for
this.
"build-max-jobs" and the "-j" option can now be set to "auto" to use
the number of CPUs in the system. (Unlike build-cores, it doesn't use
0 to imply auto-configuration, because a) magic values are a bad idea
in general; b) 0 is a legitimate value used to disable local
building.)
Fixes#1198.
The current behaviour modifies the first writeable file from amongst
.bash_profile, .bash_login and .profile. So .bash_profile (if it is
writable) would be modified even if a user has already sourced nix.sh
in, say, .profile.
This commit introduces a new environment variable,
NIX_INSTALLER_NO_MODIFY_PROFILE. If this is set during installation,
then the modifications are unconditionally skipped.
This is useful for users who have a manually curated set of dotfiles
that they are porting to a new machine. In such scenarios, nix.sh is
already sourced at a place where the user prefers. Without this
change, the nix installer would insist on modifying .bash_profile if
it exists.
This commit also add documentations for both the current behaviour and
the new override.
For example, you can now set
build-sandbox-paths = /dev/nvidiactl?
to specify that /dev/nvidiactl should only be mounted in the sandbox
if it exists in the host filesystem. This is useful e.g. for EC2
images that should support both CUDA and non-CUDA instances.
It's a slight misnomer now because it actually limits *all* downloads,
not just binary cache lookups.
Also add a "enable-http2" option to allow disabling use of HTTP/2
(enabled by default).
Once upon a time, I wrote my bachelors thesis about functional
deployment mechanisms.
I had to evaluate several szenarios where package management and
deployment were relevant. One szenario was to do distributed builds
over several machines.
I told myself: Weee, nix can do this! And with nix, this is actually
save, as you do not have side effects when building!
So I started. I use a cloud to set up four virtual machines where I
wanted to do the build. A fifth machine was used as master to distribute
the builds. All was good.
I created the necessary SSH keys, made sure every machine was reachable
by the master and configured the build in my remotes.conf.
When I started to try to build weechat from source, the build failed. It
failed, telling me
error: unable to start any build; either increase ‘--max-jobs’ or enable distributed builds
And I started to dig around. I digged long and good. But I wasn't able
to find the issue.
I double and triple checked my environment variables, my settings, the
SSH key and everything.
I reached out to fellow Nixers by asking on the nixos IRC channel. And I
got help. But we weren't able to find the issue, either.
So I became frustrated. I re-did all the environment variables.
And suddenly,... it worked! What did I change? Well... I made the
environment variables which contained pathes contain absolute pathes
rather than relatives.
And because I like to share my knowledge, this should be put into the
documentation, so others do not bang their heads against the wall
because something is not documented somewhere.
Docbook XSL got updated to version 1.79.1 in NixOS/nixpkgs@fb893a8 and
we're still referring to the hardcoded previous version.
So instead of just updating this to 1.79.1 we're going to use "current"
in the hope that this won't happen again.
I have tested this by building the manual under Nix(OS) but I haven't
tested this in a non-Nix environment, so I'm not sure whether this could
have implications.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Cc: @edolstra
The existing "nix-build" examples were failing:
error: cannot auto-call a function that has an argument without a default value (‘system’)
Thanks to @groxxda on irc for pointing out the fix!
Updated to completely remove unneeded path argument, suggested by @joachifm
Updated to remove other occurences of `all-packages.nix` from files as
suggested by @domenkozar
Some benchmarking suggested this as a good value. Running
$ benchmark -f ... -t 25 -- sh -c 'rm -f /nix/var/nix/binary-cache*; nix-store -r /nix/store/x5z8a2yvz8h6ccmhwrwrp9igg03575jg-nixos-15.09.git.5fd87e1M.drv --dry-run --option binary-caches-parallel-connections <N>'
gave the following mean elapsed times for these values of N:
N=10: 3.3541
N=20: 2.9320
N=25: 2.6690
N=30: 2.9417
N=50: 3.2021
N=100: 3.5718
N=150: 4.2079
Memory usage is also reduced (N=150 used 186 MB, N=25 only 68 MB).
Closes#708.
- rename options but leav old names as lower-priority aliases,
also "-dirs" -> "-paths" to get closer to the meaning
- update docs to reflect the new names (old aliases are not documented),
including a new file with release notes
- tests need an update after corresponding changes to nixpkgs
- __noChroot is left as it is (after discussion on the PR)
Passing "--option build-repeat <N>" will cause every build to be
repeated N times. If the build output differs between any round, the
build is rejected, and the output paths are not registered as
valid. This is primarily useful to verify build determinism. (We
already had a --check option to repeat a previously succeeded
build. However, with --check, non-deterministic builds are registered
in the DB. Preventing that is useful for Hydra to ensure that
non-deterministic builds don't end up getting published at all.)
This allows overriding the name component of the resulting Nix store
path, which is necessary if the base name of the URI contains
"illegal" characters.
This allows nix-prefetch-url to prefetch the output of fetchzip and
its wrappers (like fetchFromGitHub). For example:
$ nix-prefetch-url --unpack https://github.com/NixOS/patchelf/archive/0.8.tar.gz
or from a Nix expression:
$ nix-prefetch-url -A nix-repl.src
In the latter case, --unpack can be omitted because nix-repl.src is a
fetchFromGitHub derivation and thus has "outputHashMode" set to
"recursive".
For example,
$ nix-prefetch-url -A hello.src
will prefetch the file specified by the fetchurl call in the attribute
‘hello.src’ from the Nix expression in the current directory. This
differs from ‘nix-build -A hello.src’ in that it doesn't verify the
hash.
You can also specify a path to the Nix expression:
$ nix-prefetch-url ~/Dev/nixpkgs -A hello.src
List elements (typically used in ‘patches’ attributes) also work:
$ nix-prefetch-url -A portmidi.patches.0
