Nix veterans intuitively know what the following terms mean. They are
used in several places in the nix documentation, but never defined:
- local store
- remote store
- binary cache
- substituter
In particular, I found the last two terms to be confusingly similar.
Let's give definitions for them.
this is not as compact any more, but it more closely resembles the
chapter structure, and clearly shows that the closure property is the
key idea on which most of Nix operates.
clarify that we are copying between different stores. we have not
introduced that notion or why it would be interesting, but for now it
should be fine to keep it in context of the store directory.
we could move that later to a more detailed explanation of different
store types.
at this level of abstraction we do not really care about build instructions or what they are, and also build instructions including their arguments really amount to the build task.
group description of data instead of spreading it across the section.
that should help direct skimming. as it turns out, people do not
actually read any of that.
this leaves open implementation details, especially about store paths
and file system objects, and allows explaining them together were it is
more appropriate. also leaves room to carefully introduce the key
insight behind Nix: applying results from programming language theory to
the operating system paradigm of files and processes.
while this may eventually introduce ugly diffs, the table will now
render readably on the terminal (e.g. for `man nix` or `nix --help`)
without further intervention.
while it appears a bit much for the overview, this way we set the stage
for going directly into data types when describing the store, instead of
first having to say what build tasks are and how they relate to build
plans.
this displays correct composition again. build inputs and build results
are not part of build plans in terms of data objects.
also this is a much less complicated setup. this will be the first
impression of architecture, and we want to get it right.
convention: describe every data type in prose, and illustrate with
a class diagram, and a textual representation of an abstract
data type.
right now we save ourselves the trouble of doing class diagrams, we can
add them later. but they are important.
use file Contents instead of Data, as that flows more naturally in the
prose.
simplify explanation of the idea behind scanning for store paths
remove references to unfinished sections.
attempt to explain used and documented terminology, as well as how
the declarative programming paradigm relates to building software.
in the future one could highlight encouraged terms to shape future
material into higher consistency.
the diagram is a first approximation and only covers that same section.
of course there is much more going on, and other features should at some
point also be illustrated.
we also have to think about presentation format and technicalities
behind it. the manual has to render to `man`, but we may want something
more refined for web view.
there should be a meta section for each chapter to give motivation of
the presented structure. the structure itself is visible from the table
of contents.
idea: sections could be read in different orders by linking them in
different ways (e.g. depth-first or breadth-first). adding hard-coded
transitions makes that confusing.
trying to capture alternative terms in one go here, mirroring everyday
use:
derivation - build plan
realise - execute build
there will be more of that sort.
The idea and most of the execution are @fricklerhandwerk's. I changed a
few things best I could based on @edolstra's corrections, and a Bazel
glossary.
Valentin Gagarin <valentin@fricklerhandwerk.de>
The current docs are all "how to do things" and no "what is Nix" or "why
are things the way they are".
I see lots of misconception on the wider internet, and I also think we
would benefit from a "living document" to answer some questions people
currently turn to the thesis for.
I think a new section of the manual can address all these issues.
it is out of date, all over the place in level of detail, is really
about `nixpkgs`, and in general instructions should not be part of
a reference manual.
also:
- update redirects and internal links
- use "Nix language" consistently
Before #5150 the copy-to-store phase of the install was idempotent,
but the recursive cp isn't. This is probably baiting a few people
into trying corrective installs that will fail.
Added using the following sed scripts:
- For command-ref/opt-common.md:
s~- `(--?)([^`]+)`~- [`\1\2`]{#opt-\2}~g
- For expressions/builtin-constants.md:
s~- `(builtins\.?)([^`]+)`~- [`\1\2`]{#builtins-\2}~g
- For expressions/advanced-attributes.md
s~^ - `([^`]+)`~ - [`\1`]{#adv-attr-\1}~g
and manually adjusted outputHashAlgo & outputHashMode.
- For glossary.md
s~^ - (`([^`]+)`|(.+)) ?\\~ - [\1]{#gloss-\2\3}\\~g;
s~(gloss-\w+) ~\1-~g
and manually adjusted anchors for Nix expression, user environment, NAR, ∅ and ε.
- For command-ref/env-common.md
s~^ - `([^`]+)`~ - [`\1`]{#env-\1}~g'
Python is only pulled into the build closure by Mercurial, which might end up being removed.
Let’s port the script to jq, which is more likely to stay.
It is now possible to use the following syntax to insert anchors into the text:
[]{#anchor-name}
The anchor will allow linking to the location it is placed by appending #anchor-name to the URL.
Additionally, it is possible to create a link pointing to its own location by adding text between the square brackets:
[`--add-root`]{#opt-add-root}
Add a new `file` fetcher type, which will fetch a plain file over
http(s), or from the local file.
Because plain `http(s)://` or `file://` urls can already correspond to
`tarball` inputs (if the path ends-up with a know archive extension),
the URL parsing logic is a bit convuluted in that:
- {http,https,file}:// urls will be interpreted as either a tarball or a
file input, depending on the extensions of the path part (so
`https://foo.com/bar` will be a `file` input and
`https://foo.com/bar.tar.gz` as a `tarball` input)
- `file+{something}://` urls will be interpreted as `file` urls (with
the `file+` part removed)
- `tarball+{something}://` urls will be interpreted as `tarball` urls (with
the `tarball+` part removed)
Fix#3785
Co-Authored-By: Tony Olagbaiye <me@fron.io>
The produced path is then allowed be imported or utilized elsewhere:
```
assert (43 == import (builtins.toFile "source" "43")); "good"
```
This will still fail on write-only stores.
Requested by ppepino on the Matrix:
https://matrix.to/#/!KqkRjyTEzAGRiZFBYT:nixos.org/$Tb32BS3rVE2BSULAX4sPm0h6CDewX2hClOTGzTC7gwM?via=nixos.org&via=matrix.org&via=nixos.dev
This adds a new command, :bl, which works like :b but also creates
a GC root symlink to the various derivation outputs.
ckie@cookiemonster ~/git/nix -> ./outputs/out/bin/nix repl
Welcome to Nix 2.6.0. Type :? for help.
nix-repl> :l <nixpkgs>
Added 16118 variables.
nix-repl> :b runCommand "hello" {} "echo hi > $out"
This derivation produced the following outputs:
./repl-result-out -> /nix/store/kidqq2acdpi05c4a9mlbg2baikmzik44-hello
[1 built, 0.0 MiB DL]
ckie@cookiemonster ~/git/nix -> cat ./repl-result-out
hi
For each `nix.conf` option, add an empty html node with a unique `id`
that can be used as an anchor target. Also make the name of the option
be a link to that target so that it’s easily discoverable.
We can’t rewrite the whole list as an html definition list like it’s
done for the builtins because these options also appear in a man page,
and the manpage renderer (lowdown) can’t render arbitrary html. But the
hack here allows to keep the manpage and have the links in the html
version.
Fix https://github.com/NixOS/nix/issues/5745
gives 2-5% performance improvement across a board of tests.
LTO is broken when using clang; some libs link fine while others crash
the linker with a segfault in the llvm linker plugin. 🙁
The uninstall instructions used to accidentally remove the nix-darwin
LaunchDaemon, this was dropped. However, the original intent was to
remove the Store volume mounting LaunchDaemon.
Clarify that `/nix` being present after the uninstall is normal and it
will only disappear after a reboot.
Co-authored-by: Travis A. Everett <travis.a.everett@gmail.com>
The multi-user installation on macOS, which is now the only option, has
gotten complicated enough that it discourages some users from checking
Nix out for fear of being left with a "dirty" system. Detailed
uninstallation instructions should make this less of an issue.
The logical implication operator is included in this section but never explained. It might stump new readers with a pretty uncommon operator, and it's never referenced explicitly.
nixpkgs can save a good bit of eval memory with this primop. zipAttrsWith is
used quite a bit around nixpkgs (eg in the form of recursiveUpdate), but the
most costly application for this primop is in the module system. it improves
the implementation of zipAttrsWith from nixpkgs by not checking an attribute
multiple times if it occurs more than once in the input list, allocates less
values and set elements, and just avoids many a temporary object in general.
nixpkgs has a more generic version of this operation, zipAttrsWithNames, but
this version is only used once so isn't suitable for being the base of a new
primop. if it were to be used more we should add a second primop instead.
This function is very useful in nixpkgs, but its implementation in Nix
itself is rather slow due to it requiring a lot of attribute set and
list appends.
Because the manual is generated from default values which are themselves
generated from various sources (cpuid, bios settings (kvm), number of
cores). This commit hides non-reproducible settings from the manual
output.
This is in line with XDG Base Directory Specification, where ~/.config is supposed to be used when XDG_CONFIG_HOME is unset.
It also better matches the reality, where ~/.config/nix.conf does not seem to be used.
- Separate the generation of the manpages from their installation
- Make sure that `make` generates the manpages
- Make sure that `make install` installs them
Fix#5051
Use a dedicated make target for the man page rather than bundling the
generation as part of `install`.
Also make sure that `make install` is a fixpoint by
- Removing the generated markdown files from `MANUAL_SRCS`
- Not having the manpage generation write in its source directory so as
to not update its timestamp (it would run each time otherwise)
This looks a lot better (and is also more semantically meaningful).
Since this list is generated in a Nix expression, I don't think using
HTML here is going to be the thing that puts people off modifying this
part of the documentation!
They are equivalent according to
<https://spec.commonmark.org/0.29/#hard-line-breaks>,
and the trailing spaces tend to be a pain (because the make git
complain, editors tend to want to remove them − the `.editorconfig`
actually specifies that − etc..).
This is already used by Hydra, and is very useful when materializing
a remote builder list from service discovery. This allows the service
discovery tool to only sync one file instead of two.
This is technically a breaking change, since attempting to set plugin
files after the first non-flag argument will now throw an error. This
is acceptable given the relative lack of stability in a plugin
interface and the need to tie the knot somewhere once plugins can
actually define new subcommands.
http://nixos.org redirects to https://nixos.org and apparently the HTTP library doesn't follow the redirect, so the output is empty.
When defining https in the request it crashes because the library doesn't seem to support https.
So this switches the example to a different http library.
As we are working towards Nix 3.0 we want to make sure that we make a
huge step forward in Nix's user experience. And once 3.0 is out of the
door we need to make sure that all future commands and features keep up
the standard of user experience.
This PR adds a CLI guideline document to the Nix documentation. Consider
this document a good starting point and a checklist when somebody will
be (re)implementing commands.
Clearly this guideline does nothing to improve user experience on its
own and can only be useful as long as it is going to be read and
cared for. But it is a first step into that direction.
Since c4c3c15c19 (#4251) building Nix for
macOS with sandboxing fails:
```
getting status of /nix/var/nix/profiles/per-user/root/channels/nixpkgs: Operation not permitted
```
This happens, because `EvalSettings::getDefaultNixPath` tries to access
paths outside the sandbox. Since the state-dir is not required for
doc generation, it is set to the dummy folder. This needs to be done
for all nix invocations during doc generation, as
`EvalSettings::getDefaultNixPath` is called unconditionally.
This removes the extra-substituters and extra-sandbox-paths settings
and instead makes every array setting extensible by setting
"extra-<name> = <value>" in the configuration file or passing
"--<name> <value>" on the command line.
pkgs.fetchurl supports an executable argument, which is especially nice
when downloading a large executable. This patch adds the same option to
nix-prefetch-url.
I have tested this to work on the simple case of prefetching a little
executable:
1. nix-prefetch-url --executable https://my/little/script
2. Paste the hash into a pkgs.fetchurl-based package, script-pkg.nix
3. Delete the output from the store to avoid any misidentified artifacts
4. Realise the package script-pkg.nix
5. Run the executable
I repeated the above while using --name, as well.
I suspect --executable would have no meaningful effect if combined with
--unpack, but I have not tried it.
Otherwise, the steps advertised in this document won't actually work
(e.g. `make install` will fail, trying to access /usr, and
`./inst/bin/nix` won't exist).
Some users have their own hashed-mirrors setup, that is used to mirror
things in addition to what’s available on tarballs.nixos.org. Although
this should be feasable to do with a Binary Cache, it’s not always
easy, since you have to remember what "name" each of the tarballs has.
Continuing to support hashed-mirrors is cheap, so it’s best to leave
support in Nix. Note that NIX_HASHED_MIRRORS is also supported in
Nixpkgs through fetchurl.nix.
Note that this excludes tarballs.nixos.org from the default, as in
\#3689. All of these are available on cache.nixos.org.
The new error-format is pretty nice from a UX point-of-view, however
it's fairly hard to parse the output e.g. for editor plugins such as
vim-ale[1] that use `nix-instantiate --parse` to determine syntax errors in
Nix expression files.
This patch extends the `internal-json` logger by adding the fields
`line`, `column` and `file` to easily locate an error in a file and the
field `raw_msg` which contains the error-message itself without
code-lines and additional helpers.
An exemplary output may look like this:
```
[nix-shell]$ ./inst/bin/nix-instantiate ~/test.nix --log-format minimal
{"action":"msg","column":1,"file":"/home/ma27/test.nix","level":0,"line":4,"raw_msg":"syntax error, unexpected IF, expecting $end","msg":"<full error-msg with code-lines etc>"}
```
[1] https://github.com/dense-analysis/ale
Nix installation now requires following redirects using `curl -L`. This
is currently represented on the [Nix download page][] but not in the
manual. This change updates the manual to reflect this.
Using `curl` without the `-L` flag results in an empty body, making
installation a no-op.
[Nix download page]: https://nixos.org/download.html
This should handle installation scenarios we can handle with
anything resembling confidence. Goal is approximating the existing
setup--not enforcing a best-practice...
Approaches (+ installer-handled, - manual) and configs each covers:
+ no change needed; /nix OK on boot volume:
All pre-Catalina (regardless of T2 or FileVault use)
+ create new unencrypted volume:
Catalina, pre-T2, no FileVault
+ create new encrypted-at-rest volume:
Catalina, pre-T2, FileVault
Catalina, T2, no FileVault
- require user to pre-create encrypted volume
Catalina, T2, FileVault
By default Nix/NixOS already set a reasonable default `max-jobs = auto`
so we don't need to mention it in this tutorial.
The option is still documented in other parts of the documentation
if users ever stumble over this.
Fixes https://github.com/NixOS/nix/issues/2531
This PR proposes two changes to the "Upgrading Nix" documentation:
* Besides updating `nixpkgs.nix`, we also update `nixpkgs.cacert`, so that the certificates are up-to-date as well.
* Add the instructions for multi-user mode on Linux.
Motivation: maintain project-level configuration files.
Document the whole situation a bit better so that it corresponds to the
implementation, and add NIX_USER_CONF_FILES that allows overriding
which user files Nix will load during startup.
This line has been this way since it was written, in 9e08f5efe
in 2006.
I think it was just a small mistake then; Eelco's thesis earlier
that year says the permission on each file is set to 0444 or 0555
in a derivation's output as part of the build process. In any
case I'm pretty sure that's the behavior now.
Running `nix-store --gc --delete` will, as of Nix 2.3.3, simply fail
because the --delete option conflicts with the --delete operation.
$ nix-store --gc --delete
error: only one operation may be specified
Try 'nix-store --help' for more information.
Furthermore, it has been broken since at least Nix 0.16 (which was
released sometime in 2010), which means that any scripts which depend
on it should have been broken at least nine years ago. This commit
simply formally removes the option. There should be no actual difference
in behaviour as far as the user is concerned: it errors with the exact
same error message. The manual has been edited to remove any references
to the (now gone) --delete option.
Other information:
* Path for Nix 0.16 used:
/nix/store/rp3sgmskn0p0pj1ia2qwd5al6f6pinz4-nix-0.16
This file is licensed under the GPL. Originally, Nix was also
GPL-licensed so that was fine. However, we later changed the license
to the LGPL but missed the fact that style.css has an incompatible
license.
Since the Nix manual at nixos.org uses its own styling, we can remove
this file.
Fixes#3392.
The flag is `--pure-eval`, which can be found by looking at the test suite; it
should be in the notes describing the feature as well, since otherwise users may
assume this is referencing something like `nix-shell --pure`.
- At the top of the release notes, we announce sandboxing is now enabled by default,
then at the bottom it says it's now disabled when missing kernel support. These
can be merged into one point for clarity.
- The point about `max-jobs` defaulting to 1 appears unrelated to sandboxing.
This documents the outcome of the change in
https://github.com/NixOS/nix/issues/454:
> We can also automatically add parentheses in the generated
> `buildInputs`, so you can type `nix-shell -p "expr"`
> instead of `"(expr").
The `post-build-hook` text currently appears in the index, but not on the actual title line of the section, this follows the pattern used in a previous section to get a reference into a title.
With the merge of #2582, the syntax "tags/1.9" for refs does not work
anymore.
However, the new syntax "refs/tags/1.9" seems to support annotated tags,
such as "refs/tags/2.0".
Closes#2385.
With this patch, and this file I called `log.py`:
#!/usr/bin/env nix-shell
#!nix-shell -i python3 -p python3 --pure
import sys
from pprint import pprint
stack = []
timestack = []
for line in open(sys.argv[1]):
components = line.strip().split(" ", 2)
if components[0] != "function-trace":
continue
direction = components[1]
components = components[2].rsplit(" ", 2)
loc = components[0]
_at = components[1]
time = int(components[2])
if direction == "entered":
stack.append(loc)
timestack.append(time)
elif direction == "exited":
dur = time - timestack.pop()
vst = ";".join(stack)
print(f"{vst} {dur}")
stack.pop()
and:
nix-instantiate --trace-function-calls -vvvv ../nixpkgs/pkgs/top-level/release.nix -A unstable > log.matthewbauer 2>&1
./log.py ./log.matthewbauer > log.matthewbauer.folded
flamegraph.pl --title matthewbauer-post-pr log.matthewbauer.folded > log.matthewbauer.folded.svg
I can make flame graphs like: http://gsc.io/log.matthewbauer.folded.svg
---
Includes test cases around function call failures and tryEval. Uses
RAII so the finish is always called at the end of the function.
Make curl's low speed limit configurable via stalled-download-timeout.
Before, this limit was five minutes without receiving a single byte.
This is much too long as if the remote end may not have even
acknowledged the HTTP request.
Passing `--post-build-hook /foo/bar` to a nix-* command will cause
`/foo/bar` to be executed after each build with the following
environment variables set:
DRV_PATH=/nix/store/drv-that-has-been-built.drv
OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev
This can be useful in particular to upload all the builded artifacts to
the cache (including the ones that don't appear in the runtime closure
of the final derivation or are built because of IFD).
This new feature prints the stderr/stdout output to the `nix-build`
and `nix build` client, and the output is printed in a Nix 2
compatible format:
[nix]$ ./inst/bin/nix-build ./test.nix
these derivations will be built:
/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv
building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'...
hello!
bye!
running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'...
post-build-hook: + sleep 1
post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: + sleep 1
post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: + sleep 1
post-build-hook: + printf 'very important stuff'
/nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
[nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix
my-example-derivation> hello!
my-example-derivation> bye!
my-example-derivation (post)> + sleep 1
my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> + sleep 1
my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> + sleep 1
my-example-derivation (post)> + printf 'very important stuff'
[1 built, 0.0 MiB DL]
Co-authored-by: Graham Christensen <graham@grahamc.com>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
And probably many other distributions.
Until now, ./configure would fail silently printing a warning
./configure: line 4621: AX_CXX_COMPILE_STDCXX_17: command not found
and then continuing, later failing with a C++ #error saying that some C++11
feature isn't supported (it didn't even get to the C++17 features).
This is because older distributions don't come with the
`AX_CXX_COMPILE_STDCXX_17` m4 macro.
This commit vendors that macro accordingly.
Now ./configure complains correctly:
configure: error: *** A compiler with support for C++17 language features is required.
On Ubuntu 16.04, ./configure completes if a newer compiler is used, e.g. with
gcc-7 from https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test
using:
./bootstrap.sh
./configure CXX=g++-7 --disable-doc-gen --with-boost=$(nix-build --no-link '<nixpkgs>' -A boost.dev)
And probably other Linux distributions with long-term support releases.
Also update manual stating what version is needed;
I checked that 1.14 is the oldest version with which current nix compiles,
and added autoconf feature checks for some functions added in that release
that nix uses.
In `args@{ a ? 1 }: /* ... */` the value `a` won't be a part of `args`
unless it's specified when calling the function, the default value will
be ignored in this case.
My personal point of view is that this behavior is a matter of taste, at
least I was pretty sure that unmatched arguments will be a part of
`args@` while debugging some Nix code last week.
I decided to add a warning to the docs which hopefully reduces the
confusion of further Nix developers who thought the same about `args@`.
For text files it is possible to do it like so:
`builtins.hashString "sha256" (builtins.readFile /tmp/a)`
but that doesn't work for binary files.
With builtins.hashFile any kind of file can be conveniently hashed.
Inside a derivation, exportReferencesGraph already provides a way to
dump the Nix database for a specific closure. On the command line,
--dump-db gave us the same information, but only for the entire Nix
database at once.
With this change, one can now pass a list of paths to --dump-db to get
the Nix database dumped for just those paths. (The user is responsible
for ensuring this is a closure, like for --export).
Among other things, this is useful for deploying a closure to a new
host without using --import/--export; one can use tar to transfer the
store paths, and --dump-db/--load-db to transfer the validity
information. This is useful if the new host doesn't actually have Nix
yet, and the closure that is being deployed itself contains Nix.
- The instructions for using nix-shell as an interpreter has a Haskell script
example that doesn't work on more recent versions of Nix. Update the
instructions with a working command
This prints the references graph of the store paths in the graphML
format [1]. The graphML format is supported by several graph tools
such as the Python Networkx library or the Apache Thinkerpop project.
[1] http://graphml.graphdrawing.org
`fetchurl` will now throw if given an `md5`, and the hashes have generally
been upgraded to avoid it and use `sha256` as a default. This updates the
documentation examples in the manual accordingly.
This removes confusing documentation. It's better to remove doc than add implementation, because Nix 1.12 will surely have new GC interface anyway.
Fixes https://github.com/NixOS/nix/issues/641
While trying to understand garbage collection it was not immediately
clear that only the runtime dependency closure of output paths
would be kept (instead of the build-time dependency closure).
This commit attempts to clarify this by expanding some of the
glossary definitions and extending the Garbage Collection
section.
The overhead of sandbox builds is a problem on NixOS (since building a
NixOS configuration involves a lot of small derivations) but not for
typical non-NixOS use cases. So outside of NixOS we can enable it.
Issue #179.
builtins.path allows specifying the name of a path (which makes paths
with store-illegal names now addable), allows adding paths with flat
instead of recursive hashes, allows specifying a filter (so is a
generalization of filterSource), and allows specifying an expected
hash (enabling safe path adding in pure mode).
Instead, if a fixed-output derivation produces has an incorrect output
hash, we now unconditionally move the outputs to the path
corresponding with the actual hash and register it as valid. Thus,
after correcting the hash in the Nix expression (e.g. in a fetchurl
call), the fixed-output derivation doesn't have to be built again.
It would still be good to have a command for reporting the actual hash
of a fixed-output derivation (instead of throwing an error), but
"nix-build --hash" didn't do that.
Following discussion with Shea and Graham. It's a big enough change
from the last release. Also, from a semver perspective, 2.0 makes more
sense because we did remove some interfaces (like nix-pull/nix-push).
The name had become a misnomer since it's not only for substitution
from binary caches, but when adding/copying any
(non-content-addressed) path to a store.
Nix can now automatically run the garbage collector during builds or
while adding paths to the store. The option "min-free = <bytes>"
specifies that Nix should run the garbage collector whenever free
space in the Nix store drops below <bytes>. It will then delete
garbage until "max-free" bytes are available.
Garbage collection during builds is asynchronous; running builds are
not paused and new builds are not blocked. However, there also is a
synchronous GC run prior to the first build/substitution.
Currently, no old GC roots are deleted (as in "nix-collect-garbage
-d").
In particular, drop the "build-" and "gc-" prefixes which are
pointless. So now you can say
nix build --no-sandbox
instead of
nix build --no-build-use-sandbox
This removes the file nix-mode.el from Nix. The file is now available within the
repository https://github.com/NixOS/nix-mode.
Fixes#662Fixes#1040Fixes#1054Fixes#1055Closes#1119Fixes#1419
NOTE: all of the above should be fixed within NixOS/nix-mode. If one of those
hasn’t please reopen within NixOS/nix-mode and not within NixOS/nix.
This allows builds to call setuid binaries. This was previously
possible until we started using seccomp. Turns out that seccomp by
default disallows processes from acquiring new privileges. Generally,
any use of setuid binaries (except those created by the builder
itself) is by definition impure, but some people were relying on this
ability for certain tests.
Example:
$ nix build '(with import <nixpkgs> {}; runCommand "foo" {} "/run/wrappers/bin/ping -c 1 8.8.8.8; exit 1")' --no-allow-new-privileges
builder for ‘/nix/store/j0nd8kv85hd6r4kxgnwzvr0k65ykf6fv-foo.drv’ failed with exit code 1; last 2 log lines:
cannot raise the capability into the Ambient set
: Operation not permitted
$ nix build '(with import <nixpkgs> {}; runCommand "foo" {} "/run/wrappers/bin/ping -c 1 8.8.8.8; exit 1")' --allow-new-privileges
builder for ‘/nix/store/j0nd8kv85hd6r4kxgnwzvr0k65ykf6fv-foo.drv’ failed with exit code 1; last 6 log lines:
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=46 time=15.2 ms
Fixes#1429.
This is to simplify remote build configuration. These environment
variables predate nix.conf.
The build hook now has a sensible default (namely build-remote).
The current load is kept in the Nix state directory now.
This portion of the quick start guide may lead to confusion for
newcomers to Nix. This change clarifies the example to one that can be
copied in its entirety.
This allows various Store implementations to provide different ways to
get build logs. For example, BinaryCacheStore can get the build logs
from the binary cache.
Also, remove the log-servers option since we can use substituters for
this.
"build-max-jobs" and the "-j" option can now be set to "auto" to use
the number of CPUs in the system. (Unlike build-cores, it doesn't use
0 to imply auto-configuration, because a) magic values are a bad idea
in general; b) 0 is a legitimate value used to disable local
building.)
Fixes#1198.
The current behaviour modifies the first writeable file from amongst
.bash_profile, .bash_login and .profile. So .bash_profile (if it is
writable) would be modified even if a user has already sourced nix.sh
in, say, .profile.
This commit introduces a new environment variable,
NIX_INSTALLER_NO_MODIFY_PROFILE. If this is set during installation,
then the modifications are unconditionally skipped.
This is useful for users who have a manually curated set of dotfiles
that they are porting to a new machine. In such scenarios, nix.sh is
already sourced at a place where the user prefers. Without this
change, the nix installer would insist on modifying .bash_profile if
it exists.
This commit also add documentations for both the current behaviour and
the new override.
For example, you can now set
build-sandbox-paths = /dev/nvidiactl?
to specify that /dev/nvidiactl should only be mounted in the sandbox
if it exists in the host filesystem. This is useful e.g. for EC2
images that should support both CUDA and non-CUDA instances.
It's a slight misnomer now because it actually limits *all* downloads,
not just binary cache lookups.
Also add a "enable-http2" option to allow disabling use of HTTP/2
(enabled by default).
Once upon a time, I wrote my bachelors thesis about functional
deployment mechanisms.
I had to evaluate several szenarios where package management and
deployment were relevant. One szenario was to do distributed builds
over several machines.
I told myself: Weee, nix can do this! And with nix, this is actually
save, as you do not have side effects when building!
So I started. I use a cloud to set up four virtual machines where I
wanted to do the build. A fifth machine was used as master to distribute
the builds. All was good.
I created the necessary SSH keys, made sure every machine was reachable
by the master and configured the build in my remotes.conf.
When I started to try to build weechat from source, the build failed. It
failed, telling me
error: unable to start any build; either increase ‘--max-jobs’ or enable distributed builds
And I started to dig around. I digged long and good. But I wasn't able
to find the issue.
I double and triple checked my environment variables, my settings, the
SSH key and everything.
I reached out to fellow Nixers by asking on the nixos IRC channel. And I
got help. But we weren't able to find the issue, either.
So I became frustrated. I re-did all the environment variables.
And suddenly,... it worked! What did I change? Well... I made the
environment variables which contained pathes contain absolute pathes
rather than relatives.
And because I like to share my knowledge, this should be put into the
documentation, so others do not bang their heads against the wall
because something is not documented somewhere.
Docbook XSL got updated to version 1.79.1 in NixOS/nixpkgs@fb893a8 and
we're still referring to the hardcoded previous version.
So instead of just updating this to 1.79.1 we're going to use "current"
in the hope that this won't happen again.
I have tested this by building the manual under Nix(OS) but I haven't
tested this in a non-Nix environment, so I'm not sure whether this could
have implications.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Cc: @edolstra
The existing "nix-build" examples were failing:
error: cannot auto-call a function that has an argument without a default value (‘system’)
Thanks to @groxxda on irc for pointing out the fix!
Updated to completely remove unneeded path argument, suggested by @joachifm
Updated to remove other occurences of `all-packages.nix` from files as
suggested by @domenkozar
Some benchmarking suggested this as a good value. Running
$ benchmark -f ... -t 25 -- sh -c 'rm -f /nix/var/nix/binary-cache*; nix-store -r /nix/store/x5z8a2yvz8h6ccmhwrwrp9igg03575jg-nixos-15.09.git.5fd87e1M.drv --dry-run --option binary-caches-parallel-connections <N>'
gave the following mean elapsed times for these values of N:
N=10: 3.3541
N=20: 2.9320
N=25: 2.6690
N=30: 2.9417
N=50: 3.2021
N=100: 3.5718
N=150: 4.2079
Memory usage is also reduced (N=150 used 186 MB, N=25 only 68 MB).
Closes#708.
- rename options but leav old names as lower-priority aliases,
also "-dirs" -> "-paths" to get closer to the meaning
- update docs to reflect the new names (old aliases are not documented),
including a new file with release notes
- tests need an update after corresponding changes to nixpkgs
- __noChroot is left as it is (after discussion on the PR)
Passing "--option build-repeat <N>" will cause every build to be
repeated N times. If the build output differs between any round, the
build is rejected, and the output paths are not registered as
valid. This is primarily useful to verify build determinism. (We
already had a --check option to repeat a previously succeeded
build. However, with --check, non-deterministic builds are registered
in the DB. Preventing that is useful for Hydra to ensure that
non-deterministic builds don't end up getting published at all.)
This allows overriding the name component of the resulting Nix store
path, which is necessary if the base name of the URI contains
"illegal" characters.
This allows nix-prefetch-url to prefetch the output of fetchzip and
its wrappers (like fetchFromGitHub). For example:
$ nix-prefetch-url --unpack https://github.com/NixOS/patchelf/archive/0.8.tar.gz
or from a Nix expression:
$ nix-prefetch-url -A nix-repl.src
In the latter case, --unpack can be omitted because nix-repl.src is a
fetchFromGitHub derivation and thus has "outputHashMode" set to
"recursive".
For example,
$ nix-prefetch-url -A hello.src
will prefetch the file specified by the fetchurl call in the attribute
‘hello.src’ from the Nix expression in the current directory. This
differs from ‘nix-build -A hello.src’ in that it doesn't verify the
hash.
You can also specify a path to the Nix expression:
$ nix-prefetch-url ~/Dev/nixpkgs -A hello.src
List elements (typically used in ‘patches’ attributes) also work:
$ nix-prefetch-url -A portmidi.patches.0
This hook can be used to set system-specific per-derivation build
settings that don't fit into the derivation model and are too complex or
volatile to be hard-coded into nix. Currently, the pre-build hook can
only add chroot dirs/files through the interface, but it also has full
access to the chroot root.
The specific use case for this is systems where the operating system ABI
is more complex than just the kernel-support system calls. For example,
on OS X there is a set of system-provided frameworks that can reliably
be accessed by any program linked to them, no matter the version the
program is running on. Unfortunately, those frameworks do not
necessarily live in the same locations on each version of OS X, nor do
their dependencies, and thus nix needs to know the specific version of
OS X currently running in order to make those frameworks available. The
pre-build hook is a perfect mechanism for doing just that.
This hook can be used to set system specific per-derivation build
settings that don't fit into the derivation model and are too complex or
volatile to be hard-coded into nix. Currently, the pre-build hook can
only add chroot dirs/files.
The specific use case for this is systems where the operating system ABI
is more complex than just the kernel-supported system calls. For
example, on OS X there is a set of system-provided frameworks that can
reliably be accessed by any program linked to them, no matter the
version the program is running on. Unfortunately, those frameworks do
not necessarily live in the same locations on each version of OS X, nor
do their dependencies, and thus nix needs to know the specific version
of OS X currently running in order to make those frameworks available.
The pre-build hook is a perfect mechanism for doing just that.
If ‘build-use-chroot’ is set to ‘true’, fixed-output derivations are
now also chrooted. However, unlike normal derivations, they don't get
a private network namespace, so they can still access the
network. Also, the use of the ‘__noChroot’ derivation attribute is
no longer allowed.
Setting ‘build-use-chroot’ to ‘relaxed’ gives the old behaviour.
If ‘--option restrict-eval true’ is given, the evaluator will throw an
exception if an attempt is made to access any file outside of the Nix
search path. This is primarily intended for Hydra, where we don't want
people doing ‘builtins.readFile ~/.ssh/id_dsa’ or stuff like that.
