pkgs.fetchurl supports an executable argument, which is especially nice
when downloading a large executable. This patch adds the same option to
nix-prefetch-url.
I have tested this to work on the simple case of prefetching a little
executable:
1. nix-prefetch-url --executable https://my/little/script
2. Paste the hash into a pkgs.fetchurl-based package, script-pkg.nix
3. Delete the output from the store to avoid any misidentified artifacts
4. Realise the package script-pkg.nix
5. Run the executable
I repeated the above while using --name, as well.
I suspect --executable would have no meaningful effect if combined with
--unpack, but I have not tried it.
Otherwise, the steps advertised in this document won't actually work
(e.g. `make install` will fail, trying to access /usr, and
`./inst/bin/nix` won't exist).
Some users have their own hashed-mirrors setup, that is used to mirror
things in addition to what’s available on tarballs.nixos.org. Although
this should be feasable to do with a Binary Cache, it’s not always
easy, since you have to remember what "name" each of the tarballs has.
Continuing to support hashed-mirrors is cheap, so it’s best to leave
support in Nix. Note that NIX_HASHED_MIRRORS is also supported in
Nixpkgs through fetchurl.nix.
Note that this excludes tarballs.nixos.org from the default, as in
\#3689. All of these are available on cache.nixos.org.
The new error-format is pretty nice from a UX point-of-view, however
it's fairly hard to parse the output e.g. for editor plugins such as
vim-ale[1] that use `nix-instantiate --parse` to determine syntax errors in
Nix expression files.
This patch extends the `internal-json` logger by adding the fields
`line`, `column` and `file` to easily locate an error in a file and the
field `raw_msg` which contains the error-message itself without
code-lines and additional helpers.
An exemplary output may look like this:
```
[nix-shell]$ ./inst/bin/nix-instantiate ~/test.nix --log-format minimal
{"action":"msg","column":1,"file":"/home/ma27/test.nix","level":0,"line":4,"raw_msg":"syntax error, unexpected IF, expecting $end","msg":"<full error-msg with code-lines etc>"}
```
[1] https://github.com/dense-analysis/ale
Nix installation now requires following redirects using `curl -L`. This
is currently represented on the [Nix download page][] but not in the
manual. This change updates the manual to reflect this.
Using `curl` without the `-L` flag results in an empty body, making
installation a no-op.
[Nix download page]: https://nixos.org/download.html
This should handle installation scenarios we can handle with
anything resembling confidence. Goal is approximating the existing
setup--not enforcing a best-practice...
Approaches (+ installer-handled, - manual) and configs each covers:
+ no change needed; /nix OK on boot volume:
All pre-Catalina (regardless of T2 or FileVault use)
+ create new unencrypted volume:
Catalina, pre-T2, no FileVault
+ create new encrypted-at-rest volume:
Catalina, pre-T2, FileVault
Catalina, T2, no FileVault
- require user to pre-create encrypted volume
Catalina, T2, FileVault
By default Nix/NixOS already set a reasonable default `max-jobs = auto`
so we don't need to mention it in this tutorial.
The option is still documented in other parts of the documentation
if users ever stumble over this.
Fixes https://github.com/NixOS/nix/issues/2531
This PR proposes two changes to the "Upgrading Nix" documentation:
* Besides updating `nixpkgs.nix`, we also update `nixpkgs.cacert`, so that the certificates are up-to-date as well.
* Add the instructions for multi-user mode on Linux.
Motivation: maintain project-level configuration files.
Document the whole situation a bit better so that it corresponds to the
implementation, and add NIX_USER_CONF_FILES that allows overriding
which user files Nix will load during startup.
This line has been this way since it was written, in 9e08f5efe
in 2006.
I think it was just a small mistake then; Eelco's thesis earlier
that year says the permission on each file is set to 0444 or 0555
in a derivation's output as part of the build process. In any
case I'm pretty sure that's the behavior now.
Running `nix-store --gc --delete` will, as of Nix 2.3.3, simply fail
because the --delete option conflicts with the --delete operation.
$ nix-store --gc --delete
error: only one operation may be specified
Try 'nix-store --help' for more information.
Furthermore, it has been broken since at least Nix 0.16 (which was
released sometime in 2010), which means that any scripts which depend
on it should have been broken at least nine years ago. This commit
simply formally removes the option. There should be no actual difference
in behaviour as far as the user is concerned: it errors with the exact
same error message. The manual has been edited to remove any references
to the (now gone) --delete option.
Other information:
* Path for Nix 0.16 used:
/nix/store/rp3sgmskn0p0pj1ia2qwd5al6f6pinz4-nix-0.16
This file is licensed under the GPL. Originally, Nix was also
GPL-licensed so that was fine. However, we later changed the license
to the LGPL but missed the fact that style.css has an incompatible
license.
Since the Nix manual at nixos.org uses its own styling, we can remove
this file.
Fixes#3392.
The flag is `--pure-eval`, which can be found by looking at the test suite; it
should be in the notes describing the feature as well, since otherwise users may
assume this is referencing something like `nix-shell --pure`.
- At the top of the release notes, we announce sandboxing is now enabled by default,
then at the bottom it says it's now disabled when missing kernel support. These
can be merged into one point for clarity.
- The point about `max-jobs` defaulting to 1 appears unrelated to sandboxing.
This documents the outcome of the change in
https://github.com/NixOS/nix/issues/454:
> We can also automatically add parentheses in the generated
> `buildInputs`, so you can type `nix-shell -p "expr"`
> instead of `"(expr").
The `post-build-hook` text currently appears in the index, but not on the actual title line of the section, this follows the pattern used in a previous section to get a reference into a title.
With the merge of #2582, the syntax "tags/1.9" for refs does not work
anymore.
However, the new syntax "refs/tags/1.9" seems to support annotated tags,
such as "refs/tags/2.0".
Closes#2385.
With this patch, and this file I called `log.py`:
#!/usr/bin/env nix-shell
#!nix-shell -i python3 -p python3 --pure
import sys
from pprint import pprint
stack = []
timestack = []
for line in open(sys.argv[1]):
components = line.strip().split(" ", 2)
if components[0] != "function-trace":
continue
direction = components[1]
components = components[2].rsplit(" ", 2)
loc = components[0]
_at = components[1]
time = int(components[2])
if direction == "entered":
stack.append(loc)
timestack.append(time)
elif direction == "exited":
dur = time - timestack.pop()
vst = ";".join(stack)
print(f"{vst} {dur}")
stack.pop()
and:
nix-instantiate --trace-function-calls -vvvv ../nixpkgs/pkgs/top-level/release.nix -A unstable > log.matthewbauer 2>&1
./log.py ./log.matthewbauer > log.matthewbauer.folded
flamegraph.pl --title matthewbauer-post-pr log.matthewbauer.folded > log.matthewbauer.folded.svg
I can make flame graphs like: http://gsc.io/log.matthewbauer.folded.svg
---
Includes test cases around function call failures and tryEval. Uses
RAII so the finish is always called at the end of the function.
Make curl's low speed limit configurable via stalled-download-timeout.
Before, this limit was five minutes without receiving a single byte.
This is much too long as if the remote end may not have even
acknowledged the HTTP request.
Passing `--post-build-hook /foo/bar` to a nix-* command will cause
`/foo/bar` to be executed after each build with the following
environment variables set:
DRV_PATH=/nix/store/drv-that-has-been-built.drv
OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev
This can be useful in particular to upload all the builded artifacts to
the cache (including the ones that don't appear in the runtime closure
of the final derivation or are built because of IFD).
This new feature prints the stderr/stdout output to the `nix-build`
and `nix build` client, and the output is printed in a Nix 2
compatible format:
[nix]$ ./inst/bin/nix-build ./test.nix
these derivations will be built:
/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv
building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'...
hello!
bye!
running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'...
post-build-hook: + sleep 1
post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: + sleep 1
post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: + sleep 1
post-build-hook: + printf 'very important stuff'
/nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
[nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix
my-example-derivation> hello!
my-example-derivation> bye!
my-example-derivation (post)> + sleep 1
my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> + sleep 1
my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> + sleep 1
my-example-derivation (post)> + printf 'very important stuff'
[1 built, 0.0 MiB DL]
Co-authored-by: Graham Christensen <graham@grahamc.com>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
And probably many other distributions.
Until now, ./configure would fail silently printing a warning
./configure: line 4621: AX_CXX_COMPILE_STDCXX_17: command not found
and then continuing, later failing with a C++ #error saying that some C++11
feature isn't supported (it didn't even get to the C++17 features).
This is because older distributions don't come with the
`AX_CXX_COMPILE_STDCXX_17` m4 macro.
This commit vendors that macro accordingly.
Now ./configure complains correctly:
configure: error: *** A compiler with support for C++17 language features is required.
On Ubuntu 16.04, ./configure completes if a newer compiler is used, e.g. with
gcc-7 from https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test
using:
./bootstrap.sh
./configure CXX=g++-7 --disable-doc-gen --with-boost=$(nix-build --no-link '<nixpkgs>' -A boost.dev)
And probably other Linux distributions with long-term support releases.
Also update manual stating what version is needed;
I checked that 1.14 is the oldest version with which current nix compiles,
and added autoconf feature checks for some functions added in that release
that nix uses.
In `args@{ a ? 1 }: /* ... */` the value `a` won't be a part of `args`
unless it's specified when calling the function, the default value will
be ignored in this case.
My personal point of view is that this behavior is a matter of taste, at
least I was pretty sure that unmatched arguments will be a part of
`args@` while debugging some Nix code last week.
I decided to add a warning to the docs which hopefully reduces the
confusion of further Nix developers who thought the same about `args@`.
For text files it is possible to do it like so:
`builtins.hashString "sha256" (builtins.readFile /tmp/a)`
but that doesn't work for binary files.
With builtins.hashFile any kind of file can be conveniently hashed.
Inside a derivation, exportReferencesGraph already provides a way to
dump the Nix database for a specific closure. On the command line,
--dump-db gave us the same information, but only for the entire Nix
database at once.
With this change, one can now pass a list of paths to --dump-db to get
the Nix database dumped for just those paths. (The user is responsible
for ensuring this is a closure, like for --export).
Among other things, this is useful for deploying a closure to a new
host without using --import/--export; one can use tar to transfer the
store paths, and --dump-db/--load-db to transfer the validity
information. This is useful if the new host doesn't actually have Nix
yet, and the closure that is being deployed itself contains Nix.
- The instructions for using nix-shell as an interpreter has a Haskell script
example that doesn't work on more recent versions of Nix. Update the
instructions with a working command
This prints the references graph of the store paths in the graphML
format [1]. The graphML format is supported by several graph tools
such as the Python Networkx library or the Apache Thinkerpop project.
[1] http://graphml.graphdrawing.org
`fetchurl` will now throw if given an `md5`, and the hashes have generally
been upgraded to avoid it and use `sha256` as a default. This updates the
documentation examples in the manual accordingly.
This removes confusing documentation. It's better to remove doc than add implementation, because Nix 1.12 will surely have new GC interface anyway.
Fixes https://github.com/NixOS/nix/issues/641
While trying to understand garbage collection it was not immediately
clear that only the runtime dependency closure of output paths
would be kept (instead of the build-time dependency closure).
This commit attempts to clarify this by expanding some of the
glossary definitions and extending the Garbage Collection
section.
The overhead of sandbox builds is a problem on NixOS (since building a
NixOS configuration involves a lot of small derivations) but not for
typical non-NixOS use cases. So outside of NixOS we can enable it.
Issue #179.
builtins.path allows specifying the name of a path (which makes paths
with store-illegal names now addable), allows adding paths with flat
instead of recursive hashes, allows specifying a filter (so is a
generalization of filterSource), and allows specifying an expected
hash (enabling safe path adding in pure mode).
Instead, if a fixed-output derivation produces has an incorrect output
hash, we now unconditionally move the outputs to the path
corresponding with the actual hash and register it as valid. Thus,
after correcting the hash in the Nix expression (e.g. in a fetchurl
call), the fixed-output derivation doesn't have to be built again.
It would still be good to have a command for reporting the actual hash
of a fixed-output derivation (instead of throwing an error), but
"nix-build --hash" didn't do that.
Following discussion with Shea and Graham. It's a big enough change
from the last release. Also, from a semver perspective, 2.0 makes more
sense because we did remove some interfaces (like nix-pull/nix-push).
The name had become a misnomer since it's not only for substitution
from binary caches, but when adding/copying any
(non-content-addressed) path to a store.
Nix can now automatically run the garbage collector during builds or
while adding paths to the store. The option "min-free = <bytes>"
specifies that Nix should run the garbage collector whenever free
space in the Nix store drops below <bytes>. It will then delete
garbage until "max-free" bytes are available.
Garbage collection during builds is asynchronous; running builds are
not paused and new builds are not blocked. However, there also is a
synchronous GC run prior to the first build/substitution.
Currently, no old GC roots are deleted (as in "nix-collect-garbage
-d").
In particular, drop the "build-" and "gc-" prefixes which are
pointless. So now you can say
nix build --no-sandbox
instead of
nix build --no-build-use-sandbox
This removes the file nix-mode.el from Nix. The file is now available within the
repository https://github.com/NixOS/nix-mode.
Fixes#662Fixes#1040Fixes#1054Fixes#1055Closes#1119Fixes#1419
NOTE: all of the above should be fixed within NixOS/nix-mode. If one of those
hasn’t please reopen within NixOS/nix-mode and not within NixOS/nix.
This allows builds to call setuid binaries. This was previously
possible until we started using seccomp. Turns out that seccomp by
default disallows processes from acquiring new privileges. Generally,
any use of setuid binaries (except those created by the builder
itself) is by definition impure, but some people were relying on this
ability for certain tests.
Example:
$ nix build '(with import <nixpkgs> {}; runCommand "foo" {} "/run/wrappers/bin/ping -c 1 8.8.8.8; exit 1")' --no-allow-new-privileges
builder for ‘/nix/store/j0nd8kv85hd6r4kxgnwzvr0k65ykf6fv-foo.drv’ failed with exit code 1; last 2 log lines:
cannot raise the capability into the Ambient set
: Operation not permitted
$ nix build '(with import <nixpkgs> {}; runCommand "foo" {} "/run/wrappers/bin/ping -c 1 8.8.8.8; exit 1")' --allow-new-privileges
builder for ‘/nix/store/j0nd8kv85hd6r4kxgnwzvr0k65ykf6fv-foo.drv’ failed with exit code 1; last 6 log lines:
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=46 time=15.2 ms
Fixes#1429.
This is to simplify remote build configuration. These environment
variables predate nix.conf.
The build hook now has a sensible default (namely build-remote).
The current load is kept in the Nix state directory now.
This portion of the quick start guide may lead to confusion for
newcomers to Nix. This change clarifies the example to one that can be
copied in its entirety.
This allows various Store implementations to provide different ways to
get build logs. For example, BinaryCacheStore can get the build logs
from the binary cache.
Also, remove the log-servers option since we can use substituters for
this.
"build-max-jobs" and the "-j" option can now be set to "auto" to use
the number of CPUs in the system. (Unlike build-cores, it doesn't use
0 to imply auto-configuration, because a) magic values are a bad idea
in general; b) 0 is a legitimate value used to disable local
building.)
Fixes#1198.
The current behaviour modifies the first writeable file from amongst
.bash_profile, .bash_login and .profile. So .bash_profile (if it is
writable) would be modified even if a user has already sourced nix.sh
in, say, .profile.
This commit introduces a new environment variable,
NIX_INSTALLER_NO_MODIFY_PROFILE. If this is set during installation,
then the modifications are unconditionally skipped.
This is useful for users who have a manually curated set of dotfiles
that they are porting to a new machine. In such scenarios, nix.sh is
already sourced at a place where the user prefers. Without this
change, the nix installer would insist on modifying .bash_profile if
it exists.
This commit also add documentations for both the current behaviour and
the new override.
For example, you can now set
build-sandbox-paths = /dev/nvidiactl?
to specify that /dev/nvidiactl should only be mounted in the sandbox
if it exists in the host filesystem. This is useful e.g. for EC2
images that should support both CUDA and non-CUDA instances.
It's a slight misnomer now because it actually limits *all* downloads,
not just binary cache lookups.
Also add a "enable-http2" option to allow disabling use of HTTP/2
(enabled by default).
Once upon a time, I wrote my bachelors thesis about functional
deployment mechanisms.
I had to evaluate several szenarios where package management and
deployment were relevant. One szenario was to do distributed builds
over several machines.
I told myself: Weee, nix can do this! And with nix, this is actually
save, as you do not have side effects when building!
So I started. I use a cloud to set up four virtual machines where I
wanted to do the build. A fifth machine was used as master to distribute
the builds. All was good.
I created the necessary SSH keys, made sure every machine was reachable
by the master and configured the build in my remotes.conf.
When I started to try to build weechat from source, the build failed. It
failed, telling me
error: unable to start any build; either increase ‘--max-jobs’ or enable distributed builds
And I started to dig around. I digged long and good. But I wasn't able
to find the issue.
I double and triple checked my environment variables, my settings, the
SSH key and everything.
I reached out to fellow Nixers by asking on the nixos IRC channel. And I
got help. But we weren't able to find the issue, either.
So I became frustrated. I re-did all the environment variables.
And suddenly,... it worked! What did I change? Well... I made the
environment variables which contained pathes contain absolute pathes
rather than relatives.
And because I like to share my knowledge, this should be put into the
documentation, so others do not bang their heads against the wall
because something is not documented somewhere.
Docbook XSL got updated to version 1.79.1 in NixOS/nixpkgs@fb893a8 and
we're still referring to the hardcoded previous version.
So instead of just updating this to 1.79.1 we're going to use "current"
in the hope that this won't happen again.
I have tested this by building the manual under Nix(OS) but I haven't
tested this in a non-Nix environment, so I'm not sure whether this could
have implications.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Cc: @edolstra
The existing "nix-build" examples were failing:
error: cannot auto-call a function that has an argument without a default value (‘system’)
Thanks to @groxxda on irc for pointing out the fix!
Updated to completely remove unneeded path argument, suggested by @joachifm
Updated to remove other occurences of `all-packages.nix` from files as
suggested by @domenkozar
Some benchmarking suggested this as a good value. Running
$ benchmark -f ... -t 25 -- sh -c 'rm -f /nix/var/nix/binary-cache*; nix-store -r /nix/store/x5z8a2yvz8h6ccmhwrwrp9igg03575jg-nixos-15.09.git.5fd87e1M.drv --dry-run --option binary-caches-parallel-connections <N>'
gave the following mean elapsed times for these values of N:
N=10: 3.3541
N=20: 2.9320
N=25: 2.6690
N=30: 2.9417
N=50: 3.2021
N=100: 3.5718
N=150: 4.2079
Memory usage is also reduced (N=150 used 186 MB, N=25 only 68 MB).
Closes#708.
- rename options but leav old names as lower-priority aliases,
also "-dirs" -> "-paths" to get closer to the meaning
- update docs to reflect the new names (old aliases are not documented),
including a new file with release notes
- tests need an update after corresponding changes to nixpkgs
- __noChroot is left as it is (after discussion on the PR)
Passing "--option build-repeat <N>" will cause every build to be
repeated N times. If the build output differs between any round, the
build is rejected, and the output paths are not registered as
valid. This is primarily useful to verify build determinism. (We
already had a --check option to repeat a previously succeeded
build. However, with --check, non-deterministic builds are registered
in the DB. Preventing that is useful for Hydra to ensure that
non-deterministic builds don't end up getting published at all.)
This allows overriding the name component of the resulting Nix store
path, which is necessary if the base name of the URI contains
"illegal" characters.
This allows nix-prefetch-url to prefetch the output of fetchzip and
its wrappers (like fetchFromGitHub). For example:
$ nix-prefetch-url --unpack https://github.com/NixOS/patchelf/archive/0.8.tar.gz
or from a Nix expression:
$ nix-prefetch-url -A nix-repl.src
In the latter case, --unpack can be omitted because nix-repl.src is a
fetchFromGitHub derivation and thus has "outputHashMode" set to
"recursive".
For example,
$ nix-prefetch-url -A hello.src
will prefetch the file specified by the fetchurl call in the attribute
‘hello.src’ from the Nix expression in the current directory. This
differs from ‘nix-build -A hello.src’ in that it doesn't verify the
hash.
You can also specify a path to the Nix expression:
$ nix-prefetch-url ~/Dev/nixpkgs -A hello.src
List elements (typically used in ‘patches’ attributes) also work:
$ nix-prefetch-url -A portmidi.patches.0
This hook can be used to set system-specific per-derivation build
settings that don't fit into the derivation model and are too complex or
volatile to be hard-coded into nix. Currently, the pre-build hook can
only add chroot dirs/files through the interface, but it also has full
access to the chroot root.
The specific use case for this is systems where the operating system ABI
is more complex than just the kernel-support system calls. For example,
on OS X there is a set of system-provided frameworks that can reliably
be accessed by any program linked to them, no matter the version the
program is running on. Unfortunately, those frameworks do not
necessarily live in the same locations on each version of OS X, nor do
their dependencies, and thus nix needs to know the specific version of
OS X currently running in order to make those frameworks available. The
pre-build hook is a perfect mechanism for doing just that.
This hook can be used to set system specific per-derivation build
settings that don't fit into the derivation model and are too complex or
volatile to be hard-coded into nix. Currently, the pre-build hook can
only add chroot dirs/files.
The specific use case for this is systems where the operating system ABI
is more complex than just the kernel-supported system calls. For
example, on OS X there is a set of system-provided frameworks that can
reliably be accessed by any program linked to them, no matter the
version the program is running on. Unfortunately, those frameworks do
not necessarily live in the same locations on each version of OS X, nor
do their dependencies, and thus nix needs to know the specific version
of OS X currently running in order to make those frameworks available.
The pre-build hook is a perfect mechanism for doing just that.
If ‘build-use-chroot’ is set to ‘true’, fixed-output derivations are
now also chrooted. However, unlike normal derivations, they don't get
a private network namespace, so they can still access the
network. Also, the use of the ‘__noChroot’ derivation attribute is
no longer allowed.
Setting ‘build-use-chroot’ to ‘relaxed’ gives the old behaviour.
If ‘--option restrict-eval true’ is given, the evaluator will throw an
exception if an attempt is made to access any file outside of the Nix
search path. This is primarily intended for Hydra, where we don't want
people doing ‘builtins.readFile ~/.ssh/id_dsa’ or stuff like that.
‘--run’ is like ‘--command’, except that it runs the command in a
non-interactive shell. This is important if you do things like:
$ nix-shell --command make
Hitting Ctrl-C while make is running drops you into the interactive
Nix shell, which is probably not what you want. So you can now do
$ nix-shell --run make
instead.
'... another level of indirection not shown in the figure above ...'
but in the 'user-environments.png' figure there is '~/.nix-profile'.
the figure was updated with the commit: f982df3 on Mar 16, 2005.
‘trusted-users’ is a list of users and groups that have elevated
rights, such as the ability to specify binary caches. It defaults to
‘root’. A typical value would be ‘@wheel’ to specify all users in the
wheel group.
‘allowed-users’ is a list of users and groups that are allowed to
connect to the daemon. It defaults to ‘*’. A typical value would be
‘@users’ to specify the ‘users’ group.
If a build log is not available locally, then ‘nix-store -l’ will now
try to download it from the servers listed in the ‘log-servers’ option
in nix.conf. For instance, if you have:
log-servers = http://hydra.nixos.org/log
then it will try to get logs from http://hydra.nixos.org/log/<base
name of the store path>. So you can do things like:
$ nix-store -l $(which xterm)
and get a log even if xterm wasn't built locally.
The option '--delete-generations Nd' deletes all generations older than N
days. However, most likely the user does not want to delete the
generation that was active N days ago.
For example, say that you have these 3 generations:
1: <30 days ago>
2: <15 days ago>
3: <1 hour ago>
If you do --delete-generations 7d (say, as part of a cron job), most
likely you still want to keep generation 2, i.e. the generation that was
active 7 days ago (and for most of the past 7 days, in fact).
This patch fixes this issue. Note that this also affects
'nix-collect-garbage --delete-older-than Nd'.
Thanks to @roconnor for noticing the issue!
This allows you to easily set up a build environment containing the
specified packages from Nixpkgs. For example:
$ nix-shell -p sqlite xorg.libX11 hello
will start a shell in which the given packages are present.
The tarball can now be unpacked anywhere. The installation script
uses "sudo" to create /nix if it doesn't exist. It also fetches the
nixpkgs-unstable channel.
This allows running nix-instantiate --eval-only without performing the
evaluation in readonly mode, letting features like import from
derivation and automatic substitution of builtins.storePath paths work.
Signed-off-by: Shea Levy <shea@shealevy.com>
Combined with the previous changes, stack traces involving derivations
are now much less verbose, since something like
while evaluating the builtin function `getAttr':
while evaluating the builtin function `derivationStrict':
while instantiating the derivation named `gtk+-2.24.20' at `/home/eelco/Dev/nixpkgs/pkgs/development/libraries/gtk+/2.x.nix:11:3':
while evaluating the derivation attribute `propagatedNativeBuildInputs' at `/home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/default.nix:78:17':
while evaluating the attribute `outPath' at `/nix/store/212ngf4ph63mp6p1np2bapkfikpakfv7-nix-1.6/share/nix/corepkgs/derivation.nix:18:9':
...
now reads
while evaluating the attribute `propagatedNativeBuildInputs' of the derivation `gtk+-2.24.20' at `/home/eelco/Dev/nixpkgs/pkgs/development/libraries/gtk+/2.x.nix:11:3':
...
This is equivalent to running ‘nix-env -e '*'’ first, except that it
happens in a single transaction. Thus, ‘nix-env -i pkgs...’ replaces
the profile with the specified set of packages.
The main motivation is to support declarative package management
(similar to environment.systemPackages in NixOS). That is, if you
have a specification ‘profile.nix’ like this:
with import <nixpkgs> {};
[ thunderbird
geeqie
...
]
then after any change to ‘profile.nix’, you can run:
$ nix-env -f profile.nix -ir
to update the profile to match the specification. (Without the ‘-r’
flag, if you remove a package from ‘profile.nix’, it won't be removed
from the actual profile.)
Suggested by @zefhemel.
This allows providing additional binary caches, useful in scripts like
Hydra's build reproduction scripts, in particular because untrusted
caches are ignored.
This should make live easier for single-user (non-daemon)
installations. Note that when the daemon is used, the "calling user"
is root so we're not using any untrusted caches.
This flag causes paths that do not have a known substitute to be
quietly ignored. This is mostly useful for Charon, allowing it to
speed up deployment by letting a machine use substitutes for all
substitutable paths, instead of uploading them. The latter is
frequently faster, e.g. if the target machine has a fast Internet
connection while the source machine is on a slow ADSL line.
Binary caches can now specify a priority in their nix-cache-info file.
The binary cache substituter checks caches in order of priority. This
is to ensure that fast, static caches like nixos.org/binary-cache are
processed before slow, dynamic caches like hydra.nixos.org.
This allows disabling the use of binary caches, e.g.
$ nix-build ... --option use-binary-caches false
Note that
$ nix-build ... --option binary-caches ''
does not disable all binary caches, since the caches defined by
channels will still be used.
This operation allows fixing corrupted or accidentally deleted store
paths by redownloading them using substituters, if available.
Since the corrupted path cannot be replaced atomically, there is a
very small time window (one system call) during which neither the old
(corrupted) nor the new (repaired) contents are available. So
repairing should be used with some care on critical packages like
Glibc.
In Nixpkgs, the attribute in all-packages.nix corresponding to a
package is usually equal to the package name. However, this doesn't
work if the package contains a dash, which is fairly common. The
convention is to replace the dash with an underscore (e.g. "dbus-lib"
becomes "dbus_glib"), but that's annoying. So now dashes are valid in
variable / attribute names, allowing you to write:
dbus-glib = callPackage ../development/libraries/dbus-glib { };
and
buildInputs = [ dbus-glib ];
Since we don't have a negation or subtraction operation in Nix, this
is unambiguous.
Channels can now advertise a binary cache by creating a file
<channel-url>/binary-cache-url. The channel unpacker puts these in
its "binary-caches" subdirectory. Thus, the URLS of the binary caches
for the channels added by root appear in
/nix/var/nix/profiles/per-user/eelco/channels/binary-caches/*. The
binary cache substituter reads these and adds them to the list of
binary caches.
Mandatory features are features that MUST be present in a derivation's
requiredSystemFeatures attribute. One application is performance
testing, where we have a dedicated machine to run performance tests
(and nothing else). Then we would add the label "perf" to the
machine's mandatory features and to the performance testing
derivations.
"nix-channel --add" now accepts a second argument: the channel name.
This allows channels to have a nicer name than (say) nixpkgs_unstable.
If no name is given, it defaults to the last component of the URL
(with "-unstable" or "-stable" removed).
Also, channels are now stored in a profile
(/nix/var/nix/profiles/per-user/$USER/channels). One advantage of
this is that it allows rollbacks (e.g. if "nix-channel --update" gives
an undesirable update).
Nix now requires SQLite and bzip2 to be pre-installed. SQLite is
detected using pkg-config. We required DBD::SQLite anyway, so
depending on SQLite is not a big problem.
The --with-bzip2, --with-openssl and --with-sqlite flags are gone.
environment of the given derivation in a format that can be sourced
by the shell, e.g.
$ eval "$(nix-store --print-env $(nix-instantiate /etc/nixos/nixpkgs -A pkg))"
$ NIX_BUILD_TOP=/tmp
$ source $stdenv/setup
This is especially useful to reproduce the environment used to build
a package outside of its builder for development purposes.
TODO: add a nix-build option to do the above and fetch the
dependencies of the derivation as well.
the contents of any of the given store paths have been modified.
E.g.
$ nix-store --verify-path $(nix-store -qR /var/run/current-system)
path `/nix/store/m2smyiwbxidlprfxfz4rjlvz2c3mg58y-etc' was modified! expected hash `fc87e271c5fdf179b47939b08ad13440493805584b35e3014109d04d8436e7b8', got `20f1a47281b3c0cbe299ce47ad5ca7340b20ab34246426915fce0ee9116483aa'
All paths are checked; the exit code is 1 if any path has been
modified, 0 otherwise.
brackets, e.g.
import <nixpkgs/pkgs/lib>
are resolved by looking them up relative to the elements listed in
the search path. This allows us to get rid of hacks like
import "${builtins.getEnv "NIXPKGS_ALL"}/pkgs/lib"
The search path can be specified through the ‘-I’ command-line flag
and through the colon-separated ‘NIX_PATH’ environment variable,
e.g.,
$ nix-build -I /etc/nixos ...
If a file is not found in the search path, an error message is
lazily thrown.
(Linux) machines no longer maintain the atime because it's too
expensive, and on the machines where --use-atime is useful (like the
buildfarm), reading the atimes on the entire Nix store takes way too
much time to make it practical.
NixOS evaluation errors in particular look intimidating and
generally aren't very useful. Ideally the builtins.throw messages
should be self-contained.
UTC) rather than 0 (00:00:00). 1 is a better choice because some
programs use 0 as a special value. For instance, the Template
Toolkit uses a timestamp of 0 to denote the non-existence of a file,
so it barfs on files in the Nix store (see
template-toolkit-nix-store.patch in Nixpkgs). Similarly, Maya 2008
fails to load script directories with a timestamp of 0 and can't be
patched because it's closed source.
This will also shut up those "implausibly old time stamp" GNU tar
warnings.
logic through the `parseDrvName' and `compareVersions' primops.
This will allow expressions to easily check whether some dependency
is a specific needed version or falls in some version range. See
tests/lang/eval-okay-versions.nix for examples.
single quotes. Example (from NixOS):
job = ''
start on network-interfaces
start script
rm -f /var/run/opengl-driver
${if videoDriver == "nvidia"
then "ln -sf ${nvidiaDrivers} /var/run/opengl-driver"
else if cfg.driSupport
then "ln -sf ${mesa} /var/run/opengl-driver"
else ""
}
rm -f /var/log/slim.log
end script
'';
This style has two big advantages:
- \, ' and " aren't special, only '' and ${. So you get a lot less
escaping in shell scripts / configuration files in Nixpkgs/NixOS.
The delimiter '' is rare in scripts (and can usually be written as
""). ${ is also fairly rare.
Other delimiters such as <<...>>, {{...}} and <|...|> were also
considered but this one appears to have the fewest drawbacks
(thanks Martin).
- Indentation is intelligently stripped so that multi-line strings
can follow the nesting structure of the containing Nix
expression. E.g. in the example above 6 spaces are stripped from
the start of each line. This prevents unnecessary indentation in
generated files (which sometimes even breaks things).
See tests/lang/eval-okay-ind-string.nix for some examples.
but installations/upgrades as well. So `nix-env -ub \*' will
upgrade only those packages for which a substitute is available (or
to be precise, it will upgrade each package to the highest version
for which a substitute is available).
Nix expressions in that directory are combined into an attribute set
{file1 = import file1; file2 = import file2; ...}, i.e. each Nix
expression is an attribute with the file name as the attribute
name. Also recurses into directories.
* nix-env: removed the "--import" (-I) option which set the
~/.nix-defexpr symlink.
* nix-channel: don't use "nix-env --import", instead symlink
~/.nix-defexpr/channels. So finally nix-channel --update doesn't
override any default Nix expressions but combines with them.
This means that you can have (say) a local Nixpkgs SVN tree and use
it as a default for nix-env:
$ ln -s .../path-to-nixpkgs-tree ~/.nix-defexpr/nixpkgs_svn
and be subscribed to channels (including Nixpkgs) at the same time.
(If there is any ambiguity, the -A flag can be used to
disambiguate, e.g. "nix-env -i -A nixpkgs_svn.pan".)
need any info on substitutable paths, we just call the substituters
(such as download-using-manifests.pl) directly. This means that
it's no longer necessary for nix-pull to register substitutes or for
nix-channel to clear them, which makes those operations much faster
(NIX-95). Also, we don't have to worry about keeping nix-pull
manifests (in /nix/var/nix/manifests) and the database in sync with
each other.
The downside is that there is some overhead in calling an external
program to get the substitutes info. For instance, "nix-env -qas"
takes a bit longer.
Abolishing the substitutes table also makes the logic in
local-store.cc simpler, as we don't need to store info for invalid
paths. On the downside, you cannot do things like "nix-store -qR"
on a substitutable but invalid path (but nobody did that anyway).
* Never catch interrupts (the Interrupted exception).
by priority and version install. That is, if there are multiple
packages with the same name, then pick the package with the highest
priority, and only use the version if there are multiple packages
with the same priority.
This makes it possible to mark specific versions/variant in Nixpkgs
more or less desirable than others. A typical example would be a
beta version of some package (e.g., "gcc-4.2.0rc1") which should not
be installed even though it is the highest version, except when it
is explicitly selected (e.g., "nix-env -i gcc-4.2.0rc1").
* Idem for nix-env -u, only the semantics are a bit trickier since we
also need to take into account the priority of the currently
installed package (we never upgrade to a lower priority, unless
--always is given).
a user environment by an install or upgrade action. This is
particularly useful if you have a version installed that you don't
want to upgrade (e.g., because the newer versions are broken).
Example:
$ nix-env -u zapping --dry-run
(dry run; not doing anything)
upgrading `zapping-0.9.6' to `zapping-0.10cvs6'
$ nix-env --set-flag keep true zapping
$ nix-env -u zapping --dry-run
(dry run; not doing anything)
However, "-e" will still uninstall the package. (Maybe we should
require the keep flag to be explicitly set to false before it can be
uninstalled.)
user environment collission between two packages due to overlapping
file names, then a package with a higher priority will overwrite the
symlinks of a package with a lower priority. E.g.,
$ nix-env --set-flag priority 5 gcc
$ nix-env --set-flag priority 10 binutils
gives gcc a higher priority than binutils (higher number = lower
priority).
to show only those derivations whose output is already in the Nix
store or that can be substituted (i.e., downloaded from somewhere).
In other words, it shows the packages that can be installed “quickly”,
i.e., don’t need to be built from source.
attribute) about installed packages in user environments. Thus, an
operation like `nix-env -q --description' shows useful information
not only on available packages but also on installed packages.
* nix-env now passes the entire manifest as an argument to the Nix
expression of the user environment builder (not just a list of
paths), so that in particular the user environment builder has
access to the meta attributes.
* New operation `--set-flag' in nix-env to change meta info of
installed packages. This will be useful to pass per-package
policies to the user environment builder (e.g., how to resolve
collision or whether to disable a package (NIX-80)) or upgrade
policies in nix-env (e.g., that a package should be "masked", that
is, left untouched by upgrade actions). Example:
$ nix-env --set-flag enabled false ghc-6.4
* `sub' to subtract two numbers.
* `stringLength' to get the length of a string.
* `substring' to get a substring of a string. These should be enough
to allow most string operations to be expressed.
* nix-unpack-closure: extract the top-level paths from the closure and
print them on stdout. This allows them to be installed, e.g.,
"nix-env -i $(nix-unpack-closure)". (NIX-64)
from a source directory. All files for which a predicate function
returns true are copied to the store. Typical example is to leave
out the .svn directory:
stdenv.mkDerivation {
...
src = builtins.filterSource
(path: baseNameOf (toString path) != ".svn")
./source-dir;
# as opposed to
# src = ./source-dir;
}
This is important because the .svn directory influences the hash in
a rather unpredictable and variable way.
single derivation specified by the argument. This is useful when we
want to have a profile for a single derivation, such as a server
configuration. Then we can just say (e.g.)
$ nix-env -p /.../server-profile -f server.nix --set -A server
We can't do queries or upgrades on such a profile, but we can do
rollbacks. The advantage over -i is that we don't have to worry
about other packages having been installed in the profile
previously; --set gets rid of them.
seconds without producing output on stdout or stderr (NIX-65). This
timeout can be specified using the `--max-silent-time' option or the
`build-max-silent-time' configuration setting. The default is
infinity (0).
* Fix a tricky race condition: if we kill the build user before the
child has done its setuid() to the build user uid, then it won't be
killed, and we'll potentially lock up in pid.wait(). So also send a
conventional kill to the child.
graph to be passed to a builder. This attribute should be a list of
pairs [name1 path1 name2 path2 ...]. The references graph of each
`pathN' will be stored in a text file `nameN' in the temporary build
directory. The text files have the format used by `nix-store
--register-validity'. However, the deriver fields are left empty.
`exportReferencesGraph' is useful for builders that want to do
something with the closure of a store path. Examples: the builders
that make initrds and ISO images for NixOS.
`exportReferencesGraph' is entirely pure. It's necessary because
otherwise the only way for a builder to get this information would
be to call `nix-store' directly, which is not allowed (though
unfortunately possible).
available. For instance,
$ nix-store -l $(which svn) | less
lets you read the build log of the Subversion instance in your
profile.
* `nix-store -qb': if applied to a non-derivation, take the deriver.
e.g.,
$ nix-env -i -A subversion xorg.xorgserver
The main advantage over using symbolic names is that using attribute
names is unambiguous and much, much faster.
nix-store query options `--referer' and `--referer-closure' have
been changed to `--referrer' and `--referrer-closure' (but the old
ones are still accepted for compatibility).
The expression `with E1; E2' evaluates to E2 with all bindings in
the attribute set E1 substituted. E.g.,
with {x = 123;}; x
evaluates to 123. That is, the attribute set E1 is in scope in E2.
This is particularly useful when importing files containing lots
definitions. E.g., instead of
let {
inherit (import ./foo.nix) a b c d e f;
body = ... a ... f ...;
}
we can now say
with import ./foo.nix;
... a ... f ...
I.e., we don't have to say what variables should be brought into scope.
more common than the latter (which exists only on Linux and
FreeBSD). We don't really care about dropping the saved IDs since
there apparently is no way to quiry them in any case, so it can't
influence the build (unlike the effective IDs which are checked by
Perl for instance).
unreachable paths that haven't been used for N hours. For instance,
`nix-collect-garbage --min-age 168' only deletes paths that haven't
been accessed in the last week.
This is useful for instance in the build farm where many derivations
can be shared between consecutive builds, and we wouldn't want a
garbage collect to throw them all away. We could of course register
them as roots, but then we'd to unregister them at some point, which
would be a pain to manage. The `--min-age' flag gives us a sort of
MRU caching scheme.
BUG: this really shouldn't be in gc.cc since that violates
mechanism/policy separation.
doesn't just print the set of paths that should be deleted. So
there is no more need to pipe the result into `nix-store --delete'
(which doesn't even exist anymore).
* Builder output is written to standard error by default.
* The option `-B' is gone.
* The option `-Q' suppresses builder output.
The result of this is that most Nix invocations shouldn't need any
flags w.r.t. logging.
Whenever Nix attempts to realise a derivation for which a closure is
already known, but this closure cannot be realised, fall back on
normalising the derivation.
The most common scenario in which this is useful is when we have
registered substitutes in order to perform binary distribution from,
say, a network repository. If the repository is down, the
realisation of the derivation will fail. When this option is
specified, Nix will build the derivation instead. Thus, binary
installation falls back on a source installation. This option is
not the default since it is generally not desirable for a transient
failure in obtaining the substitutes to lead to a full build from
source (with the related consumption of resources).
much as possible. (This is similar to GNU Make's `-k' flag.)
* Refactoring to implement this: previously we just bombed out when
a build failed, but now we have to clean up. In particular this
means that goals must be freed quickly --- they shouldn't hang
around until the worker exits. So the worker now maintains weak
pointers in order not to prevent garbage collection.
* Documented the `-k' and `-j' flags.
default -> default-94-link
default-82-link -> /nix/store/cc4480...
default-83-link -> /nix/store/caeec8...
...
default-94-link -> /nix/store/2896ca...
experimental -> experimental-2-link
experimental-1-link -> /nix/store/cc4480...
experimental-2-link -> /nix/store/a3148f...
* `--profile' / `-p' -> `--switch-profile' / `-S'
* `--link' / `-l' -> `--profile' / `-p'
* The default profile is stored in $prefix/var/nix/profiles.
$prefix/var/nix/links is gone. Profiles can be stored anywhere.
* The current profile is now referenced from ~/.nix-profile, not
~/.nix-userenv.
* The roots to the garbage collector now have extension `.gcroot', not
`.id'.
