This is primarily to subsume the functionality of the
copy-from-other-stores substituter. For example, in the NixOS
installer, we can now do (assuming we're in the target chroot, and the
Nix store of the installation CD is bind-mounted on /tmp/nix):
$ nix-build ... --option substituters 'local?state=/tmp/nix/var&real=/tmp/nix/store'
However, unlike copy-from-other-stores, this also allows write access
to such a store. One application might be fetching substitutes for
/nix/store in a situation where the user doesn't have sufficient
privileges to create /nix, e.g.:
$ NIX_REMOTE="local?state=/home/alice/nix/var&real=/home/alice/nix/store" nix-build ...
Nix sometimes outputs a warning message like this:
```
directory /nix does not exist; creating it by running ‘?? using sudo
```
... when it really meant to output something that looked like this:
```
directory /nix does not exist; creating it by running 'mkdir -m 0755 /nix && chown gabriel /nix' using sudo
```
The reason why is due to some bizarre behavior in Bash where it will translate anything of the form `$x’` to `??`, leading to the incorrect warning message. I don't know what is the origin of this Bash behavior, but the easiest fix is to just use ASCII quotes instead of unicode quotes.
Just wasted a couple hours chasing shadows because the nix store got
corrupted and there was no indication of that anywhere.
Since an install is one-time only, might as well verify. Optimization
showed that the copied files aren't read-only; fixed that as well.
Also, use /bin/sh since there's a good chance that this script will be
run on systems without /bin/bash
This is currently only used by the Hydra queue runner rework, but like
eff5021eaa it presumably will be useful
for the C++ rewrite of nix-push and
download-from-binary-cache. (@shlevy)
The `set -e` at the top of the script causes the installation to fail to
complete if the shell profile is not writeable. Checking file existence
only is not enough.
The --insecure flag to curl tells curl not to bother checking if the TLS
certificate presented by the server actually matches the hostname
requested, and actually is issued by a trusted CA chain. This almost
entirely negates any benefit from using TLS in the first place.
This removes the --insecure flag to ensure we actually have a secure
connection to the intended hostname before downloading binaries.
Manually tested locally within a dev-shell; was able to download
binaries from https://cache.nixos.org without issue.
[Note: --insecure was only used for fetching NARs, whose integrity is
verified by Nix anyway using the hash from the .narinfo. But if we can
fetch the .narinfo without --insecure, we can also fetch the .nar, so
there is not much point to using --insecure. --Eelco]
Some benchmarking suggested this as a good value. Running
$ benchmark -f ... -t 25 -- sh -c 'rm -f /nix/var/nix/binary-cache*; nix-store -r /nix/store/x5z8a2yvz8h6ccmhwrwrp9igg03575jg-nixos-15.09.git.5fd87e1M.drv --dry-run --option binary-caches-parallel-connections <N>'
gave the following mean elapsed times for these values of N:
N=10: 3.3541
N=20: 2.9320
N=25: 2.6690
N=30: 2.9417
N=50: 3.2021
N=100: 3.5718
N=150: 4.2079
Memory usage is also reduced (N=150 used 186 MB, N=25 only 68 MB).
Closes#708.
This makes that option even more insecure, by also not checking the SSL host.
But without this parameter, one can still get SSL errors even when
"verify-https-binary-caches" is false, which is unexpected IMO.
This reverts commit 76f985b92d. We
shouldn't mess with $MANPATH, because on some "man" implementations
(like NixOS'), the default value on $MANPATH is derived from $PATH. So
if you set $MANPATH, you lose the default locations.
This is not strictly needed for integrity (since we already include
the NAR hash in the fingerprint) but it helps against endless data
attacks [1]. (However, this will also require
download-from-binary-cache.pl to bail out if it receives more than the
specified number of bytes.)
[1] https://isis.poly.edu/~jcappos/papers/cappos_mirror_ccs_08.pdf
In some cases the bash builtin command "cd" can print the variable $CWD
to stdout. This caused the install script to fail while copying files
because the source path was wrong.
Fixes#476.
We only need to sign the store path, NAR hash and references (the
"fingerprint"). Everything else is irrelevant to security. For
instance, the compression algorithm or the hash of the compressed NAR
don't matter as long as the contents of the uncompressed NAR are
correct.
(Maybe we should include derivers in the fingerprint, but they're
broken and nobody cares about them. Also, it might be nice in the
future if .narinfos contained signatures from multiple independent
signers. But that's impossible if the deriver is included in the
fingerprint, since everybody will tend to have a different deriver for
the same store path.)
Also renamed the "Signature" field to "Sig" since the format changed
in an incompatible way.
Sodium's Ed25519 signatures are much shorter than OpenSSL's RSA
signatures. Public keys are also much shorter, so they're now
specified directly in the nix.conf option ‘binary-cache-public-keys’.
The new command ‘nix-store --generate-binary-cache-key’ generates and
prints a public and secret key.
It moves runHook to a later position in the rcfile. After that we are
able to set the PS1 environment-variable for a nix-shell environment
e.g.:
# turn the color of the prompt to blue
shellHook = ''
export PS1="\n\[\033[1;34m\][\u@\h:\w]$\[\033[0m\] ";
'';
‘--run’ is like ‘--command’, except that it runs the command in a
non-interactive shell. This is important if you do things like:
$ nix-shell --command make
Hitting Ctrl-C while make is running drops you into the interactive
Nix shell, which is probably not what you want. So you can now do
$ nix-shell --run make
instead.
So you can have a script like:
#! /usr/bin/env nix-shell
#! nix-shell script.nix -i python
import prettytable
x = prettytable.PrettyTable(["Foo", "Bar"])
for i in range(1, 10): x.add_row([i, i**2])
print x
with a ‘script.nix’ in the same directory:
with import <nixpkgs> {};
runCommand "dummy" { buildInputs = [ python pythonPackages.prettytable ]; } ""
(Of course, in this particular case, using the ‘-p’ flag is more
convenient.)
This allows scripts to fetch their own dependencies via nix-shell. For
instance, here is a Haskell script that, when executed, pulls in GHC
and the HTTP package:
#! /usr/bin/env nix-shell
#! nix-shell -i runghc -p haskellPackages.ghc haskellPackages.HTTP
import Network.HTTP
main = do
resp <- Network.HTTP.simpleHTTP (getRequest "http://nixos.org/")
body <- getResponseBody resp
print (take 100 body)
Or a Perl script that pulls in Perl and some CPAN packages:
#! /usr/bin/env nix-shell
#! nix-shell -i perl -p perl perlPackages.HTMLTokeParserSimple perlPackages.LWP
use HTML::TokeParser::Simple;
my $p = HTML::TokeParser::Simple->new(url => 'http://nixos.org/');
while (my $token = $p->get_tag("a")) {
my $href = $token->get_attr("href");
print "$href\n" if $href;
}
Note that the options to nix-shell must be given on a separate line
that starts with the magic string ‘#! nix-shell’. This is because
‘env’ does not allow passing arguments to an interpreter directly.
Apparently, turning on utf8 encoding on stderr changes its flushing
behaviour, causing sendReply to not send anything.
http://hydra.nixos.org/build/13944384
