Commit graph

682 commits

Author SHA1 Message Date
Shea Levy
aa8bbbf69d
Merge branch 'write-failure-fixes' of git://github.com/lheckemann/nix 2018-03-02 10:59:59 -05:00
Tuomas Tynkkynen
a0e38c16bc libexpr: Recognize newline in more places in lexer
Flex's regexes have an annoying feature: the dot matches everything
except a newline. This causes problems for expressions like:

"${0}\
"

where the backslash-newline combination matches this rule instead of the
intended one mentioned in the comment:

    <STRING>\$|\\|\$\\ {
                    /* This can only occur when we reach EOF, otherwise the above
                    (...|\$[^\{\"\\]|\\.|\$\\.)+ would have triggered.
                    This is technically invalid, but we leave the problem to the
                    parser who fails with exact location. */
                    return STR;
                }
However, the parser actually accepts the resulting token sequence
('"' DOLLAR_CURLY 0 '}' STR '"'), which is a problem because the lexer
rule didn't assign anything to yylval. Ultimately this leads to a crash
when dereferencing a NULL pointer in ExprConcatStrings::bindVars().

The fix does change the syntax of the language in some corner cases
but I think it's only turning previously invalid (or crashing) syntax
to valid syntax. E.g.

"a\
b"

and

''a''\
b''

were previously syntax errors but now both result in "a\nb".

Found by afl-fuzz.
2018-03-02 17:30:48 +02:00
Will Dietz
c577186f59 tests/search.sh: simplify, don't rely on consistent ordering 2018-03-01 13:25:06 -06:00
Tuomas Tynkkynen
77e9e1ed91 libexpr: Fix prim_replaceStrings() to work on an empty source string
Otherwise, running e.g.

nix-instantiate --eval -E --strict 'builtins.replaceStrings [""] ["X"] "abc"'

would just hang in an infinite loop.

Found by afl-fuzz.

First attempt of this was reverted in e2d71bd186 because it caused
another infinite loop, which is fixed now and a test added.
2018-02-26 19:49:13 +02:00
Will Dietz
8282c60d74 tests: test nix search behavior 2018-02-25 16:40:05 -06:00
Eelco Dolstra
de4c03d201
Merge branch 'fix/dry-run-partially' of https://github.com/dtzWill/nix 2018-02-22 12:18:20 +01:00
Eelco Dolstra
e2d71bd186
Revert "libexpr: Fix prim_replaceStrings() to work on an empty source string"
This reverts commit 4ea9707591.

It causes an infinite loop in Nixpkgs evaluation,
e.g. "nix-instantiate -A hello" hung.

PR #1886.
2018-02-21 15:35:28 +01:00
Tuomas Tynkkynen
4ea9707591 libexpr: Fix prim_replaceStrings() to work on an empty source string
Otherwise, running e.g.

nix-instantiate --eval -E --strict 'builtins.replaceStrings [""] ["X"] "abc"'

would just hang in an infinite loop.

Found by afl-fuzz.
2018-02-19 23:20:26 +02:00
Eelco Dolstra
75a1d96cfd
Merge branch 'register-settings' of https://github.com/shlevy/nix 2018-02-19 13:58:34 +01:00
Shea Levy
b095c06139
Add splitVersion primop.
Fixes #1868.
2018-02-14 09:55:43 -05:00
Shea Levy
de4934ab3b
Allow plugins to define new settings. 2018-02-13 14:43:32 -05:00
Shea Levy
3fe9767dd3
Fix plugin tests on darwin 2018-02-13 12:49:14 -05:00
Linus Heckemann
78ac3eb4eb Fix #1762
nix-store --export, nix-store --dump, and nix dump-path would previously
fail silently if writing the data out failed, because
 a) FdSink::write ignored exceptions, and
 b) the commands relied on FdSink's destructor, which ignores
    exceptions, to flush the data out.

This could cause rather opaque issues with installing nixos, because
nix-store --export would happily proceed even if it couldn't write its
data out (e.g. if nix-store --import on the other side of the pipe
failed).

This commit adds tests that expose these issues in the nix-store
commands, and fixes them for all three.
2018-02-13 14:26:19 +00:00
Shea Levy
6eb1040e90
Allow includes from nix.conf 2018-02-13 08:16:32 -05:00
Eelco Dolstra
7828dca9e8
Merge branch 'register-constant' of https://github.com/shlevy/nix 2018-02-13 12:24:48 +01:00
Eelco Dolstra
1c10a74c73
Merge branch 'plugins' of https://github.com/shlevy/nix 2018-02-13 12:15:27 +01:00
Shea Levy
081f14a169
Allow using RegisterPrimop to define constants.
This enables plugins to add new constants, as well as new primops.
2018-02-08 14:35:50 -05:00
Shea Levy
88cd2d41ac
Add plugins to make Nix more extensible.
All plugins in plugin-files will be dlopened, allowing them to
statically construct instances of the various Register* types Nix
supports.
2018-02-08 12:44:37 -05:00
Eelco Dolstra
ad97a21834
nix-env: Fix parsing of --system
https://hydra.nixos.org/build/68827814
2018-02-08 15:25:03 +01:00
Eelco Dolstra
444bae44ef
dsa -> ed25519
DSS is disabled by default in NixOS 18.03.

https://hydra.nixos.org/build/68788560
2018-02-08 13:46:23 +01:00
Will Dietz
c7e0be1bfc build-dry: disable failing portion of test until issue is fixed 2018-02-07 15:20:54 -06:00
Will Dietz
3780435a0e tests: Add (failing) tests for reported --dry-run issues. 2018-02-07 15:20:54 -06:00
Shea Levy
48c192ca2d
builtins.path test: Don't rely on shlevy's XDG_RUNTIME_DIR 2018-02-07 10:26:53 -05:00
Eelco Dolstra
84989d3af2
Improve filtering of ANSI escape sequences in build logs
All ANSI sequences except color setting are now filtered out. In
particular, terminal resets (such as from NixOS VM tests) are filtered
out.

Also, fix the completely broken tab character handling.
2018-02-07 15:23:57 +01:00
Eelco Dolstra
abe6be578b
Merge pull request #1816 from shlevy/add-path
Add path primop.
2018-02-07 13:32:35 +01:00
Shea Levy
69d82e5c58
Add path primop.
builtins.path allows specifying the name of a path (which makes paths
with store-illegal names now addable), allows adding paths with flat
instead of recursive hashes, allows specifying a filter (so is a
generalization of filterSource), and allows specifying an expected
hash (enabling safe path adding in pure mode).
2018-02-06 16:48:08 -05:00
Eelco Dolstra
43f8ef73c6
realiseContext(): Add derivation outputs to the allowed paths
This makes import-from-derivation work in restricted mode again.
2018-02-06 15:38:45 +01:00
Eelco Dolstra
f539085e65
Fix evaluation 2018-02-05 21:48:09 +01:00
Eelco Dolstra
84722d67d2
Remove nix-build --hash
Instead, if a fixed-output derivation produces has an incorrect output
hash, we now unconditionally move the outputs to the path
corresponding with the actual hash and register it as valid. Thus,
after correcting the hash in the Nix expression (e.g. in a fetchurl
call), the fixed-output derivation doesn't have to be built again.

It would still be good to have a command for reporting the actual hash
of a fixed-output derivation (instead of throwing an error), but
"nix-build --hash" didn't do that.
2018-02-03 10:08:05 +01:00
Eelco Dolstra
855699855f
Remove obsolete references to manifests
Closes #323.
2018-02-01 10:39:16 +01:00
Eelco Dolstra
98f3c75a0e
Merge pull request #1797 from dezgeg/userns-tests-fix
Fix tests using user namespaces on kernels that don't have it
2018-01-24 17:17:48 +01:00
Eelco Dolstra
c382866cd2
Fix test
https://hydra.nixos.org/build/67806811
2018-01-22 17:04:08 +01:00
Eelco Dolstra
9304fde8de
Don't access tarballs.nixos.org in a test
https://hydra.nixos.org/build/67806811
2018-01-22 16:56:28 +01:00
Eelco Dolstra
3c4c30eadd
Rewrite builtin derivation environment
Also add a test.

Fixes #1803.
Closes #1805.
2018-01-19 15:00:38 +01:00
Eelco Dolstra
87e3d142cc
Add a test for --check / --repeat 2018-01-19 13:58:28 +01:00
Eelco Dolstra
d4dcffd643
Add pure evaluation mode
In this mode, the following restrictions apply:

* The builtins currentTime, currentSystem and storePath throw an
  error.

* $NIX_PATH and -I are ignored.

* fetchGit and fetchMercurial require a revision hash.

* fetchurl and fetchTarball require a sha256 attribute.

* No file system access is allowed outside of the paths returned by
  fetch{Git,Mercurial,url,Tarball}. Thus 'nix build -f ./foo.nix' is
  not allowed.

Thus, the evaluation result is completely reproducible from the
command line arguments. E.g.

  nix build --pure-eval '(
    let
      nix = fetchGit { url = https://github.com/NixOS/nixpkgs.git; rev = "9c927de4b179a6dd210dd88d34bda8af4b575680"; };
      nixpkgs = fetchGit { url = https://github.com/NixOS/nixpkgs.git; ref = "release-17.09"; rev = "66b4de79e3841530e6d9c6baf98702aa1f7124e4"; };
    in (import (nix + "/release.nix") { inherit nix nixpkgs; }).build.x86_64-linux
  )'

The goal is to enable completely reproducible and traceable
evaluation. For example, a NixOS configuration could be fully
described by a single Git commit hash. 'nixos-rebuild' would do
something like

  nix build --pure-eval '(
    (import (fetchGit { url = file:///my-nixos-config; rev = "..."; })).system
  ')

where the Git repository /my-nixos-config would use further fetchGit
calls or Git externals to fetch Nixpkgs and whatever other
dependencies it has. Either way, the commit hash would uniquely
identify the NixOS configuration and allow it to reproduced.
2018-01-16 19:23:18 +01:00
Tuomas Tynkkynen
59086e459c Fix tests using user namespaces on kernels that don't have it
Disable various tests if the kernel doesn't support unprivileged user
namespaces (e.g. Arch Linux disables them) or disable them via a sysctl
(Debian, Ubuntu).

Fixes #1521
Fixes #1625
2018-01-13 19:08:41 +02:00
Will Dietz
84d9e213d2 fetchGit.sh: Test we don't "corrupt" cache if invoke w/o git avail 2018-01-09 08:58:19 -06:00
Eelco Dolstra
44272d8719
Rename "use-substitutes" to "substitute"
Commit c2154d4c84 renamed
"build-use-substitutes" to "use-substitutes", but that broke
"nix-copy-closure --use-substitutes".
2018-01-04 16:58:39 +01:00
Eelco Dolstra
f68c2b5a78
Merge pull request #1770 from dtzWill/fix/run-test-sandbox-ubuntu
run.sh: include lib64 in sandbox-paths to fix on ubuntu 16.XX
2018-01-02 11:32:18 +01:00
Will Dietz
5afee18726 run.sh: include lib64 in sandbox-paths to fix on ubuntu 16.XX
(cc #1769)
2017-12-30 22:41:49 -06:00
Will Dietz
9dd2b8ac7b use libbrotli directly when available
* Look for both 'brotli' and 'bro' as external command,
  since upstream has renamed it in newer versions.
  If neither are found, current runtime behavior
  is preserved: try to find 'bro' on PATH.
* Limit amount handed to BrotliEncoderCompressStream
  to ensure interrupts are processed in a timely manner.
  Testing shows negligible performance impact.
  (Other compression sinks don't seem to require this)
2017-12-30 20:26:33 -06:00
Will Dietz
2e6f06c37e fetchGit: Fix handling of local repo when not using 'master' branch
Add tests checking this behavior.
2017-12-22 15:29:52 -06:00
Eelco Dolstra
6d80870832
release.nix: Use fetchTarball and fetchGit
In particular, using fetchGit means we don't need hackery to clean the
source tree when building from an unclean tree.
2017-12-22 11:35:32 +01:00
Eelco Dolstra
e3013543d3 Fix test 2017-12-07 01:07:07 +01:00
Eelco Dolstra
338f29dbd4 nix ls-{nar,store}: Return offset of files in the NAR if known
E.g.

  $ nix ls-store --json --recursive --store https://cache.nixos.org /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79 \
    | jq .entries.bin.entries.blender.narOffset
  400
2017-12-07 01:07:07 +01:00
Shea Levy
11a7f8ce14
Merge branch 'fetchGit-fast-revision-update' 2017-12-05 11:16:39 -05:00
aszlig
6567ab95a0
build-remote: Fix missing log output
The storeUri variable in the build-remote hook is declared very much to
the start of the main function and a bunch of lines later, the same
variable gets checked via hasPrefix() but it gets assigned *after* that
check when the most suitable machine for the build was choosen.

So I guess this was just a typo in d16fd24973
and what we really want is to either checkd the prefix *after* assigning
storeUri or use bestMachine->storeUri directly.

I choose the latter, because the former could introduce even more
regressions if the try block where the variable gets assigned terminates
early.

Nevertheless, the reason why the log output didn't work is because
hasPrefix() checked for "ssh://" in front of storeUri, but if the
storeUri isn't set correctly (or at all), we don't get the log file
descriptor set up properly, leading to no log output.

I've adjusted the remote-builds test to include a regression test for
this, so that we can make sure we get a build output when using remote
builds.

In addition to that I've tested this with two of my build farms and the
build logs are emitted correctly again.

Signed-off-by: aszlig <aszlig@nix.build>
2017-11-25 01:34:18 +01:00
Eelco Dolstra
90948a4e3a
nix-shell/nix-build: Support .drv files again
Fixes #1663.

Also handle '!<output-name>' (#1694).
2017-11-24 18:08:35 +01:00
Shea Levy
eedbc4e06c
fetchGit: Ignore tarballTtl if rev is set and not in the repo.
Fixes #1697.
2017-11-24 06:09:24 -05:00
Eelco Dolstra
0fc3e581e0
tests/run.sh: Fix /build conflict when building in a sandbox
https://hydra.nixos.org/build/64519355
2017-11-21 19:43:12 +01:00
Eelco Dolstra
d7da6c9ea9
fetchGit/fetchMercurial: Fix directory inclusion check
E.g. the existence of .gitignore would cause .git to be included.
2017-11-21 19:34:46 +01:00
Eelco Dolstra
4fcf44825f
Add tests for verifying/copying content-addressed paths
These don't require signatures.
2017-11-20 19:11:02 +01:00
Eelco Dolstra
0c9718aabc
Add tests for signature checking when copying between local stores 2017-11-20 19:02:57 +01:00
Eelco Dolstra
4cde04f476
Add tests for "nix run" 2017-11-20 18:36:36 +01:00
Eelco Dolstra
193330d321
Test: Replace --option with the corresponding flag 2017-11-20 17:50:49 +01:00
Eelco Dolstra
91a1987607
signed-binary-caches -> require-sigs
Unlike signed-binary-caches (which could only be '*' or ''),
require-sigs is a proper Boolean option. The default is true.
2017-11-20 17:44:07 +01:00
Eelco Dolstra
7a2b64e55c
binary-cache-public-keys -> trusted-public-keys
The name had become a misnomer since it's not only for substitution
from binary caches, but when adding/copying any
(non-content-addressed) path to a store.
2017-11-20 17:32:34 +01:00
Eelco Dolstra
27131866c9
Doh 2017-11-15 12:23:31 +01:00
Eelco Dolstra
ec5b04862b
nix sign-paths: Support binary caches 2017-11-14 18:44:05 +01:00
Eelco Dolstra
d6dbda7004
Add tests for "nix verify", "nix sign-paths" etc. 2017-11-14 18:24:20 +01:00
Eelco Dolstra
2c39e4eca0
Revert "Don't parse "x:x" as a URI"
This reverts commit f90f660b24.

This broke Hydra's release.nix, which contained

  preCheck = ''export LOGNAME=${LOGNAME:-foo}'';
2017-11-14 15:10:52 +01:00
Eelco Dolstra
5773d667ee
nix ls-{nar,store}: Don't abort on missing files 2017-11-14 14:49:06 +01:00
Eelco Dolstra
b8d446829e
Add some tests 2017-11-14 14:37:39 +01:00
Eelco Dolstra
c9c3fc710b
Rename tests/nar-index -> tests/nar-access 2017-11-14 13:27:40 +01:00
Eelco Dolstra
fd10f6f241
Show when tests are skipped
Also, don't depend on tput (ncurses). It's really not needed since
ANSI escape sequences have been standardized for 35 years or so.
2017-11-07 12:09:57 +01:00
Eelco Dolstra
ee6ac38848
fetchGit/fetchMercurial: Filter out directories with untracked files 2017-11-03 13:55:31 +01:00
Eelco Dolstra
4dee01da7c
fetchGit: Add a test 2017-11-03 13:55:30 +01:00
Eelco Dolstra
e026bc3b05
fetchMercurial: Don't fetch hashes we already have 2017-11-01 18:43:11 +01:00
Eelco Dolstra
1969f357b7
Add fetchMercurial primop
E.g.

  $ nix eval '(fetchMercurial https://www.mercurial-scm.org/repo/hello)'
  { branch = "default"; outPath = "/nix/store/alvb9y1kfz42bjishqmyy3pphnrh1pfa-source"; rev = "82e55d328c8ca4ee16520036c0aaace03a5beb65"; revCount = 1; shortRev = "82e55d328c8c"; }

  $ nix eval '(fetchMercurial { url = https://www.mercurial-scm.org/repo/hello; rev = "0a04b987be5ae354b710cefeba0e2d9de7ad41a9"; })'
  { branch = "default"; outPath = "/nix/store/alvb9y1kfz42bjishqmyy3pphnrh1pfa-source"; rev = "0a04b987be5ae354b710cefeba0e2d9de7ad41a9"; revCount = 0; shortRev = "0a04b987be5a"; }

  $ nix eval '(fetchMercurial /tmp/unclean-hg-tree)'
  { branch = "default"; outPath = "/nix/store/cm750cdw1x8wfpm3jq7mz09r30l9r024-source"; rev = "0000000000000000000000000000000000000000"; revCount = 0; shortRev = "000000000000"; }
2017-11-01 17:45:32 +01:00
Eelco Dolstra
f90f660b24
Don't parse "x:x" as a URI
URIs now have to contain "://" or start with "channel:".
2017-10-30 17:58:01 +01:00
Eelco Dolstra
812e027e1d
Add option allowed-uris
This allows network access in restricted eval mode.
2017-10-30 12:41:49 +01:00
Eelco Dolstra
82327e3cc4
exportReferencesGraph: Allow exporting a list of store paths 2017-10-25 15:18:49 +02:00
Eelco Dolstra
3395e3bbc4
Fix exportReferencesGraph in the structured attrs case 2017-10-25 14:08:29 +02:00
Eelco Dolstra
2d5b1b24bf
Pass lists/attrsets to bash as (associative) arrays 2017-10-25 13:01:50 +02:00
Eelco Dolstra
d16fd24973
Allow shorter syntax for chroot stores
You can now say '--store /tmp/nix' instead of '--store local?root=/tmp/nix'.
2017-10-24 15:32:38 +02:00
Eelco Dolstra
3460e4cf00
More progress indicator improvements
In particular, don't show superfluous "fetching path" and "building
path(s)" messages, and show the current round (with --repeat).
2017-10-24 15:32:38 +02:00
Eelco Dolstra
d4609bb3af
NIX_BUILD_HOOK variable is gone 2017-10-23 20:50:28 +02:00
Eelco Dolstra
37fbfffd8e
Pass all settings to build-remote
This ensures that command line flags such as --builders get passed
correctly.
2017-10-23 20:50:28 +02:00
Dan Peebles
3139cad9cd Fix tests
`nix copy` no longer accepts a `--recursive` argument
2017-10-02 23:45:27 -04:00
Eelco Dolstra
c2154d4c84
Rename a few configuration options
In particular, drop the "build-" and "gc-" prefixes which are
pointless. So now you can say

  nix build --no-sandbox

instead of

  nix build --no-build-use-sandbox
2017-08-31 14:28:25 +02:00
Nicolas B. Pierron
b8867a0239 Add builtins.string function.
The function 'builtins.split' takes a POSIX extended regular expression
and an arbitrary string. It returns a list of non-matching substring
interleaved by lists of matched groups of the regular expression.

```nix
with builtins;
assert split "(a)b" "abc"      == [ "" [ "a" ] "c" ];
assert split "([ac])" "abc"    == [ "" [ "a" ] "b" [ "c" ] "" ];
assert split "(a)|(c)" "abc"   == [ "" [ "a" null ] "b" [ null "c" ] "" ];
assert split "([[:upper:]]+)" "  FOO   "
                               == [ "  " [ "FOO" ] "   " ];
```
2017-08-15 20:04:11 +00:00
Jörg Thalheim
2fd8f8bb99 Replace Unicode quotes in user-facing strings by ASCII
Relevant RFC: NixOS/rfcs#4

$ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 12:32:45 +01:00
Graham Christensen
364f75e03a Update mailing list. 2017-07-18 08:04:01 -04:00
Eelco Dolstra
bf6792c0df
Always use base-16 for hashed mirror lookups
In particular, don't use base-64, which we don't support. (We do have
base-32 redirects for hysterical reasons.)

Also, add a test for the hashed mirror feature.
2017-07-17 13:13:18 +02:00
Eelco Dolstra
49304bae81
Make the hashes mirrors used by builtins.fetchurl configurable
In particular, this allows it to be disabled in our tests.
2017-07-17 13:07:08 +02:00
Graham Christensen
e0d39c8dc4
Rename PINCH_ME_IM_SILLY to ALLOW_PREEXISTING_INSTALLATION 2017-07-14 12:11:33 -04:00
Graham Christensen
12f6bb33d2
If there is no TTY, also skip verbose sudo messages 2017-07-14 12:11:30 -04:00
Graham Christensen
ce2281e6d8
Ensure PINCH_ME_IM_SILLY allows a /nix/store to stick around between builds
Also output in the status report that the user is very silly
2017-07-14 12:11:26 -04:00
Graham Christensen
1c7ce2a018
Assume yes if we have no TTY
Starve the TTY of input to ensure this works, but provide yes to the
current installer to handle the current broken case.
2017-07-14 12:11:23 -04:00
Graham Christensen
73a57a2f22
Cleanup and more specificity around set -e 2017-07-14 12:11:20 -04:00
Graham Christensen
3839dda2ec
Only clean if the file exists 2017-07-14 12:11:17 -04:00
Graham Christensen
92ca93528f
Clean up nix hints from the old insstaller 2017-07-14 12:11:13 -04:00
Graham Christensen
61ea9e9867
Run nix-build inside a fresh bash login 2017-07-14 12:11:10 -04:00
Graham Christensen
bc647fd299
chmod 2017-07-14 12:11:07 -04:00
Graham Christensen
a0369b14f4
Test the installer 2017-07-14 12:11:04 -04:00
Robert Vollmert
01722b3d2c Remove unused variable from test script 2017-07-06 22:37:53 +02:00
Eelco Dolstra
c0015e87af
Support base-64 hashes
Also simplify the Hash API.

Fixes #1437.
2017-07-04 15:07:41 +02:00
Eelco Dolstra
1888f7889b
macOS: Ugly hack to make the tests succeed
Sandboxes cannot be nested, so if Nix's build runs inside a sandbox,
it cannot use a sandbox itself. I don't see a clean way to detect
whether we're in a sandbox, so use a test-specific hack.

https://github.com/NixOS/nix/issues/1413
2017-06-19 14:26:05 +02:00
Eelco Dolstra
1d9ab273ba
Add test for setuid seccomp filter 2017-05-29 16:14:10 +02:00