Commit graph

4215 commits

Author SHA1 Message Date
Eelco Dolstra
2b0365753a Merge branch 'limit_depth_resolveExprPath' of https://github.com/d-goldin/nix 2019-12-15 00:22:35 +01:00
Eelco Dolstra
8656a2de56
Merge pull request #3269 from xzfc/nix-shell
nix-shell: don't check for "nix-shell" in shebang script name
2019-12-14 23:24:32 +01:00
Eelco Dolstra
ba6d2093c7 Fix progress bar 2019-12-14 23:19:04 +01:00
Albert Safin
a70706b025 nix-shell: don't check for "nix-shell" in shebang script name 2019-12-14 15:37:20 +00:00
Eelco Dolstra
ac9cc2ec08 Move some code 2019-12-13 19:10:39 +01:00
Eelco Dolstra
b4edc3ca61 Don't leak exceptions 2019-12-13 19:05:26 +01:00
Eelco Dolstra
e6bd88878e Improve gzip error message 2019-12-13 19:05:26 +01:00
Eelco Dolstra
ca87707c90 Get rid of CBox 2019-12-13 19:05:26 +01:00
Dima
d89d9958a7 bugfix: Adding depth limit to resolveExprPath
There is no termination condition for evaluation of cyclical
expression paths which can lead to infinite loops. This addresses
one spot in the parser in a similar fashion as utils.cc/canonPath
does.

This issue can be reproduced by something like:

```
ln -s a b
ln -s b a

nix-instantiate -E 'import ./a'
```
2019-12-13 14:51:30 +01:00
Eelco Dolstra
e8aa2290ed Only install *.sb files on macOS 2019-12-13 14:42:55 +01:00
Eelco Dolstra
3e787423c2 Remove FIXME 2019-12-13 12:55:52 +01:00
Eelco Dolstra
d1b238ec3c Simplify 2019-12-13 12:53:20 +01:00
Eelco Dolstra
2da4c61049 Merge branch 'libstore-ssh-better-exec-error-message' of https://github.com/Profpatsch/nix 2019-12-13 12:51:36 +01:00
Tom Bereknyei
c6295a3afd Initial gzip support
Closes #3256
2019-12-13 03:34:15 -05:00
Profpatsch
38b29fb72c libstore/ssh: Improve error message on failing execvp
If the `throw` is reached, this means that execvp into `ssh` wasn’t
successful. We can hint at a usual problem, which is a missing `ssh`
executable.

Test with:

```
env PATH= ./result/bin/nix-copy-closure --builders '' unusedhost
```

and the bash version with

```
env PATH= ./result/bin/nix-copy-closure --builders '' localhost
```
2019-12-12 15:32:17 +01:00
Eelco Dolstra
bbe97dff8b Make the Store API more type-safe
Most functions now take a StorePath argument rather than a Path (which
is just an alias for std::string). The StorePath constructor ensures
that the path is syntactically correct (i.e. it looks like
<store-dir>/<base32-hash>-<name>). Similarly, functions like
buildPaths() now take a StorePathWithOutputs, rather than abusing Path
by adding a '!<outputs>' suffix.

Note that the StorePath type is implemented in Rust. This involves
some hackery to allow Rust values to be used directly in C++, via a
helper type whose destructor calls the Rust type's drop()
function. The main issue is the dynamic nature of C++ move semantics:
after we have moved a Rust value, we should not call the drop function
on the original value. So when we move a value, we set the original
value to bitwise zero, and the destructor only calls drop() if the
value is not bitwise zero. This should be sufficient for most types.

Also lots of minor cleanups to the C++ API to make it more modern
(e.g. using std::optional and std::string_view in some places).
2019-12-10 22:06:05 +01:00
Eelco Dolstra
98ef11677c EvalState::callFunction(): Make FunctionCallTrace use less stack space
The FunctionCallTrace object consumes a few hundred bytes of stack
space, even when tracing is disabled. This was causing stack overflows:

  $ nix-instantiate '<nixpkgs> -A texlive.combined.scheme-full --dry-run
  error: stack overflow (possible infinite recursion)

This is with the default stack size of 8 MiB.

Putting the object on the heap reduces stack usage to < 5 MiB.
2019-12-10 13:32:30 +01:00
Eelco Dolstra
61cc9f34d2 Remove UserLock self-lock check
This is no longer needed since we're not using POSIX locks anymore.
2019-12-09 23:57:33 +01:00
Yorick van Pelt
eba82b7c88
further clean up libarchive code 2019-12-09 17:21:46 +07:00
Puck Meerburg
28ee687adf Clean up libarchive support 2019-12-07 18:12:21 +00:00
Yorick van Pelt
fe7ec70e6b
remove rust unpack_tarfile ffi 2019-12-07 23:28:31 +07:00
Yorick van Pelt
1355554d12
code 'cleanup' 2019-12-07 23:23:11 +07:00
Yorick van Pelt
f54c168031
add wrapper function around libarchive to c++ errors 2019-12-07 23:10:27 +07:00
Yorick van Pelt
232b390766
fixup! libarchive proof of concept 2019-12-07 23:00:37 +07:00
Yorick van Pelt
9ff5f6492f
libarchive proof of concept 2019-12-07 22:35:14 +07:00
Eelco Dolstra
3b9c9d34e5 Shut up clang warning
(cherry picked from commit 3392f1b778)
2019-12-05 20:41:44 +01:00
Eelco Dolstra
80ab95315d nix doctor: Fix typo
(cherry picked from commit 96c6b08ed7)
2019-12-05 20:40:52 +01:00
Eelco Dolstra
47a937d512 Show hash mismatch warnings in SRI format
(cherry picked from commit 63c5c91cc0)
2019-12-05 20:32:25 +01:00
Eelco Dolstra
0678e4d56a Move #include
(cherry picked from commit 8beedd4486)
2019-12-05 20:30:29 +01:00
Eelco Dolstra
79142cbbe1 Bindings: Add convenience method for requiring an attribute
(cherry picked from commit fb692e5f7b)
2019-12-05 20:29:15 +01:00
Eelco Dolstra
0d118ef0c9 Bindings::get(): Add convenience method
This allows writing attribute lookups as

    if (auto name = value.attrs->get(state.sName))
      ...

(cherry picked from commit f216c76c56)
2019-12-05 20:29:00 +01:00
Eelco Dolstra
5e449b43ed Initialize Command::_name
(cherry picked from commit d0a769cb06)
2019-12-05 20:21:22 +01:00
Eelco Dolstra
ac67685606 Make subcommand construction in MultiCommand lazy
(cherry picked from commit a0de58f471)
2019-12-05 20:19:26 +01:00
Eelco Dolstra
f964f428fe Move Command and MultiCommand to libutil
(cherry picked from commit f70434b1fb)
2019-12-05 20:13:47 +01:00
Eelco Dolstra
f1b5c76c1a MultiCommand: Simplify construction
(cherry picked from commit 15a16e5c05)
2019-12-05 20:10:35 +01:00
Eelco Dolstra
092af3c826 Eliminate more pass-by-value in variadic calls 2019-12-05 19:58:52 +01:00
Eelco Dolstra
603b2f583c Revert "Make fmt() non-recursive"
This reverts commit 2b761d5f50.

Also *really* make fmt() take arguments by reference.
2019-12-05 19:58:49 +01:00
Eelco Dolstra
334b8f8af1 fmt(): Pass arguments by reference rather than by value 2019-12-05 17:40:46 +01:00
Eelco Dolstra
ac2bc721d8 Merge remote-tracking branch 'origin/recursive-nix' 2019-12-02 12:34:46 +01:00
Brian Wignall
8737980e75 Fix typos 2019-11-30 19:04:14 -05:00
Eelco Dolstra
f102d793f1
Merge pull request #2748 from edolstra/rust
Make nix/unpack-channel.nix a builtin builder
2019-11-29 19:33:31 +01:00
Eelco Dolstra
2d6f1ddbb5
Remove builtins.valueSize
Fixes #3246.
2019-11-28 13:52:42 +01:00
Eelco Dolstra
949dc84894 Fix segfault on i686-linux
https://hydra.nixos.org/build/107467517

Seems that on i686-linux, gcc and rustc disagree on how to return
1-word structs: gcc has the caller pass a pointer to the result, while
rustc has the callee return the result in a register. Work around this
by using a bare pointer.
2019-11-27 14:17:15 +01:00
Eelco Dolstra
8918bae098 Drop remaining uses of external "tar"
Also, fetchGit now runs in O(1) memory since we pipe the output of
'git archive' directly into unpackTarball() (rather than first reading
it all into memory).
2019-11-26 22:07:28 +01:00
Eelco Dolstra
d33dd6e6c0 Move code around 2019-11-26 22:07:28 +01:00
Eelco Dolstra
b7fba16613 Move code around 2019-11-26 22:07:28 +01:00
Eelco Dolstra
f738cd4d97 More Rust FFI adventures
We can now convert Rust Errors to C++ exceptions. At the Rust->C++ FFI
boundary, Result<T, Error> will cause Error to be converted to and
thrown as a C++ exception.
2019-11-26 22:07:28 +01:00
Eelco Dolstra
8110b4ebb2 Rust cleanup 2019-11-26 22:07:28 +01:00
Eelco Dolstra
045708db43 Make <nix/unpack-channel.nix> a builtin builder
This was the last function using a shell script, so this allows us to
get rid of tar, coreutils, bash etc.
2019-11-26 22:07:28 +01:00
Eelco Dolstra
e60f6bd4ce Enable Rust code to call C++ Source objects 2019-11-26 22:07:28 +01:00
Eelco Dolstra
11da5b2816 Add some Rust code 2019-11-26 22:07:28 +01:00
Eelco Dolstra
abb8ef619b
Fix macOS build
https://hydra.nixos.org/build/107457009
2019-11-26 21:08:56 +01:00
Eelco Dolstra
313106d549
Fix clang warnings 2019-11-26 21:07:44 +01:00
Eelco Dolstra
425991883a
Merge pull request #3141 from xbreak/nocafile
Downloader: Log configured CA file
2019-11-26 20:52:25 +01:00
Eelco Dolstra
7c8d7c17f8
Merge pull request #3144 from matthewbauer/fix-sandbox-fallback
Fix sandbox fallback settings
2019-11-26 20:51:52 +01:00
Eelco Dolstra
ec5e7b44ff
Simplify 2019-11-26 20:26:22 +01:00
Eelco Dolstra
96e1c39bb7
Merge branch 'repair-bad-links' of https://github.com/chkno/nix 2019-11-26 20:21:48 +01:00
Eelco Dolstra
872740cf60
Merge pull request #3238 from puckipedia/attrset-overrides-dynamic
Ensure enough space in attrset bindings
2019-11-26 20:14:55 +01:00
Eelco Dolstra
c13193017f
Disallow empty store path names
Fixes #3239.
2019-11-26 20:12:15 +01:00
Eelco Dolstra
89db9353d7
Doh 2019-11-26 20:08:25 +01:00
Eelco Dolstra
1ec6e6e11e
Add feature to disable URL literals
E.g.

  $ nix-build '<nixpkgs>' -A hello --experimental-features no-url-literals
  error: URL literals are disabled, at /nix/store/vsjamkzh15r3c779q2711az826hqgvzr-nixpkgs-20.03pre194957.bef773ed53f/nixpkgs/pkgs/top-level/all-packages.nix:1236:11

Helps with implementing https://github.com/NixOS/rfcs/pull/45.
2019-11-26 19:48:34 +01:00
Puck Meerburg
cd55f91ad2 Ensure enough space in attrset bindings when using both __overrides and dynamic attributes 2019-11-25 12:37:14 +00:00
Eelco Dolstra
d12d69ea1a
Turn NIX_PATH into a config setting
This allows it to be set in nix.conf.
2019-11-22 23:07:35 +01:00
Eelco Dolstra
ec9dd9a5ae
Provide a default value for NIX_PATH 2019-11-22 22:08:51 +01:00
Eelco Dolstra
ba87b08f85
getEnv(): Return std::optional
This allows distinguishing between an empty value and no value.
2019-11-22 16:18:13 +01:00
Chuck
3e2c77d001 Check for and repair bad .links entries
A corrupt entry in .links prevents adding a fixed version of that file
to the store in any path.  The user experience is that corruption
present in the store 'spreads' to new paths added to the store:

(With store optimisation enabled)

1. A file in the store gets corrupted somehow (eg: filesystem bug).
2. The user tries to add a thing to the store which contains a good copy
   of the corrupted file.
3. The file being added to the store is hashed, found to match the bad
   .links entry, and is replaced by a link to the bad .links entry.
   (The .links entry's hash is not verified during add -- this would
   impose a substantial performance burden.)
4. The user observes that the thing in the store that is supposed to be
   a copy of what they were trying to add is not a correct copy -- some
   files have different contents!  Running "nix-store --verify
   --check-contents --repair" does not fix the problem.

This change makes "nix-store --verify --check-contents --repair" fix
this problem.  Bad .links entries are simply removed, allowing future
attempts to insert a good copy of the file to succeed.
2019-11-15 11:55:36 -08:00
Eelco Dolstra
0352c1a4f8
Typo 2019-11-13 17:18:17 +01:00
Eelco Dolstra
5ee23c35b9
Merge pull request #3219 from Ericson2314/semicolons
Fix extra semicolons warnings
2019-11-11 12:13:51 +01:00
John Ericson
8669db1dcc Clean up semicolon and comma
Thanks @bhipple for catching!
2019-11-10 16:21:59 -05:00
John Ericson
4c34054673 Remove unneeded semicolons 2019-11-10 11:24:47 -05:00
John Ericson
96e6e680c1 Fix extra ; warnings involving MakeError 2019-11-10 11:24:47 -05:00
Peter Kolloch
2ba9f22715
De-duplicate struct PrimOp forward declaration 2019-11-10 10:02:22 +01:00
Eelco Dolstra
0d6774468c
Move editorFor srom libutil to nix
libutil should not depend on libexpr.
2019-11-08 15:13:32 +01:00
Eelco Dolstra
4145cd2da0
Use upstream nlohmann_json 2019-11-07 18:23:17 +01:00
Eelco Dolstra
04bf9acd22
Remove #include 2019-11-07 10:12:35 +01:00
Eelco Dolstra
39a2e166dd
Cleanup 2019-11-06 16:53:02 +01:00
Eelco Dolstra
69326f3637
Recursive Nix: Handle concurrent client connections 2019-11-06 00:55:03 +01:00
Eelco Dolstra
c119ab9db0
Enable recursive Nix using a feature
Derivations that want to use recursion should now set

  requiredSystemFeatures = [ "recursive-nix" ];

to make the daemon socket appear.

Also, Nix should be configured with "experimental-features =
recursive-nix".
2019-11-06 00:55:03 +01:00
Eelco Dolstra
c921074c19
RestrictedStore: Implement addToStore() 2019-11-06 00:55:03 +01:00
Eelco Dolstra
c4d7c76b64
Recursive Nix support
This allows Nix builders to call Nix to build derivations, with some
limitations.

Example:

  let nixpkgs = fetchTarball channel:nixos-18.03; in

  with import <nixpkgs> {};

  runCommand "foo"
    {
      buildInputs = [ nix jq ];
      NIX_PATH = "nixpkgs=${nixpkgs}";
    }
    ''
      hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })')

      $hello/bin/hello

      mkdir -p $out/bin
      ln -s $hello/bin/hello $out/bin/hello

      nix path-info -r --json $hello | jq .
    ''

This derivation makes a recursive Nix call to build GNU Hello and
symlinks it from its $out, i.e.

  # ll ./result/bin/
  lrwxrwxrwx 1 root root 63 Jan  1  1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello

  # nix-store -qR ./result
  /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131
  /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5
  /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo

This is implemented as follows:

* Before running the outer builder, Nix creates a Unix domain socket
  '.nix-socket' in the builder's temporary directory and sets
  $NIX_REMOTE to point to it. It starts a thread to process
  connections to this socket. (Thus you don't need to have nix-daemon
  running.)

* The daemon thread uses a wrapper store (RestrictedStore) to keep
  track of paths added through recursive Nix calls, to implement some
  restrictions (see below), and to do some censorship (e.g. for
  purity, queryPathInfo() won't return impure information such as
  signatures and timestamps).

* After the build finishes, the output paths are scanned for
  references to the paths added through recursive Nix calls (in
  addition to the inputs closure). Thus, in the example above, $out
  has a reference to $hello.

The main restriction on recursive Nix calls is that they cannot do
arbitrary substitutions. For example, doing

  nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10

is forbidden unless /nix/store/kmwd... is in the inputs closure or
previously built by a recursive Nix call. This is to prevent
irreproducible derivations that have hidden dependencies on
substituters or the current store contents. Building a derivation is
fine, however, and Nix will use substitutes if available. In other
words, the builder has to present proof that it knows how to build a
desired store path from scratch by constructing a derivation graph for
that path.

Probably we should also disallow instantiating/building fixed-output
derivations (specifically, those that access the network, but
currently we have no way to mark fixed-output derivations that don't
access the network). Otherwise sandboxed derivations can bypass
sandbox restrictions and access the network.

When sandboxing is enabled, we make paths appear in the sandbox of the
builder by entering the mount namespace of the builder and
bind-mounting each path. This is tricky because we do a pivot_root()
in the builder to change the root directory of its mount namespace,
and thus the host /nix/store is not visible in the mount namespace of
the builder. To get around this, just before doing pivot_root(), we
branch a second mount namespace that shares its /nix/store mountpoint
with the parent.

Recursive Nix currently doesn't work on macOS in sandboxed mode
(because we can't change the sandbox policy of a running build) and in
non-root mode (because setns() barfs).
2019-11-06 00:52:38 +01:00
Eelco Dolstra
d823381c0a
Merge branch 'fix/nix-doctor-output' of https://github.com/bhipple/nix 2019-11-05 16:04:40 +01:00
Eelco Dolstra
852554bb16
Merge branch 'nix-repl-e' of https://github.com/zimbatm/nix 2019-11-05 11:20:53 +01:00
Eelco Dolstra
1b600ecd14
Don't use SOCK_CLOEXEC on macOS
https://hydra.nixos.org/build/105428308
2019-11-05 10:25:09 +01:00
Eelco Dolstra
f5a46ef0b1
Merge pull request #3202 from kraem/master
Update nix eval --help msg to not include deprecated command
2019-11-04 09:34:30 +01:00
Harald van Dijk
c935ad3f02
Fix progress bar when nix-prefetch-url is piped.
The intent of the code was that if the window size cannot be determined,
it would be treated as having the maximum possible size. Because of a
missing assignment, it was actually treated as having a width of 0.

The reason the width could not be determined was because it was obtained
from stdout, not stderr, even though the printing was done to stderr.

This commit addresses both issues.
2019-11-03 21:46:59 +00:00
kraem
dcd7a26063
Update nix eval --help msg to not include deprecated command 2019-11-03 18:47:28 +01:00
ng0
b811bd2172 include netinet/in.h in src/nix/main.cc
Fixes #3186
2019-11-01 14:09:42 +00:00
Eelco Dolstra
6bff1aa46d
Merge pull request #3182 from bhipple/fixup/comments
Minor updates to inline comments
2019-10-31 14:14:35 +01:00
Benjamin Hipple
80d5ec6ff4 Minor updates to inline comments
Add missing docstring on InstallableCommand. Also, some of these were wrapped
when they're right next to a line longer than the unwrapped line, so we can just
unwrap them to save vertical space.
2019-10-31 05:56:37 -04:00
Will Dietz
0e9b72e097 minor: fix mismatch of struct/class forward decl of 'Source'
Fixes the following warning and the indicate potential issue:

src/libstore/worker-protocol.hh:66:1: warning: class 'Source' was previously declared as a struct; this is valid, but may result in linker errors
under the Microsoft C++ ABI [-Wmismatched-tags]

(cherry picked from commit 6e1bb04870b1b723282d32182af286646f13bf3c)
2019-10-30 14:39:01 -05:00
Eelco Dolstra
e5319a87ce
queryPathInfoUncached(): Return const ValidPathInfo 2019-10-29 13:53:04 +01:00
Eelco Dolstra
992a2ad475
Move addToStoreFromDump to Store 2019-10-29 13:38:24 +01:00
Eelco Dolstra
05819d013f
Don't create a Store in processConnection() 2019-10-29 13:36:19 +01:00
Eelco Dolstra
63b99af85a
Move Unix domain socket creation to libutil
Also drop multithread-unfriendly hacks like doing a temporary
chmod/umask.
2019-10-29 13:30:51 +01:00
Eelco Dolstra
2d37e88319
Move most of the daemon implementation to libstore 2019-10-29 13:25:33 +01:00
Eelco Dolstra
95c727caef
Remove the check against concurrent builds in the same process 2019-10-29 12:43:20 +01:00
zimbatm
9a25059656
findDerivationFilename: add FIXME 2019-10-28 21:40:02 +01:00
zimbatm
d407f4d15f
nix repl: also handle lambda edit 2019-10-28 21:37:22 +01:00
zimbatm
3774fe55fd
editorFor: take a pos object instead 2019-10-28 21:36:34 +01:00
zimbatm
ec448f8bb6
libexpr: findDerivationFilename return Pos instead of tuple 2019-10-28 21:29:54 +01:00