max/nix-super
1
0
Fork 0
mirror of https://github.com/privatevoid-net/nix-super.git synced 2024-11-23 14:36:16 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
4665 commits 7 branches 0 tags 64 MiB
Commit graph

2638 commits

Author SHA1 Message Date
Eelco Dolstra
21ef1670b3 Fix test failures 2016-04-21 18:21:25 +02:00
Eelco Dolstra
7d14f5c331 Implement S3BinaryCacheStore::queryAllValidPaths() 
This allows commands like "nix verify --all" or "nix path-info --all"
to work on S3 caches.

Unfortunately, this requires some ugly hackery: when querying the
contents of the bucket, we don't want to have to read every .narinfo
file. But the S3 bucket keys only include the hash part of each store
path, not the name part. So as a special exception
queryAllValidPaths() can now return store paths *without* the name
part, and queryPathInfo() accepts such store paths (returning a
ValidPathInfo object containing the full name).
 2016-04-21 17:53:47 +02:00
Eelco Dolstra
d155d80155 Move S3BinaryCacheStore from Hydra 
This allows running arbitrary Nix commands against an S3 binary cache.

To do: make this a compile time option to prevent a dependency on
aws-sdk-cpp.
 2016-04-21 16:08:51 +02:00
Eelco Dolstra
1a71495273 nix path-info: Add 
Forgot to commit this earlier...
 2016-04-21 15:00:45 +02:00
Eelco Dolstra
69e3ffb076 nix --help: Show usage examples 2016-04-21 14:58:32 +02:00
Eelco Dolstra
1b0088ebb2 nix --help: Show short flags 2016-04-21 14:34:46 +02:00
Eelco Dolstra
ddea253ff8 RemoteStore: Propagate InvalidPath exceptions from the daemon 2016-04-20 15:28:07 +02:00
Eelco Dolstra
c0c4ddcd9c BinaryCacheStore: Insert new paths into the disk cache 2016-04-20 15:27:48 +02:00
Eelco Dolstra
451ebf24ce Cache path info lookups in SQLite 
This re-implements the binary cache database in C++, allowing it to be
used by other Store backends, in particular the S3 backend.
 2016-04-20 14:12:38 +02:00
Eelco Dolstra
e0204f8d46 Move path info caching from BinaryCacheStore to Store 
Caching path info is generally useful. For instance, it speeds up "nix
path-info -rS /run/current-system" (i.e. showing the closure sizes of
all paths in the closure of the current system) from 5.6s to 0.15s.

This also eliminates some APIs like Store::queryDeriver() and
Store::queryReferences().
 2016-04-19 18:52:53 +02:00
Dan Peebles
608b0265e1 Print out all bad references/requisites at once 
Also updates tests to check for new information. Fixes #799
 2016-04-16 19:58:26 -04:00
Eelco Dolstra
0423787086 Make the .narinfo cache bigger 2016-04-15 15:39:48 +02:00
Eelco Dolstra
a7d8eaba54 BinaryCacheStore: Do negative caching of .narinfo lookups 2016-04-15 15:39:48 +02:00
Eelco Dolstra
d1b0909894 BinaryCacheStore::readFile(): Return a shared_ptr to a string 
This allows readFile() to indicate that a file doesn't exist, and
might eliminate some large string copying.
 2016-04-15 15:39:48 +02:00
Eelco Dolstra
99851c6f06 Unify "nix verify-paths" and "nix verify-store" 
"verify-store" is now simply an "--all" flag to "nix verify". This
flag can be used for any other store path command as well (e.g. "nix
path-info", "nix copy-sigs", ...).
 2016-04-15 15:39:48 +02:00
Eelco Dolstra
31a1a8ed3c Merge pull request #815 from vcunat/p/outputsToInstall 
nix-env: respect meta.outputsToInstall
 2016-04-14 20:55:41 +02:00
Eelco Dolstra
5169a6da98 Make $NIX_PATH parsing more robust 2016-04-14 17:29:08 +02:00
Eelco Dolstra
c045630522 Support channel:<channel-name> URIs 
For convenience, you can now say

  $ nix-env -f channel:nixos-16.03 -iA hello

instead of

  $ nix-env -f https://nixos.org/channels/nixos-16.03/nixexprs.tar.xz -iA hello

Similarly,

  $ nix-shell -I channel:nixpkgs-unstable -p hello
  $ nix-build channel:nixos-15.09 -A hello

Abstracting over the NixOS/Nixpkgs channels location also allows us to
use a more efficient transport (e.g. Git) in the future.
 2016-04-14 17:26:57 +02:00
Eelco Dolstra
363f37d084 Make the search path lazier with non-fatal errors 
Thus, -I / $NIX_PATH entries are now downloaded only when they are
needed for evaluation. An error to download an entry is a non-fatal
warning (just like non-existant paths).

This does change the semantics of builtins.nixPath, which now returns
the original, rather than resulting path. E.g., before we had

  [ { path = "/nix/store/hgm3yxf1lrrwa3z14zpqaj5p9vs0qklk-nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ]

but now

  [ { path = "https://nixos.org/channels/nixos-16.03/nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ]

Fixes #792.
 2016-04-14 15:32:24 +02:00
Eelco Dolstra
3c1c6b8f00 Set RLIMIT_CORE to 0, infinity in builders 
This prevents the builder from being affected by whatever the host
system limits happen to be.
 2016-04-14 13:39:14 +02:00
Eelco Dolstra
12b257f045 Make primop registration pluggable 
This way we don't have to put all primops in one giant file.
 2016-04-14 12:50:01 +02:00
Eelco Dolstra
96515b0c0d Fix std::atomic_flag build failure 
http://hydra.nixos.org/build/34453794
 2016-04-14 12:50:01 +02:00
Eelco Dolstra
867967265b Remove manifest support 
Manifests have been superseded by binary caches for years. This also
gets rid of nix-pull, nix-generate-patches and bsdiff/bspatch.
 2016-04-11 16:20:15 +02:00
Eelco Dolstra
8cffec8485 Remove failed build caching 
This feature was implemented for Hydra, but Hydra no longer uses it.
 2016-04-08 18:19:04 +02:00
Eelco Dolstra
f398949b40 Make LocalStore thread-safe 
Necessary for multi-threaded commands like "nix verify-paths".
 2016-04-08 18:07:13 +02:00
Eelco Dolstra
05fbc606fc nix verify-paths: Add ‘--sigs-needed <N>’ flag 
This specifies the number of distinct signatures required to consider
each path "trusted".

Also renamed ‘--no-sigs’ to ‘--no-trust’ for the flag that disables
verifying whether a path is trusted (since a path can also be trusted
if it has no signatures, but was built locally).
 2016-04-07 15:16:57 +02:00
Eelco Dolstra
6b2ae52808 Use secret-key-files for verifying 2016-04-07 15:16:57 +02:00
Eelco Dolstra
e39999ed48 Sign locally-built paths 
Locally-built paths are now signed automatically using the secret keys
specified by the ‘secret-key-files’ option.
 2016-04-07 15:16:57 +02:00
Eelco Dolstra
dc82160164 Fix "tar: This does not look like a tar archive" with fetchTarball / -I http:// 
The 304 Not Modified was not handled correctly, so the empty result
from the conditional request would overwrite the previous tarball.
 2016-04-06 16:57:20 +02:00
Eelco Dolstra
b654381eb3 Add "nix sign-paths" command 
E.g.

  $ nix sign-paths -k ./secret -r $(type -p geeqie)

signs geeqie and all its dependencies using the key in ./secret.
 2016-04-05 16:39:29 +02:00
Eelco Dolstra
d0f5719c2a Add "nix copy-sigs" command 
This imports signatures from one store into another. E.g.

  $ nix copy-sigs -r /run/current-system -s https://cache.nixos.org/
  imported 595 signatures
 2016-04-05 16:39:23 +02:00
Eelco Dolstra
80da7a6375 Probably fix SQLITE_BUSY errors 2016-04-05 15:29:56 +02:00
Eelco Dolstra
37a337bcec throwSQLiteError(): Check for SIGINT so we don't loop forever 2016-04-04 15:08:08 +02:00
Eelco Dolstra
6e120b76ee Add missing -pthread 
https://hydra.nixos.org/build/33908385
 2016-03-31 12:42:48 +02:00
Eelco Dolstra
9cee600c88 LocalStore: Keep track of ultimately trusted paths 
These are content-addressed paths or outputs of locally performed
builds. They are trusted even if they don't have signatures, so "nix
verify-paths" won't complain about them.
 2016-03-30 17:35:48 +02:00
Eelco Dolstra
3d119f0a3b Improve the SQLite wrapper API 
In particular, this eliminates a bunch of boilerplate code.
 2016-03-30 15:50:45 +02:00
Eelco Dolstra
d9c5e3bbf0 Factour out SQLite handling 2016-03-30 13:27:25 +02:00
Eelco Dolstra
2ae43ced9a Turn retrying SQLite transactions into a higher-order function 2016-03-30 12:04:27 +02:00
Eelco Dolstra
39a6abc0bc nix verify: Support checking against signatures in other stores 
Typical usage is to check local paths using the signatures from a
binary cache:

  $ nix verify-paths -r /run/current-system -s https://cache.nixos.org
  path ‘/nix/store/c1k4zqfb74wba5sn4yflb044gvap0x6k-nixos-system-mandark-16.03.git.fc2d7a5M’ is untrusted
  ...
  checked 844 paths, 119 untrusted
 2016-03-30 11:39:34 +02:00
Eelco Dolstra
88541569a2 HttpBinaryCacheStore: Treat 403 errors as 404 2016-03-30 11:17:51 +02:00
Eelco Dolstra
de88004a9d CurlDownloader: Fix HTTP error processing 2016-03-30 11:17:35 +02:00
Eelco Dolstra
ab3ce1cc13 Improve SIGINT handling in multi-threaded programs 
The flag remembering whether an Interrupted exception was thrown is
now thread-local. Thus, all threads will (eventually) throw
Interrupted. Previously, one thread would throw Interrupted, and then
the other threads wouldn't see that they were supposed to quit.
 2016-03-29 16:37:16 +02:00
Eelco Dolstra
4f34c40398 Add "nix verify-store" command 
Like "nix-store --verify --check-contents", but with the same
advantages as "nix verify-paths".
 2016-03-29 16:37:16 +02:00
Eelco Dolstra
784ee35c80 Add "nix verify-paths" command 
Unlike "nix-store --verify-path", this command verifies signatures in
addition to store path contents, is multi-threaded (especially useful
when verifying binary caches), and has a progress indicator.

Example use:

$ nix verify-paths --store https://cache.nixos.org -r $(type -p thunderbird)
...
[17/132 checked] checking ‘/nix/store/rawakphadqrqxr6zri2rmnxh03gqkrl3-autogen-5.18.6’
 2016-03-29 16:37:16 +02:00
Eelco Dolstra
0ebe69dc67 Re-enable sync_with_stdio 
Otherwise writing to std::cerr is not thread-safe (in particular,
lines will be randomly duplicated).
 2016-03-29 16:37:16 +02:00
Eelco Dolstra
374198ad6d Move signature support from NarInfo to ValidPathInfo 2016-03-24 11:41:00 +01:00
Eelco Dolstra
11525377e1 Typos 2016-03-24 11:27:58 +01:00
Eelco Dolstra
8b7839b608 HttpBinaryCacheStore: Make thread-safe 2016-03-24 11:10:05 +01:00
Eelco Dolstra
c7d44bad00 Drop support for daemon socket path >= 108 characters 
Doing a chdir() is a bad idea in multi-threaded programs, leading to
failures such as

  error: cannot connect to daemon at ‘/nix/var/nix/daemon-socket/socket’: No such file or directory

Since Linux doesn't have a connectat() syscall like FreeBSD, there is
no way we can support this in a race-free way.
 2016-03-23 17:16:16 +01:00
Eelco Dolstra
056b3ecfa4 LocalStoreAccessor::stat: Handle ENOTDIR 
Closes https://github.com/NixOS/hydra/pull/286.
 2016-03-23 11:17:46 +01:00
Eelco Dolstra
bb1034316d Don't overload dumpPath() 2016-03-22 14:21:45 +01:00
Eelco Dolstra
712b616a84 Move signatures from NarInfo to ValidPathInfo 
This allows queryPathInfo() to return signatures.
 2016-03-21 18:05:47 +01:00
Eelco Dolstra
cebc150b7c nix: Add --store flag 
This is a bit user-friendlier than using $NIX_REMOTE.
 2016-03-21 18:03:36 +01:00
Eelco Dolstra
1c5f73f529 Add Store::dumpPath() method 
This allows applying nix-store --verify-path to binary cache stores:

  NIX_REMOTE=https://cache.nixos.org nix-store --verify-path /nix/store/s5c7...
 2016-03-21 17:55:57 +01:00
Eelco Dolstra
87295b9844 Drop support for upgrading from Nix <= 0.12 2016-03-21 15:09:03 +01:00
Eelco Dolstra
02654f782f Fix Darwin build 
http://hydra.nixos.org/build/33279996
 2016-03-15 12:11:27 +01:00
Eelco Dolstra
00a75b1cd2 Fix another mismatched tag 
http://hydra.nixos.org/build/33279570
 2016-03-14 15:32:34 +01:00
Eelco Dolstra
dc4a71aae5 Fix build on clang due to -Wmismatched-tags 
http://hydra.nixos.org/build/33073389
 2016-03-14 12:37:30 +01:00
Eelco Dolstra
e7c76f7274 BinaryCacheStore::isValidPath(): Use .narinfo cache 
If a path is in the .narinfo cache, obviously it's valid.
 2016-03-14 12:33:06 +01:00
Eelco Dolstra
674c5ff64f Merge pull request #834 from abbradar/master 
Propagate path context via builtins.readFile
 2016-03-10 20:17:03 +01:00
Eelco Dolstra
b4e0335d4d Add option binary-cache-secret-key-file for signing binary caches 2016-03-04 17:45:22 +01:00
Eelco Dolstra
af7cdb1096 BinaryCacheStore: Remove publicKeyFile argument 
The public key can be derived from the secret key, so there's no need
for the user to supply it separately.
 2016-03-04 17:45:22 +01:00
Eelco Dolstra
42bc395b63 Eliminate some large string copying 2016-03-04 16:49:56 +01:00
Eelco Dolstra
ce113c32d2 Add warning about "nix" being experimental 2016-03-04 15:54:41 +01:00
Eelco Dolstra
7c9d7a253c Merge branch 'new-cli' 2016-03-04 15:21:23 +01:00
Nikolay Amiantov
f7f0116dd7 Propagate path context via builtins.readFile 2016-03-04 14:04:51 +03:00
Eelco Dolstra
76f1ba4f3b Add file missing from 201b48de60 2016-03-03 18:03:34 +01:00
Nathan Zadoks
fe2be8f016 build.cc: fs.h doesn't appear to be necessary anymore 2016-03-03 14:11:00 +01:00
Eelco Dolstra
5a8455c85e Provide function required by Hydra 2016-03-02 18:21:48 +01:00
Eelco Dolstra
25eea97873 Force stack trace for boost format errors 2016-03-02 15:46:07 +01:00
Eelco Dolstra
201b48de60 Add an HTTP binary cache store 
Allowing stuff like

  NIX_REMOTE=https://cache.nixos.org nix-store -qR /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1

or

  NIX_REMOTE=https://cache.nixos.org nix-store --export /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 | nix-store --import
 2016-03-02 15:46:07 +01:00
Eelco Dolstra
0402b6398d Eliminate local-binary-cache-store.hh 2016-03-02 15:46:07 +01:00
Eelco Dolstra
68a5414982 Make store implementations pluggable 
This for instance allows hydra-queue-runner to add the S3 backend
at runtime.
 2016-03-02 15:46:07 +01:00
Eelco Dolstra
6055d84beb Fix reading symlinks 
The st_size field of a symlink doesn't have to be correct, e.g. for
/dev/fd symlinks.
 2016-03-02 15:46:07 +01:00
Eelco Dolstra
0a62d9b3d7 Remove bad assertion 2016-03-02 15:46:07 +01:00
Eelco Dolstra
89a2fa68ac FdSource: track number of bytes read 2016-03-02 15:46:07 +01:00
Eelco Dolstra
cca4a8dc1a importPaths(): Optionally add NARs to binary cache accessor 
This enables an optimisation in hydra-queue-runner, preventing a
download of a NAR it just uploaded to the cache when reading files
like hydra-build-products.
 2016-03-02 15:46:07 +01:00
Eelco Dolstra
e9c50064b5 Add an HTTP binary cache store 
Allowing stuff like

  NIX_REMOTE=https://cache.nixos.org nix-store -qR /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1

or

  NIX_REMOTE=https://cache.nixos.org nix-store --export /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 | nix-store --import
 2016-02-29 18:15:20 +01:00
Eelco Dolstra
6170bb474b Eliminate local-binary-cache-store.hh 2016-02-29 16:14:39 +01:00
Eelco Dolstra
0b907321cc Make store implementations pluggable 
This for instance allows hydra-queue-runner to add the S3 backend
at runtime.
 2016-02-29 16:11:11 +01:00
Eelco Dolstra
012f8d187c Fix reading symlinks 
The st_size field of a symlink doesn't have to be correct, e.g. for
/dev/fd symlinks.
 2016-02-28 03:27:34 +01:00
Eelco Dolstra
d0344dd2c1 Remove bad assertion 2016-02-26 21:43:59 +01:00
Eelco Dolstra
581e1bc5b4 FdSource: track number of bytes read 2016-02-26 16:16:08 +01:00
Eelco Dolstra
bcc9943cee importPaths(): Optionally add NARs to binary cache accessor 
This enables an optimisation in hydra-queue-runner, preventing a
download of a NAR it just uploaded to the cache when reading files
like hydra-build-products.
 2016-02-26 15:20:10 +01:00
Eelco Dolstra
00b2c05749 nix: Add commands to query contents of NARs / binary caches 
For example,

  $ NIX_REMOTE=file:///my-cache nix ls-store -lR /nix/store/f4kbgl8shhyy76rkk3nbxr0lz8d2ip7q-binutils-2.23.1
  dr-xr-xr-x                    0 ./bin
  -r-xr-xr-x                30748 ./bin/addr2line
  -r-xr-xr-x                66973 ./bin/ar
  ...

Similarly, "nix ls-nar" lists the contents of a NAR file, "nix
cat-nar" extracts a file from a NAR file, and "nix cat-store" extract
a file from a Nix store.
 2016-02-25 17:57:00 +01:00
Eelco Dolstra
24a8f9e27b Merge branch 'master' into new-cli 2016-02-25 17:48:35 +01:00
Eelco Dolstra
1042c10fd0 Add NAR / Store accessor abstraction 
This is primary to allow hydra-queue-runner to extract files like
"nix-support/hydra-build-products" from NARs in binary caches.
 2016-02-25 17:43:19 +01:00
Eelco Dolstra
c5bc571861 Fix short boolean flags 2016-02-25 13:31:34 +01:00
Eelco Dolstra
7873cfb18d Fix build 2016-02-25 11:55:05 +01:00
Eelco Dolstra
f1bdeac986 Merge branch 'master' into new-cli 2016-02-25 11:25:11 +01:00
Eelco Dolstra
152b1d6bf9 deletePath(): Succeed if path doesn't exist 
Also makes it robust against concurrent deletions.
 2016-02-24 17:44:55 +01:00
Eelco Dolstra
28e7e29abd Eliminate reserveSpace flag 2016-02-24 17:44:55 +01:00
Eelco Dolstra
5a64e66268 LocalStore::addTextToStore(): Don't read the path we just wrote 
This eliminates some unnecessary (presumably cached) I/O.
 2016-02-24 17:11:31 +01:00
Eelco Dolstra
9ccbd55c5b BinaryCacheStore: Implement addToStore() 
So now you can do

  $ NIX_REMOTE=file:///tmp/binary-cache nix-instantiate '<nixpkgs>' -A hello

and lots of other operations.
 2016-02-24 16:52:28 +01:00
Eelco Dolstra
30e9d01516 openStoreAt(): Initialise the binary cache 2016-02-24 16:52:07 +01:00
Eelco Dolstra
ba0a81d14f Show a proper error message for unimplemented binary cache operations 2016-02-24 14:57:30 +01:00
Eelco Dolstra
45c83e5f9b Typo 2016-02-24 14:49:28 +01:00
Eelco Dolstra
263187a2ec Move BinaryCacheStore / LocalBinaryCacheStore from Hydra 
So you can now do:

  $ NIX_REMOTE=file:///tmp/binary-cache nix-store -qR /nix/store/...
 2016-02-24 14:48:16 +01:00
Eelco Dolstra
b584a0e7de Merge pull request #816 from tsion/incomplete-parse 
Throw a specific error for incomplete parse errors.
 2016-02-24 13:34:56 +01:00
Eelco Dolstra
bf2adf72c4 std::condition_variable_any -> std::condition_variable 
The latter is supposed to be more efficient.
 2016-02-24 13:31:46 +01:00