max/nix-super
1
0
Fork 0
mirror of https://github.com/privatevoid-net/nix-super.git synced 2024-11-14 02:06:16 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
10527 commits 7 branches 0 tags 64 MiB
Commit graph

6663 commits

Author SHA1 Message Date
Eelco Dolstra
bccff827dc Fix deadlock in IFD through the daemon 
Fixes #4235.
 2020-11-17 13:50:36 +01:00
regnat
7de21f6664 Make the sql debug statements more useful 
Print the expanded sql query (with the variables bound to their value) rather
than the original one in case of error
 2020-11-17 10:05:45 +01:00
Eelco Dolstra
ef84c780bb
filterANSIEscapes(): Handle UTF-8 characters 2020-11-16 16:41:53 +01:00
Eelco Dolstra
0d6419ad87
Merge pull request #4262 from jbaum98/master 
Fix deadlock in nix-store when max-connections=1
 2020-11-16 10:20:36 +01:00
Jake Waksbaum
01db455733 Fix deadlock in nix-store when max-connections=1 
This fixes a bug I encountered where `nix-store -qR` will deadlock when
the `--include-outputs` flag is passed and `max-connections=1`.

The deadlock occurs because `RemoteStore::queryDerivationOutputs` takes
the only connection from the connection pool and uses it to check the
daemon version. If the version is new enough, it calls
`Store::queryDerivationOutputs`, which eventually calls
`RemoteStore::queryPartialDerivationOutputMap`, where we take another
connection from the connection pool to check the version again. Because
we still haven't released the connection from the caller, this waits for
a connection to be available, causing a deadlock.

This diff solves the issue by using `getProtocol` to check the protocol
version in the caller `RemoteStore::queryDerivationOutputs`, which
immediately frees the connection back to the pool before returning the
protocol version. That way we've already freed the connection by the
time we call `RemoteStore::queryPartialDerivationOutputMap`.
 2020-11-16 02:35:50 -05:00
Robert Hensing
ac5081d280 nix-build: Fix #4197 output order regression 2020-11-13 17:49:27 +01:00
Christian Höppner
c4c3c15c19 Fix default nix-path 
The default nix-path values for nixpkgs and root channels were
incorrect.
 2020-11-12 15:46:08 +00:00
Ben Burdette
b327de9c2d change message 2020-11-11 11:09:59 -07:00
Ben Burdette
8895fa70a4 pare down the error message 2020-11-11 11:05:21 -07:00
Ben Burdette
3edfe6090e missing argument error 2020-11-11 09:29:32 -07:00
Matthew Kenigsberg
8abb80a478 Print built derivations as json for build 
Add --json option to nix build to allow machine readable output on
stdout with all built derivations

Fixes #1930
 2020-11-11 10:27:02 -06:00
Ben Burdette
7d9037035e usage example location 2020-11-11 09:21:26 -07:00
Eelco Dolstra
4badb6943f Fix use of dirty Git/Mercurial inputs with chroot stores 
Fixes:

  $ nix build --store /tmp/nix /home/eelco/Dev/patchelf#hydraJobs.build.x86_64-linux
  warning: Git tree '/home/eelco/Dev/patchelf' is dirty
  error: --- RestrictedPathError ------------------------------------------------------------------------------------------- nix
  access to path '/tmp/nix/nix/store/xmkvfmffk7xfnazykb5kx999aika8an4-source/flake.nix' is forbidden in restricted mode
  (use '--show-trace' to show detailed location information)
 2020-11-10 23:22:45 +01:00
Eelco Dolstra
cdc840d60b
Merge pull request #4242 from wizeman/fix-stack-overflow 
Fix stack overflow introduced in #4206
 2020-11-10 11:51:00 +01:00
Eelco Dolstra
3f680c1dcc
Merge pull request #4233 from Kha/master 
nix develop: Preserve stdin with `-c`
 2020-11-10 10:51:48 +01:00
Ricardo M. Correia
108a2dab7e Fix stack overflow introduced in #4206 2020-11-10 04:25:24 +01:00
Ben Burdette
d8ef423a18 error message formatting 2020-11-09 19:16:50 -07:00
Ben Burdette
9f2b25ce55 remove unused ftn; reformat line breaks 2020-11-09 17:17:47 -07:00
Ben Burdette
6c2933a8d7 add position 2020-11-09 17:04:52 -07:00
Ben Burdette
107c91f5fe auto-call error 2020-11-09 16:48:35 -07:00
Sebastian Ullrich
fb7735e4cf nix develop: Preserve stdin with -c 2020-11-09 22:43:14 +01:00
Eelco Dolstra
b87f84cf55
Fix appending to Setting<StringSet> 
Fixes: warning: unknown setting 'extra-sandbox-paths'
 2020-11-09 15:04:34 +01:00
Maximilian Bosch
3a63fc6cd5
Allow substituting paths when building remotely using ssh-ng:// 
Until now, it was not possible to substitute missing paths from e.g.
`https://cache.nixos.org` on a remote server when building on it using
the new `ssh-ng` protocol.

This is because every store implementation except legacy `ssh://`
ignores the substitution flag passed to `Store::queryValidPaths` while
the `legacy-ssh-store` substitutes the remote store using
`cmdQueryValidPaths` when the remote store is opened with `nix-store
--serve`.

This patch slightly modifies the daemon protocol to allow passing an
integer value suggesting whether to substitute missing paths during
`wopQueryValidPaths`. To implement this on the daemon-side, the
substitution logic from `nix-store --serve` has been moved into a
protected method named `Store::substitutePaths` which gets currently
called from `LocalStore::queryValidPaths` and `Store::queryValidPaths`
if `maybeSubstitute` is `true`.

Fixes #2770
 2020-11-05 20:12:37 +01:00
Eelco Dolstra
387f824cab
Merge pull request #4206 from hercules-ci/fix-coroutine-gc 
Fix memory corruption caused by GC-invisible coroutine stacks
 2020-11-05 10:18:31 +01:00
Eelco Dolstra
5e6eabe155 Fix error message 'assertion failed at' 2020-11-03 14:45:24 +01:00
Eelco Dolstra
e8c379555f LocalStore: Get rid of recursive_mutex 2020-11-03 14:45:24 +01:00
Eelco Dolstra
797a52e31d Add FIXME 2020-11-03 14:45:24 +01:00
Eelco Dolstra
550e11f077 nix repl: Fix handling of multi-line expressions 2020-11-02 19:07:37 +01:00
Eelco Dolstra
7cf874c17d Don't use readDerivation() in addValidPath() 
readDerivation() requires a valid path.

Fixes #4210.
 2020-11-02 18:46:44 +01:00
Eelco Dolstra
ab2ef851b6
Merge pull request #4207 from hercules-ci/fix-RemoteStore-filterSource-deadlock 
Fix RemoteStore pool deadlock in filterSource etc
 2020-11-02 14:46:10 +01:00
Eelco Dolstra
db5424bf09 Don't send eval-related settings to the daemon 2020-11-02 13:57:58 +01:00
Robert Hensing
e8a45d07bc Restore RestrictedStore.addToStoreFromDump implementation 
It was accidentally removed in commit ca30abb3fb
 2020-10-31 23:56:03 +01:00
Robert Hensing
b43c13a916 BoehmGCStackAllocator: increase stack size to 8MB 
The default stack size was not based on the normal stack size and
was too small.
 2020-10-30 23:18:26 +01:00
Robert Hensing
2192cac634 Fix RemoteStore pool deadlock in filterSource etc 2020-10-30 21:47:34 +01:00
Robert Hensing
c4d903ddb0 Fix memory corruption caused by GC-invisible coroutine stacks 
Crucially this introduces BoehmGCStackAllocator, but it also
adds a bunch of wiring to avoid making libutil depend on bdw-gc.

Part of the solutions for #4178, #4200
 2020-10-30 21:21:59 +01:00
Eelco Dolstra
dc5696b84f Fix test 2020-10-30 12:00:53 +01:00
Eelco Dolstra
7f56cf67ba Fix assertion failure in tab completion for --option 2020-10-29 18:26:35 +01:00
Eelco Dolstra
ff4dea63c9 Generalize extra-* settings 
This removes the extra-substituters and extra-sandbox-paths settings
and instead makes every array setting extensible by setting
"extra-<name> = <value>" in the configuration file or passing
"--<name> <value>" on the command line.
 2020-10-29 18:17:39 +01:00
Eelco Dolstra
bb8e837e4c
Merge pull request #4199 from stefanjaax/replaceWantedBySpecified 
Alter "wanted:" to "specified:" in hash mismatch output
 2020-10-29 07:38:38 +01:00
Eelco Dolstra
662e67f8de
Merge pull request #4198 from mkenigs/capitalize-JSON 
Capitalize JSON for consistency
 2020-10-29 07:34:34 +01:00
stev
869c0321ff Alter "wanted:" to "specified:" in hash mismatch output 
This makes it even clearer which of the two hashes was specified in the
nix files. Some may think that "wanted" and "got" is obvious, but:
"got" could mean "got in nix file" and "wanted" could mean "want to see in nix file".
 2020-10-29 00:33:14 +01:00
Matthew Kenigsberg
6a4bf535d8 Capitalize JSON for consistency 2020-10-28 17:54:28 -05:00
Eelco Dolstra
a5019f0508 Consistency 2020-10-28 20:45:57 +01:00
Eelco Dolstra
02a1facbdc
Merge pull request #4056 from tweag/non-ca-depending-on-ca 
Allow non-CA derivations to depend on CA ones
 2020-10-27 17:38:29 +01:00
regnat
bc081bcd81 Inline unkownHashes 
See https://github.com/NixOS/nix/pull/4056#discussion_r493661632
 2020-10-27 07:29:25 +01:00
regnat
c092fa4702 Allow non-CA derivations to depend on CA derivations 2020-10-27 07:29:23 +01:00
Eelco Dolstra
343239fc8a Allow nix.conf options to be set in flake.nix 
This makes it possible to have per-project configuration in flake.nix,
e.g. binary caches and other stuff:

  nixConfig.bash-prompt-suffix = "ngi# ";
  nixConfig.substituters = [ "https://cache.ngi0.nixos.org/" ];
 2020-10-26 20:45:39 +01:00
Eelco Dolstra
731edf0d9b isTrivial(): Support trivial lists 2020-10-26 20:37:11 +01:00
Eelco Dolstra
14aecbb288 BaseSetting<StringMap>::set(): Don't append to previous value 2020-10-26 20:36:46 +01:00
Eelco Dolstra
b875b8f45c Remove edition field 2020-10-26 17:59:36 +01:00
Eelco Dolstra
9d5e9ef0da Move Explicit 2020-10-26 17:01:20 +01:00
Eelco Dolstra
dc7d1322ef Make the prompt used in development shells configurable 2020-10-26 14:24:25 +01:00
Eelco Dolstra
750ce500c2 Fix clang build 2020-10-22 13:40:30 +02:00
Eelco Dolstra
f9438fb64a nix develop: Add --redirect flag to redirect dependencies 
This is primarily useful if you're hacking simultaneously on a package
and one of its dependencies. E.g. if you're hacking on Hydra and Nix,
you would start a dev shell for Nix, and then a dev shell for Hydra as
follows:

  $ nix develop \
    --redirect .#hydraJobs.build.x86_64-linux.nix ~/Dev/nix/outputs/out \
    --redirect .#hydraJobs.build.x86_64-linux.nix.dev ~/Dev/nix/outputs/dev

(This assumes hydraJobs.build.x86_64-linux has a passthru.nix
attribute. You can also use a store path.)

This causes all references in the environment to those store paths to
be rewritten to ~/Dev/nix/outputs/{out,dev}. Note: unfortunately, you
may need to set LD_LIBRARY_PATH=~/Dev/nix/outputs/out/lib because
Nixpkgs' ld-wrapper only adds -rpath entries for -L flags that point
to the Nix store.
 2020-10-22 13:40:30 +02:00
Eelco Dolstra
e556a1beb7 nix develop: Handle 'declare -ax' in bash output 
Fixes 'nix develop nixpkgs#qpdfview'.
 2020-10-21 17:54:21 +02:00
Christian Kampka
461cf2b856
Add NIX_CONFIG env var for applying nix.conf overrides 2020-10-21 13:41:26 +02:00
Matthew Kenigsberg
f6aaac2b59 Make bash non-interactive for nix develop --phase 
Fix #3975: Currently if Ctrl-C is pressed during a phase, the interactive subshell
is not exited. Removing --rcfile when --phase is present makes bash
non-interactive
 2020-10-20 12:00:30 -05:00
Eelco Dolstra
9635fb77bd
Merge pull request #4080 from kquick/kwq/flake-int-doc 
Add some internal documentation for flake support objects.
 2020-10-19 11:29:12 +02:00
Eelco Dolstra
fda835b231
Merge pull request #4143 from obsidiansystems/typed-goal-maps 
Properly type the derivation and substitution goal maps
 2020-10-18 18:12:21 +02:00
Robert Hensing
bd9eb5c743 DerivationGoal: only retry if output closure incomplete is only problem 2020-10-18 14:26:37 +02:00
Robert Hensing
94f1e4a441 Typo 2020-10-18 14:26:37 +02:00
John Ericson
7ed46c1574 Explain that upcast_goal is still a static cast 2020-10-17 21:50:12 +00:00
John Ericson
57d0432b39 Just use auto in two places. 2020-10-17 21:47:52 +00:00
John Ericson
e6f8ae56d8 tab -> space 2020-10-17 21:45:31 +00:00
John Ericson
619d262c97 Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-17 21:44:27 +00:00
John Ericson
64be1c15c2 Add missing include for MAX_PATH 
And remove one that we didn't actually need to add
 2020-10-15 19:05:17 +00:00
John Ericson
fccef6a7fa Merge remote-tracking branch 'upstream/master' into fix-and-ci-static-builds 2020-10-15 18:55:03 +00:00
John Ericson
0fefc2a439 Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-14 20:49:01 +00:00
Eelco Dolstra
11882d7c7c Create /etc/passwd *after* figuring out the sandbox uid/gid 
Fixes build failures like

  # nix log /nix/store/gjaa0psfcmqvw7ivggsncx9w364p3s8s-sshd.conf-validated.drv
  No user exists for uid 30012
 2020-10-14 12:20:58 +02:00
John Ericson
55592b253f Add some more docs 2020-10-13 18:04:24 +00:00
John Ericson
13804f126e Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-13 18:02:32 +00:00
Eelco Dolstra
2653801939 Merge branch 'split_build_cc' of https://github.com/obsidiansystems/nix 2020-10-13 15:36:55 +02:00
John Ericson
1b8ebe92dc Merge remote-tracking branch 'obsidian/split_build_cc' into typed-goal-maps 2020-10-12 20:47:22 +00:00
John Ericson
a73959e6be Merge remote-tracking branch 'upstream/master' into split_build_cc 2020-10-12 17:20:46 +00:00
John Ericson
d334fd4882 Merge branches 'split_build_hh-0', 'split_build_hh-1', 'split_build_hh-2', 'split_build_hh-3' and 'split_build_hh-4' into split_build_cc 2020-10-12 17:20:12 +00:00
John Ericson
542972f029 Trim worker.hh 2020-10-12 17:16:48 +00:00
John Ericson
0e2306204a Rename to hand-hold git (worker.hh) 2020-10-12 17:16:48 +00:00
John Ericson
3ffa3546bd Trim substitution-goal.hh 2020-10-12 17:16:25 +00:00
John Ericson
e77a2344d5 Rename to hand-hold git (substitution-goal.hh) 2020-10-12 17:16:25 +00:00
John Ericson
d585b4c54f Trim hook-instance.hh 2020-10-12 17:16:13 +00:00
John Ericson
10b749a156 Rename to hand-hold git (hook-instance.hh) 2020-10-12 17:16:13 +00:00
John Ericson
8067d32f2a Trim goal.hh 2020-10-12 17:16:00 +00:00
John Ericson
0d0e345cdc Rename to hand-hold git (goal.hh) 2020-10-12 17:16:00 +00:00
John Ericson
2ce726947a Trim derivation-goal.hh 2020-10-12 17:15:32 +00:00
John Ericson
4eb8c69853 Rename to hand-hold git (derivation-goal.hh) 2020-10-12 17:15:32 +00:00
John Ericson
f7099965bf Change .cc files to use split build headers 2020-10-12 17:08:52 +00:00
John Ericson
5a97621d6d Prepare for build/*.hh headers 2020-10-12 17:07:51 +00:00
volth
eee18f88dd
Handle amount of disk space saved by hard linking being negative 
Fixes bogus messages like "currently hard linking saves 17592186044416.00 MiB".
 2020-10-12 16:06:38 +00:00
John Ericson
15fdb7cc6b Split out uds-remote-store.{cc.hh} 2020-10-11 17:37:05 +00:00
John Ericson
38e3897162 Copy {uds-,}remote-store.{cc,hh} 
This prepares for the splitting that happens in the next commit.
 2020-10-11 17:18:26 +00:00
John Ericson
5c74a6147b Properly type the derivation and substitution goal maps 
As a bonus, Worker::removeGoal is less inefficient.
 2020-10-11 17:07:14 +00:00
Eelco Dolstra
b0fbf3a653
Merge pull request #4135 from obsidiansystems/split-local-fs-store-header 
Split out `local-fs-store.hh`
 2020-10-11 18:49:40 +02:00
John Ericson
8cc510fb79 Merge branches 'build-1', 'build-2', 'build-3', 'build-4', 'build-5', 'build-6', 'build-7' and 'build-8' into split_build_cc 2020-10-11 16:44:34 +00:00
John Ericson
bcb67e1ed8 Trim lock.cc 2020-10-11 16:44:19 +00:00
John Ericson
e0be04129b Rename to hand-hold git (lock.cc) 2020-10-11 16:44:14 +00:00
John Ericson
dbc588651c Trim lock.hh 2020-10-11 16:44:07 +00:00
John Ericson
d0004bfcab Rename to hand-hold git (lock.hh) 2020-10-11 16:43:59 +00:00
John Ericson
eed53ed87a Trim build/worker.cc 2020-10-11 16:43:51 +00:00
John Ericson
904e315dae Rename to hand-hold git (build/worker.cc) 2020-10-11 16:43:42 +00:00
John Ericson
d24ffe0eb1 Trim build/substitution-goal.cc 2020-10-11 16:43:24 +00:00
John Ericson
3633b3572b Rename to hand-hold git (build/substitution-goal.cc) 2020-10-11 16:43:18 +00:00
John Ericson
4bdff7d1b0 Trim build/local-store-build.cc 2020-10-11 16:43:12 +00:00
John Ericson
dc5225cde5 Rename to hand-hold git (build/local-store-build.cc) 2020-10-11 16:43:05 +00:00
John Ericson
159054f730 Trim build/hook-instance.cc 2020-10-11 16:42:35 +00:00
John Ericson
f0b8987299 Rename to hand-hold git (build/hook-instance.cc) 2020-10-11 16:42:08 +00:00
John Ericson
819fe848ac Trim build/goal.cc 2020-10-11 16:41:58 +00:00
John Ericson
184bfc301e Rename to hand-hold git (build/goal.cc) 2020-10-11 16:41:18 +00:00
John Ericson
3bab1c5bb0 Trim build/derivation-goal.cc 2020-10-11 16:41:11 +00:00
John Ericson
9629290eda Rename to hand-hold git (build/derivation-goal.cc) 2020-10-11 16:40:52 +00:00
John Ericson
a4f0fecb03 Trim build.hh 2020-10-11 16:40:34 +00:00
John Ericson
fc72cb0760 Rename to hand-hold git (build.hh) 2020-10-11 16:40:14 +00:00
John Ericson
428536fd75 Prepare for build/* files 2020-10-11 16:39:08 +00:00
John Ericson
aef44cbaa9 Split out commonChildInit 2020-10-11 16:38:46 +00:00
John Ericson
6cc1541782 Split out local-fs-store.hh 
This matches the already-existing `local-fs-store.cc`.
 2020-10-09 20:18:08 +00:00
Eelco Dolstra
725488b892 nix develop: Unset $HOSTNAME 
This is set to "localhost" by stdenv which is probably not what you
want.
 2020-10-09 22:03:18 +02:00
Eelco Dolstra
44349064f7 nix develop: Source ~/.bashrc 
Fixes #4104.
 2020-10-09 22:02:00 +02:00
Eelco Dolstra
59bd6e87a4 Completions::add(): Guard against newlines 2020-10-09 21:55:59 +02:00
Eelco Dolstra
ea4b2b985f
Merge pull request #4128 from tweag/extended-completions 
Add a zsh completion script
 2020-10-09 21:53:12 +02:00
John Ericson
39de73550d Merge remote-tracking branch 'upstream/master' into fix-and-ci-static-builds 2020-10-09 18:26:47 +00:00
Eelco Dolstra
e845d19ae3 Remove Lazy 
This fixes a crash during startup when compiling Nix as a single
compilation unit.
 2020-10-09 17:54:59 +02:00
Eelco Dolstra
87157b2bd3 writeFile(): Add error context to writeFull() failure 
Issue #4092.
 2020-10-09 16:02:53 +02:00
Eelco Dolstra
636ec17139
Remove stray DerivationOutputsAndPaths type 2020-10-09 15:41:24 +02:00
regnat
04e5d0e704 Add a description in the completion outputs 
Make nix output completions in the form `completion\tdescription`.
This can't be used by bash (afaik), but other shells like zsh or fish
can display it along the completion choices
 2020-10-09 09:39:51 +02:00
Eelco Dolstra
97ffc1e013 Dynamically disable user namespaces if CLONE_NEWUSER fails 
This makes builds work inside nixos-enter.

Fixes #3145.
 2020-10-07 22:46:01 +02:00
Eelco Dolstra
6aa64627c8 Support user namespaces being disabled 
If max_user_namespaces is set to 0, then don't run the build in a user
namespace.

Fixes #4092.
 2020-10-07 22:02:36 +02:00
Eelco Dolstra
f66bbd8c7b Doh 2020-10-07 21:25:06 +02:00
Eelco Dolstra
e705c24294 Tweak error messages 2020-10-07 17:28:43 +02:00
Eelco Dolstra
c43e882f54 Serialize exceptions from the daemon to the client 2020-10-07 17:13:54 +02:00
Eelco Dolstra
be149acfda Serialize exceptions from the sandbox process to the parent 
Fixes #4118.
 2020-10-07 16:34:03 +02:00
Eelco Dolstra
27ca87c46a Formatting 2020-10-07 16:33:19 +02:00
John Ericson
57d960dcd1 Remove generic std::optional<T> suppport from worker proto 
See comment for rational; I think it's good to leave a comment lest
anyone is tempted to add such a sum-type instance again.

Fixes #4113
 2020-10-07 12:50:37 +00:00
Maximilian Bosch
59f2dd8e8d
libfetchers/github: allow slashes in refs 
Refs #4061
 2020-10-06 20:08:51 +02:00
Eelco Dolstra
d761485010 Prevent a deadlock when user namespace setup fails 
Observed on Centos 7 when user namespaces are disabled:
DerivationGoal::startBuilder() throws an exception, ~DerivationGoal()
waits for the child process to exit, but the child process hangs
forever in drainFD(userNamespaceSync.readSide.get()) in
DerivationGoal::runChild(). Not sure why the SIGKILL doesn't get
through.

Issue #4092.
 2020-10-06 18:57:23 +02:00
Eelco Dolstra
ad143c5b3b Shut up some clang warnings 2020-10-06 14:52:30 +02:00
Eelco Dolstra
85c8be6286 Remove static variable name clashes 
This was useful for an experiment with building Nix as a single
compilation unit. It's not very useful otherwise but also doesn't
hurt...
 2020-10-06 13:49:20 +02:00
Eelco Dolstra
0419cd2695 Remove unneeded -lboost_* flags 2020-10-06 13:34:58 +02:00
Eelco Dolstra
636455c471 Remove 'using namespace fetchers' 2020-10-06 11:16:32 +02:00
Eelco Dolstra
6691256e79 Factor out common showBytes() 2020-10-06 10:40:49 +02:00
Eelco Dolstra
d0bb544128 Add missing #pragma once 2020-10-06 10:40:07 +02:00
Eelco Dolstra
88a667e49e
Fix s3:// store 
Fixes https://github.com/NixOS/nixos-org-configurations/issues/123.
 2020-10-05 17:53:30 +02:00
Eelco Dolstra
51c299213b
Merge pull request #3935 from obsidiansystems/binary-cache-addToStoreFromDump 
Get rid of Hash::dummy from BinaryCacheStore
 2020-10-05 14:41:28 +02:00
Eelco Dolstra
f3aba88737
Merge pull request #3895 from obsidiansystems/templated-daemon-protocol 
More templated STL support for the daemon protocol
 2020-10-05 14:40:27 +02:00
Fabian Möller
d5d196b0a1
Fix profile update in nix command 2020-10-02 12:10:31 +02:00
DavHau
f3280004e2 add more examples to --help of nix run 2020-10-01 11:34:13 +07:00
Eelco Dolstra
20a1e20d91 Style 2020-09-30 13:35:56 +02:00
Eelco Dolstra
274357eb6a Simplify 2020-09-30 12:09:18 +02:00
Eelco Dolstra
002ce8449d Merge branch 'access-tokens' of https://github.com/kquick/nix 2020-09-30 11:35:15 +02:00
John Ericson
69afaeace3 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-30 00:42:28 +00:00
John Ericson
b759701652 nix::worker_proto -> worker_proto 2020-09-30 00:41:18 +00:00
John Ericson
45a0ed82f0 Revert "Use template structs instead of phantoms" 
This reverts commit 9ab07e99f5.
 2020-09-30 00:39:06 +00:00
Kevin Quick
5e7838512e
Remove github-access-token in favor of access-token. 2020-09-29 16:26:34 -07:00
Eelco Dolstra
76f000712e Merge branch 'kwq/path-dir-query' of https://github.com/kquick/nix 2020-09-29 23:42:43 +02:00
Eelco Dolstra
64e9b3c83b nix registry list: Show 'dir' attribute 
Issue #4050.
 2020-09-29 23:33:16 +02:00
Kevin Quick
66c3959e8c
Merge branch 'master' into access-tokens 2020-09-29 08:32:06 -07:00
Eelco Dolstra
5999978a05 Make Headers an optional argument 2020-09-29 13:05:19 +02:00
Eelco Dolstra
de86abbf3f Cleanup 2020-09-29 12:55:06 +02:00
Eelco Dolstra
cebd2fc35d Merge branch 'github-api-token' of https://github.com/imalsogreg/nix 2020-09-29 12:17:00 +02:00
John Ericson
00135e13f4 Clarify comment a bit 2020-09-28 18:19:10 +00:00
Kevin Quick
887be7b6f2
Switch comment format from '// ...' to '/* ... */' for consistency. 2020-09-28 09:37:26 -07:00
Kevin Quick
128c98ab09
Clarification in the description of the FlakeInput. 2020-09-28 09:34:23 -07:00
Kevin Quick
5ae164b7cf
Update description of FlakeRef, incorporating suggestion. 2020-09-28 09:23:05 -07:00
Kevin Quick
bcb3da3b6b
Fix spelling error. 2020-09-28 08:58:14 -07:00
John Ericson
80e335bb58 Use drvPath2 and give it a better name 2020-09-28 15:43:56 +00:00
John Ericson
10202bbf29 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-28 15:39:11 +00:00
John Ericson
6c31297d80
Update src/libstore/binary-cache-store.cc 
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-09-28 11:32:58 -04:00
Eelco Dolstra
649c465873
Merge pull request #4064 from serokell/balsoft/fix-max-jobs 
Fix max-jobs option
 2020-09-28 10:42:02 +02:00
Eelco Dolstra
5df687c316
Merge pull request #4077 from Ma27/left-whitespace-log 
libmain/progress-bar: don't trim whitespace on the left
 2020-09-28 10:40:00 +02:00
Kevin Quick
5885b0cfd8
Miscellaneous spelling fixes in comments. (#4071) 2020-09-27 20:04:06 +00:00
Kevin Quick
bd5328814f
Add some internal documentation for flake support objects. 2020-09-26 14:32:58 -07:00
Maximilian Bosch
a76fb07314
libmain/progress-bar: don't trim whitespace on the left 
When running `nix build -L` it can be fairly hard to read the output if
the build program intentionally renders whitespace on the left. A
typical example is `g++` displaying compilation errors.

With this patch, the whitespace on the left is retained to make the log
more readable:

```
foo> no configure script, doing nothing
foo> building
foo> foobar.cc: In function 'int main()':
foo> foobar.cc:5:5: error: 'wrong_func' was not declared in this scope
foo>     5 |     wrong_func(1);
foo>       |     ^~~~~~~~~~
error: --- Error ------------------------------------------------------------------------------------- nix
error: --- Error --- nix-daemon
builder for '/nix/store/i1q76cw6cyh91raaqg5p5isd1l2x6rx2-foo-1.0.drv' failed with exit code 1
```
 2020-09-26 17:38:11 +02:00
John Ericson
25fffdda86
Remove redundant nar hash and size setting 
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
 2020-09-26 10:17:30 -04:00
John Ericson
1832436526 Fix up BinaryCacheStore::addToStore taking a path 2020-09-26 04:56:29 +00:00
John Ericson
5db83dd771 BinaryCacheStore::addTextToStore include CA field 2020-09-26 03:21:36 +00:00
John Ericson
c40c832f19 Merge remote-tracking branch 'obsidian/master' into binary-cache-addToStoreFromDump 2020-09-26 00:21:45 +00:00
Kevin Quick
cb186f1e75
Use "?dir=..." portion of "registry add" local path specification. 
The registry targets generally follow a URL formatting schema with
support for a query parameter of "?dir=subpath" to specify a sub-path
location below the URL root.

Alternatively, an absolute path can be specified.  This specification
mode accepts the query parameter but ignores/drops it.  It would
probably be better to either (a) disallow the query parameter for the
path form, or (b) recognize the query parameter and add to the path.

This patch implements (b) for consistency, and to make it easier for
tooling that might switch between a remote git reference and a local
path reference.

See also issue #4050.
 2020-09-25 09:36:18 -07:00
Gregory Hale
faa5607f54 Merge remote-tracking branch 'origin/master' into github-api-token 2020-09-25 12:10:58 -04:00
John Ericson
cfe791a638 stdout_ -> cout 
Better to get creative than just sprinkle arbitrary underscores.
 2020-09-25 11:30:04 -04:00
Eelco Dolstra
7d81582488
Merge pull request #3953 from obsidiansystems/basic-derivation-goal-outputs 
Deduplicate basic derivation goals too
 2020-09-25 17:21:19 +02:00
Kevin Quick
5a35cc29bf
Re-add support for github-access-token, but mark as deprecated. 2020-09-25 08:09:56 -07:00
Kevin Quick
ef2a14be19
Fix reference to older name for access-tokens config value. 2020-09-25 08:08:27 -07:00
Eelco Dolstra
cbb9862cd9
Merge pull request #3626 from W95Psp/master 
Make `functionArgs` primitive accept primops (fix #3624)
 2020-09-25 15:14:18 +02:00
Eelco Dolstra
7b2ae472ff
expectArg(): Respect the 'optional' flag 2020-09-25 10:27:40 +02:00
Kevin Quick
8fba2a8b54
Update to use access-tokens configuration for github/gitlab access. 
This change provides support for using access tokens with other
instances of GitHub and GitLab beyond just github.com and
gitlab.com (especially company-specific or foundation-specific
instances).

This change also provides the ability to specify the type of access
token being used, where different types may have different handling,
based on the forge type.
 2020-09-24 22:49:44 -07:00
Kevin Quick
c2f48cfcee
Complete conversion of "url" to "host" with associated variable renaming. 
Completes the change begun in commit 56f1e0d to consistently use the
"host" attribute for "github" and "gitlab" inputs instead of a "url"
attribute.
 2020-09-24 22:46:03 -07:00
Kevin Quick
a439e9488d
Support StringMap configuration settings. 
Allows Configuration values that are space-separated key=value pairs.
 2020-09-24 22:42:59 -07:00
Paul Opiyo
4d863a9fcb Remove redundant value checks 
std::optional had redundant checks for whether it had a value.
An object is emplaced either way so it can be dereferenced
without repeating a value check
 2020-09-24 18:32:03 -05:00
Alexander Bantyev
ed218e1d6c
Fix max-jobs option 
After 0ed946aa61, max-jobs setting (-j/--max-jobs)
stopped working.

The reason was that nrLocalBuilds (which compared to maxBuildJobs to figure
out whether the limit is reached or not) is not incremented yet when tryBuild
is started; So, the solution is to move the check to tryLocalBuild.

Closes https://github.com/nixos/nix/issues/3763
 2020-09-25 00:07:42 +03:00
Kevin Quick
bd5f3dbe11
Fixes fall-through to report correct description of hash-file command. 2020-09-24 12:30:03 -07:00
Eelco Dolstra
8d9402f411
Merge pull request #4054 from edolstra/fix-4021 
registerOutputs(): Don't call canonicalisePathMetaData() twice
 2020-09-23 21:57:53 +02:00
Eelco Dolstra
4ce8a3ed45 Hopefully fix EPERM on macOS 2020-09-23 21:29:10 +02:00
Eelco Dolstra
9a24ece122 Fix exception 2020-09-23 20:21:08 +02:00
Eelco Dolstra
236d9ee7f7 lstat() cleanup 2020-09-23 19:17:28 +02:00
Eelco Dolstra
688bd4fb50 After rewriting a path, make it read-only 2020-09-23 19:10:16 +02:00
Dominique Martinet
2548347bba libutil/archive: add preallocate-contents option 
Make archive preallocation (fallocate) optional because some filesystems
like btrfs do not behave as expected with fallocate.

See #3550.
 2020-09-23 18:49:11 +02:00
Eelco Dolstra
cec9473871 DerivationGoal::registerOutputs(): Don't canonicalize twice 
Fixes #4021.
 2020-09-23 18:21:22 +02:00
Eelco Dolstra
d4f8163d10 canonicalisePathMetaData_(): Change assertion to error message 2020-09-23 18:21:22 +02:00
Eelco Dolstra
e8f0b1e996 DerivationGoal::registerOutputs(): Fix bad format string 2020-09-23 18:21:22 +02:00
John Ericson
3f226f71c1 Return more info from BinaryCacheStore::addToStoreCommon 
We don't need it yet, but we could/should in the future, and it's a
cost-free change since we already have the reference. I like it.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
 2020-09-23 14:40:41 +00:00
John Ericson
412b3a54fb
Clarify FIXME in BinaryCacheStore::addToStoreCommon 
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
 2020-09-23 10:36:55 -04:00
Eelco Dolstra
8a2e10827f Remove unused Flake::vOutputs field 2020-09-23 14:08:52 +02:00
Domen Kožar
dac8a1a5e1
Merge pull request #4052 from ujjwaljainnn/fix-typo 
fixed typo
 2020-09-23 09:58:30 +02:00
John Ericson
9fbc31a65b Get rid of Hash::dummy from BinaryCacheStore 2020-09-23 04:56:04 +00:00
ujjwal
a2f5c921d4 fixed typo 2020-09-22 23:37:06 +05:30
John Ericson
993229cdaf Deduplicate basic derivation goals too 
See comments for security concerns.

Also optimize goal creation by not traversing map twice.
 2020-09-22 17:13:59 +00:00
John Ericson
e9fc2031f0 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-22 14:18:31 +00:00
regnat
67cc356bb3 Merge remote-tracking branch 'origin/master' into non-ca-depending-on-ca 2020-09-22 16:11:25 +02:00
Eelco Dolstra
980edd1f3a RemoteStore::addCAToStore(): Don't hold connection while calling queryPathInfo() 
This leads to a deadlock if we're at the connection limit.
 2020-09-22 15:28:20 +02:00
Eelco Dolstra
5b107f2c5f
Merge pull request #4038 from maljub01/master 
Add a nix.conf option for allowing a symlinked store
 2020-09-22 13:19:22 +02:00
Eelco Dolstra
35a0ac1838 Style fixes 2020-09-22 11:40:19 +02:00
Eelco Dolstra
92ac8df0ec Merge branch 'add-ca-to-store' of https://github.com/hercules-ci/nix 2020-09-22 11:31:33 +02:00
Eelco Dolstra
e7f1109f06
Merge pull request #4040 from OmnipotentEntity/master 
Fix compatibility with nlohmann-json 3.9.1
 2020-09-22 11:09:25 +02:00
Eelco Dolstra
7dd8baafe1
Merge pull request #4041 from cole-h/enum-stringify 
Serialize SandboxMode enum to string for JSON
 2020-09-22 11:08:56 +02:00
regnat
c1e79f870c Silence a compiler warning in serialise.hh 
Explicitely cast to `uint64_t` in `readNum` to avoid a "comparison
between signed and unsigned" warning
 2020-09-22 10:39:29 +02:00
John Ericson
3786a801c3 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-22 04:15:55 +00:00
John Ericson
b92d3b2edd Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-22 00:45:55 +00:00
Michael Reilly
d860295e11 Bump nlohmann-json version to 3.9.1 2020-09-21 14:24:43 -04:00
Cole Helbling
ba37299a03
Serialize SandboxMode enum to string for JSON 
Rather than showing an integer as the default, instead show the boolean
referenced in the description.

The nix.conf.5 manpage used to show "default: 0", which is unnecessarily
opaque and confusing (doesn't 0 mean false, even though the default is
true?); now it properly shows that the default is true.
 2020-09-21 10:36:45 -07:00
Eelco Dolstra
ecc8672aa0 fmt.hh: Don't include boost/algorithm/string/replace.hpp 
This cuts compilation time by ~49s.

Issue #4045.
 2020-09-21 19:07:55 +02:00
Eelco Dolstra
557d2427ee Random header cleanup 2020-09-21 18:59:02 +02:00
Eelco Dolstra
0716adaa8b abstractsettingtojson.hh -> abstract-setting-to-json.hh 2020-09-21 18:49:43 +02:00
Eelco Dolstra
340ca382c4 Don't include nlohmann/json.hpp in globals.hh 
This reduces compilation time by 207s.

Issue #4045.
 2020-09-21 18:47:18 +02:00
Eelco Dolstra
d51ba43047 Move Callback into its own header 
This gets rid of the inclusion of <future> in util.hh, cutting
compilation time by ~20s (CPU time).

Issue #4045.
 2020-09-21 18:42:21 +02:00
Marwan Aljubeh
f80ffeb8c9 Update the variable name accordingly 2020-09-21 17:29:08 +01:00
Eelco Dolstra
e8e1d420f3 Don't include <regex> in header files 
This reduces compilation time by ~15 seconds (CPU time).

Issue #4045.
 2020-09-21 18:22:45 +02:00
Marwan Aljubeh
4e1a04733d Use a better name for the config option 2020-09-21 16:32:22 +01:00
Eelco Dolstra
cbe0bb29f4
Merge pull request #4035 from Ma27/url-attr 
libfetchers/github: allow `url` attribute
 2020-09-21 17:26:36 +02:00
Maximilian Bosch
56f1e0df05
libfetchers/github: rename url to host 2020-09-21 16:29:08 +02:00
Robert Hensing
ca30abb3fb Document addCAToStore/addToStoreFromDump source drainage 
Also checked that all usages satisfy the requirement and
removed dead code.
 2020-09-21 07:55:47 +02:00
Robert Hensing
fa08db5c4c wopAddToStore: return ValidPathInfo 
A ValidPathInfo is created anyway. By returning it we can save a
roundtrip and we have a nicer interface.
 2020-09-21 07:55:47 +02:00
Robert Hensing
7c68264085 wopAddToStore: add RepairFlag 2020-09-21 07:55:47 +02:00
Robert Hensing
fbf509c113 parseContentAddressMethodPrefix: use string_view 
Co-authored-by: John Ericson <git@JohnEricson.me>
 2020-09-21 07:55:47 +02:00
Robert Hensing
8279178b07 Move FramedSink next to FramedSource 2020-09-21 07:55:47 +02:00
Robert Hensing
ecc8088cb7 wopAddToStore: Throw to clarify unused refs 
Co-authored-by: John Ericson <git@JohnEricson.me>
 2020-09-21 07:55:47 +02:00
Robert Hensing
c602ebfb34 Refactor wopAddToStore to make wopAddTextToStore obsolete 2020-09-21 07:55:45 +02:00
Robert Hensing
e34fe47d0c Overhaul wopAddToStore 2020-09-21 07:54:05 +02:00
Marwan Aljubeh
e40772cd35 Lint issue: replacing tabs with spaces 2020-09-18 17:18:45 +01:00
Marwan Aljubeh
c00e078343 Add a nix.conf option for allowing a symlinked store 2020-09-18 17:10:39 +01:00
Bryan Richter
5fe375a8f1
nix-prefetch-url: Add --executable flag 
pkgs.fetchurl supports an executable argument, which is especially nice
when downloading a large executable. This patch adds the same option to
nix-prefetch-url.

I have tested this to work on the simple case of prefetching a little
executable:

1. nix-prefetch-url --executable https://my/little/script
2. Paste the hash into a pkgs.fetchurl-based package, script-pkg.nix
3. Delete the output from the store to avoid any misidentified artifacts
4. Realise the package script-pkg.nix
5. Run the executable

I repeated the above while using --name, as well.

I suspect --executable would have no meaningful effect if combined with
--unpack, but I have not tried it.
 2020-09-18 19:09:45 +03:00
Maximilian Bosch
2bcf8cbe7a
libfetchers/github: allow url attribute 
Since 108debef6f we allow a
`url`-attribute for the `github`-fetcher to fetch tarballs from
self-hosted `gitlab`/`github` instances.

However it's not used when defining e.g. a flake-input

    foobar = {
        type = "github";
        url = "gitlab.myserver";
        /* ... */
    }

and breaks with an evaluation-error:

    error: --- Error --------------------------------------nix
    unsupported input attribute 'url'
    (use '--show-trace' to show detailed location information)

This patch allows flake-inputs to be fetched from self-hosted instances
as well.
 2020-09-18 14:10:45 +02:00
Eelco Dolstra
958bf57123 nix build: find() -> get() 
find() returns an iterator so "!attr" doesn't work.
 2020-09-18 13:10:42 +02:00
Robert Hensing
14b30b3f3d Move FramedSource and FramedSink, extract withFramedSink 2020-09-17 20:21:04 +02:00
Robert Hensing
dfa547c6a8 Add ContentAddressMethod and parse/render it 2020-09-17 20:21:04 +02:00
Robert Hensing
29c82ccc77 Add Source.drainInto(Sink) 2020-09-17 20:21:04 +02:00
Robert Hensing
9ee3122ec7 Remove redundant import 2020-09-17 20:21:04 +02:00
Robert Hensing
3f93bc0d39 Typo 2020-09-17 20:21:04 +02:00
John Ericson
b7df353f27 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-17 16:33:10 +00:00
Eelco Dolstra
649d3aaf24
Merge pull request #3829 from obsidiansystems/remove-storetype-delegate-regStore 
Remove storetype delegate reg store -- contains #3736
 2020-09-17 13:55:01 +02:00
Eelco Dolstra
b94a35ef40
Merge pull request #4027 from tweag/fix-gc-of-ca-derivations 
Fix garbage collection of CA derivations
 2020-09-17 13:46:26 +02:00
regnat
520895b1da Fix garbage collection of CA derivations 
Fix #4026
 2020-09-17 13:36:58 +02:00
Eelco Dolstra
787469c7b6 Remove corepkgs/unpack-channel.nix 2020-09-17 09:41:02 +02:00
Eelco Dolstra
10d1865f5f Remove corepkgs/derivation.nix 2020-09-17 09:41:02 +02:00
John Ericson
f60b380a7f Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore 2020-09-16 22:35:24 +00:00
John Ericson
2741fffa35 Ensure resolved CA derivations are written 
so we can link outputs to deriver and thus properly cache.
 2020-09-16 17:57:32 +00:00
John Ericson
7fdbb377ba Start to fix floating CA + remote building 2020-09-16 17:50:48 +00:00
John Ericson
c5ccebae00 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-16 17:50:40 +00:00
Greg Hale
a303c0b6dc Fetch commits from github/gitlab using Auth header 
`nix flake info` calls the github 'commits' API, which requires
authorization when the repository is private. Currently this request
fails with a 404.

This commit adds an authorization header when calling the 'commits' API.
It also changes the way that the 'tarball' API authenticates, moving the
user's token from a query parameter into the Authorization header.

The query parameter method is recently deprecated and will be disallowed
in November 2020. Using them today triggers a warning email.
 2020-09-16 13:46:48 -04:00
Eelco Dolstra
5080d4e7b2 Merge branch 'document-store-options' of https://github.com/tweag/nix 2020-09-16 17:02:30 +02:00
Eelco Dolstra
0066ef6c59 Fix doc generation 2020-09-16 16:56:28 +02:00
Eelco Dolstra
2eacc1bc00 builtins.toFile: Fix indentation 2020-09-16 14:18:46 +02:00
Théophane Hufschmitt
77a0e2c5be
Remove useless exception copy 
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-09-16 14:00:21 +02:00
regnat
d72927aa7a Fix the s3 store 
Add some necessary casts in the initialisation of the store's config
 2020-09-16 13:53:09 +02:00
regnat
e0817cbcdc Don't include nlohmann/json.hpp in config.hh 
Instead make a separate header with the template implementation of
`BaseSetting<T>::toJSONObj` that can be included where needed
 2020-09-16 13:53:09 +02:00
regnat
93c0e14a30 Include the full nlohmann/json header in config.hh 
It is apparently required for using `toJSONObject()`, which we do inside
the header file (because it's in a template).

This was accidentally working when building Nix itself (presumably because
`config.hh` was always included after `nlohman/json.hpp`) but caused a
(pretty dirty) build failure in the perl bindings package.
 2020-09-16 13:53:09 +02:00
regnat
fc2d31c423 Add (StoreConfig*) casts to work around a GCC bug 
Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80431 that was
already there in the code but was accidentally removed in the last
commits
 2020-09-16 13:53:09 +02:00
regnat
a1e82ba450 fixup! Add a default value for the settings 2020-09-16 13:53:09 +02:00
regnat
888f7afe9f Fix build issues with gcc 2020-09-16 13:53:09 +02:00
regnat
634cb2a5ae Add a markdown output to nix describe-stores 2020-09-16 13:53:09 +02:00
regnat
b73adacc1e Add a name to the stores 
So that it can be printed by `nix describe-stores`
 2020-09-16 13:53:09 +02:00
regnat
f24f0888f9 Document the new store hierarchy 2020-09-16 13:53:09 +02:00
regnat
d65962db4d Make uri schemes grammar more RFC-compliant 
Allow `-` and `.` in the RFC schemes as stated by
[RFC3986](https://tools.ietf.org/html/rfc3986#section-3.1).

Practically, this is needed so that `ssh-ng` is a valid URI scheme
 2020-09-16 13:53:09 +02:00
regnat
29a632386e fixup! Make the store plugins more introspectable 2020-09-16 13:53:09 +02:00
regnat
1129913c4e fixup! Correctly call all the parent contructors of the stores 2020-09-16 13:53:09 +02:00
regnat
7f103dcddd Properly filter the stores according to their declared uriSchemes 
When opening a store, only try the stores whose `uriSchemes()` include
the current one
 2020-09-16 13:53:09 +02:00
regnat
5895184df4 Correctly call all the parent contructors of the stores 
Using virtual inheritance means that only the default constructors of
the parent classes will be called, which isn't what we want
 2020-09-16 13:53:09 +02:00
regnat
dae39f0a7a Make nix describe-stores functional 
Using the `*Config` class hierarchy
 2020-09-16 13:53:08 +02:00
regnat
22afa8fb4d Separate store configs from the implems 
Rework the `Store` hierarchy so that there's now one hierarchy for the
store configs and one for the implementations (where each implementation
extends the corresponding config). So a class hierarchy like

```
StoreConfig-------->Store
    |                 |
    v                 v
SubStoreConfig----->SubStore
    |                 |
    v                 v
SubSubStoreConfig-->SubSubStore
```

(with virtual inheritance to prevent DDD).

The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
 2020-09-16 13:53:08 +02:00
regnat
aa4eac3788 fixup! Separate the instantiation and initialisation of the stores 2020-09-16 13:53:08 +02:00
regnat
35042c9623 Add a default value for the settings 
The default value is initialized when creating the setting and unchanged
after that
 2020-09-16 13:53:08 +02:00
regnat
3c525d1590 Complete the toJSON instance for Setting<T> 
Don't let it just contain the value, but also the other fields of the
setting (description, aliases, etc..)
 2020-09-16 13:53:08 +02:00
regnat
3b57181f8e Separate the instantiation and initialisation of the stores 
Add a new `init()` method to the `Store` class that is supposed to
handle all the effectful initialisation needed to set-up the store.
The constructor should remain side-effect free and just initialize the
c++ data structure.

The goal behind that is that we can create “dummy” instances of each
store to query static properties about it (the parameters it accepts for
example)
 2020-09-16 13:53:08 +02:00
regnat
fa32560169 Fix the registration of stores 2020-09-16 13:53:08 +02:00
regnat
7d5bdf8b56 Make the store plugins more introspectable 
Directly register the store classes rather than a function to build an
instance of them.
This gives the possibility to introspect static members of the class or
choose different ways of instantiating them.
 2020-09-16 13:53:08 +02:00
John Ericson
3a5cdd737c Rename Derivation::pathOpt to Derivation::path 
We no longer need the `*Opt` to disambiguate.
 2020-09-15 15:21:39 +00:00
John Ericson
6387550d58 Get rid of confusing std::optional<bool> for validity 2020-09-15 15:19:45 +00:00
John Ericson
c4bf219b55 Don't link deriver until after any delayed exception is thrown 
Otherwise, we will associate fixed-output derivations with outputs that
they did indeed produce, but which had the wrong hash. That's no good.
 2020-09-15 14:28:06 +00:00
John Ericson
3ba552b245 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-15 14:17:06 +00:00
John Ericson
c08c9f08c7 Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore 2020-09-15 14:08:35 +00:00
Eelco Dolstra
2a80170920
Merge pull request #4014 from tweag/fix-queryDrvOutputMaps-for-old-daemons 
gracefully handle old daemon versions
 2020-09-15 11:25:19 +02:00
regnat
057c6203b5 gracefully handle old daemon versions 
Add a fallback path in `queryPartialDerivationOutputMap` for daemons
that don't support it.

Also upstreams a couple methods from `SSHStore` to `RemoteStore` as this
is needed to handle the fallback path.
 2020-09-15 09:58:01 +02:00
regnat
250f8a4bba Escape ${ in strings when printing Nix expressions 
Otherwise the result of the printing can't be parsed back correctly by
Nix (because the unescaped `${` will be parsed as the begining of an
anti-quotation).

Fix #3989
 2020-09-14 17:19:25 +02:00
Eelco Dolstra
a59e77d9e5 nix-daemon: Lower verbosity of restricted setting warning 
Fixes #3992.
 2020-09-14 13:48:51 +02:00
Eelco Dolstra
35ba092019
Merge pull request #4005 from Infinisil/fix-autoArgs 
Fix auto argument passing for more auto arguments than formals
 2020-09-14 13:31:23 +02:00
aszlig
525b38eee8
Fix unspecified behaviour in readStorePathCAMap 
When deploying a Hydra instance with current Nix master, most builds
would not run because of errors like this:

  queue monitor: error: --- Error --- hydra-queue-runner
  error: --- UsageError --- nix-daemon
  not a content address because it is not in the form '<prefix>:<rest>': /nix/store/...-somedrv

The last error message is from parseContentAddress, which expects a
colon-separated string, however what we got here is a store path.

Looking at the worker protocol, the following message sent to the Nix
daemon caused the error above:

  0x1E -> wopQuerySubstitutablePathInfos
  0x01 -> Number of paths
  0x16 -> Length of string
  "/nix/store/...-somedrv"
  0x00 -> Length of string
  ""

Looking at writeStorePathCAMap, the store path is indeed the first field
that's transmitted. However, readStorePathCAMap expects it to be the
*second* field *on my machine*, since expression evaluation order is a
classic form of unspecified behaviour[1] in C++.

This has been introduced in https://github.com/NixOS/nix/pull/3689,
specifically in commit 66a62b3189.

[1]: https://en.wikipedia.org/wiki/Unspecified_behavior#Order_of_evaluation_of_subexpressions

Signed-off-by: aszlig <aszlig@nix.build>
 2020-09-13 02:40:51 +02:00
Silvan Mosberger
8dbd57a6a5
Fix auto argument passing for more auto arguments than formals 
The change in 626200713b didn't account
for when the number of auto arguments is bigger than the number of
formal arguments. This causes the following:

  $ nix-instantiate --eval -E '{ ... }@args: args.foo' --argstr foo foo
  nix-instantiate: src/libexpr/attr-set.hh:55: void nix::Bindings::push_back(const nix::Attr&): Assertion `size_ < capacity_' failed.
  Aborted (core dumped)
 2020-09-11 20:41:51 +02:00
Gabriel Gonzalez
ee5906243a Add nix-shell support for preserving PS1 
Fixes https://github.com/NixOS/nix/issues/1268

`nix-shell` will now preserve `PS1` if the `NIX_SHELL_PRESERVE_PROMPT`
environment variable is set.
 2020-09-04 20:05:43 -07:00
John Ericson
98dfd7531d Fix querying outputs for CA derivations some more 
If we resolve using the known path of a derivation whose output we
didn't have, we previously blew up. Now we just fail gracefully,
returning the map of all outputs unknown.
 2020-09-04 18:33:58 +00:00
John Ericson
075d399e3f Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-09-04 16:04:35 +00:00
John Ericson
5aed6f9b25 Document mkOutputString 2020-09-04 15:58:42 +00:00
John Ericson
e9fad3006b Fix some of the issues raised by @edolstra 
- More and better comments

 - The easier renames
 2020-09-04 15:18:25 +00:00
John Ericson
e86dd59dcc
Apply suggestions from code review 
Thanks!

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-09-04 10:48:50 -04:00
John Ericson
c9f1ed912c Don't chmod symlink before moving outputs around 
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
 2020-09-04 14:41:53 +00:00
John Ericson
25f7ff16fa Merge remote-tracking branch 'upstream/master' into fix-and-ci-static-builds 2020-09-04 02:40:36 +00:00
John Ericson
aad4abcc9c Fix floating CA tests 
We will sometimes try to query the outputs of derivations we can't
resolve. That's fine; it just means we don't know what those outputs are
yet.
 2020-09-04 01:17:38 +00:00
John Ericson
975a47f7fe Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-09-03 22:09:04 +00:00
John Ericson
e7d93e7ece Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-03 15:43:17 +00:00
Eelco Dolstra
8a945d6ddb
Remove gc-check-reachability 2020-09-03 11:30:15 +02:00
Eelco Dolstra
82b77a7726
addPermRoot(): Remove indirect flag 2020-09-03 11:29:34 +02:00
Eelco Dolstra
00d25e8457
Remove the --indirect flag 
All GC roots are now indirect.
 2020-09-03 11:22:00 +02:00
Eelco Dolstra
b07167be5a
createGeneration(): Always create an indirect root 
This means profiles outside of /nix/var/nix/profiles don't get
garbage-collected. It also means we don't need to scan
/nix/var/nix/profiles for GC roots anymore, except for compatibility
with previously existing generations.
 2020-09-03 11:13:53 +02:00
Eelco Dolstra
b74f5cdd23
createGeneration(): Take a StorePath 2020-09-03 11:06:56 +02:00
Eelco Dolstra
94a043ff3b
EvalCache: Fix caching of strings 
This was broken in 50f13b06fb. Once
again it turns out that putting a bool in a std::variant is a bad
idea, since pointers get silently cast to them...
 2020-09-02 14:16:44 +02:00
Domen Kožar
dd4b56c87f
Allow HTTP binary cache to request absolute uris 2020-09-01 21:35:48 +02:00
John Ericson
ef278d00f9 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-01 18:01:48 +00:00
Eelco Dolstra
6d7f7efb89
github: Use access token when calling .../commits API 2020-09-01 15:29:04 +02:00
Eelco Dolstra
84f5cabbea Merge remote-tracking branch 'origin/master' into markdown 2020-08-31 14:24:26 +02:00
John Ericson
02e0001fc0 Merge branch 'ca-floating-upstream' of github.com:obsidiansystems/nix into ca-floating-upstream 2020-08-28 21:23:09 +00:00
John Ericson
421ed527c7
Update src/libstore/build.cc 
Thanks for catching, @regnat.
 2020-08-28 17:22:57 -04:00
John Ericson
48e8828050 Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-08-28 20:38:25 +00:00
John Ericson
8017fe7487 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-28 19:59:14 +00:00
Eelco Dolstra
f15651303f nix develop: Add convenience flags for running specific phases 
For example, for building the Nix flake, you would do:

  $ nix develop --configure
  $ nix develop --install
  $ nix develop --installcheck
 2020-08-28 19:24:29 +02:00
Eelco Dolstra
3156560d41 nix develop: Set output paths to writable locations 
Currently, they're set to $(pwd)/outputs/$outputName. This allows
commands like 'make install' to work.
 2020-08-28 18:16:03 +02:00
Eelco Dolstra
691a1bd717 Merge branch 'minimal-logger' of https://github.com/Ma27/nix 2020-08-28 10:50:52 +02:00
Griffin Smith
626200713b Pass all args when auto-calling a function with an ellipsis 
The command line options --arg and --argstr that are used by a bunch of
CLI commands to pass arguments to top-level functions in files go
through the same code-path as auto-calling top-level functions with
their default arguments - this, however, was only passing the arguments
that were *explicitly* mentioned in the formals of the function - in the
case of an as-pattern with an ellipsis (eg args @ { ... }) extra passed
arguments would get omitted. This fixes that to instead pass *all*
specified auto args in the case that our function has an ellipsis.

Fixes #598
 2020-08-27 12:38:25 -04:00
Eelco Dolstra
eb75282b8d
Merge pull request #3434 from Ericson2314/derivation-header-include-order 
Revise division of labor in deserialization of derivations
 2020-08-27 16:39:28 +02:00
Eelco Dolstra
e915fd6d2a Typo 2020-08-27 14:51:50 +02:00
Eelco Dolstra
a0f19d9f3a RemoteStore::addToStore(): Fix race between stderrThread and NAR writer 
As pointed out by @B4dM4n, the call to to.flush() on stderrThread is
unsafe because the NAR writer thread is also writing to 'to'.

Fixes #3943.
 2020-08-27 14:50:51 +02:00
Eelco Dolstra
c3efef9275 Remove obsolete comment 2020-08-26 09:28:10 +02:00
Eelco Dolstra
4bf5faf416 Merge remote-tracking branch 'origin/master' into markdown 2020-08-25 19:47:34 +02:00
Eelco Dolstra
7a02865b94
Move import docs 2020-08-25 14:06:01 +02:00
Eelco Dolstra
f53b5f1058
Add getDoc() function 2020-08-25 13:31:11 +02:00
Eelco Dolstra
2a2121d264
Use RegisterPrimOp for some undocumented primops 2020-08-25 11:25:01 +02:00
Eelco Dolstra
b8416779e3
Document some primops 2020-08-25 11:16:45 +02:00
Eelco Dolstra
24b1c2c66b
Fix tests 2020-08-25 10:51:14 +02:00
Eelco Dolstra
b42789f013
Fix clang build 2020-08-24 21:13:39 +02:00
John Ericson
8eb73a8724 CA derivations that depend on other CA derivations 
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
 2020-08-24 19:01:15 +00:00
John Ericson
e0b0e18905 Add constructor for BasicDerivation -> Derivation 2020-08-24 19:01:07 +00:00
John Ericson
59979e7053 Fix bad debug format string 2020-08-24 19:01:07 +00:00
Eelco Dolstra
d9a8619762
Don't barf if corepkgs is in the store but not a valid path 
This can happen when using a dummy store (or indeed any non-local store).
 2020-08-24 19:15:17 +02:00
Eelco Dolstra
6a67e57019
Add DummyStore (dummy://) 
DummyStore does not allow building or adding paths. This is useful for
evaluation tests when you don't want to initialize a "proper" store.
 2020-08-24 18:54:16 +02:00
Eelco Dolstra
d0690bc311
nix repl ':doc': Render using lowdown 2020-08-24 18:10:33 +02:00
Eelco Dolstra
0f314f3c25
Generate builtins section of the manual 2020-08-24 14:49:30 +02:00
Eelco Dolstra
a990f063ff
Move primop docs inline 
This makes them available to 'nix repl'.
 2020-08-24 14:31:10 +02:00
Eelco Dolstra
33b1679d75
Allow primops to have Markdown documentation 2020-08-24 13:16:02 +02:00
Eelco Dolstra
88d5c9ec58
Fix tests 2020-08-24 10:37:10 +02:00
Eelco Dolstra
2530cc6827
Merge pull request #3950 from obsidiansystems/tabs-spaces 
tabs -> spaces
 2020-08-24 10:20:28 +02:00
John Ericson
b0b59fd05a Merge remote-tracking branch 'obsidian/write-derivation-borrow' into HEAD 2020-08-23 15:11:10 +00:00
John Ericson
35e6288be1 writeDerivation just needs a plain store reference 2020-08-23 15:01:11 +00:00
John Ericson
3a7b330b64 "Downstream placeholders" should not be store paths 
Insead they should be opaque `/<hash>` like the placeholders we already
have.
 2020-08-21 19:35:35 +00:00
John Ericson
422affe102 tabs -> spaces 
Sorry I let the tab sneak in there in the first place.
 2020-08-21 19:19:24 +00:00
John Ericson
c265e0ea82 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-08-20 20:24:22 +00:00
John Ericson
27a3f82c0b Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-20 18:28:17 +00:00
John Ericson
45a2f1baab Rename drv output querying functions, like master 
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
 - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap
 2020-08-20 18:14:12 +00:00
Eelco Dolstra
4d77513d97
Merge pull request #3859 from obsidiansystems/drv-outputs-map-allow-missing 
`queryDerivationOutputMap` no longer assumes all outputs have a mapping
 2020-08-20 16:49:23 +02:00
John Ericson
9a9d834dc7 Rename drv output querying functions 
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
 - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap`
 2020-08-20 14:12:51 +00:00
Eelco Dolstra
25ecfffdc3
Remove PrimOp constructor 2020-08-20 12:34:04 +02:00
Eelco Dolstra
dc2f278c95
Allow 'nix' subcommands to provide docs in Markdown format 2020-08-20 12:21:46 +02:00
Eelco Dolstra
3c4f8c9175
List deprecated option aliases in the docs 2020-08-20 11:13:17 +02:00
Eelco Dolstra
acb99f03f9
Config: Use nlohmann/json 2020-08-20 11:02:16 +02:00
John Ericson
3df78858f2 Fix max fd calc and add test 2020-08-20 05:08:50 +00:00
Ryan Mulligan
f36793c7b9
fix spelling 2020-08-19 20:31:01 -07:00
John Ericson
a83694c7a1 Use RemoteStore to open connection for proxying daemon 
Removes duplicate websocket opening code, and also means we should be
able to to ssh-ssh-... daemon relays, not just uds-uds-... ones.
 2020-08-19 19:34:47 +00:00
Eelco Dolstra
c8fa39324a
Generate the nix.conf docs from the source code 
This means we don't have two (divergent) sets of option descriptions
anymore.
 2020-08-19 18:28:04 +02:00
John Ericson
d5af5763cf Merge branch 'master' of github.com:NixOS/nix into remove-storetype-delegate-regStore 2020-08-19 15:54:50 +00:00
Eelco Dolstra
34b22e0123
Change option descriptions to Markdown 2020-08-19 14:21:27 +02:00
John Ericson
be0d429b95 Merge branch 'master' of github.com:NixOS/nix into templated-daemon-protocol 2020-08-19 03:17:41 +00:00
John Ericson
950ddfdb82 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-08-18 14:36:44 +00:00
Eelco Dolstra
1c8b550e34
Merge pull request #3917 from obsidiansystems/output-env-var-unconditional 
Simplify code as output env vars are unconditional
 2020-08-18 16:21:17 +02:00
Eelco Dolstra
dfeb76dbf9
Merge pull request #3930 from obsidiansystems/legacy-ssh-build-paths 
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths`
 2020-08-18 16:07:40 +02:00
Eelco Dolstra
069340179e
Improve nix.1 manpage generator 2020-08-18 15:15:35 +02:00
Eelco Dolstra
6f19c776db
Start generation of the nix.1 manpage 2020-08-17 19:33:18 +02:00
Eelco Dolstra
a72a20d68f
Add 'nix dump-args' to dump all commands/flags for manpage generation 2020-08-17 17:44:52 +02:00
Eelco Dolstra
7cdc739ece
Merge remote-tracking branch 'origin/master' into markdown 2020-08-17 13:43:39 +02:00
Eelco Dolstra
e849b19872
Merge pull request #3932 from chkno/no-show-signature 
Don't try to parse signature check as commit timestamp
 2020-08-17 11:00:24 +02:00
Eelco Dolstra
847a5392f4 Merge branch 'ca-no-need-trust' of https://github.com/obsidiansystems/nix 2020-08-17 10:57:02 +02:00
Eelco Dolstra
3c619f6290 Merge branch 'test-RemoteStore-buildDerivation' of https://github.com/obsidiansystems/nix 2020-08-17 10:53:18 +02:00
John Ericson
6f7ac5e865 Remove extra closing paren 2020-08-14 21:59:31 +00:00
John Ericson
f899a7c6d7 Work around clang bug 2020-08-14 18:51:31 +00:00
John Ericson
3c8b5b6219 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-14 17:00:13 +00:00
John Ericson
4b571ea321
Update src/libstore/daemon.cc 
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-08-14 11:52:37 -04:00
Eelco Dolstra
13e49be660
Merge pull request #3875 from obsidiansystems/new-interface-for-path-pathOpt 
Offer a safer interface for path and pathOpt
 2020-08-14 17:19:19 +02:00
Eelco Dolstra
7714d9a943
Merge pull request #3924 from obsidiansystems/features-per-store 
Make `system-features` a store setting
 2020-08-14 17:13:07 +02:00
Eelco Dolstra
9b9d5297ba
Merge pull request #3909 from matthewbauer/readd-hashed-mirrors 
Add hashed-mirrors back
 2020-08-14 17:05:47 +02:00
Eelco Dolstra
d81f13f7cb
Merge pull request #3899 from obsidiansystems/make-narHash-not-optional 
Make narHash in ValidPathInfo not optional
 2020-08-14 17:00:18 +02:00
Chuck
ed026f7206 Don't try to parse signature check as commit timestamp 
When the log.showSignature git setting is enabled, the output of
"git log" contains signature verification information in addition to the
timestamp GitInputScheme::fetch wants:

  $ git log -1 --format=%ct
  gpg: Signature made Sat 07 Sep 2019 02:02:03 PM PDT
  gpg:                using RSA key 0123456789ABCDEF0123456789ABCDEF01234567
  gpg:                issuer "user@example.com"
  gpg: Good signature from "User <user@example.com>" [ultimate] 1567890123
  1567890123

For folks that had log.showSignature set, this caused all nix operations
on flakes to fail:

  $ nix build
  error: stoull
 2020-08-13 17:44:42 -07:00
John Ericson
e1308b1211 Define LegacySSHStore::buildPaths using cmdBuildPaths 
Evidentally this was never implemented because Nix switched to using
`buildDerivation` exclusively before `build-remote.pl` was rewritten.

The `nix-copy-ssh` test (already) tests this.
 2020-08-13 21:27:55 +00:00
John Ericson
5ccd94501d Allow trustless building of CA derivations 
Include a long comment explaining the policy. Perhaps this can be moved
to the manual at some point in the future.

Also bump the daemon protocol minor version, so clients can tell whether
`wopBuildDerivation` supports trustless CA derivation building. I hope
to take advantage of this in a follow-up PR to support trustless remote
building with the minimal sending of derivation closures.
 2020-08-13 18:15:57 +00:00
John Ericson
85aacbee64 Use TeeSink and TeeSouce in a few more places 2020-08-13 14:51:17 +00:00
John Ericson
d2f2be0f70 Test RemoteStore::buildDerivation 
Fix `wopNarFromPath` which needed a `toRealPath`.
 2020-08-13 04:07:14 +00:00
John Ericson
5d67f18c86 Merge branch 'daemon-auth-cleanup' of github.com:obsidiansystems/nix into HEAD 2020-08-12 18:22:31 +00:00
John Ericson
4720853129 Make system-features a store setting 
This seems more correct. It also means one can specify the features a
store should support with --store and remote-store=..., which is useful.
I use this to clean up the build remotes test.
 2020-08-12 18:13:00 +00:00
John Ericson
8d4162ff9e Separate auth and logic for the daemon 
Before, processConnection wanted to know a user name and user id, and
`nix-daemon --stdio`, when it isn't proxying to an underlying daemon,
would just assume "root" and 0. But `nix-daemon --stdio` (no proxying)
shouldn't make guesses about who holds the other end of its standard
streams.

Now processConnection takes an "auth hook", so `nix-daemon` can provide
the appropriate policy and daemon.cc doesn't need to know or care what
it is.
 2020-08-12 15:22:33 +00:00
John Ericson
5f80aea795 Break out lambda so output can be matched just once 
This is much better.
 2020-08-12 02:23:31 +00:00
John Ericson
18834f7764 Recheck path validity after acquiring lock 
It might have changed, and in any event this is how the cod used to work
so let's just keep it.
 2020-08-11 23:44:02 +00:00
John Ericson
2de201254e Don't assume a total output map in two places in build.cc 
Thanks @regnat for catching one of them. The other follows for many of
the same reasons. I'm find fixing others on a need-to-fix basis,
provided their are no regressions.
 2020-08-11 23:07:50 +00:00
John Ericson
4c6aac8fdf Clarify comment on sandbox and temp fresh paths 2020-08-11 22:46:05 +00:00
John Ericson
6d57139050 Clarify outputReferences variable with self-describing type 
Thanks for the idea, @Regnat!
 2020-08-11 22:34:09 +00:00
John Ericson
8a068bd025 Remove redundant equality check 2020-08-11 21:25:40 +00:00
John Ericson
07e3466eb4 Float comment to out describe gaveUpOnSubstitution in general 2020-08-11 21:16:14 +00:00
John Ericson
d0f6e338dd
Apply suggestions from code review 
Thanks!!

Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
 2020-08-11 16:49:10 -04:00
Ben Burdette
1a281ec07f demote remote build message to Info 2020-08-11 10:29:43 -06:00
Eelco Dolstra
7a911b6783
Merge pull request #3914 from obsidiansystems/small-drv-serialize-cleanup 
Two small derivation serialization cleanups
 2020-08-11 07:14:38 +02:00
John Ericson
d3fa8c04c6 Simplify code as output env vars are unconditional 
Since the jsonObject unique ptr is reset to flush the string to make
`__json`, all these `!jsonObject` conditions will always be true.
 2020-08-11 01:13:26 +00:00
John Ericson
2a0902634e Fix error in merge breaking floating CA drvs 
Forgot to add this hunk!
 2020-08-11 00:13:19 +00:00
Maximilian Bosch
5f8ae16c8b
Always reset ANSI colors in progress-bar line 
When having a message like `waiting for a machine to build X` and
building with `nix build -L`, the log-prefix is always colored yellow[1]
on a small terminal-width as everything (including the ANSI color-reset) is
stripped away.

To work around that problem, this patch explicitly adds an `ANSI_NORMAL`
to the end of the line.

[1] https://imgur.com/a/FjtJOk3
 2020-08-10 17:44:17 +02:00
John Ericson
1b5c24662b Merge branch 'small-drv-serialize-cleanup' of github.com:obsidiansystems/nix into single-ca-drv-build 2020-08-10 01:57:54 +00:00
John Ericson
581183d4d5 Deduplicate parsing and reading derivations 2020-08-10 01:40:50 +00:00
John Ericson
bcd0629c2e Remove name parameter from writeDerivation 
The name is now stored with the derivation itself.
 2020-08-10 01:35:59 +00:00
John Ericson
e913a2989f Squashed get CA derivations building 2020-08-07 19:51:55 +00:00
Eelco Dolstra
edfd676e05 Fix .ls file names in binary caches 
These are not supposed to include the 'name' part of the store
path. This was broken by 759947bf72.
 2020-08-07 21:18:29 +02:00
John Ericson
8f92bb5ad9 Merge branch 'drv-outputs-map-allow-missing' of github.com:obsidiansystems/nix into templated-daemon-protocol 2020-08-07 18:51:01 +00:00
John Ericson
f7ba16f9cb Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing 2020-08-07 17:46:39 +00:00
John Ericson
47644e49ca Specialize std::optional<StorePath> so this is backwards compatible 
While I am cautious to break parametricity, I think it's OK in this
cases---we're not about to try to do some crazy polymorphic protocol
anytime soon.
 2020-08-07 17:05:14 +00:00
John Ericson
39ae9a3d4a Merge remote-tracking branch 'upstream/master' into fix-and-ci-static-builds 2020-08-07 14:44:47 +00:00
Eelco Dolstra
3c75ddc16b nix build (and others): Force re-evaluation of cached errors 
Fixes #3872.

This is a bit hacky. Ideally we would automatically re-evaluate the
failed attribute iff we need to print the error message (so in
commands like 'nix search' we wouldn't re-evaluate because we're
suppressing errors).
 2020-08-07 14:47:23 +02:00
Eelco Dolstra
2ffc058950 Make --no-eval-cache a global setting 2020-08-07 14:13:24 +02:00
Matthew Bauer
96c158d6e1 Fix build 2020-08-06 21:04:31 -05:00
Matthew Bauer
8abc577cc2 Merge remote-tracking branch 'origin/master' into readd-hashed-mirrors 2020-08-06 18:58:59 -05:00
Carlo Nucera
46f9dd56da Fix bug due to non-deterministic arg eval order 2020-08-06 19:30:05 -04:00
Matthew Bauer
641c950701 Add hashed-mirrors back 
Some users have their own hashed-mirrors setup, that is used to mirror
things in addition to what’s available on tarballs.nixos.org. Although
this should be feasable to do with a Binary Cache, it’s not always
easy, since you have to remember what "name" each of the tarballs has.
Continuing to support hashed-mirrors is cheap, so it’s best to leave
support in Nix. Note that NIX_HASHED_MIRRORS is also supported in
Nixpkgs through fetchurl.nix.

Note that this excludes tarballs.nixos.org from the default, as in
\#3689. All of these are available on cache.nixos.org.
 2020-08-06 18:19:09 -05:00
Carlo Nucera
9ab07e99f5 Use template structs instead of phantoms 2020-08-06 18:04:13 -04:00
Carlo Nucera
3d8240c32e Remove leftover commented code 2020-08-06 16:04:18 -04:00
Carlo Nucera
f795f0fabc Merge branch 'drv-outputs-map-allow-missing-namespace' of github.com:obsidiansystems/nix into templated-daemon-protocol 2020-08-06 15:53:09 -04:00
John Ericson
e89b5bd0bf Minimize the usage of Hash::dummy 2020-08-06 18:31:48 +00:00
John Ericson
5e59b25a23 Merge remote-tracking branch 'upstream/master' into make-narHash-not-optional 2020-08-06 15:34:15 +00:00
Eelco Dolstra
6146447842
Merge pull request #3856 from obsidiansystems/buildable-variant 
Make `Buildable` a `std::variant`
 2020-08-06 14:14:06 +02:00
Eelco Dolstra
59067f0f58 repl.cc: Check for HAVE_BOEHMGC 
Fixes #3906.
 2020-08-06 11:40:41 +02:00
Carlo Nucera
8b175f58d1 Simplify the namespace 2020-08-05 17:57:07 -04:00
Carlo Nucera
0739d428e0 Solve template deduction problem 
We had to predeclare our template functions
 2020-08-05 17:49:45 -04:00
John Ericson
6c66331d5c WIP: Put the worker protocol read and write in a namespace to disambig 2020-08-05 20:37:48 +00:00
Carlo Nucera
1d2e80ddd6 Merge branch 'master' of github.com:NixOS/nix into new-interface-for-path-pathOpt 2020-08-05 15:45:33 -04:00
John Ericson
ed96e603e1 Proxy -> Phantom to match Rust 
Sorry, Haskell.
 2020-08-05 19:44:08 +00:00
Carlo Nucera
8241e660ba Remove Hash::operator bool () 
Since the hash is not optional anymore
 2020-08-05 15:30:38 -04:00
Carlo Nucera
be6e1c6457 Merge branch 'master' of github.com:NixOS/nix into make-narHash-not-optional 2020-08-05 15:14:47 -04:00
Carlo Nucera
1ad6394b33 Add Hash::dummy to signal default value 
We did this in the same spirit of the dummy value that's present in
libstore/path.hh
 2020-08-05 15:11:49 -04:00
Carlo Nucera
1d71028f4d Remove optionality in ValidPathInfo::narInfo 2020-08-05 14:42:48 -04:00
John Ericson
7302761f64 Merge remote-tracking branch 'obsidian/drv-outputs-map-allow-missing' into templated-daemon-protocol 2020-08-05 17:53:24 +00:00
Ben Burdette
31f1af0cab don't crash if there's no drvPath 2020-08-05 11:26:06 -06:00
Ben Burdette
e4eae078a5 add derivation path to hint 2020-08-05 11:21:36 -06:00
Ben Burdette
f1a47a96b6 error messages for issue 2238 2020-08-05 10:58:00 -06:00
John Ericson
cf939055c8 Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing 2020-08-05 16:43:30 +00:00
John Ericson
a9bbfaa851 Fix --profile with multiple opaque paths 2020-08-05 16:27:15 +00:00
John Ericson
d89472a912 Merge remote-tracking branch 'upstream/master' into buildable-variant 2020-08-05 15:41:57 +00:00
John Ericson
b3e73547a0
Update src/libexpr/primops.cc 
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-08-05 11:05:46 -04:00
John Ericson
0559ff3d8b Merge branch 'master' of github.com:NixOS/nix into derivation-primop-floating-output 2020-08-05 14:56:40 +00:00
John Ericson
92ad550e96 Merge remote-tracking branch 'obsidian/misc-ca' into derivation-primop-floating-output 2020-08-05 14:51:41 +00:00
Eelco Dolstra
790b694be7 Style fix 2020-08-05 16:51:06 +02:00
Eelco Dolstra
e48f944e9d Merge branch 'misc-ca' of https://github.com/obsidiansystems/nix 2020-08-05 16:50:05 +02:00
John Ericson
b9ebe373bb Sed some names to perhaps avoid conflicts 2020-08-05 14:49:25 +00:00
Eelco Dolstra
25f7912156 Style fix 2020-08-05 16:47:48 +02:00
John Ericson
e561a13a58 Reanme DerivationType::Regular defintion too 
This is the one non-prefixed occurence
 2020-08-05 14:45:56 +00:00
John Ericson
e7b0847f2d Make names more consistent 2020-08-05 14:44:39 +00:00
John Ericson
839f0fe095 Merge remote-tracking branch 'upstream/master' into misc-ca 2020-08-05 14:40:01 +00:00
John Ericson
03f4fafc27 Merge remote-tracking branch 'upstream/master' into misc-ca 2020-08-05 14:36:25 +00:00
Eelco Dolstra
b91dc7ebad
Merge pull request #3730 from obsidiansystems/better-ca-parse-errors 
Improve hash parsing and errors
 2020-08-05 16:33:07 +02:00
Eelco Dolstra
75f220a595
Merge pull request #3864 from obsidiansystems/more-topo-sort 
Abstract out topo sorting logic
 2020-08-05 16:07:29 +02:00
Eelco Dolstra
088dcea0e8 Typo 2020-08-05 15:41:51 +02:00
John Ericson
16c98bf57c Get rid of some unneeded temporaries 2020-08-04 22:36:31 +00:00
John Ericson
1dfcbebc95 Organize and format code a bit 2020-08-04 22:28:10 +00:00
John Ericson
45b6fdb22b Remove unused functions 2020-08-04 22:10:13 +00:00
John Ericson
1bab8a321f Remove unneeded definition 
Template instantiations will cover this case fine.
 2020-08-04 21:56:42 +00:00
John Ericson
2f2ae993dc WIP systematize more of the worker protocol 
This refactor should *not* change the wire protocol.
 2020-08-04 19:02:05 +00:00
John Ericson
fbeb8695fb Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing 2020-08-04 18:35:33 +00:00
John Ericson
d3452a5ed6 Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors 2020-08-04 16:13:20 +00:00
Eelco Dolstra
5b22a2c0d4
Merge pull request #3894 from obsidiansystems/json-test-order-agnostic 
Make JSON equality tests agnostic to ordering
 2020-08-04 17:24:46 +02:00
Eelco Dolstra
4e7f1c7f11 S3BinaryCacheStore: Fix size determination 2020-08-04 16:01:13 +02:00
Eelco Dolstra
dfe66420e7 Revert "Remove putBytes" 
This reverts commit b8eea7e81a.
 2020-08-04 15:56:10 +02:00
Eelco Dolstra
327b1bf378 BinaryCacheStore: Explicitly flush file sink 
The file sink is also flushed in its destructor, but we ignore any
exceptions in the destructor.

Issue #3886.
 2020-08-04 14:54:07 +02:00
Eelco Dolstra
ed52cf632b
Merge pull request #3892 from obsidiansystems/path-info-header 
Move ValidPathInfo to its own header
 2020-08-04 10:44:47 +02:00
John Ericson
fe7e57a80d tab -> space 2020-08-04 03:47:09 +00:00
John Ericson
062533f7cd Merge remote-tracking branch 'upstream/master' into path-info-header 2020-08-03 21:02:28 +00:00
Eelco Dolstra
24e07c428f Delete compressed NARs 
Fixes #3891.
 2020-08-03 18:34:52 +02:00
John Ericson
54281f3ac1 addToStore in terms of addToStoreFromDump is not local-store-specific 2020-08-03 04:13:45 +00:00
John Ericson
9357512d73 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-08-01 19:38:35 +00:00
John Ericson
c4ada76e86 Fix error message and avoid recalculation 2020-08-01 16:22:50 +00:00
John Ericson
bc165e28ae Embelish documentation of new Hash functions 2020-08-01 15:32:20 +00:00
John Ericson
3cbee1e840 Convert to C-style comments 2020-08-01 15:26:57 +00:00
Carlo Nucera
b6d97fdbf4 Merge branch 'master' of github.com:NixOS/nix into drv-outputs-map-allow-missing 2020-07-31 13:12:51 -04:00
Eelco Dolstra
a3f9625818 Tweak description 2020-07-31 17:32:40 +02:00
Eelco Dolstra
935723eb3b Merge branch 'nix-command-build-check' of https://github.com/Ma27/nix 2020-07-31 17:30:55 +02:00
Eelco Dolstra
f56dfce734 nix bundle: Set category 2020-07-31 17:30:12 +02:00
Maximilian Bosch
5a09eb86f1
nix/build: add --rebuild option 
Occasionally, `nix-build --check` is fairly helpful and I'd like to be
able to use this feature for flakes that need to be built with `nix
build` as well.
 2020-07-31 17:17:03 +02:00
Eelco Dolstra
bf290c2306
Merge remote-tracking branch 'origin/master' into markdown 2020-07-31 16:07:04 +02:00
Eelco Dolstra
1d0a7b54fa
Enable syntax highlighting 2020-07-31 15:43:25 +02:00
Eelco Dolstra
0604cfd0eb
Merge pull request #3880 from matthewbauer/nix-bundle 
Add "bundle" command to Nix
 2020-07-31 11:45:53 +02:00
John Ericson
e3a2154f5a Fix indentation 2020-07-31 01:07:59 +00:00
Carlo Nucera
eee6ef86cd Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors 2020-07-30 18:27:25 -04:00
Matthew Bauer
cdc2386644 Make expectedHash optional in prim_path 
This fixes an error found in builtins.path that looks like:

store path mismatch in (possibly filtered) path added from '/private/tmp/nix-shell.CyXViH/nix-test/filter-source/filterin'

when no hash is specified
 2020-07-30 16:40:40 -05:00
Matthew Bauer
fa2d1fb36e Pass system to bundler 2020-07-30 15:37:05 -05:00
Matthew Bauer
22fcfdf18a Address misc review 2020-07-30 15:18:57 -05:00
Matthew Bauer
1a705637ce Remove single file restriction for bundler 2020-07-30 15:16:29 -05:00
Matthew Bauer
52407f83a1 exporter -> bundler 2020-07-30 15:16:23 -05:00
Matthew Bauer
d7ffe327ae Merge remote-tracking branch 'origin/master' into fix-and-ci-static-builds 2020-07-30 14:59:57 -05:00
Matthew Bauer
05ac4db39a Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-07-30 12:38:24 -05:00
Matthew Bauer
5d04a4db9b Handle exporters checking correctly 2020-07-30 11:45:47 -05:00
Matthew Bauer
2f4250a416 Add "export" to Nix 
This adds a ‘nix export’ command which hooks into nix-bundle. It can
be used in a similar way as nix-bundle, with the benefit of hooking
into the new “app” functionality. For instance,

$ nix export nixpkgs#jq
$ ./jq --help
jq - commandline JSON processor [version 1.6]
...
$ scp jq machine-without-nix:
$ ssh machine-without-nix ./jq
jq - commandline JSON processor [version 1.6]
...

Note that nix-bundle currently requires Linux to run. Other exporters
might not have that requirement.

“exporters” are meant to be reusable, so that, other repos can
implement their own bundling.

Fixes #3705
 2020-07-30 11:33:22 -05:00
Eelco Dolstra
a785b3eddf Fix build 2020-07-30 15:27:28 +02:00
Eelco Dolstra
3f6e88a552 unsigned long long -> uint64_t 2020-07-30 13:34:04 +02:00
Eelco Dolstra
ebee2b7852 receiveContents(): unsigned int -> size_t 2020-07-30 13:00:30 +02:00
Mat Marini
d9e23bfee2 Prevent nix-build from accepting --run/--command 2020-07-29 20:53:02 -04:00
Mat Marini
e4f6f8da77 Prevent nix-build from accepting --packages 2020-07-29 20:50:06 -04:00
Eelco Dolstra
0c94c17644 Merge remote-tracking branch 'origin/master' into markdown 2020-07-29 18:08:57 +02:00
Eelco Dolstra
f63839bfa4 Cleanup 2020-07-29 18:04:51 +02:00
Eelco Dolstra
39311e7009
Merge pull request #3549 from Ma27/fetchgit-hash 
Merge legacy `fetchGit`-builtin with the generic `fetchTree`-function
 2020-07-29 11:33:27 +02:00
Eelco Dolstra
7c097275c4
Merge pull request #3877 from matthewbauer/develop-continuous-regex 
Set regex_constants::match_continuous for quicker search in nix develop
 2020-07-29 10:07:52 +02:00
Matthew Bauer
fa8515d7ec Set regex_constants::match_continuous for quicker search 
match_continuous limits the search to the current start position,
instead of searching the entire file.

On libc++, this improves performance dramatically:

$ time /nix/store/70ai68dfm6xbzwn26j5n4li9di52ylia-nix-3.0pre20200728_c159f48/bin/nix print-dev-env >/dev/null
/nix/store/70ai68dfm6xbzwn26j5n4li9di52ylia-nix-3.0pre20200728_c159f48/bin/ni  2.39s user 0.19s system 64% cpu 4.032 total
$ time /nix/store/cwjfxxlp83zln4mfyy1d2dbsx7f6s962-nix-3.0pre20200728_dirty/bin/nix print-dev-env >/dev/null
/nix/store/cwjfxxlp83zln4mfyy1d2dbsx7f6s962-nix-3.0pre20200728_dirty/bin/nix   0.09s user 0.05s system 65% cpu 0.204 total

Fixes #3874
 2020-07-29 00:57:20 -05:00
Eelco Dolstra
4c0077a07d Fix RemoteStore::addToStore() latency 
Since 6185d25e52, this was very
latency-bound since it required a round-trip for every 32 KiB. So for
example copying a 514 MiB closure over a virtual ethernet device with
a articial delay of just 1 ms took 343s. Now it takes 2.7s.

Fixes #3372.
 2020-07-29 00:48:39 +02:00
Eelco Dolstra
c159f48a39 Cleanup 2020-07-29 00:24:55 +02:00
John Ericson
df707d05d1 Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca 2020-07-28 21:12:36 +00:00
Maximilian Bosch
e4940e90f3
Restore backwards-compat for current builtins.fetchGit 
If a repo is dirty, it used to return a `rev` object with an "empty"
sha1 (0000000000000000000000000000000000000000). Please note that this
only applies for `builtins.fetchGit` and *not* for `builtins.fetchTree{
type = "git"; }`.
 2020-07-28 22:46:39 +02:00
Carlo Nucera
2980b244b7 Use assert(false) instead of abort() 2020-07-28 15:39:45 -04:00
Carlo Nucera
cfe96bd7ce Merge branch 'master' of github.com:NixOS/nix into ca-drv 2020-07-28 15:36:45 -04:00
Carlo Nucera
c318d398f3 Merge branch 'misc-ca' of github.com:obsidiansystems/nix into new-interface-for-path-pathOpt 2020-07-28 14:22:24 -04:00
Carlo Nucera
7ef1e3cd14 Use the new interface 2020-07-28 13:59:24 -04:00
Carlo Nucera
d564ac1c50 Offer a safer interface for pathOpt 
The new interface we offer provides a way of getting all the
DerivationOutputs with the storePaths directly, based on the observation
that it's the most common usecase.
 2020-07-28 13:55:15 -04:00
Eelco Dolstra
b8eea7e81a Remove putBytes 
istream->tellg() returns -1 so we can't get the number of bytes
written.

Fixes 'uploaded 's3://nix-cache/nar/00819r9lp5kajr6baxfw5dhhc0cx8ndxaz43qmd2f0gn1hk1ynlp.nar.xz' (-1 bytes) in 11620 ms' messages.
 2020-07-28 13:27:56 +02:00
Maximilian Bosch
f742438465
Merge legacy fetchGit-builtin with the generic fetchTree-function 
The original idea was to implement a git-fetcher in Nix's core that
supports content hashes[1]. In #3549[2] it has been suggested to
actually use `fetchTree` for this since it's a fairly generic wrapper
over the new fetcher-API[3] and already supports content-hashes.

This patch implements a new git-fetcher based on `fetchTree` by
incorporating the following changes:

* Removed the original `fetchGit`-implementation and replaced it with an
  alias on the `fetchTree` implementation.

* Ensured that the `git`-fetcher from `libfetchers` always computes a
  content-hash and returns an "empty" revision on dirty trees (the
  latter one is needed to retain backwards-compatibility).

* The hash-mismatch error in the fetcher-API exits with code 102 as it
  usually happens whenever a hash-mismatch is detected by Nix.

* Removed the `flakes`-feature-flag: I didn't see a reason why this API
  is so tightly coupled to the flakes-API and at least `fetchGit` should
  remain usable without any feature-flags.

* It's only possible to specify a `narHash` for a `git`-tree if either a
  `ref` or a `rev` is given[4].

* It's now possible to specify an URL without a protocol. If it's missing,
  `file://` is automatically added as it was the case in the original
  `fetchGit`-implementation.

[1] https://github.com/NixOS/nix/pull/3216
[2] https://github.com/NixOS/nix/pull/3549#issuecomment-625194383
[3] https://github.com/NixOS/nix/pull/3459
[4] https://github.com/NixOS/nix/pull/3216#issuecomment-553956703
 2020-07-28 00:44:39 +02:00
John Ericson
8065c6d160 Abstract out topo sorting logic 2020-07-27 20:45:34 +00:00
John Ericson
951415b568 Require ca-derivations everywhere we create a CA derivation 
"create" as in read one in from a serialized form, or build one from
scratch in memory.
 2020-07-27 17:56:36 +00:00
John Ericson
e32a9e124b Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output 2020-07-27 17:50:06 +00:00
John Ericson
7cf978440c Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca 2020-07-27 17:47:40 +00:00
John Ericson
e6115ca004 Merge remote-tracking branch 'upstream/master' into ca-derivation-data-types 2020-07-27 17:41:43 +00:00
John Ericson
1d7d94ceea Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-07-27 16:17:50 +00:00
John Ericson
43f2bd8dc5 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-27 16:13:57 +00:00
John Ericson
387699fe7f Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca 2020-07-27 15:00:28 +00:00
John Ericson
78466bcb2f Merge branch 'optional-derivation-output-storepath' into ca-derivation-data-types 2020-07-27 14:40:08 +00:00
John Ericson
d5bb67cfa4 Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath 2020-07-27 14:29:32 +00:00
John Ericson
699fc89b39 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-27 14:22:45 +00:00
Maximilian Bosch
6ccfdb79c7
libutil/logging: extend internal-json logger to make it more machine-readable 
The new error-format is pretty nice from a UX point-of-view, however
it's fairly hard to parse the output e.g. for editor plugins such as
vim-ale[1] that use `nix-instantiate --parse` to determine syntax errors in
Nix expression files.

This patch extends the `internal-json` logger by adding the fields
`line`, `column` and `file` to easily locate an error in a file and the
field `raw_msg` which contains the error-message itself without
code-lines and additional helpers.

An exemplary output may look like this:

```
[nix-shell]$ ./inst/bin/nix-instantiate ~/test.nix --log-format minimal
{"action":"msg","column":1,"file":"/home/ma27/test.nix","level":0,"line":4,"raw_msg":"syntax error, unexpected IF, expecting $end","msg":"<full error-msg with code-lines etc>"}
```

[1] https://github.com/dense-analysis/ale
 2020-07-26 02:29:57 +02:00
John Ericson
2c7557481b queryDerivationOutputMap no longer assumes all outputs have a mapping 
This assumption is broken by CA derivations. Making a PR now to do the
breaking daemon change as soon as possible (if it is already too late,
we can bump protocol intead).
 2020-07-24 21:14:06 +00:00
Eelco Dolstra
a71d1cedff
printVersion(): Show system types 2020-07-24 11:34:01 +02:00
Eelco Dolstra
2292814049
createUnixDomainSocket(): Fix off-by-one error in copying the socket path 
Reported by Kane York.
 2020-07-24 11:19:17 +02:00
John Ericson
58e55c0923 Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output 2020-07-23 19:17:52 +00:00
John Ericson
a2af25c55a Merge remote-tracking branch 'upstream/master' into misc-ca 2020-07-23 19:14:17 +00:00
John Ericson
e1de1fe0d8 Make Buildable a std::variant 
I think this better captures the intent of what's going on: we either
have an opaque store path, or a drv path with some outputs.

Having this structure will also help us support CA derivations: we'll
have to allow the outpath paths to be optional, so the structure we gain
now makes up for the structure we loose then.
 2020-07-23 19:02:57 +00:00
John Ericson
c36916eca0 Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output 2020-07-23 15:06:56 +00:00
John Ericson
6fcd9966bf Merge remote-tracking branch 'upstream/master' into misc-ca 2020-07-23 15:02:29 +00:00
John Ericson
66a2067288 Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors 2020-07-23 14:56:35 +00:00
John Ericson
2274f63453 Get rid of basicDerivation::findOutput 
It's a tiny function which is:

 - hardly worth abstrating over, and also only used once.

 - doesn't work once we get CA drvs

I rewrote the one callsite to be forwards compatable with CA
derivations, and also potentially more performant: instead of reading in
the derivation it can ust consult the SQLite DB in the common case.
 2020-07-23 14:39:02 +00:00
Fabian Möller
b9ead08ca8
Save changes made by "nix registry pin" to user registry 2020-07-23 14:21:27 +02:00
John Ericson
9423f64ee2 Parse CA derivations using new output variants 
We no longer need `ParsedDerivation` because everything libstore needs
to know about is in the `BasicDerivation` proper.
 2020-07-22 23:59:25 +00:00
John Ericson
c56356bacc Separate concerns in scanForReferences with TeeSink 
This also will make it easier to use a `HashModuloSink` instead for CA
derivations.
 2020-07-22 22:13:44 +00:00
Eelco Dolstra
ff314f186e
Merge pull request #3736 from obsidiansystems/allow-relative-paths-in-store-option 
Allow relative paths in --store option
 2020-07-21 19:43:07 +02:00
Eelco Dolstra
0951330680
Merge pull request #3714 from obsidiansystems/add-body-to-network-errors 
Add response body to network errors
 2020-07-21 18:09:42 +02:00
John Ericson
02639716ea Merge branch 'allow-relative-paths-in-store-option' into remove-storetype-delegate-regStore 2020-07-21 15:55:27 +00:00
John Ericson
6cce32c8e8 Change logic for deciding what is a relative path for the local store 
The was Eelco's prefered logic, and it looks good to me!
 2020-07-21 15:39:47 +00:00
Carlo Nucera
922a845ffc Update chunkSize to the suggested value 
This was a suggested course of action in a review in one of our earlier
commits, https://github.com/NixOS/nix/pull/3801#discussion_r457557079
 2020-07-21 10:24:19 -04:00
John Ericson
54e507a7aa Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors 2020-07-21 14:17:59 +00:00
John Ericson
5ce95b9529
Update src/libstore/build.cc 2020-07-21 09:47:40 -04:00
John Ericson
cdb3f39b64 Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors 2020-07-21 13:43:20 +00:00
John Ericson
5055c595bd Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into ca-derivation-data-types 2020-07-21 01:20:53 +00:00
John Ericson
5a37413796 Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into from-dump-stream 2020-07-21 01:20:14 +00:00
John Ericson
bf0dca7790 Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into misc-ca 2020-07-21 01:18:33 +00:00
Carlo Nucera
9aae179f34 Correct bug, thoroughly document addToStoreSlow 2020-07-20 20:18:12 -04:00
Carlo Nucera
0ca9744694 Use heuristics to decide when to show the response 
Due to https://github.com/NixOS/nix/issues/3841 we don't know how print
different messages for different verbosity levels.
 2020-07-20 15:57:58 -04:00
Carlo Nucera
6357b1b0fb Add another Unimplemented case 2020-07-20 14:17:25 -04:00
Carlo Nucera
007bf6d17a Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca 2020-07-20 14:14:39 -04:00
Carlo Nucera
362ae93851 Add UnimplementedError to ease grepping for these 2020-07-20 14:13:37 -04:00
John Ericson
c58c6165c5 Remove period at the end of the exception message 2020-07-20 17:43:19 +00:00
John Ericson
bf61871271 parser.hh -> split.hh 2020-07-20 17:42:34 +00:00
John Ericson
eab945f591 Merge remote-tracking branch 'upstream/master' into from-dump-stream 2020-07-20 17:29:23 +00:00
Eelco Dolstra
1c5f8bbfb5
Merge pull request #3822 from obsidiansystems/dump-thrice-fixme 
Optimize `addToStoreSlow` and remove `TeeParseSink`
 2020-07-20 18:55:05 +02:00
John Ericson
ac2fc7ba1f
Apply suggestions from code review 
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-07-20 11:29:46 -04:00
Carlo Nucera
0aa79dcc6f Remove StoreType abstraction and delegate regStore 
to each Store implementation. The generic regStore implementation will
only be for the ambiguous shorthands, like "" and "auto".

This also could get us close to simplifying the daemon command.
 2020-07-17 17:24:02 -04:00
John Ericson
6756cecfcf Add DerivationType::CAFloating 2020-07-17 19:55:41 +00:00
Carlo Nucera
fefd6c9e5f Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option 2020-07-17 14:34:20 -04:00
Carlo Nucera
bbc633c98c Revert "Don't anticipate CA but not fixed outputs for now" 
This reverts commit 3a9e4c3262.
 2020-07-17 13:10:32 -04:00
Carlo Nucera
205dcd140d Revert "Don't anticipate multiple CA outputs for now" 
This reverts commit 74b251b2f3.
 2020-07-17 12:43:46 -04:00
Carlo Nucera
1feb8981df Revert "Don't anticipate hash algo without hash in derivation for now" 
This reverts commit 3804e3df9b.
 2020-07-17 12:05:54 -04:00
Lars Mühmel
bc73590151 nix edit: call restoreSignals() before execvp-ing the $EDITOR 
Currently resizing of the terminal doesn't play nicely with
nix edit when using kakoune as the editor, as it relies on the
SIGWINCH signal which is trapped by nix. How this is not a problem
with e.g. vim is beyond me.

Virtually all other exec* calls are following a call to
restoreSignals(). This commit adds this behavior to nix edit
as well.
 2020-07-17 17:35:59 +02:00
Carlo Nucera
487c5751c6 Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca 2020-07-17 11:32:59 -04:00
Carlo Nucera
a395c12830 Merge branch 'master' of github.com:NixOS/nix into misc-ca 2020-07-17 11:06:54 -04:00
Carlo Nucera
5cb840541b Merge branch 'multi-output-hashDerivationModulo' of github.com:Ericson2314/nix into misc-ca 2020-07-17 10:28:33 -04:00
Eelco Dolstra
17f75f9cc4 parseFlakeRef(): Only search for the top-level directory for CLI flakerefs 2020-07-17 14:54:21 +02:00
Eelco Dolstra
52c8be38e0 nix profile diff-closures: Don't inherit EvalCommand 2020-07-17 12:36:12 +02:00
Carlo Nucera
745a03cef5 Merge branch 'optional-derivation-output-storepath' of github.com:obsidiansystems/nix into ca-derivation-data-types 2020-07-16 13:36:01 -04:00
Carlo Nucera
048e916f64 Merge branch 'master' of github.com:NixOS/nix into optional-derivation-output-storepath 2020-07-16 13:32:28 -04:00
John Ericson
cc0d77f8c9 Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-07-16 17:28:52 +00:00
Carlo Nucera
e3a3406db8 Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors 2020-07-16 12:04:10 -04:00
Eelco Dolstra
16c9f6762d Add command 'nix profile diff-closure' 
This shows all changes between generations of a profile. E.g.

  $ nix profile diff-closures --profile /nix/var/nix/profiles/system
  Generation 654 -> 655:
    nix: 2.4pre20200617_5d69bbf → 2.4pre20200701_6ff9aa8, +42.2 KiB

  Generation 655 -> 656:
    blender-bin: 2.83.0 → 2.83.1, -294.2 KiB

  Generation 656 -> 657:
    curl: 7.68.0 → 7.70.0, +19.1 KiB
    firmware-linux-nonfree: 2020-01-22 → 2020-05-19, +30827.7 KiB
    ibus: -21.8 KiB
    initrd-linux: 5.4.46 → 5.4.49
    ...
 2020-07-16 17:00:42 +02:00
John Ericson
5ea817dace Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-16 14:58:53 +00:00
John Ericson
3dcca18c30 Fix bug in TeeSource 
We use this to simplify `LocalStore::addToStoreFromDump`.

Also, hope I fixed build error with old clang (used in Darwin CI).
 2020-07-16 13:39:27 +00:00
Carlo Nucera
e3b394b6e8 Small namespace fix 2020-07-16 09:36:02 -04:00
Eelco Dolstra
5517eee17e Generations API cleanup 2020-07-16 15:18:21 +02:00
Eelco Dolstra
8807ff902e nix diff-closures: Fix build 2020-07-16 14:25:51 +02:00
Eelco Dolstra
716eafaca4 Merge branch 'diff-closures' 2020-07-16 14:19:35 +02:00
Eelco Dolstra
3f264916db Merge remote-tracking branch 'origin/flakes' 2020-07-16 14:07:32 +02:00
John Ericson
68dfb8c6ae Optimize addToStoreSlow and remove TeeParseSink 2020-07-16 05:09:41 +00:00
John Ericson
5602637d9e Revert "LocalStore::addToStoreFromDump copy in chunks" 
This reverts commit 592851fb67. We don't
need this extra feature anymore
 2020-07-15 23:37:49 +00:00
John Ericson
bc109648c4 Get rid of LocalStore::addToStoreCommon 
I got it to just become `LocalStore::addToStoreFromDump`, cleanly taking
a store and then doing nothing too fancy with it.

`LocalStore::addToStore(...Path...)` is now just a simple wrapper with a
bare-bones sinkToSource of the right dump command.
 2020-07-15 23:14:30 +00:00
Carlo Nucera
455bdee205 Merge branch 'master' of github.com:NixOS/nix into derivation-header-include-order 2020-07-15 17:58:30 -04:00
John Ericson
64b7421741 Merge branch 'handle-flat-case' into from-dump-stream 2020-07-15 21:22:55 +00:00
Carlo Nucera
d090562348 Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type 2020-07-15 17:21:01 -04:00
John Ericson
d087cf4855 Revert "Revert "LocalStore::addToStore(srcPath): Handle the flat case"" 
This reverts commit cff2157185.
 2020-07-15 21:10:33 +00:00
John Ericson
aff69e229c Merge remote-tracking branch 'obsidian/fix-add-to-star-narhash' into from-dump-stream 2020-07-15 21:05:29 +00:00
Eelco Dolstra
36a1242603 nix why-depends: Fix shortest path calculation 
This was completely broken since d8972317fc.
 2020-07-15 21:11:07 +02:00
Eelco Dolstra
2d6d53bc87 nix: Fix examples 2020-07-15 20:28:16 +02:00
Eelco Dolstra
3624c042ac nix: Add --derivation flag to operate on .drv paths 
For instance, 'nix why-depends --use-derivation nixpkgs#hello
nixpkgs#glibc' shows why hello's .drv depends on glibc's .drv.
 2020-07-15 20:25:10 +02:00
Eelco Dolstra
dfe8f3ebc6 nix why-depends: Fix misleading message 2020-07-15 20:09:50 +02:00
Eelco Dolstra
94eb5fad76 Clean up RealiseMode 2020-07-15 20:05:42 +02:00
Eelco Dolstra
e3c2b00237 Make InstallableStorePath behave consistently with InstallableValue 
That is, the commands 'nix path-info nixpkgs#hello' and 'nix path-info
/nix/store/00ls0qi49qkqpqblmvz5s1ajl3gc63lr-hello-2.10.drv' now do the
same thing (i.e. build the derivation and operate on the output store
path, rather than the .drv path).
 2020-07-15 19:50:32 +02:00
Carlo Nucera
2d2a10e79a Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors 2020-07-15 13:06:08 -04:00
Eelco Dolstra
0356f14459 Add 'nix diff-closures' command 
This command makes it easier to see what changed between two closures,
i.e. what packages/versions got added or removed, and whether there
were any notable changes in path size.

For example:

  $ nix diff-closures /nix/var/nix/profiles/system-655-link /nix/var/nix/profiles/system-658-link
  blender-bin: 2.83.0 → 2.83.2, -294.2 KiB
  curl: 7.68.0 → 7.70.0, +19.1 KiB
  firmware-linux-nonfree: 2020-01-22 → 2020-05-19, +30827.7 KiB
  ibus: -21.8 KiB
  initrd-linux: 5.4.46 → 5.4.51, +16.9 KiB
  libexif: 0.6.21 → 0.6.22, +497.6 KiB
  linux: 5.4.46 → 5.4.51, +13.2 KiB
  mesa: 19.3.3 → 19.3.5, -183.9 KiB
  nix: 2.4pre20200701_6ff9aa8 → 2.4pre20200708_9223603, +9.7 KiB
  nix-bash-completions: 0.6.8 → ∅, -57.6 KiB
  nixos-system-hagbard: 20.03.20200615.a84b797 → 20.03.20200713.add5529
  nvidia-persistenced: 440.82 → 440.100
  nvidia-settings: 440.82 → 440.100
  nvidia-x11: 440.82-5.4.46 → 440.100-5.4.51, +664.7 KiB
  pcre: 8.43 → 8.44
  php: 7.3.16 → 7.3.20, -26.2 KiB
  python3.7-youtube-dl: 2020.06.06 → 2020.06.16.1, +8.4 KiB
  samba: 4.11.5 → 4.11.9, +30.1 KiB
  sane-backends: 1.0.28 → 1.0.30, +680.5 KiB
  source: -182.0 KiB
  zfs-kernel: 0.8.3-5.4.46 → 0.8.4-5.4.51, +9.9 KiB
  zfs-user: 0.8.3 → 0.8.4, +20.1 KiB
 2020-07-15 16:24:25 +02:00
Eelco Dolstra
cff2157185 Revert "LocalStore::addToStore(srcPath): Handle the flat case" 
This reverts commit a2c27022e9. See
addToStoreSlow(), we don't need to handle this case efficiently
anymore. In fact, we can almost remove the method/hashAlgo arguments
since the non-recursive and/or non-SHA256 are almost not used anymore.
 2020-07-15 12:49:03 +02:00
John Ericson
650c2c6558 Rename variable nar -> dump according to TODO 2020-07-14 21:28:50 +00:00
John Ericson
8173e7bfef Fix localhost::addToStore(...Path...) 
We were calculating the nar hash wrong when the file ingestion method
was flat. I don't think there's anything we can do in that case but dump
the file again, so that's what I do.

As an optomization, we again could reuse the original dump for just the
recursive and non-sha256 case, but I rather do that after this fix, and
after my other PRs which deduplicate this code.
 2020-07-14 21:12:11 +00:00
John Ericson
592851fb67 LocalStore::addToStoreFromDump copy in chunks 
Rather than copying byte-by-byte, we let the coroutine know how much
data we would like it to send back to us.
 2020-07-14 13:56:36 +00:00
John Ericson
9de96ef7d4 Dedup LocalStore::addToStore* 
The downsides is that the coroutine has byte-by-byte loop transfer. Will
fix that next.
 2020-07-14 13:56:36 +00:00
John Ericson
c86fc3a965 Crudely make addToStoreFromDump take Source not string 
I just as little beyond the type as possible, so the implementation
changes this enables can be reviewed separately.
 2020-07-14 13:56:36 +00:00
John Ericson
9ec10046e0 Narrow scope of temporary value 2020-07-14 13:54:29 +00:00
Eelco Dolstra
298ff6af8f
Merge pull request #3809 from Ma27/gitlab-refs 
Fix gitlab-fetcher to obtain tags and branches
 2020-07-14 15:22:52 +02:00
Eelco Dolstra
da3aea291d EvalCache: Ignore SQLite errors 
Fixes #3794.
 2020-07-14 15:17:38 +02:00
Eelco Dolstra
832e111494 Merge remote-tracking branch 'origin/master' into flakes 2020-07-14 13:56:18 +02:00
Eelco Dolstra
926c3a6664 Doh 2020-07-14 11:55:54 +02:00
Eelco Dolstra
43b8e96d30 Fix 'nix verify --all' on a binary cache (cached case) 2020-07-13 20:17:00 +02:00
Eelco Dolstra
9502c0e2eb nix verify: Show correct path when using --all on a binary cache 2020-07-13 20:12:44 +02:00
Eelco Dolstra
7c2fef0a81 Make 'nix copy' to s3:// binary caches run in constant memory 2020-07-13 20:07:19 +02:00
Maximilian Bosch
cf9f33995b
Fix gitlab-fetcher to obtain tags and branches 
Until now, the `gitlab`-fetcher determined the source's rev by checking
the latest commit of the given `ref` using the
`/repository/branches`-API.

This breaks however when trying to fetch a gitlab-repo by its tag:

```
$ nix repl
nix-repl> builtins.fetchTree gitlab:Ma27/nvim.nix/0.2.0
error: --- Error ------------------------------------------------------------------------------------- nix
unable to download 'https://gitlab.com/api/v4/projects/Ma27%2Fnvim.nix/repository/branches/0.2.0': HTTP error 404 ('')
```

When using the `/commits?ref_name`-endpoint[1] you can pass any kind of
valid ref to the `gitlab`-fetcher.

Please note that this fetches the only first 20 commits on a ref,
unfortunately there's currently no endpoint which only retrieves the
latest commit of any kind of `ref`.

[1] https://docs.gitlab.com/ee/api/commits.html#list-repository-commits
 2020-07-13 19:22:59 +02:00
Eelco Dolstra
493961b689 Remove istringstream_nocopy 2020-07-13 18:31:19 +02:00
Eelco Dolstra
545bb2ed03 Remove 'accessor' from addToStore() 
This is only used by hydra-queue-runner and it's better to implement
it there.
 2020-07-13 18:31:19 +02:00
Eelco Dolstra
0a9da00a10 NarAccessor: Run in constant memory 2020-07-13 17:30:42 +02:00
Eelco Dolstra
fc84c358d9 Make 'nix copy' to file:// binary caches run in constant memory 2020-07-13 16:28:45 +02:00
Eelco Dolstra
400f1a9b59 Store::pathInfoToJSON(): Use consistent format for downloadHash 2020-07-13 16:26:09 +02:00
Eelco Dolstra
c0dd05131e toStorePath(): Return a StorePath and the suffix 2020-07-13 16:25:48 +02:00
Eelco Dolstra
1d01ae816b Fix 'nix verify --all' on a binary cache and add a test 2020-07-13 14:35:01 +02:00
John Ericson
c466cb2091 Merge branch 'hash-always-has-type' into better-ca-parse-errors 2020-07-13 03:02:09 +00:00
John Ericson
4415765385 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-13 03:01:44 +00:00
John Ericson
230c9b4329 Change types to prepare the way for CA derivations 
We've added the variant to `DerivationOutput` to support them, but made
`DerivationOutput::path` partial to avoid actually implementing them.

With this chage, we can all collaborate on "just" removing
`DerivationOutput::path` calls to implement CA derivations.
 2020-07-12 23:53:33 +00:00
John Ericson
fedfc913ad Use more std::visit to prepare for new variant 
N.B. not using `std::visit` for fetchurl because there is no attempt to
handle all the cases (e.g. no `else`) and lambda complicates early
return.
 2020-07-12 23:51:40 +00:00
John Ericson
ffc18583b1 Move C++17 "pattern matching" boilerplat to utils.hh 2020-07-12 22:15:14 +00:00
John Ericson
503b425690 DerivationOutputExtensional -> DerivationOutputInputAddressed 
Thanks @regnat for the great name.
 2020-07-12 15:56:20 +00:00
John Ericson
a8d4707107 Undo erroneous indentation change 2020-07-12 15:54:12 +00:00
John Ericson
18152406ce String .drv suffix to create derivation name 2020-07-12 15:40:14 +00:00
John Ericson
5d0b75e5b6 std::string_view for new derivation name parameters 2020-07-12 15:02:36 +00:00
Maximilian Bosch
64f03635d7
Fix ANSI color constants 
The `m` acts as termination-symbol when declaring graphics. Because
of this, the `;1m` doesn't have any effect and is directly printed to
the console:

```
$ nix repl
> builtins.fetchGit { /* ... */ }
{ outPath = "/nix/store/s0f0iz4a41cxx2h055lmh6p2d5k5bc6r-source"; rev = "e73e45b723a9a6eecb98bd5f3df395d9ab3633b6"; revCount = ;1m428; shortRev = "e73e45b"; submodules = ;1mfalse; }
```

Introduced by 6403508f5a.
 2020-07-12 16:52:20 +02:00
John Ericson
13ec627e0a Set derivation name in dervationStrict 2020-07-12 03:03:12 +00:00
John Ericson
1c9bec226f Don't improperly assume path is store path 2020-07-12 02:38:03 +00:00
John Ericson
abea26a968 BasicDerivation::findOutput cannot return reference anymore 2020-07-12 01:57:06 +00:00
John Ericson
442d43178b Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath 2020-07-11 02:27:45 +00:00
Matthew Bauer
acb74d4d94 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-07-10 18:13:45 -04:00
Eelco Dolstra
8efa23bb99 Avoid a redundant hash 2020-07-10 15:56:24 +02:00
Eelco Dolstra
5dff49f661 Factor out commonality between nix-prefetch-url and nix-store --add-fixed 2020-07-10 13:21:37 +02:00
Eelco Dolstra
7f1a86d57c nix-store --add-fixed: Run in constant memory 2020-07-10 12:51:56 +02:00
Eelco Dolstra
06e3dd9005 nix-prefetch-url: Run in constant memory when using RemoteStore 
Fixes #3684.
 2020-07-10 11:22:48 +02:00
John Ericson
323d51a7a0 Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors 2020-07-09 19:07:06 +00:00
Matthew Bauer
a7884970c5 Fix DerivationOutputExtensional name 2020-07-09 11:37:18 -04:00
Eelco Dolstra
a2c27022e9 LocalStore::addToStore(srcPath): Handle the flat case 
This helps nix-prefetch-url when using a local store.
 2020-07-09 15:54:32 +02:00
Matthew Bauer
06a4e15478 Fix build.cc on linux 2020-07-08 19:27:51 -04:00
Matthew Bauer
8e0d0689be Only store hash of fixed derivation output 
we don’t need a full storepath for a fixedoutput derivation. So just
putting the ingestion method + the hash is sufficient.
 2020-07-08 19:11:39 -04:00
Eelco Dolstra
b981e5aacf Cleanup 2020-07-08 22:07:21 +02:00
Eelco Dolstra
34f25124ba Make LocalStore::addToStore(srcPath) run in constant memory 
This reduces memory consumption of

  nix-instantiate \
    -E 'with import <nixpkgs> {}; runCommand "foo" { src = ./blender; } "echo foo"' \
    --option nar-buffer-size 10000

(where ./blender is a 1.1 GiB tree) from 1716 to 36 MiB, while still
ensuring that we don't do any write I/O for small source paths (up to
'nar-buffer-size' bytes). The downside is that large paths are now
always written to a temporary location in the store, even if they
produce an already valid store path. Thus, adding large paths might be
slower and run out of disk space. ¯\_(ツ)_/¯ Of course, you can always
restore the old behaviour by setting 'nar-buffer-size' to a very high
value.
 2020-07-08 22:07:21 +02:00