max/nix-super
1
0
Fork 0
mirror of https://github.com/privatevoid-net/nix-super.git synced 2024-11-14 02:06:16 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
9432 commits 7 branches 0 tags 64 MiB
Commit graph

2713 commits

Author SHA1 Message Date
Eelco Dolstra
e48f944e9d Merge branch 'misc-ca' of https://github.com/obsidiansystems/nix 2020-08-05 16:50:05 +02:00
John Ericson
b9ebe373bb Sed some names to perhaps avoid conflicts 2020-08-05 14:49:25 +00:00
Eelco Dolstra
25f7912156 Style fix 2020-08-05 16:47:48 +02:00
John Ericson
e561a13a58 Reanme DerivationType::Regular defintion too 
This is the one non-prefixed occurence
 2020-08-05 14:45:56 +00:00
John Ericson
e7b0847f2d Make names more consistent 2020-08-05 14:44:39 +00:00
John Ericson
839f0fe095 Merge remote-tracking branch 'upstream/master' into misc-ca 2020-08-05 14:40:01 +00:00
John Ericson
03f4fafc27 Merge remote-tracking branch 'upstream/master' into misc-ca 2020-08-05 14:36:25 +00:00
Eelco Dolstra
b91dc7ebad
Merge pull request #3730 from obsidiansystems/better-ca-parse-errors 
Improve hash parsing and errors
 2020-08-05 16:33:07 +02:00
Eelco Dolstra
75f220a595
Merge pull request #3864 from obsidiansystems/more-topo-sort 
Abstract out topo sorting logic
 2020-08-05 16:07:29 +02:00
John Ericson
16c98bf57c Get rid of some unneeded temporaries 2020-08-04 22:36:31 +00:00
John Ericson
1dfcbebc95 Organize and format code a bit 2020-08-04 22:28:10 +00:00
John Ericson
45b6fdb22b Remove unused functions 2020-08-04 22:10:13 +00:00
John Ericson
1bab8a321f Remove unneeded definition 
Template instantiations will cover this case fine.
 2020-08-04 21:56:42 +00:00
John Ericson
2f2ae993dc WIP systematize more of the worker protocol 
This refactor should *not* change the wire protocol.
 2020-08-04 19:02:05 +00:00
John Ericson
fbeb8695fb Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing 2020-08-04 18:35:33 +00:00
John Ericson
d3452a5ed6 Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors 2020-08-04 16:13:20 +00:00
Eelco Dolstra
4e7f1c7f11 S3BinaryCacheStore: Fix size determination 2020-08-04 16:01:13 +02:00
Eelco Dolstra
dfe66420e7 Revert "Remove putBytes" 
This reverts commit b8eea7e81a.
 2020-08-04 15:56:10 +02:00
Eelco Dolstra
327b1bf378 BinaryCacheStore: Explicitly flush file sink 
The file sink is also flushed in its destructor, but we ignore any
exceptions in the destructor.

Issue #3886.
 2020-08-04 14:54:07 +02:00
Eelco Dolstra
ed52cf632b
Merge pull request #3892 from obsidiansystems/path-info-header 
Move ValidPathInfo to its own header
 2020-08-04 10:44:47 +02:00
John Ericson
062533f7cd Merge remote-tracking branch 'upstream/master' into path-info-header 2020-08-03 21:02:28 +00:00
Eelco Dolstra
24e07c428f Delete compressed NARs 
Fixes #3891.
 2020-08-03 18:34:52 +02:00
John Ericson
54281f3ac1 addToStore in terms of addToStoreFromDump is not local-store-specific 2020-08-03 04:13:45 +00:00
John Ericson
9357512d73 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-08-01 19:38:35 +00:00
Carlo Nucera
b6d97fdbf4 Merge branch 'master' of github.com:NixOS/nix into drv-outputs-map-allow-missing 2020-07-31 13:12:51 -04:00
Eelco Dolstra
bf290c2306
Merge remote-tracking branch 'origin/master' into markdown 2020-07-31 16:07:04 +02:00
Eelco Dolstra
1d0a7b54fa
Enable syntax highlighting 2020-07-31 15:43:25 +02:00
Carlo Nucera
eee6ef86cd Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors 2020-07-30 18:27:25 -04:00
Matthew Bauer
05ac4db39a Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-07-30 12:38:24 -05:00
Eelco Dolstra
3f6e88a552 unsigned long long -> uint64_t 2020-07-30 13:34:04 +02:00
Eelco Dolstra
ebee2b7852 receiveContents(): unsigned int -> size_t 2020-07-30 13:00:30 +02:00
Eelco Dolstra
f63839bfa4 Cleanup 2020-07-29 18:04:51 +02:00
Eelco Dolstra
4c0077a07d Fix RemoteStore::addToStore() latency 
Since 6185d25e52, this was very
latency-bound since it required a round-trip for every 32 KiB. So for
example copying a 514 MiB closure over a virtual ethernet device with
a articial delay of just 1 ms took 343s. Now it takes 2.7s.

Fixes #3372.
 2020-07-29 00:48:39 +02:00
Eelco Dolstra
c159f48a39 Cleanup 2020-07-29 00:24:55 +02:00
John Ericson
df707d05d1 Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca 2020-07-28 21:12:36 +00:00
Carlo Nucera
2980b244b7 Use assert(false) instead of abort() 2020-07-28 15:39:45 -04:00
Carlo Nucera
cfe96bd7ce Merge branch 'master' of github.com:NixOS/nix into ca-drv 2020-07-28 15:36:45 -04:00
Carlo Nucera
c318d398f3 Merge branch 'misc-ca' of github.com:obsidiansystems/nix into new-interface-for-path-pathOpt 2020-07-28 14:22:24 -04:00
Carlo Nucera
7ef1e3cd14 Use the new interface 2020-07-28 13:59:24 -04:00
Carlo Nucera
d564ac1c50 Offer a safer interface for pathOpt 
The new interface we offer provides a way of getting all the
DerivationOutputs with the storePaths directly, based on the observation
that it's the most common usecase.
 2020-07-28 13:55:15 -04:00
Eelco Dolstra
b8eea7e81a Remove putBytes 
istream->tellg() returns -1 so we can't get the number of bytes
written.

Fixes 'uploaded 's3://nix-cache/nar/00819r9lp5kajr6baxfw5dhhc0cx8ndxaz43qmd2f0gn1hk1ynlp.nar.xz' (-1 bytes) in 11620 ms' messages.
 2020-07-28 13:27:56 +02:00
John Ericson
8065c6d160 Abstract out topo sorting logic 2020-07-27 20:45:34 +00:00
John Ericson
951415b568 Require ca-derivations everywhere we create a CA derivation 
"create" as in read one in from a serialized form, or build one from
scratch in memory.
 2020-07-27 17:56:36 +00:00
John Ericson
e32a9e124b Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output 2020-07-27 17:50:06 +00:00
John Ericson
7cf978440c Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca 2020-07-27 17:47:40 +00:00
John Ericson
e6115ca004 Merge remote-tracking branch 'upstream/master' into ca-derivation-data-types 2020-07-27 17:41:43 +00:00
John Ericson
1d7d94ceea Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-07-27 16:17:50 +00:00
John Ericson
43f2bd8dc5 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-27 16:13:57 +00:00
John Ericson
387699fe7f Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca 2020-07-27 15:00:28 +00:00
John Ericson
78466bcb2f Merge branch 'optional-derivation-output-storepath' into ca-derivation-data-types 2020-07-27 14:40:08 +00:00
John Ericson
d5bb67cfa4 Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath 2020-07-27 14:29:32 +00:00
John Ericson
699fc89b39 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-27 14:22:45 +00:00
John Ericson
2c7557481b queryDerivationOutputMap no longer assumes all outputs have a mapping 
This assumption is broken by CA derivations. Making a PR now to do the
breaking daemon change as soon as possible (if it is already too late,
we can bump protocol intead).
 2020-07-24 21:14:06 +00:00
John Ericson
58e55c0923 Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output 2020-07-23 19:17:52 +00:00
John Ericson
a2af25c55a Merge remote-tracking branch 'upstream/master' into misc-ca 2020-07-23 19:14:17 +00:00
John Ericson
c36916eca0 Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output 2020-07-23 15:06:56 +00:00
John Ericson
6fcd9966bf Merge remote-tracking branch 'upstream/master' into misc-ca 2020-07-23 15:02:29 +00:00
John Ericson
66a2067288 Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors 2020-07-23 14:56:35 +00:00
John Ericson
2274f63453 Get rid of basicDerivation::findOutput 
It's a tiny function which is:

 - hardly worth abstrating over, and also only used once.

 - doesn't work once we get CA drvs

I rewrote the one callsite to be forwards compatable with CA
derivations, and also potentially more performant: instead of reading in
the derivation it can ust consult the SQLite DB in the common case.
 2020-07-23 14:39:02 +00:00
John Ericson
9423f64ee2 Parse CA derivations using new output variants 
We no longer need `ParsedDerivation` because everything libstore needs
to know about is in the `BasicDerivation` proper.
 2020-07-22 23:59:25 +00:00
John Ericson
c56356bacc Separate concerns in scanForReferences with TeeSink 
This also will make it easier to use a `HashModuloSink` instead for CA
derivations.
 2020-07-22 22:13:44 +00:00
Eelco Dolstra
ff314f186e
Merge pull request #3736 from obsidiansystems/allow-relative-paths-in-store-option 
Allow relative paths in --store option
 2020-07-21 19:43:07 +02:00
Eelco Dolstra
0951330680
Merge pull request #3714 from obsidiansystems/add-body-to-network-errors 
Add response body to network errors
 2020-07-21 18:09:42 +02:00
John Ericson
02639716ea Merge branch 'allow-relative-paths-in-store-option' into remove-storetype-delegate-regStore 2020-07-21 15:55:27 +00:00
John Ericson
6cce32c8e8 Change logic for deciding what is a relative path for the local store 
The was Eelco's prefered logic, and it looks good to me!
 2020-07-21 15:39:47 +00:00
Carlo Nucera
922a845ffc Update chunkSize to the suggested value 
This was a suggested course of action in a review in one of our earlier
commits, https://github.com/NixOS/nix/pull/3801#discussion_r457557079
 2020-07-21 10:24:19 -04:00
John Ericson
54e507a7aa Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors 2020-07-21 14:17:59 +00:00
John Ericson
5ce95b9529
Update src/libstore/build.cc 2020-07-21 09:47:40 -04:00
John Ericson
cdb3f39b64 Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors 2020-07-21 13:43:20 +00:00
John Ericson
5055c595bd Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into ca-derivation-data-types 2020-07-21 01:20:53 +00:00
John Ericson
5a37413796 Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into from-dump-stream 2020-07-21 01:20:14 +00:00
John Ericson
bf0dca7790 Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into misc-ca 2020-07-21 01:18:33 +00:00
Carlo Nucera
9aae179f34 Correct bug, thoroughly document addToStoreSlow 2020-07-20 20:18:12 -04:00
Carlo Nucera
0ca9744694 Use heuristics to decide when to show the response 
Due to https://github.com/NixOS/nix/issues/3841 we don't know how print
different messages for different verbosity levels.
 2020-07-20 15:57:58 -04:00
Carlo Nucera
6357b1b0fb Add another Unimplemented case 2020-07-20 14:17:25 -04:00
Carlo Nucera
007bf6d17a Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca 2020-07-20 14:14:39 -04:00
Carlo Nucera
362ae93851 Add UnimplementedError to ease grepping for these 2020-07-20 14:13:37 -04:00
John Ericson
bf61871271 parser.hh -> split.hh 2020-07-20 17:42:34 +00:00
John Ericson
eab945f591 Merge remote-tracking branch 'upstream/master' into from-dump-stream 2020-07-20 17:29:23 +00:00
Eelco Dolstra
1c5f8bbfb5
Merge pull request #3822 from obsidiansystems/dump-thrice-fixme 
Optimize `addToStoreSlow` and remove `TeeParseSink`
 2020-07-20 18:55:05 +02:00
John Ericson
ac2fc7ba1f
Apply suggestions from code review 
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-07-20 11:29:46 -04:00
Carlo Nucera
0aa79dcc6f Remove StoreType abstraction and delegate regStore 
to each Store implementation. The generic regStore implementation will
only be for the ambiguous shorthands, like "" and "auto".

This also could get us close to simplifying the daemon command.
 2020-07-17 17:24:02 -04:00
John Ericson
6756cecfcf Add DerivationType::CAFloating 2020-07-17 19:55:41 +00:00
Carlo Nucera
fefd6c9e5f Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option 2020-07-17 14:34:20 -04:00
Carlo Nucera
bbc633c98c Revert "Don't anticipate CA but not fixed outputs for now" 
This reverts commit 3a9e4c3262.
 2020-07-17 13:10:32 -04:00
Carlo Nucera
205dcd140d Revert "Don't anticipate multiple CA outputs for now" 
This reverts commit 74b251b2f3.
 2020-07-17 12:43:46 -04:00
Carlo Nucera
1feb8981df Revert "Don't anticipate hash algo without hash in derivation for now" 
This reverts commit 3804e3df9b.
 2020-07-17 12:05:54 -04:00
Carlo Nucera
487c5751c6 Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca 2020-07-17 11:32:59 -04:00
Carlo Nucera
5cb840541b Merge branch 'multi-output-hashDerivationModulo' of github.com:Ericson2314/nix into misc-ca 2020-07-17 10:28:33 -04:00
Carlo Nucera
745a03cef5 Merge branch 'optional-derivation-output-storepath' of github.com:obsidiansystems/nix into ca-derivation-data-types 2020-07-16 13:36:01 -04:00
Carlo Nucera
048e916f64 Merge branch 'master' of github.com:NixOS/nix into optional-derivation-output-storepath 2020-07-16 13:32:28 -04:00
John Ericson
cc0d77f8c9 Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-07-16 17:28:52 +00:00
Carlo Nucera
e3a3406db8 Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors 2020-07-16 12:04:10 -04:00
John Ericson
5ea817dace Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-16 14:58:53 +00:00
John Ericson
3dcca18c30 Fix bug in TeeSource 
We use this to simplify `LocalStore::addToStoreFromDump`.

Also, hope I fixed build error with old clang (used in Darwin CI).
 2020-07-16 13:39:27 +00:00
Carlo Nucera
e3b394b6e8 Small namespace fix 2020-07-16 09:36:02 -04:00
Eelco Dolstra
5517eee17e Generations API cleanup 2020-07-16 15:18:21 +02:00
Eelco Dolstra
3f264916db Merge remote-tracking branch 'origin/flakes' 2020-07-16 14:07:32 +02:00
John Ericson
68dfb8c6ae Optimize addToStoreSlow and remove TeeParseSink 2020-07-16 05:09:41 +00:00
John Ericson
5602637d9e Revert "LocalStore::addToStoreFromDump copy in chunks" 
This reverts commit 592851fb67. We don't
need this extra feature anymore
 2020-07-15 23:37:49 +00:00
John Ericson
bc109648c4 Get rid of LocalStore::addToStoreCommon 
I got it to just become `LocalStore::addToStoreFromDump`, cleanly taking
a store and then doing nothing too fancy with it.

`LocalStore::addToStore(...Path...)` is now just a simple wrapper with a
bare-bones sinkToSource of the right dump command.
 2020-07-15 23:14:30 +00:00
Carlo Nucera
455bdee205 Merge branch 'master' of github.com:NixOS/nix into derivation-header-include-order 2020-07-15 17:58:30 -04:00
Carlo Nucera
d090562348 Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type 2020-07-15 17:21:01 -04:00
John Ericson
aff69e229c Merge remote-tracking branch 'obsidian/fix-add-to-star-narhash' into from-dump-stream 2020-07-15 21:05:29 +00:00
Carlo Nucera
2d2a10e79a Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors 2020-07-15 13:06:08 -04:00
Eelco Dolstra
cff2157185 Revert "LocalStore::addToStore(srcPath): Handle the flat case" 
This reverts commit a2c27022e9. See
addToStoreSlow(), we don't need to handle this case efficiently
anymore. In fact, we can almost remove the method/hashAlgo arguments
since the non-recursive and/or non-SHA256 are almost not used anymore.
 2020-07-15 12:49:03 +02:00
John Ericson
650c2c6558 Rename variable nar -> dump according to TODO 2020-07-14 21:28:50 +00:00
John Ericson
8173e7bfef Fix localhost::addToStore(...Path...) 
We were calculating the nar hash wrong when the file ingestion method
was flat. I don't think there's anything we can do in that case but dump
the file again, so that's what I do.

As an optomization, we again could reuse the original dump for just the
recursive and non-sha256 case, but I rather do that after this fix, and
after my other PRs which deduplicate this code.
 2020-07-14 21:12:11 +00:00
John Ericson
592851fb67 LocalStore::addToStoreFromDump copy in chunks 
Rather than copying byte-by-byte, we let the coroutine know how much
data we would like it to send back to us.
 2020-07-14 13:56:36 +00:00
John Ericson
9de96ef7d4 Dedup LocalStore::addToStore* 
The downsides is that the coroutine has byte-by-byte loop transfer. Will
fix that next.
 2020-07-14 13:56:36 +00:00
John Ericson
c86fc3a965 Crudely make addToStoreFromDump take Source not string 
I just as little beyond the type as possible, so the implementation
changes this enables can be reviewed separately.
 2020-07-14 13:56:36 +00:00
John Ericson
9ec10046e0 Narrow scope of temporary value 2020-07-14 13:54:29 +00:00
Eelco Dolstra
832e111494 Merge remote-tracking branch 'origin/master' into flakes 2020-07-14 13:56:18 +02:00
Eelco Dolstra
926c3a6664 Doh 2020-07-14 11:55:54 +02:00
Eelco Dolstra
43b8e96d30 Fix 'nix verify --all' on a binary cache (cached case) 2020-07-13 20:17:00 +02:00
Eelco Dolstra
7c2fef0a81 Make 'nix copy' to s3:// binary caches run in constant memory 2020-07-13 20:07:19 +02:00
Eelco Dolstra
493961b689 Remove istringstream_nocopy 2020-07-13 18:31:19 +02:00
Eelco Dolstra
545bb2ed03 Remove 'accessor' from addToStore() 
This is only used by hydra-queue-runner and it's better to implement
it there.
 2020-07-13 18:31:19 +02:00
Eelco Dolstra
0a9da00a10 NarAccessor: Run in constant memory 2020-07-13 17:30:42 +02:00
Eelco Dolstra
fc84c358d9 Make 'nix copy' to file:// binary caches run in constant memory 2020-07-13 16:28:45 +02:00
Eelco Dolstra
400f1a9b59 Store::pathInfoToJSON(): Use consistent format for downloadHash 2020-07-13 16:26:09 +02:00
Eelco Dolstra
c0dd05131e toStorePath(): Return a StorePath and the suffix 2020-07-13 16:25:48 +02:00
Eelco Dolstra
1d01ae816b Fix 'nix verify --all' on a binary cache and add a test 2020-07-13 14:35:01 +02:00
John Ericson
c466cb2091 Merge branch 'hash-always-has-type' into better-ca-parse-errors 2020-07-13 03:02:09 +00:00
John Ericson
4415765385 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-13 03:01:44 +00:00
John Ericson
230c9b4329 Change types to prepare the way for CA derivations 
We've added the variant to `DerivationOutput` to support them, but made
`DerivationOutput::path` partial to avoid actually implementing them.

With this chage, we can all collaborate on "just" removing
`DerivationOutput::path` calls to implement CA derivations.
 2020-07-12 23:53:33 +00:00
John Ericson
fedfc913ad Use more std::visit to prepare for new variant 
N.B. not using `std::visit` for fetchurl because there is no attempt to
handle all the cases (e.g. no `else`) and lambda complicates early
return.
 2020-07-12 23:51:40 +00:00
John Ericson
ffc18583b1 Move C++17 "pattern matching" boilerplat to utils.hh 2020-07-12 22:15:14 +00:00
John Ericson
503b425690 DerivationOutputExtensional -> DerivationOutputInputAddressed 
Thanks @regnat for the great name.
 2020-07-12 15:56:20 +00:00
John Ericson
a8d4707107 Undo erroneous indentation change 2020-07-12 15:54:12 +00:00
John Ericson
18152406ce String .drv suffix to create derivation name 2020-07-12 15:40:14 +00:00
John Ericson
5d0b75e5b6 std::string_view for new derivation name parameters 2020-07-12 15:02:36 +00:00
John Ericson
abea26a968 BasicDerivation::findOutput cannot return reference anymore 2020-07-12 01:57:06 +00:00
John Ericson
442d43178b Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath 2020-07-11 02:27:45 +00:00
Matthew Bauer
acb74d4d94 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-07-10 18:13:45 -04:00
Eelco Dolstra
8efa23bb99 Avoid a redundant hash 2020-07-10 15:56:24 +02:00
Eelco Dolstra
5dff49f661 Factor out commonality between nix-prefetch-url and nix-store --add-fixed 2020-07-10 13:21:37 +02:00
John Ericson
323d51a7a0 Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors 2020-07-09 19:07:06 +00:00
Matthew Bauer
a7884970c5 Fix DerivationOutputExtensional name 2020-07-09 11:37:18 -04:00
Eelco Dolstra
a2c27022e9 LocalStore::addToStore(srcPath): Handle the flat case 
This helps nix-prefetch-url when using a local store.
 2020-07-09 15:54:32 +02:00
Matthew Bauer
06a4e15478 Fix build.cc on linux 2020-07-08 19:27:51 -04:00
Matthew Bauer
8e0d0689be Only store hash of fixed derivation output 
we don’t need a full storepath for a fixedoutput derivation. So just
putting the ingestion method + the hash is sufficient.
 2020-07-08 19:11:39 -04:00
Eelco Dolstra
b981e5aacf Cleanup 2020-07-08 22:07:21 +02:00
Eelco Dolstra
34f25124ba Make LocalStore::addToStore(srcPath) run in constant memory 
This reduces memory consumption of

  nix-instantiate \
    -E 'with import <nixpkgs> {}; runCommand "foo" { src = ./blender; } "echo foo"' \
    --option nar-buffer-size 10000

(where ./blender is a 1.1 GiB tree) from 1716 to 36 MiB, while still
ensuring that we don't do any write I/O for small source paths (up to
'nar-buffer-size' bytes). The downside is that large paths are now
always written to a temporary location in the store, even if they
produce an already valid store path. Thus, adding large paths might be
slower and run out of disk space. ¯\_(ツ)_/¯ Of course, you can always
restore the old behaviour by setting 'nar-buffer-size' to a very high
value.
 2020-07-08 22:07:21 +02:00
Matthew Bauer
af95a7c16b Add name to BasicDerivation 
We always have a name for BasicDerivation, since we have a derivation
store path that has a name.
 2020-07-08 15:38:01 -04:00
Eelco Dolstra
9223603908 Merge remote-tracking branch 'origin/master' into flakes 2020-07-08 15:55:19 +02:00
Eelco Dolstra
16ec7785ca Fix 'got unknown message type 1 from Nix daemon' 
Example:

  $ nix-build -E 'with import <nixpkgs> {}; runCommand "foo" { x = runCommand "bar" {} "exit 1"; } "echo foo; exit 1"'
  warning: unknown setting 'auto-allocate-uids'
  these 2 derivations will be built:
    /nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv
    /nix/store/k4fsvrjl7cp2xpz7927iv7g0dqj1zyhs-foo.drv
  warning: unknown setting 'auto-allocate-uids'
  building '/nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv'...
  error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------- nix-daemon
  builder for '/nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv' failed with exit code 1
  error: --- Error ------------------------------------------------------------------------------------------------------------------------------------------------------------------ nix-build
  got unknown message type 1 from Nix daemon
 2020-07-08 15:53:14 +02:00
Eelco Dolstra
1ab9da9154 Merge remote-tracking branch 'origin/master' into flakes 2020-07-07 14:38:57 +02:00
Eelco Dolstra
7c9ece5dca exportReferencesGraph: Fix support for non-top-level store paths 
Fixes #3471.
 2020-07-07 14:25:43 +02:00
Carlo Nucera
0f05a36e20 Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors 2020-07-06 16:57:09 -04:00
Eelco Dolstra
54712aaf8a Merge remote-tracking branch 'origin/master' into flakes 2020-07-06 16:40:10 +02:00
John Ericson
a38ab99d57 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-07-05 21:49:01 +00:00
John Ericson
465daa9396 Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors 2020-07-03 17:08:39 +00:00
John Ericson
3134db1a83 Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-07-03 14:12:38 +00:00
John Ericson
dbffd309fe Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type 2020-07-03 14:11:38 +00:00
Ben Burdette
b29a4ea1dc Merge branch 'master' into add-trace 2020-07-03 07:57:36 -06:00
regnat
223fbe644a Shorten the path to the test root 
Fix a socket length failure on the OSX builders
 2020-07-03 09:20:01 +02:00
John Ericson
13796be78d Have splitPrefix and splitPrefixTo parser helpers 2020-07-02 23:18:22 +00:00
John Ericson
2f93d9f2ba Merge branch 'hash-always-has-type' into HEAD 2020-07-02 21:47:51 +00:00
John Ericson
1be279af26 Fix Narinfo corruption detection bug 
The aim of this check was just to ensure each key occurs once.
 2020-07-02 21:46:10 +00:00
Carlo Nucera
1fc835aa22 Tighten parsing for drv files and pathinfo 2020-07-02 11:57:21 -04:00
Matthew Bauer
fc2ab42e86 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-07-02 11:14:04 -04:00
Matthew Bauer
d2e8b9ff0e Store subPath in SubstitutionGoal 2020-07-02 11:12:05 -04:00
Ben Burdette
bf2788e4c1 move showTrace to new loggerSettings 2020-07-02 09:04:31 -06:00
Matthew Bauer
1f9cb06db2 Try next when no ca exists and have different store dirs 2020-07-02 10:59:24 -04:00
Carlo Nucera
263ccdd489 Rename two hash constructors to proper functions 2020-07-01 18:34:18 -04:00
Eelco Dolstra
38ccf2e241 Cleanup 2020-07-01 15:31:34 +02:00
Eelco Dolstra
86a4aba6c4 Merge branch 'remote-query-outputs' of https://github.com/tweag/nix 2020-07-01 15:10:29 +02:00
Ben Burdette
9159dfe3d8 comments and cleanup 2020-06-30 16:31:55 -06:00
Matthew Bauer
09833126f3 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-30 16:35:08 -04:00
Ben Burdette
ddb81ca126 Merge branch 'master' into add-trace 2020-06-30 12:21:45 -06:00
Carlo Nucera
77b51f4598 Factor the prefix splitting in content-address 2020-06-30 11:57:46 -04:00
John Ericson
a83566e5bc Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-06-29 18:40:34 +00:00
John Ericson
987a4a0be9 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-06-29 18:03:21 +00:00
Ben Burdette
8f81fae116 showTrace flag in loggers 2020-06-29 10:20:51 -06:00
Eelco Dolstra
58bc3b6578
Merge pull request #3729 from obsidiansystems/simpler-hased-mirror 
hashed-mirrors: Use parsed derivation output rather than reconstructing it
 2020-06-29 14:04:12 +02:00
Eelco Dolstra
64232f3ea6
Merge pull request #3749 from rodarima/master 
Fall back to copyPath if link fails with EPERM
 2020-06-29 13:31:24 +02:00
Carlo Nucera
278a679782 Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option 2020-06-26 10:40:13 -04:00
Eelco Dolstra
adf2fbbdc2 Merge remote-tracking branch 'origin/master' into flakes 2020-06-26 08:46:46 +02:00
John Ericson
d92d4f85a5 Move ValidPathInfo to its own header 2020-06-25 17:48:49 +00:00
Eelco Dolstra
de2641ae99 Fix empty std::optional dereference in writeDerivation() 
https://hydra.nixos.org/build/123017579
 2020-06-25 15:50:30 +02:00
Rodrigo
3a642187c3 Fall back to copyPath if link fails with EPERM 
BeeGFS doesn't allow hard-links and returns EPERM, so we fall back
to copyPath. See https://github.com/NixOS/nix/issues/3748
 2020-06-25 12:03:26 +02:00
regnat
d38f860c3e Add a way to get all the outputs of a derivation with their label 
Generalize `queryDerivationOutputNames` and `queryDerivationOutputs` by
adding a `queryDerivationOutputMap` that returns the map
`outputName=>outputPath`

(not that this is not equivalent to merging the results of
`queryDerivationOutputs` and `queryDerivationOutputNames` as sets don't
preserve the order, so we would end up with an incorrect mapping).

squash! Add a way to get all the outputs of a derivation with their label

Rename StorePathMap to OutputPathMap
 2020-06-24 20:38:40 +02:00
John Ericson
d9fb9e9f26 Merge remote-tracking branch 'upstream/master' into ca-drv 2020-06-24 15:41:29 +00:00
John Ericson
98e5d1af03 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-06-23 17:03:37 +00:00
Carlo Nucera
e197bc6229 Enable the --store option to take relative paths 
In nix commands which accept --store options, we can now specify a
relative path, which will be canonicalized.
 2020-06-23 11:13:18 -04:00
John Ericson
55d4bd6e0e Improve content address parsing 
- Ensure hash is in form <algo>-<prefix> and not SRI.

 - Better errors if something goes wrong

 - string_view for no coppying
 2020-06-22 18:08:27 +00:00
Matthew Bauer
66a62b3189 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-22 13:08:11 -04:00
John Ericson
f4a5913125 hashed-mirrors: Use parsed derivation output rather than reconstructing it 
Now the derivation outputs are parsed up front, we can avoid a reparse
by doing it. Also, this just feels a bit better as the `output*` env
vars are more of a `libnixexpr` interface than `libnixstore` interface:
ultimately, it's the derivation outputs that decide whether the
derivation is fixed-output.

Yes, hashed mirrors might go away with #3689, but this bit of code would
be moved rather than deleted, so it's worth doing a cleanup anyways I
think.
 2020-06-22 15:17:20 +00:00
John Ericson
93bbe6e8ab Merge remote-tracking branch 'upstream/master' into ca-drv 2020-06-21 21:11:58 +00:00
John Ericson
3804e3df9b Don't anticipate hash algo without hash in derivation for now 
When we merge with master, the new lack of string types make this case
impossible (after parsing). Later, when we actually implemenent
CA-derivations, we'll change the types to allow that.
 2020-06-21 21:05:37 +00:00
John Ericson
8313f0e939 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-06-21 20:39:10 +00:00
John Ericson
bf9f040112 Tweak declaration 
I think this is clearer
 2020-06-21 16:51:39 +00:00
John Ericson
fdeabf7160 Merge remote-tracking branch 'upstream/master' into multi-output-hashDerivationModulo 2020-06-21 16:43:17 +00:00
Matthew Bauer
f2a6cee334 Update worker protocol to support sending storepath maps 
We need to also send the ca to daemon in addition to the path.
 2020-06-19 18:06:19 -04:00
John Ericson
507aa48739 WIP: Make Hash always store a valid hash type 2020-06-19 21:48:57 +00:00
Matthew Bauer
e8e1f5282f Replace error message in getDerivationCA 2020-06-19 15:19:09 -04:00
Matthew Bauer
0c9c1b8826 Return map of StorePaths in copyPaths 
This allows the caller to know what values were actually added to the
store.
 2020-06-19 14:48:57 -04:00
Matthew Bauer
7e11cf3399 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-19 14:40:49 -04:00
Matthew Bauer
2796b01e86 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-19 14:22:06 -04:00
John Ericson
68294746ae Merge remote-tracking branch 'upstream/master' into no-hash-type-unknown 2020-06-19 17:53:34 +00:00
John Ericson
911fc88bcb More designated initializers 2020-06-19 17:42:56 +00:00
John Ericson
2f0e395c99 Merge remote-tracking branch 'me/no-stringly-typed-derivation-output' into validPathInfo-ca-proper-datatype 2020-06-19 15:26:59 +00:00
John Ericson
fb39a5e00c Remove unneeded constructor for DerivationOutputHash 2020-06-19 15:11:11 +00:00
John Ericson
01dc8b0bab Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output 2020-06-19 14:59:05 +00:00
John Ericson
145d88cb2a Use designated initializers for DerivationOutputHash 2020-06-19 14:58:30 +00:00
John Ericson
237d88c97e FileSystemHash -> DerivationOutputHash 2020-06-19 14:47:10 +00:00
John Ericson
3fc58a9638 Remove some Base:: that crept in 2020-06-19 00:24:47 +00:00
John Ericson
3f8dcfe3fd Merge branch 'validPathInfo-temp' into validPathInfo-ca-proper-datatype 2020-06-18 23:01:58 +00:00
John Ericson
669c3992e8 Merge branch 'no-hash-type-unknown' into validPathInfo-temp 2020-06-18 22:33:07 +00:00
John Ericson
15abb2aa2b Revert the enum struct change 
Not a regular git revert as there have been many merges and things.
 2020-06-18 22:11:26 +00:00
John Ericson
bbbf3602a3 Merge branch 'enum-class' into no-hash-type-unknown 2020-06-18 22:11:19 +00:00
John Ericson
40526fbea5 Merge remote-tracking branch 'upstream/master' into enum-class 2020-06-18 21:38:15 +00:00
John Ericson
1b23fe4afb Fix bugs 
- Bad dynamic cast target ...classic

 - std::shared_ptr need explicit deref
 2020-06-18 19:03:10 +00:00
John Ericson
fb432b0bd2 Merge remote-tracking branch 'obsidian/fix-url-format' into add-body-to-network-errors 2020-06-18 18:08:02 +00:00
John Ericson
639e20dc3e Prevent '%' in URL from causing crashes 
We have a larger problem that passsing computed strings to the first
variable argument of many exception constructors is unsafe because that
first variable argument is interpreted not as a plain string, but format
string, and if it contains '%' boost::format will abort, since there are
no arguments to the format string.

In this particular instance '%' was used as part of an escape code in a
URL, which, when the download failed, caused Nix to abort displaying the
`FileTransferError`.
 2020-06-18 17:54:16 +00:00
John Ericson
74b219ef6e Adjust FileTransferError message to use opt response 2020-06-18 14:48:45 +00:00
regnat
4fef2ba7e4 Rename content-addressed-paths into ca-derivations 
See <https://github.com/NixOS/nix/pull/3710#issuecomment-645480333>
 2020-06-18 09:25:55 +02:00
Carlo Nucera
004570a377 Add HTTP responses to FileTransferErrors 2020-06-17 22:34:31 +00:00
Carlo Nucera
a835c740ca Replace TransferItem::status with a local variable 
Everywhere seems to use `getHTTPStatus` now.
 2020-06-17 22:16:24 +00:00
Carlo Nucera
fb05a6adcf Eliminate old TeeSink abstraction 
This was introduced in fa125b9b28, and
then "reverted" in 1cf4801108, except that
revert left the struct around doing nothing useful.

We're removing it all the way now because we want to make a new
`TeeSink` complementing the already-exiting `TeeSource`, that is
actually a completely different concept as far as the class hierarchy is
concerned.
 2020-06-17 22:07:22 +00:00
Eelco Dolstra
2b8f33bf5f
Merge pull request #3713 from matthewbauer/cleanup-warnings 
Cleanup class StorePath warning
 2020-06-17 21:19:30 +02:00
Matthew Bauer
5e631e3304 Add StorePathCAMap for querySubstitutablePathInfos 
I’m not 100% sure this is wanted since it kind of makes everything
have to know about ca even if they don’t really want to. But it also
make things easier in dealing with looking up ca.
 2020-06-17 15:03:05 -04:00
Matthew Bauer
be50de1142 Make sure references are empty for store path replacing 
also copy info2 instead of casting
 2020-06-17 14:15:08 -04:00
Matthew Bauer
8974755d19 Add assert for replaced storePath 2020-06-17 14:04:46 -04:00
Eelco Dolstra
3078404e35
Merge pull request #3712 from obsidiansystems/make-http-successful-states-coherent 
Make successful states coherent
 2020-06-17 19:37:46 +02:00
Matthew Bauer
7bd88cc1dc Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-17 13:32:29 -04:00
Matthew Bauer
22d7d36703 Remove unused narInfoFile in binary-cache-store 2020-06-17 13:27:19 -04:00
Matthew Bauer
f767bedfac Replace struct StorePath with class StorePath 
also a similar case with struct Goal
 2020-06-17 13:26:37 -04:00
Carlo Nucera
4930cb48a2 Include review comments 2020-06-17 12:58:59 -04:00
Matthew Bauer
6438ba1e99 Update strings from review comment 2020-06-17 12:31:01 -04:00
Eelco Dolstra
2f51cd8dc9
Merge pull request #3710 from tweag/reserve_ca_derivations 
Reserve the `__contentAddressed` derivation parameter
 2020-06-17 18:28:26 +02:00
Eelco Dolstra
4d5169bdd5
Merge pull request #3707 from p01arst0rm/outdated-function-fix 
replaced uncaught_exception with uncaught_exceptions
 2020-06-17 18:26:01 +02:00
regnat
480b54e1c6 fixup! Reserve the __contentAddressed derivation parameter 2020-06-17 17:37:04 +02:00
Carlo Nucera
079c6e87de Make successful states coherent 
The successful states used in these two places in the code were slightly
different. Should they be the same list?
 2020-06-17 11:16:16 -04:00
regnat
56d75bf4fc Reserve the __contentAddressed derivation parameter 
Not implementing anything here, just throwing an error if a derivation
sets `__contentAddressed = true` without
`--experimental-features content-addressed-paths`
(and also with it as there's nothing implemented yet)
 2020-06-17 15:41:17 +02:00
Eelco Dolstra
fdff09e57c Fix coverage build 2020-06-17 15:18:10 +02:00
Eelco Dolstra
ccbea8255c
Merge pull request #3657 from obsidiansystems/sligthly-improve-store-path-documentation 
Clarify the description of StorePath inputs
 2020-06-17 14:54:37 +02:00
Eelco Dolstra
1524752c17 Merge remote-tracking branch 'origin/master' into flakes 2020-06-17 10:26:52 +02:00
John Ericson
517f5980e2 Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output 2020-06-17 04:58:43 +00:00
John Ericson
18493fd9c4 Move some Store functions from derivations.cc to store-api.cc 
This further continues with the dependency inverstion. Also I just went
ahead and exposed `parseDerivation`: it seems like the more proper
building block, and not a bad thing to expose if we are trying to be
less wedded to drv files on disk anywas.
 2020-06-17 04:30:16 +00:00
John Ericson
7130f0a3a6 Don't need abstract struct Derivation in local-store 2020-06-17 04:05:38 +00:00
John Ericson
21ef342172 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-06-17 03:52:01 +00:00
John Ericson
02928f76fd Merge remote-tracking branch 'upstream/master' into multi-output-hashDerivationModulo 2020-06-17 03:51:02 +00:00
John Ericson
49308ef039 Merge remote-tracking branch 'upstream/master' into ca-drv 2020-06-17 03:39:39 +00:00
p01arst0rm
c9d06558b6 replaced uncaught_exception with uncaught_exceptions 2020-06-17 03:15:47 +01:00
Eelco Dolstra
29542865ce Remove StorePath::clone() and related functions 2020-06-16 22:20:18 +02:00
Eelco Dolstra
759947bf72 StorePath: Rewrite in C++ 
On nix-env -qa -f '<nixpkgs>', this reduces maximum RSS by 20970 KiB
and runtime by 0.8%. This is mostly because we're not parsing the hash
part as a hash anymore (just validating that it consists of base-32
characters).

Also, replace storePathToHash() by StorePath::hashPart().
 2020-06-16 14:28:41 +02:00
Eelco Dolstra
72e17290d4 Fix FTP support 
Fixes #3618.
 2020-06-16 11:53:04 +02:00
Eelco Dolstra
a588b6b19d Print only one error message if a build fails 
E.g. instead of

  error: --- BuildError ----------------------------------------------- nix
  builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1
  error: --- Error ---------------------------------------------------- nix
  build of '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed

we now get

  error: --- Error ---------------------------------------------------- nix
  builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1
 2020-06-15 19:35:31 +02:00