Eelco Dolstra
b8eea7e81a
Remove putBytes
...
istream->tellg() returns -1 so we can't get the number of bytes
written.
Fixes 'uploaded 's3://nix-cache/nar/00819r9lp5kajr6baxfw5dhhc0cx8ndxaz43qmd2f0gn1hk1ynlp.nar.xz' (-1 bytes) in 11620 ms' messages.
2020-07-28 13:27:56 +02:00
John Ericson
43f2bd8dc5
Merge remote-tracking branch 'upstream/master' into hash-always-has-type
2020-07-27 16:13:57 +00:00
John Ericson
d5bb67cfa4
Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath
2020-07-27 14:29:32 +00:00
John Ericson
699fc89b39
Merge remote-tracking branch 'upstream/master' into hash-always-has-type
2020-07-27 14:22:45 +00:00
John Ericson
2274f63453
Get rid of basicDerivation::findOutput
...
It's a tiny function which is:
- hardly worth abstrating over, and also only used once.
- doesn't work once we get CA drvs
I rewrote the one callsite to be forwards compatable with CA
derivations, and also potentially more performant: instead of reading in
the derivation it can ust consult the SQLite DB in the common case.
2020-07-23 14:39:02 +00:00
John Ericson
c56356bacc
Separate concerns in scanForReferences
with TeeSink
...
This also will make it easier to use a `HashModuloSink` instead for CA
derivations.
2020-07-22 22:13:44 +00:00
Eelco Dolstra
ff314f186e
Merge pull request #3736 from obsidiansystems/allow-relative-paths-in-store-option
...
Allow relative paths in --store option
2020-07-21 19:43:07 +02:00
Eelco Dolstra
0951330680
Merge pull request #3714 from obsidiansystems/add-body-to-network-errors
...
Add response body to network errors
2020-07-21 18:09:42 +02:00
John Ericson
6cce32c8e8
Change logic for deciding what is a relative path for the local store
...
The was Eelco's prefered logic, and it looks good to me!
2020-07-21 15:39:47 +00:00
Carlo Nucera
922a845ffc
Update chunkSize to the suggested value
...
This was a suggested course of action in a review in one of our earlier
commits, https://github.com/NixOS/nix/pull/3801#discussion_r457557079
2020-07-21 10:24:19 -04:00
John Ericson
54e507a7aa
Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors
2020-07-21 14:17:59 +00:00
John Ericson
5a37413796
Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into from-dump-stream
2020-07-21 01:20:14 +00:00
Carlo Nucera
9aae179f34
Correct bug, thoroughly document addToStoreSlow
2020-07-20 20:18:12 -04:00
Carlo Nucera
0ca9744694
Use heuristics to decide when to show the response
...
Due to https://github.com/NixOS/nix/issues/3841 we don't know how print
different messages for different verbosity levels.
2020-07-20 15:57:58 -04:00
John Ericson
eab945f591
Merge remote-tracking branch 'upstream/master' into from-dump-stream
2020-07-20 17:29:23 +00:00
Eelco Dolstra
1c5f8bbfb5
Merge pull request #3822 from obsidiansystems/dump-thrice-fixme
...
Optimize `addToStoreSlow` and remove `TeeParseSink`
2020-07-20 18:55:05 +02:00
John Ericson
ac2fc7ba1f
Apply suggestions from code review
...
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-07-20 11:29:46 -04:00
Carlo Nucera
fefd6c9e5f
Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option
2020-07-17 14:34:20 -04:00
Carlo Nucera
048e916f64
Merge branch 'master' of github.com:NixOS/nix into optional-derivation-output-storepath
2020-07-16 13:32:28 -04:00
Carlo Nucera
e3a3406db8
Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors
2020-07-16 12:04:10 -04:00
John Ericson
5ea817dace
Merge remote-tracking branch 'upstream/master' into hash-always-has-type
2020-07-16 14:58:53 +00:00
John Ericson
3dcca18c30
Fix bug in TeeSource
...
We use this to simplify `LocalStore::addToStoreFromDump`.
Also, hope I fixed build error with old clang (used in Darwin CI).
2020-07-16 13:39:27 +00:00
Eelco Dolstra
5517eee17e
Generations API cleanup
2020-07-16 15:18:21 +02:00
Eelco Dolstra
3f264916db
Merge remote-tracking branch 'origin/flakes'
2020-07-16 14:07:32 +02:00
John Ericson
68dfb8c6ae
Optimize addToStoreSlow
and remove TeeParseSink
2020-07-16 05:09:41 +00:00
John Ericson
5602637d9e
Revert "LocalStore::addToStoreFromDump copy in chunks"
...
This reverts commit 592851fb67
. We don't
need this extra feature anymore
2020-07-15 23:37:49 +00:00
John Ericson
bc109648c4
Get rid of LocalStore::addToStoreCommon
...
I got it to just become `LocalStore::addToStoreFromDump`, cleanly taking
a store and then doing nothing too fancy with it.
`LocalStore::addToStore(...Path...)` is now just a simple wrapper with a
bare-bones sinkToSource of the right dump command.
2020-07-15 23:14:30 +00:00
Carlo Nucera
d090562348
Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type
2020-07-15 17:21:01 -04:00
John Ericson
aff69e229c
Merge remote-tracking branch 'obsidian/fix-add-to-star-narhash' into from-dump-stream
2020-07-15 21:05:29 +00:00
Carlo Nucera
2d2a10e79a
Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors
2020-07-15 13:06:08 -04:00
Eelco Dolstra
cff2157185
Revert "LocalStore::addToStore(srcPath): Handle the flat case"
...
This reverts commit a2c27022e9
. See
addToStoreSlow(), we don't need to handle this case efficiently
anymore. In fact, we can almost remove the method/hashAlgo arguments
since the non-recursive and/or non-SHA256 are almost not used anymore.
2020-07-15 12:49:03 +02:00
John Ericson
650c2c6558
Rename variable nar
-> dump
according to TODO
2020-07-14 21:28:50 +00:00
John Ericson
8173e7bfef
Fix localhost::addToStore(...Path...)
...
We were calculating the nar hash wrong when the file ingestion method
was flat. I don't think there's anything we can do in that case but dump
the file again, so that's what I do.
As an optomization, we again could reuse the original dump for just the
recursive and non-sha256 case, but I rather do that after this fix, and
after my other PRs which deduplicate this code.
2020-07-14 21:12:11 +00:00
John Ericson
592851fb67
LocalStore::addToStoreFromDump copy in chunks
...
Rather than copying byte-by-byte, we let the coroutine know how much
data we would like it to send back to us.
2020-07-14 13:56:36 +00:00
John Ericson
9de96ef7d4
Dedup LocalStore::addToStore*
...
The downsides is that the coroutine has byte-by-byte loop transfer. Will
fix that next.
2020-07-14 13:56:36 +00:00
John Ericson
c86fc3a965
Crudely make addToStoreFromDump
take Source
not string
...
I just as little beyond the type as possible, so the implementation
changes this enables can be reviewed separately.
2020-07-14 13:56:36 +00:00
John Ericson
9ec10046e0
Narrow scope of temporary value
2020-07-14 13:54:29 +00:00
Eelco Dolstra
832e111494
Merge remote-tracking branch 'origin/master' into flakes
2020-07-14 13:56:18 +02:00
Eelco Dolstra
926c3a6664
Doh
2020-07-14 11:55:54 +02:00
Eelco Dolstra
43b8e96d30
Fix 'nix verify --all' on a binary cache (cached case)
2020-07-13 20:17:00 +02:00
Eelco Dolstra
7c2fef0a81
Make 'nix copy' to s3:// binary caches run in constant memory
2020-07-13 20:07:19 +02:00
Eelco Dolstra
493961b689
Remove istringstream_nocopy
2020-07-13 18:31:19 +02:00
Eelco Dolstra
545bb2ed03
Remove 'accessor' from addToStore()
...
This is only used by hydra-queue-runner and it's better to implement
it there.
2020-07-13 18:31:19 +02:00
Eelco Dolstra
0a9da00a10
NarAccessor: Run in constant memory
2020-07-13 17:30:42 +02:00
Eelco Dolstra
fc84c358d9
Make 'nix copy' to file:// binary caches run in constant memory
2020-07-13 16:28:45 +02:00
Eelco Dolstra
400f1a9b59
Store::pathInfoToJSON(): Use consistent format for downloadHash
2020-07-13 16:26:09 +02:00
Eelco Dolstra
c0dd05131e
toStorePath(): Return a StorePath and the suffix
2020-07-13 16:25:48 +02:00
Eelco Dolstra
1d01ae816b
Fix 'nix verify --all' on a binary cache and add a test
2020-07-13 14:35:01 +02:00
John Ericson
4415765385
Merge remote-tracking branch 'upstream/master' into hash-always-has-type
2020-07-13 03:01:44 +00:00
John Ericson
503b425690
DerivationOutputExtensional -> DerivationOutputInputAddressed
...
Thanks @regnat for the great name.
2020-07-12 15:56:20 +00:00