2414 commits

Author	SHA1	Message	Date
John Ericson	0e9e599b00	Merge remote-tracking branch 'upstream/master' into path-info	2020-08-01 00:06:24 +00:00
Matthew Bauer	05ac4db39a	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-07-30 12:38:24 -05:00
Eelco Dolstra	3f6e88a552	unsigned long long -> uint64_t	2020-07-30 13:34:04 +02:00
Eelco Dolstra	ebee2b7852	receiveContents(): unsigned int -> size_t	2020-07-30 13:00:30 +02:00
Eelco Dolstra	f63839bfa4	Cleanup	2020-07-29 18:04:51 +02:00
Eelco Dolstra	4c0077a07d	Fix RemoteStore::addToStore() latency ... Since 6185d25e52, this was very latency-bound since it required a round-trip for every 32 KiB. So for example copying a 514 MiB closure over a virtual ethernet device with a articial delay of just 1 ms took 343s. Now it takes 2.7s. Fixes #3372.	2020-07-29 00:48:39 +02:00
Eelco Dolstra	c159f48a39	Cleanup	2020-07-29 00:24:55 +02:00
Eelco Dolstra	b8eea7e81a	Remove putBytes ... istream->tellg() returns -1 so we can't get the number of bytes written. Fixes 'uploaded 's3://nix-cache/nar/00819r9lp5kajr6baxfw5dhhc0cx8ndxaz43qmd2f0gn1hk1ynlp.nar.xz' (-1 bytes) in 11620 ms' messages.	2020-07-28 13:27:56 +02:00
John Ericson	43f2bd8dc5	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-07-27 16:13:57 +00:00
John Ericson	d5bb67cfa4	Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath	2020-07-27 14:29:32 +00:00
John Ericson	699fc89b39	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-07-27 14:22:45 +00:00
John Ericson	2274f63453	Get rid of basicDerivation::findOutput ... It's a tiny function which is: - hardly worth abstrating over, and also only used once. - doesn't work once we get CA drvs I rewrote the one callsite to be forwards compatable with CA derivations, and also potentially more performant: instead of reading in the derivation it can ust consult the SQLite DB in the common case.	2020-07-23 14:39:02 +00:00
John Ericson	c56356bacc	Separate concerns in scanForReferences with TeeSink ... This also will make it easier to use a `HashModuloSink` instead for CA derivations.	2020-07-22 22:13:44 +00:00
Eelco Dolstra	ff314f186e	Merge pull request #3736 from obsidiansystems/allow-relative-paths-in-store-option ... Allow relative paths in --store option	2020-07-21 19:43:07 +02:00
Eelco Dolstra	0951330680	Merge pull request #3714 from obsidiansystems/add-body-to-network-errors ... Add response body to network errors	2020-07-21 18:09:42 +02:00
John Ericson	6cce32c8e8	Change logic for deciding what is a relative path for the local store ... The was Eelco's prefered logic, and it looks good to me!	2020-07-21 15:39:47 +00:00
Carlo Nucera	922a845ffc	Update chunkSize to the suggested value ... This was a suggested course of action in a review in one of our earlier commits, https://github.com/NixOS/nix/pull/3801#discussion_r457557079	2020-07-21 10:24:19 -04:00
John Ericson	54e507a7aa	Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors	2020-07-21 14:17:59 +00:00
John Ericson	5a37413796	Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into from-dump-stream	2020-07-21 01:20:14 +00:00
Carlo Nucera	9aae179f34	Correct bug, thoroughly document addToStoreSlow	2020-07-20 20:18:12 -04:00
Carlo Nucera	0ca9744694	Use heuristics to decide when to show the response ... Due to https://github.com/NixOS/nix/issues/3841 we don't know how print different messages for different verbosity levels.	2020-07-20 15:57:58 -04:00
John Ericson	eab945f591	Merge remote-tracking branch 'upstream/master' into from-dump-stream	2020-07-20 17:29:23 +00:00
Eelco Dolstra	1c5f8bbfb5	Merge pull request #3822 from obsidiansystems/dump-thrice-fixme ... Optimize `addToStoreSlow` and remove `TeeParseSink`	2020-07-20 18:55:05 +02:00
John Ericson	ac2fc7ba1f	Apply suggestions from code review ... Co-authored-by: Eelco Dolstra <edolstra@gmail.com>	2020-07-20 11:29:46 -04:00
Carlo Nucera	fefd6c9e5f	Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option	2020-07-17 14:34:20 -04:00
Carlo Nucera	048e916f64	Merge branch 'master' of github.com:NixOS/nix into optional-derivation-output-storepath	2020-07-16 13:32:28 -04:00
Carlo Nucera	e3a3406db8	Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors	2020-07-16 12:04:10 -04:00
John Ericson	5ea817dace	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-07-16 14:58:53 +00:00
John Ericson	3dcca18c30	Fix bug in TeeSource ... We use this to simplify `LocalStore::addToStoreFromDump`. Also, hope I fixed build error with old clang (used in Darwin CI).	2020-07-16 13:39:27 +00:00
Eelco Dolstra	5517eee17e	Generations API cleanup	2020-07-16 15:18:21 +02:00
Eelco Dolstra	3f264916db	Merge remote-tracking branch 'origin/flakes'	2020-07-16 14:07:32 +02:00
John Ericson	68dfb8c6ae	Optimize addToStoreSlow and remove TeeParseSink	2020-07-16 05:09:41 +00:00
John Ericson	5602637d9e	Revert "LocalStore::addToStoreFromDump copy in chunks" ... This reverts commit 592851fb67. We don't need this extra feature anymore	2020-07-15 23:37:49 +00:00
John Ericson	bc109648c4	Get rid of LocalStore::addToStoreCommon ... I got it to just become `LocalStore::addToStoreFromDump`, cleanly taking a store and then doing nothing too fancy with it. `LocalStore::addToStore(...Path...)` is now just a simple wrapper with a bare-bones sinkToSource of the right dump command.	2020-07-15 23:14:30 +00:00
Carlo Nucera	d090562348	Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type	2020-07-15 17:21:01 -04:00
John Ericson	aff69e229c	Merge remote-tracking branch 'obsidian/fix-add-to-star-narhash' into from-dump-stream	2020-07-15 21:05:29 +00:00
Carlo Nucera	2d2a10e79a	Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors	2020-07-15 13:06:08 -04:00
Eelco Dolstra	cff2157185	Revert "LocalStore::addToStore(srcPath): Handle the flat case" ... This reverts commit a2c27022e9. See addToStoreSlow(), we don't need to handle this case efficiently anymore. In fact, we can almost remove the method/hashAlgo arguments since the non-recursive and/or non-SHA256 are almost not used anymore.	2020-07-15 12:49:03 +02:00
John Ericson	650c2c6558	Rename variable nar -> dump according to TODO	2020-07-14 21:28:50 +00:00
John Ericson	8173e7bfef	Fix localhost::addToStore(...Path...) ... We were calculating the nar hash wrong when the file ingestion method was flat. I don't think there's anything we can do in that case but dump the file again, so that's what I do. As an optomization, we again could reuse the original dump for just the recursive and non-sha256 case, but I rather do that after this fix, and after my other PRs which deduplicate this code.	2020-07-14 21:12:11 +00:00
John Ericson	592851fb67	LocalStore::addToStoreFromDump copy in chunks ... Rather than copying byte-by-byte, we let the coroutine know how much data we would like it to send back to us.	2020-07-14 13:56:36 +00:00
John Ericson	9de96ef7d4	Dedup LocalStore::addToStore* ... The downsides is that the coroutine has byte-by-byte loop transfer. Will fix that next.	2020-07-14 13:56:36 +00:00
John Ericson	c86fc3a965	Crudely make addToStoreFromDump take Source not string ... I just as little beyond the type as possible, so the implementation changes this enables can be reviewed separately.	2020-07-14 13:56:36 +00:00
John Ericson	9ec10046e0	Narrow scope of temporary value	2020-07-14 13:54:29 +00:00
Eelco Dolstra	832e111494	Merge remote-tracking branch 'origin/master' into flakes	2020-07-14 13:56:18 +02:00
Eelco Dolstra	926c3a6664	Doh	2020-07-14 11:55:54 +02:00
Eelco Dolstra	43b8e96d30	Fix 'nix verify --all' on a binary cache (cached case)	2020-07-13 20:17:00 +02:00
Eelco Dolstra	7c2fef0a81	Make 'nix copy' to s3:// binary caches run in constant memory	2020-07-13 20:07:19 +02:00
Eelco Dolstra	493961b689	Remove istringstream_nocopy	2020-07-13 18:31:19 +02:00
Eelco Dolstra	545bb2ed03	Remove 'accessor' from addToStore() ... This is only used by hydra-queue-runner and it's better to implement it there.	2020-07-13 18:31:19 +02:00