3104 commits

Author	SHA1	Message	Date
Eelco Dolstra	b07167be5a	createGeneration(): Always create an indirect root ... This means profiles outside of /nix/var/nix/profiles don't get garbage-collected. It also means we don't need to scan /nix/var/nix/profiles for GC roots anymore, except for compatibility with previously existing generations.	2020-09-03 11:13:53 +02:00
Eelco Dolstra	b74f5cdd23	createGeneration(): Take a StorePath	2020-09-03 11:06:56 +02:00
Domen Kožar	dd4b56c87f	Allow HTTP binary cache to request absolute uris	2020-09-01 21:35:48 +02:00
John Ericson	ef278d00f9	Merge remote-tracking branch 'upstream/master' into single-ca-drv-build	2020-09-01 18:01:48 +00:00
Eelco Dolstra	84f5cabbea	Merge remote-tracking branch 'origin/master' into markdown	2020-08-31 14:24:26 +02:00
John Ericson	02e0001fc0	Merge branch 'ca-floating-upstream' of github.com:obsidiansystems/nix into ca-floating-upstream	2020-08-28 21:23:09 +00:00
John Ericson	421ed527c7	Update src/libstore/build.cc ... Thanks for catching, @regnat.	2020-08-28 17:22:57 -04:00
John Ericson	48e8828050	Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream	2020-08-28 20:38:25 +00:00
John Ericson	8017fe7487	Merge remote-tracking branch 'upstream/master' into single-ca-drv-build	2020-08-28 19:59:14 +00:00
Eelco Dolstra	eb75282b8d	Merge pull request #3434 from Ericson2314/derivation-header-include-order ... Revise division of labor in deserialization of derivations	2020-08-27 16:39:28 +02:00
Eelco Dolstra	a0f19d9f3a	RemoteStore::addToStore(): Fix race between stderrThread and NAR writer ... As pointed out by @B4dM4n, the call to to.flush() on stderrThread is unsafe because the NAR writer thread is also writing to 'to'. Fixes #3943.	2020-08-27 14:50:51 +02:00
Eelco Dolstra	4bf5faf416	Merge remote-tracking branch 'origin/master' into markdown	2020-08-25 19:47:34 +02:00
John Ericson	8eb73a8724	CA derivations that depend on other CA derivations ... Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>	2020-08-24 19:01:15 +00:00
John Ericson	e0b0e18905	Add constructor for BasicDerivation -> Derivation	2020-08-24 19:01:07 +00:00
John Ericson	59979e7053	Fix bad debug format string	2020-08-24 19:01:07 +00:00
Eelco Dolstra	6a67e57019	Add DummyStore (dummy://) ... DummyStore does not allow building or adding paths. This is useful for evaluation tests when you don't want to initialize a "proper" store.	2020-08-24 18:54:16 +02:00
Eelco Dolstra	2530cc6827	Merge pull request #3950 from obsidiansystems/tabs-spaces ... tabs -> spaces	2020-08-24 10:20:28 +02:00
John Ericson	b0b59fd05a	Merge remote-tracking branch 'obsidian/write-derivation-borrow' into HEAD	2020-08-23 15:11:10 +00:00
John Ericson	35e6288be1	writeDerivation just needs a plain store reference	2020-08-23 15:01:11 +00:00
John Ericson	3a7b330b64	"Downstream placeholders" should not be store paths ... Insead they should be opaque `/<hash>` like the placeholders we already have.	2020-08-21 19:35:35 +00:00
John Ericson	422affe102	tabs -> spaces ... Sorry I let the tab sneak in there in the first place.	2020-08-21 19:19:24 +00:00
John Ericson	c265e0ea82	Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol	2020-08-20 20:24:22 +00:00
John Ericson	27a3f82c0b	Merge remote-tracking branch 'upstream/master' into single-ca-drv-build	2020-08-20 18:28:17 +00:00
John Ericson	45a2f1baab	Rename drv output querying functions, like master ... - `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap` - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap	2020-08-20 18:14:12 +00:00
Eelco Dolstra	4d77513d97	Merge pull request #3859 from obsidiansystems/drv-outputs-map-allow-missing ... `queryDerivationOutputMap` no longer assumes all outputs have a mapping	2020-08-20 16:49:23 +02:00
John Ericson	9a9d834dc7	Rename drv output querying functions ... - `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap` - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap`	2020-08-20 14:12:51 +00:00
Eelco Dolstra	acb99f03f9	Config: Use nlohmann/json	2020-08-20 11:02:16 +02:00
John Ericson	a83694c7a1	Use RemoteStore to open connection for proxying daemon ... Removes duplicate websocket opening code, and also means we should be able to to ssh-ssh-... daemon relays, not just uds-uds-... ones.	2020-08-19 19:34:47 +00:00
Eelco Dolstra	c8fa39324a	Generate the nix.conf docs from the source code ... This means we don't have two (divergent) sets of option descriptions anymore.	2020-08-19 18:28:04 +02:00
John Ericson	d5af5763cf	Merge branch 'master' of github.com:NixOS/nix into remove-storetype-delegate-regStore	2020-08-19 15:54:50 +00:00
John Ericson	be0d429b95	Merge branch 'master' of github.com:NixOS/nix into templated-daemon-protocol	2020-08-19 03:17:41 +00:00
John Ericson	950ddfdb82	Merge remote-tracking branch 'upstream/master' into derivation-header-include-order	2020-08-18 14:36:44 +00:00
Eelco Dolstra	dfeb76dbf9	Merge pull request #3930 from obsidiansystems/legacy-ssh-build-paths ... Define `LegacySSHStore::buildPaths` using `cmdBuildPaths`	2020-08-18 16:07:40 +02:00
Eelco Dolstra	7cdc739ece	Merge remote-tracking branch 'origin/master' into markdown	2020-08-17 13:43:39 +02:00
Eelco Dolstra	847a5392f4	Merge branch 'ca-no-need-trust' of https://github.com/obsidiansystems/nix	2020-08-17 10:57:02 +02:00
Eelco Dolstra	3c619f6290	Merge branch 'test-RemoteStore-buildDerivation' of https://github.com/obsidiansystems/nix	2020-08-17 10:53:18 +02:00
John Ericson	6f7ac5e865	Remove extra closing paren	2020-08-14 21:59:31 +00:00
John Ericson	3c8b5b6219	Merge remote-tracking branch 'upstream/master' into single-ca-drv-build	2020-08-14 17:00:13 +00:00
John Ericson	4b571ea321	Update src/libstore/daemon.cc ... Co-authored-by: Eelco Dolstra <edolstra@gmail.com>	2020-08-14 11:52:37 -04:00
Eelco Dolstra	13e49be660	Merge pull request #3875 from obsidiansystems/new-interface-for-path-pathOpt ... Offer a safer interface for path and pathOpt	2020-08-14 17:19:19 +02:00
Eelco Dolstra	7714d9a943	Merge pull request #3924 from obsidiansystems/features-per-store ... Make `system-features` a store setting	2020-08-14 17:13:07 +02:00
Eelco Dolstra	9b9d5297ba	Merge pull request #3909 from matthewbauer/readd-hashed-mirrors ... Add hashed-mirrors back	2020-08-14 17:05:47 +02:00
Eelco Dolstra	d81f13f7cb	Merge pull request #3899 from obsidiansystems/make-narHash-not-optional ... Make narHash in ValidPathInfo not optional	2020-08-14 17:00:18 +02:00
John Ericson	e1308b1211	Define LegacySSHStore::buildPaths using cmdBuildPaths ... Evidentally this was never implemented because Nix switched to using `buildDerivation` exclusively before `build-remote.pl` was rewritten. The `nix-copy-ssh` test (already) tests this.	2020-08-13 21:27:55 +00:00
John Ericson	5ccd94501d	Allow trustless building of CA derivations ... Include a long comment explaining the policy. Perhaps this can be moved to the manual at some point in the future. Also bump the daemon protocol minor version, so clients can tell whether `wopBuildDerivation` supports trustless CA derivation building. I hope to take advantage of this in a follow-up PR to support trustless remote building with the minimal sending of derivation closures.	2020-08-13 18:15:57 +00:00
John Ericson	85aacbee64	Use TeeSink and TeeSouce in a few more places	2020-08-13 14:51:17 +00:00
John Ericson	d2f2be0f70	Test RemoteStore::buildDerivation ... Fix `wopNarFromPath` which needed a `toRealPath`.	2020-08-13 04:07:14 +00:00
John Ericson	5d67f18c86	Merge branch 'daemon-auth-cleanup' of github.com:obsidiansystems/nix into HEAD	2020-08-12 18:22:31 +00:00
John Ericson	4720853129	Make system-features a store setting ... This seems more correct. It also means one can specify the features a store should support with --store and remote-store=..., which is useful. I use this to clean up the build remotes test.	2020-08-12 18:13:00 +00:00
John Ericson	8d4162ff9e	Separate auth and logic for the daemon ... Before, processConnection wanted to know a user name and user id, and `nix-daemon --stdio`, when it isn't proxying to an underlying daemon, would just assume "root" and 0. But `nix-daemon --stdio` (no proxying) shouldn't make guesses about who holds the other end of its standard streams. Now processConnection takes an "auth hook", so `nix-daemon` can provide the appropriate policy and daemon.cc doesn't need to know or care what it is.	2020-08-12 15:22:33 +00:00
John Ericson	5f80aea795	Break out lambda so output can be matched just once ... This is much better.	2020-08-12 02:23:31 +00:00
John Ericson	18834f7764	Recheck path validity after acquiring lock ... It might have changed, and in any event this is how the cod used to work so let's just keep it.	2020-08-11 23:44:02 +00:00
John Ericson	2de201254e	Don't assume a total output map in two places in build.cc ... Thanks @regnat for catching one of them. The other follows for many of the same reasons. I'm find fixing others on a need-to-fix basis, provided their are no regressions.	2020-08-11 23:07:50 +00:00
John Ericson	4c6aac8fdf	Clarify comment on sandbox and temp fresh paths	2020-08-11 22:46:05 +00:00
John Ericson	6d57139050	Clarify outputReferences variable with self-describing type ... Thanks for the idea, @Regnat!	2020-08-11 22:34:09 +00:00
John Ericson	8a068bd025	Remove redundant equality check	2020-08-11 21:25:40 +00:00
John Ericson	07e3466eb4	Float comment to out describe gaveUpOnSubstitution in general	2020-08-11 21:16:14 +00:00
John Ericson	d0f6e338dd	Apply suggestions from code review ... Thanks!! Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>	2020-08-11 16:49:10 -04:00
John Ericson	2a0902634e	Fix error in merge breaking floating CA drvs ... Forgot to add this hunk!	2020-08-11 00:13:19 +00:00
John Ericson	1b5c24662b	Merge branch 'small-drv-serialize-cleanup' of github.com:obsidiansystems/nix into single-ca-drv-build	2020-08-10 01:57:54 +00:00
John Ericson	581183d4d5	Deduplicate parsing and reading derivations	2020-08-10 01:40:50 +00:00
John Ericson	bcd0629c2e	Remove name parameter from writeDerivation ... The name is now stored with the derivation itself.	2020-08-10 01:35:59 +00:00
John Ericson	e913a2989f	Squashed get CA derivations building	2020-08-07 19:51:55 +00:00
Eelco Dolstra	edfd676e05	Fix .ls file names in binary caches ... These are not supposed to include the 'name' part of the store path. This was broken by 759947bf72.	2020-08-07 21:18:29 +02:00
John Ericson	8f92bb5ad9	Merge branch 'drv-outputs-map-allow-missing' of github.com:obsidiansystems/nix into templated-daemon-protocol	2020-08-07 18:51:01 +00:00
John Ericson	f7ba16f9cb	Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing	2020-08-07 17:46:39 +00:00
John Ericson	47644e49ca	Specialize std::optional<StorePath> so this is backwards compatible ... While I am cautious to break parametricity, I think it's OK in this cases---we're not about to try to do some crazy polymorphic protocol anytime soon.	2020-08-07 17:05:14 +00:00
Matthew Bauer	96c158d6e1	Fix build	2020-08-06 21:04:31 -05:00
Matthew Bauer	8abc577cc2	Merge remote-tracking branch 'origin/master' into readd-hashed-mirrors	2020-08-06 18:58:59 -05:00
Carlo Nucera	46f9dd56da	Fix bug due to non-deterministic arg eval order	2020-08-06 19:30:05 -04:00
Matthew Bauer	641c950701	Add hashed-mirrors back ... Some users have their own hashed-mirrors setup, that is used to mirror things in addition to what’s available on tarballs.nixos.org. Although this should be feasable to do with a Binary Cache, it’s not always easy, since you have to remember what "name" each of the tarballs has. Continuing to support hashed-mirrors is cheap, so it’s best to leave support in Nix. Note that NIX_HASHED_MIRRORS is also supported in Nixpkgs through fetchurl.nix. Note that this excludes tarballs.nixos.org from the default, as in \#3689. All of these are available on cache.nixos.org.	2020-08-06 18:19:09 -05:00
Carlo Nucera	9ab07e99f5	Use template structs instead of phantoms	2020-08-06 18:04:13 -04:00
Carlo Nucera	3d8240c32e	Remove leftover commented code	2020-08-06 16:04:18 -04:00
Carlo Nucera	f795f0fabc	Merge branch 'drv-outputs-map-allow-missing-namespace' of github.com:obsidiansystems/nix into templated-daemon-protocol	2020-08-06 15:53:09 -04:00
John Ericson	e89b5bd0bf	Minimize the usage of Hash::dummy	2020-08-06 18:31:48 +00:00
John Ericson	5e59b25a23	Merge remote-tracking branch 'upstream/master' into make-narHash-not-optional	2020-08-06 15:34:15 +00:00
Carlo Nucera	8b175f58d1	Simplify the namespace	2020-08-05 17:57:07 -04:00
Carlo Nucera	0739d428e0	Solve template deduction problem ... We had to predeclare our template functions	2020-08-05 17:49:45 -04:00
John Ericson	6c66331d5c	WIP: Put the worker protocol read and write in a namespace to disambig	2020-08-05 20:37:48 +00:00
Carlo Nucera	1d2e80ddd6	Merge branch 'master' of github.com:NixOS/nix into new-interface-for-path-pathOpt	2020-08-05 15:45:33 -04:00
John Ericson	ed96e603e1	Proxy -> Phantom to match Rust ... Sorry, Haskell.	2020-08-05 19:44:08 +00:00
Carlo Nucera	8241e660ba	Remove Hash::operator bool () ... Since the hash is not optional anymore	2020-08-05 15:30:38 -04:00
Carlo Nucera	be6e1c6457	Merge branch 'master' of github.com:NixOS/nix into make-narHash-not-optional	2020-08-05 15:14:47 -04:00
Carlo Nucera	1ad6394b33	Add Hash::dummy to signal default value ... We did this in the same spirit of the dummy value that's present in libstore/path.hh	2020-08-05 15:11:49 -04:00
Carlo Nucera	1d71028f4d	Remove optionality in ValidPathInfo::narInfo	2020-08-05 14:42:48 -04:00
John Ericson	7302761f64	Merge remote-tracking branch 'obsidian/drv-outputs-map-allow-missing' into templated-daemon-protocol	2020-08-05 17:53:24 +00:00
Ben Burdette	f1a47a96b6	error messages for issue 2238	2020-08-05 10:58:00 -06:00
John Ericson	cf939055c8	Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing	2020-08-05 16:43:30 +00:00
John Ericson	0559ff3d8b	Merge branch 'master' of github.com:NixOS/nix into derivation-primop-floating-output	2020-08-05 14:56:40 +00:00
John Ericson	92ad550e96	Merge remote-tracking branch 'obsidian/misc-ca' into derivation-primop-floating-output	2020-08-05 14:51:41 +00:00
Eelco Dolstra	790b694be7	Style fix	2020-08-05 16:51:06 +02:00
Eelco Dolstra	e48f944e9d	Merge branch 'misc-ca' of https://github.com/obsidiansystems/nix	2020-08-05 16:50:05 +02:00
John Ericson	b9ebe373bb	Sed some names to perhaps avoid conflicts	2020-08-05 14:49:25 +00:00
Eelco Dolstra	25f7912156	Style fix	2020-08-05 16:47:48 +02:00
John Ericson	e561a13a58	Reanme DerivationType::Regular defintion too ... This is the one non-prefixed occurence	2020-08-05 14:45:56 +00:00
John Ericson	e7b0847f2d	Make names more consistent	2020-08-05 14:44:39 +00:00
John Ericson	839f0fe095	Merge remote-tracking branch 'upstream/master' into misc-ca	2020-08-05 14:40:01 +00:00
John Ericson	03f4fafc27	Merge remote-tracking branch 'upstream/master' into misc-ca	2020-08-05 14:36:25 +00:00
Eelco Dolstra	b91dc7ebad	Merge pull request #3730 from obsidiansystems/better-ca-parse-errors ... Improve hash parsing and errors	2020-08-05 16:33:07 +02:00
Eelco Dolstra	75f220a595	Merge pull request #3864 from obsidiansystems/more-topo-sort ... Abstract out topo sorting logic	2020-08-05 16:07:29 +02:00
John Ericson	16c98bf57c	Get rid of some unneeded temporaries	2020-08-04 22:36:31 +00:00
John Ericson	1dfcbebc95	Organize and format code a bit	2020-08-04 22:28:10 +00:00
John Ericson	45b6fdb22b	Remove unused functions	2020-08-04 22:10:13 +00:00
John Ericson	1bab8a321f	Remove unneeded definition ... Template instantiations will cover this case fine.	2020-08-04 21:56:42 +00:00
John Ericson	2f2ae993dc	WIP systematize more of the worker protocol ... This refactor should *not* change the wire protocol.	2020-08-04 19:02:05 +00:00
John Ericson	fbeb8695fb	Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing	2020-08-04 18:35:33 +00:00
John Ericson	d3452a5ed6	Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors	2020-08-04 16:13:20 +00:00
Eelco Dolstra	4e7f1c7f11	S3BinaryCacheStore: Fix size determination	2020-08-04 16:01:13 +02:00
Eelco Dolstra	dfe66420e7	Revert "Remove putBytes" ... This reverts commit b8eea7e81a.	2020-08-04 15:56:10 +02:00
Eelco Dolstra	327b1bf378	BinaryCacheStore: Explicitly flush file sink ... The file sink is also flushed in its destructor, but we ignore any exceptions in the destructor. Issue #3886.	2020-08-04 14:54:07 +02:00
Eelco Dolstra	ed52cf632b	Merge pull request #3892 from obsidiansystems/path-info-header ... Move ValidPathInfo to its own header	2020-08-04 10:44:47 +02:00
John Ericson	062533f7cd	Merge remote-tracking branch 'upstream/master' into path-info-header	2020-08-03 21:02:28 +00:00
Eelco Dolstra	24e07c428f	Delete compressed NARs ... Fixes #3891.	2020-08-03 18:34:52 +02:00
John Ericson	54281f3ac1	addToStore in terms of addToStoreFromDump is not local-store-specific	2020-08-03 04:13:45 +00:00
John Ericson	9357512d73	Merge remote-tracking branch 'upstream/master' into derivation-header-include-order	2020-08-01 19:38:35 +00:00
Carlo Nucera	b6d97fdbf4	Merge branch 'master' of github.com:NixOS/nix into drv-outputs-map-allow-missing	2020-07-31 13:12:51 -04:00
Eelco Dolstra	bf290c2306	Merge remote-tracking branch 'origin/master' into markdown	2020-07-31 16:07:04 +02:00
Eelco Dolstra	1d0a7b54fa	Enable syntax highlighting	2020-07-31 15:43:25 +02:00
Carlo Nucera	eee6ef86cd	Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors	2020-07-30 18:27:25 -04:00
Matthew Bauer	05ac4db39a	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-07-30 12:38:24 -05:00
Eelco Dolstra	3f6e88a552	unsigned long long -> uint64_t	2020-07-30 13:34:04 +02:00
Eelco Dolstra	ebee2b7852	receiveContents(): unsigned int -> size_t	2020-07-30 13:00:30 +02:00
Eelco Dolstra	f63839bfa4	Cleanup	2020-07-29 18:04:51 +02:00
Eelco Dolstra	4c0077a07d	Fix RemoteStore::addToStore() latency ... Since 6185d25e52, this was very latency-bound since it required a round-trip for every 32 KiB. So for example copying a 514 MiB closure over a virtual ethernet device with a articial delay of just 1 ms took 343s. Now it takes 2.7s. Fixes #3372.	2020-07-29 00:48:39 +02:00
Eelco Dolstra	c159f48a39	Cleanup	2020-07-29 00:24:55 +02:00
John Ericson	df707d05d1	Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca	2020-07-28 21:12:36 +00:00
Carlo Nucera	2980b244b7	Use assert(false) instead of abort()	2020-07-28 15:39:45 -04:00
Carlo Nucera	cfe96bd7ce	Merge branch 'master' of github.com:NixOS/nix into ca-drv	2020-07-28 15:36:45 -04:00
Carlo Nucera	c318d398f3	Merge branch 'misc-ca' of github.com:obsidiansystems/nix into new-interface-for-path-pathOpt	2020-07-28 14:22:24 -04:00
Carlo Nucera	7ef1e3cd14	Use the new interface	2020-07-28 13:59:24 -04:00
Carlo Nucera	d564ac1c50	Offer a safer interface for pathOpt ... The new interface we offer provides a way of getting all the DerivationOutputs with the storePaths directly, based on the observation that it's the most common usecase.	2020-07-28 13:55:15 -04:00
Eelco Dolstra	b8eea7e81a	Remove putBytes ... istream->tellg() returns -1 so we can't get the number of bytes written. Fixes 'uploaded 's3://nix-cache/nar/00819r9lp5kajr6baxfw5dhhc0cx8ndxaz43qmd2f0gn1hk1ynlp.nar.xz' (-1 bytes) in 11620 ms' messages.	2020-07-28 13:27:56 +02:00
John Ericson	8065c6d160	Abstract out topo sorting logic	2020-07-27 20:45:34 +00:00
John Ericson	951415b568	Require ca-derivations everywhere we create a CA derivation ... "create" as in read one in from a serialized form, or build one from scratch in memory.	2020-07-27 17:56:36 +00:00
John Ericson	e32a9e124b	Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output	2020-07-27 17:50:06 +00:00
John Ericson	7cf978440c	Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca	2020-07-27 17:47:40 +00:00
John Ericson	e6115ca004	Merge remote-tracking branch 'upstream/master' into ca-derivation-data-types	2020-07-27 17:41:43 +00:00
John Ericson	1d7d94ceea	Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors	2020-07-27 16:17:50 +00:00
John Ericson	43f2bd8dc5	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-07-27 16:13:57 +00:00
John Ericson	387699fe7f	Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca	2020-07-27 15:00:28 +00:00
John Ericson	78466bcb2f	Merge branch 'optional-derivation-output-storepath' into ca-derivation-data-types	2020-07-27 14:40:08 +00:00
John Ericson	d5bb67cfa4	Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath	2020-07-27 14:29:32 +00:00
John Ericson	699fc89b39	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-07-27 14:22:45 +00:00
John Ericson	2c7557481b	queryDerivationOutputMap no longer assumes all outputs have a mapping ... This assumption is broken by CA derivations. Making a PR now to do the breaking daemon change as soon as possible (if it is already too late, we can bump protocol intead).	2020-07-24 21:14:06 +00:00
John Ericson	58e55c0923	Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output	2020-07-23 19:17:52 +00:00
John Ericson	a2af25c55a	Merge remote-tracking branch 'upstream/master' into misc-ca	2020-07-23 19:14:17 +00:00
John Ericson	c36916eca0	Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output	2020-07-23 15:06:56 +00:00
John Ericson	6fcd9966bf	Merge remote-tracking branch 'upstream/master' into misc-ca	2020-07-23 15:02:29 +00:00
John Ericson	66a2067288	Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors	2020-07-23 14:56:35 +00:00
John Ericson	2274f63453	Get rid of basicDerivation::findOutput ... It's a tiny function which is: - hardly worth abstrating over, and also only used once. - doesn't work once we get CA drvs I rewrote the one callsite to be forwards compatable with CA derivations, and also potentially more performant: instead of reading in the derivation it can ust consult the SQLite DB in the common case.	2020-07-23 14:39:02 +00:00
John Ericson	9423f64ee2	Parse CA derivations using new output variants ... We no longer need `ParsedDerivation` because everything libstore needs to know about is in the `BasicDerivation` proper.	2020-07-22 23:59:25 +00:00
John Ericson	c56356bacc	Separate concerns in scanForReferences with TeeSink ... This also will make it easier to use a `HashModuloSink` instead for CA derivations.	2020-07-22 22:13:44 +00:00
Eelco Dolstra	ff314f186e	Merge pull request #3736 from obsidiansystems/allow-relative-paths-in-store-option ... Allow relative paths in --store option	2020-07-21 19:43:07 +02:00
Eelco Dolstra	0951330680	Merge pull request #3714 from obsidiansystems/add-body-to-network-errors ... Add response body to network errors	2020-07-21 18:09:42 +02:00
John Ericson	02639716ea	Merge branch 'allow-relative-paths-in-store-option' into remove-storetype-delegate-regStore	2020-07-21 15:55:27 +00:00
John Ericson	6cce32c8e8	Change logic for deciding what is a relative path for the local store ... The was Eelco's prefered logic, and it looks good to me!	2020-07-21 15:39:47 +00:00
Carlo Nucera	922a845ffc	Update chunkSize to the suggested value ... This was a suggested course of action in a review in one of our earlier commits, https://github.com/NixOS/nix/pull/3801#discussion_r457557079	2020-07-21 10:24:19 -04:00
John Ericson	54e507a7aa	Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors	2020-07-21 14:17:59 +00:00
John Ericson	5ce95b9529	Update src/libstore/build.cc	2020-07-21 09:47:40 -04:00
John Ericson	cdb3f39b64	Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors	2020-07-21 13:43:20 +00:00
John Ericson	5055c595bd	Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into ca-derivation-data-types	2020-07-21 01:20:53 +00:00
John Ericson	5a37413796	Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into from-dump-stream	2020-07-21 01:20:14 +00:00
John Ericson	bf0dca7790	Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into misc-ca	2020-07-21 01:18:33 +00:00
Carlo Nucera	9aae179f34	Correct bug, thoroughly document addToStoreSlow	2020-07-20 20:18:12 -04:00
Carlo Nucera	0ca9744694	Use heuristics to decide when to show the response ... Due to https://github.com/NixOS/nix/issues/3841 we don't know how print different messages for different verbosity levels.	2020-07-20 15:57:58 -04:00
Carlo Nucera	6357b1b0fb	Add another Unimplemented case	2020-07-20 14:17:25 -04:00
Carlo Nucera	007bf6d17a	Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca	2020-07-20 14:14:39 -04:00
Carlo Nucera	362ae93851	Add UnimplementedError to ease grepping for these	2020-07-20 14:13:37 -04:00
John Ericson	bf61871271	parser.hh -> split.hh	2020-07-20 17:42:34 +00:00
John Ericson	eab945f591	Merge remote-tracking branch 'upstream/master' into from-dump-stream	2020-07-20 17:29:23 +00:00
Eelco Dolstra	1c5f8bbfb5	Merge pull request #3822 from obsidiansystems/dump-thrice-fixme ... Optimize `addToStoreSlow` and remove `TeeParseSink`	2020-07-20 18:55:05 +02:00
John Ericson	ac2fc7ba1f	Apply suggestions from code review ... Co-authored-by: Eelco Dolstra <edolstra@gmail.com>	2020-07-20 11:29:46 -04:00
Carlo Nucera	0aa79dcc6f	Remove StoreType abstraction and delegate regStore ... to each Store implementation. The generic regStore implementation will only be for the ambiguous shorthands, like "" and "auto". This also could get us close to simplifying the daemon command.	2020-07-17 17:24:02 -04:00
John Ericson	6756cecfcf	Add DerivationType::CAFloating	2020-07-17 19:55:41 +00:00
Carlo Nucera	fefd6c9e5f	Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option	2020-07-17 14:34:20 -04:00
Carlo Nucera	bbc633c98c	Revert "Don't anticipate CA but not fixed outputs for now" ... This reverts commit 3a9e4c3262.	2020-07-17 13:10:32 -04:00
Carlo Nucera	205dcd140d	Revert "Don't anticipate multiple CA outputs for now" ... This reverts commit 74b251b2f3.	2020-07-17 12:43:46 -04:00
Carlo Nucera	1feb8981df	Revert "Don't anticipate hash algo without hash in derivation for now" ... This reverts commit 3804e3df9b.	2020-07-17 12:05:54 -04:00
Carlo Nucera	487c5751c6	Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca	2020-07-17 11:32:59 -04:00
Carlo Nucera	5cb840541b	Merge branch 'multi-output-hashDerivationModulo' of github.com:Ericson2314/nix into misc-ca	2020-07-17 10:28:33 -04:00
Carlo Nucera	745a03cef5	Merge branch 'optional-derivation-output-storepath' of github.com:obsidiansystems/nix into ca-derivation-data-types	2020-07-16 13:36:01 -04:00
Carlo Nucera	048e916f64	Merge branch 'master' of github.com:NixOS/nix into optional-derivation-output-storepath	2020-07-16 13:32:28 -04:00
John Ericson	cc0d77f8c9	Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors	2020-07-16 17:28:52 +00:00
Carlo Nucera	e3a3406db8	Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors	2020-07-16 12:04:10 -04:00
John Ericson	5ea817dace	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-07-16 14:58:53 +00:00
John Ericson	3dcca18c30	Fix bug in TeeSource ... We use this to simplify `LocalStore::addToStoreFromDump`. Also, hope I fixed build error with old clang (used in Darwin CI).	2020-07-16 13:39:27 +00:00
Carlo Nucera	e3b394b6e8	Small namespace fix	2020-07-16 09:36:02 -04:00
Eelco Dolstra	5517eee17e	Generations API cleanup	2020-07-16 15:18:21 +02:00
Eelco Dolstra	3f264916db	Merge remote-tracking branch 'origin/flakes'	2020-07-16 14:07:32 +02:00
John Ericson	68dfb8c6ae	Optimize addToStoreSlow and remove TeeParseSink	2020-07-16 05:09:41 +00:00
John Ericson	5602637d9e	Revert "LocalStore::addToStoreFromDump copy in chunks" ... This reverts commit 592851fb67. We don't need this extra feature anymore	2020-07-15 23:37:49 +00:00
John Ericson	bc109648c4	Get rid of LocalStore::addToStoreCommon ... I got it to just become `LocalStore::addToStoreFromDump`, cleanly taking a store and then doing nothing too fancy with it. `LocalStore::addToStore(...Path...)` is now just a simple wrapper with a bare-bones sinkToSource of the right dump command.	2020-07-15 23:14:30 +00:00
Carlo Nucera	455bdee205	Merge branch 'master' of github.com:NixOS/nix into derivation-header-include-order	2020-07-15 17:58:30 -04:00
Carlo Nucera	d090562348	Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type	2020-07-15 17:21:01 -04:00
John Ericson	aff69e229c	Merge remote-tracking branch 'obsidian/fix-add-to-star-narhash' into from-dump-stream	2020-07-15 21:05:29 +00:00
Carlo Nucera	2d2a10e79a	Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors	2020-07-15 13:06:08 -04:00
Eelco Dolstra	cff2157185	Revert "LocalStore::addToStore(srcPath): Handle the flat case" ... This reverts commit a2c27022e9. See addToStoreSlow(), we don't need to handle this case efficiently anymore. In fact, we can almost remove the method/hashAlgo arguments since the non-recursive and/or non-SHA256 are almost not used anymore.	2020-07-15 12:49:03 +02:00
John Ericson	650c2c6558	Rename variable nar -> dump according to TODO	2020-07-14 21:28:50 +00:00
John Ericson	8173e7bfef	Fix localhost::addToStore(...Path...) ... We were calculating the nar hash wrong when the file ingestion method was flat. I don't think there's anything we can do in that case but dump the file again, so that's what I do. As an optomization, we again could reuse the original dump for just the recursive and non-sha256 case, but I rather do that after this fix, and after my other PRs which deduplicate this code.	2020-07-14 21:12:11 +00:00
John Ericson	592851fb67	LocalStore::addToStoreFromDump copy in chunks ... Rather than copying byte-by-byte, we let the coroutine know how much data we would like it to send back to us.	2020-07-14 13:56:36 +00:00
John Ericson	9de96ef7d4	Dedup LocalStore::addToStore* ... The downsides is that the coroutine has byte-by-byte loop transfer. Will fix that next.	2020-07-14 13:56:36 +00:00
John Ericson	c86fc3a965	Crudely make addToStoreFromDump take Source not string ... I just as little beyond the type as possible, so the implementation changes this enables can be reviewed separately.	2020-07-14 13:56:36 +00:00
John Ericson	9ec10046e0	Narrow scope of temporary value	2020-07-14 13:54:29 +00:00
Eelco Dolstra	832e111494	Merge remote-tracking branch 'origin/master' into flakes	2020-07-14 13:56:18 +02:00
Eelco Dolstra	926c3a6664	Doh	2020-07-14 11:55:54 +02:00
Eelco Dolstra	43b8e96d30	Fix 'nix verify --all' on a binary cache (cached case)	2020-07-13 20:17:00 +02:00
Eelco Dolstra	7c2fef0a81	Make 'nix copy' to s3:// binary caches run in constant memory	2020-07-13 20:07:19 +02:00
Eelco Dolstra	493961b689	Remove istringstream_nocopy	2020-07-13 18:31:19 +02:00
Eelco Dolstra	545bb2ed03	Remove 'accessor' from addToStore() ... This is only used by hydra-queue-runner and it's better to implement it there.	2020-07-13 18:31:19 +02:00
Eelco Dolstra	0a9da00a10	NarAccessor: Run in constant memory	2020-07-13 17:30:42 +02:00
Eelco Dolstra	fc84c358d9	Make 'nix copy' to file:// binary caches run in constant memory	2020-07-13 16:28:45 +02:00
Eelco Dolstra	400f1a9b59	Store::pathInfoToJSON(): Use consistent format for downloadHash	2020-07-13 16:26:09 +02:00
Eelco Dolstra	c0dd05131e	toStorePath(): Return a StorePath and the suffix	2020-07-13 16:25:48 +02:00
Eelco Dolstra	1d01ae816b	Fix 'nix verify --all' on a binary cache and add a test	2020-07-13 14:35:01 +02:00
John Ericson	c466cb2091	Merge branch 'hash-always-has-type' into better-ca-parse-errors	2020-07-13 03:02:09 +00:00
John Ericson	4415765385	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-07-13 03:01:44 +00:00
John Ericson	230c9b4329	Change types to prepare the way for CA derivations ... We've added the variant to `DerivationOutput` to support them, but made `DerivationOutput::path` partial to avoid actually implementing them. With this chage, we can all collaborate on "just" removing `DerivationOutput::path` calls to implement CA derivations.	2020-07-12 23:53:33 +00:00
John Ericson	fedfc913ad	Use more std::visit to prepare for new variant ... N.B. not using `std::visit` for fetchurl because there is no attempt to handle all the cases (e.g. no `else`) and lambda complicates early return.	2020-07-12 23:51:40 +00:00
John Ericson	ffc18583b1	Move C++17 "pattern matching" boilerplat to utils.hh	2020-07-12 22:15:14 +00:00
John Ericson	503b425690	DerivationOutputExtensional -> DerivationOutputInputAddressed ... Thanks @regnat for the great name.	2020-07-12 15:56:20 +00:00
John Ericson	a8d4707107	Undo erroneous indentation change	2020-07-12 15:54:12 +00:00
John Ericson	18152406ce	String .drv suffix to create derivation name	2020-07-12 15:40:14 +00:00
John Ericson	5d0b75e5b6	std::string_view for new derivation name parameters	2020-07-12 15:02:36 +00:00
John Ericson	abea26a968	BasicDerivation::findOutput cannot return reference anymore	2020-07-12 01:57:06 +00:00
John Ericson	442d43178b	Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath	2020-07-11 02:27:45 +00:00
Matthew Bauer	acb74d4d94	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-07-10 18:13:45 -04:00
Eelco Dolstra	8efa23bb99	Avoid a redundant hash	2020-07-10 15:56:24 +02:00
Eelco Dolstra	5dff49f661	Factor out commonality between nix-prefetch-url and nix-store --add-fixed	2020-07-10 13:21:37 +02:00
John Ericson	323d51a7a0	Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors	2020-07-09 19:07:06 +00:00
Matthew Bauer	a7884970c5	Fix DerivationOutputExtensional name	2020-07-09 11:37:18 -04:00
Eelco Dolstra	a2c27022e9	LocalStore::addToStore(srcPath): Handle the flat case ... This helps nix-prefetch-url when using a local store.	2020-07-09 15:54:32 +02:00
Matthew Bauer	06a4e15478	Fix build.cc on linux	2020-07-08 19:27:51 -04:00
Matthew Bauer	8e0d0689be	Only store hash of fixed derivation output ... we don’t need a full storepath for a fixedoutput derivation. So just putting the ingestion method + the hash is sufficient.	2020-07-08 19:11:39 -04:00
Eelco Dolstra	b981e5aacf	Cleanup	2020-07-08 22:07:21 +02:00
Eelco Dolstra	34f25124ba	Make LocalStore::addToStore(srcPath) run in constant memory ... This reduces memory consumption of nix-instantiate \ -E 'with import <nixpkgs> {}; runCommand "foo" { src = ./blender; } "echo foo"' \ --option nar-buffer-size 10000 (where ./blender is a 1.1 GiB tree) from 1716 to 36 MiB, while still ensuring that we don't do any write I/O for small source paths (up to 'nar-buffer-size' bytes). The downside is that large paths are now always written to a temporary location in the store, even if they produce an already valid store path. Thus, adding large paths might be slower and run out of disk space. ¯\_(ツ)_/¯ Of course, you can always restore the old behaviour by setting 'nar-buffer-size' to a very high value.	2020-07-08 22:07:21 +02:00
Matthew Bauer	af95a7c16b	Add name to BasicDerivation ... We always have a name for BasicDerivation, since we have a derivation store path that has a name.	2020-07-08 15:38:01 -04:00
Eelco Dolstra	9223603908	Merge remote-tracking branch 'origin/master' into flakes	2020-07-08 15:55:19 +02:00
Eelco Dolstra	16ec7785ca	Fix 'got unknown message type 1 from Nix daemon' ... Example: $ nix-build -E 'with import <nixpkgs> {}; runCommand "foo" { x = runCommand "bar" {} "exit 1"; } "echo foo; exit 1"' warning: unknown setting 'auto-allocate-uids' these 2 derivations will be built: /nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv /nix/store/k4fsvrjl7cp2xpz7927iv7g0dqj1zyhs-foo.drv warning: unknown setting 'auto-allocate-uids' building '/nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv'... error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------- nix-daemon builder for '/nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv' failed with exit code 1 error: --- Error ------------------------------------------------------------------------------------------------------------------------------------------------------------------ nix-build got unknown message type 1 from Nix daemon	2020-07-08 15:53:14 +02:00
Eelco Dolstra	1ab9da9154	Merge remote-tracking branch 'origin/master' into flakes	2020-07-07 14:38:57 +02:00
Eelco Dolstra	7c9ece5dca	exportReferencesGraph: Fix support for non-top-level store paths ... Fixes #3471.	2020-07-07 14:25:43 +02:00
Carlo Nucera	0f05a36e20	Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors	2020-07-06 16:57:09 -04:00
Eelco Dolstra	54712aaf8a	Merge remote-tracking branch 'origin/master' into flakes	2020-07-06 16:40:10 +02:00
John Ericson	a38ab99d57	Merge remote-tracking branch 'upstream/master' into derivation-header-include-order	2020-07-05 21:49:01 +00:00
John Ericson	465daa9396	Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors	2020-07-03 17:08:39 +00:00
John Ericson	3134db1a83	Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors	2020-07-03 14:12:38 +00:00
John Ericson	dbffd309fe	Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type	2020-07-03 14:11:38 +00:00
Ben Burdette	b29a4ea1dc	Merge branch 'master' into add-trace	2020-07-03 07:57:36 -06:00
regnat	223fbe644a	Shorten the path to the test root ... Fix a socket length failure on the OSX builders	2020-07-03 09:20:01 +02:00
John Ericson	13796be78d	Have splitPrefix and splitPrefixTo parser helpers	2020-07-02 23:18:22 +00:00
John Ericson	2f93d9f2ba	Merge branch 'hash-always-has-type' into HEAD	2020-07-02 21:47:51 +00:00
John Ericson	1be279af26	Fix Narinfo corruption detection bug ... The aim of this check was just to ensure each key occurs once.	2020-07-02 21:46:10 +00:00
Carlo Nucera	1fc835aa22	Tighten parsing for drv files and pathinfo	2020-07-02 11:57:21 -04:00
Matthew Bauer	fc2ab42e86	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-07-02 11:14:04 -04:00
Matthew Bauer	d2e8b9ff0e	Store subPath in SubstitutionGoal	2020-07-02 11:12:05 -04:00
Ben Burdette	bf2788e4c1	move showTrace to new loggerSettings	2020-07-02 09:04:31 -06:00
Matthew Bauer	1f9cb06db2	Try next when no ca exists and have different store dirs	2020-07-02 10:59:24 -04:00
Carlo Nucera	263ccdd489	Rename two hash constructors to proper functions	2020-07-01 18:34:18 -04:00
Eelco Dolstra	38ccf2e241	Cleanup	2020-07-01 15:31:34 +02:00
Eelco Dolstra	86a4aba6c4	Merge branch 'remote-query-outputs' of https://github.com/tweag/nix	2020-07-01 15:10:29 +02:00
Ben Burdette	9159dfe3d8	comments and cleanup	2020-06-30 16:31:55 -06:00
Matthew Bauer	09833126f3	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-06-30 16:35:08 -04:00
Ben Burdette	ddb81ca126	Merge branch 'master' into add-trace	2020-06-30 12:21:45 -06:00
Carlo Nucera	77b51f4598	Factor the prefix splitting in content-address	2020-06-30 11:57:46 -04:00
John Ericson	a83566e5bc	Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors	2020-06-29 18:40:34 +00:00
John Ericson	987a4a0be9	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-06-29 18:03:21 +00:00
Ben Burdette	8f81fae116	showTrace flag in loggers	2020-06-29 10:20:51 -06:00
Eelco Dolstra	58bc3b6578	Merge pull request #3729 from obsidiansystems/simpler-hased-mirror ... hashed-mirrors: Use parsed derivation output rather than reconstructing it	2020-06-29 14:04:12 +02:00
Eelco Dolstra	64232f3ea6	Merge pull request #3749 from rodarima/master ... Fall back to copyPath if link fails with EPERM	2020-06-29 13:31:24 +02:00
Carlo Nucera	278a679782	Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option	2020-06-26 10:40:13 -04:00
Eelco Dolstra	adf2fbbdc2	Merge remote-tracking branch 'origin/master' into flakes	2020-06-26 08:46:46 +02:00
John Ericson	d92d4f85a5	Move ValidPathInfo to its own header	2020-06-25 17:48:49 +00:00
Eelco Dolstra	de2641ae99	Fix empty std::optional dereference in writeDerivation() ... https://hydra.nixos.org/build/123017579	2020-06-25 15:50:30 +02:00
Rodrigo	3a642187c3	Fall back to copyPath if link fails with EPERM ... BeeGFS doesn't allow hard-links and returns EPERM, so we fall back to copyPath. See https://github.com/NixOS/nix/issues/3748	2020-06-25 12:03:26 +02:00
regnat	d38f860c3e	Add a way to get all the outputs of a derivation with their label ... Generalize `queryDerivationOutputNames` and `queryDerivationOutputs` by adding a `queryDerivationOutputMap` that returns the map `outputName=>outputPath` (not that this is not equivalent to merging the results of `queryDerivationOutputs` and `queryDerivationOutputNames` as sets don't preserve the order, so we would end up with an incorrect mapping). squash! Add a way to get all the outputs of a derivation with their label Rename StorePathMap to OutputPathMap	2020-06-24 20:38:40 +02:00
John Ericson	d9fb9e9f26	Merge remote-tracking branch 'upstream/master' into ca-drv	2020-06-24 15:41:29 +00:00
John Ericson	98e5d1af03	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-06-23 17:03:37 +00:00
Carlo Nucera	e197bc6229	Enable the --store option to take relative paths ... In nix commands which accept --store options, we can now specify a relative path, which will be canonicalized.	2020-06-23 11:13:18 -04:00
John Ericson	55d4bd6e0e	Improve content address parsing ... - Ensure hash is in form <algo>-<prefix> and not SRI. - Better errors if something goes wrong - string_view for no coppying	2020-06-22 18:08:27 +00:00
Matthew Bauer	66a62b3189	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-06-22 13:08:11 -04:00
John Ericson	f4a5913125	hashed-mirrors: Use parsed derivation output rather than reconstructing it ... Now the derivation outputs are parsed up front, we can avoid a reparse by doing it. Also, this just feels a bit better as the `output*` env vars are more of a `libnixexpr` interface than `libnixstore` interface: ultimately, it's the derivation outputs that decide whether the derivation is fixed-output. Yes, hashed mirrors might go away with #3689, but this bit of code would be moved rather than deleted, so it's worth doing a cleanup anyways I think.	2020-06-22 15:17:20 +00:00
John Ericson	93bbe6e8ab	Merge remote-tracking branch 'upstream/master' into ca-drv	2020-06-21 21:11:58 +00:00
John Ericson	3804e3df9b	Don't anticipate hash algo without hash in derivation for now ... When we merge with master, the new lack of string types make this case impossible (after parsing). Later, when we actually implemenent CA-derivations, we'll change the types to allow that.	2020-06-21 21:05:37 +00:00
John Ericson	8313f0e939	Merge remote-tracking branch 'upstream/master' into derivation-header-include-order	2020-06-21 20:39:10 +00:00
John Ericson	bf9f040112	Tweak declaration ... I think this is clearer	2020-06-21 16:51:39 +00:00
John Ericson	fdeabf7160	Merge remote-tracking branch 'upstream/master' into multi-output-hashDerivationModulo	2020-06-21 16:43:17 +00:00
Matthew Bauer	f2a6cee334	Update worker protocol to support sending storepath maps ... We need to also send the ca to daemon in addition to the path.	2020-06-19 18:06:19 -04:00
John Ericson	507aa48739	WIP: Make Hash always store a valid hash type	2020-06-19 21:48:57 +00:00
Matthew Bauer	e8e1f5282f	Replace error message in getDerivationCA	2020-06-19 15:19:09 -04:00
Matthew Bauer	0c9c1b8826	Return map of StorePaths in copyPaths ... This allows the caller to know what values were actually added to the store.	2020-06-19 14:48:57 -04:00
Matthew Bauer	7e11cf3399	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-06-19 14:40:49 -04:00
Matthew Bauer	2796b01e86	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-06-19 14:22:06 -04:00
John Ericson	68294746ae	Merge remote-tracking branch 'upstream/master' into no-hash-type-unknown	2020-06-19 17:53:34 +00:00
John Ericson	911fc88bcb	More designated initializers	2020-06-19 17:42:56 +00:00
John Ericson	2f0e395c99	Merge remote-tracking branch 'me/no-stringly-typed-derivation-output' into validPathInfo-ca-proper-datatype	2020-06-19 15:26:59 +00:00
John Ericson	fb39a5e00c	Remove unneeded constructor for DerivationOutputHash	2020-06-19 15:11:11 +00:00
John Ericson	01dc8b0bab	Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output	2020-06-19 14:59:05 +00:00
John Ericson	145d88cb2a	Use designated initializers for DerivationOutputHash	2020-06-19 14:58:30 +00:00
John Ericson	237d88c97e	FileSystemHash -> DerivationOutputHash	2020-06-19 14:47:10 +00:00
John Ericson	3fc58a9638	Remove some Base:: that crept in	2020-06-19 00:24:47 +00:00
John Ericson	3f8dcfe3fd	Merge branch 'validPathInfo-temp' into validPathInfo-ca-proper-datatype	2020-06-18 23:01:58 +00:00
John Ericson	669c3992e8	Merge branch 'no-hash-type-unknown' into validPathInfo-temp	2020-06-18 22:33:07 +00:00
John Ericson	15abb2aa2b	Revert the enum struct change ... Not a regular git revert as there have been many merges and things.	2020-06-18 22:11:26 +00:00
John Ericson	bbbf3602a3	Merge branch 'enum-class' into no-hash-type-unknown	2020-06-18 22:11:19 +00:00
John Ericson	40526fbea5	Merge remote-tracking branch 'upstream/master' into enum-class	2020-06-18 21:38:15 +00:00
John Ericson	1b23fe4afb	Fix bugs ... - Bad dynamic cast target ...classic - std::shared_ptr need explicit deref	2020-06-18 19:03:10 +00:00
John Ericson	fb432b0bd2	Merge remote-tracking branch 'obsidian/fix-url-format' into add-body-to-network-errors	2020-06-18 18:08:02 +00:00
John Ericson	639e20dc3e	Prevent '%' in URL from causing crashes ... We have a larger problem that passsing computed strings to the first variable argument of many exception constructors is unsafe because that first variable argument is interpreted not as a plain string, but format string, and if it contains '%' boost::format will abort, since there are no arguments to the format string. In this particular instance '%' was used as part of an escape code in a URL, which, when the download failed, caused Nix to abort displaying the `FileTransferError`.	2020-06-18 17:54:16 +00:00
John Ericson	74b219ef6e	Adjust FileTransferError message to use opt response	2020-06-18 14:48:45 +00:00
regnat	4fef2ba7e4	Rename content-addressed-paths into ca-derivations ... See <https://github.com/NixOS/nix/pull/3710#issuecomment-645480333>	2020-06-18 09:25:55 +02:00
Carlo Nucera	004570a377	Add HTTP responses to FileTransferErrors	2020-06-17 22:34:31 +00:00
Carlo Nucera	a835c740ca	Replace TransferItem::status with a local variable ... Everywhere seems to use `getHTTPStatus` now.	2020-06-17 22:16:24 +00:00
Carlo Nucera	fb05a6adcf	Eliminate old TeeSink abstraction ... This was introduced in fa125b9b28, and then "reverted" in 1cf4801108, except that revert left the struct around doing nothing useful. We're removing it all the way now because we want to make a new `TeeSink` complementing the already-exiting `TeeSource`, that is actually a completely different concept as far as the class hierarchy is concerned.	2020-06-17 22:07:22 +00:00
Eelco Dolstra	2b8f33bf5f	Merge pull request #3713 from matthewbauer/cleanup-warnings ... Cleanup class StorePath warning	2020-06-17 21:19:30 +02:00
Matthew Bauer	5e631e3304	Add StorePathCAMap for querySubstitutablePathInfos ... I’m not 100% sure this is wanted since it kind of makes everything have to know about ca even if they don’t really want to. But it also make things easier in dealing with looking up ca.	2020-06-17 15:03:05 -04:00
Matthew Bauer	be50de1142	Make sure references are empty for store path replacing ... also copy info2 instead of casting	2020-06-17 14:15:08 -04:00
Matthew Bauer	8974755d19	Add assert for replaced storePath	2020-06-17 14:04:46 -04:00
Eelco Dolstra	3078404e35	Merge pull request #3712 from obsidiansystems/make-http-successful-states-coherent ... Make successful states coherent	2020-06-17 19:37:46 +02:00
Matthew Bauer	7bd88cc1dc	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-06-17 13:32:29 -04:00
Matthew Bauer	22d7d36703	Remove unused narInfoFile in binary-cache-store	2020-06-17 13:27:19 -04:00
Matthew Bauer	f767bedfac	Replace struct StorePath with class StorePath ... also a similar case with struct Goal	2020-06-17 13:26:37 -04:00
Carlo Nucera	4930cb48a2	Include review comments	2020-06-17 12:58:59 -04:00
Matthew Bauer	6438ba1e99	Update strings from review comment	2020-06-17 12:31:01 -04:00
Eelco Dolstra	2f51cd8dc9	Merge pull request #3710 from tweag/reserve_ca_derivations ... Reserve the `__contentAddressed` derivation parameter	2020-06-17 18:28:26 +02:00
Eelco Dolstra	4d5169bdd5	Merge pull request #3707 from p01arst0rm/outdated-function-fix ... replaced uncaught_exception with uncaught_exceptions	2020-06-17 18:26:01 +02:00
regnat	480b54e1c6	fixup! Reserve the __contentAddressed derivation parameter	2020-06-17 17:37:04 +02:00
Carlo Nucera	079c6e87de	Make successful states coherent ... The successful states used in these two places in the code were slightly different. Should they be the same list?	2020-06-17 11:16:16 -04:00
regnat	56d75bf4fc	Reserve the __contentAddressed derivation parameter ... Not implementing anything here, just throwing an error if a derivation sets `__contentAddressed = true` without `--experimental-features content-addressed-paths` (and also with it as there's nothing implemented yet)	2020-06-17 15:41:17 +02:00
Eelco Dolstra	fdff09e57c	Fix coverage build	2020-06-17 15:18:10 +02:00
Eelco Dolstra	ccbea8255c	Merge pull request #3657 from obsidiansystems/sligthly-improve-store-path-documentation ... Clarify the description of StorePath inputs	2020-06-17 14:54:37 +02:00
Eelco Dolstra	1524752c17	Merge remote-tracking branch 'origin/master' into flakes	2020-06-17 10:26:52 +02:00
John Ericson	517f5980e2	Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output	2020-06-17 04:58:43 +00:00
John Ericson	18493fd9c4	Move some Store functions from derivations.cc to store-api.cc ... This further continues with the dependency inverstion. Also I just went ahead and exposed `parseDerivation`: it seems like the more proper building block, and not a bad thing to expose if we are trying to be less wedded to drv files on disk anywas.	2020-06-17 04:30:16 +00:00
John Ericson	7130f0a3a6	Don't need abstract struct Derivation in local-store	2020-06-17 04:05:38 +00:00
John Ericson	21ef342172	Merge remote-tracking branch 'upstream/master' into derivation-header-include-order	2020-06-17 03:52:01 +00:00
John Ericson	02928f76fd	Merge remote-tracking branch 'upstream/master' into multi-output-hashDerivationModulo	2020-06-17 03:51:02 +00:00
John Ericson	49308ef039	Merge remote-tracking branch 'upstream/master' into ca-drv	2020-06-17 03:39:39 +00:00
p01arst0rm	c9d06558b6	replaced uncaught_exception with uncaught_exceptions	2020-06-17 03:15:47 +01:00
Eelco Dolstra	29542865ce	Remove StorePath::clone() and related functions	2020-06-16 22:20:18 +02:00
Eelco Dolstra	759947bf72	StorePath: Rewrite in C++ ... On nix-env -qa -f '<nixpkgs>', this reduces maximum RSS by 20970 KiB and runtime by 0.8%. This is mostly because we're not parsing the hash part as a hash anymore (just validating that it consists of base-32 characters). Also, replace storePathToHash() by StorePath::hashPart().	2020-06-16 14:28:41 +02:00
Eelco Dolstra	72e17290d4	Fix FTP support ... Fixes #3618.	2020-06-16 11:53:04 +02:00
Eelco Dolstra	a588b6b19d	Print only one error message if a build fails ... E.g. instead of error: --- BuildError ----------------------------------------------- nix builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1 error: --- Error ---------------------------------------------------- nix build of '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed we now get error: --- Error ---------------------------------------------------- nix builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1	2020-06-15 19:35:31 +02:00
Eelco Dolstra	31707735b6	Remove unnecessary amDone() overrides	2020-06-15 16:47:21 +02:00
Eelco Dolstra	5ed5d7acbd	Improve "waiting for locks" messages ... These are now shown in the progress bar. Closes #3577.	2020-06-15 16:03:29 +02:00
Eelco Dolstra	e14e62fddd	Remove trailing whitespace	2020-06-15 14:12:39 +02:00
Eelco Dolstra	1fb762d11f	Get rid of explicit ErrorInfo constructors	2020-06-15 14:06:58 +02:00
Eelco Dolstra	7a77762961	Merge branch 'errors-phase-2' of https://github.com/bburdette/nix	2020-06-15 11:46:31 +02:00
Matthew Bauer	c214cda940	Correctly substitute from different storeDir ... Originally, the test was only checking for different “real” storeDir. That’s an easy case to handle, but the much harder one is if different virtual store dirs are used. To do this, we need the SubstitutionGoal to know about the ca, so it can recalculate the path to copy it over. An important note here is that the store path passed to copyStorePath needs to be one for srcStore - so that queryPathInfo works properly. This also adds an error message when the store path from queryPathInfo is different from the one we requested.	2020-06-13 00:07:42 -05:00
Matthew Bauer	b2cb288cdd	Add makeFixedOutputPathFromCA function ... This puts what we are already doing into a shared method. It just needs a path name and a ca and produces a store path.	2020-06-12 16:36:35 -05:00
John Ericson	f6f01416b7	Use std::string_view in a few more places	2020-06-12 21:32:30 +00:00
Matthew Bauer	88120442d2	Debug when storePath changes ... these rewrites should be transparent, but they are important to know about when debugging	2020-06-12 15:32:52 -05:00
Matthew Bauer	5b386b05f5	Recompute storePath based on isContentAddressed	2020-06-12 14:57:45 -05:00
Matthew Bauer	85d01e1f65	Don’t use makeStorePath	2020-06-12 14:53:30 -05:00
Matthew Bauer	9077b9e6b2	Separate dstStore path from srcStore path	2020-06-12 14:27:28 -05:00
Matthew Bauer	2f2ac850b5	Compute new store path correctly	2020-06-12 14:03:17 -05:00
Matthew Bauer	11c97070f3	Fix storeDir != storeDir condition ... this needs to only continue if the path replacement fails.	2020-06-12 10:14:03 -05:00
Matthew Bauer	3e3eaa90dd	Remove hashed-mirrors	2020-06-12 09:51:44 -05:00
Matthew Bauer	79c169d1c6	Allow substituting from different storeDir ... Substituters can substitute from one store dir to another with a little bit of help. The store api just needs to have a CA so it can recompute the store path based on the new store dir. We can only do this for fixed output derivations with no references, though.	2020-06-12 09:49:09 -05:00
Eelco Dolstra	9f736dd89d	Add Store::readDerivation() convenience function	2020-06-12 13:04:52 +02:00
Eelco Dolstra	045b07200c	Remove Store::queryDerivationOutputNames() ... This function was used in only one place, where it could easily be replaced by readDerivation() since it's not performance-critical. (This function appears to have been modelled after queryDerivationOutputs(), which exists only to make the garbage collector faster.)	2020-06-12 12:46:33 +02:00
Ben Burdette	ef1b3f21b6	Merge remote-tracking branch 'upstream/master' into errors-phase-2	2020-06-11 14:06:35 -06:00
Eelco Dolstra	ac4d43a31b	Merge pull request #3073 from tweag/machine-logs ... Add an option to print the logs in a machine-readable format	2020-06-11 15:45:18 +02:00
Eelco Dolstra	dd9bb11d0d	Move names.{cc,hh} to libstore	2020-06-11 15:42:18 +02:00
Eelco Dolstra	b9ae1bdd7a	Merge pull request #3655 from zimbatm/hash-encoding-prepare ... libutils/hash: remove default encoding	2020-06-10 11:48:38 +02:00
Matthew Bauer	b2c8061b44	Disable extra-platforms = i686-linux on wsl1 (#3676) ... WSL1 doesn’t support i686-linux emulation, see https://github.com/microsoft/wsl/issues/2468	2020-06-09 21:53:53 +00:00
Eelco Dolstra	29e0748847	Show HTTP status message ... For example: warning: unable to download 'https://api.github.com/repos/edolstra/dwarffs/commits/master': HTTP error 403 ('rate limit exceeded'); using cached version	2020-06-09 14:20:22 +02:00
Eelco Dolstra	447ea52b07	FileTransfer: Don't store status since curl already does that	2020-06-09 14:05:15 +02:00
regnat	4983401440	Unify the printing of the logs between bar-with-logs and raw ... Make the printing of the build logs systematically go through the logger, and replicate the behavior of `no-build-output` by having two different loggers (one that prints the build logs and one that doesn't)	2020-06-08 09:31:15 +02:00
Carlo Nucera	d614166cb6	Fix condition error and make test suite pass	2020-06-04 17:21:21 -04:00
John Ericson	e5cc1ebc5d	Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output	2020-06-04 21:04:35 +00:00
John Ericson	94ddea9e2f	Use readString rather than >> temporary ... Fixed the rest of these before, but this one slipped through.	2020-06-04 20:55:08 +00:00
John Ericson	2041499b5e	Flip boolean ... Thanks Matt!	2020-06-04 20:42:25 +00:00
John Ericson	ed86acf02a	Use some std::optional::has_value for clarity	2020-06-04 20:42:02 +00:00
Matthew Bauer	2299ef705c	Add error message when FileIngestionMethod is out of bounds ... bool coerces anything >0 to true, but in the future we may have other file ingestion methods. This shows a better error message when the “recursive” byte isn’t 1.	2020-06-04 11:32:39 -05:00
John Ericson	53bc8ff152	No C++ designated initializers yet with Clang 7	2020-06-03 20:45:14 -04:00
Ben Burdette	4335ba999b	Merge remote-tracking branch 'upstream/master' into errors-phase-2	2020-06-03 17:00:00 -06:00
Carlo Nucera	132d6f2c24	Clarify the description of StorePath construction	2020-06-03 16:08:32 -04:00
John Ericson	74b251b2f3	Don't anticipate multiple CA outputs for now	2020-06-03 18:53:04 +00:00
John Ericson	3a9e4c3262	Don't anticipate CA but not fixed outputs for now	2020-06-03 18:50:45 +00:00
John Ericson	2500403059	Use enum and predicates rather than bitfile for derivation type	2020-06-03 17:38:54 +00:00
John Ericson	6b7f4ec4ab	Merge remote-tracking branch 'upstream/master' into ca-drv	2020-06-03 16:36:06 +00:00
Eelco Dolstra	c20591ddc3	Merge remote-tracking branch 'origin/master' into flakes	2020-06-03 16:15:22 +02:00
zimbatm	6ee03b8444	libutils/hash: remove default encoding ... This will make it easier to reason about the hash encoding and switch to SRI everywhere where possible.	2020-06-03 13:49:51 +02:00
John Ericson	3c78ac348c	Merge remote-tracking branch 'obsidian/no-hash-type-unknown' into validPathInfo-ca-proper-datatype	2020-06-03 04:44:24 +00:00
John Ericson	406dbb7fce	outputHashAlgo can be blank so parse accordingly ... It is blank for SRI hashes.	2020-06-02 21:09:15 +00:00
Carlo Nucera	75d2581390	Typo	2020-06-02 16:21:18 -04:00
Carlo Nucera	78f137e931	Validate text version instead, throw Errors	2020-06-02 16:20:22 -04:00
Carlo Nucera	a5cdf1867e	Add assertions for SHA256 in fixed case	2020-06-02 16:13:08 -04:00
Carlo Nucera	fd2eb41e64	Move file-hash to content-address	2020-06-02 15:44:58 -04:00
Carlo Nucera	343c20a404	WIP Completed implementation	2020-06-02 15:23:21 -04:00
Carlo Nucera	390bf64858	WIP	2020-06-02 14:15:58 -04:00
John Ericson	a33270ce1d	Clean up ValidPathInfo::isContentAddressed with std::visit	2020-06-02 17:04:21 +00:00
John Ericson	25e61812f3	Apply suggestions from code review ... Co-authored-by: Matthew Bauer <mjbauer95@gmail.com>	2020-06-02 12:47:18 -04:00
John Ericson	64cffb804a	Merge remote-tracking branch 'upstream/master' into no-hash-type-unknown	2020-06-02 16:07:25 +00:00
John Ericson	450dcf2c1b	Remove HashType::Unknown ... Instead, `Hash` uses `std::optional<HashType>`. In the future, we may also make `Hash` itself require a known hash type, encoraging people to use `std::optional<Hash>` instead.	2020-06-02 15:52:13 +00:00
Carlo Nucera	a5d820a0a3	Change parseCa(Opt) to parseContentAddress(Opt)	2020-06-02 11:00:10 -04:00
John Ericson	1b6461f671	Merge remote-tracking branch 'upstream/master' into validPathInfo-ca-proper-datatype	2020-06-02 14:31:18 +00:00
Ben Burdette	d82d230b40	elide the 'ErrorInfo' in logError and logWarning calls	2020-06-02 08:22:24 -06:00
John Ericson	efcd30da89	WIP	2020-06-02 00:37:43 +00:00
John Ericson	754c910953	WIP more progress	2020-06-01 19:26:40 -04:00
Carlo Nucera	da39092a39	WIP	2020-06-01 18:53:31 -04:00
Carlo Nucera	0e9438b6d3	Create new file-hash files	2020-06-01 17:32:40 -04:00
Carlo Nucera	0cb67ecbd3	Merge branch 'derivation-header-include-order' of github.com:Ericson2314/nix into validPathInfo-ca-proper-datatype	2020-06-01 17:13:11 -04:00
Carlo Nucera	f4b89e11a4	Merge branch 'no-stringly-typed-derivation-output' of github.com:Ericson2314/nix into validPathInfo-ca-proper-datatype	2020-06-01 17:12:50 -04:00
John Ericson	fac0c2d54a	Remove addToStore variant as requested by FIXME ... The idea is it's always more flexible to consumer a `Source` than a plain string, and it might even reduce memory consumption. I also looked at `addToStoreFromDump` with its `// FIXME: remove?`, but the worked needed for that is far more up for interpretation, so I punted for now.	2020-05-29 17:02:32 -04:00
Ben Burdette	734283d636	Merge remote-tracking branch 'upstream/master' into errors-phase-2	2020-05-29 09:51:37 -06:00
Carlo Nucera	6dd471ebf6	Fixing the result of merge	2020-05-28 12:14:36 -04:00
Carlo Nucera	4f597fb901	Merge branch 'master' of github.com:NixOS/nix into enum-class	2020-05-28 10:58:22 -04:00
John Ericson	5b4cd84bc2	Merge remote-tracking branch 'me/more-rust-ffi' into no-stringly-typed-derivation-output	2020-05-28 10:35:53 -04:00
John Ericson	ef71caba29	Merge remote-tracking branch 'upstream/master' into more-rust-ffi	2020-05-28 10:31:46 -04:00
Eelco Dolstra	17ca997fc6	Merge remote-tracking branch 'origin/master' into flakes	2020-05-28 12:55:24 +02:00
Matthew Bauer	c66441a646	Rename some variables named “recursive” to “method” ... This is much less confusing since recursive is no longer a boolean.	2020-05-27 13:21:26 -05:00
Carlo Nucera	d49e65ba9d	Merge remote-tracking branch 'john-ericson/enum-FileIngestionMethod' into no-stringly-typed-derivation-output	2020-05-26 12:30:48 -04:00
Carlo Nucera	0f3f901071	Merge remote-tracking branch 'origin/master' into more-rust-ffi	2020-05-26 11:46:42 -04:00
Carlo Nucera	6d73c10041	Merge remote-tracking branch 'origin/master' into enum-FileIngestionMethod	2020-05-26 11:14:08 -04:00
Eelco Dolstra	5ef64f05e6	Cleanup	2020-05-18 15:50:29 +02:00
Eelco Dolstra	0ed946aa61	Merge branch 'wait-for-builders' of https://github.com/serokell/nix	2020-05-18 13:48:45 +02:00
Ben Burdette	92123c6c79	Merge remote-tracking branch 'upstream/master' into errors-phase-2	2020-05-15 07:00:36 -06:00
Alexander Bantyev	183dd28266	Don't lock a user while doing remote builds	2020-05-14 17:00:54 +03:00
Ben Burdette	ef9dd9f9bc	formatting and a few minor changes	2020-05-13 15:56:39 -06:00
Ben Burdette	c79d4addab	consistent capitalization	2020-05-13 10:02:18 -06:00
Ben Burdette	bfca5fc395	change status messages to info level	2020-05-13 09:52:36 -06:00
Ben Burdette	19cffc29c9	remove unused extra json fields	2020-05-12 12:09:12 -06:00
Ben Burdette	ec870b9c85	new pos format for more errors	2020-05-12 10:52:26 -06:00
Eelco Dolstra	ebc024df22	Show hint how to enable experimental features	2020-05-12 15:47:09 +02:00
Ben Burdette	b93c1bf3d6	fixes to merged code	2020-05-11 15:52:15 -06:00
Ben Burdette	59b1f5c701	Merge branch 'master' into errors-phase-2	2020-05-11 14:35:30 -06:00
Ben Burdette	958e81987b	switch from printError warnings to logWarnings	2020-05-11 13:02:16 -06:00
Domen Kožar	5bdb67c843	Merge pull request #3568 from kolloch/outputHashModeError ... libstore/build.cc: more explicit error about form of output	2020-05-11 18:14:32 +02:00
Domen Kožar	1d8144e36b	Update src/libstore/build.cc	2020-05-11 18:14:23 +02:00
Benjamin Hipple	146f9c114f	doc: consistently refer to 'fixed-output' with a dash ... General cleanup that makes it easier to search for the term.	2020-05-09 10:58:43 -04:00
Alexander Bantyev	772e5db828	Mention build users in the 'waiting for' message	2020-05-08 12:29:00 +03:00
Alexander Bantyev	14073fb76b	Don't block while waiting for build users	2020-05-08 12:22:39 +03:00
Ben Burdette	e3901638b5	todo removal	2020-05-06 15:01:13 -06:00
Eelco Dolstra	2f8ee4578f	Merge remote-tracking branch 'origin/master' into flakes	2020-05-06 12:01:40 +02:00
Eelco Dolstra	74a1bfdcab	Merge pull request #3546 from guibou/nix_readfile_on_0_sized_files ... builtins.readFile: do not truncate content	2020-05-06 11:33:55 +02:00
Peter Kolloch	9be46859a9	libstore/build.cc: more explicit about form of output ... Be more explicit about why we expect a regular file as output when outputHashMode=flat for a fixed output derivation.	2020-05-06 11:21:12 +02:00
Eelco Dolstra	02c5914ea4	Merge pull request #3562 from pikajude/master ... Use fragment size for autoGC capacity calculation	2020-05-06 11:19:30 +02:00
Eelco Dolstra	fd4911269f	Revert "Merge pull request #3558 from LnL7/ssh-ng-stderr" ... This reverts commit 3ebfbecdd1, reversing changes made to c089c52d5f. https://github.com/NixOS/nix/pull/3558	2020-05-06 10:54:18 +02:00
Eelco Dolstra	6f3244ce45	Merge remote-tracking branch 'origin/master' into flakes	2020-05-05 18:59:33 +02:00
Alexander Bantyev	04967dee9d	Wait for build users when none are available	2020-05-05 13:04:36 +03:00
Ben Burdette	7ffb5efdbc	appending to hints; remove _printError	2020-05-04 16:19:57 -06:00
Jude Taylor	e2fc575c61	nix auto-gc: use fragment size	2020-05-04 14:42:06 -07:00
Eelco Dolstra	a721a0b114	Flag: Use designated initializers	2020-05-04 22:40:19 +02:00
Ben Burdette	ab6f0b9641	convert some printError calls to logError	2020-05-03 08:01:25 -06:00
Daiderd Jordan	f16e24f95e	remote-store: don't log raw stderr by default ... For remote stores the log messages are already forwarded as structured STDERR_RESULT messages so the old format is duplicate information. But still included with -vvv since it could be useful for debugging problems. $ nix build -L /nix/store/nl71b2niws857ffiaggyrkjwgx9jjzc0-foo.drv --store ssh-ng://localhost Hello World! foo> Hello World! [1/0/1 built] building foo Fixes #3556	2020-05-02 23:40:50 +02:00
Ben Burdette	4b99c09f5c	convert some errors	2020-05-01 14:32:06 -06:00
Eelco Dolstra	941f95284a	Merge remote-tracking branch 'origin/master' into flakes	2020-05-01 11:59:56 +02:00
Ben Burdette	39ff80d031	errorinfo constructor test	2020-04-29 18:57:05 -06:00
Guillaume Bouchard	7afcb5af98	Remove the drain argument from readFile ... Now it is always `drain` (see previous commit).	2020-04-29 18:43:45 +02:00
Antoine Eiche	ca93b26db6	Only call grantpt on MacOS systems ... The commit 3cc1125595 adds a `grantpt` call on the builder pseudo terminal fd. This call is actually only required for MacOS, but it however requires a RW access to /dev/pts which is only RO bindmounted in the Bazel Linux sandbox. So, Nix can not be actually run in the Bazel Linux sandbox for unneeded reasons.	2020-04-29 15:43:20 +02:00
Ben Burdette	22e6490311	Error classname as name	2020-04-28 21:06:08 -06:00
Eelco Dolstra	ef4d3fc111	Merge remote-tracking branch 'origin/master' into flakes	2020-04-23 15:16:18 +02:00
Dustin DeWeese	c0d940978a	Replace select() with poll() to allow waiting on more than FD_SETSIZE fds	2020-04-21 16:21:28 -07:00
Ben Burdette	e4fb9a3849	remove 'format' from Error constructor calls	2020-04-21 17:07:07 -06:00
Ben Burdette	15e9564fd1	logEI for tunnelLogger and progressbar	2020-04-19 17:16:51 -06:00
Ben Burdette	3d5b1032a1	logError, logWarning; Logger functions; switch to Verbosity enum	2020-04-17 15:07:44 -06:00
Eelco Dolstra	aaa109565e	Use a more space/time-efficient representation for the eval cache	2020-04-17 23:04:21 +02:00
Eelco Dolstra	7a9687ba30	SQLiteStmt: Use std::string_view	2020-04-17 01:00:56 +02:00
Eelco Dolstra	0858738355	Merge remote-tracking branch 'origin/master' into flakes	2020-04-16 18:27:37 +02:00
Eelco Dolstra	a118293bd0	Merge pull request #3458 from zimbatm/nix-user-conf-dir ... NIX_USER_CONF_FILES	2020-04-15 13:00:28 +02:00
zimbatm	895516cadf	add NIX_USER_CONF_FILES ... Motivation: maintain project-level configuration files. Document the whole situation a bit better so that it corresponds to the implementation, and add NIX_USER_CONF_FILES that allows overriding which user files Nix will load during startup.	2020-04-14 18:45:06 +02:00
Eelco Dolstra	c0c2cb871d	Merge remote-tracking branch 'origin/master' into flakes	2020-04-14 13:02:55 +02:00
Bruce Toll	8132d0a12e	Fix nix-build --check -K in sandbox w/o root ... Temporarily add user-write permission to build directory so that it can be moved out of the sandbox to the store with a .check suffix. This is necessary because the build directory has already had its permissions set read-only, but write permission is required to update the directory's parent link to move it out of the sandbox. Updated the related --check "derivation may not be deterministic" messages to consistently use the real store paths. Added test for non-root sandbox nix-build --check -K to demonstrate issue and help prevent regressions.	2020-04-10 16:23:10 -04:00
Domen Kožar	db25a6d7bb	Merge pull request #2689 from tollb/fix/delete_tmp_dir_when_build_check_ok ... Delete temporary directory on successful build	2020-04-10 09:47:57 +02:00
Bruce Toll	16a4864759	Delete temporary directory on successful build ... With --check and the --keep-failed (-K) flag, the temporary directory was being retained regardless of whether the build was successful and reproducible. This removes the temporary directory, as expected, on a reproducible check build. Added tests to verify that temporary build directories are not retained unnecessarily, particularly when using --check with --keep-failed.	2020-04-09 16:37:41 -04:00
Philipp Middendorf	04bedda0b6	gc.cc: Ignore hidden files in temproots	2020-04-09 17:05:29 +02:00
Eelco Dolstra	d103c79144	Merge remote-tracking branch 'origin/master' into flakes	2020-04-09 15:54:21 +02:00
Nikola Knezevic	f5095594e7	datatransfer.{cc,hh} -> filetransfer.{cc,hh}	2020-04-08 22:26:57 +02:00
Nikola Knezevic	c330109bfa	DataTransfer -> FileTransfer	2020-04-08 22:26:57 +02:00
Nikola Knezevic	7848372b0f	Add upload method	2020-04-08 22:26:57 +02:00
Nikola Knezevic	a0c5931208	actDownload -> actDataTransfer	2020-04-08 22:26:57 +02:00
Nikola Knezevic	c4c1ae0a00	DownloadError -> DataTransferError	2020-04-08 22:26:57 +02:00
Nikola Knezevic	213d124277	DownloadItem -> TransferItem	2020-04-08 22:26:57 +02:00
Nikola Knezevic	cd391206e6	{get,make,new}Downloader -> DataTransfer	2020-04-08 22:26:57 +02:00
Nikola Knezevic	2df2741ec6	enqueueDownload -> enqueueDataTransfer	2020-04-08 22:26:57 +02:00
Nikola Knezevic	142ed7fe45	DownloadResult -> DataTransferResult	2020-04-08 22:26:57 +02:00
Nikola Knezevic	e5cc53beec	DownloadSettings -> DataTransferSettings	2020-04-08 22:26:57 +02:00
Nikola Knezevic	741e9012d3	Rename src/lib/download.* to src/lib/datatransfer.*	2020-04-08 22:26:57 +02:00
Nikola Knezevic	65ef57e0cb	DownloadRequest -> DataTransferRequest	2020-04-08 22:26:57 +02:00
Domen Kožar	bf81b31559	build.cc: improve message if home directory exists	2020-04-08 18:27:10 +02:00
Eelco Dolstra	1ab8d6ac18	Downloader: Only write data to the sink on a 200 response ... Hopefully fixes #3278.	2020-04-08 15:27:09 +02:00
Eelco Dolstra	54955867a6	Merge remote-tracking branch 'origin/master' into flakes	2020-04-07 14:29:45 +02:00
Eelco Dolstra	cd39709003	Cleanup	2020-04-07 09:27:17 +02:00
Eelco Dolstra	462421d345	Backport libfetchers from the flakes branch ... This provides a pluggable mechanism for defining new fetchers. It adds a builtin function 'fetchTree' that generalizes existing fetchers like 'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a set of attributes, e.g. fetchTree { type = "git"; url = "https://example.org/repo.git"; ref = "some-branch"; rev = "abcdef..."; } The existing fetchers are just wrappers around this. Note that the input attributes to fetchTree are the same as flake input specifications and flake lock file entries. All fetchers share a common cache stored in ~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}). This also adds support for Git worktrees (c169ea5904).	2020-04-07 09:03:14 +02:00
Cole Helbling	c976cb0b8a	Don't retry on "unsupported protocol" error ... When encountering an unsupported protocol, there's no need to retry. Chances are, it won't suddenly be supported between retry attempts; error instead. Otherwise, you see something like the following: $ nix-env -i -f git://git@github.com/foo/bar warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 335 ms warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 604 ms warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 1340 ms warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 2685 ms With this change, you now see: $ nix-env -i -f git://git@github.com/foo/bar error: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1)	2020-04-05 09:00:34 -07:00
Eelco Dolstra	4fdec5f61d	Merge remote-tracking branch 'origin/master' into flakes	2020-04-01 00:03:50 +02:00
mlatus	12556e5709	fix placeholder not substituted in passAsFile	2020-03-31 19:40:16 +08:00
John Ericson	51afea3af2	Never cast FileIngestionMethod to or from boolean	2020-03-30 22:31:51 +00:00
John Ericson	c251b011cd	Merge remote-tracking branch 'upstream/master' into enum-FileIngestionMethod	2020-03-30 18:16:44 -04:00
John Ericson	832bd534dc	Store parsed hashes in DerivationOutput ... It's best to detect invalid data as soon as possible, with data types that make storing it impossible.	2020-03-30 11:33:35 -04:00
John Ericson	f5494d9442	Merge remote-tracking branch 'me/enum-FileIngestionMethod' into HEAD	2020-03-30 11:08:13 -04:00
Eelco Dolstra	d15d91cad1	Makefile cleanup	2020-03-30 16:50:13 +02:00
Eelco Dolstra	e322a16523	Remove global -I flags ... (cherry picked from commit 2c692a3b14)	2020-03-30 15:30:19 +02:00
Eelco Dolstra	2c692a3b14	Remove global -I flags	2020-03-30 14:39:33 +02:00
Eelco Dolstra	e0a0ae0467	Move fetchers from libstore to libfetchers	2020-03-30 14:04:53 +02:00
Eelco Dolstra	2287cc6486	Fix segfault	2020-03-30 13:31:55 +02:00
John Ericson	225e62a56a	Replace some bool recursive with a new FileIngestionMethod enum	2020-03-29 15:16:20 -04:00
John Ericson	87b32bab05	Use enum struct and drop prefixes ... This does a few enums; the rest will be gotten in subsequent commits.	2020-03-29 11:23:15 -04:00
Eelco Dolstra	2287e2f279	nix flake info: Show flake subdirectory	2020-03-28 18:05:50 +01:00
John Ericson	e433d4af4c	Extend Rust FFI ... Do idiomatic C++ copy and move constructors for a few things, so wrapping structs' defaults can work.	2020-03-25 16:12:14 -04:00
John Ericson	bcde5456cc	Flip dependency so store-api.hh includes derivations.hh ... I think it makes more sense to define the data model (derivations), before the operations (store api).	2020-03-24 20:39:45 +00:00
Eelco Dolstra	0a10854f85	Misc changes from the flakes branch	2020-03-24 14:34:47 +01:00
Eelco Dolstra	777e21e596	nix path-info --json: Print hash in SRI format ... (cherry picked from commit 442e665d6d)	2020-03-24 14:26:23 +01:00
Eelco Dolstra	f9611c7ae4	buildenv: Eliminate global variables, other cleanup ... (cherry picked from commit b82f75464d)	2020-03-24 14:06:47 +01:00
Eelco Dolstra	edc34cc1a2	Add function for quoting strings ... (cherry picked from commit 7dcf5b011a)	2020-03-24 13:44:04 +01:00
Eelco Dolstra	5a7e7fc35f	Use std::string_view ... (cherry picked from commit 6529490cc1)	2020-03-24 13:26:37 +01:00
Eelco Dolstra	7abe3bde8a	Merge remote-tracking branch 'origin/master' into flakes	2020-03-23 13:27:54 +01:00
Eelco Dolstra	5885e20404	Merge pull request #3429 from LnL7/darwin-sandbox ... darwin sandbox	2020-03-23 09:13:36 +01:00
Daiderd Jordan	2e9bc1245c	sandbox: fix /bin/sh on catalina ... Sadly 10.15 changed /bin/sh to a shim which executes bash, this means it can't be used anymore without also opening up the sandbox to allow bash. Failed to exec /bin/bash as variant for /bin/sh (1: Operation not permitted).	2020-03-20 22:12:30 +01:00
Daiderd Jordan	f6c122aaeb	sandbox: allow pty devices ... Nix now runs builds with a pseudo-terminal to enable colored build output.	2020-03-20 21:58:45 +01:00
Daiderd Jordan	7f2df903d9	libstore: relax default sandbox-paths on darwin	2020-03-20 21:31:20 +01:00
Daiderd Jordan	afb78ebd34	libstore: disable resolve-system-dependencies hook ... This is used to determine the dependency tree of impure libraries so nix knows what paths to open in the sandbox. With the less restrictive defaults it isn't needed anymore.	2020-03-20 21:21:56 +01:00
John Ericson	e317324236	Apply suggestions from code review	2020-03-19 23:38:51 -04:00
John Ericson	d5b3328dd1	Apply suggestions from code review ... Co-Authored-By: Cole Helbling <cole.e.helbling@outlook.com>	2020-03-19 23:37:52 -04:00
John Ericson	f1cf3ab870	hashDerivationModulo: Generalize for multiple fixed ouputs per drv ... See documentattion in header and comments in implementation for details. This is actually done in preparation for floating ca derivations, not multi-output fixed ca derivations, but the distinction doesn't yet mattter. Thanks @cole-h for finding and fixing a bunch of typos.	2020-03-19 10:30:49 -04:00
Eelco Dolstra	8aa354fdfd	Register flake-registry.json as a GC root again	2020-03-19 11:45:34 +01:00
Eelco Dolstra	1c127e6a82	downloadFile(): Use expired file if the download fails	2020-03-19 11:42:50 +01:00
John Ericson	049179ba07	Fix typos ... Thanks @asymmetric I failed to do them all in one batch Co-Authored-By: asymmetric <lorenzo@mailbox.org>	2020-03-18 19:07:05 -04:00
Eelco Dolstra	f6ddf48882	Get rid of downloadCached() ... Everything uses the generic caching system now.	2020-03-18 17:24:19 +01:00
Eelco Dolstra	c5ec95e2c7	tarball.cc: Use ETags	2020-03-18 15:14:23 +01:00
Eelco Dolstra	1e7ce1d6da	tarball / github fetchers: Use generic caching system	2020-03-18 14:08:25 +01:00
Eelco Dolstra	38e360154d	Git: Use unified caching system	2020-03-17 22:35:29 +01:00
Eelco Dolstra	d1165d8791	Require shallow clones to be requested explicitly ... If you do a fetchTree on a Git repository, whether the result contains a revCount attribute should not depend on whether that repository happens to be a shallow clone or not. That would complicate caching a lot and would be semantically messy. So applying fetchTree/fetchGit to a shallow repository is now an error unless you pass the attribute 'shallow = true'. If 'shallow = true', we don't return revCount, even if the repository is not actually shallow. Note that Nix itself is not doing shallow clones at the moment. But it could do so as an optimisation if the user specifies 'shallow = true'. Issue #2988.	2020-03-17 22:35:29 +01:00
Eelco Dolstra	2a4e4f6a6e	Unified fetcher caching system	2020-03-17 22:35:29 +01:00
John Ericson	e5178fd22d	Fix typos ... Thanks @asymmetric! Co-Authored-By: asymmetric <lorenzo@mailbox.org>	2020-03-16 16:40:13 -04:00
Eelco Dolstra	fbcb897e21	Add a test for shallow Git clones ... Also, don't return a revCount anymore for shallow or dirty Git trees, since it's incorrect. Closes #2988.	2020-03-16 13:20:32 +01:00
John Ericson	2be64efb02	Generalize isFixedOutput in preparation for CA drvs ... Today's fixed output derivations and regular derivations differ in a few ways which are largely orthogonal. This replaces `isFixedOutput` with a `type` that returns an enum of possible combinations.	2020-03-15 11:05:37 -04:00
Eelco Dolstra	0c2088d438	Merge remote-tracking branch 'origin/master' into flakes	2020-03-13 17:03:30 +01:00
Eelco Dolstra	b816515f61	Fix ca-references feature check ... Fixes #3406.	2020-03-13 13:15:51 +01:00
Robert Hensing	3f55f8a8fb	pathInfoCache: Respect disk cache TTLs #3398	2020-03-12 10:30:28 +01:00
Eelco Dolstra	e188fe7c6d	Move call-flake.nix into libexpr	2020-03-11 17:04:51 +01:00
Eelco Dolstra	73b6d87e17	Merge remote-tracking branch 'origin/master' into flakes	2020-03-04 13:58:42 +01:00
Eelco Dolstra	d37dc71e3c	nix-build: Fix !<output> handling ... This was broken by 22a754c091. https://hydra.nixos.org/eval/1573669	2020-03-04 13:56:17 +01:00
Eelco Dolstra	75db069f92	Optimise Derivation::unparse() ... In nix-instantiate --dry-run '<nixpkgs/nixos/release-combined.nix>' -A nixos.tests.simple.x86_64-linux this reduces time spent in unparse() from 9.15% to 4.31%. The main culprit was appending characters one at a time to the destination string. Even though the string has enough capacity, push_back() still needs to check this on every call.	2020-03-04 11:44:45 +01:00
Eelco Dolstra	22a754c091	Fix GC failures on bad store path names ... It failed on names like '/nix/store/9ip48nkc9rfy0a4yaw98lp6gipqlib1a-'.	2020-02-28 18:07:10 +01:00
Eelco Dolstra	4a4521f462	Fix nlohmann::json exception	2020-02-20 23:47:02 +01:00
Eelco Dolstra	73c9840569	Restore subdir -> dir ... Got this mixed up somewhere.	2020-02-20 23:44:06 +01:00
Eelco Dolstra	d068f9ffff	Restore subdir support in registries ... Hacky...	2020-02-20 22:14:44 +01:00
Eelco Dolstra	1d99c4ab25	Merge pull request #3229 from Ma27/flakes-fetchgit-worktree-support ... builtins.fetchGit: Fix build when fetching a git worktree	2020-02-19 14:11:18 +01:00
Maximilian Bosch	c169ea5904	builtins.fetchGit: Fix build when fetching a git worktree ... Worktrees[1] are a feature of git which allow you to check out a ref in a different directory. While playing around with flakes I realized that git repositories in a worktree checkout break when trying to build a flake: ``` $ git worktree add ../nixpkgs-flakes nixpkgs-flakes $ cd ../nixpkgs-flakes $ nix build .#hello error: opening directory '/home/ma27/Projects/nixpkgs-flakes/.git/refs/heads': Not a directory ``` This issue has been fixed by determining with `git rev-parse --git-common-dir` where the actual `.git` directory is. Please note that this issue only exists on the `flakes` branch, fetching worktree checkouts with Nix master seems to work fine. [1] https://git-scm.com/docs/git-worktree	2020-02-19 14:00:36 +01:00
Eelco Dolstra	f3505a7899	Merge remote-tracking branch 'origin/master' into flakes	2020-02-19 12:57:45 +01:00
Eelco Dolstra	c4d3674de6	Merge pull request #3353 from tbsmoest/priv_tobias_pr_set_deathsig-1.4 ... Fix PR_SET_PDEATHSIG results in Broken pipe (#2395)	2020-02-19 12:29:12 +01:00
Eelco Dolstra	553e584f92	LocalStore::checkDerivationOutputs(): Improve error message	2020-02-18 17:51:48 +01:00
Eelco Dolstra	b5e3c04c03	Fix URL parser ... Fixes #3361.	2020-02-18 12:51:26 +01:00
Eelco Dolstra	6529490cc1	nix eval-hydra-jobs: Support job names as aggregate constituents ... Fixes https://github.com/NixOS/hydra/issues/715.	2020-02-17 15:53:59 +01:00
Eelco Dolstra	8a78bcf6a2	LocalStore::checkDerivationOutputs(): Improve error message	2020-02-17 15:46:07 +01:00
Eelco Dolstra	6208d24c38	Merge remote-tracking branch 'origin/master' into flakes	2020-02-14 22:42:35 +01:00
Tobias Möst	3e347220c8	Fix PR_SET_PDEATHSIG results in Broken pipe (#2395) ... The ssh client is lazily started by the first worker thread, that requires a ssh connection. To avoid the ssh client to be killed, when the worker process is stopped, do not set PR_SET_PDEATHSIG.	2020-02-14 07:51:44 +01:00
Eelco Dolstra	d8972317fc	Prevent uninitialized StorePath creation	2020-02-13 16:12:16 +01:00
Eelco Dolstra	d4df99a334	Parse narHash attribute for all input types	2020-02-11 23:53:24 +01:00