aszlig
525b38eee8
Fix unspecified behaviour in readStorePathCAMap
...
When deploying a Hydra instance with current Nix master, most builds
would not run because of errors like this:
queue monitor: error: --- Error --- hydra-queue-runner
error: --- UsageError --- nix-daemon
not a content address because it is not in the form '<prefix>:<rest>': /nix/store/...-somedrv
The last error message is from parseContentAddress, which expects a
colon-separated string, however what we got here is a store path.
Looking at the worker protocol, the following message sent to the Nix
daemon caused the error above:
0x1E -> wopQuerySubstitutablePathInfos
0x01 -> Number of paths
0x16 -> Length of string
"/nix/store/...-somedrv"
0x00 -> Length of string
""
Looking at writeStorePathCAMap, the store path is indeed the first field
that's transmitted. However, readStorePathCAMap expects it to be the
*second* field *on my machine*, since expression evaluation order is a
classic form of unspecified behaviour[1] in C++.
This has been introduced in https://github.com/NixOS/nix/pull/3689 ,
specifically in commit 66a62b3189https://en.wikipedia.org/wiki/Unspecified_behavior#Order_of_evaluation_of_subexpressions
Signed-off-by: aszlig <aszlig@nix.build>
2020-09-13 02:40:51 +02:00
John Ericson
98dfd7531d
Fix querying outputs for CA derivations some more
...
If we resolve using the known path of a derivation whose output we
didn't have, we previously blew up. Now we just fail gracefully,
returning the map of all outputs unknown.
2020-09-04 18:33:58 +00:00
John Ericson
075d399e3f
Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream
2020-09-04 16:04:35 +00:00
John Ericson
e9fad3006b
Fix some of the issues raised by @edolstra
...
- More and better comments
- The easier renames
2020-09-04 15:18:25 +00:00
John Ericson
e86dd59dcc
Apply suggestions from code review
...
Thanks!
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-09-04 10:48:50 -04:00
John Ericson
c9f1ed912c
Don't chmod symlink before moving outputs around
...
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-09-04 14:41:53 +00:00
John Ericson
aad4abcc9c
Fix floating CA tests
...
We will sometimes try to query the outputs of derivations we can't
resolve. That's fine; it just means we don't know what those outputs are
yet.
2020-09-04 01:17:38 +00:00
John Ericson
975a47f7fe
Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream
2020-09-03 22:09:04 +00:00
John Ericson
e7d93e7ece
Merge remote-tracking branch 'upstream/master' into single-ca-drv-build
2020-09-03 15:43:17 +00:00
Eelco Dolstra
8a945d6ddb
Remove gc-check-reachability
2020-09-03 11:30:15 +02:00
Eelco Dolstra
82b77a7726
addPermRoot(): Remove indirect flag
2020-09-03 11:29:34 +02:00
Eelco Dolstra
b07167be5a
createGeneration(): Always create an indirect root
...
This means profiles outside of /nix/var/nix/profiles don't get
garbage-collected. It also means we don't need to scan
/nix/var/nix/profiles for GC roots anymore, except for compatibility
with previously existing generations.
2020-09-03 11:13:53 +02:00
Eelco Dolstra
b74f5cdd23
createGeneration(): Take a StorePath
2020-09-03 11:06:56 +02:00
Domen Kožar
dd4b56c87f
Allow HTTP binary cache to request absolute uris
2020-09-01 21:35:48 +02:00
John Ericson
ef278d00f9
Merge remote-tracking branch 'upstream/master' into single-ca-drv-build
2020-09-01 18:01:48 +00:00
Eelco Dolstra
84f5cabbea
Merge remote-tracking branch 'origin/master' into markdown
2020-08-31 14:24:26 +02:00
John Ericson
02e0001fc0
Merge branch 'ca-floating-upstream' of github.com:obsidiansystems/nix into ca-floating-upstream
2020-08-28 21:23:09 +00:00
John Ericson
421ed527c7
Update src/libstore/build.cc
...
Thanks for catching, @regnat.
2020-08-28 17:22:57 -04:00
John Ericson
48e8828050
Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream
2020-08-28 20:38:25 +00:00
John Ericson
8017fe7487
Merge remote-tracking branch 'upstream/master' into single-ca-drv-build
2020-08-28 19:59:14 +00:00
Eelco Dolstra
eb75282b8d
Merge pull request #3434 from Ericson2314/derivation-header-include-order
...
Revise division of labor in deserialization of derivations
2020-08-27 16:39:28 +02:00
Eelco Dolstra
a0f19d9f3a
RemoteStore::addToStore(): Fix race between stderrThread and NAR writer
...
As pointed out by @B4dM4n, the call to to.flush() on stderrThread is
unsafe because the NAR writer thread is also writing to 'to'.
Fixes #3943 .
2020-08-27 14:50:51 +02:00
Eelco Dolstra
4bf5faf416
Merge remote-tracking branch 'origin/master' into markdown
2020-08-25 19:47:34 +02:00
John Ericson
8eb73a8724
CA derivations that depend on other CA derivations
...
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-08-24 19:01:15 +00:00
John Ericson
e0b0e18905
Add constructor for BasicDerivation -> Derivation
2020-08-24 19:01:07 +00:00
John Ericson
59979e7053
Fix bad debug format string
2020-08-24 19:01:07 +00:00
Eelco Dolstra
6a67e57019
Add DummyStore (dummy://)
...
DummyStore does not allow building or adding paths. This is useful for
evaluation tests when you don't want to initialize a "proper" store.
2020-08-24 18:54:16 +02:00
Eelco Dolstra
2530cc6827
Merge pull request #3950 from obsidiansystems/tabs-spaces
...
tabs -> spaces
2020-08-24 10:20:28 +02:00
John Ericson
b0b59fd05a
Merge remote-tracking branch 'obsidian/write-derivation-borrow' into HEAD
2020-08-23 15:11:10 +00:00
John Ericson
35e6288be1
writeDerivation just needs a plain store reference
2020-08-23 15:01:11 +00:00
John Ericson
3a7b330b64
"Downstream placeholders" should not be store paths
...
Insead they should be opaque `/<hash>` like the placeholders we already
have.
2020-08-21 19:35:35 +00:00
John Ericson
422affe102
tabs -> spaces
...
Sorry I let the tab sneak in there in the first place.
2020-08-21 19:19:24 +00:00
John Ericson
c265e0ea82
Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol
2020-08-20 20:24:22 +00:00
John Ericson
27a3f82c0b
Merge remote-tracking branch 'upstream/master' into single-ca-drv-build
2020-08-20 18:28:17 +00:00
John Ericson
45a2f1baab
Rename drv output querying functions, like master
...
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
- `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap
2020-08-20 18:14:12 +00:00
Eelco Dolstra
4d77513d97
Merge pull request #3859 from obsidiansystems/drv-outputs-map-allow-missing
...
`queryDerivationOutputMap` no longer assumes all outputs have a mapping
2020-08-20 16:49:23 +02:00
John Ericson
9a9d834dc7
Rename drv output querying functions
...
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
- `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap`
2020-08-20 14:12:51 +00:00
Eelco Dolstra
acb99f03f9
Config: Use nlohmann/json
2020-08-20 11:02:16 +02:00
John Ericson
a83694c7a1
Use RemoteStore to open connection for proxying daemon
...
Removes duplicate websocket opening code, and also means we should be
able to to ssh-ssh-... daemon relays, not just uds-uds-... ones.
2020-08-19 19:34:47 +00:00
Eelco Dolstra
c8fa39324a
Generate the nix.conf docs from the source code
...
This means we don't have two (divergent) sets of option descriptions
anymore.
2020-08-19 18:28:04 +02:00
John Ericson
d5af5763cf
Merge branch 'master' of github.com:NixOS/nix into remove-storetype-delegate-regStore
2020-08-19 15:54:50 +00:00
John Ericson
be0d429b95
Merge branch 'master' of github.com:NixOS/nix into templated-daemon-protocol
2020-08-19 03:17:41 +00:00
John Ericson
950ddfdb82
Merge remote-tracking branch 'upstream/master' into derivation-header-include-order
2020-08-18 14:36:44 +00:00
Eelco Dolstra
dfeb76dbf9
Merge pull request #3930 from obsidiansystems/legacy-ssh-build-paths
...
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths`
2020-08-18 16:07:40 +02:00
Eelco Dolstra
7cdc739ece
Merge remote-tracking branch 'origin/master' into markdown
2020-08-17 13:43:39 +02:00
Eelco Dolstra
847a5392f4
Merge branch 'ca-no-need-trust' of https://github.com/obsidiansystems/nix
2020-08-17 10:57:02 +02:00
Eelco Dolstra
3c619f6290
Merge branch 'test-RemoteStore-buildDerivation' of https://github.com/obsidiansystems/nix
2020-08-17 10:53:18 +02:00
John Ericson
6f7ac5e865
Remove extra closing paren
2020-08-14 21:59:31 +00:00
John Ericson
3c8b5b6219
Merge remote-tracking branch 'upstream/master' into single-ca-drv-build
2020-08-14 17:00:13 +00:00
John Ericson
4b571ea321
Update src/libstore/daemon.cc
...
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-08-14 11:52:37 -04:00
Eelco Dolstra
13e49be660
Merge pull request #3875 from obsidiansystems/new-interface-for-path-pathOpt
...
Offer a safer interface for path and pathOpt
2020-08-14 17:19:19 +02:00
Eelco Dolstra
7714d9a943
Merge pull request #3924 from obsidiansystems/features-per-store
...
Make `system-features` a store setting
2020-08-14 17:13:07 +02:00
Eelco Dolstra
9b9d5297ba
Merge pull request #3909 from matthewbauer/readd-hashed-mirrors
...
Add hashed-mirrors back
2020-08-14 17:05:47 +02:00
Eelco Dolstra
d81f13f7cb
Merge pull request #3899 from obsidiansystems/make-narHash-not-optional
...
Make narHash in ValidPathInfo not optional
2020-08-14 17:00:18 +02:00
John Ericson
e1308b1211
Define LegacySSHStore::buildPaths using cmdBuildPaths
...
Evidentally this was never implemented because Nix switched to using
`buildDerivation` exclusively before `build-remote.pl` was rewritten.
The `nix-copy-ssh` test (already) tests this.
2020-08-13 21:27:55 +00:00
John Ericson
5ccd94501d
Allow trustless building of CA derivations
...
Include a long comment explaining the policy. Perhaps this can be moved
to the manual at some point in the future.
Also bump the daemon protocol minor version, so clients can tell whether
`wopBuildDerivation` supports trustless CA derivation building. I hope
to take advantage of this in a follow-up PR to support trustless remote
building with the minimal sending of derivation closures.
2020-08-13 18:15:57 +00:00
John Ericson
85aacbee64
Use TeeSink and TeeSouce in a few more places
2020-08-13 14:51:17 +00:00
John Ericson
d2f2be0f70
Test RemoteStore::buildDerivation
...
Fix `wopNarFromPath` which needed a `toRealPath`.
2020-08-13 04:07:14 +00:00
John Ericson
5d67f18c86
Merge branch 'daemon-auth-cleanup' of github.com:obsidiansystems/nix into HEAD
2020-08-12 18:22:31 +00:00
John Ericson
4720853129
Make system-features a store setting
...
This seems more correct. It also means one can specify the features a
store should support with --store and remote-store=..., which is useful.
I use this to clean up the build remotes test.
2020-08-12 18:13:00 +00:00
John Ericson
8d4162ff9e
Separate auth and logic for the daemon
...
Before, processConnection wanted to know a user name and user id, and
`nix-daemon --stdio`, when it isn't proxying to an underlying daemon,
would just assume "root" and 0. But `nix-daemon --stdio` (no proxying)
shouldn't make guesses about who holds the other end of its standard
streams.
Now processConnection takes an "auth hook", so `nix-daemon` can provide
the appropriate policy and daemon.cc doesn't need to know or care what
it is.
2020-08-12 15:22:33 +00:00
John Ericson
5f80aea795
Break out lambda so output can be matched just once
...
This is much better.
2020-08-12 02:23:31 +00:00
John Ericson
18834f7764
Recheck path validity after acquiring lock
...
It might have changed, and in any event this is how the cod used to work
so let's just keep it.
2020-08-11 23:44:02 +00:00
John Ericson
2de201254e
Don't assume a total output map in two places in build.cc
...
Thanks @regnat for catching one of them. The other follows for many of
the same reasons. I'm find fixing others on a need-to-fix basis,
provided their are no regressions.
2020-08-11 23:07:50 +00:00
John Ericson
4c6aac8fdf
Clarify comment on sandbox and temp fresh paths
2020-08-11 22:46:05 +00:00
John Ericson
6d57139050
Clarify outputReferences variable with self-describing type
...
Thanks for the idea, @Regnat!
2020-08-11 22:34:09 +00:00
John Ericson
8a068bd025
Remove redundant equality check
2020-08-11 21:25:40 +00:00
John Ericson
07e3466eb4
Float comment to out describe gaveUpOnSubstitution in general
2020-08-11 21:16:14 +00:00
John Ericson
d0f6e338dd
Apply suggestions from code review
...
Thanks!!
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-08-11 16:49:10 -04:00
John Ericson
2a0902634e
Fix error in merge breaking floating CA drvs
...
Forgot to add this hunk!
2020-08-11 00:13:19 +00:00
John Ericson
1b5c24662b
Merge branch 'small-drv-serialize-cleanup' of github.com:obsidiansystems/nix into single-ca-drv-build
2020-08-10 01:57:54 +00:00
John Ericson
581183d4d5
Deduplicate parsing and reading derivations
2020-08-10 01:40:50 +00:00
John Ericson
bcd0629c2e
Remove name parameter from writeDerivation
...
The name is now stored with the derivation itself.
2020-08-10 01:35:59 +00:00
John Ericson
e913a2989f
Squashed get CA derivations building
2020-08-07 19:51:55 +00:00
Eelco Dolstra
edfd676e05
Fix .ls file names in binary caches
...
These are not supposed to include the 'name' part of the store
path. This was broken by 759947bf72
2020-08-07 21:18:29 +02:00
John Ericson
8f92bb5ad9
Merge branch 'drv-outputs-map-allow-missing' of github.com:obsidiansystems/nix into templated-daemon-protocol
2020-08-07 18:51:01 +00:00
John Ericson
f7ba16f9cb
Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing
2020-08-07 17:46:39 +00:00
John Ericson
47644e49ca
Specialize std::optional<StorePath> so this is backwards compatible
...
While I am cautious to break parametricity, I think it's OK in this
cases---we're not about to try to do some crazy polymorphic protocol
anytime soon.
2020-08-07 17:05:14 +00:00
Matthew Bauer
96c158d6e1
Fix build
2020-08-06 21:04:31 -05:00
Matthew Bauer
8abc577cc2
Merge remote-tracking branch 'origin/master' into readd-hashed-mirrors
2020-08-06 18:58:59 -05:00
Carlo Nucera
46f9dd56da
Fix bug due to non-deterministic arg eval order
2020-08-06 19:30:05 -04:00
Matthew Bauer
641c950701
Add hashed-mirrors back
...
Some users have their own hashed-mirrors setup, that is used to mirror
things in addition to what’s available on tarballs.nixos.org. Although
this should be feasable to do with a Binary Cache, it’s not always
easy, since you have to remember what "name" each of the tarballs has.
Continuing to support hashed-mirrors is cheap, so it’s best to leave
support in Nix. Note that NIX_HASHED_MIRRORS is also supported in
Nixpkgs through fetchurl.nix.
Note that this excludes tarballs.nixos.org from the default, as in
\#3689. All of these are available on cache.nixos.org.
2020-08-06 18:19:09 -05:00
Carlo Nucera
9ab07e99f5
Use template structs instead of phantoms
2020-08-06 18:04:13 -04:00
Carlo Nucera
3d8240c32e
Remove leftover commented code
2020-08-06 16:04:18 -04:00
Carlo Nucera
f795f0fabc
Merge branch 'drv-outputs-map-allow-missing-namespace' of github.com:obsidiansystems/nix into templated-daemon-protocol
2020-08-06 15:53:09 -04:00
John Ericson
e89b5bd0bf
Minimize the usage of Hash::dummy
2020-08-06 18:31:48 +00:00
John Ericson
5e59b25a23
Merge remote-tracking branch 'upstream/master' into make-narHash-not-optional
2020-08-06 15:34:15 +00:00
Carlo Nucera
8b175f58d1
Simplify the namespace
2020-08-05 17:57:07 -04:00
Carlo Nucera
0739d428e0
Solve template deduction problem
...
We had to predeclare our template functions
2020-08-05 17:49:45 -04:00
John Ericson
6c66331d5c
WIP: Put the worker protocol read and write in a namespace to disambig
2020-08-05 20:37:48 +00:00
Carlo Nucera
1d2e80ddd6
Merge branch 'master' of github.com:NixOS/nix into new-interface-for-path-pathOpt
2020-08-05 15:45:33 -04:00
John Ericson
ed96e603e1
Proxy -> Phantom to match Rust
...
Sorry, Haskell.
2020-08-05 19:44:08 +00:00
Carlo Nucera
8241e660ba
Remove Hash::operator bool ()
...
Since the hash is not optional anymore
2020-08-05 15:30:38 -04:00
Carlo Nucera
be6e1c6457
Merge branch 'master' of github.com:NixOS/nix into make-narHash-not-optional
2020-08-05 15:14:47 -04:00
Carlo Nucera
1ad6394b33
Add Hash::dummy to signal default value
...
We did this in the same spirit of the dummy value that's present in
libstore/path.hh
2020-08-05 15:11:49 -04:00
Carlo Nucera
1d71028f4d
Remove optionality in ValidPathInfo::narInfo
2020-08-05 14:42:48 -04:00
John Ericson
7302761f64
Merge remote-tracking branch 'obsidian/drv-outputs-map-allow-missing' into templated-daemon-protocol
2020-08-05 17:53:24 +00:00
Ben Burdette
f1a47a96b6
error messages for issue 2238
2020-08-05 10:58:00 -06:00
John Ericson
cf939055c8
Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing
2020-08-05 16:43:30 +00:00
John Ericson
0559ff3d8b
Merge branch 'master' of github.com:NixOS/nix into derivation-primop-floating-output
2020-08-05 14:56:40 +00:00
John Ericson
92ad550e96
Merge remote-tracking branch 'obsidian/misc-ca' into derivation-primop-floating-output
2020-08-05 14:51:41 +00:00
Eelco Dolstra
790b694be7
Style fix
2020-08-05 16:51:06 +02:00
Eelco Dolstra
e48f944e9d
Merge branch 'misc-ca' of https://github.com/obsidiansystems/nix
2020-08-05 16:50:05 +02:00
John Ericson
b9ebe373bb
Sed some names to perhaps avoid conflicts
2020-08-05 14:49:25 +00:00
Eelco Dolstra
25f7912156
Style fix
2020-08-05 16:47:48 +02:00
John Ericson
e561a13a58
Reanme DerivationType::Regular defintion too
...
This is the one non-prefixed occurence
2020-08-05 14:45:56 +00:00
John Ericson
e7b0847f2d
Make names more consistent
2020-08-05 14:44:39 +00:00
John Ericson
839f0fe095
Merge remote-tracking branch 'upstream/master' into misc-ca
2020-08-05 14:40:01 +00:00
John Ericson
03f4fafc27
Merge remote-tracking branch 'upstream/master' into misc-ca
2020-08-05 14:36:25 +00:00
Eelco Dolstra
b91dc7ebad
Merge pull request #3730 from obsidiansystems/better-ca-parse-errors
...
Improve hash parsing and errors
2020-08-05 16:33:07 +02:00
Eelco Dolstra
75f220a595
Merge pull request #3864 from obsidiansystems/more-topo-sort
...
Abstract out topo sorting logic
2020-08-05 16:07:29 +02:00
John Ericson
16c98bf57c
Get rid of some unneeded temporaries
2020-08-04 22:36:31 +00:00
John Ericson
1dfcbebc95
Organize and format code a bit
2020-08-04 22:28:10 +00:00
John Ericson
45b6fdb22b
Remove unused functions
2020-08-04 22:10:13 +00:00
John Ericson
1bab8a321f
Remove unneeded definition
...
Template instantiations will cover this case fine.
2020-08-04 21:56:42 +00:00
John Ericson
2f2ae993dc
WIP systematize more of the worker protocol
...
This refactor should *not* change the wire protocol.
2020-08-04 19:02:05 +00:00
John Ericson
fbeb8695fb
Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing
2020-08-04 18:35:33 +00:00
John Ericson
d3452a5ed6
Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors
2020-08-04 16:13:20 +00:00
Eelco Dolstra
4e7f1c7f11
S3BinaryCacheStore: Fix size determination
2020-08-04 16:01:13 +02:00
Eelco Dolstra
dfe66420e7
Revert "Remove putBytes"
...
This reverts commit b8eea7e81a
2020-08-04 15:56:10 +02:00
Eelco Dolstra
327b1bf378
BinaryCacheStore: Explicitly flush file sink
...
The file sink is also flushed in its destructor, but we ignore any
exceptions in the destructor.
Issue #3886 .
2020-08-04 14:54:07 +02:00
Eelco Dolstra
ed52cf632b
Merge pull request #3892 from obsidiansystems/path-info-header
...
Move ValidPathInfo to its own header
2020-08-04 10:44:47 +02:00
John Ericson
062533f7cd
Merge remote-tracking branch 'upstream/master' into path-info-header
2020-08-03 21:02:28 +00:00
Eelco Dolstra
24e07c428f
Delete compressed NARs
...
Fixes #3891 .
2020-08-03 18:34:52 +02:00
John Ericson
54281f3ac1
addToStore in terms of addToStoreFromDump is not local-store-specific
2020-08-03 04:13:45 +00:00
John Ericson
9357512d73
Merge remote-tracking branch 'upstream/master' into derivation-header-include-order
2020-08-01 19:38:35 +00:00
Carlo Nucera
b6d97fdbf4
Merge branch 'master' of github.com:NixOS/nix into drv-outputs-map-allow-missing
2020-07-31 13:12:51 -04:00
Eelco Dolstra
bf290c2306
Merge remote-tracking branch 'origin/master' into markdown
2020-07-31 16:07:04 +02:00
Eelco Dolstra
1d0a7b54fa
Enable syntax highlighting
2020-07-31 15:43:25 +02:00
Carlo Nucera
eee6ef86cd
Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors
2020-07-30 18:27:25 -04:00
Matthew Bauer
05ac4db39a
Merge remote-tracking branch 'origin/master' into substitute-other-storedir
2020-07-30 12:38:24 -05:00
Eelco Dolstra
3f6e88a552
unsigned long long -> uint64_t
2020-07-30 13:34:04 +02:00
Eelco Dolstra
ebee2b7852
receiveContents(): unsigned int -> size_t
2020-07-30 13:00:30 +02:00
Eelco Dolstra
f63839bfa4
Cleanup
2020-07-29 18:04:51 +02:00
Eelco Dolstra
4c0077a07d
Fix RemoteStore::addToStore() latency
...
Since 6185d25e52Fixes #3372 .
2020-07-29 00:48:39 +02:00
Eelco Dolstra
c159f48a39
Cleanup
2020-07-29 00:24:55 +02:00
John Ericson
df707d05d1
Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca
2020-07-28 21:12:36 +00:00
Carlo Nucera
2980b244b7
Use assert(false) instead of abort()
2020-07-28 15:39:45 -04:00
Carlo Nucera
cfe96bd7ce
Merge branch 'master' of github.com:NixOS/nix into ca-drv
2020-07-28 15:36:45 -04:00
Carlo Nucera
c318d398f3
Merge branch 'misc-ca' of github.com:obsidiansystems/nix into new-interface-for-path-pathOpt
2020-07-28 14:22:24 -04:00
Carlo Nucera
7ef1e3cd14
Use the new interface
2020-07-28 13:59:24 -04:00
Carlo Nucera
d564ac1c50
Offer a safer interface for pathOpt
...
The new interface we offer provides a way of getting all the
DerivationOutputs with the storePaths directly, based on the observation
that it's the most common usecase.
2020-07-28 13:55:15 -04:00
Eelco Dolstra
b8eea7e81a
Remove putBytes
...
istream->tellg() returns -1 so we can't get the number of bytes
written.
Fixes 'uploaded 's3://nix-cache/nar/00819r9lp5kajr6baxfw5dhhc0cx8ndxaz43qmd2f0gn1hk1ynlp.nar.xz' (-1 bytes) in 11620 ms' messages.
2020-07-28 13:27:56 +02:00
John Ericson
8065c6d160
Abstract out topo sorting logic
2020-07-27 20:45:34 +00:00
John Ericson
951415b568
Require ca-derivations everywhere we create a CA derivation
...
"create" as in read one in from a serialized form, or build one from
scratch in memory.
2020-07-27 17:56:36 +00:00
John Ericson
e32a9e124b
Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output
2020-07-27 17:50:06 +00:00
John Ericson
7cf978440c
Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca
2020-07-27 17:47:40 +00:00
John Ericson
e6115ca004
Merge remote-tracking branch 'upstream/master' into ca-derivation-data-types
2020-07-27 17:41:43 +00:00
John Ericson
1d7d94ceea
Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors
2020-07-27 16:17:50 +00:00
John Ericson
43f2bd8dc5
Merge remote-tracking branch 'upstream/master' into hash-always-has-type
2020-07-27 16:13:57 +00:00
John Ericson
387699fe7f
Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca
2020-07-27 15:00:28 +00:00
John Ericson
78466bcb2f
Merge branch 'optional-derivation-output-storepath' into ca-derivation-data-types
2020-07-27 14:40:08 +00:00
John Ericson
d5bb67cfa4
Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath
2020-07-27 14:29:32 +00:00
John Ericson
699fc89b39
Merge remote-tracking branch 'upstream/master' into hash-always-has-type
2020-07-27 14:22:45 +00:00
John Ericson
2c7557481b
queryDerivationOutputMap no longer assumes all outputs have a mapping
...
This assumption is broken by CA derivations. Making a PR now to do the
breaking daemon change as soon as possible (if it is already too late,
we can bump protocol intead).
2020-07-24 21:14:06 +00:00
John Ericson
58e55c0923
Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output
2020-07-23 19:17:52 +00:00
John Ericson
a2af25c55a
Merge remote-tracking branch 'upstream/master' into misc-ca
2020-07-23 19:14:17 +00:00
John Ericson
c36916eca0
Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output
2020-07-23 15:06:56 +00:00
John Ericson
6fcd9966bf
Merge remote-tracking branch 'upstream/master' into misc-ca
2020-07-23 15:02:29 +00:00
John Ericson
66a2067288
Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors
2020-07-23 14:56:35 +00:00
John Ericson
2274f63453
Get rid of basicDerivation::findOutput
...
It's a tiny function which is:
- hardly worth abstrating over, and also only used once.
- doesn't work once we get CA drvs
I rewrote the one callsite to be forwards compatable with CA
derivations, and also potentially more performant: instead of reading in
the derivation it can ust consult the SQLite DB in the common case.
2020-07-23 14:39:02 +00:00
John Ericson
9423f64ee2
Parse CA derivations using new output variants
...
We no longer need `ParsedDerivation` because everything libstore needs
to know about is in the `BasicDerivation` proper.
2020-07-22 23:59:25 +00:00
John Ericson
c56356bacc
Separate concerns in scanForReferences with TeeSink
...
This also will make it easier to use a `HashModuloSink` instead for CA
derivations.
2020-07-22 22:13:44 +00:00
Eelco Dolstra
ff314f186e
Merge pull request #3736 from obsidiansystems/allow-relative-paths-in-store-option
...
Allow relative paths in --store option
2020-07-21 19:43:07 +02:00
Eelco Dolstra
0951330680
Merge pull request #3714 from obsidiansystems/add-body-to-network-errors
...
Add response body to network errors
2020-07-21 18:09:42 +02:00
John Ericson
02639716ea
Merge branch 'allow-relative-paths-in-store-option' into remove-storetype-delegate-regStore
2020-07-21 15:55:27 +00:00
John Ericson
6cce32c8e8
Change logic for deciding what is a relative path for the local store
...
The was Eelco's prefered logic, and it looks good to me!
2020-07-21 15:39:47 +00:00
Carlo Nucera
922a845ffc
Update chunkSize to the suggested value
...
This was a suggested course of action in a review in one of our earlier
commits, https://github.com/NixOS/nix/pull/3801#discussion_r457557079
2020-07-21 10:24:19 -04:00
John Ericson
54e507a7aa
Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors
2020-07-21 14:17:59 +00:00
John Ericson
5ce95b9529
Update src/libstore/build.cc
2020-07-21 09:47:40 -04:00
John Ericson
cdb3f39b64
Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors
2020-07-21 13:43:20 +00:00
John Ericson
5055c595bd
Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into ca-derivation-data-types
2020-07-21 01:20:53 +00:00
John Ericson
5a37413796
Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into from-dump-stream
2020-07-21 01:20:14 +00:00
John Ericson
bf0dca7790
Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into misc-ca
2020-07-21 01:18:33 +00:00
Carlo Nucera
9aae179f34
Correct bug, thoroughly document addToStoreSlow
2020-07-20 20:18:12 -04:00
Carlo Nucera
0ca9744694
Use heuristics to decide when to show the response
...
Due to https://github.com/NixOS/nix/issues/3841 we don't know how print
different messages for different verbosity levels.
2020-07-20 15:57:58 -04:00
Carlo Nucera
6357b1b0fb
Add another Unimplemented case
2020-07-20 14:17:25 -04:00
Carlo Nucera
007bf6d17a
Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca
2020-07-20 14:14:39 -04:00
Carlo Nucera
362ae93851
Add UnimplementedError to ease grepping for these
2020-07-20 14:13:37 -04:00
John Ericson
bf61871271
parser.hh -> split.hh
2020-07-20 17:42:34 +00:00
John Ericson
eab945f591
Merge remote-tracking branch 'upstream/master' into from-dump-stream
2020-07-20 17:29:23 +00:00
Eelco Dolstra
1c5f8bbfb5
Merge pull request #3822 from obsidiansystems/dump-thrice-fixme
...
Optimize `addToStoreSlow` and remove `TeeParseSink`
2020-07-20 18:55:05 +02:00
John Ericson
ac2fc7ba1f
Apply suggestions from code review
...
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-07-20 11:29:46 -04:00
Carlo Nucera
0aa79dcc6f
Remove StoreType abstraction and delegate regStore
...
to each Store implementation. The generic regStore implementation will
only be for the ambiguous shorthands, like "" and "auto".
This also could get us close to simplifying the daemon command.
2020-07-17 17:24:02 -04:00
John Ericson
6756cecfcf
Add DerivationType::CAFloating
2020-07-17 19:55:41 +00:00
Carlo Nucera
fefd6c9e5f
Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option
2020-07-17 14:34:20 -04:00
Carlo Nucera
bbc633c98c
Revert "Don't anticipate CA but not fixed outputs for now"
...
This reverts commit 3a9e4c3262
2020-07-17 13:10:32 -04:00
Carlo Nucera
205dcd140d
Revert "Don't anticipate multiple CA outputs for now"
...
This reverts commit 74b251b2f3
2020-07-17 12:43:46 -04:00
Carlo Nucera
1feb8981df
Revert "Don't anticipate hash algo without hash in derivation for now"
...
This reverts commit 3804e3df9b
2020-07-17 12:05:54 -04:00
Carlo Nucera
487c5751c6
Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca
2020-07-17 11:32:59 -04:00
Carlo Nucera
5cb840541b
Merge branch 'multi-output-hashDerivationModulo' of github.com:Ericson2314/nix into misc-ca
2020-07-17 10:28:33 -04:00
Carlo Nucera
745a03cef5
Merge branch 'optional-derivation-output-storepath' of github.com:obsidiansystems/nix into ca-derivation-data-types
2020-07-16 13:36:01 -04:00
Carlo Nucera
048e916f64
Merge branch 'master' of github.com:NixOS/nix into optional-derivation-output-storepath
2020-07-16 13:32:28 -04:00
John Ericson
cc0d77f8c9
Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors
2020-07-16 17:28:52 +00:00
Carlo Nucera
e3a3406db8
Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors
2020-07-16 12:04:10 -04:00
John Ericson
5ea817dace
Merge remote-tracking branch 'upstream/master' into hash-always-has-type
2020-07-16 14:58:53 +00:00
John Ericson
3dcca18c30
Fix bug in TeeSource
...
We use this to simplify `LocalStore::addToStoreFromDump`.
Also, hope I fixed build error with old clang (used in Darwin CI).
2020-07-16 13:39:27 +00:00
Carlo Nucera
e3b394b6e8
Small namespace fix
2020-07-16 09:36:02 -04:00
Eelco Dolstra
5517eee17e
Generations API cleanup
2020-07-16 15:18:21 +02:00
Eelco Dolstra
3f264916db
Merge remote-tracking branch 'origin/flakes'
2020-07-16 14:07:32 +02:00
John Ericson
68dfb8c6ae
Optimize addToStoreSlow and remove TeeParseSink
2020-07-16 05:09:41 +00:00
John Ericson
5602637d9e
Revert "LocalStore::addToStoreFromDump copy in chunks"
...
This reverts commit 592851fb67
2020-07-15 23:37:49 +00:00
John Ericson
bc109648c4
Get rid of LocalStore::addToStoreCommon
...
I got it to just become `LocalStore::addToStoreFromDump`, cleanly taking
a store and then doing nothing too fancy with it.
`LocalStore::addToStore(...Path...)` is now just a simple wrapper with a
bare-bones sinkToSource of the right dump command.
2020-07-15 23:14:30 +00:00
Carlo Nucera
455bdee205
Merge branch 'master' of github.com:NixOS/nix into derivation-header-include-order
2020-07-15 17:58:30 -04:00
Carlo Nucera
d090562348
Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type
2020-07-15 17:21:01 -04:00
John Ericson
aff69e229c
Merge remote-tracking branch 'obsidian/fix-add-to-star-narhash' into from-dump-stream
2020-07-15 21:05:29 +00:00
Carlo Nucera
2d2a10e79a
Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors
2020-07-15 13:06:08 -04:00
Eelco Dolstra
cff2157185
Revert "LocalStore::addToStore(srcPath): Handle the flat case"
...
This reverts commit a2c27022e9
2020-07-15 12:49:03 +02:00
John Ericson
650c2c6558
Rename variable nar -> dump according to TODO
2020-07-14 21:28:50 +00:00
John Ericson
8173e7bfef
Fix localhost::addToStore(...Path...)
...
We were calculating the nar hash wrong when the file ingestion method
was flat. I don't think there's anything we can do in that case but dump
the file again, so that's what I do.
As an optomization, we again could reuse the original dump for just the
recursive and non-sha256 case, but I rather do that after this fix, and
after my other PRs which deduplicate this code.
2020-07-14 21:12:11 +00:00
John Ericson
592851fb67
LocalStore::addToStoreFromDump copy in chunks
...
Rather than copying byte-by-byte, we let the coroutine know how much
data we would like it to send back to us.
2020-07-14 13:56:36 +00:00
John Ericson
9de96ef7d4
Dedup LocalStore::addToStore*
...
The downsides is that the coroutine has byte-by-byte loop transfer. Will
fix that next.
2020-07-14 13:56:36 +00:00
John Ericson
c86fc3a965
Crudely make addToStoreFromDump take Source not string
...
I just as little beyond the type as possible, so the implementation
changes this enables can be reviewed separately.
2020-07-14 13:56:36 +00:00
John Ericson
9ec10046e0
Narrow scope of temporary value
2020-07-14 13:54:29 +00:00
Eelco Dolstra
832e111494
Merge remote-tracking branch 'origin/master' into flakes
2020-07-14 13:56:18 +02:00
Eelco Dolstra
926c3a6664
Doh
2020-07-14 11:55:54 +02:00
Eelco Dolstra
43b8e96d30
Fix 'nix verify --all' on a binary cache (cached case)
2020-07-13 20:17:00 +02:00
Eelco Dolstra
7c2fef0a81
Make 'nix copy' to s3:// binary caches run in constant memory
2020-07-13 20:07:19 +02:00
Eelco Dolstra
493961b689
Remove istringstream_nocopy
2020-07-13 18:31:19 +02:00
Eelco Dolstra
545bb2ed03
Remove 'accessor' from addToStore()
...
This is only used by hydra-queue-runner and it's better to implement
it there.
2020-07-13 18:31:19 +02:00
Eelco Dolstra
0a9da00a10
NarAccessor: Run in constant memory
2020-07-13 17:30:42 +02:00
Eelco Dolstra
fc84c358d9
Make 'nix copy' to file:// binary caches run in constant memory
2020-07-13 16:28:45 +02:00
Eelco Dolstra
400f1a9b59
Store::pathInfoToJSON(): Use consistent format for downloadHash
2020-07-13 16:26:09 +02:00
Eelco Dolstra
c0dd05131e
toStorePath(): Return a StorePath and the suffix
2020-07-13 16:25:48 +02:00
Eelco Dolstra
1d01ae816b
Fix 'nix verify --all' on a binary cache and add a test
2020-07-13 14:35:01 +02:00
John Ericson
c466cb2091
Merge branch 'hash-always-has-type' into better-ca-parse-errors
2020-07-13 03:02:09 +00:00
John Ericson
4415765385
Merge remote-tracking branch 'upstream/master' into hash-always-has-type
2020-07-13 03:01:44 +00:00
John Ericson
230c9b4329
Change types to prepare the way for CA derivations
...
We've added the variant to `DerivationOutput` to support them, but made
`DerivationOutput::path` partial to avoid actually implementing them.
With this chage, we can all collaborate on "just" removing
`DerivationOutput::path` calls to implement CA derivations.
2020-07-12 23:53:33 +00:00
John Ericson
fedfc913ad
Use more std::visit to prepare for new variant
...
N.B. not using `std::visit` for fetchurl because there is no attempt to
handle all the cases (e.g. no `else`) and lambda complicates early
return.
2020-07-12 23:51:40 +00:00
John Ericson
ffc18583b1
Move C++17 "pattern matching" boilerplat to utils.hh
2020-07-12 22:15:14 +00:00
John Ericson
503b425690
DerivationOutputExtensional -> DerivationOutputInputAddressed
...
Thanks @regnat for the great name.
2020-07-12 15:56:20 +00:00
John Ericson
a8d4707107
Undo erroneous indentation change
2020-07-12 15:54:12 +00:00
John Ericson
18152406ce
String .drv suffix to create derivation name
2020-07-12 15:40:14 +00:00
John Ericson
5d0b75e5b6
std::string_view for new derivation name parameters
2020-07-12 15:02:36 +00:00
John Ericson
abea26a968
BasicDerivation::findOutput cannot return reference anymore
2020-07-12 01:57:06 +00:00
John Ericson
442d43178b
Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath
2020-07-11 02:27:45 +00:00
Matthew Bauer
acb74d4d94
Merge remote-tracking branch 'origin/master' into substitute-other-storedir
2020-07-10 18:13:45 -04:00
Eelco Dolstra
8efa23bb99
Avoid a redundant hash
2020-07-10 15:56:24 +02:00
Eelco Dolstra
5dff49f661
Factor out commonality between nix-prefetch-url and nix-store --add-fixed
2020-07-10 13:21:37 +02:00
John Ericson
323d51a7a0
Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors
2020-07-09 19:07:06 +00:00
Matthew Bauer
a7884970c5
Fix DerivationOutputExtensional name
2020-07-09 11:37:18 -04:00
Eelco Dolstra
a2c27022e9
LocalStore::addToStore(srcPath): Handle the flat case
...
This helps nix-prefetch-url when using a local store.
2020-07-09 15:54:32 +02:00
Matthew Bauer
06a4e15478
Fix build.cc on linux
2020-07-08 19:27:51 -04:00
Matthew Bauer
8e0d0689be
Only store hash of fixed derivation output
...
we don’t need a full storepath for a fixedoutput derivation. So just
putting the ingestion method + the hash is sufficient.
2020-07-08 19:11:39 -04:00
Eelco Dolstra
b981e5aacf
Cleanup
2020-07-08 22:07:21 +02:00
Eelco Dolstra
34f25124ba
Make LocalStore::addToStore(srcPath) run in constant memory
...
This reduces memory consumption of
nix-instantiate \
-E 'with import <nixpkgs> {}; runCommand "foo" { src = ./blender; } "echo foo"' \
--option nar-buffer-size 10000
(where ./blender is a 1.1 GiB tree) from 1716 to 36 MiB, while still
ensuring that we don't do any write I/O for small source paths (up to
'nar-buffer-size' bytes). The downside is that large paths are now
always written to a temporary location in the store, even if they
produce an already valid store path. Thus, adding large paths might be
slower and run out of disk space. ¯\_(ツ)_/¯ Of course, you can always
restore the old behaviour by setting 'nar-buffer-size' to a very high
value.
2020-07-08 22:07:21 +02:00
Matthew Bauer
af95a7c16b
Add name to BasicDerivation
...
We always have a name for BasicDerivation, since we have a derivation
store path that has a name.
2020-07-08 15:38:01 -04:00
Eelco Dolstra
9223603908
Merge remote-tracking branch 'origin/master' into flakes
2020-07-08 15:55:19 +02:00
Eelco Dolstra
16ec7785ca
Fix 'got unknown message type 1 from Nix daemon'
...
Example:
$ nix-build -E 'with import <nixpkgs> {}; runCommand "foo" { x = runCommand "bar" {} "exit 1"; } "echo foo; exit 1"'
warning: unknown setting 'auto-allocate-uids'
these 2 derivations will be built:
/nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv
/nix/store/k4fsvrjl7cp2xpz7927iv7g0dqj1zyhs-foo.drv
warning: unknown setting 'auto-allocate-uids'
building '/nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv'...
error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------- nix-daemon
builder for '/nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv' failed with exit code 1
error: --- Error ------------------------------------------------------------------------------------------------------------------------------------------------------------------ nix-build
got unknown message type 1 from Nix daemon
2020-07-08 15:53:14 +02:00
Eelco Dolstra
1ab9da9154
Merge remote-tracking branch 'origin/master' into flakes
2020-07-07 14:38:57 +02:00
Eelco Dolstra
7c9ece5dca
exportReferencesGraph: Fix support for non-top-level store paths
...
Fixes #3471 .
2020-07-07 14:25:43 +02:00
Carlo Nucera
0f05a36e20
Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors
2020-07-06 16:57:09 -04:00
Eelco Dolstra
54712aaf8a
Merge remote-tracking branch 'origin/master' into flakes
2020-07-06 16:40:10 +02:00
John Ericson
a38ab99d57
Merge remote-tracking branch 'upstream/master' into derivation-header-include-order
2020-07-05 21:49:01 +00:00
John Ericson
465daa9396
Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors
2020-07-03 17:08:39 +00:00
John Ericson
3134db1a83
Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors
2020-07-03 14:12:38 +00:00
John Ericson
dbffd309fe
Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type
2020-07-03 14:11:38 +00:00
Ben Burdette
b29a4ea1dc
Merge branch 'master' into add-trace
2020-07-03 07:57:36 -06:00
regnat
223fbe644a
Shorten the path to the test root
...
Fix a socket length failure on the OSX builders
2020-07-03 09:20:01 +02:00
John Ericson
13796be78d
Have splitPrefix and splitPrefixTo parser helpers
2020-07-02 23:18:22 +00:00
John Ericson
2f93d9f2ba
Merge branch 'hash-always-has-type' into HEAD
2020-07-02 21:47:51 +00:00
John Ericson
1be279af26
Fix Narinfo corruption detection bug
...
The aim of this check was just to ensure each key occurs once.
2020-07-02 21:46:10 +00:00
Carlo Nucera
1fc835aa22
Tighten parsing for drv files and pathinfo
2020-07-02 11:57:21 -04:00
Matthew Bauer
fc2ab42e86
Merge remote-tracking branch 'origin/master' into substitute-other-storedir
2020-07-02 11:14:04 -04:00
Matthew Bauer
d2e8b9ff0e
Store subPath in SubstitutionGoal
2020-07-02 11:12:05 -04:00
Ben Burdette
bf2788e4c1
move showTrace to new loggerSettings
2020-07-02 09:04:31 -06:00
Matthew Bauer
1f9cb06db2
Try next when no ca exists and have different store dirs
2020-07-02 10:59:24 -04:00
Carlo Nucera
263ccdd489
Rename two hash constructors to proper functions
2020-07-01 18:34:18 -04:00
Eelco Dolstra
38ccf2e241
Cleanup
2020-07-01 15:31:34 +02:00
Eelco Dolstra
86a4aba6c4
Merge branch 'remote-query-outputs' of https://github.com/tweag/nix
2020-07-01 15:10:29 +02:00
Ben Burdette
9159dfe3d8
comments and cleanup
2020-06-30 16:31:55 -06:00
Matthew Bauer
09833126f3
Merge remote-tracking branch 'origin/master' into substitute-other-storedir
2020-06-30 16:35:08 -04:00
Ben Burdette
ddb81ca126
Merge branch 'master' into add-trace
2020-06-30 12:21:45 -06:00
Carlo Nucera
77b51f4598
Factor the prefix splitting in content-address
2020-06-30 11:57:46 -04:00
John Ericson
a83566e5bc
Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors
2020-06-29 18:40:34 +00:00
John Ericson
987a4a0be9
Merge remote-tracking branch 'upstream/master' into hash-always-has-type
2020-06-29 18:03:21 +00:00
Ben Burdette
8f81fae116
showTrace flag in loggers
2020-06-29 10:20:51 -06:00
Eelco Dolstra
58bc3b6578
Merge pull request #3729 from obsidiansystems/simpler-hased-mirror
...
hashed-mirrors: Use parsed derivation output rather than reconstructing it
2020-06-29 14:04:12 +02:00
Eelco Dolstra
64232f3ea6
Merge pull request #3749 from rodarima/master
...
Fall back to copyPath if link fails with EPERM
2020-06-29 13:31:24 +02:00
Carlo Nucera
278a679782
Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option
2020-06-26 10:40:13 -04:00
Eelco Dolstra
adf2fbbdc2
Merge remote-tracking branch 'origin/master' into flakes
2020-06-26 08:46:46 +02:00
John Ericson
d92d4f85a5
Move ValidPathInfo to its own header
2020-06-25 17:48:49 +00:00
Eelco Dolstra
de2641ae99
Fix empty std::optional dereference in writeDerivation()
...
https://hydra.nixos.org/build/123017579
2020-06-25 15:50:30 +02:00
Rodrigo
3a642187c3
Fall back to copyPath if link fails with EPERM
...
BeeGFS doesn't allow hard-links and returns EPERM, so we fall back
to copyPath. See https://github.com/NixOS/nix/issues/3748
2020-06-25 12:03:26 +02:00
regnat
d38f860c3e
Add a way to get all the outputs of a derivation with their label
...
Generalize `queryDerivationOutputNames` and `queryDerivationOutputs` by
adding a `queryDerivationOutputMap` that returns the map
`outputName=>outputPath`
(not that this is not equivalent to merging the results of
`queryDerivationOutputs` and `queryDerivationOutputNames` as sets don't
preserve the order, so we would end up with an incorrect mapping).
squash! Add a way to get all the outputs of a derivation with their label
Rename StorePathMap to OutputPathMap
2020-06-24 20:38:40 +02:00
John Ericson
d9fb9e9f26
Merge remote-tracking branch 'upstream/master' into ca-drv
2020-06-24 15:41:29 +00:00
John Ericson
98e5d1af03
Merge remote-tracking branch 'upstream/master' into hash-always-has-type
2020-06-23 17:03:37 +00:00
Carlo Nucera
e197bc6229
Enable the --store option to take relative paths
...
In nix commands which accept --store options, we can now specify a
relative path, which will be canonicalized.
2020-06-23 11:13:18 -04:00
John Ericson
55d4bd6e0e
Improve content address parsing
...
- Ensure hash is in form <algo>-<prefix> and not SRI.
- Better errors if something goes wrong
- string_view for no coppying
2020-06-22 18:08:27 +00:00
Matthew Bauer
66a62b3189
Merge remote-tracking branch 'origin/master' into substitute-other-storedir
2020-06-22 13:08:11 -04:00
John Ericson
f4a5913125
hashed-mirrors: Use parsed derivation output rather than reconstructing it
...
Now the derivation outputs are parsed up front, we can avoid a reparse
by doing it. Also, this just feels a bit better as the `output*` env
vars are more of a `libnixexpr` interface than `libnixstore` interface:
ultimately, it's the derivation outputs that decide whether the
derivation is fixed-output.
Yes, hashed mirrors might go away with #3689 , but this bit of code would
be moved rather than deleted, so it's worth doing a cleanup anyways I
think.
2020-06-22 15:17:20 +00:00
John Ericson
93bbe6e8ab
Merge remote-tracking branch 'upstream/master' into ca-drv
2020-06-21 21:11:58 +00:00
John Ericson
3804e3df9b
Don't anticipate hash algo without hash in derivation for now
...
When we merge with master, the new lack of string types make this case
impossible (after parsing). Later, when we actually implemenent
CA-derivations, we'll change the types to allow that.
2020-06-21 21:05:37 +00:00
John Ericson
8313f0e939
Merge remote-tracking branch 'upstream/master' into derivation-header-include-order
2020-06-21 20:39:10 +00:00
John Ericson
bf9f040112
Tweak declaration
...
I think this is clearer
2020-06-21 16:51:39 +00:00
John Ericson
fdeabf7160
Merge remote-tracking branch 'upstream/master' into multi-output-hashDerivationModulo
2020-06-21 16:43:17 +00:00
Matthew Bauer
f2a6cee334
Update worker protocol to support sending storepath maps
...
We need to also send the ca to daemon in addition to the path.
2020-06-19 18:06:19 -04:00
John Ericson
507aa48739
WIP: Make Hash always store a valid hash type
2020-06-19 21:48:57 +00:00
Matthew Bauer
e8e1f5282f
Replace error message in getDerivationCA
2020-06-19 15:19:09 -04:00
Matthew Bauer
0c9c1b8826
Return map of StorePaths in copyPaths
...
This allows the caller to know what values were actually added to the
store.
2020-06-19 14:48:57 -04:00
Matthew Bauer
7e11cf3399
Merge remote-tracking branch 'origin/master' into substitute-other-storedir
2020-06-19 14:40:49 -04:00
Matthew Bauer
2796b01e86
Merge remote-tracking branch 'origin/master' into substitute-other-storedir
2020-06-19 14:22:06 -04:00
John Ericson
68294746ae
Merge remote-tracking branch 'upstream/master' into no-hash-type-unknown
2020-06-19 17:53:34 +00:00
John Ericson
911fc88bcb
More designated initializers
2020-06-19 17:42:56 +00:00
John Ericson
2f0e395c99
Merge remote-tracking branch 'me/no-stringly-typed-derivation-output' into validPathInfo-ca-proper-datatype
2020-06-19 15:26:59 +00:00
John Ericson
fb39a5e00c
Remove unneeded constructor for DerivationOutputHash
2020-06-19 15:11:11 +00:00
John Ericson
01dc8b0bab
Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output
2020-06-19 14:59:05 +00:00
John Ericson
145d88cb2a
Use designated initializers for DerivationOutputHash
2020-06-19 14:58:30 +00:00
John Ericson
237d88c97e
FileSystemHash -> DerivationOutputHash
2020-06-19 14:47:10 +00:00
John Ericson
3fc58a9638
Remove some Base:: that crept in
2020-06-19 00:24:47 +00:00
John Ericson
3f8dcfe3fd
Merge branch 'validPathInfo-temp' into validPathInfo-ca-proper-datatype
2020-06-18 23:01:58 +00:00
John Ericson
669c3992e8
Merge branch 'no-hash-type-unknown' into validPathInfo-temp
2020-06-18 22:33:07 +00:00
John Ericson
15abb2aa2b
Revert the enum struct change
...
Not a regular git revert as there have been many merges and things.
2020-06-18 22:11:26 +00:00
John Ericson
bbbf3602a3
Merge branch 'enum-class' into no-hash-type-unknown
2020-06-18 22:11:19 +00:00
John Ericson
40526fbea5
Merge remote-tracking branch 'upstream/master' into enum-class
2020-06-18 21:38:15 +00:00
John Ericson
1b23fe4afb
Fix bugs
...
- Bad dynamic cast target ...classic
- std::shared_ptr need explicit deref
2020-06-18 19:03:10 +00:00
John Ericson
fb432b0bd2
Merge remote-tracking branch 'obsidian/fix-url-format' into add-body-to-network-errors
2020-06-18 18:08:02 +00:00
John Ericson
639e20dc3e
Prevent '%' in URL from causing crashes
...
We have a larger problem that passsing computed strings to the first
variable argument of many exception constructors is unsafe because that
first variable argument is interpreted not as a plain string, but format
string, and if it contains '%' boost::format will abort, since there are
no arguments to the format string.
In this particular instance '%' was used as part of an escape code in a
URL, which, when the download failed, caused Nix to abort displaying the
`FileTransferError`.
2020-06-18 17:54:16 +00:00
John Ericson
74b219ef6e
Adjust FileTransferError message to use opt response
2020-06-18 14:48:45 +00:00
regnat
4fef2ba7e4
Rename content-addressed-paths into ca-derivations
...
See <https://github.com/NixOS/nix/pull/3710#issuecomment-645480333 >
2020-06-18 09:25:55 +02:00
Carlo Nucera
004570a377
Add HTTP responses to FileTransferErrors
2020-06-17 22:34:31 +00:00
Carlo Nucera
a835c740ca
Replace TransferItem::status with a local variable
...
Everywhere seems to use `getHTTPStatus` now.
2020-06-17 22:16:24 +00:00
Carlo Nucera
fb05a6adcf
Eliminate old TeeSink abstraction
...
This was introduced in fa125b9b281cf4801108
2020-06-17 22:07:22 +00:00
Eelco Dolstra
2b8f33bf5f
Merge pull request #3713 from matthewbauer/cleanup-warnings
...
Cleanup class StorePath warning
2020-06-17 21:19:30 +02:00
Matthew Bauer
5e631e3304
Add StorePathCAMap for querySubstitutablePathInfos
...
I’m not 100% sure this is wanted since it kind of makes everything
have to know about ca even if they don’t really want to. But it also
make things easier in dealing with looking up ca.
2020-06-17 15:03:05 -04:00
Matthew Bauer
be50de1142
Make sure references are empty for store path replacing
...
also copy info2 instead of casting
2020-06-17 14:15:08 -04:00
Matthew Bauer
8974755d19
Add assert for replaced storePath
2020-06-17 14:04:46 -04:00
Eelco Dolstra
3078404e35
Merge pull request #3712 from obsidiansystems/make-http-successful-states-coherent
...
Make successful states coherent
2020-06-17 19:37:46 +02:00
Matthew Bauer
7bd88cc1dc
Merge remote-tracking branch 'origin/master' into substitute-other-storedir
2020-06-17 13:32:29 -04:00
Matthew Bauer
22d7d36703
Remove unused narInfoFile in binary-cache-store
2020-06-17 13:27:19 -04:00
Matthew Bauer
f767bedfac
Replace struct StorePath with class StorePath
...
also a similar case with struct Goal
2020-06-17 13:26:37 -04:00
Carlo Nucera
4930cb48a2
Include review comments
2020-06-17 12:58:59 -04:00
Matthew Bauer
6438ba1e99
Update strings from review comment
2020-06-17 12:31:01 -04:00
Eelco Dolstra
2f51cd8dc9
Merge pull request #3710 from tweag/reserve_ca_derivations
...
Reserve the `__contentAddressed` derivation parameter
2020-06-17 18:28:26 +02:00
Eelco Dolstra
4d5169bdd5
Merge pull request #3707 from p01arst0rm/outdated-function-fix
...
replaced uncaught_exception with uncaught_exceptions
2020-06-17 18:26:01 +02:00
regnat
480b54e1c6
fixup! Reserve the __contentAddressed derivation parameter
2020-06-17 17:37:04 +02:00
Carlo Nucera
079c6e87de
Make successful states coherent
...
The successful states used in these two places in the code were slightly
different. Should they be the same list?
2020-06-17 11:16:16 -04:00
regnat
56d75bf4fc
Reserve the __contentAddressed derivation parameter
...
Not implementing anything here, just throwing an error if a derivation
sets `__contentAddressed = true` without
`--experimental-features content-addressed-paths`
(and also with it as there's nothing implemented yet)
2020-06-17 15:41:17 +02:00
Eelco Dolstra
fdff09e57c
Fix coverage build
2020-06-17 15:18:10 +02:00
Eelco Dolstra
ccbea8255c
Merge pull request #3657 from obsidiansystems/sligthly-improve-store-path-documentation
...
Clarify the description of StorePath inputs
2020-06-17 14:54:37 +02:00
Eelco Dolstra
1524752c17
Merge remote-tracking branch 'origin/master' into flakes
2020-06-17 10:26:52 +02:00
John Ericson
517f5980e2
Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output
2020-06-17 04:58:43 +00:00
John Ericson
18493fd9c4
Move some Store functions from derivations.cc to store-api.cc
...
This further continues with the dependency inverstion. Also I just went
ahead and exposed `parseDerivation`: it seems like the more proper
building block, and not a bad thing to expose if we are trying to be
less wedded to drv files on disk anywas.
2020-06-17 04:30:16 +00:00
John Ericson
7130f0a3a6
Don't need abstract struct Derivation in local-store
2020-06-17 04:05:38 +00:00
John Ericson
21ef342172
Merge remote-tracking branch 'upstream/master' into derivation-header-include-order
2020-06-17 03:52:01 +00:00
John Ericson
02928f76fd
Merge remote-tracking branch 'upstream/master' into multi-output-hashDerivationModulo
2020-06-17 03:51:02 +00:00
John Ericson
49308ef039
Merge remote-tracking branch 'upstream/master' into ca-drv
2020-06-17 03:39:39 +00:00
p01arst0rm
c9d06558b6
replaced uncaught_exception with uncaught_exceptions
2020-06-17 03:15:47 +01:00
Eelco Dolstra
29542865ce
Remove StorePath::clone() and related functions
2020-06-16 22:20:18 +02:00
Eelco Dolstra
759947bf72
StorePath: Rewrite in C++
...
On nix-env -qa -f '<nixpkgs>', this reduces maximum RSS by 20970 KiB
and runtime by 0.8%. This is mostly because we're not parsing the hash
part as a hash anymore (just validating that it consists of base-32
characters).
Also, replace storePathToHash() by StorePath::hashPart().
2020-06-16 14:28:41 +02:00
Eelco Dolstra
72e17290d4
Fix FTP support
...
Fixes #3618 .
2020-06-16 11:53:04 +02:00
Eelco Dolstra
a588b6b19d
Print only one error message if a build fails
...
E.g. instead of
error: --- BuildError ----------------------------------------------- nix
builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1
error: --- Error ---------------------------------------------------- nix
build of '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed
we now get
error: --- Error ---------------------------------------------------- nix
builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1
2020-06-15 19:35:31 +02:00
Eelco Dolstra
31707735b6
Remove unnecessary amDone() overrides
2020-06-15 16:47:21 +02:00
Eelco Dolstra
5ed5d7acbd
Improve "waiting for locks" messages
...
These are now shown in the progress bar.
Closes #3577 .
2020-06-15 16:03:29 +02:00
Eelco Dolstra
e14e62fddd
Remove trailing whitespace
2020-06-15 14:12:39 +02:00
Eelco Dolstra
1fb762d11f
Get rid of explicit ErrorInfo constructors
2020-06-15 14:06:58 +02:00
Eelco Dolstra
7a77762961
Merge branch 'errors-phase-2' of https://github.com/bburdette/nix
2020-06-15 11:46:31 +02:00
Matthew Bauer
c214cda940
Correctly substitute from different storeDir
...
Originally, the test was only checking for different “real” storeDir.
That’s an easy case to handle, but the much harder one is if different
virtual store dirs are used. To do this, we need the SubstitutionGoal
to know about the ca, so it can recalculate the path to copy it over.
An important note here is that the store path passed to copyStorePath
needs to be one for srcStore - so that queryPathInfo works properly.
This also adds an error message when the store path from queryPathInfo
is different from the one we requested.
2020-06-13 00:07:42 -05:00
Matthew Bauer
b2cb288cdd
Add makeFixedOutputPathFromCA function
...
This puts what we are already doing into a shared method. It just
needs a path name and a ca and produces a store path.
2020-06-12 16:36:35 -05:00
John Ericson
f6f01416b7
Use std::string_view in a few more places
2020-06-12 21:32:30 +00:00
Matthew Bauer
88120442d2
Debug when storePath changes
...
these rewrites should be transparent, but they are important to know
about when debugging
2020-06-12 15:32:52 -05:00
Matthew Bauer
5b386b05f5
Recompute storePath based on isContentAddressed
2020-06-12 14:57:45 -05:00
Matthew Bauer
85d01e1f65
Don’t use makeStorePath
2020-06-12 14:53:30 -05:00
Matthew Bauer
9077b9e6b2
Separate dstStore path from srcStore path
2020-06-12 14:27:28 -05:00
Matthew Bauer
2f2ac850b5
Compute new store path correctly
2020-06-12 14:03:17 -05:00
Matthew Bauer
11c97070f3
Fix storeDir != storeDir condition
...
this needs to only continue if the path replacement fails.
2020-06-12 10:14:03 -05:00
Matthew Bauer
3e3eaa90dd
Remove hashed-mirrors
2020-06-12 09:51:44 -05:00
Matthew Bauer
79c169d1c6
Allow substituting from different storeDir
...
Substituters can substitute from one store dir to another with a
little bit of help. The store api just needs to have a CA so it can
recompute the store path based on the new store dir. We can only do
this for fixed output derivations with no references, though.
2020-06-12 09:49:09 -05:00
Eelco Dolstra
9f736dd89d
Add Store::readDerivation() convenience function
2020-06-12 13:04:52 +02:00
Eelco Dolstra
045b07200c
Remove Store::queryDerivationOutputNames()
...
This function was used in only one place, where it could easily be
replaced by readDerivation() since it's not
performance-critical. (This function appears to have been modelled
after queryDerivationOutputs(), which exists only to make the garbage
collector faster.)
2020-06-12 12:46:33 +02:00
Ben Burdette
ef1b3f21b6
Merge remote-tracking branch 'upstream/master' into errors-phase-2
2020-06-11 14:06:35 -06:00
Eelco Dolstra
ac4d43a31b
Merge pull request #3073 from tweag/machine-logs
...
Add an option to print the logs in a machine-readable format
2020-06-11 15:45:18 +02:00
Eelco Dolstra
dd9bb11d0d
Move names.{cc,hh} to libstore
2020-06-11 15:42:18 +02:00
Eelco Dolstra
b9ae1bdd7a
Merge pull request #3655 from zimbatm/hash-encoding-prepare
...
libutils/hash: remove default encoding
2020-06-10 11:48:38 +02:00
Matthew Bauer
b2c8061b44
Disable extra-platforms = i686-linux on wsl1 ( #3676 )
...
WSL1 doesn’t support i686-linux emulation, see https://github.com/microsoft/wsl/issues/2468
2020-06-09 21:53:53 +00:00
Eelco Dolstra
29e0748847
Show HTTP status message
...
For example:
warning: unable to download 'https://api.github.com/repos/edolstra/dwarffs/commits/master ': HTTP error 403 ('rate limit exceeded'); using cached version
2020-06-09 14:20:22 +02:00
Eelco Dolstra
447ea52b07
FileTransfer: Don't store status since curl already does that
2020-06-09 14:05:15 +02:00
regnat
4983401440
Unify the printing of the logs between bar-with-logs and raw
...
Make the printing of the build logs systematically go through the
logger, and replicate the behavior of `no-build-output` by having two
different loggers (one that prints the build logs and one that doesn't)
2020-06-08 09:31:15 +02:00
Carlo Nucera
d614166cb6
Fix condition error and make test suite pass
2020-06-04 17:21:21 -04:00
John Ericson
e5cc1ebc5d
Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output
2020-06-04 21:04:35 +00:00
John Ericson
94ddea9e2f
Use readString rather than >> temporary
...
Fixed the rest of these before, but this one slipped through.
2020-06-04 20:55:08 +00:00
John Ericson
2041499b5e
Flip boolean
...
Thanks Matt!
2020-06-04 20:42:25 +00:00
John Ericson
ed86acf02a
Use some std::optional::has_value for clarity
2020-06-04 20:42:02 +00:00
Matthew Bauer
2299ef705c
Add error message when FileIngestionMethod is out of bounds
...
bool coerces anything >0 to true, but in the future we may have other
file ingestion methods. This shows a better error message when the
“recursive” byte isn’t 1.
2020-06-04 11:32:39 -05:00
John Ericson
53bc8ff152
No C++ designated initializers yet with Clang 7
2020-06-03 20:45:14 -04:00
Ben Burdette
4335ba999b
Merge remote-tracking branch 'upstream/master' into errors-phase-2
2020-06-03 17:00:00 -06:00
Carlo Nucera
132d6f2c24
Clarify the description of StorePath construction
2020-06-03 16:08:32 -04:00
John Ericson
74b251b2f3
Don't anticipate multiple CA outputs for now
2020-06-03 18:53:04 +00:00
John Ericson
3a9e4c3262
Don't anticipate CA but not fixed outputs for now
2020-06-03 18:50:45 +00:00
John Ericson
2500403059
Use enum and predicates rather than bitfile for derivation type
2020-06-03 17:38:54 +00:00
John Ericson
6b7f4ec4ab
Merge remote-tracking branch 'upstream/master' into ca-drv
2020-06-03 16:36:06 +00:00
Eelco Dolstra
c20591ddc3
Merge remote-tracking branch 'origin/master' into flakes
2020-06-03 16:15:22 +02:00
zimbatm
6ee03b8444
libutils/hash: remove default encoding
...
This will make it easier to reason about the hash encoding and switch to
SRI everywhere where possible.
2020-06-03 13:49:51 +02:00
John Ericson
3c78ac348c
Merge remote-tracking branch 'obsidian/no-hash-type-unknown' into validPathInfo-ca-proper-datatype
2020-06-03 04:44:24 +00:00
John Ericson
406dbb7fce
outputHashAlgo can be blank so parse accordingly
...
It is blank for SRI hashes.
2020-06-02 21:09:15 +00:00
Carlo Nucera
75d2581390
Typo
2020-06-02 16:21:18 -04:00
Carlo Nucera
78f137e931
Validate text version instead, throw Errors
2020-06-02 16:20:22 -04:00
Carlo Nucera
a5cdf1867e
Add assertions for SHA256 in fixed case
2020-06-02 16:13:08 -04:00
Carlo Nucera
fd2eb41e64
Move file-hash to content-address
2020-06-02 15:44:58 -04:00
Carlo Nucera
343c20a404
WIP Completed implementation
2020-06-02 15:23:21 -04:00
Carlo Nucera
390bf64858
WIP
2020-06-02 14:15:58 -04:00
John Ericson
a33270ce1d
Clean up ValidPathInfo::isContentAddressed with std::visit
2020-06-02 17:04:21 +00:00
John Ericson
25e61812f3
Apply suggestions from code review
...
Co-authored-by: Matthew Bauer <mjbauer95@gmail.com>
2020-06-02 12:47:18 -04:00
John Ericson
64cffb804a
Merge remote-tracking branch 'upstream/master' into no-hash-type-unknown
2020-06-02 16:07:25 +00:00
John Ericson
450dcf2c1b
Remove HashType::Unknown
...
Instead, `Hash` uses `std::optional<HashType>`. In the future, we may
also make `Hash` itself require a known hash type, encoraging people to
use `std::optional<Hash>` instead.
2020-06-02 15:52:13 +00:00
Carlo Nucera
a5d820a0a3
Change parseCa(Opt) to parseContentAddress(Opt)
2020-06-02 11:00:10 -04:00
John Ericson
1b6461f671
Merge remote-tracking branch 'upstream/master' into validPathInfo-ca-proper-datatype
2020-06-02 14:31:18 +00:00
Ben Burdette
d82d230b40
elide the 'ErrorInfo' in logError and logWarning calls
2020-06-02 08:22:24 -06:00
John Ericson
efcd30da89
WIP
2020-06-02 00:37:43 +00:00
John Ericson
754c910953
WIP more progress
2020-06-01 19:26:40 -04:00
Carlo Nucera
da39092a39
WIP
2020-06-01 18:53:31 -04:00
Carlo Nucera
0e9438b6d3
Create new file-hash files
2020-06-01 17:32:40 -04:00
Carlo Nucera
0cb67ecbd3
Merge branch 'derivation-header-include-order' of github.com:Ericson2314/nix into validPathInfo-ca-proper-datatype
2020-06-01 17:13:11 -04:00
Carlo Nucera
f4b89e11a4
Merge branch 'no-stringly-typed-derivation-output' of github.com:Ericson2314/nix into validPathInfo-ca-proper-datatype
2020-06-01 17:12:50 -04:00
John Ericson
fac0c2d54a
Remove addToStore variant as requested by FIXME
...
The idea is it's always more flexible to consumer a `Source` than a
plain string, and it might even reduce memory consumption.
I also looked at `addToStoreFromDump` with its `// FIXME: remove?`, but
the worked needed for that is far more up for interpretation, so I
punted for now.
2020-05-29 17:02:32 -04:00
Ben Burdette
734283d636
Merge remote-tracking branch 'upstream/master' into errors-phase-2
2020-05-29 09:51:37 -06:00
Carlo Nucera
6dd471ebf6
Fixing the result of merge
2020-05-28 12:14:36 -04:00
Carlo Nucera
4f597fb901
Merge branch 'master' of github.com:NixOS/nix into enum-class
2020-05-28 10:58:22 -04:00
John Ericson
5b4cd84bc2
Merge remote-tracking branch 'me/more-rust-ffi' into no-stringly-typed-derivation-output
2020-05-28 10:35:53 -04:00
John Ericson
ef71caba29
Merge remote-tracking branch 'upstream/master' into more-rust-ffi
2020-05-28 10:31:46 -04:00
Eelco Dolstra
17ca997fc6
Merge remote-tracking branch 'origin/master' into flakes
2020-05-28 12:55:24 +02:00
Matthew Bauer
c66441a646
Rename some variables named “recursive” to “method”
...
This is much less confusing since recursive is no longer a boolean.
2020-05-27 13:21:26 -05:00
Carlo Nucera
d49e65ba9d
Merge remote-tracking branch 'john-ericson/enum-FileIngestionMethod' into no-stringly-typed-derivation-output
2020-05-26 12:30:48 -04:00
Carlo Nucera
0f3f901071
Merge remote-tracking branch 'origin/master' into more-rust-ffi
2020-05-26 11:46:42 -04:00
Carlo Nucera
6d73c10041
Merge remote-tracking branch 'origin/master' into enum-FileIngestionMethod
2020-05-26 11:14:08 -04:00
Eelco Dolstra
5ef64f05e6
Cleanup
2020-05-18 15:50:29 +02:00
Eelco Dolstra
0ed946aa61
Merge branch 'wait-for-builders' of https://github.com/serokell/nix
2020-05-18 13:48:45 +02:00
Ben Burdette
92123c6c79
Merge remote-tracking branch 'upstream/master' into errors-phase-2
2020-05-15 07:00:36 -06:00
Alexander Bantyev
183dd28266
Don't lock a user while doing remote builds
2020-05-14 17:00:54 +03:00
Ben Burdette
ef9dd9f9bc
formatting and a few minor changes
2020-05-13 15:56:39 -06:00
Ben Burdette
c79d4addab
consistent capitalization
2020-05-13 10:02:18 -06:00
Ben Burdette
bfca5fc395
change status messages to info level
2020-05-13 09:52:36 -06:00
Ben Burdette
19cffc29c9
remove unused extra json fields
2020-05-12 12:09:12 -06:00
Ben Burdette
ec870b9c85
new pos format for more errors
2020-05-12 10:52:26 -06:00
Eelco Dolstra
ebc024df22
Show hint how to enable experimental features
2020-05-12 15:47:09 +02:00
Ben Burdette
b93c1bf3d6
fixes to merged code
2020-05-11 15:52:15 -06:00
Ben Burdette
59b1f5c701
Merge branch 'master' into errors-phase-2
2020-05-11 14:35:30 -06:00
Ben Burdette
958e81987b
switch from printError warnings to logWarnings
2020-05-11 13:02:16 -06:00
Domen Kožar
5bdb67c843
Merge pull request #3568 from kolloch/outputHashModeError
...
libstore/build.cc: more explicit error about form of output
2020-05-11 18:14:32 +02:00
Domen Kožar
1d8144e36b
Update src/libstore/build.cc
2020-05-11 18:14:23 +02:00
Benjamin Hipple
146f9c114f
doc: consistently refer to 'fixed-output' with a dash
...
General cleanup that makes it easier to search for the term.
2020-05-09 10:58:43 -04:00
Alexander Bantyev
772e5db828
Mention build users in the 'waiting for' message
2020-05-08 12:29:00 +03:00
Alexander Bantyev
14073fb76b
Don't block while waiting for build users
2020-05-08 12:22:39 +03:00
Ben Burdette
e3901638b5
todo removal
2020-05-06 15:01:13 -06:00
Eelco Dolstra
2f8ee4578f
Merge remote-tracking branch 'origin/master' into flakes
2020-05-06 12:01:40 +02:00
Eelco Dolstra
74a1bfdcab
Merge pull request #3546 from guibou/nix_readfile_on_0_sized_files
...
builtins.readFile: do not truncate content
2020-05-06 11:33:55 +02:00
Peter Kolloch
9be46859a9
libstore/build.cc: more explicit about form of output
...
Be more explicit about why we expect a regular file as output
when outputHashMode=flat for a fixed output derivation.
2020-05-06 11:21:12 +02:00
Eelco Dolstra
02c5914ea4
Merge pull request #3562 from pikajude/master
...
Use fragment size for autoGC capacity calculation
2020-05-06 11:19:30 +02:00
Eelco Dolstra
fd4911269f
Revert "Merge pull request #3558 from LnL7/ssh-ng-stderr"
...
This reverts commit 3ebfbecdd1c089c52d5fhttps://github.com/NixOS/nix/pull/3558
2020-05-06 10:54:18 +02:00
Eelco Dolstra
6f3244ce45
Merge remote-tracking branch 'origin/master' into flakes
2020-05-05 18:59:33 +02:00
Alexander Bantyev
04967dee9d
Wait for build users when none are available
2020-05-05 13:04:36 +03:00
Ben Burdette
7ffb5efdbc
appending to hints; remove _printError
2020-05-04 16:19:57 -06:00
Jude Taylor
e2fc575c61
nix auto-gc: use fragment size
2020-05-04 14:42:06 -07:00
Eelco Dolstra
a721a0b114
Flag: Use designated initializers
2020-05-04 22:40:19 +02:00
Ben Burdette
ab6f0b9641
convert some printError calls to logError
2020-05-03 08:01:25 -06:00
Daiderd Jordan
f16e24f95e
remote-store: don't log raw stderr by default
...
For remote stores the log messages are already forwarded as structured
STDERR_RESULT messages so the old format is duplicate information. But
still included with -vvv since it could be useful for debugging
problems.
$ nix build -L /nix/store/nl71b2niws857ffiaggyrkjwgx9jjzc0-foo.drv --store ssh-ng://localhost
Hello World!
foo> Hello World!
[1/0/1 built] building foo
Fixes #3556
2020-05-02 23:40:50 +02:00
Ben Burdette
4b99c09f5c
convert some errors
2020-05-01 14:32:06 -06:00
Eelco Dolstra
941f95284a
Merge remote-tracking branch 'origin/master' into flakes
2020-05-01 11:59:56 +02:00
Ben Burdette
39ff80d031
errorinfo constructor test
2020-04-29 18:57:05 -06:00
Guillaume Bouchard
7afcb5af98
Remove the drain argument from readFile
...
Now it is always `drain` (see previous commit).
2020-04-29 18:43:45 +02:00
Antoine Eiche
ca93b26db6
Only call grantpt on MacOS systems
...
The commit 3cc1125595
2020-04-29 15:43:20 +02:00
Ben Burdette
22e6490311
Error classname as name
2020-04-28 21:06:08 -06:00
Eelco Dolstra
ef4d3fc111
Merge remote-tracking branch 'origin/master' into flakes
2020-04-23 15:16:18 +02:00
Dustin DeWeese
c0d940978a
Replace select() with poll() to allow waiting on more than FD_SETSIZE fds
2020-04-21 16:21:28 -07:00
Ben Burdette
e4fb9a3849
remove 'format' from Error constructor calls
2020-04-21 17:07:07 -06:00
Ben Burdette
15e9564fd1
logEI for tunnelLogger and progressbar
2020-04-19 17:16:51 -06:00
Ben Burdette
3d5b1032a1
logError, logWarning; Logger functions; switch to Verbosity enum
2020-04-17 15:07:44 -06:00
Eelco Dolstra
aaa109565e
Use a more space/time-efficient representation for the eval cache
2020-04-17 23:04:21 +02:00
Eelco Dolstra
7a9687ba30
SQLiteStmt: Use std::string_view
2020-04-17 01:00:56 +02:00
Eelco Dolstra
0858738355
Merge remote-tracking branch 'origin/master' into flakes
2020-04-16 18:27:37 +02:00
Eelco Dolstra
a118293bd0
Merge pull request #3458 from zimbatm/nix-user-conf-dir
...
NIX_USER_CONF_FILES
2020-04-15 13:00:28 +02:00
zimbatm
895516cadf
add NIX_USER_CONF_FILES
...
Motivation: maintain project-level configuration files.
Document the whole situation a bit better so that it corresponds to the
implementation, and add NIX_USER_CONF_FILES that allows overriding
which user files Nix will load during startup.
2020-04-14 18:45:06 +02:00
Eelco Dolstra
c0c2cb871d
Merge remote-tracking branch 'origin/master' into flakes
2020-04-14 13:02:55 +02:00
Bruce Toll
8132d0a12e
Fix nix-build --check -K in sandbox w/o root
...
Temporarily add user-write permission to build directory so that it
can be moved out of the sandbox to the store with a .check suffix.
This is necessary because the build directory has already had its
permissions set read-only, but write permission is required
to update the directory's parent link to move it out of the sandbox.
Updated the related --check "derivation may not be deterministic"
messages to consistently use the real store paths.
Added test for non-root sandbox nix-build --check -K to demonstrate
issue and help prevent regressions.
2020-04-10 16:23:10 -04:00
Domen Kožar
db25a6d7bb
Merge pull request #2689 from tollb/fix/delete_tmp_dir_when_build_check_ok
...
Delete temporary directory on successful build
2020-04-10 09:47:57 +02:00
Bruce Toll
16a4864759
Delete temporary directory on successful build
...
With --check and the --keep-failed (-K) flag, the temporary directory
was being retained regardless of whether the build was successful and
reproducible. This removes the temporary directory, as expected, on
a reproducible check build.
Added tests to verify that temporary build directories are not
retained unnecessarily, particularly when using --check with
--keep-failed.
2020-04-09 16:37:41 -04:00
Philipp Middendorf
04bedda0b6
gc.cc: Ignore hidden files in temproots
2020-04-09 17:05:29 +02:00
Eelco Dolstra
d103c79144
Merge remote-tracking branch 'origin/master' into flakes
2020-04-09 15:54:21 +02:00
Nikola Knezevic
f5095594e7
datatransfer.{cc,hh} -> filetransfer.{cc,hh}
2020-04-08 22:26:57 +02:00
Nikola Knezevic
c330109bfa
DataTransfer -> FileTransfer
2020-04-08 22:26:57 +02:00
Nikola Knezevic
7848372b0f
Add upload method
2020-04-08 22:26:57 +02:00
Nikola Knezevic
a0c5931208
actDownload -> actDataTransfer
2020-04-08 22:26:57 +02:00
Nikola Knezevic
c4c1ae0a00
DownloadError -> DataTransferError
2020-04-08 22:26:57 +02:00
Nikola Knezevic
213d124277
DownloadItem -> TransferItem
2020-04-08 22:26:57 +02:00
Nikola Knezevic
cd391206e6
{get,make,new}Downloader -> DataTransfer
2020-04-08 22:26:57 +02:00
Nikola Knezevic
2df2741ec6
enqueueDownload -> enqueueDataTransfer
2020-04-08 22:26:57 +02:00
Nikola Knezevic
142ed7fe45
DownloadResult -> DataTransferResult
2020-04-08 22:26:57 +02:00
Nikola Knezevic
e5cc53beec
DownloadSettings -> DataTransferSettings
2020-04-08 22:26:57 +02:00
Nikola Knezevic
741e9012d3
Rename src/lib/download.* to src/lib/datatransfer.*
2020-04-08 22:26:57 +02:00
Nikola Knezevic
65ef57e0cb
DownloadRequest -> DataTransferRequest
2020-04-08 22:26:57 +02:00
Domen Kožar
bf81b31559
build.cc: improve message if home directory exists
2020-04-08 18:27:10 +02:00
Eelco Dolstra
1ab8d6ac18
Downloader: Only write data to the sink on a 200 response
...
Hopefully fixes #3278 .
2020-04-08 15:27:09 +02:00
Eelco Dolstra
54955867a6
Merge remote-tracking branch 'origin/master' into flakes
2020-04-07 14:29:45 +02:00
Eelco Dolstra
cd39709003
Cleanup
2020-04-07 09:27:17 +02:00
Eelco Dolstra
462421d345
Backport libfetchers from the flakes branch
...
This provides a pluggable mechanism for defining new fetchers. It adds
a builtin function 'fetchTree' that generalizes existing fetchers like
'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a
set of attributes, e.g.
fetchTree {
type = "git";
url = "https://example.org/repo.git ";
ref = "some-branch";
rev = "abcdef...";
}
The existing fetchers are just wrappers around this. Note that the
input attributes to fetchTree are the same as flake input
specifications and flake lock file entries.
All fetchers share a common cache stored in
~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching
mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}).
This also adds support for Git worktrees (c169ea5904
2020-04-07 09:03:14 +02:00
Cole Helbling
c976cb0b8a
Don't retry on "unsupported protocol" error
...
When encountering an unsupported protocol, there's no need to retry.
Chances are, it won't suddenly be supported between retry attempts;
error instead. Otherwise, you see something like the following:
$ nix-env -i -f git://git@github.com/foo/bar
warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 335 ms
warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 604 ms
warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 1340 ms
warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 2685 ms
With this change, you now see:
$ nix-env -i -f git://git@github.com/foo/bar
error: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1)
2020-04-05 09:00:34 -07:00
Eelco Dolstra
4fdec5f61d
Merge remote-tracking branch 'origin/master' into flakes
2020-04-01 00:03:50 +02:00
mlatus
12556e5709
fix placeholder not substituted in passAsFile
2020-03-31 19:40:16 +08:00
John Ericson
51afea3af2
Never cast FileIngestionMethod to or from boolean
2020-03-30 22:31:51 +00:00
John Ericson
c251b011cd
Merge remote-tracking branch 'upstream/master' into enum-FileIngestionMethod
2020-03-30 18:16:44 -04:00
John Ericson
832bd534dc
Store parsed hashes in DerivationOutput
...
It's best to detect invalid data as soon as possible, with data types
that make storing it impossible.
2020-03-30 11:33:35 -04:00
John Ericson
f5494d9442
Merge remote-tracking branch 'me/enum-FileIngestionMethod' into HEAD
2020-03-30 11:08:13 -04:00
Eelco Dolstra
d15d91cad1
Makefile cleanup
2020-03-30 16:50:13 +02:00
Eelco Dolstra
e322a16523
Remove global -I flags
...
(cherry picked from commit 2c692a3b14
2020-03-30 15:30:19 +02:00
Eelco Dolstra
2c692a3b14
Remove global -I flags
2020-03-30 14:39:33 +02:00
Eelco Dolstra
e0a0ae0467
Move fetchers from libstore to libfetchers
2020-03-30 14:04:53 +02:00
Eelco Dolstra
2287cc6486
Fix segfault
2020-03-30 13:31:55 +02:00
John Ericson
225e62a56a
Replace some bool recursive with a new FileIngestionMethod enum
2020-03-29 15:16:20 -04:00
John Ericson
87b32bab05
Use enum struct and drop prefixes
...
This does a few enums; the rest will be gotten in subsequent commits.
2020-03-29 11:23:15 -04:00
Eelco Dolstra
2287e2f279
nix flake info: Show flake subdirectory
2020-03-28 18:05:50 +01:00
John Ericson
e433d4af4c
Extend Rust FFI
...
Do idiomatic C++ copy and move constructors for a few things, so
wrapping structs' defaults can work.
2020-03-25 16:12:14 -04:00
John Ericson
bcde5456cc
Flip dependency so store-api.hh includes derivations.hh
...
I think it makes more sense to define the data model (derivations),
before the operations (store api).
2020-03-24 20:39:45 +00:00
Eelco Dolstra
0a10854f85
Misc changes from the flakes branch
2020-03-24 14:34:47 +01:00
Eelco Dolstra
777e21e596
nix path-info --json: Print hash in SRI format
...
(cherry picked from commit 442e665d6d
2020-03-24 14:26:23 +01:00
Eelco Dolstra
f9611c7ae4
buildenv: Eliminate global variables, other cleanup
...
(cherry picked from commit b82f75464d
2020-03-24 14:06:47 +01:00
Eelco Dolstra
edc34cc1a2
Add function for quoting strings
...
(cherry picked from commit 7dcf5b011a
2020-03-24 13:44:04 +01:00
Eelco Dolstra
5a7e7fc35f
Use std::string_view
...
(cherry picked from commit 6529490cc1
2020-03-24 13:26:37 +01:00
Eelco Dolstra
7abe3bde8a
Merge remote-tracking branch 'origin/master' into flakes
2020-03-23 13:27:54 +01:00
Eelco Dolstra
5885e20404
Merge pull request #3429 from LnL7/darwin-sandbox
...
darwin sandbox
2020-03-23 09:13:36 +01:00
Daiderd Jordan
2e9bc1245c
sandbox: fix /bin/sh on catalina
...
Sadly 10.15 changed /bin/sh to a shim which executes bash, this means it
can't be used anymore without also opening up the sandbox to allow bash.
Failed to exec /bin/bash as variant for /bin/sh (1: Operation not permitted).
2020-03-20 22:12:30 +01:00
Daiderd Jordan
f6c122aaeb
sandbox: allow pty devices
...
Nix now runs builds with a pseudo-terminal to enable colored build
output.
2020-03-20 21:58:45 +01:00
Daiderd Jordan
7f2df903d9
libstore: relax default sandbox-paths on darwin
2020-03-20 21:31:20 +01:00
Daiderd Jordan
afb78ebd34
libstore: disable resolve-system-dependencies hook
...
This is used to determine the dependency tree of impure libraries so nix
knows what paths to open in the sandbox. With the less restrictive
defaults it isn't needed anymore.
2020-03-20 21:21:56 +01:00
John Ericson
e317324236
Apply suggestions from code review
2020-03-19 23:38:51 -04:00
John Ericson
d5b3328dd1
Apply suggestions from code review
...
Co-Authored-By: Cole Helbling <cole.e.helbling@outlook.com>
2020-03-19 23:37:52 -04:00
John Ericson
f1cf3ab870
hashDerivationModulo: Generalize for multiple fixed ouputs per drv
...
See documentattion in header and comments in implementation for details.
This is actually done in preparation for floating ca derivations, not
multi-output fixed ca derivations, but the distinction doesn't yet
mattter.
Thanks @cole-h for finding and fixing a bunch of typos.
2020-03-19 10:30:49 -04:00
Eelco Dolstra
8aa354fdfd
Register flake-registry.json as a GC root again
2020-03-19 11:45:34 +01:00
Eelco Dolstra
1c127e6a82
downloadFile(): Use expired file if the download fails
2020-03-19 11:42:50 +01:00
John Ericson
049179ba07
Fix typos
...
Thanks @asymmetric
I failed to do them all in one batch
Co-Authored-By: asymmetric <lorenzo@mailbox.org>
2020-03-18 19:07:05 -04:00
Eelco Dolstra
f6ddf48882
Get rid of downloadCached()
...
Everything uses the generic caching system now.
2020-03-18 17:24:19 +01:00
Eelco Dolstra
c5ec95e2c7
tarball.cc: Use ETags
2020-03-18 15:14:23 +01:00
Eelco Dolstra
1e7ce1d6da
tarball / github fetchers: Use generic caching system
2020-03-18 14:08:25 +01:00
Eelco Dolstra
38e360154d
Git: Use unified caching system
2020-03-17 22:35:29 +01:00
Eelco Dolstra
d1165d8791
Require shallow clones to be requested explicitly
...
If you do a fetchTree on a Git repository, whether the result contains
a revCount attribute should not depend on whether that repository
happens to be a shallow clone or not. That would complicate caching a
lot and would be semantically messy. So applying fetchTree/fetchGit to
a shallow repository is now an error unless you pass the attribute
'shallow = true'. If 'shallow = true', we don't return revCount, even
if the repository is not actually shallow.
Note that Nix itself is not doing shallow clones at the moment. But it
could do so as an optimisation if the user specifies 'shallow = true'.
Issue #2988 .
2020-03-17 22:35:29 +01:00
Eelco Dolstra
2a4e4f6a6e
Unified fetcher caching system
2020-03-17 22:35:29 +01:00
John Ericson
e5178fd22d
Fix typos
...
Thanks @asymmetric!
Co-Authored-By: asymmetric <lorenzo@mailbox.org>
2020-03-16 16:40:13 -04:00
Eelco Dolstra
fbcb897e21
Add a test for shallow Git clones
...
Also, don't return a revCount anymore for shallow or dirty Git trees,
since it's incorrect.
Closes #2988 .
2020-03-16 13:20:32 +01:00
John Ericson
2be64efb02
Generalize isFixedOutput in preparation for CA drvs
...
Today's fixed output derivations and regular derivations differ in a few
ways which are largely orthogonal. This replaces `isFixedOutput` with a
`type` that returns an enum of possible combinations.
2020-03-15 11:05:37 -04:00
Eelco Dolstra
0c2088d438
Merge remote-tracking branch 'origin/master' into flakes
2020-03-13 17:03:30 +01:00
Eelco Dolstra
b816515f61
Fix ca-references feature check
...
Fixes #3406 .
2020-03-13 13:15:51 +01:00
Robert Hensing
3f55f8a8fb
pathInfoCache: Respect disk cache TTLs #3398
2020-03-12 10:30:28 +01:00
Eelco Dolstra
e188fe7c6d
Move call-flake.nix into libexpr
2020-03-11 17:04:51 +01:00
Eelco Dolstra
73b6d87e17
Merge remote-tracking branch 'origin/master' into flakes
2020-03-04 13:58:42 +01:00
Eelco Dolstra
d37dc71e3c
nix-build: Fix !<output> handling
...
This was broken by 22a754c091https://hydra.nixos.org/eval/1573669
2020-03-04 13:56:17 +01:00
Eelco Dolstra
75db069f92
Optimise Derivation::unparse()
...
In
nix-instantiate --dry-run '<nixpkgs/nixos/release-combined.nix>' -A nixos.tests.simple.x86_64-linux
this reduces time spent in unparse() from 9.15% to 4.31%. The main
culprit was appending characters one at a time to the destination
string. Even though the string has enough capacity, push_back() still
needs to check this on every call.
2020-03-04 11:44:45 +01:00
Eelco Dolstra
22a754c091
Fix GC failures on bad store path names
...
It failed on names like '/nix/store/9ip48nkc9rfy0a4yaw98lp6gipqlib1a-'.
2020-02-28 18:07:10 +01:00
Eelco Dolstra
4a4521f462
Fix nlohmann::json exception
2020-02-20 23:47:02 +01:00
Eelco Dolstra
73c9840569
Restore subdir -> dir
...
Got this mixed up somewhere.
2020-02-20 23:44:06 +01:00
Eelco Dolstra
d068f9ffff
Restore subdir support in registries
...
Hacky...
2020-02-20 22:14:44 +01:00
Eelco Dolstra
1d99c4ab25
Merge pull request #3229 from Ma27/flakes-fetchgit-worktree-support
...
builtins.fetchGit: Fix build when fetching a git worktree
2020-02-19 14:11:18 +01:00
