max/nix-super
1
0
Fork 0
mirror of https://github.com/privatevoid-net/nix-super.git synced 2024-11-27 16:16:16 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
9466 commits 7 branches 0 tags 64 MiB
Commit graph

2727 commits

Author SHA1 Message Date
John Ericson
993229cdaf Deduplicate basic derivation goals too 
See comments for security concerns.

Also optimize goal creation by not traversing map twice.
 2020-09-22 17:13:59 +00:00
John Ericson
e9fc2031f0 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-22 14:18:31 +00:00
regnat
67cc356bb3 Merge remote-tracking branch 'origin/master' into non-ca-depending-on-ca 2020-09-22 16:11:25 +02:00
Eelco Dolstra
980edd1f3a RemoteStore::addCAToStore(): Don't hold connection while calling queryPathInfo() 
This leads to a deadlock if we're at the connection limit.
 2020-09-22 15:28:20 +02:00
Eelco Dolstra
5b107f2c5f
Merge pull request #4038 from maljub01/master 
Add a nix.conf option for allowing a symlinked store
 2020-09-22 13:19:22 +02:00
Eelco Dolstra
35a0ac1838 Style fixes 2020-09-22 11:40:19 +02:00
Eelco Dolstra
92ac8df0ec Merge branch 'add-ca-to-store' of https://github.com/hercules-ci/nix 2020-09-22 11:31:33 +02:00
John Ericson
3786a801c3 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-22 04:15:55 +00:00
John Ericson
b92d3b2edd Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-22 00:45:55 +00:00
Cole Helbling
ba37299a03
Serialize SandboxMode enum to string for JSON 
Rather than showing an integer as the default, instead show the boolean
referenced in the description.

The nix.conf.5 manpage used to show "default: 0", which is unnecessarily
opaque and confusing (doesn't 0 mean false, even though the default is
true?); now it properly shows that the default is true.
 2020-09-21 10:36:45 -07:00
Eelco Dolstra
0716adaa8b abstractsettingtojson.hh -> abstract-setting-to-json.hh 2020-09-21 18:49:43 +02:00
Eelco Dolstra
340ca382c4 Don't include nlohmann/json.hpp in globals.hh 
This reduces compilation time by 207s.

Issue #4045.
 2020-09-21 18:47:18 +02:00
Eelco Dolstra
d51ba43047 Move Callback into its own header 
This gets rid of the inclusion of <future> in util.hh, cutting
compilation time by ~20s (CPU time).

Issue #4045.
 2020-09-21 18:42:21 +02:00
Marwan Aljubeh
f80ffeb8c9 Update the variable name accordingly 2020-09-21 17:29:08 +01:00
Eelco Dolstra
e8e1d420f3 Don't include <regex> in header files 
This reduces compilation time by ~15 seconds (CPU time).

Issue #4045.
 2020-09-21 18:22:45 +02:00
Marwan Aljubeh
4e1a04733d Use a better name for the config option 2020-09-21 16:32:22 +01:00
Robert Hensing
ca30abb3fb Document addCAToStore/addToStoreFromDump source drainage 
Also checked that all usages satisfy the requirement and
removed dead code.
 2020-09-21 07:55:47 +02:00
Robert Hensing
fa08db5c4c wopAddToStore: return ValidPathInfo 
A ValidPathInfo is created anyway. By returning it we can save a
roundtrip and we have a nicer interface.
 2020-09-21 07:55:47 +02:00
Robert Hensing
7c68264085 wopAddToStore: add RepairFlag 2020-09-21 07:55:47 +02:00
Robert Hensing
fbf509c113 parseContentAddressMethodPrefix: use string_view 
Co-authored-by: John Ericson <git@JohnEricson.me>
 2020-09-21 07:55:47 +02:00
Robert Hensing
8279178b07 Move FramedSink next to FramedSource 2020-09-21 07:55:47 +02:00
Robert Hensing
ecc8088cb7 wopAddToStore: Throw to clarify unused refs 
Co-authored-by: John Ericson <git@JohnEricson.me>
 2020-09-21 07:55:47 +02:00
Robert Hensing
c602ebfb34 Refactor wopAddToStore to make wopAddTextToStore obsolete 2020-09-21 07:55:45 +02:00
Robert Hensing
e34fe47d0c Overhaul wopAddToStore 2020-09-21 07:54:05 +02:00
Marwan Aljubeh
e40772cd35 Lint issue: replacing tabs with spaces 2020-09-18 17:18:45 +01:00
Marwan Aljubeh
c00e078343 Add a nix.conf option for allowing a symlinked store 2020-09-18 17:10:39 +01:00
Robert Hensing
14b30b3f3d Move FramedSource and FramedSink, extract withFramedSink 2020-09-17 20:21:04 +02:00
Robert Hensing
dfa547c6a8 Add ContentAddressMethod and parse/render it 2020-09-17 20:21:04 +02:00
Robert Hensing
9ee3122ec7 Remove redundant import 2020-09-17 20:21:04 +02:00
John Ericson
b7df353f27 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-17 16:33:10 +00:00
Eelco Dolstra
649d3aaf24
Merge pull request #3829 from obsidiansystems/remove-storetype-delegate-regStore 
Remove storetype delegate reg store -- contains #3736
 2020-09-17 13:55:01 +02:00
Eelco Dolstra
b94a35ef40
Merge pull request #4027 from tweag/fix-gc-of-ca-derivations 
Fix garbage collection of CA derivations
 2020-09-17 13:46:26 +02:00
regnat
520895b1da Fix garbage collection of CA derivations 
Fix #4026
 2020-09-17 13:36:58 +02:00
John Ericson
f60b380a7f Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore 2020-09-16 22:35:24 +00:00
John Ericson
2741fffa35 Ensure resolved CA derivations are written 
so we can link outputs to deriver and thus properly cache.
 2020-09-16 17:57:32 +00:00
John Ericson
7fdbb377ba Start to fix floating CA + remote building 2020-09-16 17:50:48 +00:00
John Ericson
c5ccebae00 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-16 17:50:40 +00:00
Greg Hale
a303c0b6dc Fetch commits from github/gitlab using Auth header 
`nix flake info` calls the github 'commits' API, which requires
authorization when the repository is private. Currently this request
fails with a 404.

This commit adds an authorization header when calling the 'commits' API.
It also changes the way that the 'tarball' API authenticates, moving the
user's token from a query parameter into the Authorization header.

The query parameter method is recently deprecated and will be disallowed
in November 2020. Using them today triggers a warning email.
 2020-09-16 13:46:48 -04:00
Théophane Hufschmitt
77a0e2c5be
Remove useless exception copy 
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-09-16 14:00:21 +02:00
regnat
d72927aa7a Fix the s3 store 
Add some necessary casts in the initialisation of the store's config
 2020-09-16 13:53:09 +02:00
regnat
e0817cbcdc Don't include nlohmann/json.hpp in config.hh 
Instead make a separate header with the template implementation of
`BaseSetting<T>::toJSONObj` that can be included where needed
 2020-09-16 13:53:09 +02:00
regnat
fc2d31c423 Add (StoreConfig*) casts to work around a GCC bug 
Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80431 that was
already there in the code but was accidentally removed in the last
commits
 2020-09-16 13:53:09 +02:00
regnat
888f7afe9f Fix build issues with gcc 2020-09-16 13:53:09 +02:00
regnat
b73adacc1e Add a name to the stores 
So that it can be printed by `nix describe-stores`
 2020-09-16 13:53:09 +02:00
regnat
f24f0888f9 Document the new store hierarchy 2020-09-16 13:53:09 +02:00
regnat
29a632386e fixup! Make the store plugins more introspectable 2020-09-16 13:53:09 +02:00
regnat
1129913c4e fixup! Correctly call all the parent contructors of the stores 2020-09-16 13:53:09 +02:00
regnat
7f103dcddd Properly filter the stores according to their declared uriSchemes 
When opening a store, only try the stores whose `uriSchemes()` include
the current one
 2020-09-16 13:53:09 +02:00
regnat
5895184df4 Correctly call all the parent contructors of the stores 
Using virtual inheritance means that only the default constructors of
the parent classes will be called, which isn't what we want
 2020-09-16 13:53:09 +02:00
regnat
22afa8fb4d Separate store configs from the implems 
Rework the `Store` hierarchy so that there's now one hierarchy for the
store configs and one for the implementations (where each implementation
extends the corresponding config). So a class hierarchy like

```
StoreConfig-------->Store
    |                 |
    v                 v
SubStoreConfig----->SubStore
    |                 |
    v                 v
SubSubStoreConfig-->SubSubStore
```

(with virtual inheritance to prevent DDD).

The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
 2020-09-16 13:53:08 +02:00
regnat
aa4eac3788 fixup! Separate the instantiation and initialisation of the stores 2020-09-16 13:53:08 +02:00
regnat
3c525d1590 Complete the toJSON instance for Setting<T> 
Don't let it just contain the value, but also the other fields of the
setting (description, aliases, etc..)
 2020-09-16 13:53:08 +02:00
regnat
3b57181f8e Separate the instantiation and initialisation of the stores 
Add a new `init()` method to the `Store` class that is supposed to
handle all the effectful initialisation needed to set-up the store.
The constructor should remain side-effect free and just initialize the
c++ data structure.

The goal behind that is that we can create “dummy” instances of each
store to query static properties about it (the parameters it accepts for
example)
 2020-09-16 13:53:08 +02:00
regnat
fa32560169 Fix the registration of stores 2020-09-16 13:53:08 +02:00
regnat
7d5bdf8b56 Make the store plugins more introspectable 
Directly register the store classes rather than a function to build an
instance of them.
This gives the possibility to introspect static members of the class or
choose different ways of instantiating them.
 2020-09-16 13:53:08 +02:00
John Ericson
3a5cdd737c Rename Derivation::pathOpt to Derivation::path 
We no longer need the `*Opt` to disambiguate.
 2020-09-15 15:21:39 +00:00
John Ericson
6387550d58 Get rid of confusing std::optional<bool> for validity 2020-09-15 15:19:45 +00:00
John Ericson
c4bf219b55 Don't link deriver until after any delayed exception is thrown 
Otherwise, we will associate fixed-output derivations with outputs that
they did indeed produce, but which had the wrong hash. That's no good.
 2020-09-15 14:28:06 +00:00
John Ericson
3ba552b245 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-15 14:17:06 +00:00
John Ericson
c08c9f08c7 Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore 2020-09-15 14:08:35 +00:00
regnat
057c6203b5 gracefully handle old daemon versions 
Add a fallback path in `queryPartialDerivationOutputMap` for daemons
that don't support it.

Also upstreams a couple methods from `SSHStore` to `RemoteStore` as this
is needed to handle the fallback path.
 2020-09-15 09:58:01 +02:00
Eelco Dolstra
a59e77d9e5 nix-daemon: Lower verbosity of restricted setting warning 
Fixes #3992.
 2020-09-14 13:48:51 +02:00
aszlig
525b38eee8
Fix unspecified behaviour in readStorePathCAMap 
When deploying a Hydra instance with current Nix master, most builds
would not run because of errors like this:

  queue monitor: error: --- Error --- hydra-queue-runner
  error: --- UsageError --- nix-daemon
  not a content address because it is not in the form '<prefix>:<rest>': /nix/store/...-somedrv

The last error message is from parseContentAddress, which expects a
colon-separated string, however what we got here is a store path.

Looking at the worker protocol, the following message sent to the Nix
daemon caused the error above:

  0x1E -> wopQuerySubstitutablePathInfos
  0x01 -> Number of paths
  0x16 -> Length of string
  "/nix/store/...-somedrv"
  0x00 -> Length of string
  ""

Looking at writeStorePathCAMap, the store path is indeed the first field
that's transmitted. However, readStorePathCAMap expects it to be the
*second* field *on my machine*, since expression evaluation order is a
classic form of unspecified behaviour[1] in C++.

This has been introduced in https://github.com/NixOS/nix/pull/3689,
specifically in commit 66a62b3189.

[1]: https://en.wikipedia.org/wiki/Unspecified_behavior#Order_of_evaluation_of_subexpressions

Signed-off-by: aszlig <aszlig@nix.build>
 2020-09-13 02:40:51 +02:00
John Ericson
98dfd7531d Fix querying outputs for CA derivations some more 
If we resolve using the known path of a derivation whose output we
didn't have, we previously blew up. Now we just fail gracefully,
returning the map of all outputs unknown.
 2020-09-04 18:33:58 +00:00
John Ericson
075d399e3f Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-09-04 16:04:35 +00:00
John Ericson
e9fad3006b Fix some of the issues raised by @edolstra 
- More and better comments

 - The easier renames
 2020-09-04 15:18:25 +00:00
John Ericson
e86dd59dcc
Apply suggestions from code review 
Thanks!

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-09-04 10:48:50 -04:00
John Ericson
c9f1ed912c Don't chmod symlink before moving outputs around 
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
 2020-09-04 14:41:53 +00:00
John Ericson
aad4abcc9c Fix floating CA tests 
We will sometimes try to query the outputs of derivations we can't
resolve. That's fine; it just means we don't know what those outputs are
yet.
 2020-09-04 01:17:38 +00:00
John Ericson
975a47f7fe Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-09-03 22:09:04 +00:00
John Ericson
e7d93e7ece Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-03 15:43:17 +00:00
Eelco Dolstra
8a945d6ddb
Remove gc-check-reachability 2020-09-03 11:30:15 +02:00
Eelco Dolstra
82b77a7726
addPermRoot(): Remove indirect flag 2020-09-03 11:29:34 +02:00
Eelco Dolstra
b07167be5a
createGeneration(): Always create an indirect root 
This means profiles outside of /nix/var/nix/profiles don't get
garbage-collected. It also means we don't need to scan
/nix/var/nix/profiles for GC roots anymore, except for compatibility
with previously existing generations.
 2020-09-03 11:13:53 +02:00
Eelco Dolstra
b74f5cdd23
createGeneration(): Take a StorePath 2020-09-03 11:06:56 +02:00
Domen Kožar
dd4b56c87f
Allow HTTP binary cache to request absolute uris 2020-09-01 21:35:48 +02:00
John Ericson
ef278d00f9 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-01 18:01:48 +00:00
Eelco Dolstra
84f5cabbea Merge remote-tracking branch 'origin/master' into markdown 2020-08-31 14:24:26 +02:00
John Ericson
02e0001fc0 Merge branch 'ca-floating-upstream' of github.com:obsidiansystems/nix into ca-floating-upstream 2020-08-28 21:23:09 +00:00
John Ericson
421ed527c7
Update src/libstore/build.cc 
Thanks for catching, @regnat.
 2020-08-28 17:22:57 -04:00
John Ericson
48e8828050 Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-08-28 20:38:25 +00:00
John Ericson
8017fe7487 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-28 19:59:14 +00:00
Eelco Dolstra
eb75282b8d
Merge pull request #3434 from Ericson2314/derivation-header-include-order 
Revise division of labor in deserialization of derivations
 2020-08-27 16:39:28 +02:00
Eelco Dolstra
a0f19d9f3a RemoteStore::addToStore(): Fix race between stderrThread and NAR writer 
As pointed out by @B4dM4n, the call to to.flush() on stderrThread is
unsafe because the NAR writer thread is also writing to 'to'.

Fixes #3943.
 2020-08-27 14:50:51 +02:00
Eelco Dolstra
4bf5faf416 Merge remote-tracking branch 'origin/master' into markdown 2020-08-25 19:47:34 +02:00
John Ericson
8eb73a8724 CA derivations that depend on other CA derivations 
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
 2020-08-24 19:01:15 +00:00
John Ericson
e0b0e18905 Add constructor for BasicDerivation -> Derivation 2020-08-24 19:01:07 +00:00
John Ericson
59979e7053 Fix bad debug format string 2020-08-24 19:01:07 +00:00
Eelco Dolstra
6a67e57019
Add DummyStore (dummy://) 
DummyStore does not allow building or adding paths. This is useful for
evaluation tests when you don't want to initialize a "proper" store.
 2020-08-24 18:54:16 +02:00
Eelco Dolstra
2530cc6827
Merge pull request #3950 from obsidiansystems/tabs-spaces 
tabs -> spaces
 2020-08-24 10:20:28 +02:00
John Ericson
b0b59fd05a Merge remote-tracking branch 'obsidian/write-derivation-borrow' into HEAD 2020-08-23 15:11:10 +00:00
John Ericson
35e6288be1 writeDerivation just needs a plain store reference 2020-08-23 15:01:11 +00:00
John Ericson
3a7b330b64 "Downstream placeholders" should not be store paths 
Insead they should be opaque `/<hash>` like the placeholders we already
have.
 2020-08-21 19:35:35 +00:00
John Ericson
422affe102 tabs -> spaces 
Sorry I let the tab sneak in there in the first place.
 2020-08-21 19:19:24 +00:00
John Ericson
c265e0ea82 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-08-20 20:24:22 +00:00
John Ericson
27a3f82c0b Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-20 18:28:17 +00:00
John Ericson
45a2f1baab Rename drv output querying functions, like master 
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
 - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap
 2020-08-20 18:14:12 +00:00
Eelco Dolstra
4d77513d97
Merge pull request #3859 from obsidiansystems/drv-outputs-map-allow-missing 
`queryDerivationOutputMap` no longer assumes all outputs have a mapping
 2020-08-20 16:49:23 +02:00
John Ericson
9a9d834dc7 Rename drv output querying functions 
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
 - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap`
 2020-08-20 14:12:51 +00:00
Eelco Dolstra
acb99f03f9
Config: Use nlohmann/json 2020-08-20 11:02:16 +02:00