Commit graph

35 commits

Author SHA1 Message Date
Graham Christensen
1130b28824
distributed builds: load remote builder host key from the machines file
This is already used by Hydra, and is very useful when materializing
a remote builder list from service discovery. This allows the service
discovery tool to only sync one file instead of two.
2021-02-25 09:17:34 -05:00
John Ericson
1a1af75338 Overhaul store subclassing
We embrace virtual the rest of the way, and get rid of the
`assert(false)` 0-param constructors.

We also list config base classes first, so the constructor order is
always:

  1. all the configs
  2. all the stores

Each in the same order
2020-12-20 15:47:14 +00:00
Eelco Dolstra
85c8be6286 Remove static variable name clashes
This was useful for an experiment with building Nix as a single
compilation unit. It's not very useful otherwise but also doesn't
hurt...
2020-10-06 13:49:20 +02:00
John Ericson
f60b380a7f Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore 2020-09-16 22:35:24 +00:00
regnat
888f7afe9f Fix build issues with gcc 2020-09-16 13:53:09 +02:00
regnat
b73adacc1e Add a name to the stores
So that it can be printed by `nix describe-stores`
2020-09-16 13:53:09 +02:00
regnat
f24f0888f9 Document the new store hierarchy 2020-09-16 13:53:09 +02:00
regnat
7f103dcddd Properly filter the stores according to their declared uriSchemes
When opening a store, only try the stores whose `uriSchemes()` include
the current one
2020-09-16 13:53:09 +02:00
regnat
5895184df4 Correctly call all the parent contructors of the stores
Using virtual inheritance means that only the default constructors of
the parent classes will be called, which isn't what we want
2020-09-16 13:53:09 +02:00
regnat
22afa8fb4d Separate store configs from the implems
Rework the `Store` hierarchy so that there's now one hierarchy for the
store configs and one for the implementations (where each implementation
extends the corresponding config). So a class hierarchy like

```
StoreConfig-------->Store
    |                 |
    v                 v
SubStoreConfig----->SubStore
    |                 |
    v                 v
SubSubStoreConfig-->SubSubStore
```

(with virtual inheritance to prevent DDD).

The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-16 13:53:08 +02:00
regnat
3b57181f8e Separate the instantiation and initialisation of the stores
Add a new `init()` method to the `Store` class that is supposed to
handle all the effectful initialisation needed to set-up the store.
The constructor should remain side-effect free and just initialize the
c++ data structure.

The goal behind that is that we can create “dummy” instances of each
store to query static properties about it (the parameters it accepts for
example)
2020-09-16 13:53:08 +02:00
regnat
fa32560169 Fix the registration of stores 2020-09-16 13:53:08 +02:00
regnat
7d5bdf8b56 Make the store plugins more introspectable
Directly register the store classes rather than a function to build an
instance of them.
This gives the possibility to introspect static members of the class or
choose different ways of instantiating them.
2020-09-16 13:53:08 +02:00
John Ericson
c08c9f08c7 Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore 2020-09-15 14:08:35 +00:00
regnat
057c6203b5 gracefully handle old daemon versions
Add a fallback path in `queryPartialDerivationOutputMap` for daemons
that don't support it.

Also upstreams a couple methods from `SSHStore` to `RemoteStore` as this
is needed to handle the fallback path.
2020-09-15 09:58:01 +02:00
John Ericson
a83694c7a1 Use RemoteStore to open connection for proxying daemon
Removes duplicate websocket opening code, and also means we should be
able to to ssh-ssh-... daemon relays, not just uds-uds-... ones.
2020-08-19 19:34:47 +00:00
Eelco Dolstra
d82b78bf51
Fix segfault in gcc on i686-linux
src/libstore/ssh-store.cc: In constructor 'nix::SSHStore::SSHStore(const string&, const Params&)':
  src/libstore/ssh-store.cc:31:21: internal compiler error: Segmentation fault
               compress)
                       ^
  Please submit a full bug report,
  with preprocessed source if appropriate.

https://hydra.nixos.org/build/111545609
2020-02-07 13:01:48 +01:00
Daiderd Jordan
8745c63d3c
ssh-store: add remote-store and remote-program query params
Brings the functionality of ssh-ng:// in sync with the legacy ssh://
implementation.  Specifying the remote store uri enables various useful
things. eg.

    $ nix copy --to ssh-ng://cache?remote-store=file://mnt/cache --all
2020-02-03 23:22:28 +01:00
Eelco Dolstra
bbe97dff8b Make the Store API more type-safe
Most functions now take a StorePath argument rather than a Path (which
is just an alias for std::string). The StorePath constructor ensures
that the path is syntactically correct (i.e. it looks like
<store-dir>/<base32-hash>-<name>). Similarly, functions like
buildPaths() now take a StorePathWithOutputs, rather than abusing Path
by adding a '!<outputs>' suffix.

Note that the StorePath type is implemented in Rust. This involves
some hackery to allow Rust values to be used directly in C++, via a
helper type whose destructor calls the Rust type's drop()
function. The main issue is the dynamic nature of C++ move semantics:
after we have moved a Rust value, we should not call the drop function
on the original value. So when we move a value, we set the original
value to bitwise zero, and the destructor only calls drop() if the
value is not bitwise zero. This should be sufficient for most types.

Also lots of minor cleanups to the C++ API to make it more modern
(e.g. using std::optional and std::string_view in some places).
2019-12-10 22:06:05 +01:00
Eelco Dolstra
313106d549
Fix clang warnings 2019-11-26 21:07:44 +01:00
Eelco Dolstra
906d56a96b
ssh-ng: Don't set CPU affinity on the remote
Fixes #3138.
2019-10-11 18:49:46 +02:00
Eelco Dolstra
47f7e5585b Make 'nix copy --from ssh://...' run in constant memory
For instance, this reduced the memory consumption of

  $ nix copy --from ssh://localhost --to ~/my-nix /nix/store/1n7x0yv8vq6zi90hfmian84vdhd04bgp-blender-2.79a

from 632 MiB to 16 MiB.
2018-03-21 23:10:40 +01:00
Shea Levy
088ef81759
ssh-ng: Don't forward options to the daemon.
This can be iterated on and currently leaves out settings we know we
want to forward, but it fixes #1713 and fixes #1935 and isn't
fundamentally broken like the status quo. Future changes are suggested
in a comment.
2018-03-05 07:42:15 -05:00
Daiderd Jordan
2cb8aaa610
ssh-store: fix length when writing nar
This fixes nix copy and other things that use copyStorePath.
2017-12-06 11:41:08 +01:00
Eelco Dolstra
2040240e23
Add a Config class to simplify adding configuration settings
The typical use is to inherit Config and add Setting<T> members:

  class MyClass : private Config
  {
    Setting<int> foo{this, 123, "foo", "the number of foos to use"};
    Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"};

    MyClass() : Config(readConfigFile("/etc/my-app.conf"))
    {
      std::cout << foo << "\n"; // will print 123 unless overriden
    }
  };

Currently, this is used by Store and its subclasses for store
parameters. You now get a warning if you specify a non-existant store
parameter in a store URI.
2017-04-13 16:03:31 +02:00
Eelco Dolstra
287084d688
ssh:// -> ssh-ng://, legacy-ssh:// -> ssh:// 2017-03-16 14:19:32 +01:00
Eelco Dolstra
577ebeaefb
Improve SSH handling
* Unify SSH code in SSHStore and LegacySSHStore.

* Fix a race starting the SSH master. We now wait synchronously for
  the SSH master to finish starting. This prevents the SSH clients
  from starting their own connections.

* Don't use a master if max-connections == 1.

* Add a "max-connections" store parameter.

* Add a "compress" store parameter.
2017-03-03 19:05:50 +01:00
Eelco Dolstra
6f4682ad36
Merge branch 'nix-copy-closure-c++' of https://github.com/shlevy/nix 2017-02-07 20:47:45 +01:00
Eelco Dolstra
7a58ad0ef5
SSHStore: uri -> host 2017-02-07 19:20:15 +01:00
Eelco Dolstra
7a65b2470e
Restore default signal handling in child processes
In particular, this fixes Ctrl-C in nix-shell sessions.
2017-02-01 13:00:21 +01:00
Shea Levy
bfa41eb671 nix-copy-closure: Implement in C++.
Tests fail currently because the database is not given proper hashes in the VM
2017-01-20 09:47:58 -05:00
Shea Levy
167d12b02c build-remote: Implement in C++ 2016-11-10 11:09:15 -05:00
Shea Levy
196815f700 ssh-store: Start master on-demand 2016-09-12 08:07:50 -04:00
Shea Levy
ab31f9986c Inline ssh-store.hh into ssh-store.cc 2016-09-12 08:03:29 -04:00
Shea Levy
ecba88de93 Add ssh store implementation 2016-09-02 14:31:38 -04:00