Commit graph

19 commits

Author SHA1 Message Date
John Ericson
903700c5e1 Simplify ContentAddress
Whereas `ContentAddressWithReferences` is a sum type complex because different
varieties support different notions of reference, and
`ContentAddressMethod` is a nested enum to support that,
`ContentAddress` can be a simple pair of a method and hash.

`ContentAddress` does not need to be a sum type on the outside because
the choice of method doesn't effect what type of hashes we can use.

Co-Authored-By: Cale Gibbard <cgibbard@gmail.com>
2023-07-07 07:30:01 -04:00
John Ericson
9f69b7dee9 Create worker_proto::{Read,Write}Conn
Pass this around instead of `Source &` and `Sink &` directly. This will
give us something to put the protocol version on once the time comes.

To do this ergonomically, we need to expose `RemoteStore::Connection`,
so do that too. Give it some more API docs while we are at it.
2023-06-19 12:08:23 -04:00
John Ericson
95eae0c002 Put worker protocol items inside a WorkerProto struct
See API docs on that struct for why. The pasing as as template argument
doesn't yet happen in that commit, but will instead happen in later
commit.

Also make `WorkerOp` (now `Op`) and enum struct. This led us to catch
that two operations were not handled!

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-06-19 12:08:23 -04:00
John Ericson
469d06f9bc Split out worker protocol template definitions from declarations
This is generally a fine practice: Putting implementations in headers
makes them harder to read and slows compilation. Unfortunately it is
necessary for templates, but we can ameliorate that by putting them in a
separate header. Only files which need to instantiate those templates
will need to include the header with the implementation; the rest can
just include the declaration.

This is now documenting in the contributing guide.

Also, it just happens that these polymorphic serializers are the
protocol agnostic ones. (Worker and serve protocol have the same logic
for these container types.) This means by doing this general template
cleanup, we are also getting a head start on better indicating which
code is protocol-specific and which code is shared between protocols.
2023-06-19 11:45:59 -04:00
Théophane Hufschmitt
673fe85976
Merge pull request #8365 from obsidiansystems/proto-structs
Revert "Revert "Use template structs instead of phantoms""
2023-05-22 09:34:34 +02:00
John Ericson
9923403d90 Don't use store-api.hh in worker-protocol.hh
Using abstract types like can help cut down on compilation time, both
from scratch, and especially incremental builds during development. The
idea is that `worker-protocol.hh` can declare all the (de)serializers, but
only again abstract types; when code needs to use some (de)serializers, it can
include headers just for the data types it needs to (de)serialize.

`store-api.hh` in particular is a bit of a sledgehammer, and the data
types we want to serialize have their own headers.
2023-05-18 00:20:24 -04:00
John Ericson
cb5052d98f Revert "Revert "Use template structs instead of phantoms""
This is the more typically way to do [Argument-dependent
lookup](https://en.cppreference.com/w/cpp/language/adl)-leveraging
generic serializers in C++. It makes the relationship between the `read`
and `write` methods more clear and rigorous, and also looks more
familiar to users coming from other languages that do not have C++'s
libertine ad-hoc overloading.

I am returning to this because during the review in
https://github.com/NixOS/nix/pull/6223, it came up as something that
would make the code easier to read --- easier today hopefully already,
but definitely easier if we were have multiple codified protocols with
code sharing between them as that PR seeks to accomplish.

If I recall correctly, the main criticism of this the first time around
(in 2020) was that having to specify the type when writing, e.g.
`WorkerProto<MyType>::write`, was too verbose and cumbersome. This is
now addressed with the `workerProtoWrite` wrapper function.

This method is also the way `nlohmann::json`, which we have used for a
number of years now, does its serializers, for what its worth.

This reverts commit 45a0ed82f0. That
commit in turn reverted 9ab07e99f5.
2023-05-17 22:44:47 -04:00
John Ericson
c51d554c93 Use "raw pattern" for content address types
We weren't because this ancient PR predated it!

This is actually a new version of the pattern which addresses some
issues identified in #7479.
2023-03-30 17:12:49 -04:00
John Ericson
d381248ec0 No inheritance for TextInfo and FixedOutputInfo 2023-02-28 12:14:11 -05:00
John Ericson
85bb865d20 Revert "Remove some designated initializers"
This reverts commit ee9eb83a84.
2023-02-28 11:57:20 -05:00
John Ericson
c36b584f8e Fix typo in the method name 2023-02-28 11:34:18 -05:00
John Ericson
ee9eb83a84 Remove some designated initializers
With the switch to C++20, the rules became more strict, and we can no
longer initialize base classes. Make them comments instead.

(BTW
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2287r1.html
this offers some new syntax for this use-case. Hopefully this will be
adopted and we can eventually use it.)
2023-02-01 11:25:56 -05:00
John Ericson
4540e7b940 Don't add StorePathDescriptor for now
We don't need it yet, we can add it back later.
2023-01-23 12:58:27 -05:00
John Ericson
b3d91239ae Make ValidPathInfo have plain StorePathSet references like before
This change can wait for another PR.
2023-01-14 16:42:03 -05:00
John Ericson
056cc1c1b9 Merge remote-tracking branch 'upstream/master' into path-info 2023-01-14 14:27:28 -05:00
John Ericson
a416476217 Move ValidPathInfo defintions to path-info.cc
Originally there was no `path-info.*`, then there was `path-info.hh`,
then there was `path-info.cc`, but only for new things. Moving this
stuff over makes everything consistent.
2023-01-13 15:39:19 -05:00
John Ericson
2e7be46e73 Move new ValidPathInfo methods to path-info.cc
We'll move the old ones separately, so as not to clutter the diff.
2023-01-13 15:06:07 -05:00
John Ericson
f4f3203aa7 Merge remote-tracking branch 'upstream/master' into path-info 2021-09-30 22:41:53 +00:00
Eelco Dolstra
fe1f34fa60 Low-latency closure copy
This adds a new store operation 'addMultipleToStore' that reads a
number of NARs and ValidPathInfos from a Source, allowing any number
of store paths to be copied in a single call. This is much faster on
high-latency links when copying a lot of small files, like .drv
closures.

For example, on a connection with an 50 ms delay:

Before:

  $ nix copy --to 'unix:///tmp/proxy-socket?root=/tmp/dest-chroot' \
    /nix/store/90jjw94xiyg5drj70whm9yll6xjj0ca9-hello-2.10.drv \
    --derivation --no-check-sigs
  real    0m57.868s
  user    0m0.103s
  sys     0m0.056s

After:

  real    0m0.690s
  user    0m0.017s
  sys     0m0.011s
2021-07-26 13:31:09 +02:00