max/nix-super
1
0
Fork 0
mirror of https://github.com/privatevoid-net/nix-super.git synced 2024-11-28 08:36:15 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
nix-super/src/libstore/worker-protocol-connection.cc

326 lines
11 KiB
C++
Raw Normal View History

Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
#include "worker-protocol-connection.hh"
#include "worker-protocol-impl.hh"
#include "build-result.hh"
#include "derivations.hh"
namespace nix {
WorkerProto: Support fine-grained protocol feature negotiation Currently, the worker protocol has a version number that we increment whenever we change something in the protocol. However, this can cause a collision between Nix PRs / forks that make protocol changes (e.g. PR #9857 increments the version, which could collide with another PR). So instead, the client and daemon now exchange a set of protocol features (such as `auth-forwarding`). They will use the intersection of the sets of features, i.e. the features they both support. Note that protocol features are completely distinct from `ExperimentalFeature`s.
2024-07-19 16:48:19 +03:00
const std::set<WorkerProto::Feature> WorkerProto::allFeatures{};
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
WorkerProto::BasicClientConnection::~BasicClientConnection()
{
try {
to.flush();
} catch (...) {
Split ignoreException for destructors or interrupt-safe
2024-09-30 12:49:53 +03:00
ignoreExceptionInDestructor();
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
}
}
static Logger::Fields readFields(Source & from)
{
Logger::Fields fields;
size_t size = readInt(from);
for (size_t n = 0; n < size; n++) {
auto type = (decltype(Logger::Field::type)) readInt(from);
if (type == Logger::Field::tInt)
fields.push_back(readNum<uint64_t>(from));
else if (type == Logger::Field::tString)
fields.push_back(readString(from));
else
throw Error("got unsupported field type %x from Nix daemon", (int) type);
}
return fields;
}
withFramedSink(): Don't use a thread to monitor the other side Since withFramedSink() is now used a lot more than in the past (for every addToStore() variant), we were creating a lot of threads, e.g. nix flake show --no-eval-cache --all-systems github:NixOS/nix/afdd12be5e19c0001ff3297dea544301108d298 would create 46418 threads. While threads on Linux are cheap, this is still substantial overhead. So instead, just poll from FramedSink before every write whether there are pending messages from the daemon. This could slightly increase the latency on log messages from the daemon, but not on exceptions (which were only synchronously checked from FramedSink anyway). This speeds up the command above from 19.2s to 17.5s on my machine (a 9% speedup).
2024-08-19 18:49:26 +03:00
std::exception_ptr
WorkerProto::BasicClientConnection::processStderrReturn(Sink * sink, Source * source, bool flush, bool block)
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
{
if (flush)
to.flush();
std::exception_ptr ex;
while (true) {
withFramedSink(): Don't use a thread to monitor the other side Since withFramedSink() is now used a lot more than in the past (for every addToStore() variant), we were creating a lot of threads, e.g. nix flake show --no-eval-cache --all-systems github:NixOS/nix/afdd12be5e19c0001ff3297dea544301108d298 would create 46418 threads. While threads on Linux are cheap, this is still substantial overhead. So instead, just poll from FramedSink before every write whether there are pending messages from the daemon. This could slightly increase the latency on log messages from the daemon, but not on exceptions (which were only synchronously checked from FramedSink anyway). This speeds up the command above from 19.2s to 17.5s on my machine (a 9% speedup).
2024-08-19 18:49:26 +03:00
if (!block && !from.hasData())
break;
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
auto msg = readNum<uint64_t>(from);
if (msg == STDERR_WRITE) {
auto s = readString(from);
if (!sink)
throw Error("no sink");
(*sink)(s);
}
else if (msg == STDERR_READ) {
if (!source)
throw Error("no source");
size_t len = readNum<size_t>(from);
auto buf = std::make_unique<char[]>(len);
writeString({(const char *) buf.get(), source->read(buf.get(), len)}, to);
to.flush();
}
else if (msg == STDERR_ERROR) {
Factor out commonality between WorkerProto::Basic{Client,Server}Connection This also renames clientVersion and daemonVersion to the more correct protoVersion (since it's the version agreed to by both sides).
2024-07-17 16:27:15 +03:00
if (GET_PROTOCOL_MINOR(protoVersion) >= 26) {
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
ex = std::make_exception_ptr(readError(from));
} else {
auto error = readString(from);
unsigned int status = readInt(from);
ex = std::make_exception_ptr(Error(status, error));
}
break;
}
else if (msg == STDERR_NEXT)
printError(chomp(readString(from)));
else if (msg == STDERR_START_ACTIVITY) {
auto act = readNum<ActivityId>(from);
auto lvl = (Verbosity) readInt(from);
auto type = (ActivityType) readInt(from);
auto s = readString(from);
auto fields = readFields(from);
auto parent = readNum<ActivityId>(from);
logger->startActivity(act, lvl, type, s, fields, parent);
}
else if (msg == STDERR_STOP_ACTIVITY) {
auto act = readNum<ActivityId>(from);
logger->stopActivity(act);
}
else if (msg == STDERR_RESULT) {
auto act = readNum<ActivityId>(from);
auto type = (ResultType) readInt(from);
auto fields = readFields(from);
logger->result(act, type, fields);
}
withFramedSink(): Don't use a thread to monitor the other side Since withFramedSink() is now used a lot more than in the past (for every addToStore() variant), we were creating a lot of threads, e.g. nix flake show --no-eval-cache --all-systems github:NixOS/nix/afdd12be5e19c0001ff3297dea544301108d298 would create 46418 threads. While threads on Linux are cheap, this is still substantial overhead. So instead, just poll from FramedSink before every write whether there are pending messages from the daemon. This could slightly increase the latency on log messages from the daemon, but not on exceptions (which were only synchronously checked from FramedSink anyway). This speeds up the command above from 19.2s to 17.5s on my machine (a 9% speedup).
2024-08-19 18:49:26 +03:00
else if (msg == STDERR_LAST) {
assert(block);
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
break;
withFramedSink(): Don't use a thread to monitor the other side Since withFramedSink() is now used a lot more than in the past (for every addToStore() variant), we were creating a lot of threads, e.g. nix flake show --no-eval-cache --all-systems github:NixOS/nix/afdd12be5e19c0001ff3297dea544301108d298 would create 46418 threads. While threads on Linux are cheap, this is still substantial overhead. So instead, just poll from FramedSink before every write whether there are pending messages from the daemon. This could slightly increase the latency on log messages from the daemon, but not on exceptions (which were only synchronously checked from FramedSink anyway). This speeds up the command above from 19.2s to 17.5s on my machine (a 9% speedup).
2024-08-19 18:49:26 +03:00
}
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
else
throw Error("got unknown message type %x from Nix daemon", msg);
}
if (!ex) {
return ex;
} else {
try {
std::rethrow_exception(ex);
} catch (const Error & e) {
// Nix versions before #4628 did not have an adequate
// behavior for reporting that the derivation format was
// upgraded. To avoid having to add compatibility logic in
// many places, we expect to catch almost all occurrences of
// the old incomprehensible error here, so that we can
// explain to users what's going on when their daemon is
// older than #4628 (2023).
if (experimentalFeatureSettings.isEnabled(Xp::DynamicDerivations)
Factor out commonality between WorkerProto::Basic{Client,Server}Connection This also renames clientVersion and daemonVersion to the more correct protoVersion (since it's the version agreed to by both sides).
2024-07-17 16:27:15 +03:00
&& GET_PROTOCOL_MINOR(protoVersion) <= 35) {
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
auto m = e.msg();
if (m.find("parsing derivation") != std::string::npos && m.find("expected string") != std::string::npos
&& m.find("Derive([") != std::string::npos)
return std::make_exception_ptr(Error(
"%s, this might be because the daemon is too old to understand dependencies on dynamic derivations. Check to see if the raw derivation is in the form '%s'",
std::move(m),
"Drv WithVersion(..)"));
}
return std::current_exception();
}
}
}
withFramedSink(): Don't use a thread to monitor the other side Since withFramedSink() is now used a lot more than in the past (for every addToStore() variant), we were creating a lot of threads, e.g. nix flake show --no-eval-cache --all-systems github:NixOS/nix/afdd12be5e19c0001ff3297dea544301108d298 would create 46418 threads. While threads on Linux are cheap, this is still substantial overhead. So instead, just poll from FramedSink before every write whether there are pending messages from the daemon. This could slightly increase the latency on log messages from the daemon, but not on exceptions (which were only synchronously checked from FramedSink anyway). This speeds up the command above from 19.2s to 17.5s on my machine (a 9% speedup).
2024-08-19 18:49:26 +03:00
void WorkerProto::BasicClientConnection::processStderr(
bool * daemonException, Sink * sink, Source * source, bool flush, bool block)
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
{
withFramedSink(): Don't use a thread to monitor the other side Since withFramedSink() is now used a lot more than in the past (for every addToStore() variant), we were creating a lot of threads, e.g. nix flake show --no-eval-cache --all-systems github:NixOS/nix/afdd12be5e19c0001ff3297dea544301108d298 would create 46418 threads. While threads on Linux are cheap, this is still substantial overhead. So instead, just poll from FramedSink before every write whether there are pending messages from the daemon. This could slightly increase the latency on log messages from the daemon, but not on exceptions (which were only synchronously checked from FramedSink anyway). This speeds up the command above from 19.2s to 17.5s on my machine (a 9% speedup).
2024-08-19 18:49:26 +03:00
auto ex = processStderrReturn(sink, source, flush, block);
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
if (ex) {
*daemonException = true;
std::rethrow_exception(ex);
}
}
WorkerProto: Support fine-grained protocol feature negotiation Currently, the worker protocol has a version number that we increment whenever we change something in the protocol. However, this can cause a collision between Nix PRs / forks that make protocol changes (e.g. PR #9857 increments the version, which could collide with another PR). So instead, the client and daemon now exchange a set of protocol features (such as `auth-forwarding`). They will use the intersection of the sets of features, i.e. the features they both support. Note that protocol features are completely distinct from `ExperimentalFeature`s.
2024-07-19 16:48:19 +03:00
static std::set<WorkerProto::Feature>
intersectFeatures(const std::set<WorkerProto::Feature> & a, const std::set<WorkerProto::Feature> & b)
{
std::set<WorkerProto::Feature> res;
for (auto & x : a)
if (b.contains(x))
res.insert(x);
return res;
}
std::tuple<WorkerProto::Version, std::set<WorkerProto::Feature>> WorkerProto::BasicClientConnection::handshake(
BufferedSink & to,
Source & from,
WorkerProto::Version localVersion,
const std::set<WorkerProto::Feature> & supportedFeatures)
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
{
to << WORKER_MAGIC_1 << localVersion;
to.flush();
unsigned int magic = readInt(from);
if (magic != WORKER_MAGIC_2)
throw Error("nix-daemon protocol mismatch from");
auto daemonVersion = readInt(from);
if (GET_PROTOCOL_MAJOR(daemonVersion) != GET_PROTOCOL_MAJOR(PROTOCOL_VERSION))
throw Error("Nix daemon protocol version not supported");
if (GET_PROTOCOL_MINOR(daemonVersion) < 10)
throw Error("the Nix daemon version is too old");
WorkerProto: Support fine-grained protocol feature negotiation Currently, the worker protocol has a version number that we increment whenever we change something in the protocol. However, this can cause a collision between Nix PRs / forks that make protocol changes (e.g. PR #9857 increments the version, which could collide with another PR). So instead, the client and daemon now exchange a set of protocol features (such as `auth-forwarding`). They will use the intersection of the sets of features, i.e. the features they both support. Note that protocol features are completely distinct from `ExperimentalFeature`s.
2024-07-19 16:48:19 +03:00
auto protoVersion = std::min(daemonVersion, localVersion);
/* Exchange features. */
std::set<WorkerProto::Feature> daemonFeatures;
if (GET_PROTOCOL_MINOR(protoVersion) >= 38) {
to << supportedFeatures;
to.flush();
daemonFeatures = readStrings<std::set<WorkerProto::Feature>>(from);
}
return {protoVersion, intersectFeatures(daemonFeatures, supportedFeatures)};
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
}
WorkerProto: Support fine-grained protocol feature negotiation Currently, the worker protocol has a version number that we increment whenever we change something in the protocol. However, this can cause a collision between Nix PRs / forks that make protocol changes (e.g. PR #9857 increments the version, which could collide with another PR). So instead, the client and daemon now exchange a set of protocol features (such as `auth-forwarding`). They will use the intersection of the sets of features, i.e. the features they both support. Note that protocol features are completely distinct from `ExperimentalFeature`s.
2024-07-19 16:48:19 +03:00
std::tuple<WorkerProto::Version, std::set<WorkerProto::Feature>> WorkerProto::BasicServerConnection::handshake(
BufferedSink & to,
Source & from,
WorkerProto::Version localVersion,
const std::set<WorkerProto::Feature> & supportedFeatures)
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
{
unsigned int magic = readInt(from);
if (magic != WORKER_MAGIC_1)
throw Error("protocol mismatch");
to << WORKER_MAGIC_2 << localVersion;
to.flush();
auto clientVersion = readInt(from);
WorkerProto: Support fine-grained protocol feature negotiation Currently, the worker protocol has a version number that we increment whenever we change something in the protocol. However, this can cause a collision between Nix PRs / forks that make protocol changes (e.g. PR #9857 increments the version, which could collide with another PR). So instead, the client and daemon now exchange a set of protocol features (such as `auth-forwarding`). They will use the intersection of the sets of features, i.e. the features they both support. Note that protocol features are completely distinct from `ExperimentalFeature`s.
2024-07-19 16:48:19 +03:00
auto protoVersion = std::min(clientVersion, localVersion);
/* Exchange features. */
std::set<WorkerProto::Feature> clientFeatures;
if (GET_PROTOCOL_MINOR(protoVersion) >= 38) {
clientFeatures = readStrings<std::set<WorkerProto::Feature>>(from);
to << supportedFeatures;
to.flush();
}
return {protoVersion, intersectFeatures(clientFeatures, supportedFeatures)};
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
}
WorkerProto::ClientHandshakeInfo WorkerProto::BasicClientConnection::postHandshake(const StoreDirConfig & store)
{
WorkerProto::ClientHandshakeInfo res;
Factor out commonality between WorkerProto::Basic{Client,Server}Connection This also renames clientVersion and daemonVersion to the more correct protoVersion (since it's the version agreed to by both sides).
2024-07-17 16:27:15 +03:00
if (GET_PROTOCOL_MINOR(protoVersion) >= 14) {
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
// Obsolete CPU affinity.
to << 0;
}
Factor out commonality between WorkerProto::Basic{Client,Server}Connection This also renames clientVersion and daemonVersion to the more correct protoVersion (since it's the version agreed to by both sides).
2024-07-17 16:27:15 +03:00
if (GET_PROTOCOL_MINOR(protoVersion) >= 11)
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
to << false; // obsolete reserveSpace
Factor out commonality between WorkerProto::Basic{Client,Server}Connection This also renames clientVersion and daemonVersion to the more correct protoVersion (since it's the version agreed to by both sides).
2024-07-17 16:27:15 +03:00
if (GET_PROTOCOL_MINOR(protoVersion) >= 33)
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
to.flush();
return WorkerProto::Serialise<ClientHandshakeInfo>::read(store, *this);
}
void WorkerProto::BasicServerConnection::postHandshake(const StoreDirConfig & store, const ClientHandshakeInfo & info)
{
Factor out commonality between WorkerProto::Basic{Client,Server}Connection This also renames clientVersion and daemonVersion to the more correct protoVersion (since it's the version agreed to by both sides).
2024-07-17 16:27:15 +03:00
if (GET_PROTOCOL_MINOR(protoVersion) >= 14 && readInt(from)) {
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
// Obsolete CPU affinity.
readInt(from);
}
Factor out commonality between WorkerProto::Basic{Client,Server}Connection This also renames clientVersion and daemonVersion to the more correct protoVersion (since it's the version agreed to by both sides).
2024-07-17 16:27:15 +03:00
if (GET_PROTOCOL_MINOR(protoVersion) >= 11)
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
readInt(from); // obsolete reserveSpace
WorkerProto::write(store, *this, info);
}
UnkeyedValidPathInfo WorkerProto::BasicClientConnection::queryPathInfo(
const StoreDirConfig & store, bool * daemonException, const StorePath & path)
{
to << WorkerProto::Op::QueryPathInfo << store.printStorePath(path);
try {
processStderr(daemonException);
} catch (Error & e) {
// Ugly backwards compatibility hack.
if (e.msg().find("is not valid") != std::string::npos)
throw InvalidPath(std::move(e.info()));
throw;
}
Factor out commonality between WorkerProto::Basic{Client,Server}Connection This also renames clientVersion and daemonVersion to the more correct protoVersion (since it's the version agreed to by both sides).
2024-07-17 16:27:15 +03:00
if (GET_PROTOCOL_MINOR(protoVersion) >= 17) {
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
bool valid;
from >> valid;
if (!valid)
throw InvalidPath("path '%s' is not valid", store.printStorePath(path));
}
return WorkerProto::Serialise<UnkeyedValidPathInfo>::read(store, *this);
}
StorePathSet WorkerProto::BasicClientConnection::queryValidPaths(
const StoreDirConfig & store, bool * daemonException, const StorePathSet & paths, SubstituteFlag maybeSubstitute)
{
Factor out commonality between WorkerProto::Basic{Client,Server}Connection This also renames clientVersion and daemonVersion to the more correct protoVersion (since it's the version agreed to by both sides).
2024-07-17 16:27:15 +03:00
assert(GET_PROTOCOL_MINOR(protoVersion) >= 12);
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
to << WorkerProto::Op::QueryValidPaths;
WorkerProto::write(store, *this, paths);
Factor out commonality between WorkerProto::Basic{Client,Server}Connection This also renames clientVersion and daemonVersion to the more correct protoVersion (since it's the version agreed to by both sides).
2024-07-17 16:27:15 +03:00
if (GET_PROTOCOL_MINOR(protoVersion) >= 27) {
Factor our connection code for worker proto like serve proto This increases test coverage, and gets the worker protocol ready to be used by Hydra. Why don't we just try to use the store interface in Hydra? Well, the problem is that the store interface works on connection pools, with each opreation getting potentially a different connection, but the way temp roots work requires that we keep one logical "transaction" (temp root session) using the same connection. The longer-term solution probably is making connections themselves implement the store interface, but that is something that builds on this, so I feel OK that this is not churn in the wrong direction. Fixes #9584
2024-05-23 23:40:05 +03:00
to << maybeSubstitute;
}
processStderr(daemonException);
return WorkerProto::Serialise<StorePathSet>::read(store, *this);
}
void WorkerProto::BasicClientConnection::addTempRoot(
const StoreDirConfig & store, bool * daemonException, const StorePath & path)
{
to << WorkerProto::Op::AddTempRoot << store.printStorePath(path);
processStderr(daemonException);
readInt(from);
}
void WorkerProto::BasicClientConnection::putBuildDerivationRequest(
const StoreDirConfig & store,
bool * daemonException,
const StorePath & drvPath,
const BasicDerivation & drv,
BuildMode buildMode)
{
to << WorkerProto::Op::BuildDerivation << store.printStorePath(drvPath);
writeDerivation(to, store, drv);
to << buildMode;
}
BuildResult
WorkerProto::BasicClientConnection::getBuildDerivationResponse(const StoreDirConfig & store, bool * daemonException)
{
return WorkerProto::Serialise<BuildResult>::read(store, *this);
}
void WorkerProto::BasicClientConnection::narFromPath(
const StoreDirConfig & store, bool * daemonException, const StorePath & path, std::function<void(Source &)> fun)
{
to << WorkerProto::Op::NarFromPath << store.printStorePath(path);
processStderr(daemonException);
fun(from);
}
void WorkerProto::BasicClientConnection::importPaths(
const StoreDirConfig & store, bool * daemonException, Source & source)
{
to << WorkerProto::Op::ImportPaths;
processStderr(daemonException, 0, &source);
auto importedPaths = WorkerProto::Serialise<StorePathSet>::read(store, *this);
assert(importedPaths.size() <= importedPaths.size());
}
}
Reference in a new issue Copy permalink