When doing local builds, we get phase reporting lines in the log file,
they look like '@nix {"action":"setPhase","phase":"unpackPhase"}'.
With the ssh-ng protocol, we do have access to these messages, but since we
are only including messages of type resBuildLogLine in the logs, the phase
information does not end up in the log file.
The phase reporting could probably be improved altoghether (it looks like it
is kind of accidental that these JSON messages for phase reporting show up
but others don't, just because they are actually emitted by nixpkgs' stdenv),
but as a first step I propose to make ssh-ng behave in the same way as local builds do.
The brings a number of advantages, including:
- Easier to update test data if design changes (and I do think our
derivation JSON is not yet complaint with the guidelines).
- Easier to reuse test data in other implementations, inching closer to
compliance tests for Nix *the concept* rather than any one
implementation.
Before the change builder ID exhaustion printed the following message:
[0/1 built] waiting for UID to build '/nix/store/hiy9136x0iyib4ssh3w3r5m8pxjnad50-python3.11-breathe-4.35.0.drv'
After the change it should be:
[0/1 built] waiting for a free build user ID for '/nix/store/hiy9136x0iyib4ssh3w3r5m8pxjnad50-python3.11-breathe-4.35.0.drv'
Rather than having a misc tutorial page in the grab-bag "package management" section, this information should just be part of the S3 store docs.
---------
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
It does not belong with the data type itself.
This also materializes the fact that `copyPath` does not do any version
negotiation just just hard-codes "16".
The non-standard interface of these serializers makes it harder to test,
but this is fixed in the next commit which then adds those tests.
Worker Protocol:
Note that the worker protocol already had a serialization for
`BuildResult`; this was added in
a4604f1928. It didn't have any versioning
support because at that time reusable seralizers were not away for the protocol
version. It could thus only be used for new messages also introduced in
that commit.
Now that we do support versioning in reusable serializers, we can expand
it to support all known versions and use it in many more places.
The exist test data becomes the version 1.29 tests: note that those
files' contents are unchanged. 1.28 and 1.27 tests are added to cover
the older code-paths.
The keyered build result test only has 1.29 because the keying was also
added in a4604f19284254ac98f19a13ff7c2216de7fe176; the older
serializations are always used unkeyed.
Serve Protocol:
Conversely, no attempt was made to factor out such a serializer for the
serve protocol, so our work there in this commit for that protocol
proceeds from scratch.
It was some ad-hoc functions to account for versions, while the already
factored-out serializer just supported the latest version.
Now, we can fold that version-specific logic into the factored out one,
and so we do.
It is dead code. It was added in
8e0946e8df as part of the repeated /
enforce-determinism feature, but that was removed in
8fdd156a65.
It is not good because it skips many fields. For testing purposes we
will soon want to add a new one that doesn't skip fields, but we want to
make sure making == sensitive to those fields won't change how Nix
works. Proving in this commit that the old version is dead code achieves
that.
hashBase is ambiguous, since it's not about the digital bases, but about
the format of hashes. Base16, Base32 and Base64 are all character maps
for binary encoding.
Rename the enum Base to HashFormat.
Rename variables of type HashFormat from [hash]Base to hashFormat,
including CmdHashBase::hashFormat and CmdToBase::hashFormat.
This will allow us to factor out logic, which is currently scattered
inline, into several reusable instances
The tests are also updated to support versioning. Currently all Worker
and Serve protocol tests are using the minimum version, since no
version-specific serialisers have been created yet. But in subsequent
commits when that changes, we will test individual versions to ensure
complete coverage.
16591eb3cc (diff-19f999107b609d37cfb22c58e7f0bc1cf76edf1180e238dd6389e03cc279b604) (2013) added support for files to doBind
This is work towards allowing users to change the location of chrootRootDir, to, for example, a tmpfs.
inspired by trofi on matrix
> It looks like build sandbox created by nix-daemon runs on the same filesystem, as /nix/store including things like /tmp which makes all small temporary files hit the disk. Is it intentional? If it is is there an easy way to redirect chroot's root to be tmpfs?
dirsInChroot -> pathsInChroot
Add a new experimental `impure-env` setting that is a key-value list of
environment variables to inject into FOD derivations that specify the
corresponding `impureEnvVars`.
This allows clients to make use of this feature (without having to change the
environment of the daemon itself) and might eventually deprecate the current
behaviour (pick whatever is in the environment of the daemon) as it's more
principled and might prevent information leakage.
To start, it is just a clone of the common protocol. But now that we
have the separate protocol implementations, we can add versioning
information without the versions of one protocol leaking into another.
Using the infrastructure from the previous commit, we don't have to
duplicate code for shared behavior.
Motivation: No more perverse incentives. [0] did some awkward things
because the serialisers did not store the version. I don't want anyone
making changes to be pushed towards keeping the serialization logic with
the core data types just because it's easier or the alternative is
tedious.
The actual versioning of the Worker and Serve protocol serialisers
(Common remains unversioned as the underlying mini-protocols are not
versioned) will happen in subsequent commits / PRs.
[0]: fe1f34fa60
Copy the relevant tests to ensure the new interfaces added in the last
commit are tested.
Perhaps I should try to deduplicat these tests some more. However its
not clear how to do that outside of a big ugly C++ macro.
https://github.com/google/googletest/blob/main/docs/advanced.md has some
stuff but it is cumbersome and I didn't figure it out yet.
This is done in a separate commit in order to be sure that the first
commit really didn't change any behavior; if we changed the
implementation and the tests at once, it would be harder to tell whether
or not some behavioral changes slipped in what is supposed to be a "pure
refactor".
Co-Authored-By: Valentin Gagarin <valentin.gagarin@tweag.io>
This introduces some shared infrastructure for our notion of protocols.
We can then define multiple protocols in terms of that notion.
We an also express how particular protocols depend on each other.
For example, we can define a common protocol and a worker protocol,
where the second depends on the first in terms of the data types it can
read and write.
The "serve" protocol can just use the common one for now, but will
eventually need its own machinary just like the worker protocol for
version-aware serialisers
I think it is bad for these reasons when `tests/` contains a mix of
functional and integration tests
- Concepts is harder to understand, the documentation makes a good
unit vs functional vs integration distinction, but when the
integration tests are just two subdirs within `tests/` this is not
clear.
- Source filtering in the `flake.nix` is more complex. We need to
filter out some of the dirs from `tests/`, rather than simply pick
the dirs we want and take all of them. This is a good sign the
structure of what we are trying to do is not matching the structure
of the files.
With this change we have a clean:
```shell-session
$ git show 'HEAD:tests'
tree HEAD:tests
functional/
installer/
nixos/
```
This has been the behaviour before Nix 2.4. It was dropped in a rewrite
in 759947bf72, allowing the creation of
store paths that aren't considered valid by older Nix versions or other
Nix tooling.
Nix 2.4 didn't ship in NixOS until 22.05, and stdenv.mkDerivation in
nixpkgs drops leading periods since April 2022, so it's unlikely anyone
is relying on the current lax behaviour.
Closes#9091.
Change-Id: I4a57bd9899e1b0dba56870ae5a1b680918a18ce9
This was somewhat of a false alarm. The problem was not that the
protocol implementation actually failed to round trip, but that two of
the fields were ignored entirely --- not serialized and deserialized at
all.
For reference, those fields were added in
fa68eb367e.
This reverts commit 5e3986f59c. This
un-implements RFC 92 but fixes the critical bug #9052 which many people
are hitting. This is a decent stop-gap until a minimal reproduction of
that bug is found and a proper fix can be made.
Mostly fixed#9052, but I would like to leave that issue open until we
have a regression test, so I can then properly fix the bug (unbreaking
RFC 92) later.
