Commit graph

3977 commits

Author SHA1 Message Date
Eelco Dolstra
622191c2b5
Merge pull request #8965 from Artturin/bindfilesinchroot
Bindmount files instead of hardlinking or copying to chroot
2023-10-25 19:10:03 +02:00
Jacek Galowicz
f555c98a34 Improve loop over gid container 2023-10-25 16:10:35 +02:00
Jacek Galowicz
b113d925de Fix warning 2023-10-24 19:57:38 +02:00
Jacek Galowicz
eaced12c94 Fix signed vs. unsigned comparison warning and improve code 2023-10-24 19:57:38 +02:00
Robert Hensing
cde3c63617
system-features: Typo
There I was, thinking all of Apple's OSes started with lower case.
2023-10-23 19:30:00 +02:00
Valentin Gagarin
cd680bd53d
Merge how-to section on S3 buckets into S3 store docs (#7972)
Rather than having a misc tutorial page in the grab-bag "package management" section, this information should just be part of the S3 store docs.

---------

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2023-10-23 13:22:33 -04:00
John Ericson
8b68bbb777
Merge pull request #6223 from obsidiansystems/worker-proto-with-version
Give `nix daemon` and `nix-store --serve` protocols separate serializers with version info
2023-10-23 09:16:23 -04:00
Eelco Dolstra
955bbe53c5
Merge pull request #9177 from edolstra/input-accessors
Backport FSInputAccessor and MemoryInputAccessor from lazy-trees
2023-10-23 11:42:04 +02:00
John Ericson
70f8b96c11 Factor out UnkeyedValidPathInfo and test
This makes the path info serialisers ideomatic again, which allows me to
test them.
2023-10-20 15:21:04 -04:00
John Ericson
596bd469cc Move ValidPathInfo serialization code to worker-protocol.{cc.hh}
It does not belong with the data type itself.

This also materializes the fact that `copyPath` does not do any version
negotiation just just hard-codes "16".

The non-standard interface of these serializers makes it harder to test,
but this is fixed in the next commit which then adds those tests.
2023-10-20 15:20:48 -04:00
John Ericson
ab822af0df Factor out serialization for BuildResult
Worker Protocol:

Note that the worker protocol already had a serialization for
`BuildResult`; this was added in
a4604f1928. It didn't have any versioning
support because at that time reusable seralizers were not away for the protocol
version. It could thus only be used for new messages also introduced in
that commit.

Now that we do support versioning in reusable serializers, we can expand
it to support all known versions and use it in many more places.

The exist test data becomes the version 1.29 tests: note that those
files' contents are unchanged. 1.28 and 1.27 tests are added to cover
the older code-paths.

The keyered build result test only has 1.29 because the keying was also
added in a4604f19284254ac98f19a13ff7c2216de7fe176; the older
serializations are always used unkeyed.

Serve Protocol:

Conversely, no attempt was made to factor out such a serializer for the
serve protocol, so our work there in this commit for that protocol
proceeds from scratch.
2023-10-20 15:19:28 -04:00
John Ericson
4372738efe Systematize the worker protocol derived path serialiser
It was some ad-hoc functions to account for versions, while the already
factored-out serializer just supported the latest version.

Now, we can fold that version-specific logic into the factored out one,
and so we do.
2023-10-20 15:19:28 -04:00
John Ericson
96c58550b8 Test more derived paths 2023-10-20 15:19:28 -04:00
John Ericson
862d16436b Remove the ValidPathInfo == operator
It is dead code. It was added in
8e0946e8df as part of the repeated /
enforce-determinism feature, but that was removed in
8fdd156a65.

It is not good because it skips many fields. For testing purposes we
will soon want to add a new one that doesn't skip fields, but we want to
make sure making == sensitive to those fields won't change how Nix
works. Proving in this commit that the old version is dead code achieves
that.
2023-10-20 12:39:17 -04:00
John Ericson
3e6b9f9357 Remove prevInfos as its dead code
It is unused since 8e0946e8df removed
support for the repeat and enforce-determinism options.
2023-10-20 12:39:17 -04:00
Robert Hensing
4d17c59d8d
Merge pull request #9157 from obsidiansystems/protocol-versions
Add protocol versions to `{Worker,Serve}Proto::*Conn`
2023-10-20 15:34:26 +02:00
Robert Hensing
bb645c5d02 system-features doc: kvm is Linux-only 2023-10-20 10:21:39 +02:00
Robert Hensing
9277eb276b libstore: Add apple-virt to system features when available
I'm sure that we'll adjust the implementation over time, but this
at least discerns between an apple silicon bare metal machine and
a tart VM.
2023-10-20 10:21:39 +02:00
Yueh-Shun Li
e026f3e1ae treewide: Reference HashFormat members with scope
Base* -> HashFormat::Base*
2023-10-19 00:56:41 +08:00
Yueh-Shun Li
838c70f621 treewide: Rename hashBase to hashFormat
hashBase is ambiguous, since it's not about the digital bases, but about
the format of hashes. Base16, Base32 and Base64 are all character maps
for binary encoding.

Rename the enum Base to HashFormat.

Rename variables of type HashFormat from [hash]Base to hashFormat,
including CmdHashBase::hashFormat and CmdToBase::hashFormat.
2023-10-19 00:38:32 +08:00
Eelco Dolstra
ea38605d11 Introduce FSInputAccessor and use it
Backported from the lazy-trees branch. Note that this doesn't yet use
the access control features of FSInputAccessor.
2023-10-18 17:37:32 +02:00
John Ericson
e36c9175f4 Add protocol versions to {Worker,Serve}Proto::*Conn
This will allow us to factor out logic, which is currently scattered
inline, into several reusable instances

The tests are also updated to support versioning. Currently all Worker
and Serve protocol tests are using the minimum version, since no
version-specific serialisers have been created yet. But in subsequent
commits when that changes, we will test individual versions to ensure
complete coverage.
2023-10-17 11:21:10 -04:00
John Ericson
ff68426095 Name the protocol version types
This makes the code clearer, and will help us replace them with proper
structs and get rid of the macros later.
2023-10-17 11:20:39 -04:00
Artturin
b8dfa3d53b use doBind in addDependency 2023-10-17 01:26:34 +03:00
Artturin
11e47e7dfb factor out doBind from runChild 2023-10-17 01:26:34 +03:00
Artturin
630c2545d1 remove linkOrCopy and use bindmounts for files in addDependency 2023-10-17 01:26:34 +03:00
Artturin
5649229394 Bindmount files instead of hardlinking or copying to chroot
16591eb3cc (diff-19f999107b609d37cfb22c58e7f0bc1cf76edf1180e238dd6389e03cc279b604) (2013) added support for files to doBind

This is work towards allowing users to change the location of chrootRootDir, to, for example, a tmpfs.

inspired by trofi on matrix

> It looks like build sandbox created by nix-daemon runs on the same filesystem, as /nix/store including things like /tmp which makes all small temporary files hit the disk. Is it intentional? If it is is there an easy way to redirect chroot's root to be tmpfs?

dirsInChroot -> pathsInChroot
2023-10-17 01:26:34 +03:00
Jacek Galowicz
54b350d517 Drop some moves that would happen anyway but forbid NRVO where appicable 2023-10-16 21:48:35 +01:00
Jacek Galowicz
abf7df2b37 Fix moves that accidentally copy anyway 2023-10-16 21:48:35 +01:00
John Ericson
d070d8b746
Merge pull request #9137 from obsidiansystems/serve-protocol
Introduce separate Serve protocol serialisers
2023-10-13 10:51:46 -04:00
Robert Hensing
da2b59a088
Merge pull request #8047 from lovesegfault/always-allow-substitutes
feat: add always-allow-substitutes
2023-10-13 15:42:11 +02:00
Ninlives
94e91566ed
Allow CLI to pass environment variables to FOD builder (#8830)
Add a new experimental `impure-env` setting that is a key-value list of
environment variables to inject into FOD derivations that specify the
corresponding `impureEnvVars`.

This allows clients to make use of this feature (without having to change the
environment of the daemon itself) and might eventually deprecate the current
behaviour (pick whatever is in the environment of the daemon) as it's more
principled and might prevent information leakage.
2023-10-11 11:58:42 +00:00
John Ericson
f7b8f8aff6 Introduce separate Serve protocol serialisers
To start, it is just a clone of the common protocol. But now that we
have the separate protocol implementations, we can add versioning
information without the versions of one protocol leaking into another.

Using the infrastructure from the previous commit, we don't have to
duplicate code for shared behavior.

Motivation: No more perverse incentives. [0] did some awkward things
because the serialisers did not store the version. I don't want anyone
making changes to be pushed towards keeping the serialization logic with
the core data types just because it's easier or the alternative is
tedious.

The actual versioning of the Worker and Serve protocol serialisers
(Common remains unversioned as the underlying mini-protocols are not
versioned) will happen in subsequent commits / PRs.

[0]: fe1f34fa60
2023-10-10 11:52:45 -04:00
John Ericson
4de54b2190 Unit test the "common protocol" too
Copy the relevant tests to ensure the new interfaces added in the last
commit are tested.

Perhaps I should try to deduplicat these tests some more. However its
not clear how to do that outside of a big ugly C++ macro.
https://github.com/google/googletest/blob/main/docs/advanced.md has some
stuff but it is cumbersome and I didn't figure it out yet.

This is done in a separate commit in order to be sure that the first
commit really didn't change any behavior; if we changed the
implementation and the tests at once, it would be harder to tell whether
or not some behavioral changes slipped in what is supposed to be a "pure
refactor".

Co-Authored-By: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-10-09 16:57:03 -04:00
John Ericson
be81764320 Factor out bits of the worker protocol to use elsewhere
This introduces some shared infrastructure for our notion of protocols.
We can then define multiple protocols in terms of that notion.
We an also express how particular protocols depend on each other.

For example, we can define a common protocol and a worker protocol,
where the second depends on the first in terms of the data types it can
read and write.

The "serve" protocol can just use the common one for now, but will
eventually need its own machinary just like the worker protocol for
version-aware serialisers
2023-10-09 16:55:12 -04:00
John Ericson
68c81c7375 Put functional tests in tests/functional
I think it is bad for these reasons when `tests/` contains a mix of
functional and integration tests

 - Concepts is harder to understand, the documentation makes a good
   unit vs functional vs integration distinction, but when the
   integration tests are just two subdirs within `tests/` this is not
   clear.

 - Source filtering in the `flake.nix` is more complex. We need to
   filter out some of the dirs from `tests/`, rather than simply pick
   the dirs we want and take all of them. This is a good sign the
   structure of what we are trying to do is not matching the structure
   of the files.

With this change we have a clean:
```shell-session
$ git show 'HEAD:tests'
tree HEAD:tests

functional/
installer/
nixos/
```
2023-10-06 09:05:56 -04:00
Eelco Dolstra
62434951d9
Merge pull request #9095 from edef1c/reject-dot-paths
StorePath: reject names starting with '.'
2023-10-06 14:12:53 +02:00
edef
24bda0c7b3 StorePath: reject names starting with '.'
This has been the behaviour before Nix 2.4. It was dropped in a rewrite
in 759947bf72, allowing the creation of
store paths that aren't considered valid by older Nix versions or other
Nix tooling.

Nix 2.4 didn't ship in NixOS until 22.05, and stdenv.mkDerivation in
nixpkgs drops leading periods since April 2022, so it's unlikely anyone
is relying on the current lax behaviour.

Closes #9091.

Change-Id: I4a57bd9899e1b0dba56870ae5a1b680918a18ce9
2023-10-04 22:10:52 +00:00
John Ericson
e1af175707 Enable most of the third BuildResult worker protocol test
This was somewhat of a false alarm. The problem was not that the
protocol implementation actually failed to round trip, but that two of
the fields were ignored entirely --- not serialized and deserialized at
all.

For reference, those fields were added in
fa68eb367e.
2023-10-04 18:03:50 -04:00
John Ericson
632f24166d Test the rest of the worker protocol serializers
Part of the `BuildResult` test is commented out because we have caught a
roundtrip bug! A future PR will fix the bug and uncomment that test.
2023-10-04 15:31:52 -04:00
John Ericson
8440afbed7 Revert "Adapt scheduler to work with dynamic derivations"
This reverts commit 5e3986f59c. This
un-implements RFC 92 but fixes the critical bug #9052 which many people
are hitting. This is a decent stop-gap until a minimal reproduction of
that bug is found and a proper fix can be made.

Mostly fixed #9052, but I would like to leave that issue open until we
have a regression test, so I can then properly fix the bug (unbreaking
RFC 92) later.
2023-10-01 23:43:12 -04:00
Théophane Hufschmitt
c6faef61a6
Merge pull request #8923 from obsidiansystems/test-proto
Unit test some worker protocol serializers
2023-09-26 17:12:24 +02:00
Théophane Hufschmitt
1da1642527
Merge pull request #9041 from trofi/profiles-sign
src/libstore/profiles.cc: fix comparison of sign difference
2023-09-26 07:50:17 +02:00
Valentin Gagarin
60a155d01c
Merge pull request #8706 from fricklerhandwerk/doc-system-features
document system features
2023-09-26 04:21:31 +02:00
Sergei Trofimovich
ad213103d8 src/libstore/profiles.cc: fix comparison of sign difference
Detected by `gcc` as:

      CXX    src/libstore/profiles.o
    src/libstore/profiles.cc: In function 'void nix::deleteGenerationsGreaterThan(const Path&, GenerationNumber, bool)':
    src/libstore/profiles.cc:186:50: warning: comparison of integer expressions of different signedness: 'int' and 'nix::GenerationNumber' {aka 'long unsigned int'} [-Wsign-compare]
      186 |     for (auto keep = 0; i != gens.rend() && keep < max; ++i, ++keep);
          |                                             ~~~~~^~~~~
2023-09-25 17:45:57 +01:00
Eelco Dolstra
126e2645f2 Disable rapidcheck tests in the coverage run
https://hydra.nixos.org/build/233688539
2023-09-19 16:04:00 +02:00
John Ericson
80d7994f52 Special-case error message to add extra information
The Derivation parser and old ATerm unfortunately leaves few ways to get
nice errors when an old version of Nix encounters a new version of the
format. The most likely scenario for this to occur is with a new client
making a derivation that the old daemon it is communicating with cannot
understand.

The extensions we just created for dynamic derivation deps will add a
version field, solving the problem going forward, but there is still the
issue of what to do about old versions of Nix up to now.

The solution here is to carefully catch the bad error from the daemon
that is likely to indicate this problem, and add some extra context to
it.

There is another "Ugly backwards compatibility hack" in
`remote-store.cc` that also works by transforming an error.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-07 10:39:37 -04:00
John Ericson
7ad66cb3ef Allow dynamic derivation deps in inputDrvs
We use the same nested map representation we used for goals, again in
order to save space. We might someday want to combine with `inputDrvs`,
by doing `V = bool` instead of `V = std::set<OutputName>`, but we are
not doing that yet for sake of a smaller diff.

The ATerm format for Derivations also needs to be extended, in addition
to the in-memory format. To accomodate this, we added a new basic
versioning scheme, so old versions of Nix will get nice errors. (And
going forward, if the ATerm format changes again the errors will be even
better.)

`parsedStrings`, an internal function used as part of parsing
derivations in A-Term format, used to consume the final `]` but expect
the initial `[` to already be consumed. This made for what looked like
unbalanced brackets at callsites, which was confusing. Now it consumes
both which is hopefully less confusing.

As part of testing, we also created a unit test for the A-Term format for
regular non-experimental derivations too.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

Apply suggestions from code review

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-07 10:39:37 -04:00
John Ericson
b7edc2099f Improve derivation parsing
- Don't assert: Derivation ATerms are not necessarily produced by Nix,
  and parsers should always throw graceful errors

- Improve error message from `static void except(..)`, shows both what
  we expected and what we actually got.

The intention is that we backport it, and then hopefully a few people
might get slightly better errors if they try out new experimental drv
files (for RFC 92) with an old version of Nix.
2023-09-06 11:44:06 -04:00
John Ericson
754528241a
Merge pull request #8927 from obsidiansystems/test-derivation-aterm
Test and begin documentation of the ATerm format for derivations
2023-09-06 09:24:52 -04:00