Commit graph

3998 commits

Author SHA1 Message Date
Eelco Dolstra
b0455e9931 Fix uninitialized variable warnings on i686-linux
https://hydra.nixos.org/build/239849607
2023-11-03 12:04:20 +01:00
Eelco Dolstra
55dd1244d2 parseDerivation(): Fix warning about uninitialized 'version' variable 2023-11-03 12:04:20 +01:00
John Ericson
d15c3a33e6 Don't use std::invocable C++ concept yet
It s not supported on all platforms yet. Can revert this once it is.
2023-11-02 12:06:29 -04:00
John Ericson
b107431816 Systematize characterization tests a bit more
Deduplicating code moreover enforcing the pattern means:

 - It is easier to write new characterization tests because less boilerplate

 - It is harder to mess up new tests because there are fewer places to
   make mistakes.

Co-authored-by: Jacek Galowicz <jacek@galowicz.de>
2023-11-02 12:06:08 -04:00
John Ericson
90de958637
Merge pull request #9265 from obsidiansystems/better-parse-sink
Make `ParseSink` a bit better
2023-11-02 09:28:29 -04:00
Eelco Dolstra
5223114c93
Merge pull request #9269 from edolstra/unify-accessor
Unify `FSAccessor` and `SourceAccessor`
2023-11-02 14:23:10 +01:00
Eelco Dolstra
e47984ce0b
Fix whitespace
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-11-01 20:19:08 +01:00
Zbigniew Jędrzejewski-Szmek
eab9292738 fix: gcc complains about if which doesn't guard the indented statement 2023-11-01 18:10:06 +01:00
Eelco Dolstra
2f5c1a27dc LocalStoreAccessor: Reuse PosixSourceAccessor 2023-11-01 17:22:25 +01:00
Eelco Dolstra
1a902f5fa7 Merge FSAccessor into SourceAccessor 2023-11-01 17:09:28 +01:00
Eelco Dolstra
50aae0a14c FSAccessor: Make the fileSize and narOffset fields optional
The narOffset field only applies to NAR accessors. The fileSize field
may be too expensive to compute for certain accessors (e.g. libgit).
2023-11-01 15:39:40 +01:00
Eelco Dolstra
5381123879 Unify DirEntries types 2023-11-01 15:33:35 +01:00
Eelco Dolstra
8ffd1695ce Unify FSAccessor::Type and SourceAccessor::Type 2023-11-01 14:43:20 +01:00
Eelco Dolstra
b2ac6fc040 Remove FSAccessor::Type::tMissing
Instead stat() now returns std::nullopt to denote that the file
doesn't exist.
2023-11-01 14:36:40 +01:00
John Ericson
1093d6585f Make ParseSink a bit better
I wouldn't call it *good* yet, but this will do for now.

- `RetrieveRegularNARSink` renamed to `RegularFileSink` and moved
  accordingly because it actually has nothing to do with NARs in
  particular.

  - its `fd` field is also marked private

- `copyRecursive` introduced to dump a `SourceAccessor` into a
  `ParseSink`.

- `NullParseSink` made so `ParseSink` no longer has sketchy default
  methods.

This was done while updating #8918 to work with the new
`SourceAccessor`.
2023-11-01 02:36:43 -04:00
John Ericson
b2cae33aef Remove bug-avoiding StoreConfig * casts for settings
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80431 has been fixed, and
per the previous commit we now check that is the case at build time.
2023-10-31 12:09:46 -04:00
John Ericson
a419b61497 Turn derivation unit tests into unit characterization tests
The brings a number of advantages, including:

- Easier to update test data if design changes (and I do think our
  derivation JSON is not yet complaint with the guidelines).

- Easier to reuse test data in other implementations, inching closer to
  compliance tests for Nix *the concept* rather than any one
  implementation.
2023-10-26 18:09:01 -04:00
Sergei Trofimovich
e69c764708 local-derivation-goal.cc: slightly clarify waiting message
Before the change builder ID exhaustion printed the following message:

    [0/1 built] waiting for UID to build '/nix/store/hiy9136x0iyib4ssh3w3r5m8pxjnad50-python3.11-breathe-4.35.0.drv'

After the change it should be:

    [0/1 built] waiting for a free build user ID for '/nix/store/hiy9136x0iyib4ssh3w3r5m8pxjnad50-python3.11-breathe-4.35.0.drv'
2023-10-26 20:54:21 +01:00
John Ericson
1dc6a65d36
Merge pull request #9238 from tfc/small-improvements2
Small improvements 2
2023-10-26 09:35:03 -04:00
Jacek Galowicz
28c39c370c Provide default value for id to fix warning 2023-10-26 09:47:40 +02:00
Jacek Galowicz
b66381e8d8 Use using instead of typedef 2023-10-26 09:47:40 +02:00
Eelco Dolstra
622191c2b5
Merge pull request #8965 from Artturin/bindfilesinchroot
Bindmount files instead of hardlinking or copying to chroot
2023-10-25 19:10:03 +02:00
Jacek Galowicz
f555c98a34 Improve loop over gid container 2023-10-25 16:10:35 +02:00
Jacek Galowicz
b113d925de Fix warning 2023-10-24 19:57:38 +02:00
Jacek Galowicz
eaced12c94 Fix signed vs. unsigned comparison warning and improve code 2023-10-24 19:57:38 +02:00
Robert Hensing
cde3c63617
system-features: Typo
There I was, thinking all of Apple's OSes started with lower case.
2023-10-23 19:30:00 +02:00
Valentin Gagarin
cd680bd53d
Merge how-to section on S3 buckets into S3 store docs (#7972)
Rather than having a misc tutorial page in the grab-bag "package management" section, this information should just be part of the S3 store docs.

---------

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2023-10-23 13:22:33 -04:00
John Ericson
8b68bbb777
Merge pull request #6223 from obsidiansystems/worker-proto-with-version
Give `nix daemon` and `nix-store --serve` protocols separate serializers with version info
2023-10-23 09:16:23 -04:00
Eelco Dolstra
955bbe53c5
Merge pull request #9177 from edolstra/input-accessors
Backport FSInputAccessor and MemoryInputAccessor from lazy-trees
2023-10-23 11:42:04 +02:00
John Ericson
70f8b96c11 Factor out UnkeyedValidPathInfo and test
This makes the path info serialisers ideomatic again, which allows me to
test them.
2023-10-20 15:21:04 -04:00
John Ericson
596bd469cc Move ValidPathInfo serialization code to worker-protocol.{cc.hh}
It does not belong with the data type itself.

This also materializes the fact that `copyPath` does not do any version
negotiation just just hard-codes "16".

The non-standard interface of these serializers makes it harder to test,
but this is fixed in the next commit which then adds those tests.
2023-10-20 15:20:48 -04:00
John Ericson
ab822af0df Factor out serialization for BuildResult
Worker Protocol:

Note that the worker protocol already had a serialization for
`BuildResult`; this was added in
a4604f1928. It didn't have any versioning
support because at that time reusable seralizers were not away for the protocol
version. It could thus only be used for new messages also introduced in
that commit.

Now that we do support versioning in reusable serializers, we can expand
it to support all known versions and use it in many more places.

The exist test data becomes the version 1.29 tests: note that those
files' contents are unchanged. 1.28 and 1.27 tests are added to cover
the older code-paths.

The keyered build result test only has 1.29 because the keying was also
added in a4604f19284254ac98f19a13ff7c2216de7fe176; the older
serializations are always used unkeyed.

Serve Protocol:

Conversely, no attempt was made to factor out such a serializer for the
serve protocol, so our work there in this commit for that protocol
proceeds from scratch.
2023-10-20 15:19:28 -04:00
John Ericson
4372738efe Systematize the worker protocol derived path serialiser
It was some ad-hoc functions to account for versions, while the already
factored-out serializer just supported the latest version.

Now, we can fold that version-specific logic into the factored out one,
and so we do.
2023-10-20 15:19:28 -04:00
John Ericson
96c58550b8 Test more derived paths 2023-10-20 15:19:28 -04:00
John Ericson
862d16436b Remove the ValidPathInfo == operator
It is dead code. It was added in
8e0946e8df as part of the repeated /
enforce-determinism feature, but that was removed in
8fdd156a65.

It is not good because it skips many fields. For testing purposes we
will soon want to add a new one that doesn't skip fields, but we want to
make sure making == sensitive to those fields won't change how Nix
works. Proving in this commit that the old version is dead code achieves
that.
2023-10-20 12:39:17 -04:00
John Ericson
3e6b9f9357 Remove prevInfos as its dead code
It is unused since 8e0946e8df removed
support for the repeat and enforce-determinism options.
2023-10-20 12:39:17 -04:00
Robert Hensing
4d17c59d8d
Merge pull request #9157 from obsidiansystems/protocol-versions
Add protocol versions to `{Worker,Serve}Proto::*Conn`
2023-10-20 15:34:26 +02:00
Robert Hensing
bb645c5d02 system-features doc: kvm is Linux-only 2023-10-20 10:21:39 +02:00
Robert Hensing
9277eb276b libstore: Add apple-virt to system features when available
I'm sure that we'll adjust the implementation over time, but this
at least discerns between an apple silicon bare metal machine and
a tart VM.
2023-10-20 10:21:39 +02:00
Yueh-Shun Li
e026f3e1ae treewide: Reference HashFormat members with scope
Base* -> HashFormat::Base*
2023-10-19 00:56:41 +08:00
Yueh-Shun Li
838c70f621 treewide: Rename hashBase to hashFormat
hashBase is ambiguous, since it's not about the digital bases, but about
the format of hashes. Base16, Base32 and Base64 are all character maps
for binary encoding.

Rename the enum Base to HashFormat.

Rename variables of type HashFormat from [hash]Base to hashFormat,
including CmdHashBase::hashFormat and CmdToBase::hashFormat.
2023-10-19 00:38:32 +08:00
Eelco Dolstra
ea38605d11 Introduce FSInputAccessor and use it
Backported from the lazy-trees branch. Note that this doesn't yet use
the access control features of FSInputAccessor.
2023-10-18 17:37:32 +02:00
John Ericson
e36c9175f4 Add protocol versions to {Worker,Serve}Proto::*Conn
This will allow us to factor out logic, which is currently scattered
inline, into several reusable instances

The tests are also updated to support versioning. Currently all Worker
and Serve protocol tests are using the minimum version, since no
version-specific serialisers have been created yet. But in subsequent
commits when that changes, we will test individual versions to ensure
complete coverage.
2023-10-17 11:21:10 -04:00
John Ericson
ff68426095 Name the protocol version types
This makes the code clearer, and will help us replace them with proper
structs and get rid of the macros later.
2023-10-17 11:20:39 -04:00
Artturin
b8dfa3d53b use doBind in addDependency 2023-10-17 01:26:34 +03:00
Artturin
11e47e7dfb factor out doBind from runChild 2023-10-17 01:26:34 +03:00
Artturin
630c2545d1 remove linkOrCopy and use bindmounts for files in addDependency 2023-10-17 01:26:34 +03:00
Artturin
5649229394 Bindmount files instead of hardlinking or copying to chroot
16591eb3cc (diff-19f999107b609d37cfb22c58e7f0bc1cf76edf1180e238dd6389e03cc279b604) (2013) added support for files to doBind

This is work towards allowing users to change the location of chrootRootDir, to, for example, a tmpfs.

inspired by trofi on matrix

> It looks like build sandbox created by nix-daemon runs on the same filesystem, as /nix/store including things like /tmp which makes all small temporary files hit the disk. Is it intentional? If it is is there an easy way to redirect chroot's root to be tmpfs?

dirsInChroot -> pathsInChroot
2023-10-17 01:26:34 +03:00
Jacek Galowicz
54b350d517 Drop some moves that would happen anyway but forbid NRVO where appicable 2023-10-16 21:48:35 +01:00
Jacek Galowicz
abf7df2b37 Fix moves that accidentally copy anyway 2023-10-16 21:48:35 +01:00