Commit graph

4011 commits

Author SHA1 Message Date
Jacek Galowicz
df8bfe84cc Fix consts and casts 2023-11-08 17:29:55 +01:00
Jacek Galowicz
c581143e0c Use structured binding for json iteration 2023-11-08 17:29:55 +01:00
Jacek Galowicz
77dceb2844 Drop obsolete assert and cast 2023-11-08 17:29:55 +01:00
Jacek Galowicz
6a47629530 Fix initialization of struct members (wrong order) 2023-11-08 17:29:55 +01:00
John Ericson
dcc49744ed
Merge pull request #9293 from ThinkChaos/ssh-misc-improvments
SSH small improvments
2023-11-06 13:33:51 -05:00
John Ericson
cc46ea1630 Make nix path-info --json return an object not array
Before it returned a list of JSON objects with store object information,
including the path in each object. Now, it maps the paths to JSON
objects with the metadata sans path.

This matches how `nix derivation show` works.

Quite hillariously, none of our existing functional tests caught this
change to `path-info --json` though they did use it. So just new
functional tests need to be added.
2023-11-06 11:06:31 -05:00
John Ericson
a7212e169b Include compression in the NarInfo JSON format
It was forgotten before.
2023-11-06 11:06:31 -05:00
John Ericson
937e02e7b9 Shuffle ValidPathInfo JSON rendering
`Store::pathInfoToJSON` was a rather baroque functions, being full of
parameters to support both parsed derivations and `nix path-info`. The
common core of each, a simple `dValidPathInfo::toJSON` function, is
factored out, but the rest of the logic is just duplicated and then
specialized to its use-case (at which point it is no longer that
duplicated).

This keeps the human oriented CLI logic (which is currently unstable)
and the core domain logic (export reference graphs with structured
attrs, which is stable), separate, which I think is better.
2023-11-06 11:06:31 -05:00
John Ericson
0b0d1b5214 Add comparison functions for NarInfo
We will need these for tests.
2023-11-06 09:51:21 -05:00
ThinkChaos
2fb49759b8
fix(ssh): log first line of stdout
Spent a while debugging why `nix-copy-closure` wasn't working anymore
and it was my shell RC printing something I added for debug.
Hopefully this can save someone else some time.
2023-11-06 08:46:19 -05:00
ThinkChaos
6472c3bf0d
fix(ssh): extraneous master processes 2023-11-06 08:45:14 -05:00
John Ericson
ac89bb064a Split up util.{hh,cc}
All OS and IO operations should be moved out, leaving only some misc
portable pure functions.

This is useful to avoid copious CPP when doing things like Windows and
Emscripten ports.

Newly exposed functions to break cycles:

 - `restoreSignals`
 - `updateWindowSize`
2023-11-05 12:20:02 -05:00
John Ericson
2678b51b31 Narrower scope for nativeSystem
I don't think we need a CPP defininition and a header entry, and this
way allows constant expression elimination.
2023-11-05 12:18:28 -05:00
Eelco Dolstra
b0455e9931 Fix uninitialized variable warnings on i686-linux
https://hydra.nixos.org/build/239849607
2023-11-03 12:04:20 +01:00
Eelco Dolstra
55dd1244d2 parseDerivation(): Fix warning about uninitialized 'version' variable 2023-11-03 12:04:20 +01:00
John Ericson
d15c3a33e6 Don't use std::invocable C++ concept yet
It s not supported on all platforms yet. Can revert this once it is.
2023-11-02 12:06:29 -04:00
John Ericson
b107431816 Systematize characterization tests a bit more
Deduplicating code moreover enforcing the pattern means:

 - It is easier to write new characterization tests because less boilerplate

 - It is harder to mess up new tests because there are fewer places to
   make mistakes.

Co-authored-by: Jacek Galowicz <jacek@galowicz.de>
2023-11-02 12:06:08 -04:00
John Ericson
90de958637
Merge pull request #9265 from obsidiansystems/better-parse-sink
Make `ParseSink` a bit better
2023-11-02 09:28:29 -04:00
Eelco Dolstra
5223114c93
Merge pull request #9269 from edolstra/unify-accessor
Unify `FSAccessor` and `SourceAccessor`
2023-11-02 14:23:10 +01:00
Eelco Dolstra
e47984ce0b
Fix whitespace
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-11-01 20:19:08 +01:00
Zbigniew Jędrzejewski-Szmek
eab9292738 fix: gcc complains about if which doesn't guard the indented statement 2023-11-01 18:10:06 +01:00
Eelco Dolstra
2f5c1a27dc LocalStoreAccessor: Reuse PosixSourceAccessor 2023-11-01 17:22:25 +01:00
Eelco Dolstra
1a902f5fa7 Merge FSAccessor into SourceAccessor 2023-11-01 17:09:28 +01:00
Eelco Dolstra
50aae0a14c FSAccessor: Make the fileSize and narOffset fields optional
The narOffset field only applies to NAR accessors. The fileSize field
may be too expensive to compute for certain accessors (e.g. libgit).
2023-11-01 15:39:40 +01:00
Eelco Dolstra
5381123879 Unify DirEntries types 2023-11-01 15:33:35 +01:00
Eelco Dolstra
8ffd1695ce Unify FSAccessor::Type and SourceAccessor::Type 2023-11-01 14:43:20 +01:00
Eelco Dolstra
b2ac6fc040 Remove FSAccessor::Type::tMissing
Instead stat() now returns std::nullopt to denote that the file
doesn't exist.
2023-11-01 14:36:40 +01:00
John Ericson
1093d6585f Make ParseSink a bit better
I wouldn't call it *good* yet, but this will do for now.

- `RetrieveRegularNARSink` renamed to `RegularFileSink` and moved
  accordingly because it actually has nothing to do with NARs in
  particular.

  - its `fd` field is also marked private

- `copyRecursive` introduced to dump a `SourceAccessor` into a
  `ParseSink`.

- `NullParseSink` made so `ParseSink` no longer has sketchy default
  methods.

This was done while updating #8918 to work with the new
`SourceAccessor`.
2023-11-01 02:36:43 -04:00
John Ericson
b2cae33aef Remove bug-avoiding StoreConfig * casts for settings
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80431 has been fixed, and
per the previous commit we now check that is the case at build time.
2023-10-31 12:09:46 -04:00
John Ericson
a419b61497 Turn derivation unit tests into unit characterization tests
The brings a number of advantages, including:

- Easier to update test data if design changes (and I do think our
  derivation JSON is not yet complaint with the guidelines).

- Easier to reuse test data in other implementations, inching closer to
  compliance tests for Nix *the concept* rather than any one
  implementation.
2023-10-26 18:09:01 -04:00
Sergei Trofimovich
e69c764708 local-derivation-goal.cc: slightly clarify waiting message
Before the change builder ID exhaustion printed the following message:

    [0/1 built] waiting for UID to build '/nix/store/hiy9136x0iyib4ssh3w3r5m8pxjnad50-python3.11-breathe-4.35.0.drv'

After the change it should be:

    [0/1 built] waiting for a free build user ID for '/nix/store/hiy9136x0iyib4ssh3w3r5m8pxjnad50-python3.11-breathe-4.35.0.drv'
2023-10-26 20:54:21 +01:00
John Ericson
1dc6a65d36
Merge pull request #9238 from tfc/small-improvements2
Small improvements 2
2023-10-26 09:35:03 -04:00
Jacek Galowicz
28c39c370c Provide default value for id to fix warning 2023-10-26 09:47:40 +02:00
Jacek Galowicz
b66381e8d8 Use using instead of typedef 2023-10-26 09:47:40 +02:00
Eelco Dolstra
622191c2b5
Merge pull request #8965 from Artturin/bindfilesinchroot
Bindmount files instead of hardlinking or copying to chroot
2023-10-25 19:10:03 +02:00
Jacek Galowicz
f555c98a34 Improve loop over gid container 2023-10-25 16:10:35 +02:00
Jacek Galowicz
b113d925de Fix warning 2023-10-24 19:57:38 +02:00
Jacek Galowicz
eaced12c94 Fix signed vs. unsigned comparison warning and improve code 2023-10-24 19:57:38 +02:00
Robert Hensing
cde3c63617
system-features: Typo
There I was, thinking all of Apple's OSes started with lower case.
2023-10-23 19:30:00 +02:00
Valentin Gagarin
cd680bd53d
Merge how-to section on S3 buckets into S3 store docs (#7972)
Rather than having a misc tutorial page in the grab-bag "package management" section, this information should just be part of the S3 store docs.

---------

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2023-10-23 13:22:33 -04:00
John Ericson
8b68bbb777
Merge pull request #6223 from obsidiansystems/worker-proto-with-version
Give `nix daemon` and `nix-store --serve` protocols separate serializers with version info
2023-10-23 09:16:23 -04:00
Eelco Dolstra
955bbe53c5
Merge pull request #9177 from edolstra/input-accessors
Backport FSInputAccessor and MemoryInputAccessor from lazy-trees
2023-10-23 11:42:04 +02:00
John Ericson
70f8b96c11 Factor out UnkeyedValidPathInfo and test
This makes the path info serialisers ideomatic again, which allows me to
test them.
2023-10-20 15:21:04 -04:00
John Ericson
596bd469cc Move ValidPathInfo serialization code to worker-protocol.{cc.hh}
It does not belong with the data type itself.

This also materializes the fact that `copyPath` does not do any version
negotiation just just hard-codes "16".

The non-standard interface of these serializers makes it harder to test,
but this is fixed in the next commit which then adds those tests.
2023-10-20 15:20:48 -04:00
John Ericson
ab822af0df Factor out serialization for BuildResult
Worker Protocol:

Note that the worker protocol already had a serialization for
`BuildResult`; this was added in
a4604f1928. It didn't have any versioning
support because at that time reusable seralizers were not away for the protocol
version. It could thus only be used for new messages also introduced in
that commit.

Now that we do support versioning in reusable serializers, we can expand
it to support all known versions and use it in many more places.

The exist test data becomes the version 1.29 tests: note that those
files' contents are unchanged. 1.28 and 1.27 tests are added to cover
the older code-paths.

The keyered build result test only has 1.29 because the keying was also
added in a4604f19284254ac98f19a13ff7c2216de7fe176; the older
serializations are always used unkeyed.

Serve Protocol:

Conversely, no attempt was made to factor out such a serializer for the
serve protocol, so our work there in this commit for that protocol
proceeds from scratch.
2023-10-20 15:19:28 -04:00
John Ericson
4372738efe Systematize the worker protocol derived path serialiser
It was some ad-hoc functions to account for versions, while the already
factored-out serializer just supported the latest version.

Now, we can fold that version-specific logic into the factored out one,
and so we do.
2023-10-20 15:19:28 -04:00
John Ericson
96c58550b8 Test more derived paths 2023-10-20 15:19:28 -04:00
John Ericson
862d16436b Remove the ValidPathInfo == operator
It is dead code. It was added in
8e0946e8df as part of the repeated /
enforce-determinism feature, but that was removed in
8fdd156a65.

It is not good because it skips many fields. For testing purposes we
will soon want to add a new one that doesn't skip fields, but we want to
make sure making == sensitive to those fields won't change how Nix
works. Proving in this commit that the old version is dead code achieves
that.
2023-10-20 12:39:17 -04:00
John Ericson
3e6b9f9357 Remove prevInfos as its dead code
It is unused since 8e0946e8df removed
support for the repeat and enforce-determinism options.
2023-10-20 12:39:17 -04:00
Robert Hensing
4d17c59d8d
Merge pull request #9157 from obsidiansystems/protocol-versions
Add protocol versions to `{Worker,Serve}Proto::*Conn`
2023-10-20 15:34:26 +02:00