Commit graph

469 commits

Author SHA1 Message Date
Eelco Dolstra
eab2919119 Use SourcePath in more places
Now that SourcePath uses a SourceAccessor instead of an InputAccessor,
we can use it in function signatures instead of passing a
SourceAccessor and CanonPath separately.
2024-05-06 19:05:42 +02:00
John Ericson
8433027e35 Build a minimized Nix with MinGW
At this point many features are stripped out, but this works:

- Can run libnix{util,store,expr} unit tests
- Can run some Nix commands

Co-Authored-By volth <volth@volth.com>
Co-Authored-By Brian McKenna <brian@brianmckenna.org>
2024-04-17 12:26:10 -04:00
Valentin Gagarin
13c2005e7d
add intermediate variables and clarifying comments (#9274)
* add intermediate variables and clarifying comments

Co-authored-by: Alexander Groleau <alex@proof.construction>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-04-12 17:43:35 +02:00
John Ericson
c99c80f075 Merge remote-tracking branch 'upstream/master' into overlayfs-store 2024-04-05 16:32:02 -04:00
John Ericson
e4d9b207c2 Factor out isRootUser function 2024-03-29 16:33:01 -04:00
John Ericson
cb4f85f11c Merge branch 'master' into overlayfs-store 2024-02-29 09:54:21 -05:00
John Ericson
d4ad1fcf30 Avoid creating temporary store object for git over the wire
Instead, serialize as NAR and send that over, then rehash sever side.
This is alorithmically simpler, but comes at the cost of a newer
parameter to `Store::addToStoreFromDump`.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2024-02-27 11:27:34 -05:00
John Ericson
201551c937 Add Git object hashing to the store layer
Part of RFC 133

Extracted from our old IPFS branches.

Co-Authored-By: Matthew Bauer <mjbauer95@gmail.com>
Co-Authored-By: Carlo Nucera <carlo.nucera@protonmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Florian Klink <flokli@flokli.de>
2024-02-27 11:27:34 -05:00
John Ericson
5169f5f4d9
Apply suggestions from code review
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-02-12 12:01:54 -05:00
John Ericson
eb76b35efa Merge remote-tracking branch 'upstream/master' into store-path-complete-construction 2024-02-12 11:22:54 -05:00
Eelco Dolstra
619ca631d0 Fix "may be used uninitialized" warning 2024-02-12 15:29:48 +01:00
John Ericson
31881d651a Merge remote-tracking branch 'upstream/master' into overlayfs-store 2024-02-01 11:07:47 -05:00
John Ericson
966d6fcd01 ParseSink -> FileSystemObjectSink
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-01-22 18:01:18 -05:00
Robert Hensing
382fa51ff0
Merge pull request #9794 from hercules-ci/queryPathInfoFromClientCache
refactor: Extract `Store::queryPathInfoFromClientCache`
2024-01-19 17:33:52 +01:00
Robert Hensing
d19627e8b4 refactor: Remove throw from queryPathInfoFromClientCache
Return a value instead of throwing.

Rather than the more trivial refactor of wrapping the return value in
another std::optional, we retain the meaning of the outer optional:
"we know at least something."

So we have changed:
return nullopt    -> return nullopt
throw InvalidPath -> return make_optional(nullptr)
return vpi        -> return make_optional(vpi)
2024-01-19 17:01:19 +01:00
Robert Hensing
8983ee8b2e refactor: Un-callback transform Store::queryPathInfoFromClientCache
This part of the code was not necessarily callback based.
Removing CPS is always nice; particularly if there's no loss of
functionality, like here.
2024-01-19 17:01:19 +01:00
John Ericson
a34ec0bd12 Include store path exact spec in the docs
This is niche, but deserves to be in the manual because it is describing
behavior visible to the outside world, not mere implementation details.
2024-01-18 22:19:14 -05:00
Eelco Dolstra
dca0a80240 copyStorePath(): Bail out early if the store path already exists
In rare cases (e.g. when using allowSubstitutes = false), it's
possible that we simultaneously have a DerivationGoal *and* a
SubstitutionGoal building the same path. So if a DerivationGoal
already built the path while the SubstitutionGoal was waiting for a
download slot, it saves us a superfluous download to exit early.
2024-01-18 17:19:15 +01:00
Robert Hensing
d0a284284b refactor: Extract simply, awkwardly Store::queryPathInfoFromClientCache
This is useful for determining quickly which substituters to query.

An alternative would be for users to invoke the narinfo cache db directly,
so why do we need this change?

 - It is easier to use. I believe Nix itself should also use it.

 - This way, the narinfo cache db remains an implementation detail.

 - Callers get to use the in-memory cache as well.
2024-01-17 17:54:19 +01:00
John Ericson
ff6de4a9ee
Merge pull request #9662 from shlevy/flat-fixed-references-assert
Improve error message for fixed-outputs with references.
2024-01-08 10:46:37 -05:00
Shea Levy
eeb2f083c5
Improve error message for fixed-outputs with references.
This codepath is possible, e.g. with a dockerTools.pullImage of an image with a Nix store.
2024-01-07 07:32:31 -05:00
John Ericson
12bb8cdd38 Signer infrastructure: Prep for #9076
This sets up infrastructure in libutil to allow for signing other than
by a secret key in memory. #9076 uses this to implement remote signing.

(Split from that PR to allow reviewing in smaller chunks.)

Co-Authored-By: Raito Bezarius <masterancpp@gmail.com>
2024-01-03 16:13:55 -05:00
John Ericson
ed26b186fb Remove now-redundant text-hashing store methods
`addTextToStore` and `computeStorePathFromDump` are now redundant.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-12-18 10:44:10 -05:00
John Ericson
dfc876531f Organize content addressing, use SourceAccessor with Store::addToStore
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-12-18 10:41:54 -05:00
John Ericson
245af3ea02 Merge remote-tracking branch 'upstream/master' into overlayfs-store 2023-12-11 13:12:09 -05:00
John Ericson
5f30c8acc7 Give Store::queryDerivationOutputMap and evalStore argument
Picking up where https://github.com/NixOS/nix/pull/9563 left off.
2023-12-11 10:39:08 -05:00
Peter Kolloch
5334c9c792 HashType: Rename to HashAlgorithm
To be consistent with CLI, nix API
and many other references.

As part of this, we also converted it to a scoped enum.

https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Robert Hensing
fcf09813c6
Merge pull request #6236 from obsidiansystems/store-dir-config
Factor out `StoreDirConfig`
2023-12-01 15:38:14 +01:00
John Ericson
937e02e7b9 Shuffle ValidPathInfo JSON rendering
`Store::pathInfoToJSON` was a rather baroque functions, being full of
parameters to support both parsed derivations and `nix path-info`. The
common core of each, a simple `dValidPathInfo::toJSON` function, is
factored out, but the rest of the logic is just duplicated and then
specialized to its use-case (at which point it is no longer that
duplicated).

This keeps the human oriented CLI logic (which is currently unstable)
and the core domain logic (export reference graphs with structured
attrs, which is stable), separate, which I think is better.
2023-11-06 11:06:31 -05:00
John Ericson
ac89bb064a Split up util.{hh,cc}
All OS and IO operations should be moved out, leaving only some misc
portable pure functions.

This is useful to avoid copious CPP when doing things like Windows and
Emscripten ports.

Newly exposed functions to break cycles:

 - `restoreSignals`
 - `updateWindowSize`
2023-11-05 12:20:02 -05:00
John Ericson
dde1d86338 Restrict some code to StoreDirConfig
- part of eval cache
 - part of derivations
 - derived path
 - store path with outputs
 - serializers
2023-11-04 19:05:36 -04:00
John Ericson
e97ac09abe Factor out StoreDirConfig
More progress on #5729.
2023-11-04 19:05:36 -04:00
Eelco Dolstra
b0455e9931 Fix uninitialized variable warnings on i686-linux
https://hydra.nixos.org/build/239849607
2023-11-03 12:04:20 +01:00
John Ericson
90de958637
Merge pull request #9265 from obsidiansystems/better-parse-sink
Make `ParseSink` a bit better
2023-11-02 09:28:29 -04:00
Eelco Dolstra
1a902f5fa7 Merge FSAccessor into SourceAccessor 2023-11-01 17:09:28 +01:00
John Ericson
1093d6585f Make ParseSink a bit better
I wouldn't call it *good* yet, but this will do for now.

- `RetrieveRegularNARSink` renamed to `RegularFileSink` and moved
  accordingly because it actually has nothing to do with NARs in
  particular.

  - its `fd` field is also marked private

- `copyRecursive` introduced to dump a `SourceAccessor` into a
  `ParseSink`.

- `NullParseSink` made so `ParseSink` no longer has sketchy default
  methods.

This was done while updating #8918 to work with the new
`SourceAccessor`.
2023-11-01 02:36:43 -04:00
John Ericson
250c3541bb Use local-overlay:// not local-overlay for store URL
This is a bit uglier, but allows us to avoid an ad-hoc special case in
`store-api.cc`.
2023-10-25 15:29:11 -04:00
John Ericson
5c1cb0b696 Merge remote-tracking branch 'upstream/master' into overlayfs-store 2023-10-23 13:13:37 -04:00
John Ericson
8b68bbb777
Merge pull request #6223 from obsidiansystems/worker-proto-with-version
Give `nix daemon` and `nix-store --serve` protocols separate serializers with version info
2023-10-23 09:16:23 -04:00
Eelco Dolstra
955bbe53c5
Merge pull request #9177 from edolstra/input-accessors
Backport FSInputAccessor and MemoryInputAccessor from lazy-trees
2023-10-23 11:42:04 +02:00
John Ericson
596bd469cc Move ValidPathInfo serialization code to worker-protocol.{cc.hh}
It does not belong with the data type itself.

This also materializes the fact that `copyPath` does not do any version
negotiation just just hard-codes "16".

The non-standard interface of these serializers makes it harder to test,
but this is fixed in the next commit which then adds those tests.
2023-10-20 15:20:48 -04:00
Yueh-Shun Li
e026f3e1ae treewide: Reference HashFormat members with scope
Base* -> HashFormat::Base*
2023-10-19 00:56:41 +08:00
Yueh-Shun Li
838c70f621 treewide: Rename hashBase to hashFormat
hashBase is ambiguous, since it's not about the digital bases, but about
the format of hashes. Base16, Base32 and Base64 are all character maps
for binary encoding.

Rename the enum Base to HashFormat.

Rename variables of type HashFormat from [hash]Base to hashFormat,
including CmdHashBase::hashFormat and CmdToBase::hashFormat.
2023-10-19 00:38:32 +08:00
Eelco Dolstra
ea38605d11 Introduce FSInputAccessor and use it
Backported from the lazy-trees branch. Note that this doesn't yet use
the access control features of FSInputAccessor.
2023-10-18 17:37:32 +02:00
John Ericson
4b9a621812 Guard the local overlay store behind an experimental feature 2023-08-02 20:30:42 -04:00
John Ericson
3fc838c8a8 Merge remote-tracking branch 'layered-store/experimental-stores' into overlayfs-store 2023-08-02 19:03:34 -04:00
John Ericson
3b592c880a Add infra for experimental store implemenations
This is analogous to that for experimental settings and flags that we
have also added as of late.
2023-08-02 15:46:38 -04:00
John Ericson
2cabf85b53 Merge remote-tracking branch 'upstream/master' into overlayfs-store 2023-07-24 15:39:36 -04:00
John Ericson
fe1fbdb5a1
Merge pull request #8724 from obsidiansystems/queryPartialDerivationOutputMap-evalStore
Give `queryPartialDerivationOutputMap` an `evalStore` parameter
2023-07-21 08:53:18 -04:00
John Ericson
6bc98c7fba Give queryPartialDerivationOutputMap an evalStore parameter
This makes it more useful. In general, the derivation will be in one
store, and the realisation info is in another.

This also helps us avoid duplication. See how `resolveDerivedPath` is
now simpler because it uses `queryPartialDerivationOutputMap`. In #8369
we get more flavors of derived path, and need more code to resolve them
all, and this problem only gets worse.

The fact that we need a new method to deal with the multiple dispatch is
unfortunate, but this generally relates to the fact that `Store` is a
sub-par interface, too bulky/unwieldy and conflating separate concerns.
Solving that is out of scope of this PR.

This is part of the RFC 92 work. See tracking issue #6316
2023-07-20 15:59:52 -04:00