Commit graph

4269 commits

Author SHA1 Message Date
John Ericson
cb4f85f11c Merge branch 'master' into overlayfs-store 2024-02-29 09:54:21 -05:00
Théophane Hufschmitt
da90be789d Fix a too smart implicit cast
Apparently gcc is able to implicitly cast from `FileIngestionMethod` to
`ContentAddressMethod`, but clang isn't. So explicit the cast
2024-02-28 08:00:17 +01:00
John Ericson
d4ad1fcf30 Avoid creating temporary store object for git over the wire
Instead, serialize as NAR and send that over, then rehash sever side.
This is alorithmically simpler, but comes at the cost of a newer
parameter to `Store::addToStoreFromDump`.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2024-02-27 11:27:34 -05:00
John Ericson
201551c937 Add Git object hashing to the store layer
Part of RFC 133

Extracted from our old IPFS branches.

Co-Authored-By: Matthew Bauer <mjbauer95@gmail.com>
Co-Authored-By: Carlo Nucera <carlo.nucera@protonmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Florian Klink <flokli@flokli.de>
2024-02-27 11:27:34 -05:00
Yueh-Shun Li
fd47f76da9 treewide: hash type -> hash algorithm
"hash type" -> "hash algorithm" in all comments, documentation, and
messages.

ht -> ha, [Hh]ashType -> [HhashAlgo] for all local variables and
function arguments. No API change is made.

Continuation of 5334c9c792 and 837b889c41.
2024-02-26 18:09:06 +08:00
Eelco Dolstra
cabee98152 Tarball fetcher: Use the content-addressed Git cache
Backported from the lazy-trees branch.
2024-02-20 12:57:36 +01:00
Eelco Dolstra
2b946aad8e
Merge pull request #10006 from edolstra/fix-impure-fetchurl
<nix/fetchurl.nix>: Restore support for "impure = true"
2024-02-13 22:50:55 +01:00
John Ericson
41dd9857c7 Proper parse and render functions for FileIngestionMethod and ContentAddressMethod
No outward facing behavior is changed.

Older methods with same names that operate on on method + algo pair (for
old-style `<method>:algo`) are renamed to `*WithAlgo`.)

The functions are unit-tested in the same way the names for the hash
algorithms are tested.
2024-02-13 10:30:16 -05:00
Eelco Dolstra
b439b1fc66
Merge pull request #9993 from edolstra/builtin-paths
builtin:{unpack-channel,buildenv}: Get output path from the derivation
2024-02-13 14:57:13 +01:00
Valentin Gagarin
39c353f6fa
reword description of the cores setting (#9522)
* reword description of the `cores` setting

- be precise about the `builder` executable
- clearly distinguish between `builder` and job parallelism
- clarify the role of `mkDerivation` in the example
- remove prose for the default, it's shown programmatically
- mention relation to `max-jobs`
2024-02-13 13:52:51 +00:00
Eelco Dolstra
bb63bd50e6 <nix/fetchurl.nix>: Restore support for "impure = true" 2024-02-13 14:14:20 +01:00
Valentin Gagarin
fd82ba0985
extract reference documentation on remote builds (#9526)
- move all reference documentation to the `builders` configuration setting
- reword documentation on machine specification, add examples
- disable showing the default value, as it rendered as `@/dummy/machines`, which is wrong
- highlight the examples
- link to the configuration docs for distributed builds
- builder -> build machine

Co-authored-by: Janik H <janik@aq0.de>
2024-02-13 14:13:56 +01:00
Valentin Gagarin
e37d502895
add instructions to wipe the substituter lookup cache (#9498)
* add instructions to wipe the substituter lookup cache
2024-02-13 13:34:51 +01:00
Anton Samokhvalov
64cbd4c05a
Update nar-info-disk-cache.cc
fix case when asserts are no-op, like in release build
2024-02-12 23:37:40 +03:00
John Ericson
5169f5f4d9
Apply suggestions from code review
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-02-12 12:01:54 -05:00
John Ericson
eb76b35efa Merge remote-tracking branch 'upstream/master' into store-path-complete-construction 2024-02-12 11:22:54 -05:00
Eelco Dolstra
a9b69b2fff builtin:{unpack-channel,buildenv}: Get output path from the derivation
Similar to 1ee42c5b88, get the "out"
path from the derivation (and complain if it doesn't exist), rather
than getting it from the environment.
2024-02-12 16:34:59 +01:00
Eelco Dolstra
619ca631d0 Fix "may be used uninitialized" warning 2024-02-12 15:29:48 +01:00
Eelco Dolstra
fb5a792280 runPostBuildHook(): Be less chatty
Don't spam the user with "running post-build-hook" messages. It's up
to the post-build hook if it has something interesting to say.
2024-02-09 15:55:24 +01:00
Rebecca Turner
c0e7f50c1a
Rename hintfmt to HintFmt 2024-02-08 11:58:25 -08:00
Rebecca Turner
149bd63afb
Cleanup fmt.hh
When I started contributing to Nix, I found the mix of definitions and
names in `fmt.hh` to be rather confusing, especially the small
difference between `hintfmt` and `hintformat`. I've renamed many classes
and added documentation to most definitions.

- `formatHelper` is no longer exported.
- `fmt`'s documentation is now with `fmt` rather than (misleadingly)
  above `formatHelper`.
- `yellowtxt` is renamed to `Magenta`.

  `yellowtxt` wraps its value with `ANSI_WARNING`, but `ANSI_WARNING`
  has been equal to `ANSI_MAGENTA` for a long time. Now the name is
  updated.
- `normaltxt` is renamed to `Uncolored`.
- `hintfmt` has been merged into `hintformat` as extra constructor
  functions.
- `hintformat` has been renamed to `hintfmt`.
- The single-argument `hintformat(std::string)` constructor has been
  renamed to a static member `hintformat::interpolate` to avoid pitfalls
  with using user-generated strings as format strings.
2024-02-08 11:51:03 -08:00
Théophane Hufschmitt
1ba9780cf5
Merge pull request #9834 from 9999years/structured-errors
Towards structured error classes
2024-02-08 20:00:25 +01:00
Théophane Hufschmitt
fb78a99e04
Merge pull request #9924 from 9999years/rename-yellowtxt
Rename `yellowtxt` -> `magentatxt`
2024-02-08 10:01:30 +01:00
Eelco Dolstra
a6737b7e17 CanonPath, SourcePath: Change operator + to /
This is less confusing and makes it more similar to std::filesystem::path.
2024-02-05 15:17:39 +01:00
Rebecca Turner
a7939a6c2a
Rename yellowtxt -> magentatxt
`yellowtxt` wraps its value with `ANSI_WARNING`, but `ANSI_WARNING` has
been equal to `ANSI_MAGENTA` for a long time. Now the name is updated.
2024-02-03 19:28:11 -08:00
Rebecca Turner
c6a89c1a16
libexpr: Support structured error classes
While preparing PRs like #9753, I've had to change error messages in
dozens of code paths. It would be nice if instead of

    EvalError("expected 'boolean' but found '%1%'", showType(v))

we could write

    TypeError(v, "boolean")

or similar. Then, changing the error message could be a mechanical
refactor with the compiler pointing out places the constructor needs to
be changed, rather than the error-prone process of grepping through the
codebase. Structured errors would also help prevent the "same" error
from having multiple slightly different messages, and could be a first
step towards error codes / an error index.

This PR reworks the exception infrastructure in `libexpr` to
support exception types with different constructor signatures than
`BaseError`. Actually refactoring the exceptions to use structured data
will come in a future PR (this one is big enough already, as it has to
touch every exception in `libexpr`).

The core design is in `eval-error.hh`. Generally, errors like this:

    state.error("'%s' is not a string", getAttrPathStr())
      .debugThrow<TypeError>()

are transformed like this:

    state.error<TypeError>("'%s' is not a string", getAttrPathStr())
      .debugThrow()

The type annotation has moved from `ErrorBuilder::debugThrow` to
`EvalState::error`.
2024-02-01 16:39:38 -08:00
Eelco Dolstra
b8b739e484 builtin:fetchurl: Get output hash info from the drv 2024-02-01 22:01:02 +01:00
Eelco Dolstra
1ee42c5b88 builtin:fetchurl: Ensure a fixed-output derivation
Previously we didn't check that the derivation was fixed-output, so
you could use builtin:fetchurl to impurely fetch a file.
2024-02-01 21:46:01 +01:00
John Ericson
dc439eaf23 Fill in missing markdown link dest 2024-02-01 11:20:19 -05:00
John Ericson
31881d651a Merge remote-tracking branch 'upstream/master' into overlayfs-store 2024-02-01 11:07:47 -05:00
Eelco Dolstra
ef6d055ace
Merge pull request #9896 from hercules-ci/tidy-tidy
Disable a slightly annoying clang-tidy check
2024-02-01 09:02:46 +01:00
Robert Hensing
0f2e9e6bd2 Typo 2024-02-01 01:01:04 +01:00
Robert Hensing
4072a8fea0
Merge pull request #9867 from hercules-ci/issue-912
#912 allow leading period
2024-01-31 19:10:59 +01:00
Robert Hensing
f1b4663805 Disallow store path names that are . or .. (plus opt. -)
As discussed in the maintainer meeting on 2024-01-29.

Mainly this is to avoid a situation where the name is parsed and
treated as a file name, mostly to protect users.
.-* and ..-* are also considered invalid because they might strip
on that separator to remove versions. Doesn't really work, but that's
what we decided, and I won't argue with it, because .-* probably
doesn't seem to have a real world application anyway.
We do still permit a 1-character name that's just "-", which still
poses a similar risk in such a situation. We can't start disallowing
trailing -, because a non-zero number of users will need it and we've
seen how annoying and painful such a change is.

What matters most is preventing a situation where . or .. can be
injected, and to just get this done.
2024-01-31 18:35:19 +01:00
Robert Hensing
40254092dd
Merge pull request #9770 from hercules-ci/refactor-rename-derivation-isPure
Refactor rename derivation type `isPure`
2024-01-27 11:24:20 +01:00
Robert Hensing
9ddd0f2af8 Revert "StorePath: reject names starting with '.'"
This reverts commit 24bda0c7b3.
2024-01-27 11:18:03 +01:00
Robert Hensing
49b25ea85c refactor: Impure derivation type isPure -> isImpure
To quote the method doc:

Non-impure derivations can still behave impurely, to the degree permitted
by the sandbox. Hence why this method isn't `isPure`: impure derivations
are not the negation of pure derivations. Purity can not be ascertained
except by rather heavy tools.
2024-01-27 11:00:10 +01:00
John Ericson
a9e10a1dbd Make StoreConfig::getDefaultSystemFeatures a static method
This makes something in Hydra bit simpler. If someday the default
depends on the other config options, we can always change it back.
2024-01-24 21:34:31 -05:00
John Ericson
1e24db6f9a Convert Machine::speedFactor from a non-neg int to a non-neg float
The short motivation is to match Hydra, so we can de-dup.

The long version is layed out in
https://github.com/NixOS/nix/issues/9840.
2024-01-24 01:03:07 -05:00
John Ericson
0aa85088de Factor out Machine::systemSupported
There's just enough logic (the `"builtin"` special case) that makes this
worthy of its own method.
2024-01-23 12:53:51 -05:00
John Ericson
870acc2892 Add API docs to Machine methods 2024-01-23 12:50:48 -05:00
John Ericson
739032762a Make Machine::systemTypes a set not vector
This is more conceptually correct (the order does not matter), and also
matches what Hydra already does.

(Nix and Hydra matching is needed for dedup
https://github.com/NixOS/hydra/issues/1164)
2024-01-23 12:30:26 -05:00
Eelco Dolstra
b6aee9a93f
Merge pull request #9833 from obsidiansystems/ssh-arg-split
Make `SSHMaster::startCommand` work on an args list
2024-01-23 14:34:41 +01:00
John Ericson
6365bbfa81 Improve the FileSystemObjectSink interface
More invariants are enforced in the type, and less state needs to be
stored in the main sink itself. The method here is roughly that known as
"session types".

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-01-22 18:01:21 -05:00
John Ericson
966d6fcd01 ParseSink -> FileSystemObjectSink
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-01-22 18:01:18 -05:00
John Ericson
b71673109c Make SSHMaster::startCommand work on an args list
This avoids split-on-whitespace errors:

- No more `bash -c` needed

- No more `shellEscape` needed

- `remote-program` ssh store setting also cleanly supports args (e.g.
  `nix daemon`)

- `ssh` uses `--` to separate args for SSH from args for the command to
  run.

and will help with Hydra dedup.

Some code taken from #6628.

Co-Authored-By: Alexander Bantyev <balsoft@balsoft.ru>
2024-01-22 17:46:57 -05:00
John Ericson
e960b28230 Factor our ServeProto::BasicServerConnection::handshake
We'll need this for unit testing.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-01-22 12:43:11 -05:00
John Ericson
4a5ca576da Factor out ServeProto::BasicClientConnection::handshake Hydra to share 2024-01-22 12:43:11 -05:00
John Ericson
4580bed3e4 LegacySSHStore::openConnection move more logic inside catch block
Broader error handling logic is more robust.
2024-01-22 12:43:11 -05:00
John Ericson
ce2f714e6d Start factoring out the serve protocol for Hydra to share
Factor out `ServeProto::BasicClientConnection` for Hydra to share

- `queryValidPaths`: Hydra uses the lock argument differently than Nix,
  so we un-hard-code it.

- `buildDerivationRequest`: Just the request half, as Hydra does some
  things between requesting and responding.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-01-22 12:43:03 -05:00