Commit graph

16261 commits

Author SHA1 Message Date
Robert Hensing
3f9589f17e
Merge pull request #6312 from obsidiansystems/keyed-build-result
Shuffle `BuildResult` data definition, make state machine clearer, introduce `SingleDrvOutputs`
2023-04-17 18:08:05 +02:00
John Ericson
2eb493ca51 Fix DerivationOutput::fromJSON 2023-04-17 10:28:54 -04:00
Valentin Gagarin
39e0f609cc
Merge pull request #7514 from fricklerhandwerk/opt-I
remove incorrect reference to `NIX_PATH` documentation
2023-04-17 16:27:23 +02:00
John Ericson
e12efa3654 Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2023-04-17 10:16:57 -04:00
Robert Hensing
e641de085b
Merge pull request #3746 from obsidiansystems/path-info
Introduce `StoreReferences` and `ContentAddressWithReferences`
2023-04-17 15:49:48 +02:00
John Ericson
e95db8f2b9 nix-testing -> daemon-trust-override
And only enable in the tests that need it. This makes it less of a
sledgehammer.
2023-04-17 09:35:43 -04:00
John Ericson
b1343e8ad1 Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-17 09:27:17 -04:00
John Ericson
537e8719f2
Explain various .self = false,
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-17 09:15:11 -04:00
Valentin Gagarin
f4119a67cc
use @docroot@ link 2023-04-17 14:54:30 +02:00
Valentin Gagarin
d0cf615cbb
add link to nix-conf setting
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-04-17 14:27:26 +02:00
John Ericson
72ffa7fedb
Merge pull request #7732 from hercules-ci/make-initLibStore-viable-alternative
Make `initLibStore` a viable alternative
2023-04-17 08:04:41 -04:00
Robert Hensing
cb2615cf47 Merge remote-tracking branch 'upstream/master' into source-path 2023-04-17 11:41:50 +02:00
Robert Hensing
9af9c260fc
Merge pull request #8193 from hercules-ci/dry-strings
Deduplicate string literal rendering, fix 4909
2023-04-17 11:19:40 +02:00
Robert Hensing
36a473c5e8
Merge pull request #8196 from obsidiansystems/fix-8162
Mark experimental configuration settings programmatically
2023-04-17 11:18:11 +02:00
John Ericson
9800c1e807 Mark experimental configuration settings programmatically
Fix #8162

The test is changed to compare `nlohmann::json` values, not strings of dumped
JSON, which allows us to format things more nicely.
2023-04-16 10:58:04 -04:00
John Ericson
ba9ae691b6 Add optionalString to manual Nix lang utilities
Use it everywhere it could be also.
2023-04-16 10:57:46 -04:00
Robert Hensing
b6125772d7 libexpr: Move identifier-like printing to print.cc 2023-04-16 14:07:35 +02:00
Robert Hensing
28a5cdde02 libexpr/value/print.* -> libexpr/print.*
Generalizes the file to sensibly allow printing any part of the
language syntax.
2023-04-16 13:10:45 +02:00
Robert Hensing
1e2dd669bc printLiteral: Do not overload 2023-04-16 13:04:35 +02:00
Robert Hensing
9c74df5bb4
Format
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-04-15 20:56:51 +02:00
John Ericson
24866b71c4 Introduce SingleDrvOutputs
In many cases we are dealing with a collection of realisations, they are
all outputs of the same derivation. In that case, we don't need
"derivation hashes modulos" to be part of our map key, because the
output names alone will be unique. Those hashes are still part of the
realisation proper, so we aren't loosing any information, we're just
"normalizing our schema" by narrowing the "primary key".

Besides making our data model a bit "tighter" this allows us to avoid a
double `for` loop in `DerivationGoal::waiteeDone`. The inner `for` loop
was previously just to select the output we cared about without knowing
its hash. Now we can just select the output by name directly.

Note that neither protocol is changed as part of this: we are still
transferring `DrvOutputs` over the wire for `BuildResult`s. I would only
consider revising this once #6223 is merged, and we can mention protocol
versions inside factored-out serialization logic. Until then it is
better not change anything because it would come a the cost of code
reuse.
2023-04-15 12:51:19 -04:00
John Ericson
0f2b5146c7 Make restarting state machines explicit
If my memory is correct, @edolstra objected to modifying `wantedOutputs`
upon falling back to doing a build (as we did before), because we should
only modify it in response to new requests --- *actual* wants --- and
not because we are "incidentally" building all the outptus beyond what
may have been requested.

That's a fair point, and the alternative is to replace the boolean soup
with proper enums: Instead of modifying `wantedOuputs` som more, we'll
modify `needsRestart` to indicate we are passed the need.
2023-04-15 11:01:31 -04:00
John Ericson
37fca662b0 Make KeyedBuildResult, BuildResult like before, and fix bug another way
In https://github.com/NixOS/nix/pull/6311#discussion_r834863823, I
realized since derivation goals' wanted outputs can "grow" due to
overlapping dependencies (See `DerivationGoal::addWantedOutputs`, called
by `Worker::makeDerivationGoalCommon`), the previous bug fix had an
unfortunate side effect of causing more pointless rebuilds.

In paticular, we have this situation:

1. Goal made from `DerivedPath::Built { foo, {a} }`.

2. Goal gives on on substituting, starts building.

3. Goal made from `DerivedPath::Built { foo, {b} }`, in fact is just
   modified original goal.

4. Though the goal had gotten as far as building, so all outputs were
   going to be produced, `addWantedOutputs` no longer knows that and so
   the goal is flagged to be restarted.

This might sound far-fetched with input-addressed drvs, where we usually
basically have all our goals "planned out" before we start doing
anything, but with CA derivation goals and especially RFC 92, where *drv
resolution* means goals are created after some building is completed, it
is more likely to happen.

So the first thing to do was restore the clearing of `wantedOutputs` we
used to do, and then filter the outputs in `buildPathsWithResults` to
only get the ones we care about.

But fix also has its own side effect in that the `DerivedPath` in the
`BuildResult` in `DerivationGoal` cannot be trusted; it is merely the
*first* `DerivedPath` for which this goal was originally created.

To remedy this, I made `BuildResult` be like it was before, and instead
made `KeyedBuildResult` be a subclass wit the path. Only
`buildPathsWithResults` returns `KeyedBuildResult`s, everything else
just becomes like it was before, where the "key" is unambiguous from
context.

I think separating the "primary key" field(s) from the other fields is
good practical in general anyways. (I would like to do the same thing
for `ValidPathInfo`.) Among other things, it allows constructions like
`std::map<Key, ThingWithKey>` where doesn't contain duplicate keys and
just precludes the possibility of those duplicate keys being out of
sync.

We might leverage the above someday to overload `buildPathsWithResults`
to take a *set* of return a *map* per the above.

-----

Unfortunately, we need to avoid C++20 strictness on designated
initializers.

(BTW
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2287r1.html
this offers some new syntax for this use-case. Hopefully this will be
adopted and we can eventually use it.)

No having that yet, maybe it would be better to not make
`KeyedBuildResult` a subclass to just avoid this.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-15 11:01:31 -04:00
John Ericson
9df7f3f537 Introduce Worker::makeGoal
This takes a `DerivedPath` so the caller doesn't need to care about
which sort of goal does what.
2023-04-15 11:01:31 -04:00
Robert Hensing
ab228d73db
Merge pull request #8214 from raphaelr/installable-derived-path-warn
Display valid installable in `InstallableDerivedPath::parse` warning
2023-04-15 14:15:02 +02:00
John Ericson
ee420ac64e Legacy vs non-legacy to_string/parse for DerivedPath
As requested by @roberth, it is good to call out the specific instances
we care about, which is `!` for the RPC protocols, and `^` for humans.

This doesn't take advantage of parametricity as much, but since the
human and computer interfaces are good to decouple anyways (we don't
care if they drift further apart over time in the slightest) some
separation and slight duplication is fine.

Also, unit test both round trips.
2023-04-14 20:45:11 -04:00
Archit Gupta
bfc558c972 Whitelist commit-lockfile-summary in flake nixConfig 2023-04-14 11:33:38 -07:00
John Ericson
b41f739068
Merge pull request #7864 from obsidiansystems/quickstart-long-options
Use long options instead of short ones in the "quick start"
2023-04-14 09:13:16 -04:00
John Ericson
92346d1ed4
Merge pull request #8174 from NixOS/gloss-xp-features-note-installables
Start cross-referencing experimental features
2023-04-14 08:48:10 -04:00
John Ericson
a6f85e052c Support repairPath on most stores.
More progress on issue #5729

The method trivially generalizes to be store-implementation-agnostic, in
fact.

However, we force it to continue to be unimplemented with `RemoteStore`
and `LegacySSHStore` because the implementation we'd get via the
generalization is probably not the one users expect. This keeps our
hands untied to do it right going forward.

For more about the tension between the scheduler logic being
store-type-agnostic and remote stores doing their own scheduling, see
issues #5025 and #5056.
2023-04-14 08:29:06 -04:00
Robert Hensing
f5ab38a688
Merge pull request #8217 from obsidiansystems/push-get-fs-accessor-unsupported-down
Push `getFSAccessor` `unsupported(...)` down `Store` class hierarchy
2023-04-14 14:23:23 +02:00
John Ericson
d93e76fbb8 Start cross-referencing experimental features
- Create a glossary entry for experimental features.

- Have the man page experimental feature notice link `nix-commmand`.

  (Eventually this should be programmed, based on whether the command is
  experimental, and if so what experimental feature does it depend on.)

- Document which installables depend on which experimental features.

  I tried to use the same style (bold warning and block quote) that the
  top of the man page uses.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-04-14 07:45:08 -04:00
Eelco Dolstra
7eac8838df
Merge pull request #8199 from tweag/fix-sqlite-busy-reporting
Fix unnecessary reporting of SQLite busy errors
2023-04-14 10:50:50 +02:00
Eelco Dolstra
33fc09c2a5
Merge pull request #8176 from tweag/rename-confusing-write-method
Rename and protect `BufferedSink::write`
2023-04-14 10:44:36 +02:00
Raphael Robatsch
9e8f209036 Display valid installable in InstallableDerivedPath::parse warning
The warning message should produce an installable name that can be
passed to `nix build`, `nix path-info`, etc. again. Since the CLI
expects that the .drv path and the output names are separated by
a caret, the warning message must also separate the .drv path and output
names with a caret.

However, `DerivedPath::Built.to_string()` uses an exclamation point as
the separator instead. This commit adds a `separator` argument to the
to_string method.

This changes the warning message from:
If this command is now failing try again with '/nix/store/foo.drv!*'
to:
If this command is now failing try again with '/nix/store/foo.drv^*'
2023-04-14 10:32:03 +02:00
John Ericson
ee97f107e8 Push getFSAccessor unsupported(...) down Store class hierarchy
More progress on issue #5729.

Instead of having it by the default method in `Store` itself, have it be
the implementation in `DummyStore` and `LegacySSHStore`. Then just the
implementations which fail to provide the method pay the "penalty" of
dealing with the icky `unimplemented` function for non-compliance.

Combined with my other recent PRs, this finally makes `Store` have no
`unsupported` calls!
2023-04-13 13:39:44 -04:00
John Ericson
615c25b0dd Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-12 00:00:12 -04:00
Eelco Dolstra
ef0b48377d
Merge pull request #8206 from NixOS/bump-2.16.0
Bump version
2023-04-11 20:59:11 +02:00
Eelco Dolstra
a4b6d1d9a3 Bump version 2023-04-11 20:16:37 +02:00
Eelco Dolstra
37046432b9
Merge pull request #8201 from obsidiansystems/hacky-xp-setting-fix
Do not gate or hide experimental settings
2023-04-11 17:37:09 +02:00
Ben Radford
de3df3009b
Move warning timing logic into handleSQLiteBusy. 2023-04-11 16:14:16 +01:00
John Ericson
450e5ec618 Do not gate or hide experimental settings
This is somewhat hacky fix just for 2.15. I unintentionally hid them
from the manual, when no one wanted to hide them that (including
myself). I also required the experimental feature to be enabled in an
order-dependent way, which is not good.

The simplest fix for this immanent release is just to always show them,
and always allow them to be set.

Effectively undoes some changes from aa663b7e89
2023-04-11 10:56:48 -04:00
Eelco Dolstra
0a54624275
Merge pull request #8200 from edolstra/release-notes
2.15 release notes
2023-04-11 13:18:30 +02:00
Eelco Dolstra
e570a91661 Release notes 2023-04-11 12:40:56 +02:00
Eelco Dolstra
162c3f5f7c
Merge pull request #8170 from tweag/fix-aarch64-build
Explicitely define `LockFile::operator!=`
2023-04-11 12:35:23 +02:00
Ben Radford
da322ebda6
Revert "Add script to reproduce issue by inducing heavy load."
This reverts commit 213b838f9cfb820d2bc76d7c6edc468b27029945.
2023-04-11 11:15:34 +01:00
Ben Radford
7c56e84213
Warn after a second of being busy instead of immediately.
Getting the occasional SQLITE_BUSY is expected when the database is being
accessed concurrently. The retry will likely succeed so it is pointless to warn
immediately. Instead we track how long each retrySQLite block has been running,
and only begin warning after a second has elapsed (and then every 10 seconds
subsequently).
2023-04-11 11:15:34 +01:00
Ben Radford
7f5ca6192d
Add script to reproduce issue by inducing heavy load. 2023-04-11 11:15:34 +01:00
Eelco Dolstra
45ca4e6432
Merge pull request #8158 from tweag/harden-profiles-test
Fix the flaky `nix-profile` test
2023-04-11 12:12:09 +02:00
Eelco Dolstra
8fd91b3585
Merge pull request #8153 from obsidiansystems/more-labeler
Try auto-labeling a few more things
2023-04-11 12:10:34 +02:00