diff --git a/doc/manual/src/release-notes/rl-2.4.md b/doc/manual/src/release-notes/rl-2.4.md index f7ab9f6ad..def4a7b66 100644 --- a/doc/manual/src/release-notes/rl-2.4.md +++ b/doc/manual/src/release-notes/rl-2.4.md @@ -1,8 +1,498 @@ -# Release 2.4 (202X-XX-XX) +# Release 2.4 (2021-10-XX) - - It is now an error to modify the `plugin-files` setting via a - command-line flag that appears after the first non-flag argument - to any command, including a subcommand to `nix`. For example, - `nix-instantiate default.nix --plugin-files ""` must now become - `nix-instantiate --plugin-files "" default.nix`. - - Plugins that add new `nix` subcommands are now actually respected. +This is the first release in more than two years and is the result of +more than 2800 commits from 195 contributors since release 2.3. + +## Highlights + +* Nix's **error messages** have been improved a lot. For instance, + evaluation errors now point out the location of the error: + + ``` + $ nix build + error: undefined variable 'bzip3' + + at /nix/store/449lv242z0zsgwv95a8124xi11sp419f-source/flake.nix:88:13: + + 87| [ curl + 88| bzip3 xz brotli editline + | ^ + 89| openssl sqlite + ``` + +* The **`nix` command** has seen a lot of work and is now almost at + feature parity with the old command-line interface (the `nix-*` + commands). It aims to be [more modern, consistent and pleasant to + use](../contributing/cli-guideline.md) than the old CLI. It is still + marked as experimental but its interface should not change much + anymore in future releases. + +* **Flakes** are a new format to package Nix-based projects in a more + discoverable, composable, consistent and reproducible way. A flake + is just a repository or tarball containing a file named `flake.nix` + that specifies dependencies on other flakes and returns any Nix + assets such as packages, Nixpkgs overlays, NixOS modules or CI + tests. The new `nix` CLI is primarily based around flakes; for + example, a command like `nix run nixpkgs#hello` runs the `hello` + application from the `nixpkgs` flake. + + Flakes are currently marked as experimental. For an introduction, + see [this blog + post](https://www.tweag.io/blog/2020-05-25-flakes/). For detailed + information about flake syntax and semantics, see the [`nix flake` + manual page](../command-ref/new-cli/nix3-flake.md). + +* Nix's store can now be **content-addressed**, meaning that the hash + component of a store path is the hash of the path's + contents. Previously Nix could only build **input-addressed** store + paths, where the hash is computed from the derivation dependency + graph. Content-addressing allows deduplication, early cutoff in + build systems, and unprivileged closure copying. This is still [an + experimental + feature](https://discourse.nixos.org/t/content-addressed-nix-call-for-testers/12881). + +* The Nix manual has been converted into Markdown, making it easier to + contribute. In addition, every `nix` subcommand now has a manual + page, documenting every option. + +* A new setting that allows **experimental features** to be enabled + selectively. This allows us to merge unstable features into Nix more + quickly and do more frequent releases. + +## Other features + +* There are many new `nix` subcommands: + + - `nix develop` is intended to replace `nix-shell`. It has a number + of new features: + + * It automatically sets the output environment variables (such as + `$out`) to writable locations (such as `./outputs/out`). + + * It can store the environment in a profile. This is useful for + offline work. + + * It can run specific phases directly. For instance, `nix develop + --build` runs `buildPhase`. + + - It allows dependencies in the Nix store to be "redirected" to + arbitrary directories using the `--redirect` flag. This is + useful if you want to hack on a package *and* some of its + dependencies at the same time. + + - `nix print-dev-env` prints the environment variables and bash + functions defined by a derivation. This is useful for users of + other shells than bash (especially with `--json`). + + - `nix shell` was previously named `nix run` and is intended to + replace `nix-shell -p`, but without the `stdenv` overhead. It + simply starts a shell where some packages have been added to + `$PATH`. + + - `nix run` (not to be confused with the old subcommand that has + been renamed to `nix shell`) runs an "app", a flake output that + specifies a command to run, or an eponymous program from a + package. For example, `nix run nixpkgs#hello` runs the `hello` + program from the `hello` package in `nixpkgs`. + + - `nix flake` is the container for flake-related operations, such as + creating a new flake, querying the contents of a flake or updating + flake lock files. + + - `nix registry` allows you to query and update the flake registry, + which maps identifiers such as `nixpkgs` to concrete flake URLs. + + - `nix profile` is intended to replace `nix-env`. Its main advantage + is that it keeps track of the provenance of installed packages + (e.g. exactly which flake version a package came from). It also + has some helpful subcommands: + + * `nix profile history` shows what packages were added, upgraded + or removed between each version of a profile. + + * `nix profile diff-closures` shows the changes between the + closures of each version of a profile. This allows you to + discover the addition or removal of dependencies or size + changes. + + **Warning**: after a profile has been updated using `nix profile`, + it is no longer usable with `nix-env`. + + - `nix store diff-closures` shows the differences between the + closures of two store paths in terms of the versions and sizes of + dependencies in the closures. + + - `nix store make-content-addressable` rewrites an arbitrary closure + to make it content-addressed. Such paths can be copied into other + stores without requiring signatures. + + - `nix bundle` uses the [`nix-bundle` + program](https://github.com/matthewbauer/nix-bundle) to convert a + closure into a self-extracting executable. + + - Various other replacements for the old CLI, e.g. `nix store gc`, + `nix store delete`, `nix store repair`, `nix nar dump-path`, `nix + store prefetch-file`, `nix store prefetch-tarball`, `nix key` and + `nix daemon`. + +* Nix now has an **evaluation cache** for flake outputs. For example, + a second invocation of the command `nix run nixpkgs#firefox` will + not need to evaluate the `firefox` attribute because it's already in + the evaluation cache. This is made possible by the hermetic + evaluation model of flakes. + +* The new `--offline` flag disables substituters and causes all + locally cached tarballs and repositories to be considered + up-to-date. + +* The new `--refresh` flag causes all locally cached tarballs and + repositories to be considered out-of-date. + +* Many `nix` subcommands now have a `--json` option to produce + machine-readable output. + +* `nix repl` has a new `:doc` command to show documentation about + builtin functions (e.g. `:doc builtins.map`). + +* Binary cache stores now have an option `index-debug-info` to create + an index of DWARF debuginfo files for use by + [`dwarffs`](https://github.com/edolstra/dwarffs). + +* To support flakes, Nix now has an extensible mechanism for fetching + source trees. Currently it has the following backends: + + * Git repositories + + * Mercurial repositories + + * GitHub and GitLab repositories (an optimisation for faster + fetching than Git) + + * Tarballs + + * Arbitrary directories + + The fetcher infrastructure is exposed via flake input specifications + and via the `fetchTree` built-in. + +* **Languages changes**: the only new language feature is that you can + now have antiquotations in paths, e.g. `./${foo}` instead of `./. + + foo`. + +* **New built-in functions**: + + - `builtins.fetchTree` allows fetching a source tree using any + backends supported by the fetcher infrastructure. It subsumes the + functionality of existing built-ins like `fetchGit`, + `fetchMercurial` and `fetchTarball`. + + - `builtins.getFlake` fetches a flake and returns its output + attributes. This function should not be used inside flakes! Use + flake inputs instead. + + - `builtins.floor` and `builtins.ceil` round a floating-point number + down and up, respectively. + +* Experimental support for recursive Nix. This means that Nix + derivations can now call Nix to build other derivations. This is not + in a stable state yet and not well + [documented](https://github.com/NixOS/nix/commit/c4d7c76b641d82b2696fef73ce0ac160043c18da). + +* The new experimental feature `no-url-literals` disables URL + literals. This helps to implement [RFC + 45](https://github.com/NixOS/rfcs/pull/45). + +* Nix now uses `libarchive` to decompress and unpack tarballs and zip + files, so `tar` is no longer required. + +* The priority of substituters can now be overridden using the + `priority` substituter setting (e.g. `--substituters + 'http://cache.nixos.org?priority=100 daemon?priority=10'`). + +* `nix edit` now supports non-derivation attributes, e.g. `nix edit + .#nixosConfigurations.bla`. + +* The `nix` command now provides command line completion for `bash`, + `zsh` and `fish`. Since the support for getting completions is built + into `nix`, it's easy to add support for other shells. + +* The new `--log-format` flag selects what Nix's output looks like. It + defaults to a terse progress indicator. There is a new + `internal-json` output format for use by other programs. + +* `nix eval` has a new `--apply` flag that applies a function to the + evaluation result. + +* `nix eval` has a new `--write-to` flag that allows it to write a + nested attribute set of string leaves to a corresponding directory + tree. + +* Memory improvements: many operations that add paths to the store or + copy paths between stores now run in constant memory. + +* Many `nix` commands now support the flag `--derivation` to operate + on a `.drv` file itself instead of its outputs. + +* There is a new store called `dummy://` that does not support + building or adding paths. This is useful if you want to use the Nix + evaluator but don't have a Nix store. + +* The `ssh-ng://` store now allows substituting paths on the remote, + as `ssh://` already did. + +* When auto-calling a function with an ellipsis, all arguments are now + passed. + +* New `nix-shell` features: + + - It preserves the `PS1` environment variable if + `NIX_SHELL_PRESERVE_PROMPT` is set. + + - With `-p`, it passes any `--arg`s as Nixpkgs arguments. + + - Support for structured attributes. + +* `nix-prefetch-url` has a new `--executable` flag. + +* On `x86_64` systems, [`x86_64` microarchitecture + levels](https://lwn.net/Articles/844831/) are mapped to additional + system types (e.g. `x86_64-v1-linux`). + +* The new `--eval-store` flag allows you to use a different store for + evaluation than for building or storing the build result. This is + primarily useful when you want to query whether something exists in + a read-only store, such as a binary cache: + + ``` + # nix path-info --json --store https://cache.nixos.org \ + --eval-store auto nixpkgs#hello + ``` + + (Here `auto` indicates the local store.) + +* The Nix daemon has a new low-latency mechanism for copying + closures. This is useful when building on remote stores such as + `ssh-ng://`. + +* Plugins can now register `nix` subcommands. + +## Incompatible changes + +* The `nix` command is now marked as an experimental feature. This + means that you need to add + + > experimental-features = nix-command + + to your `nix.conf` if you want to use it, or pass + `--extra-experimental-features nix-command` on the command line. + +* The old `nix run` has been renamed to `nix shell` (and there is a + new `nix run` that does something else, as described above). + +* It is now an error to modify the `plugin-files` setting via a + command-line flag that appears after the first non-flag argument to + any command, including a subcommand to `nix`. For example, + `nix-instantiate default.nix --plugin-files ""` must now become + `nix-instantiate --plugin-files "" default.nix`. + +* We no longer release source tarballs. If you want to build from + source, please build from the tags in the Git repository. + +## Contributors + +This release has contributions from +Adam Höse, +Albert Safin, +Alex Kovar, +Alex Zero, +Alexander Bantyev, +Alexandre Esteves, +Alyssa Ross, +Anatole Lucet, +Anders Kaseorg, +Andreas Rammhold, +Antoine Eiche, +Antoine Martin, +Arnout Engelen, +Arthur Gautier, +aszlig, +Ben Burdette, +Benjamin Hipple, +Bernardo Meurer, +Björn Gohla, +Bjørn Forsman, +Bob van der Linden, +Brian Leung, +Brian McKenna, +Brian Wignall, +Bruce Toll, +Bryan Richter, +Calle Rosenquist, +Calvin Loncaric, +Carlo Nucera, +Carlos D'Agostino, +Chaz Schlarp, +Christian Höppner, +Christian Kampka, +Chua Hou, +Chuck, +Cole Helbling, +Daiderd Jordan, +Dan Callahan, +Dani, +Daniel Fitzpatrick, +Danila Fedorin, +Daniël de Kok, +Danny Bautista, +DavHau, +David McFarland, +Dima, +Domen Kožar, +Dominik Schrempf, +Dominique Martinet, +dramforever, +Dustin DeWeese, +edef, +Eelco Dolstra, +Emilio Karakey, +Emily, +Eric Culp, +Ersin Akinci, +Fabian Möller, +Farid Zakaria, +Federico Pellegrin, +Finn Behrens, +Florian Franzen, +Félix Baylac-Jacqué, +Gabriel Gonzalez, +Geoff Reedy, +Georges Dubus, +Graham Christensen, +Greg Hale, +Greg Price, +Gregor Kleen, +Gregory Hale, +Griffin Smith, +Guillaume Bouchard, +Harald van Dijk, +illustris, +Ivan Zvonimir Horvat, +Jade, +Jake Waksbaum, +jakobrs, +James Ottaway, +Jan Tojnar, +Janne Heß, +Jaroslavas Pocepko, +Jarrett Keifer, +Jeremy Schlatter, +Joachim Breitner, +Joe Hermaszewski, +Joe Pea, +John Ericson, +Jonathan Ringer, +Josef Kemetmüller, +Joseph Lucas, +Jude Taylor, +Julian Stecklina, +Julien Tanguy, +Jörg Thalheim, +Kai Wohlfahrt, +keke, +Keshav Kini, +Kevin Quick, +Kevin Stock, +Kjetil Orbekk, +Krzysztof Gogolewski, +kvtb, +Lars Mühmel, +Leonhard Markert, +Lily Ballard, +Linus Heckemann, +Lorenzo Manacorda, +Lucas Desgouilles, +Lucas Franceschino, +Lucas Hoffmann, +Luke Granger-Brown, +Madeline Haraj, +Marwan Aljubeh, +Mat Marini, +Mateusz Piotrowski, +Matthew Bauer, +Matthew Kenigsberg, +Mauricio Scheffer, +Maximilian Bosch, +Michael Adler, +Michael Bishop, +Michael Fellinger, +Michael Forney, +Michael Reilly, +mlatus, +Mykola Orliuk, +Nathan van Doorn, +Naïm Favier, +ng0, +Nick Van den Broeck, +Nicolas Stig124 Formichella, +Niels Egberts, +Niklas Hambüchen, +Nikola Knezevic, +oxalica, +p01arst0rm, +Pamplemousse, +Patrick Hilhorst, +Paul Opiyo, +Pavol Rusnak, +Peter Kolloch, +Philipp Bartsch, +Philipp Middendorf, +Piotr Szubiakowski, +Profpatsch, +Puck Meerburg, +Ricardo M. Correia, +Rickard Nilsson, +Robert Hensing, +Robin Gloster, +Rodrigo, +Rok Garbas, +Ronnie Ebrin, +Rovanion Luckey, +Ryan Burns, +Ryan Mulligan, +Ryne Everett, +Sam Doshi, +Sam Lidder, +Samir Talwar, +Samuel Dionne-Riel, +Sebastian Ullrich, +Sergei Trofimovich, +Sevan Janiyan, +Shao Cheng, +Shea Levy, +Silvan Mosberger, +Stefan Frijters, +Stefan Jaax, +sternenseemann, +Steven Shaw, +Stéphan Kochen, +SuperSandro2000, +Suraj Barkale, +Taeer Bar-Yam, +Thomas Churchman, +Théophane Hufschmitt, +Timothy DeHerrera, +Timothy Klim, +Tobias Möst, +Tobias Pflug, +Tom Bereknyei, +Travis A. Everett, +Ujjwal Jain, +Vladimír Čunát, +Wil Taylor, +Will Dietz, +Yaroslav Bolyukin, +Yestin L. Harrison, +YI, +Yorick van Pelt, +Yuriy Taraday and +zimbatm. diff --git a/src/libcmd/command.cc b/src/libcmd/command.cc index 0d61e12d0..fd3edfc46 100644 --- a/src/libcmd/command.cc +++ b/src/libcmd/command.cc @@ -203,10 +203,10 @@ void MixProfile::updateProfile(const BuiltPaths & buildables) for (auto & buildable : buildables) { std::visit(overloaded { - [&](BuiltPath::Opaque bo) { + [&](const BuiltPath::Opaque & bo) { result.push_back(bo.path); }, - [&](BuiltPath::Built bfd) { + [&](const BuiltPath::Built & bfd) { for (auto & output : bfd.outputs) { result.push_back(output.second); } diff --git a/src/libcmd/installables.cc b/src/libcmd/installables.cc index 8015cff4d..0f0fcf39e 100644 --- a/src/libcmd/installables.cc +++ b/src/libcmd/installables.cc @@ -697,13 +697,13 @@ std::shared_ptr SourceExprCommand::parseInstallable( BuiltPaths getBuiltPaths(ref evalStore, ref store, const DerivedPaths & hopefullyBuiltPaths) { BuiltPaths res; - for (auto & b : hopefullyBuiltPaths) + for (const auto & b : hopefullyBuiltPaths) std::visit( overloaded{ - [&](DerivedPath::Opaque bo) { + [&](const DerivedPath::Opaque & bo) { res.push_back(BuiltPath::Opaque{bo.path}); }, - [&](DerivedPath::Built bfd) { + [&](const DerivedPath::Built & bfd) { OutputPathMap outputs; auto drv = evalStore->readDerivation(bfd.drvPath); auto outputHashes = staticOutputHashes(*evalStore, drv); // FIXME: expensive @@ -823,10 +823,10 @@ StorePathSet toDerivations( { StorePathSet drvPaths; - for (auto & i : installables) - for (auto & b : i->toDerivedPaths()) + for (const auto & i : installables) + for (const auto & b : i->toDerivedPaths()) std::visit(overloaded { - [&](DerivedPath::Opaque bo) { + [&](const DerivedPath::Opaque & bo) { if (!useDeriver) throw Error("argument '%s' did not evaluate to a derivation", i->what()); auto derivers = store->queryValidDerivers(bo.path); @@ -835,7 +835,7 @@ StorePathSet toDerivations( // FIXME: use all derivers? drvPaths.insert(*derivers.begin()); }, - [&](DerivedPath::Built bfd) { + [&](const DerivedPath::Built & bfd) { drvPaths.insert(bfd.drvPath); }, }, b.raw()); diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index fd7a34926..25b9c32b2 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -1178,7 +1178,7 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * * // hash per output. auto hashModulo = hashDerivationModulo(*state.store, Derivation(drv), true); std::visit(overloaded { - [&](Hash h) { + [&](Hash & h) { for (auto & i : outputs) { auto outPath = state.store->makeOutputPath(i, h, drvName); drv.env[i] = state.store->printStorePath(outPath); @@ -1190,11 +1190,11 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * * }); } }, - [&](CaOutputHashes) { + [&](CaOutputHashes &) { // Shouldn't happen as the toplevel derivation is not CA. assert(false); }, - [&](DeferredHash _) { + [&](DeferredHash &) { for (auto & i : outputs) { drv.outputs.insert_or_assign(i, DerivationOutput { diff --git a/src/libstore/build/entry-points.cc b/src/libstore/build/entry-points.cc index 96deb81d1..2b77e4354 100644 --- a/src/libstore/build/entry-points.cc +++ b/src/libstore/build/entry-points.cc @@ -11,12 +11,12 @@ void Store::buildPaths(const std::vector & reqs, BuildMode buildMod Worker worker(*this, evalStore ? *evalStore : *this); Goals goals; - for (auto & br : reqs) { + for (const auto & br : reqs) { std::visit(overloaded { - [&](DerivedPath::Built bfd) { + [&](const DerivedPath::Built & bfd) { goals.insert(worker.makeDerivationGoal(bfd.drvPath, bfd.outputs, buildMode)); }, - [&](DerivedPath::Opaque bo) { + [&](const DerivedPath::Opaque & bo) { goals.insert(worker.makePathSubstitutionGoal(bo.path, buildMode == bmRepair ? Repair : NoRepair)); }, }, br.raw()); diff --git a/src/libstore/build/local-derivation-goal.cc b/src/libstore/build/local-derivation-goal.cc index 3dd3a48b7..f574e64cf 100644 --- a/src/libstore/build/local-derivation-goal.cc +++ b/src/libstore/build/local-derivation-goal.cc @@ -1094,10 +1094,10 @@ void LocalDerivationGoal::writeStructuredAttrs() static StorePath pathPartOfReq(const DerivedPath & req) { return std::visit(overloaded { - [&](DerivedPath::Opaque bo) { + [&](const DerivedPath::Opaque & bo) { return bo.path; }, - [&](DerivedPath::Built bfd) { + [&](const DerivedPath::Built & bfd) { return bfd.drvPath; }, }, req.raw()); @@ -2155,8 +2155,8 @@ void LocalDerivationGoal::registerOutputs() /* Since we'll use the already installed versions of these, we can treat them as leaves and ignore any references they have. */ - [&](AlreadyRegistered _) { return StringSet {}; }, - [&](PerhapsNeedToRegister refs) { + [&](const AlreadyRegistered &) { return StringSet {}; }, + [&](const PerhapsNeedToRegister & refs) { StringSet referencedOutputs; /* FIXME build inverted map up front so no quadratic waste here */ for (auto & r : refs.refs) @@ -2192,11 +2192,11 @@ void LocalDerivationGoal::registerOutputs() }; std::optional referencesOpt = std::visit(overloaded { - [&](AlreadyRegistered skippedFinalPath) -> std::optional { + [&](const AlreadyRegistered & skippedFinalPath) -> std::optional { finish(skippedFinalPath.path); return std::nullopt; }, - [&](PerhapsNeedToRegister r) -> std::optional { + [&](const PerhapsNeedToRegister & r) -> std::optional { return r.refs; }, }, outputReferencesIfUnregistered.at(outputName)); @@ -2262,10 +2262,10 @@ void LocalDerivationGoal::registerOutputs() std::string oldHashPart { scratchPath.hashPart() }; HashModuloSink caSink { outputHash.hashType, oldHashPart }; std::visit(overloaded { - [&](TextHashMethod _) { + [&](const TextHashMethod &) { readFile(actualPath, caSink); }, - [&](FileIngestionMethod m2) { + [&](const FileIngestionMethod & m2) { switch (m2) { case FileIngestionMethod::Recursive: dumpPath(actualPath, caSink); @@ -2312,7 +2312,7 @@ void LocalDerivationGoal::registerOutputs() }; ValidPathInfo newInfo = std::visit(overloaded { - [&](DerivationOutputInputAddressed output) { + [&](const DerivationOutputInputAddressed & output) { /* input-addressed case */ auto requiredFinalPath = output.path; /* Preemptively add rewrite rule for final hash, as that is @@ -2328,7 +2328,7 @@ void LocalDerivationGoal::registerOutputs() static_cast &>(newInfo0) = rewriteRefs(); return newInfo0; }, - [&](DerivationOutputCAFixed dof) { + [&](const DerivationOutputCAFixed & dof) { auto wanted = getContentAddressHash(dof.ca); auto newInfo0 = newInfoFromCA(DerivationOutputCAFloating { diff --git a/src/libstore/content-address.cc b/src/libstore/content-address.cc index 6a695fe68..e8c6b94be 100644 --- a/src/libstore/content-address.cc +++ b/src/libstore/content-address.cc @@ -23,7 +23,7 @@ std::string makeFileIngestionPrefix(FileIngestionMethod m) std::string makeContentAddressingPrefix(ContentAddressMethod m) { return std::visit(overloaded { - [](TextHashMethod _) -> std::string { return "text:"; }, + [](TextHashMethod) -> std::string { return "text:"; }, [](FileIngestionMethod m2) { /* Not prefixed for back compat with things that couldn't produce text before. */ return makeFileIngestionPrefix(m2); @@ -52,11 +52,11 @@ std::string makeFixedOutputCA(FileIngestionMethod method, const Hash & hash) std::string renderContentAddress(ContentAddress ca) { return std::visit(overloaded { - [](TextHash th) { + [](TextHash & th) { return "text:" + th.hash.to_string(Base32, true); }, - [](FixedOutputHash fsh) { + [](FixedOutputHash & fsh) { return "fixed:" + makeFileIngestionPrefix(fsh.method) + fsh.hash.to_string(Base32, true); @@ -128,12 +128,12 @@ ContentAddress parseContentAddress(std::string_view rawCa) { auto hashType = hashType_; // work around clang bug return std::visit(overloaded { - [&](TextHashMethod _) { + [&](TextHashMethod &) { return ContentAddress(TextHash { .hash = Hash::parseNonSRIUnprefixed(rest, hashType) }); }, - [&](FileIngestionMethod fim) { + [&](FileIngestionMethod & fim) { return ContentAddress(FixedOutputHash { .method = fim, .hash = Hash::parseNonSRIUnprefixed(rest, hashType), @@ -185,10 +185,10 @@ ContentAddressWithReferences contentAddressFromMethodHashAndRefs( ContentAddressMethod getContentAddressMethod(const ContentAddressWithReferences & ca) { return std::visit(overloaded { - [](TextInfo th) -> ContentAddressMethod { + [](const TextInfo & th) -> ContentAddressMethod { return TextHashMethod {}; }, - [](FixedOutputInfo fsh) -> ContentAddressMethod { + [](const FixedOutputInfo & fsh) -> ContentAddressMethod { return fsh.method; }, }, ca); @@ -197,10 +197,10 @@ ContentAddressMethod getContentAddressMethod(const ContentAddressWithReferences Hash getContentAddressHash(const ContentAddress & ca) { return std::visit(overloaded { - [](TextHash th) { + [](const TextHash & th) { return th.hash; }, - [](FixedOutputHash fsh) { + [](const FixedOutputHash & fsh) { return fsh.hash; }, }, ca); @@ -208,10 +208,10 @@ Hash getContentAddressHash(const ContentAddress & ca) ContentAddressWithReferences caWithoutRefs(const ContentAddress & ca) { return std::visit(overloaded { - [&](TextHash h) -> ContentAddressWithReferences { + [&](const TextHash & h) -> ContentAddressWithReferences { return TextInfo { h, {}}; }, - [&](FixedOutputHash h) -> ContentAddressWithReferences { + [&](const FixedOutputHash & h) -> ContentAddressWithReferences { return FixedOutputInfo { h, {}}; }, }, ca); @@ -220,10 +220,10 @@ ContentAddressWithReferences caWithoutRefs(const ContentAddress & ca) { Hash getContentAddressHash(const ContentAddressWithReferences & ca) { return std::visit(overloaded { - [](TextInfo th) { + [](const TextInfo & th) { return th.hash; }, - [](FixedOutputInfo fsh) { + [](const FixedOutputInfo & fsh) { return fsh.hash; }, }, ca); diff --git a/src/libstore/daemon.cc b/src/libstore/daemon.cc index 2b8772a8d..74dd11cbd 100644 --- a/src/libstore/daemon.cc +++ b/src/libstore/daemon.cc @@ -396,7 +396,7 @@ static void performOp(TunnelLogger * logger, ref store, FramedSource source(from); // TODO this is essentially RemoteStore::addCAToStore. Move it up to Store. return std::visit(overloaded { - [&](TextHashMethod _) { + [&](TextHashMethod &) { if (hashType != htSHA256) throw UnimplementedError("Only SHA-256 is supported for adding text-hashed data, but '%1' was given", printHashType(hashType)); @@ -405,7 +405,7 @@ static void performOp(TunnelLogger * logger, ref store, auto path = store->addTextToStore(name, contents, refs, repair); return store->queryPathInfo(path); }, - [&](FileIngestionMethod fim) { + [&](FileIngestionMethod & fim) { if (!refs.empty()) throw UnimplementedError("cannot yet have refs with flat or nar-hashed data"); auto path = store->addToStoreFromDump(source, name, fim, hashType, repair); diff --git a/src/libstore/derivations.cc b/src/libstore/derivations.cc index b0aaa0f51..7868c7f94 100644 --- a/src/libstore/derivations.cc +++ b/src/libstore/derivations.cc @@ -11,18 +11,18 @@ namespace nix { std::optional DerivationOutput::path(const Store & store, std::string_view drvName, std::string_view outputName) const { return std::visit(overloaded { - [](DerivationOutputInputAddressed doi) -> std::optional { + [](const DerivationOutputInputAddressed & doi) -> std::optional { return { doi.path }; }, - [&](DerivationOutputCAFixed dof) -> std::optional { + [&](const DerivationOutputCAFixed & dof) -> std::optional { return { dof.path(store, drvName, outputName) }; }, - [](DerivationOutputCAFloating dof) -> std::optional { + [](const DerivationOutputCAFloating & dof) -> std::optional { return std::nullopt; }, - [](DerivationOutputDeferred) -> std::optional { + [](const DerivationOutputDeferred &) -> std::optional { return std::nullopt; }, }, output); @@ -330,22 +330,22 @@ string Derivation::unparse(const Store & store, bool maskOutputs, if (first) first = false; else s += ','; s += '('; printUnquotedString(s, i.first); std::visit(overloaded { - [&](DerivationOutputInputAddressed doi) { + [&](const DerivationOutputInputAddressed & doi) { s += ','; printUnquotedString(s, maskOutputs ? "" : store.printStorePath(doi.path)); s += ','; printUnquotedString(s, ""); s += ','; printUnquotedString(s, ""); }, - [&](DerivationOutputCAFixed dof) { + [&](const DerivationOutputCAFixed & dof) { s += ','; printUnquotedString(s, maskOutputs ? "" : store.printStorePath(dof.path(store, name, i.first))); s += ','; printUnquotedString(s, printMethodAlgo(dof.ca)); s += ','; printUnquotedString(s, getContentAddressHash(dof.ca).to_string(Base16, false)); }, - [&](DerivationOutputCAFloating dof) { + [&](const DerivationOutputCAFloating & dof) { s += ','; printUnquotedString(s, ""); s += ','; printUnquotedString(s, makeContentAddressingPrefix(dof.method) + printHashType(dof.hashType)); s += ','; printUnquotedString(s, ""); }, - [&](DerivationOutputDeferred) { + [&](const DerivationOutputDeferred &) { s += ','; printUnquotedString(s, ""); s += ','; printUnquotedString(s, ""); s += ','; printUnquotedString(s, ""); @@ -418,13 +418,13 @@ DerivationType BasicDerivation::type() const std::optional floatingHashType; for (auto & i : outputs) { std::visit(overloaded { - [&](DerivationOutputInputAddressed _) { + [&](const DerivationOutputInputAddressed &) { inputAddressedOutputs.insert(i.first); }, - [&](DerivationOutputCAFixed _) { + [&](const DerivationOutputCAFixed &) { fixedCAOutputs.insert(i.first); }, - [&](DerivationOutputCAFloating dof) { + [&](const DerivationOutputCAFloating & dof) { floatingCAOutputs.insert(i.first); if (!floatingHashType) { floatingHashType = dof.hashType; @@ -433,7 +433,7 @@ DerivationType BasicDerivation::type() const throw Error("All floating outputs must use the same hash type"); } }, - [&](DerivationOutputDeferred _) { + [&](const DerivationOutputDeferred &) { deferredIAOutputs.insert(i.first); }, }, i.second.output); @@ -536,15 +536,15 @@ DrvHashModulo hashDerivationModulo(Store & store, const Derivation & drv, bool m const auto & res = pathDerivationModulo(store, i.first); std::visit(overloaded { // Regular non-CA derivation, replace derivation - [&](Hash drvHash) { + [&](const Hash & drvHash) { inputs2.insert_or_assign(drvHash.to_string(Base16, false), i.second); }, - [&](DeferredHash deferredHash) { + [&](const DeferredHash & deferredHash) { isDeferred = true; inputs2.insert_or_assign(deferredHash.hash.to_string(Base16, false), i.second); }, // CA derivation's output hashes - [&](CaOutputHashes outputHashes) { + [&](const CaOutputHashes & outputHashes) { std::set justOut = { "out" }; for (auto & output : i.second) { /* Put each one in with a single "out" output.. */ @@ -570,17 +570,17 @@ std::map staticOutputHashes(Store & store, const Derivation & { std::map res; std::visit(overloaded { - [&](Hash drvHash) { + [&](const Hash & drvHash) { for (auto & outputName : drv.outputNames()) { res.insert({outputName, drvHash}); } }, - [&](DeferredHash deferredHash) { + [&](const DeferredHash & deferredHash) { for (auto & outputName : drv.outputNames()) { res.insert({outputName, deferredHash.hash}); } }, - [&](CaOutputHashes outputHashes) { + [&](const CaOutputHashes & outputHashes) { res = outputHashes; }, }, hashDerivationModulo(store, drv, true)); @@ -664,22 +664,22 @@ void writeDerivation(Sink & out, const Store & store, const BasicDerivation & dr for (auto & i : drv.outputs) { out << i.first; std::visit(overloaded { - [&](DerivationOutputInputAddressed doi) { + [&](const DerivationOutputInputAddressed & doi) { out << store.printStorePath(doi.path) << "" << ""; }, - [&](DerivationOutputCAFixed dof) { + [&](const DerivationOutputCAFixed & dof) { out << store.printStorePath(dof.path(store, drv.name, i.first)) << printMethodAlgo(dof.ca) << getContentAddressHash(dof.ca).to_string(Base16, false); }, - [&](DerivationOutputCAFloating dof) { + [&](const DerivationOutputCAFloating & dof) { out << "" << (makeContentAddressingPrefix(dof.method) + printHashType(dof.hashType)) << ""; }, - [&](DerivationOutputDeferred) { + [&](const DerivationOutputDeferred &) { out << "" << "" << ""; diff --git a/src/libstore/derived-path.cc b/src/libstore/derived-path.cc index 8da81d0ac..e55af21e9 100644 --- a/src/libstore/derived-path.cc +++ b/src/libstore/derived-path.cc @@ -24,8 +24,8 @@ StorePathSet BuiltPath::outPaths() const { return std::visit( overloaded{ - [](BuiltPath::Opaque p) { return StorePathSet{p.path}; }, - [](BuiltPath::Built b) { + [](const BuiltPath::Opaque & p) { return StorePathSet{p.path}; }, + [](const BuiltPath::Built & b) { StorePathSet res; for (auto & [_, path] : b.outputs) res.insert(path); @@ -94,8 +94,8 @@ RealisedPath::Set BuiltPath::toRealisedPaths(Store & store) const RealisedPath::Set res; std::visit( overloaded{ - [&](BuiltPath::Opaque p) { res.insert(p.path); }, - [&](BuiltPath::Built p) { + [&](const BuiltPath::Opaque & p) { res.insert(p.path); }, + [&](const BuiltPath::Built & p) { auto drvHashes = staticOutputHashes(store, store.readDerivation(p.drvPath)); for (auto& [outputName, outputPath] : p.outputs) { diff --git a/src/libstore/legacy-ssh-store.cc b/src/libstore/legacy-ssh-store.cc index 571719043..770827b85 100644 --- a/src/libstore/legacy-ssh-store.cc +++ b/src/libstore/legacy-ssh-store.cc @@ -247,6 +247,10 @@ private: conn.to << settings.buildRepeat << settings.enforceDeterminism; + + if (GET_PROTOCOL_MINOR(conn.remoteVersion) >= 7) { + conn.to << ((int) settings.keepFailed); + } } public: @@ -289,10 +293,10 @@ public: for (auto & p : drvPaths) { auto sOrDrvPath = StorePathWithOutputs::tryFromDerivedPath(p); std::visit(overloaded { - [&](StorePathWithOutputs s) { + [&](const StorePathWithOutputs & s) { ss.push_back(s.to_string(*this)); }, - [&](StorePath drvPath) { + [&](const StorePath & drvPath) { throw Error("wanted to fetch '%s' but the legacy ssh protocol doesn't support merely substituting drv files via the build paths command. It would build them instead. Try using ssh-ng://", printStorePath(drvPath)); }, }, sOrDrvPath); diff --git a/src/libstore/local-store.cc b/src/libstore/local-store.cc index 3d6de09d3..8fcedd2fc 100644 --- a/src/libstore/local-store.cc +++ b/src/libstore/local-store.cc @@ -681,7 +681,7 @@ void LocalStore::checkDerivationOutputs(const StorePath & drvPath, const Derivat std::optional h; for (auto & i : drv.outputs) { std::visit(overloaded { - [&](DerivationOutputInputAddressed doia) { + [&](const DerivationOutputInputAddressed & doia) { if (!h) { // somewhat expensive so we do lazily auto temp = hashDerivationModulo(*this, drv, true); @@ -693,14 +693,14 @@ void LocalStore::checkDerivationOutputs(const StorePath & drvPath, const Derivat printStorePath(drvPath), printStorePath(doia.path), printStorePath(recomputed)); envHasRightPath(doia.path, i.first); }, - [&](DerivationOutputCAFixed dof) { + [&](const DerivationOutputCAFixed & dof) { auto path = dof.path(*this, drvName, i.first); envHasRightPath(path, i.first); }, - [&](DerivationOutputCAFloating _) { + [&](const DerivationOutputCAFloating &) { /* Nothing to check */ }, - [&](DerivationOutputDeferred) { + [&](const DerivationOutputDeferred &) { }, }, i.second.output); } diff --git a/src/libstore/misc.cc b/src/libstore/misc.cc index 44809e014..6a98934ef 100644 --- a/src/libstore/misc.cc +++ b/src/libstore/misc.cc @@ -88,12 +88,12 @@ std::optional getDerivationCA(const BasicDerivation & drv) return std::nullopt; if (auto dof = std::get_if(&out->second.output)) { return std::visit(overloaded { - [&](TextInfo ti) -> std::optional { + [&](const TextInfo & ti) -> std::optional { if (!ti.references.empty()) return std::nullopt; return static_cast(ti); }, - [&](FixedOutputInfo fi) -> std::optional { + [&](const FixedOutputInfo & fi) -> std::optional { if (fi.references != PathReferences {}) return std::nullopt; return static_cast(fi); @@ -177,7 +177,7 @@ void Store::queryMissing(const std::vector & targets, } std::visit(overloaded { - [&](DerivedPath::Built bfd) { + [&](const DerivedPath::Built & bfd) { if (!isValidPath(bfd.drvPath)) { // FIXME: we could try to substitute the derivation. auto state(state_.lock()); @@ -210,7 +210,7 @@ void Store::queryMissing(const std::vector & targets, mustBuildDrv(bfd.drvPath, *drv); }, - [&](DerivedPath::Opaque bo) { + [&](const DerivedPath::Opaque & bo) { if (isValidPath(bo.path)) return; diff --git a/src/libstore/path-with-outputs.cc b/src/libstore/path-with-outputs.cc index 865d64cf2..e5a121e00 100644 --- a/src/libstore/path-with-outputs.cc +++ b/src/libstore/path-with-outputs.cc @@ -31,14 +31,14 @@ std::vector toDerivedPaths(const std::vector std::variant StorePathWithOutputs::tryFromDerivedPath(const DerivedPath & p) { return std::visit(overloaded { - [&](DerivedPath::Opaque bo) -> std::variant { + [&](const DerivedPath::Opaque & bo) -> std::variant { if (bo.path.isDerivation()) { // drv path gets interpreted as "build", not "get drv file itself" return bo.path; } return StorePathWithOutputs { bo.path }; }, - [&](DerivedPath::Built bfd) -> std::variant { + [&](const DerivedPath::Built & bfd) -> std::variant { return StorePathWithOutputs { bfd.drvPath, bfd.outputs }; }, }, p.raw()); diff --git a/src/libstore/remote-store.cc b/src/libstore/remote-store.cc index 85f050c6d..0df432898 100644 --- a/src/libstore/remote-store.cc +++ b/src/libstore/remote-store.cc @@ -530,7 +530,7 @@ ref RemoteStore::addCAToStore( if (repair) throw Error("repairing is not supported when building through the Nix daemon protocol < 1.25"); std::visit(overloaded { - [&](TextHashMethod thm) -> void { + [&](const TextHashMethod & thm) -> void { if (hashType != htSHA256) throw UnimplementedError("Only SHA-256 is supported for adding text-hashed data, but '%1' was given", printHashType(hashType)); @@ -539,7 +539,7 @@ ref RemoteStore::addCAToStore( worker_proto::write(*this, conn->to, references); conn.processStderr(); }, - [&](FileIngestionMethod fim) -> void { + [&](const FileIngestionMethod & fim) -> void { conn->to << wopAddToStore << name @@ -710,10 +710,10 @@ static void writeDerivedPaths(RemoteStore & store, ConnectionHandle & conn, cons for (auto & p : reqs) { auto sOrDrvPath = StorePathWithOutputs::tryFromDerivedPath(p); std::visit(overloaded { - [&](StorePathWithOutputs s) { + [&](const StorePathWithOutputs & s) { ss.push_back(s.to_string(store)); }, - [&](StorePath drvPath) { + [&](const StorePath & drvPath) { throw Error("trying to request '%s', but daemon protocol %d.%d is too old (< 1.29) to request a derivation file", store.printStorePath(drvPath), GET_PROTOCOL_MAJOR(conn->daemonVersion), diff --git a/src/libstore/serve-protocol.hh b/src/libstore/serve-protocol.hh index 02d0810cc..3f76baa82 100644 --- a/src/libstore/serve-protocol.hh +++ b/src/libstore/serve-protocol.hh @@ -5,7 +5,7 @@ namespace nix { #define SERVE_MAGIC_1 0x390c9deb #define SERVE_MAGIC_2 0x5452eecb -#define SERVE_PROTOCOL_VERSION (2 << 8 | 6) +#define SERVE_PROTOCOL_VERSION (2 << 8 | 7) #define GET_PROTOCOL_MAJOR(x) ((x) & 0xff00) #define GET_PROTOCOL_MINOR(x) ((x) & 0x00ff) diff --git a/src/libstore/store-api.cc b/src/libstore/store-api.cc index 3cb3356f3..a57eab3e6 100644 --- a/src/libstore/store-api.cc +++ b/src/libstore/store-api.cc @@ -208,10 +208,10 @@ StorePath Store::makeFixedOutputPathFromCA(const StorePathDescriptor & desc) con { // New template return std::visit(overloaded { - [&](TextInfo ti) { + [&](const TextInfo & ti) { return makeTextPath(desc.name, ti); }, - [&](FixedOutputInfo foi) { + [&](const FixedOutputInfo & foi) { return makeFixedOutputPath(desc.name, foi); } }, desc.info); @@ -1150,13 +1150,13 @@ std::optional ValidPathInfo::fullStorePathDescriptorOpt() c return StorePathDescriptor { .name = std::string { path.name() }, .info = std::visit(overloaded { - [&](TextHash th) { + [&](const TextHash & th) { TextInfo info { th }; assert(!hasSelfReference); info.references = references; return ContentAddressWithReferences { info }; }, - [&](FixedOutputHash foh) { + [&](const FixedOutputHash & foh) { FixedOutputInfo info { foh }; info.references = static_cast>(*this); return ContentAddressWithReferences { info }; @@ -1218,11 +1218,11 @@ ValidPathInfo::ValidPathInfo( , narHash(narHash) { std::visit(overloaded { - [this](TextInfo ti) { + [this](const TextInfo & ti) { this->references = ti.references; this->ca = TextHash { std::move(ti) }; }, - [this](FixedOutputInfo foi) { + [this](const FixedOutputInfo & foi) { *(static_cast *>(this)) = foi.references; this->ca = FixedOutputHash { (FixedOutputHash) std::move(foi) }; }, diff --git a/src/nix-store/nix-store.cc b/src/nix-store/nix-store.cc index 7947e65fb..24a488d43 100644 --- a/src/nix-store/nix-store.cc +++ b/src/nix-store/nix-store.cc @@ -807,6 +807,9 @@ static void opServe(Strings opFlags, Strings opArgs) settings.enforceDeterminism = readInt(in); settings.runDiffHook = true; } + if (GET_PROTOCOL_MINOR(clientVersion) >= 7) { + settings.keepFailed = (bool) readInt(in); + } settings.printRepeatedBuilds = false; }; diff --git a/src/nix/build.cc b/src/nix/build.cc index ce6df7df8..6e31757a2 100644 --- a/src/nix/build.cc +++ b/src/nix/build.cc @@ -66,12 +66,12 @@ struct CmdBuild : InstallablesCommand, MixDryRun, MixJSON, MixProfile for (const auto & [_i, buildable] : enumerate(buildables)) { auto i = _i; std::visit(overloaded { - [&](BuiltPath::Opaque bo) { + [&](const BuiltPath::Opaque & bo) { std::string symlink = outLink; if (i) symlink += fmt("-%d", i); store2->addPermRoot(bo.path, absPath(symlink)); }, - [&](BuiltPath::Built bfd) { + [&](const BuiltPath::Built & bfd) { for (auto & output : bfd.outputs) { std::string symlink = outLink; if (i) symlink += fmt("-%d", i); diff --git a/src/nix/log.cc b/src/nix/log.cc index 962c47525..fd3c1d787 100644 --- a/src/nix/log.cc +++ b/src/nix/log.cc @@ -35,10 +35,10 @@ struct CmdLog : InstallableCommand RunPager pager; for (auto & sub : subs) { auto log = std::visit(overloaded { - [&](DerivedPath::Opaque bo) { + [&](const DerivedPath::Opaque & bo) { return sub->getBuildLog(bo.path); }, - [&](DerivedPath::Built bfd) { + [&](const DerivedPath::Built & bfd) { return sub->getBuildLog(bfd.drvPath); }, }, b.raw()); diff --git a/src/nix/profile.cc b/src/nix/profile.cc index bd5042d8f..916966997 100644 --- a/src/nix/profile.cc +++ b/src/nix/profile.cc @@ -269,11 +269,11 @@ struct CmdProfileInstall : InstallablesCommand, MixDefaultProfile ProfileElement element; std::visit(overloaded { - [&](BuiltPath::Opaque bo) { + [&](const BuiltPath::Opaque & bo) { pathsToBuild.push_back(bo); element.storePaths.insert(bo.path); }, - [&](BuiltPath::Built bfd) { + [&](const BuiltPath::Built & bfd) { // TODO: Why are we querying if we know the output // names already? Is it just to figure out what the // default one is? diff --git a/src/nix/show-derivation.cc b/src/nix/show-derivation.cc index d13960280..f07fedaf0 100644 --- a/src/nix/show-derivation.cc +++ b/src/nix/show-derivation.cc @@ -65,19 +65,19 @@ struct CmdShowDerivation : InstallablesCommand auto & outputName = _outputName; // work around clang bug auto outputObj { outputsObj.object(outputName) }; std::visit(overloaded { - [&](DerivationOutputInputAddressed doi) { + [&](const DerivationOutputInputAddressed & doi) { outputObj.attr("path", store->printStorePath(doi.path)); }, - [&](DerivationOutputCAFixed dof) { + [&](const DerivationOutputCAFixed & dof) { outputObj.attr("path", store->printStorePath(dof.path(*store, drv.name, outputName))); outputObj.attr("hashAlgo", printMethodAlgo(dof.ca)); outputObj.attr("hash", getContentAddressHash(dof.ca).to_string(Base16, false)); // FIXME print refs? }, - [&](DerivationOutputCAFloating dof) { + [&](const DerivationOutputCAFloating & dof) { outputObj.attr("hashAlgo", makeContentAddressingPrefix(dof.method) + printHashType(dof.hashType)); }, - [&](DerivationOutputDeferred) {}, + [&](const DerivationOutputDeferred &) {}, }, output.output); } } diff --git a/tests/build-remote.sh b/tests/build-remote.sh index 27d85a83d..806c6d261 100644 --- a/tests/build-remote.sh +++ b/tests/build-remote.sh @@ -53,3 +53,16 @@ nix path-info --store $TEST_ROOT/machine3 --all \ | grep -v builder-build-remote-input-1.sh \ | grep -v builder-build-remote-input-2.sh \ | grep builder-build-remote-input-3.sh + +# Behavior of keep-failed +out="$(nix-build 2>&1 failing.nix \ + --builders "$(join_by '; ' "${builders[@]}")" \ + --keep-failed \ + --store $TEST_ROOT/machine0 \ + -j0 \ + --arg busybox $busybox)" || true + +[[ "$out" =~ .*"note: keeping build directory".* ]] + +build_dir="$(grep "note: keeping build" <<< "$out" | sed -E "s/^(.*)note: keeping build directory '(.*)'(.*)$/\2/")" +[[ "foo" = $(<"$build_dir"/bar) ]] diff --git a/tests/failing.nix b/tests/failing.nix new file mode 100644 index 000000000..2a0350d4d --- /dev/null +++ b/tests/failing.nix @@ -0,0 +1,22 @@ +{ busybox }: +with import ./config.nix; +let + + mkDerivation = args: + derivation ({ + inherit system; + builder = busybox; + args = ["sh" "-e" args.builder or (builtins.toFile "builder-${args.name}.sh" "if [ -e .attrs.sh ]; then source .attrs.sh; fi; eval \"$buildCommand\"")]; + } // removeAttrs args ["builder" "meta"]) + // { meta = args.meta or {}; }; +in +{ + + failing = mkDerivation { + name = "failing"; + buildCommand = '' + echo foo > bar + exit 1 + ''; + }; +}