Commit graph

73 commits

Author SHA1 Message Date
Eelco Dolstra
4ed66735b6 RunOptions: Use designated initializers
Also get rid of _killStderr because it wasn't actually checked
anywhere.
2021-09-13 23:31:04 +02:00
regnat
497225b07d Don’t create lockfiles with an invalid path name
Store paths are only allowed to contain a limited subset of the
alphabet, which doesn’t include `!`. So don’t create lockfiles that
contain this `!` character as that would otherwise confuse (and break)
the gc.

Fix #5176
2021-09-02 09:57:41 +02:00
Eelco Dolstra
a7b7fcfb16 Remove redundant RealisedPath::closure() call 2021-07-22 22:43:18 +02:00
Eelco Dolstra
eb6db4fd38 buildPaths(): Add an evalStore argument
With this, we don't have to copy the entire .drv closure to the
destination store ahead of time (or at all). Instead, buildPaths()
reads .drv files from the eval store and copies inputSrcs to the
destination store if it needs to build a derivation.

Issue #5025.
2021-07-22 09:59:51 +02:00
regnat
8707773965 Properly lock the builds of CA derivations
Make sure that we can’t build the same derivation twice at the same
time.

Fix https://github.com/NixOS/nix/issues/5029
2021-07-20 06:57:56 +02:00
regnat
9b1f3cbc13 Forward the whole Nix config to the post-build-hook
Fill `NIX_CONFIG` with the value of the current Nix configuration before
calling the post-build-hook.
That way the whole configuration (including the possible
`experimental-features`, a possibly `--store` option or whatever) will
be made available to the hook
2021-07-15 18:41:56 +02:00
Maximilian Bosch
04cd2da84c
Merge branch 'master' into structured-attrs-shell
Conflicts:
        src/nix/develop.cc
        src/nix/get-env.sh
        tests/shell.nix
2021-07-12 15:49:39 +02:00
regnat
be7a4a6a13 Make the post-build-hook also run for unresolved CA derivations
Fix #4837
2021-06-24 11:41:57 +02:00
Maximilian Bosch
3b5429aec1
Source complete env in nix-shell with __structuredAttrs = true;
This is needed to push the adoption of structured attrs[1] forward. It's
now checked if a `__json` exists in the environment-map of the derivation
to be openend in a `nix-shell`.

Derivations with structured attributes enabled also make use of a file
named `.attrs.json` containing every environment variable represented as
JSON which is useful for e.g. `exportReferencesGraph`[2]. To
provide an environment similar to the build sandbox, `nix-shell` now
adds a `.attrs.json` to `cwd` (which is mostly equal to the one in the
build sandbox) and removes it using an exit hook when closing the shell.

To avoid leaking internals of the build-process to the `nix-shell`, the
entire logic to generate JSON and shell code for structured attrs was
moved into the `ParsedDerivation` class.

[1] https://nixos.mayflower.consulting/blog/2020/01/20/structured-attrs/
[2] https://nixos.org/manual/nix/unstable/expressions/advanced-attributes.html#advanced-attributes
2021-06-22 19:15:57 +02:00
regnat
1f3ff0d193 Aso track the output path of the realisation dependencies 2021-05-26 17:09:21 +02:00
regnat
8c30acc3e8 Properly track the drvoutput references when building 2021-05-26 16:59:09 +02:00
regnat
9161e02039 Always register the realisations of input-addressed drvs
Fix #4725
2021-04-22 20:07:02 +02:00
Alyssa Ross
9ac6534f7c
Include sys/wait.h everywhere WIFEXITED etc is used
This is required on NetBSD, and I think FreeBSD too.
2021-04-19 18:31:58 +00:00
John Ericson
9b805d36ac Rename Buildable 2021-04-05 09:52:25 -04:00
John Ericson
9dfb97c987 "newtype" BuildableReq
This makes for better types errors and allows us to give it methods.
2021-04-05 09:35:55 -04:00
John Ericson
255d145ba7 Use BuildableReq for buildPaths and ensurePath
This avoids an ambiguity where the `StorePathWithOutputs { drvPath, {}
}` could mean "build `brvPath`" or "substitute `drvPath`" depending on
context.

It also brings the internals closer in line to the new CLI, by
generalizing the `Buildable` type is used there and makes that
distinction already.

In doing so, relegate `StorePathWithOutputs` to being a type just for
backwards compatibility (CLI and RPC).
2021-04-05 08:33:00 -04:00
Eelco Dolstra
ccb8a403ee
Merge pull request #4587 from obsidiansystems/derivation-goal-detect-invalid-output
Throw error for derivation goal with bogus wanted output
2021-03-15 16:49:44 +01:00
regnat
703c98c6cb Properly sign the unresolved drvs
Don't let them inherit the signature from the parent one (because it
makes no sense to do so), but re-sign them after they have been built
2021-03-15 16:35:17 +01:00
John Ericson
7ce10924c7 Fix bad wanted output error as requested
- UsageError -> Error

- include drv path too
2021-03-01 15:07:09 +00:00
regnat
df9d4f88d5 Allow substituting drv outputs when building 2021-03-01 14:00:17 +01:00
regnat
5d1c05b075 SubstitutionGoal -> PathSubstitutionGoal
To prepare for the upcoming DrvOutputSubstitutionGoal
2021-03-01 14:00:17 +01:00
John Ericson
4bbd80c536 Throw error for derivation goal with bogus wanted output 2021-02-28 00:19:35 +00:00
John Ericson
68f4c728ec Split {,local-}derivation-goal.{cc,hh}
This separates the scheduling logic (including simple hook pathway) from
the local-store needing code.

This should be the final split for now. I'm reasonably happy with how
it's turning out, even before I'm done moving code into
`local-derivation-goal`. Benefits:

1. This will help "witness" that the hook case is indeed a lot simpler,
   and also compensate for the increased complexity that comes from
   content-addressed derivation outputs.

2. It also moves us ever so slightly towards a world where we could use
   off-the-shelf storage or sandboxing, since `local-derivation-goal`
   would be gutted in those cases, but `derivation-goal` should remain
   nearly the same.

The new `#if 0` in the new files will be deleted in the following
commit. I keep it here so if it turns out more stuff can be moved over,
it's easy to do so in a way that preserves ordering --- and thus
prevents conflicts.

N.B.
```sh
git diff HEAD^^ --color-moved --find-copies-harder --patience --stat
```
makes nicer output.
2021-02-26 16:10:26 +00:00
Eelco Dolstra
94637cd7e5
Merge pull request #4477 from NixOS/ca/build-remote
Build ca derivations remotely
2021-02-26 16:54:44 +01:00
Eelco Dolstra
076d2b04da
Update src/libstore/build/derivation-goal.cc 2021-02-26 16:30:12 +01:00
Eelco Dolstra
20ea1de77d Use std::make_unique 2021-02-26 12:35:29 +01:00
regnat
ba1a256d08 Make DerivationGoal::drv a full Derivation
This field used to be a `BasicDerivation`, but this `BasicDerivation`
was downcasted to a `Derivation` when needed (implicitely or not), so we
might as well make it a full `Derivation` and upcast it when needed.

This also allows getting rid of a weird duplication in the way we
compute the static output hashes for the derivation. We had to
do it differently and in a different place depending on whether the
derivation was a full derivation or just a basic drv, but we can now do
it unconditionally on the full derivation.

Fix #4559
2021-02-23 14:15:45 +01:00
regnat
527da73690 Properly bypass the registering step when all outputs are present
There was already some logic for that, but it didn't handle the case of
content-addressed outputs, so extend it a bit for that
2021-02-23 08:04:03 +01:00
regnat
8c385d16ee Also send ca outputs to the build hook
Otherwise they don't get registered, triggering an assertion failure
at some point later
2021-02-23 08:04:03 +01:00
regnat
6fbf3fe636 Make the build-hook work with ca derivations
- Pass it the name of the outputs rather than their output paths (as
  these don't exist for ca derivations)
- Get the built output paths from the remote builder
- Register the new received realisations
2021-02-23 08:04:03 +01:00
regnat
4bc28c44f2 Store the output hashes in the initialOutputs of the drv goal
That way we
1. Don't have to recompute them several times
2. Can compute them in a place where we know the type of the parent
  derivation, meaning that we don't need the casting dance we had before
2021-02-19 15:48:31 +01:00
regnat
87c8d3d702 Register the realisations for unresolved drvs
Once a build is done, get back to the original derivation, and register
all the newly built outputs for this derivation.

This allows Nix to work properly with derivations that don't have all
their build inputs available − thus allowing garbage collection and
(once it's implemented) binary substitution
2021-02-19 15:48:31 +01:00
Eelco Dolstra
480426a364 Add more instrumentation for #4270 2021-02-05 15:57:33 +01:00
Eelco Dolstra
0187838e2e Add a trace to readLine() failures
Hopefully this helps to diagnose 'error: unexpected EOF reading a
line' on macOS.
2021-02-05 12:18:11 +01:00
Eelco Dolstra
b19aec7eeb
Merge pull request #4461 from NixOS/ca/error-logging-fixes
Fix some logging with ca derivations
2021-01-29 16:12:50 +01:00
regnat
9da11bac57 Fix the error message when a dep is missing
Fix a mismatch in the errors thrown when a needed output was missing
from an input derivation that was leading to a wrong and quite misleading error
message
2021-01-26 14:49:23 +01:00
Eelco Dolstra
3ba98ba8f0 Tell user to run 'nix log' to get full build logs 2021-01-25 17:15:38 +01:00
Eelco Dolstra
488a826842
Merge pull request #4467 from edolstra/error-formatting
Improve error formatting
2021-01-25 12:50:57 +01:00
John Ericson
53a709535b Apply suggestions from code review
Thanks!

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-01-22 15:58:58 +00:00
Eelco Dolstra
8d4268d190 Improve error formatting
Changes:

* The divider lines are gone. These were in practice a bit confusing,
  in particular with --show-trace or --keep-going, since then there
  were multiple lines, suggesting a start/end which wasn't the case.

* Instead, multi-line error messages are now indented to align with
  the prefix (e.g. "error: ").

* The 'description' field is gone since we weren't really using it.

* 'hint' is renamed to 'msg' since it really wasn't a hint.

* The error is now printed *before* the location info.

* The 'name' field is no longer printed since most of the time it
  wasn't very useful since it was just the name of the exception (like
  EvalError). Ideally in the future this would be a unique, easily
  googleable error ID (like rustc).

* "trace:" is now just "…". This assumes error contexts start with
  something like "while doing X".

Example before:

  error: --- AssertionError ---------------------------------------------------------------------------------------- nix
  at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

       6|
       7|   x = assert false; 1;
        |       ^
       8|

  assertion 'false' failed
  ----------------------------------------------------- show-trace -----------------------------------------------------
  trace: while evaluating the attribute 'x' of the derivation 'hello-2.10'
  at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix

     191|         // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) {
     192|           name = "${attrs.pname}-${attrs.version}";
        |           ^
     193|         } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {

Example after:

  error: assertion 'false' failed

         at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

              6|
              7|   x = assert false; 1;
               |       ^
              8|

         … while evaluating the attribute 'x' of the derivation 'hello-2.10'

         at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix

            191|         // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) {
            192|           name = "${attrs.pname}-${attrs.version}";
               |           ^
            193|         } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {
2021-01-21 11:02:09 +01:00
John Ericson
0027b05a15 Merge remote-tracking branch 'upstream/master' into non-local-store-build 2021-01-15 02:01:24 +00:00
Eelco Dolstra
4e9cec79bf
Merge pull request #4444 from matthewbauer/unset-curproc-arch-affinity
Set kern.curproc_arch_affinity=0 to escape Rosetta
2021-01-13 12:16:53 +01:00
Rickard Nilsson
0ca1a50132 Remove a redundant condition in DerivationGoal::tryLocalBuild() 2021-01-13 10:13:51 +01:00
Matthew Bauer
f69820417f Set kern.curproc_arch_affinity=0 to escape Rosetta
By default, once you enter x86_64 Rosetta 2, macOS will try to run
everything in x86_64. So an x86_64 Nix will still try to use x86_64
even when system = aarch64-darwin. To avoid this we can set
kern.curproc_arch_affinity sysctl. With kern.curproc_arch_affinity=0,
we ignore this preference.

This is based on how
https://opensource.apple.com/source/system_cmds/system_cmds-880.40.5/arch.tproj/arch.c.auto.html
works. Completely undocumented, but seems to work!

Note, you can verify this works with this impure Nix expression:

```
  {
    a = derivation {
      name = "a";
      system = "aarch64-darwin";
      builder = "/bin/sh";
      args = [ "-e" (builtins.toFile "builder" ''
        [ "$(/usr/bin/arch)" = arm64 ]
        [ "$(/usr/bin/arch -arch x86_64 /bin/sh -c /usr/bin/arch)" = i386 ]
        [ "$(/usr/bin/arch -arch arm64 /bin/sh -c /usr/bin/arch)" = arm64 ]
        /usr/bin/touch $out
      '') ];
    };

    b = derivation {
      name = "b";
      system = "x86_64-darwin";
      builder = "/bin/sh";
      args = [ "-e" (builtins.toFile "builder" ''
        [ "$(/usr/bin/arch)" = i386 ]
        [ "$(/usr/bin/arch -arch x86_64 /bin/sh -c /usr/bin/arch)" = i386 ]
        [ "$(/usr/bin/arch -arch arm64 /bin/sh -c /usr/bin/arch)" = arm64 ]
        /usr/bin/touch $out
      '') ];
    };
  }
```
2021-01-11 22:40:21 -06:00
Eelco Dolstra
6548b89cc4 string2Int(): Return std::optional 2021-01-08 12:22:21 +01:00
Eelco Dolstra
3edcb198e5
Merge pull request #4310 from matthewbauer/rosetta2-extra-platforms
Add x86_64-darwin and aarch64 to "extra-platforms" automatically when Rosetta2 is detected
2021-01-06 11:31:13 +01:00
John Ericson
fed1237246 Test nix-build with non-local-store --store
Just a few small things needed fixing!
2020-12-23 22:42:06 +00:00
John Ericson
450c3500f1 Crudely make worker only provide a Store, not LocalStore
We downcast in a few places, this will be refactored to be better later.
2020-12-23 22:42:06 +00:00
John Ericson
1a1af75338 Overhaul store subclassing
We embrace virtual the rest of the way, and get rid of the
`assert(false)` 0-param constructors.

We also list config base classes first, so the constructor order is
always:

  1. all the configs
  2. all the stores

Each in the same order
2020-12-20 15:47:14 +00:00
regnat
e9b39f6004 Restrict the operations on drv outputs in recursive Nix
There's currently no way to properly filter them, so disallow them
altogether instead.
2020-12-11 21:17:25 +01:00