Commit graph

7031 commits

Author SHA1 Message Date
Ben Burdette
e7847ad7a1 map1/2 for stack usage 2021-05-11 15:38:49 -06:00
Ben Burdette
a8fef9a6b1 throwTypeError with debugger/env 2021-05-10 18:36:57 -06:00
Eelco Dolstra
7f9759b18d
Merge pull request #4781 from NixOS/locally_cache_the_remote_realisations
Add a realisations disk cache
2021-05-10 20:37:57 +02:00
Eelco Dolstra
8768398d5f
Merge pull request #4787 from Ma27/builtins-ceil-floor
Implement `builtins.floor` and `builtins.ceil` using the C library functions internally
2021-05-10 20:37:27 +02:00
regnat
d5d19582ef Simplify the realisations disk cache 2021-05-10 17:47:14 +02:00
regnat
ab96c1ee50 Remove useless parents
I never remember the exact syntax of the `switch` statement
2021-05-10 17:36:49 +02:00
Maximilian Bosch
bec33858bb
primops: math.h -> cmath
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-05-10 16:41:10 +02:00
Eelco Dolstra
b68a85df0a
Merge pull request #4776 from NixOS/fix-ca-normalization
Properly normalize the content-addressed paths
2021-05-10 16:11:38 +02:00
Maximilian Bosch
7f7f99f350
Implement builtins.floor and builtins.ceil using the C library functions internally
Closes #4782

Note: even though the type is internally called `NixFloat`, it's
actually a `double`.
2021-05-10 12:19:32 +02:00
Eelco Dolstra
2bb49fb6d0
Merge pull request #4786 from matthewbauer/mkdir-local-share-nix
Create parent for trusted list path before writing
2021-05-10 11:14:45 +02:00
Matthew Bauer
2f63cc02de Create parent trusted list path before writing
This makes sure that $HOME/.local/share/nix exists before we try to
write to it.
2021-05-08 21:31:28 -05:00
regnat
b66234134f Add a realisations disk cache
Similar to the nar-info disk cache (and using the same db).
This makes rebuilds muuch faster.

- This works regardless of the ca-derivations experimental feature.
  I could modify the logic to not touch the db if the flag isn’t there,
  but given that this is a trash-able local cache, it doesn’t seem to be
  really worth it.
- We could unify the `NARs` and `Realisation` tables to only have one
  generic kv table. This is left as an exercise to the reader.
- I didn’t update the cache db version number as the new schema just
  adds a new table to the previous one, so the db will be transparently
  migrated and is backwards-compatible.

Fix #4746
2021-05-06 17:38:20 +02:00
Matthew Bauer
1e3a9033b7 Use derivation output name from toDerivation
This was previously done in https://github.com/NixOS/nix/pull/4515 but
got clobbered away in https://github.com/NixOS/nix/pull/4594.

--------------------------------------------------------------------------------

This fixes an issue where derivations with a primary output that is
not "out" would fail with:

$ nix profile install nixpkgs#sqlite
error: opening directory '/nix/store/2a2ydlgyydly5czcc8lg12n6qqkfz863-sqlite-3.34.1-bin': No such file or directory

This happens because while derivations produce every output when
built, you might not have them if you didn't build the derivation
yourself (for instance, the store path was fetch from a binary cache).
This uses outputName provided from DerivationInfo which appears to
match the first output of the derivation.
2021-05-05 16:33:05 -05:00
regnat
bf485dcf46 Properly normalize the content-addressed paths
Make sure that their timestamp are always normalized.
Otherwise, strange − and non-deterministic − things might happen, like
https://github.com/NixOS/nixpkgs/issues/121813

Fix #4775
2021-05-05 21:00:08 +02:00
regnat
dadfbce318 Fix the double-slash in the realisations path
Make sure that we always access the realisations under
`binaryCacheUrl/realisations` and not `binaryCacheUrl//realisations`

Fix #4766
2021-05-04 10:35:34 +02:00
Ben Burdette
2dd61411af debugger on autoCallFunction error 2021-05-03 14:37:33 -06:00
Eelco Dolstra
ef13c9c223
Merge pull request #4750 from NinjaTrappeur/nin-build-dry-run-json
nix build: make dry-run to print a json output if --json is enabled
2021-05-03 16:14:29 +02:00
Eelco Dolstra
40378fbcba
Merge pull request #4761 from emilazy/issue-4658-mark-impure-host-deps-as-optional
Mark `__impureHostDeps` paths as optional
2021-05-03 16:13:36 +02:00
Maximilian Bosch
e5951a6b2f
Bump version number for DerivedPath changes
I guess I misunderstood John's initial explanation about why wildcards
for outputs are sent to older stores[1]. My `nix-daemon` from 2021-03-26
also has version 1.29, but misses the wildcard[2]. So bumping seems to
be the right call.

[1] https://github.com/NixOS/nix/pull/4759#issuecomment-830812464
[2] 255d145ba7
2021-05-03 01:12:23 +02:00
Emily
c4355a52fa Mark __impureHostDeps paths as optional
Starting in macOS 11, the on-disk dylib bundles are no longer available,
but nixpkgs needs to be able to keep compatibility with older versions
that require `/usr/lib/libSystem.B.dylib` in `__impureHostDeps`. Allow
it to keep backwards compatibility with these versions by marking these
dependencies as optional.

Fixes #4658.
2021-05-02 05:30:50 +01:00
Ben Burdette
f32c687f03 move repl.cc to libcmd for linkage 2021-04-28 15:50:11 -06:00
Ben Burdette
57c2dd5d85 fixes 2021-04-28 09:55:08 -06:00
Ben Burdette
b466d51952 Merge branch 'master' into debugger-merge 2021-04-27 12:31:01 -06:00
Domen Kožar
e3e78ee2a2
Merge pull request #4751 from Ma27/storepath-pos
primops/storePath: add trace to pure mode error
2021-04-27 19:05:23 +02:00
Maximilian Bosch
e6a360a4d9
primops/storePath: add trace to pure mode error
As described in #4745 it's otherwise fairly hard to understand where
this is coming from. Say you have an expression which uses e.g.
`types.package`:

``` nix
{ outputs = { self, nixpkgs }: {
    packages.x86_64-linux.hello = let
      foo = nixpkgs.lib.evalModules {
        modules = [
          {
            options.foo.bar = with nixpkgs.lib; mkOption { type = types.package; };
          }
          {
            foo.bar = ./.;
          }
        ];
      };
    in builtins.trace foo.config.foo.bar.outPath nixpkgs.legacyPackages.x86_64-linux.hello;
    defaultPackage.x86_64-linux = self.packages.x86_64-linux.hello;
  };
}
```

Then you'll get an error trace like this:

```
error: 'builtins.storePath' is not allowed in pure evaluation mode

       at /nix/store/p4h2x6r80njkb0j2rc1xjhhl99yri3zb-source/lib/attrsets.nix:328:15:

          327|     let
          328|       path' = builtins.storePath path;
             |               ^
          329|       res =

       … while evaluating the attribute 'config.foo.bar.outPath'

       at /nix/store/p4h2x6r80njkb0j2rc1xjhhl99yri3zb-source/lib/attrsets.nix:332:11:

          331|           name = sanitizeDerivationName (builtins.substring 33 (-1) (baseNameOf path'));
          332|           outPath = path';
             |           ^
          333|           outputs = [ "out" ];

       … while evaluating the attribute 'packages.x86_64-linux.hello'

       at /nix/store/6c1rfsqzrhjw1235palzjmf5vihcpci7-source/flake.nix:3:5:

            2| { outputs = { self, nixpkgs }: {
            3|     packages.x86_64-linux.hello = let
             |     ^
            4|       foo = nixpkgs.lib.evalModules {
```

Fixes #4745
2021-04-27 17:24:38 +02:00
Félix Baylac-Jacqué
7dca9c24c4
nix build: make dry-run to print a json output if --json is enabled 2021-04-27 11:28:47 +02:00
Eelco Dolstra
fe2bf464cf Merge branch 'remove-trailing-spaces' of github.com:NixOS/nix 2021-04-23 15:09:44 +02:00
regnat
31313d1401 Replace the trailing markdown spaces by a backslash
They are equivalent according to
<https://spec.commonmark.org/0.29/#hard-line-breaks>,
and the trailing spaces tend to be a pain (because the make git
complain, editors tend to want to remove them − the `.editorconfig`
actually specifies that − etc..).
2021-04-23 14:37:21 +02:00
Eelco Dolstra
4549d65b4f
Merge pull request #4732 from NixOS/4725-always-register-the-realisations
Aways register the realisations
2021-04-23 14:22:03 +02:00
Eelco Dolstra
293220bed5
Merge pull request #4440 from Ma27/misc-pos-fixes
Miscellaneous improvements for positioning in eval-errors
2021-04-23 11:10:59 +02:00
regnat
d3df747cb6 Revert "Make nix shell fallback to static outputs when needed"
This reverts commit 8d66f5f110.

This fix isn't needed anymore now that the realisations are always
properly registered
2021-04-23 09:34:43 +02:00
p01arst0rm
45473d02c9 unified macro style for ENABLE_S3 2021-04-23 07:30:05 +01:00
regnat
9161e02039 Always register the realisations of input-addressed drvs
Fix #4725
2021-04-22 20:07:02 +02:00
Eelco Dolstra
d9864be4b7 Merge branch 'master' of github.com:NixOS/nix 2021-04-22 10:33:00 +02:00
Eelco Dolstra
78a2303b3d Merge branch 'nix-derivation-query-store-path' of https://github.com/matthewbauer/nix 2021-04-22 10:29:18 +02:00
Eelco Dolstra
dd83b29b90
Merge pull request #4724 from NixOS/nix-shell-missing-outputs
Make `nix shell` fallback to static outputs when needed
2021-04-22 10:25:55 +02:00
Eelco Dolstra
5385755227 Remove makeDecompressionSource()
This function doesn't support all compression methods (i.e. 'none' and
'br') so it shouldn't be exposed.

Also restore the original decompress() as a wrapper around
makeDecompressionSink().
2021-04-22 10:23:20 +02:00
Eelco Dolstra
a6eebcff36 Merge branch 's3-decompress' of https://github.com/lukegb/nix 2021-04-22 10:20:40 +02:00
Luke Granger-Brown
97dde3cdd9 libutil: allow decompression with none/empty method
The S3 store relies on the ability to be able to decompress things with
an empty method, because it just passes the value of the Content-Encoding
directly to decompress.

If the file is not compressed, then this will cause the compression
routine to get confused.

This caused NixOS/nixpkgs#120120.
2021-04-22 02:33:59 +00:00
Antoine Martin
ac38c4cdc2 Fix typo in flake doc 2021-04-22 03:07:57 +02:00
Matthew Bauer
428062e578 Get deriver when passing --derivation
This makes Nix look up paths derivations when they are passed as a
store paths. So:

$ nix path-info --derivation /nix/store/0pisd259nldh8yfjvw663mspm60cn5v8-hello-2.10

now gives

/nix/store/qp724w90516n4bk5r9gfb37vzmqdh3z7-hello-2.10.drv

instead of "".

If no deriver is available, Nix now errors instead of silently
ignoring that argument.
2021-04-21 15:32:33 -05:00
regnat
8d66f5f110 Make nix shell fallback to static outputs when needed
In case of pure input-addressed derivations, the build loop doesn't
guaranty that the realisations are stored in the db (if the output paths
are already there or can be substituted, the realisations won't be
registered). This caused `nix shell` to fail in some cases because it
was assuming that the realisations were always existing.

A better (but more involved) fix would probably to ensure that we always
register the realisations, but in the meantime this patches the surface
issue.

Fix #4721
2021-04-21 17:46:27 +02:00
Eelco Dolstra
8d651a1f68 Fix brotli compression of files > 128 KiB
This has been broken since faa31f4084.
2021-04-20 22:54:49 +02:00
Eelco Dolstra
b60b0d62d6
Merge pull request #4720 from alyssais/wait
Include sys/wait.h everywhere WIFEXITED etc is used
2021-04-20 16:09:30 +02:00
Alyssa Ross
9ac6534f7c
Include sys/wait.h everywhere WIFEXITED etc is used
This is required on NetBSD, and I think FreeBSD too.
2021-04-19 18:31:58 +00:00
Maximilian Bosch
6905291daf
libcmd/installables: force re-evaluation of cached failures
I think that it's not very helpful to get "cached failures" in a wrong
`flake.nix`. This can be very confusing when debugging a Nix expression.
See for instance NixOS/nixpkgs#118115.

In fact, the eval cache allows a forced reevaluation which is used for
e.g. `nix eval`.

This change makes sure that this is the case for `nix build` as well. So
rather than

    λ ma27 [~/Projects/exp] → ../nix/outputs/out/bin/nix build -L --rebuild --experimental-features 'nix-command flakes'
    error: cached failure of attribute 'defaultPackage.x86_64-linux'

the evaluation of already-evaluated (and failed) attributes looks like
this now:

    λ ma27 [~/Projects/exp] → ../nix/outputs/out/bin/nix build -L --rebuild --experimental-features 'nix-command flakes'
    error: attribute 'hell' missing

           at /nix/store/mrnvi9ss8zn5wj6gpn4bcd68vbh42mfh-source/flake.nix:6:35:

                5|
                6|     packages.x86_64-linux.hello = nixpkgs.legacyPackages.x86_64-linux.hell;
                 |                                   ^
                7|
    (use '--show-trace' to show detailed location information)
2021-04-19 19:51:04 +02:00
Eelco Dolstra
3ee0ecdda0
Merge pull request #4679 from ony/feature/one-pass-canon-path
Optimize canonPath to resolve relative symlinks in one pass
2021-04-15 14:11:51 +02:00
Eelco Dolstra
15f4d4fd43 Drop libbz2 / zlib / lzma dependency + style fixes 2021-04-15 13:55:22 +02:00
Eelco Dolstra
6fb7582413 Merge branch 'libarchive-decompress' of https://github.com/serokell/nix 2021-04-15 13:39:04 +02:00
Eelco Dolstra
f716779c1f Merge branch 'revertRemoveJSON' of https://github.com/cgohla/nix 2021-04-15 13:26:56 +02:00
Eelco Dolstra
709a60a045
Merge pull request #4707 from grahamc/defaul-path-restricted-eval
EvalSettings::getDefaultNixPath: respect {restrict,pure}Eval
2021-04-14 11:02:58 +02:00
Eelco Dolstra
b47b5f4061
Merge pull request #4708 from NixOS/fix-remote-registerDrvOutput
Fix registerDrvOutput with the daemon
2021-04-14 11:01:36 +02:00
Maximilian Bosch
864ef0e93d
libexpr/primops: review 2021-04-13 23:12:39 +02:00
Maximilian Bosch
ad7b47dc85
primops/libexpr: use new attr-call extractor everywhere; use function's pos if attr-set pos == noPos 2021-04-13 23:12:38 +02:00
Maximilian Bosch
f60473a077
libexpr/primops: Move attr name extraction into its own function
This now takes care of providing positioning for both the faulting value
and the faulting function call in case of an error.
2021-04-13 23:12:38 +02:00
Maximilian Bosch
7c76964daa
libexpr: misc improvements for proper error position
When working on some more complex Nix code, there are sometimes rather
unhelpful or misleading error messages, especially if coerce-errors are
thrown.

This patch is a first steps towards improving that. I'm happy to file
more changes after that, but I'd like to gather some feedback first.

To summarize, this patch does the following things:

* Attrsets (a.k.a. `Bindings` in `libexpr`) now have a `Pos`. This is
  helpful e.g. to identify which attribute-set in `listToAttrs` is
  invalid.

* The `Value`-struct has a new method named `determinePos` which tries
  to guess the position of a value and falls back to a default if that's
  not possible.

  This can be used to provide better messages if a coercion fails.

* The new `determinePos`-API is used by `builtins.concatMap` now. With
  that change, Nix shows the exact position in the error where a wrong
  value was returned by the lambda.

  To make sure it's still obvious that `concatMap` is the problem,
  another stack-frame was added.

* The changes described above can be added to every other `primop`, but
  first I'd like to get some feedback about the overall approach.
2021-04-13 23:12:38 +02:00
Maximilian Bosch
3550a32b25
primops/derivation: use position of currently evaluated attribute
* The position of the `name`-attribute appears in the trace.
* If e.g. `meta` has no `outPath`-attribute, a `cannot coerce set to
  string` error will be thrown where `pos` points to `name =` which is
  highly misleading.
2021-04-13 23:12:38 +02:00
regnat
ed29610cc6 Fix registerDrvOutput with the daemon
Resolve a protocol issue that caused the daemon to endlessly wait for
some information that the client doesn't ever send
2021-04-13 13:58:22 +02:00
Graham Christensen
d04969ffa5 EvalSettings::getDefaultNixPath: respect {restrict,pure}Eval
Otherwise Nix may look to invalid locations for channels.
2021-04-12 12:37:15 -04:00
Domen Kožar
c2dfda007e
add tests for zstd compression 2021-04-09 23:13:08 +02:00
Eelco Dolstra
42f0246698
Revert "libfetchers/tarball: Lock on effectiveUrl"
This reverts commit fc6bfb261d.

Fixes #4672.
2021-04-09 13:51:57 +02:00
Eelco Dolstra
906adadacd Restore stack size in child processes
Fixes #4673.
2021-04-07 13:40:13 +02:00
Eelco Dolstra
9b9e703df4 restoreSignals() + restoreAffinity() -> restoreProcessContext() 2021-04-07 13:10:02 +02:00
Eelco Dolstra
8a29052cb2 PathSubstitutionGoal: Clean up pipe
If there were many top-level goals (which are not destroyed until the
very end), commands like

  $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' \
    /run/current-system --no-check-sigs --substitute-on-destination

could fail with "Too many open files". So now we do some explicit
cleanup from amDone(). It would be cleaner to separate goals from
their temporary internal state, but that would be a bigger refactor.
2021-04-07 12:21:31 +02:00
Björn Gohla
9f28dd97ae Revert "Use upstream nlohmann_json"
This reverts commit 4145cd2da0.
2021-04-05 21:24:55 +01:00
John Ericson
125a824228 Document the derived path types. 2021-04-05 10:56:48 -04:00
John Ericson
d8fa7517fa buildable.{cc,hh} -> derived-path.{cc,hh} 2021-04-05 10:33:28 -04:00
John Ericson
179582872d Make DerivedPathWithHints a newtype
This allows us to namespace its constructors under it.
2021-04-05 10:05:21 -04:00
John Ericson
9b805d36ac Rename Buildable 2021-04-05 09:52:25 -04:00
John Ericson
9dfb97c987 "newtype" BuildableReq
This makes for better types errors and allows us to give it methods.
2021-04-05 09:35:55 -04:00
John Ericson
4fe41c6db3 No templates for Buildable and BuildableReq 2021-04-05 09:15:25 -04:00
John Ericson
255d145ba7 Use BuildableReq for buildPaths and ensurePath
This avoids an ambiguity where the `StorePathWithOutputs { drvPath, {}
}` could mean "build `brvPath`" or "substitute `drvPath`" depending on
context.

It also brings the internals closer in line to the new CLI, by
generalizing the `Buildable` type is used there and makes that
distinction already.

In doing so, relegate `StorePathWithOutputs` to being a type just for
backwards compatibility (CLI and RPC).
2021-04-05 08:33:00 -04:00
John Ericson
32f4454b9f Move StorePathWithOutput utilities out of store class
These are by no means part of the notion of a store, but rather are
things that happen to use stores. (Or put another way, there's no way
we'd make them virtual methods any time soon.) It's better to move them
out of that too-big class then.

Also, this helps us remove StorePathWithOutputs from the Store interface
altogether next commit.
2021-04-05 08:31:37 -04:00
John Ericson
7a2b566dc8 Move StorePathWithOutputs into its own header/file
In the following commits it will become less prevalent.
2021-04-05 08:31:37 -04:00
John Ericson
f7d9f7c338 Pull out Buildable into its own file/header in libnixstore 2021-04-05 08:31:37 -04:00
Mykola Orliuk
f3f228700a canonPath in one pass 2021-03-31 04:58:49 +02:00
Bjørn Forsman
edd606ae62 fetchGit: don't prefix "refs/heads/" on ref = "HEAD"
This fixes builtins.fetchGit { url = ...; ref = "HEAD"; }, that works in
stable nix (v2.3.10), but is broken in nix master:

  $ ./result/bin/nix repl
  Welcome to Nix version 2.4pre19700101_dd77f71. Type :? for help.

  nix-repl> builtins.fetchGit { url = "https://github.com/NixOS/nix"; ref = "HEAD"; }
  fetching Git repository 'https://github.com/NixOS/nix'fatal: couldn't find remote ref refs/heads/HEAD
  error: program 'git' failed with exit code 128

The documentation for builtins.fetchGit says ref = "HEAD" is the
default, so it should also be supported to explicitly pass it.

I came across this issue because poetry2nix can use ref = "HEAD" in some
situations.

Fixes #4674.
2021-03-30 11:21:38 +02:00
Eelco Dolstra
dd77f71afe LocalBinaryCacheStore::upsertFile(): Fix race
When multiple threads try to upsert the same file, this could fail.

Fixes #4667.
2021-03-26 17:10:15 +01:00
Eelco Dolstra
4638bcfb2c Fix some typos
Fixes #4671.
2021-03-26 16:14:38 +01:00
Eelco Dolstra
d1cb956bf2
Merge pull request #4650 from samueldr/fix/registry-config-dir
Use the appropriate config dir for the registry
2021-03-24 13:37:19 +01:00
Eelco Dolstra
1581c3e8ef
Merge pull request #4603 from DavHau/davhau-improve-nix-conf-help-builders
improve man page for nix.conf (builders)
2021-03-23 16:13:02 +01:00
Eelco Dolstra
6749d9e057
Merge pull request #4646 from Ma27/support-scoped-v6
Fix Nix to properly work with stores using a scoped IPv6 address
2021-03-23 15:03:47 +01:00
DavHau
0f40561c78 nix.conf builders: refer to manual page 2021-03-23 10:19:00 +07:00
John Ericson
f44206e719 Fix typos in the last PR #4656 2021-03-22 15:18:48 +00:00
John Ericson
9d309de0de Clean up serialization for BuildResult
A few versioning mistakes were corrected:

- In 27b5747ca7, Daemon protocol had some
  version `>= 0xc` that should have been `>= 0x1c`, or `28` since the
  other conditions used decimal.

- In a2b69660a9, legacy SSH gated new CAS
  info on version 6, but version 5 in the server. It is now 6
  everywhere.

Additionally, legacy ssh was sending over more metadata than the daemon
one was. The daemon now sends that data too.

CC @regnat

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2021-03-22 14:57:41 +00:00
Samuel Dionne-Riel
66b857244f Use the appropriate config dir for the registry 2021-03-19 15:20:47 -04:00
Eelco Dolstra
3e0e443181 ProgressBar: Respect verbosity level
This makes its behaviour consistent with SimpleLogger.
2021-03-19 17:56:39 +01:00
Eelco Dolstra
ef83ced4e1 Restore 'nix flake info' as a deprecated alias 2021-03-19 17:21:37 +01:00
Eelco Dolstra
8a5203d3b8
Merge pull request #4648 from hercules-ci/remove-unimplemented-hashAlgoOpt
Remove unimplemented hashAlgoOpt
2021-03-17 20:51:09 +01:00
Robert Hensing
a61112aadf Remove unimplemented hashAlgoOpt
It was in the header but never implemented.
2021-03-17 11:27:11 +01:00
Maximilian Bosch
ddcef3bcbb
Fix Nix to properly work with stores using a scoped IPv6 address
According to RFC4007[1], IPv6 addresses can have a so-called zone_id
separated from the actual address with `%` as delimiter. In contrast to
Nix 2.3, the version on `master` doesn't recognize it as such:

    $ nix ping-store --store ssh://root@fe80::1%18 --experimental-features nix-command
    warning: 'ping-store' is a deprecated alias for 'store ping'
    error: --- Error ----------------------------------------------------------------- nix
    don't know how to open Nix store 'ssh://root@fe80::1%18'

I modified the IPv6 match-regex accordingly to optionally detect this
part of the address. As we don't seem to do anything special with it, I
decided to leave it as part of the URL for now.

Fixes #4490

[1] https://tools.ietf.org/html/rfc4007
2021-03-16 19:14:42 +01:00
Eelco Dolstra
66fa1c7375 Merge 'nix flake {info,list-inputs}' into 'nix flake metadata'
Fixes #4613.
2021-03-16 17:19:04 +01:00
Eelco Dolstra
77f5d171e1 --override-input: Imply --no-write-lock-file
Fixes #3779.
2021-03-16 16:53:39 +01:00
Eelco Dolstra
ccb8a403ee
Merge pull request #4587 from obsidiansystems/derivation-goal-detect-invalid-output
Throw error for derivation goal with bogus wanted output
2021-03-15 16:49:44 +01:00
Eelco Dolstra
c0073f6268
Merge pull request #4580 from obsidiansystems/restore-test-build-remote-ca-fixed
Restore now-working build-remote-content-addressed-fixed test
2021-03-15 16:48:28 +01:00
Eelco Dolstra
91ea9c52ee
Merge pull request #4624 from NixOS/ca/realisation-nix-command
Add a `nix realisation` command for working on realisations
2021-03-15 16:37:09 +01:00
regnat
703c98c6cb Properly sign the unresolved drvs
Don't let them inherit the signature from the parent one (because it
makes no sense to do so), but re-sign them after they have been built
2021-03-15 16:35:17 +01:00
regnat
54ced9072b Check the signatures when copying store paths around
Broken atm
2021-03-15 16:35:14 +01:00
regnat
3e6017f911 pathInfoIsTrusted -> pathInfoIsUntrusted
I guess the rationale behind the old name wath that
`pathInfoIsTrusted(info)` returns `true` iff we would need to `blindly`
trust the path (because it has no valid signature and `requireSigs` is
set), but I find it to be a really confusing footgun because it's quite
natural to give it the opposite meaning.
2021-03-15 16:34:49 +01:00
regnat
826877cabf Add some logic for signing realisations
Not exposed anywhere, but built realisations are now signed (and this
should be forwarded when copy-ing them around)
2021-03-15 16:34:49 +01:00
Eelco Dolstra
306c154632
Merge pull request #4592 from NixOS/ca/remote-cache
Substitute content-addressed derivations
2021-03-15 16:22:42 +01:00
Eelco Dolstra
3bb1becdbb
Merge pull request #4566 from orbekk/master
Add support for bare git repositories when using git+file
2021-03-11 10:38:07 +01:00
Yorick van Pelt
8a0c00b856
Use libarchive for all compression 2021-03-10 22:34:29 +01:00
regnat
89013bdd7e Add a nix realisation command for working on realisations
Currently only has `nix realisation info`, more to come probably
2021-03-09 10:16:44 +01:00
Eelco Dolstra
1c0e3e453d
Merge pull request #4601 from lovesegfault/fix-4598
nix-build: check that envCommand exists
2021-03-08 10:13:39 +01:00
Eelco Dolstra
c5a232dda7
Merge pull request #4615 from NixOS/nix-show-stats-with-nix-cmd
Make NIX_SHOW_STATS work with new-style commands
2021-03-08 09:56:59 +01:00
Bernardo Meurer
6e849e3b0a
nix-build: set execfail
When starting a nix-shell with `-i` it was previously possible for it to
silently fail in the scenario where the specified interpreter didn't
exist. This happened due to the `exec` call masking the issue.

With this change we enable `execfail`, which causes the script using
`nix-shell` as interpreter to correctly exit with code 127.

Fixes: #4598
2021-03-04 18:54:45 -08:00
John Ericson
6212e89bf6 Avoid some StorePath -> Path -> StorePath roundtrips
There were done when StorePath was defined in Rust and there were some
FFI issues. This is no longer an issue.
2021-03-05 00:49:46 +00:00
DavHau
e16431b466 improve man page for nix.conf (builders) 2021-03-04 16:14:23 +07:00
Eelco Dolstra
665d4ec2da nix repl :doc: Don't return docs for partially applied primops
This gives misleading results for Nixpkgs functions like lib.toUpper.

Fixes #4596.
2021-03-03 17:52:57 +01:00
regnat
7331da99ab Make NIX_SHOW_STATS work with new-style commands 2021-03-02 14:59:12 +01:00
dramforever
fc6bfb261d libfetchers/tarball: Lock on effectiveUrl
Basically, if a tarball URL is used as a flake input, and the URL leads
to a redirect, the final redirect destination would be recorded as the
locked URL.

This allows tarballs under https://nixos.org/channels to be used as
flake inputs. If we, as before, lock on to the original URL it would
break every time the channel updates.
2021-03-02 21:56:50 +08:00
John Ericson
7ce10924c7 Fix bad wanted output error as requested
- UsageError -> Error

- include drv path too
2021-03-01 15:07:09 +00:00
Kjetil Orbekk
9931f18c2d Add support for bare git repositories with git+file
Local git repositories are normally used directly instead of
cloning. This commit checks if a repo is bare and forces a
clone.

Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2021-03-01 09:03:25 -05:00
regnat
df9d4f88d5 Allow substituting drv outputs when building 2021-03-01 14:00:17 +01:00
regnat
5d1c05b075 SubstitutionGoal -> PathSubstitutionGoal
To prepare for the upcoming DrvOutputSubstitutionGoal
2021-03-01 14:00:17 +01:00
Eelco Dolstra
e64cf8e0a3
Merge pull request #4574 from grahamc/libstore-ssh-host-key
libstore: support passing a builder's public SSH host key
2021-03-01 13:12:18 +01:00
John Ericson
4bbd80c536 Throw error for derivation goal with bogus wanted output 2021-02-28 00:19:35 +00:00
John Ericson
ae1441e548 Fix testing fixed-output derivations in double sandboxes
What happened was that Nix was trying to unconditionally mount these
paths in fixed-output derivations, but since the outer derivation was
pure, those paths did not exist. The solution is to only mount those
paths when they exist.
2021-02-27 05:23:14 +00:00
Puck Meerburg
7241fdc3d2 Properly propagate libseccomp linker flags 2021-02-26 23:01:16 +00:00
John Ericson
553b79f8c9 Remove unused redirectedBadOutputs 2021-02-26 16:10:54 +00:00
John Ericson
d560311f76 Remove temporary #if 0...#endif from previous commit 2021-02-26 16:10:52 +00:00
John Ericson
68f4c728ec Split {,local-}derivation-goal.{cc,hh}
This separates the scheduling logic (including simple hook pathway) from
the local-store needing code.

This should be the final split for now. I'm reasonably happy with how
it's turning out, even before I'm done moving code into
`local-derivation-goal`. Benefits:

1. This will help "witness" that the hook case is indeed a lot simpler,
   and also compensate for the increased complexity that comes from
   content-addressed derivation outputs.

2. It also moves us ever so slightly towards a world where we could use
   off-the-shelf storage or sandboxing, since `local-derivation-goal`
   would be gutted in those cases, but `derivation-goal` should remain
   nearly the same.

The new `#if 0` in the new files will be deleted in the following
commit. I keep it here so if it turns out more stuff can be moved over,
it's easy to do so in a way that preserves ordering --- and thus
prevents conflicts.

N.B.
```sh
git diff HEAD^^ --color-moved --find-copies-harder --patience --stat
```
makes nicer output.
2021-02-26 16:10:26 +00:00
John Ericson
05cc5a8587 Copy {,local-}derivation-goal.{cc,h}
Doing this prior to splitting, so we get better diff with default
options (e.g. on GitHub).
2021-02-26 16:01:47 +00:00
Eelco Dolstra
94637cd7e5
Merge pull request #4477 from NixOS/ca/build-remote
Build ca derivations remotely
2021-02-26 16:54:44 +01:00
regnat
f54976d77b Simplify the case where the drv is a purely input-addressed one 2021-02-26 16:35:05 +01:00
Eelco Dolstra
076d2b04da
Update src/libstore/build/derivation-goal.cc 2021-02-26 16:30:12 +01:00
Eelco Dolstra
17c98e03ea
Update src/build-remote/build-remote.cc 2021-02-26 16:29:37 +01:00
Eelco Dolstra
14f51880ba
Update src/build-remote/build-remote.cc 2021-02-26 16:29:30 +01:00
Eelco Dolstra
453c3a603f nix flake update: Recreate the lock file
This is probably what most people expect it to do. Fixes #3781.

There is a new command 'nix flake lock' that has the old behaviour of
'nix flake update', i.e. it just adds missing lock file entries unless
overriden using --update-input.
2021-02-26 14:55:54 +01:00
Eelco Dolstra
20ea1de77d Use std::make_unique 2021-02-26 12:35:29 +01:00
Eelco Dolstra
73daffb81b Merge remote-tracking branch 'origin/deduplicate-static-hashes-computation' 2021-02-26 12:30:25 +01:00
Eelco Dolstra
d2803406b5
Merge pull request #4487 from NixOS/ca/copy-drv-outputs
Copy ca derivation outputs
2021-02-26 12:20:42 +01:00
Eelco Dolstra
5edab777d1
Merge pull request #4530 from alyssais/kill
libutil: EPERM from kill(-1, ...) is fine
2021-02-26 12:19:16 +01:00
regnat
c43f446f4e Make nix copy work without the ca-derivations flag
The experimental feature was by mistake required for `nix copy` to work
at oll
2021-02-25 17:21:51 +01:00
Théophane Hufschmitt
c182aac98a Apply @edolstra stylistic suggestions
Mostly removing useless comments and adding spaces before `&`

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-02-25 17:21:51 +01:00
regnat
3b76f8f252 Ensure that the ca-derivations bit is set when copying realisations
This should already hold, but better ensure it for future-proof-nees
2021-02-25 17:19:59 +01:00
regnat
f67ff1f575 Don't crash when copying realisations to a non-ca remote
Rather throw a proper exception, and catch&log it on the client side
2021-02-25 17:19:59 +01:00
regnat
2e199673a5 Use RealisedPaths in copyPaths
That way we can copy the realisations too (in addition to the store
paths themselves)
2021-02-25 17:18:48 +01:00
Graham Christensen
1130b28824
distributed builds: load remote builder host key from the machines file
This is already used by Hydra, and is very useful when materializing
a remote builder list from service discovery. This allows the service
discovery tool to only sync one file instead of two.
2021-02-25 09:17:34 -05:00
Shea Levy
f6c5b05488
Respect command registrations in plugins. 2021-02-24 08:25:45 -05:00
Shea Levy
98d1b64400
Initialize plugins after handling initial command line flags
This is technically a breaking change, since attempting to set plugin
files after the first non-flag argument will now throw an error. This
is acceptable given the relative lack of stability in a plugin
interface and the need to tie the knot somewhere once plugins can
actually define new subcommands.
2021-02-24 08:22:17 -05:00
Shea Levy
ec3497c1d6
Bail if plugin-files is set after plugins have been loaded.
We know the flag will be ignored but the user wants it to take effect.
2021-02-24 08:20:48 -05:00
Eelco Dolstra
a878c448d8
Merge pull request #4551 from danieldk/system-features-compute-level
Add x86_64 compute levels as system features
2021-02-23 14:30:10 +01:00
regnat
ba1a256d08 Make DerivationGoal::drv a full Derivation
This field used to be a `BasicDerivation`, but this `BasicDerivation`
was downcasted to a `Derivation` when needed (implicitely or not), so we
might as well make it a full `Derivation` and upcast it when needed.

This also allows getting rid of a weird duplication in the way we
compute the static output hashes for the derivation. We had to
do it differently and in a different place depending on whether the
derivation was a full derivation or just a basic drv, but we can now do
it unconditionally on the full derivation.

Fix #4559
2021-02-23 14:15:45 +01:00
regnat
527da73690 Properly bypass the registering step when all outputs are present
There was already some logic for that, but it didn't handle the case of
content-addressed outputs, so extend it a bit for that
2021-02-23 08:04:03 +01:00
regnat
69666b951e build-remote: Always register the missing outputs
It's possible that all the paths are already there, but just not
associated to the current drv output
2021-02-23 08:04:03 +01:00
regnat
8c385d16ee Also send ca outputs to the build hook
Otherwise they don't get registered, triggering an assertion failure
at some point later
2021-02-23 08:04:03 +01:00
regnat
27b5747ca7 RemoteStore: Send back the new realisations
To allow it to build ca derivations remotely
2021-02-23 08:04:03 +01:00
regnat
a2b69660a9 LegacySSHStore: Send back the new realisations
To allow it to build ca derivations remotely
2021-02-23 08:04:03 +01:00
regnat
5687564a27 LocalStore: Send back the new realisations
To allow it to build ca derivations remotely
2021-02-23 08:04:03 +01:00
regnat
6fbf3fe636 Make the build-hook work with ca derivations
- Pass it the name of the outputs rather than their output paths (as
  these don't exist for ca derivations)
- Get the built output paths from the remote builder
- Register the new received realisations
2021-02-23 08:04:03 +01:00
Shea Levy
35205e2e92
Warn about instability of plugin API 2021-02-22 17:10:55 -05:00
regnat
e2f3b2eb42 Make missing auto-call arguments throw an eval error
The PR #4240 changed messag of the error that was thrown when an auto-called
function was missing an argument.
However this change also changed the type of the error, from `EvalError`
to a new `MissingArgumentError`. This broke hydra which was relying on
an `EvalError` being thrown.

Make `MissingArgumentError` a subclass of `EvalError` to un-break hydra.
2021-02-22 16:13:09 +01:00
Eelco Dolstra
574eb2be81
Tweak error message 2021-02-22 15:24:14 +01:00
Daniël de Kok
2de232d2b3 Add x86_64 compute levels as additional system types
When performing distributed builds of machine learning packages, it
would be nice if builders without the required SIMD instructions can
be excluded as build nodes.

Since x86_64 has accumulated a large number of different instruction
set extensions, listing all possible extensions would be unwieldy.
AMD, Intel, Red Hat, and SUSE have recently defined four different
microarchitecture levels that are now part of the x86-64 psABI
supplement and will be used in glibc 2.33:

https://gitlab.com/x86-psABIs/x86-64-ABI
https://lwn.net/Articles/844831/

This change uses libcpuid to detect CPU features and then uses them to
add the supported x86_64 levels to the additional system types. For
example on a Ryzen 3700X:

$ ~/aps/bin/nix -vv --version | grep "Additional system"
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
2021-02-22 09:11:15 +01:00
Eelco Dolstra
548437c234
Merge pull request #4541 from obsidiansystems/simpler-store-path-command
Deeper `Command` hierarchy to remove redundancy
2021-02-19 16:18:53 +01:00
regnat
f483b623e9 Remove the drv resolution caching mechanism
It isn't needed anymore now that don't need to eagerly resolve
everything like we used to do. So we can safely get rid of it
2021-02-19 15:48:31 +01:00
regnat
4bc28c44f2 Store the output hashes in the initialOutputs of the drv goal
That way we
1. Don't have to recompute them several times
2. Can compute them in a place where we know the type of the parent
  derivation, meaning that we don't need the casting dance we had before
2021-02-19 15:48:31 +01:00
Théophane Hufschmitt
0bfbd04369 Don't expose the "bang" drvoutput syntax
It's not fixed nor useful atm, so better keep it hidden

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-02-19 15:48:31 +01:00
Théophane Hufschmitt
93d9eb78a0 Syntactic fixes
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-02-19 15:48:31 +01:00
regnat
87c8d3d702 Register the realisations for unresolved drvs
Once a build is done, get back to the original derivation, and register
all the newly built outputs for this derivation.

This allows Nix to work properly with derivations that don't have all
their build inputs available − thus allowing garbage collection and
(once it's implemented) binary substitution
2021-02-19 15:48:31 +01:00
regnat
263f6dbd1c Don't crash nix-build when not all outputs are realised
Change the `nix-build` logic for linking/printing the output paths to allow for
some outputs to be missing. This might happen when the toplevel
derivation didn't have to be built, either because all the required
outputs were already there, or because they have all been substituted.
2021-02-19 15:48:31 +01:00
Eelco Dolstra
cd44c0af71 Increase default stack size on Linux
Workaround for #4550.
2021-02-18 19:22:37 +01:00
Eelco Dolstra
1b57825524 Document meta.mainProgram
Issue #4498.
2021-02-17 17:58:47 +01:00
Eelco Dolstra
7bd9898d5c nix run: Allow program name to be set in meta.mainProgram
This is useful when the program name doesn't match the package name
(e.g. ripgrep vs rg).

Fixes #4498.
2021-02-17 17:55:15 +01:00
Eelco Dolstra
13897afbe6 Throw an error if --arg / --argstr is used with a flake
Fixes #3949.
2021-02-17 17:32:10 +01:00
Eelco Dolstra
f33878b656 Make 'nix --version -vv' work
Fixes #3743.
2021-02-17 17:11:21 +01:00
Eelco Dolstra
cced73496b nix flake show: Handle 'overlays' output
Fixes #4542.
2021-02-17 16:53:39 +01:00
Eelco Dolstra
063de66909 nix develop: Fix quoted string handling
Fixes #4540.
2021-02-17 16:42:03 +01:00
John Ericson
ad337c8697 Deeper Command hierarchy to remove redundancy
Simply put, we now have `StorePathCommand : public StorePathsCommand` so
`StorePathCommand` doesn't reimplement work.
2021-02-12 17:48:09 +00:00
Graham Christensen
f2245091d0
Revert "narinfo: Change NAR URLs to be addressed on the NAR hash instead of the compressed hash" 2021-02-09 12:26:41 -05:00
Eelco Dolstra
ee3846b587
Merge pull request #4464 from tweag/nar-narhash-addressed
narinfo: Change NAR URLs to be addressed on the NAR hash instead of the compressed hash
2021-02-09 14:47:39 +01:00
Eelco Dolstra
fd6eaa1766
Merge pull request #4525 from sternenseemann/lowdown-0.8.0
libcmd/markdown: handle allocation errors in lowdown_term_rndr
2021-02-07 21:24:16 +01:00
Eelco Dolstra
37352aa7e1 Support --no-net for backwards compatibility 2021-02-07 20:44:56 +01:00
Alyssa Ross
7c112351d9
libutil: EPERM from kill(-1, ...) is fine
I tested a trivial program that called kill(-1, SIGKILL), which was
run as the only process for an unpriveleged user, on Linux and
FreeBSD.  On Linux, kill reported success, while on FreeBSD it failed
with EPERM.

POSIX says:

> If pid is -1, sig shall be sent to all processes (excluding an
> unspecified set of system processes) for which the process has
> permission to send that signal.

and

> The kill() function is successful if the process has permission to
> send sig to any of the processes specified by pid.  If kill() fails,
> no signal shall be sent.

and

> [EPERM]
>     The process does not have permission to send the signal to any
>     receiving process.

My reading of this is that kill(-1, ...) may fail with EPERM when
there are no other processes to kill (since the current process is
ignored).  Since kill(-1, ...) only attempts to kill processes the
user has permission to kill, it can't mean that we tried to do
something we didn't have permission to kill, so it should be fine to
interpret EPERM the same as success here for any POSIX-compliant
system.

This fixes an issue that Mic92 encountered[1] when he tried to review a
Nixpkgs PR on FreeBSD.

[1]: https://github.com/NixOS/nixpkgs/pull/81459#issuecomment-606073668
2021-02-07 13:56:50 +00:00
sternenseemann
d0e34c85f8 libcmd/markdown: handle allocation errors in lowdown_term_rndr
We upgrade to lowdown 0.8.0 [1] which contains a fix/improvement to a
behavior mentioned in this issue thread [2] where a big part of
lowdown's API would just call exit(1) on allocation errors since that
is a satisfying behavior for the lowdown binary.

Now lowdown_term_rndr returns 0 if an allocation error occurred which we
check for in libcmd/markdown.cc.

Also the extern "C" { } wrapper around lowdown.h has been removed as it
is not necessary.

[1]: 6ca7c855a0/versions.xml (L987-L1006)
[2]: https://github.com/kristapsdz/lowdown/issues/45#issuecomment-756681153
2021-02-06 13:14:57 +01:00
Eelco Dolstra
480426a364 Add more instrumentation for #4270 2021-02-05 15:57:33 +01:00
Eelco Dolstra
d7c27f21ab
Merge pull request #4372 from tweag/ca/drvoutputs-commands
Add a new Cmd type working on RealisedPaths
2021-02-05 13:03:50 +01:00
Eelco Dolstra
c77f4a9012
Merge pull request #4515 from matthewbauer/fix-nix-profile-install-first-output
Use derivation output name from toDerivation
2021-02-05 12:19:10 +01:00
Eelco Dolstra
0187838e2e Add a trace to readLine() failures
Hopefully this helps to diagnose 'error: unexpected EOF reading a
line' on macOS.
2021-02-05 12:18:11 +01:00
regnat
e69cfdebb0 Remove the visit machinery in RealisedPath
In addition to being some ugly template trickery, it was also totally
useless as it was used in only one place where I could replace it by
just a few extra characters
2021-02-05 11:42:33 +01:00
regnat
d2091af231 Move the GENERATE_CMP macro to its own file
Despite being an ugly hack, it can probably be useful in a couple extra
places
2021-02-05 11:42:33 +01:00
Théophane Hufschmitt
43d409f669
Fix a whitespace issue
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-02-04 14:47:56 +01:00
Théophane Hufschmitt
ca8facefb6
Normalize some error messages
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-02-04 14:47:28 +01:00
Matthew Bauer
e38cd5becb Always enter first level of attrset in nix search
This makes nix search always go through the first level of an
attribute set, even if it's not a top level attribute. For instance,
you can now list all GHC compilers with:

$ nix search nixpkgs#haskell.compiler
...

This is similar to how nix-env works when you pass in -A.
2021-02-03 21:22:11 -06:00
sternenseemann
76d8bdfe35 Include note about type of catched errors in tryEval documentation
Reference #356.
2021-02-03 17:14:40 +01:00
Matthew Bauer
3d1bbabe55 Use derivation output name from toDerivation
This fixes an issue where derivations with a primary output that is
not "out" would fail with:

$ nix profile install nixpkgs#sqlite
error: opening directory '/nix/store/2a2ydlgyydly5czcc8lg12n6qqkfz863-sqlite-3.34.1-bin': No such file or directory

This happens because while derivations produce every output when
built, you might not have them if you didn't build the derivation
yourself (for instance, the store path was fetch from a binary cache).
This uses outputName provided from DerivationInfo which appears to
match the first output of the derivation.
2021-02-02 19:53:19 -06:00
Eelco Dolstra
e486996cef
Rename to --debugger
Co-authored-by: Domen Kožar <domen@dev.si>
2021-02-01 15:50:58 +01:00
Domen Kožar
d0b74e2d25
--no-net -> --offline 2021-02-01 13:11:42 +00:00
Eelco Dolstra
b19aec7eeb
Merge pull request #4461 from NixOS/ca/error-logging-fixes
Fix some logging with ca derivations
2021-01-29 16:12:50 +01:00
James Ottaway
991edaace5
Shorten mktemp flag for macOS
Address `mktemp: illegal option -- -`.
2021-01-29 13:55:18 +10:00
regnat
9355ecd543 Add a new Cmd type working on RealisedPaths
Where a `RealisedPath` is a store path with its history, meaning either
an opaque path for stuff that has been directly added to the store, or a
`Realisation` for stuff that has been built by a derivation

This is a low-level refactoring that doesn't bring anything by itself
(except a few dozen extra lines of code :/ ), but raising the
abstraction level a bit is important on a number of levels:

- Commands like `nix build` have to query for the realisations after the
  build is finished which is fragile (see
  27905f12e4a7207450abe37c9ed78e31603b67e1 for example). Having them
  oprate directly at the realisation level would avoid that
- Others like `nix copy` currently operate directly on (built) store
  paths, but need a bit more information as they will need to register
  the realisations on the remote side
2021-01-28 09:38:44 +01:00
Eelco Dolstra
b8f345b29a
Merge pull request #4483 from shlevy/libcmd
Move command plugin interface to libnixcmd
2021-01-27 17:17:58 +01:00
Eelco Dolstra
12de0466fe Add trace to build errors during import-from-derivation
Example:

error: builder for '/nix/store/9ysqfidhipyzfiy54mh77iqn29j6cpsb-failing.drv' failed with exit code 1;
       last 1 log lines:
       > FAIL
       For full logs, run 'nix log /nix/store/9ysqfidhipyzfiy54mh77iqn29j6cpsb-failing.drv'.

       … while importing '/nix/store/pfp4a4bjh642ylxyipncqs03z6kkgfvy-failing'

       at /nix/store/25wgzr2qrqqiqfbdb1chpiry221cjglc-source/flake.nix:58:15:

           57|
           58|         ifd = import self.hydraJobs.broken;
             |               ^
           59|
2021-01-27 14:46:10 +01:00
Eelco Dolstra
965dc6070a Drop trailing whitespace 2021-01-27 14:04:49 +01:00
Eelco Dolstra
c03f41055d Add traces to errors while updating flake lock file
Example:

$ nix build --show-trace
error: unable to download 'https://api.github.com/repos/NixOS/nixpkgs/commits/no-such-branch': HTTP error 422 ('')

       response body:

       {
         "message": "No commit found for SHA: no-such-branch",
         "documentation_url": "https://docs.github.com/rest/reference/repos#get-a-commit"
       }

       … while fetching the input 'github:NixOS/nixpkgs/no-such-branch'

       … while updating the flake input 'nixpkgs'

       … while updating the lock file of flake 'git+file:///home/eelco/Dev/nix'
2021-01-27 14:02:54 +01:00
Eelco Dolstra
8e758d402b Remove mkFlag() 2021-01-27 12:06:03 +01:00
regnat
9da11bac57 Fix the error message when a dep is missing
Fix a mismatch in the errors thrown when a needed output was missing
from an input derivation that was leading to a wrong and quite misleading error
message
2021-01-26 14:49:23 +01:00
regnat
d3c4284133 Make the error message for missing outputs more useful
Don't only show the name of the output, but also the derivation to which
this output belongs (as otherwise it's very hard to track back what went
wrong)
2021-01-26 14:49:23 +01:00
Shea Levy
6af6e41df0
Move command plugin interface to libnixcmd 2021-01-26 06:22:24 -05:00
Eelco Dolstra
36c4d6f592 Group common options 2021-01-25 19:03:13 +01:00
Eelco Dolstra
807d963ee8 Group subcommands by category 2021-01-25 18:19:32 +01:00
Eelco Dolstra
3ba98ba8f0 Tell user to run 'nix log' to get full build logs 2021-01-25 17:15:38 +01:00
Eelco Dolstra
a32073e7e8 Add FIXME 2021-01-25 14:43:16 +01:00
Eelco Dolstra
b159d23800 Make '--help' do the same as 'help' (i.e. show a manpage) 2021-01-25 14:38:15 +01:00
Eelco Dolstra
488a826842
Merge pull request #4467 from edolstra/error-formatting
Improve error formatting
2021-01-25 12:50:57 +01:00
Eelco Dolstra
c5b42c5a42
Merge pull request #4470 from matthewbauer/fix-4469
Handle missing etag in 304 Not Modified response
2021-01-25 12:50:13 +01:00
Eelco Dolstra
680d8a5b86
Merge pull request #4387 from obsidiansystems/non-local-store-build
Make `nix-build --store whatever` work
2021-01-25 12:24:23 +01:00
Matthew Bauer
1ea5f0b66c Remove expectedETag assert in tarball.cc 2021-01-22 23:19:52 -06:00
Matthew Bauer
a766824660 Handle missing etag in 304 Not Modified response
GitHub now omits the etag, but 304 implies it matches the one we
provided. Just use that one to avoid having an etag-less resource.

Fixes #4469
2021-01-22 14:47:45 -06:00
John Ericson
53a709535b Apply suggestions from code review
Thanks!

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-01-22 15:58:58 +00:00
John Ericson
8c07ed1dda Improve documentation and test and requested 2021-01-22 15:58:58 +00:00
Eelco Dolstra
b7bfc7ee52
Add FIXME 2021-01-22 12:36:50 +01:00
Eelco Dolstra
0eb22db311 Fix macOS build 2021-01-21 12:46:22 +01:00
Eelco Dolstra
55849e153e Change error position formatting
It's now

  at /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix:7:7:

instead of

  at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

The new format is more standard and clickable.
2021-01-21 11:02:09 +01:00
Eelco Dolstra
40608342cb Remove trailing whitespace 2021-01-21 11:02:09 +01:00
Eelco Dolstra
8d4268d190 Improve error formatting
Changes:

* The divider lines are gone. These were in practice a bit confusing,
  in particular with --show-trace or --keep-going, since then there
  were multiple lines, suggesting a start/end which wasn't the case.

* Instead, multi-line error messages are now indented to align with
  the prefix (e.g. "error: ").

* The 'description' field is gone since we weren't really using it.

* 'hint' is renamed to 'msg' since it really wasn't a hint.

* The error is now printed *before* the location info.

* The 'name' field is no longer printed since most of the time it
  wasn't very useful since it was just the name of the exception (like
  EvalError). Ideally in the future this would be a unique, easily
  googleable error ID (like rustc).

* "trace:" is now just "…". This assumes error contexts start with
  something like "while doing X".

Example before:

  error: --- AssertionError ---------------------------------------------------------------------------------------- nix
  at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

       6|
       7|   x = assert false; 1;
        |       ^
       8|

  assertion 'false' failed
  ----------------------------------------------------- show-trace -----------------------------------------------------
  trace: while evaluating the attribute 'x' of the derivation 'hello-2.10'
  at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix

     191|         // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) {
     192|           name = "${attrs.pname}-${attrs.version}";
        |           ^
     193|         } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {

Example after:

  error: assertion 'false' failed

         at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

              6|
              7|   x = assert false; 1;
               |       ^
              8|

         … while evaluating the attribute 'x' of the derivation 'hello-2.10'

         at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix

            191|         // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) {
            192|           name = "${attrs.pname}-${attrs.version}";
               |           ^
            193|         } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {
2021-01-21 11:02:09 +01:00
adisbladis
144cad9069
narinfo: Change NAR URLs to be addressed on the NAR hash instead of the compressed hash
This change is to simplify [Trustix](https://github.com/tweag/trustix) indexing and makes it possible to reconstruct this URL regardless of the compression used.

In particular this means that 7c2e9ca597/contrib/nix/nar/nar.go (L61-L71) can be removed and only the bits that are required to establish trust needs to be published in the Trustix build logs.
2021-01-21 10:32:56 +01:00
Eelco Dolstra
259100332f Fix clang build 2021-01-21 10:29:51 +01:00
Eelco Dolstra
d04d8463fa
Merge pull request #4281 from lilyball/shebang
Escape filename given to nix-shell in shebang mode
2021-01-20 10:00:10 +01:00
Eelco Dolstra
bc90252cec nix profile install: Support installing non-flakes
Fixes #4458.
2021-01-18 23:08:58 +01:00
Eelco Dolstra
555940f065 Use enumerate() 2021-01-18 22:50:39 +01:00
Eelco Dolstra
ea756b3654 --refresh: Imply setting .narinfo disk cache TTL to 0 2021-01-18 14:38:31 +01:00
Eelco Dolstra
1bbc66f865 Merge branch 'slashes-in-github-branches' of https://github.com/Ma27/nix 2021-01-18 12:55:06 +01:00
regnat
11b63740e3 Fix content-addressed flake outputs
Prevent some `nix flake` commands to crash by trying to parse a
placeholder output as a store path
2021-01-18 11:31:59 +01:00
regnat
9432c170e7 Fix the drv output map for non ca derivations
With the `ca-derivation` experimental features, non-ca derivations used
to have their output paths returned as unknown as long as they weren't
built (because of a mistake in the code that systematically erased the
previous value)
2021-01-18 11:01:39 +01:00
Eelco Dolstra
1acbb61696 Tweak 2021-01-17 19:49:28 +01:00
ryneeverett
1e13c79a91 Document expected output of 'nix store ping'.
While interpreting the output is fairly intuitive it would be better to
explicitly specify what a good invocation looks like.

That this isn't completely obvious (or at least causes folks to
second-guess themselves) can be seen in a couple user threads:

- https://discourse.nixos.org/t/nixos-cache-fetching-issue/3575/11
- https://discourse.nixos.org/t/newbie-question-cant-get-trivial-example-of-nixops-to-work-on-my-mac/1125/8
2021-01-16 19:11:10 +00:00
Domen Kožar
00f99fdfe6
Merge pull request #4240 from bburdette/2259-error-message
2259 error message - "auto-call" error
2021-01-15 18:26:21 +00:00
John Ericson
7af743470c Make public keys and requireSigs local-store specific again
Thanks @regnat and @edolstra for catching this and comming up with the
solution.

They way I had generalized those is wrong, because local settings for
non-local stores is confusing default. And due to the nature of C++
inheritance, fixing the defaults is more annoying than it should be.
Additionally, I thought we might just drop the check in the substitution
logic since `Store::addToStore` is now streaming, but @regnat rightfully
pointed out that as it downloads dependencies first, that would still be
too late, and also waste effort on possibly unneeded/unwanted
dependencies.

The simple and correct thing to do is just make a store method for the
boolean logic, keeping all the setting and key stuff the way it was
before. That new method is both used by `LocalStore::addToStore` and the
substitution goal check. Perhaps we might eventually make it fancier,
e.g. sending the ValidPathInfo to remote stores for them to validate,
but this is good enough for now.
2021-01-15 16:37:41 +00:00
Jonathan Ringer
86a2ceeb98 Fix gcc10 build 2021-01-15 10:41:35 +01:00
John Ericson
0027b05a15 Merge remote-tracking branch 'upstream/master' into non-local-store-build 2021-01-15 02:01:24 +00:00
Eelco Dolstra
7a472a76d4 Add 'nix daemon' command 2021-01-14 00:05:04 +01:00
Eelco Dolstra
28ef6ebf91 Typo 2021-01-13 23:51:27 +01:00
Eelco Dolstra
d33eca8539 Rename 'nix store sign-paths' to 'nix store sign' 2021-01-13 23:32:37 +01:00
Eelco Dolstra
61216d32e1 Add 'nix store repair' command 2021-01-13 23:27:39 +01:00
Eelco Dolstra
3da9a9241c Convert option descriptions to Markdown 2021-01-13 14:18:04 +01:00
Eelco Dolstra
4e9cec79bf
Merge pull request #4444 from matthewbauer/unset-curproc-arch-affinity
Set kern.curproc_arch_affinity=0 to escape Rosetta
2021-01-13 12:16:53 +01:00
Eelco Dolstra
fbfa70dc02
Merge pull request #4443 from rickynils/prefer-local-build-respect-zero-max-jobs
Don't let 'preferLocalBuild' override 'max-jobs=0'
2021-01-13 12:15:54 +01:00
Rickard Nilsson
0ca1a50132 Remove a redundant condition in DerivationGoal::tryLocalBuild() 2021-01-13 10:13:51 +01:00
Eelco Dolstra
2f463e90ed Add 'nix profile history' command
Replaces 'nix-env --list-generations'. Similar to 'nix profile
diff-closures' but shows only the changes in top-level packages.
2021-01-12 23:53:53 +01:00
Eelco Dolstra
29007f8bc6 nix profile info -> nix profile list 2021-01-12 19:57:05 +01:00
Matthew Bauer
f69820417f Set kern.curproc_arch_affinity=0 to escape Rosetta
By default, once you enter x86_64 Rosetta 2, macOS will try to run
everything in x86_64. So an x86_64 Nix will still try to use x86_64
even when system = aarch64-darwin. To avoid this we can set
kern.curproc_arch_affinity sysctl. With kern.curproc_arch_affinity=0,
we ignore this preference.

This is based on how
https://opensource.apple.com/source/system_cmds/system_cmds-880.40.5/arch.tproj/arch.c.auto.html
works. Completely undocumented, but seems to work!

Note, you can verify this works with this impure Nix expression:

```
  {
    a = derivation {
      name = "a";
      system = "aarch64-darwin";
      builder = "/bin/sh";
      args = [ "-e" (builtins.toFile "builder" ''
        [ "$(/usr/bin/arch)" = arm64 ]
        [ "$(/usr/bin/arch -arch x86_64 /bin/sh -c /usr/bin/arch)" = i386 ]
        [ "$(/usr/bin/arch -arch arm64 /bin/sh -c /usr/bin/arch)" = arm64 ]
        /usr/bin/touch $out
      '') ];
    };

    b = derivation {
      name = "b";
      system = "x86_64-darwin";
      builder = "/bin/sh";
      args = [ "-e" (builtins.toFile "builder" ''
        [ "$(/usr/bin/arch)" = i386 ]
        [ "$(/usr/bin/arch -arch x86_64 /bin/sh -c /usr/bin/arch)" = i386 ]
        [ "$(/usr/bin/arch -arch arm64 /bin/sh -c /usr/bin/arch)" = arm64 ]
        /usr/bin/touch $out
      '') ];
    };
  }
```
2021-01-11 22:40:21 -06:00
Rickard Nilsson
44fd7a05b6 Don't let 'preferLocalBuild' override 'max-jobs=0'
This resolves #3810 by changing the behavior of `max-jobs = 0`, so
that specifying the option also avoids local building of derivations
with the attribute `preferLocalBuild = true`.
2021-01-12 01:28:00 +01:00
Eelco Dolstra
6254b1f5d2 Add 'nix store delete' command 2021-01-11 19:46:59 +01:00
Eelco Dolstra
93ad6430ed nix store prefetch-tarball -> nix flake prefetch 2021-01-11 12:36:39 +01:00
Eelco Dolstra
7480f2bf20
Merge pull request #4435 from DanilaFe/flake-input-types
Allow Flake inputs to accept boolean and integer attributes
2021-01-11 11:38:53 +01:00
Eelco Dolstra
fdcd62eec5 Add 'nix store gc' command 2021-01-10 23:29:14 +01:00
Danila
1db3f84bac
Upcase "Boolean" in Flake attribute type error
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-01-08 16:12:21 -08:00
Eelco Dolstra
e21aee58f6 Fix tests 2021-01-08 14:17:06 +01:00
Eelco Dolstra
17beae299d Support binary unit prefixes in command line arguments 2021-01-08 12:51:19 +01:00
Eelco Dolstra
6548b89cc4 string2Int(): Return std::optional 2021-01-08 12:22:21 +01:00
Eelco Dolstra
29a445840a Remove unused mkFlag1 2021-01-08 11:42:44 +01:00
Eelco Dolstra
1d4954e73e Remove mkFlag integer specialisation 2021-01-08 11:40:36 +01:00
Eelco Dolstra
48a9be2aab Remove mkIntFlag 2021-01-08 10:44:55 +01:00
Danila Fedorin
ba0f841a07 Use switch statement instead of sequence of ifs 2021-01-08 03:13:42 +00:00
Danila Fedorin
93f1678ec6 Allow Flake inputs to accept boolean and integer attributes
I believe that this makes it possible to do things like
Git inputs with submodules, but it also likely applies
to other input types from libfetchers.
2021-01-08 01:53:57 +00:00
Eelco Dolstra
0813350349 Add 'nix store prefetch-{file,tarball}'
These replace nix-prefetch-url and nix-prefetch-url --unpack,
respectively.
2021-01-07 21:51:46 +01:00
Eelco Dolstra
0df69d96e0 Make sodium a required dependency 2021-01-06 17:56:53 +01:00
Eelco Dolstra
9374c2baea Add commands for generating secret/public keys 2021-01-06 17:49:31 +01:00
Eelco Dolstra
555152ffe8 crypto.cc: API cleanup and add generate() / to_string() methods 2021-01-06 17:04:46 +01:00
Eelco Dolstra
146af4ee9b Move sodium_init() call 2021-01-06 16:43:09 +01:00
Eelco Dolstra
3edcb198e5
Merge pull request #4310 from matthewbauer/rosetta2-extra-platforms
Add x86_64-darwin and aarch64 to "extra-platforms" automatically when Rosetta2 is detected
2021-01-06 11:31:13 +01:00
Eelco Dolstra
8af4f886e2 Fix deadlock in LocalStore::addSignatures()
Fixes #4367.
2021-01-05 11:47:29 +01:00
Danila Fedorin
988dd0a65f Fix conversion from JSON to fetch attributes
It appears as through the fetch attribute, which
is simply a variant with 3 elements, implicitly
converts boolean arguments to integers. One must
use Explicit<bool> to correctly populate it with
a boolean. This was missing from the implementation,
and resulted in clearly boolean JSON fields being
treated as numbers.
2021-01-05 02:06:25 +00:00
Eelco Dolstra
368a972a38
Merge pull request #4411 from corngood/env-assert-fix
Fix insufficent attribute capacity in user profile
2020-12-30 21:47:21 +01:00
David McFarland
d27eb0ef57 Fix insufficent attribute capacity in user profile 2020-12-30 16:20:03 -04:00
Eelco Dolstra
bff706e27c
Merge pull request #4397 from matthewbauer/support-libcxx10
Cast variants fully for libc++10
2020-12-30 21:07:09 +01:00
Matthew Bauer
64904b9d5d Fixup 2020-12-28 19:40:04 -06:00
Eelco Dolstra
6262a70363
scanForReferences: Remove misleading comment
References have always been determined only by the hash part, not the
name or the store prefix.

Fixes #4396.
2020-12-28 17:21:19 +01:00
Matthew Bauer
ede534a3a1
Merge branch 'master' into support-libcxx10 2020-12-24 14:16:09 -06:00
Matthew Bauer
d4870462f8 Cast variants fully for libc++10
libc++10 seems to be stricter on what it allows in variant conversion.
I'm not sure what the rules are here, but this is the minimal change
needed to get through the compilation errors.
2020-12-23 23:41:58 -06:00
John Ericson
fed1237246 Test nix-build with non-local-store --store
Just a few small things needed fixing!
2020-12-23 22:42:06 +00:00
John Ericson
85f2e9e8fa Expose schedule entrypoints to all stores
Remote stores still override so the other end schedules.
2020-12-23 22:42:06 +00:00
John Ericson
450c3500f1 Crudely make worker only provide a Store, not LocalStore
We downcast in a few places, this will be refactored to be better later.
2020-12-23 22:42:06 +00:00
John Ericson
12f7a1f65b build-remote no longer requires local store be local 2020-12-23 22:42:06 +00:00
John Ericson
57062179ce Move some PKI stuff from LocalStore to Store 2020-12-23 22:42:06 +00:00
Eelco Dolstra
a93916b190
Merge pull request #4336 from NixOS/manpages
Documentation for nix subcommands
2020-12-23 21:10:32 +01:00
Eelco Dolstra
5178211e96
Add 'nix' manpage 2020-12-23 18:33:42 +01:00
Eelco Dolstra
26e502ceb5
Add TODO 2020-12-23 18:26:40 +01:00
Eelco Dolstra
1047cb1e53
Command: Remove examples() 2020-12-23 18:26:40 +01:00
Eelco Dolstra
c9279b831e
Add 'nix flake' manpages 2020-12-23 18:26:40 +01:00
Eelco Dolstra
8927cba62f
Merge pull request #4366 from NixOS/readInvalidDerivation-on-remote-caches
Use the fs accessor for readInvalidDerivation
2020-12-23 11:55:52 +01:00
Eelco Dolstra
f4a9fb67da Merge branch 'git-rev-error' of https://github.com/Ma27/nix into master 2020-12-22 16:39:32 +01:00
Eelco Dolstra
e27044216b Fix tests 2020-12-22 16:23:57 +01:00
Eelco Dolstra
75efa42134 Move <nix/fetchurl.nix> into the nix binary
This makes the statically linked nix binary just work, without needing
any additional files.
2020-12-22 14:43:20 +01:00
Eelco Dolstra
5373f4be3b chrootHelper: Handle symlinks in the root directory
This is necessary on Ubuntu where /bin and /lib* are symlinks.
2020-12-22 12:28:50 +01:00
Maximilian Bosch
e54971d019
Document allRefs argument of builtins.fetchTree 2020-12-22 12:02:08 +01:00
Eelco Dolstra
724b7f4fb6 Don't log from inside the logger
This deadlocks ProgressBar, e.g.

  # nix run --impure --no-substitute --store '/tmp/nix2?store=/foo' --expr 'derivation { builder = /nix/store/zi90rxslsm4mlr46l2xws1rm94g7pk8p-busybox-1.31.1-x86_64-unknown-linux-musl/bin/busybox; }'

leads to

  Thread 1 (Thread 0x7ffff6126e80 (LWP 12250)):
  #0  0x00007ffff7215d62 in __lll_lock_wait () from /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libpthread.so.0
  #1  0x00007ffff720e721 in pthread_mutex_lock () from /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libpthread.so.0
  #2  0x00007ffff7ad17fa in __gthread_mutex_lock (__mutex=0x6c5448) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/x86_64-unknown-linux-gnu/bits/gthr-default.h:749
  #3  std::mutex::lock (this=0x6c5448) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/std_mutex.h:100
  #4  std::unique_lock<std::mutex>::lock (this=0x7fffffff09a8, this=0x7fffffff09a8) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/unique_lock.h:141
  #5  std::unique_lock<std::mutex>::unique_lock (__m=..., this=0x7fffffff09a8) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/unique_lock.h:71
  #6  nix::Sync<nix::ProgressBar::State, std::mutex>::Lock::Lock (s=0x6c5448, this=0x7fffffff09a0) at src/libutil/sync.hh:45
  #7  nix::Sync<nix::ProgressBar::State, std::mutex>::lock (this=0x6c5448) at src/libutil/sync.hh:85
  #8  nix::ProgressBar::logEI (this=0x6c5440, ei=...) at src/libmain/progress-bar.cc:131
  #9  0x00007ffff7608cfd in nix::Logger::logEI (ei=..., lvl=nix::lvlError, this=0x6c5440) at src/libutil/logging.hh:88
  #10 nix::getCodeLines (errPos=...) at src/libutil/error.cc:66
  #11 0x00007ffff76073f2 in nix::showErrorInfo (out=..., einfo=..., showTrace=<optimized out>) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/optional:897
  #12 0x00007ffff7ad19e7 in nix::ProgressBar::logEI (this=0x6c5440, ei=...) at src/libmain/progress-bar.cc:134
  #13 0x00007ffff7ab9d10 in nix::Logger::logEI (ei=..., lvl=nix::lvlError, this=0x6c5440) at src/libutil/logging.hh:88
  #14 nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) (programName="/home/eelco/Dev/nix/outputs/out/bin/nix", fun=...) at src/libmain/shared.cc:328
  #15 0x000000000046226b in main (argc=<optimized out>, argv=<optimized out>) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/ext/new_allocator.h:80
2020-12-22 11:15:29 +01:00
Maximilian Bosch
2857b1baaf
Add explicit allRefs = true; argument to fetchGit
Sometimes it's necessary to fetch a git repository at a revision and
it's unknown which ref contains the revision in question. An example
would be a Cargo.lock which only provides the URL and the revision when
using a git repository as build input.

However it's considered a bad practice to perform a full checkout of a
repository since this may take a lot of time and can eat up a lot of
disk space. This patch makes a full checkout explicit by adding an
`allRefs` argument to `builtins.fetchGit` which fetches all refs if
explicitly set to true.

Closes #2409
2020-12-22 10:46:00 +01:00
Maximilian Bosch
629af83b2d
Provide a more meaningful error-message for builtins.fetchGit if a revision can't be checked out
A common pitfall when using e.g. `builtins.fetchGit` is the `fatal: not
a tree object`-error when trying to fetch a revision of a git-repository
that isn't on the `master` branch and no `ref` is specified.

In order to make clear what's the problem, I added a simple check
whether the revision in question exists and if it doesn't a more
meaningful error-message is displayed:

```
nix-repl> builtins.fetchGit { url = "https://github.com/owner/myrepo"; rev = "<commit not on master>"; }
moderror: --- Error -------------------------------------------------------------------- nix
Cannot find Git revision 'bf1cc5c648e6aed7360448a3745bb2fe4fbbf0e9' in ref 'master' of repository 'https://gitlab.com/Ma27/nvim.nix'! Please make sure that the rev exists on the ref you've specified or add allRefs = true; to fetchGit.
```

Closes #2431
2020-12-22 10:46:00 +01:00
Eelco Dolstra
16e34085e8
Add 'nix profile' manpage 2020-12-21 13:32:29 +01:00
Eelco Dolstra
0c09f63de8
Add 'nix bundle' manpage
Fixes #4375.
2020-12-21 13:32:29 +01:00
Eelco Dolstra
f4e9d4fcb3
Add 'nix store diff-closures' manpage 2020-12-21 13:32:29 +01:00
Eelco Dolstra
4f3e7f4eec
Add 'nix show-derivation' manpage 2020-12-21 13:32:29 +01:00
Eelco Dolstra
3b123a6ee6
nix show-derivation: Say "system" instead of "platform"
There is really no good reason to use "platform" except that that's
what we use internally (also for no good reason).
2020-12-21 13:32:29 +01:00
Eelco Dolstra
daf365b0b7
Add 'nix help' manpage 2020-12-21 13:32:29 +01:00
Eelco Dolstra
e6bea9c9b1
Add 'nix store make-content-addressable' manpage 2020-12-21 13:32:29 +01:00
Eelco Dolstra
cdf20e04b7
Doh 2020-12-21 13:32:29 +01:00
Eelco Dolstra
2e599dbb88
Add 'nix path-info' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra
cb25a89f1c
Add 'nix store optimise' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra
8dd7d7e9db
Add 'nix store verify' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra
6b32551aba
Add 'nix upgrade-nix' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra
19540744ad
Add 'nix why-depends' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra
c14ed3f8b2
Add 'nix store' NAR-related manpages 2020-12-21 13:32:28 +01:00
Eelco Dolstra
2cc02bbe76
Add 'nix nar' manpages 2020-12-21 13:32:28 +01:00
Eelco Dolstra
a407d14339
Add 'nix eval' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra
53ce20eab7
Add 'nix store ping' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra
f34b1801a4
Tweak 2020-12-21 13:32:28 +01:00
Eelco Dolstra
58bacc85e7
Add 'nix log' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra
6ce393392b
Add 'nix repl' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra
b2262be19b
Add 'nix edit' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra
4e065229c7
Typo 2020-12-21 13:32:27 +01:00
Eelco Dolstra
e90e745232
Add 'nix registry' manpages
This also documents the registry format and matching/unification
semantics (though not quite correctly).
2020-12-21 13:32:27 +01:00
Eelco Dolstra
42cc98f8d6
Add 'nix develop' and `nix print-dev-env' manpages 2020-12-21 13:32:27 +01:00
Eelco Dolstra
e9de689a6e
Add 'nix search' manpage 2020-12-21 13:32:27 +01:00
Eelco Dolstra
28ee307fd8
Add 'nix copy' manpage 2020-12-21 13:32:27 +01:00
Eelco Dolstra
09660b8557
Add 'nix run' and 'nix shell' manpages 2020-12-21 13:32:27 +01:00
Eelco Dolstra
ae7351dbee
Add 'nix build' manpage 2020-12-21 13:32:27 +01:00
Eelco Dolstra
346baec783
Move doc() to Args 2020-12-21 13:32:23 +01:00
Eelco Dolstra
9fab14adbc
Merge pull request #4385 from obsidiansystems/store-subclass
Overhaul store subclassing
2020-12-21 12:51:36 +01:00
Eelco Dolstra
ec4a5c5b0b
Merge pull request #4355 from Infinisil/private-value-type
Refactoring for private Value type
2020-12-21 12:38:47 +01:00
John Ericson
1a1af75338 Overhaul store subclassing
We embrace virtual the rest of the way, and get rid of the
`assert(false)` 0-param constructors.

We also list config base classes first, so the constructor order is
always:

  1. all the configs
  2. all the stores

Each in the same order
2020-12-20 15:47:14 +00:00
Silvan Mosberger
b70d22baca
Replace Value type setters with mk* functions
Move clearValue inside Value

mkInt instead of setInt

mkBool instead of setBool

mkString instead of setString

mkPath instead of setPath

mkNull instead of setNull

mkAttrs instead of setAttrs

mkList instead of setList*

mkThunk instead of setThunk

mkApp instead of setApp

mkLambda instead of setLambda

mkBlackhole instead of setBlackhole

mkPrimOp instead of setPrimOp

mkPrimOpApp instead of setPrimOpApp

mkExternal instead of setExternal

mkFloat instead of setFloat

Add note that the static mk* function should be removed eventually
2020-12-18 21:48:22 +01:00
Silvan Mosberger
12e65078ef
Rename Value::normalType() -> Value::type() 2020-12-17 14:45:45 +01:00
Silvan Mosberger
d67e02919c
Rename ValueType -> InternalType, NormalType -> ValueType
And Value::type to Value::internalType, such that type() can be used in
the next commit to get the new ValueType
2020-12-17 14:45:22 +01:00
regnat
4d45839499 Fix the detection of already built drv outputs
PRs #4370 and #4348 had a bad interaction in that the second broke the fist
one in a not trivial way.

The issue was that since #4348 the logic for detecting whether a
derivation output is already built requires some logic that was specific
to the `LocalStore`.

It happens though that most of this logic could be upstreamed to any `Store`,
which is what this commit does.
2020-12-17 11:35:24 +01:00
Eelco Dolstra
ae3c3e3bb2
Merge pull request #4370 from NixOS/ca/more-precise-build-noop
Better detect when `buildPaths` would be a no-op
2020-12-16 14:54:04 +01:00
regnat
cac8d5b742 Don't ignore an absent drv file in queryPartialDrvOutputMap
This ignore was here because `queryPartialDrvOutputMap` was used both
1. as a cache to avoid having to re-read the derivation (when gc-ing for
example), and
2. as the source of truth for ca realisations

The use-case 2. required it to be able to work even when the derivation
wasn't there anymore (see https://github.com/NixOS/nix/issues/4138).
However, this use-case is now handled by `queryRealisation`, meaning
that we can safely error out if the derivation isn't there anymore
2020-12-16 13:36:17 +01:00
Eelco Dolstra
3765174691
Merge pull request #4348 from NixOS/ca/use-hashmodulo
Use the hash modulo in the derivation outputs
2020-12-16 12:48:44 +01:00
regnat
962b82ef25 Fix BinaryCacheStore::registerDrvOutput
Was crashing because coercing a json document into a string is only
valid if the json is a string, otherwise we need to call `.dump()`
2020-12-16 10:54:36 +01:00
regnat
6e899278d3 Better detect when buildPaths would be a no-op
`buildPaths` can be called even for stores where it's not defined in case it's
bound to be a no-op.
The “no-op detection” mechanism was only detecting the case wher `buildPaths`
was called on a set of (non-drv) paths that were already present on the store.

This commit extends this mechanism to also detect the case where `buildPaths`
is called on a set of derivation outputs which are already built on the store.

This only works with the ca-derivations flag. It could be possible to
extend this to also work without it, but it would add quite a bit of
complexity, and it's not used without it anyways.
2020-12-16 10:36:16 +01:00
regnat
7080321618 Use the fs accessor for readInvalidDerivation
Extend `FSAccessor::readFile` to allow not checking that the path is a
valid one, and rewrite `readInvalidDerivation` using this extended
`readFile`.

Several places in the code use `readInvalidDerivation`, either because
they need to read a derivation that has been written in the store but
not registered yet, or more generally to prevent a deadlock because
`readDerivation` tries to lock the state, so can't be called from a
place where the lock is already held.
However, `readInvalidDerivation` implicitely assumes that the store is a
`LocalFSStore`, which isn't always the case.

The concrete motivation for this is that it's required for `nix copy
--from someBinaryCache` to work, which is tremendously useful for the
tests.
2020-12-15 20:10:46 +01:00
regnat
44c3fbc6e0 Fix addTextToStore for binary caches
Because of a too eager refactoring, `addTextToStore` used to throw an
error because the input wasn't a valid nar.

Partially revert that refactoring to wrap the text into a proper nar
(using `dumpString`) to make this method work again
2020-12-15 09:38:19 +01:00
Eelco Dolstra
f2f60bf5d6
Merge pull request #4330 from NixOS/ca/properly-store-outputs
Properly store the outputs of CA derivations − take 2
2020-12-14 15:01:23 +01:00
Eelco Dolstra
27b5ff354e
Merge pull request #4351 from Ma27/json-errtrace
primops/fromJSON: add error position in case of parse error
2020-12-14 11:27:44 +01:00
Maximilian Bosch
f890830b33
primops/fromJSON: add error position in case of parse error
This makes it easier to track down where invalid JSON was passed to
`builtins.fromJSON`.
2020-12-13 13:55:32 +01:00
Eelco Dolstra
92438c70d2
Merge pull request #4352 from jonringer/allow-private-caches
treat s3 permission errors as file-not-found
2020-12-13 13:39:20 +01:00
Silvan Mosberger
730b152b19
Make Value::type private
This is an implementation detail and shouldn't be used. Use normalType()
and the various is<Type> functions instead
2020-12-12 03:31:52 +01:00
Silvan Mosberger
bf98903967
Add ValueType checking functions for types that have the same NormalType 2020-12-12 03:31:50 +01:00
Silvan Mosberger
22ead43a0b
Use Value::normalType on all forced values instead of Value::type 2020-12-12 03:31:48 +01:00
Silvan Mosberger
9f056f7afd
Introduce Value type setters and make use of them 2020-12-12 03:31:48 +01:00
Silvan Mosberger
fa307875e9
Introduce NormalType for the normal type of a Value
This will be useful to abstract over the ValueType implementation
details

Make use of it already to replace the showType(ValueType) function
2020-12-12 03:31:46 +01:00
regnat
e9b39f6004 Restrict the operations on drv outputs in recursive Nix
There's currently no way to properly filter them, so disallow them
altogether instead.
2020-12-11 21:17:25 +01:00
regnat
bab1cda0e6 Use the hash modulo in the derivation outputs
Rather than storing the derivation outputs as `drvPath!outputName` internally,
store them as `drvHashModulo!outputName` (or `outputHash!outputName` for
fixed-output derivations).

This makes the storage slightly more opaque, but enables an earlier
cutoff in cases where a fixed-output dependency changes (but keeps the
same output hash) − same as what we already do for input-addressed
derivations.
2020-12-11 21:17:23 +01:00
regnat
8914e01e37 Store the realisations as JSON in the binary cache
Fix #4332
2020-12-11 21:05:09 +01:00
regnat
3ac9d74eb1 Rework the db schema for derivation outputs
Add a new table for tracking the derivation output mappings.

We used to hijack the `DerivationOutputs` table for that, but (despite its
name), it isn't a really good fit:

- Its entries depend on the drv being a valid path, making it play badly with
  garbage collection and preventing us to copy a drv output without copying
  the whole drv closure too;
- It dosen't guaranty that the output path exists;

By using a different table, we can experiment with a different schema better
suited for tracking the output mappings of CA derivations.
(incidentally, this also fixes #4138)
2020-12-11 20:41:32 +01:00
regnat
58cdab64ac Store metadata about drv outputs realisations
For each known realisation, store:
- its output
- its output path

This comes with a set of needed changes:

- New `realisations` module declaring the types needed for describing
  these mappings
- New `Store::registerDrvOutput` method registering all the needed informations
  about a derivation output (also replaces `LocalStore::linkDeriverToPath`)
- new `Store::queryRealisation` method to retrieve the informations for a
  derivations

This introcudes some redundancy on the remote-store side between
`wopQueryDerivationOutputMap` and `wopQueryRealisation`.
However we might need to keep both (regardless of backwards compat)
because we sometimes need to get some infos for all the outputs of a
derivation (where `wopQueryDerivationOutputMap` is handy), but all the
stores can't implement it − because listing all the outputs of a
derivation isn't really possible for binary caches where the server
doesn't allow to list a directory.
2020-12-11 20:41:32 +01:00
Michael Bishop
63b3536f50
treat s3 permission errors as file-not-found
Signed-off-by: Jonathan Ringer <jonringer117@gmail.com>
2020-12-11 09:49:24 -08:00
Eelco Dolstra
9c143c411b
Merge pull request #4350 from NixOS/ca/fix-build-with-nix-command
Fix the `nix` command with CA derivations
2020-12-11 15:01:49 +01:00
regnat
eb45308109 Fix the nix command with CA derivations
Prevents a crash because most `nix` subcommands assumed that derivations
know their output path, which isn't the case for CA derivations
2020-12-11 10:28:09 +01:00
Eelco Dolstra
c6a1bcd0ec nix store make-content-addressable: Show rewritten path 2020-12-10 17:11:56 +01:00
Eelco Dolstra
a8f533b664 Add lvlNotice log level
This is like syslog's LOG_NOTICE: "normal, but significant,
condition".
2020-12-10 16:41:24 +01:00
Eelco Dolstra
253571e4ec
Merge pull request #4342 from tweag/fix-remote-build-hook
fix remote build hook
2020-12-09 12:40:00 +01:00
Maximilian Bosch
93a8a005de
libstore/openStore: fix stores with IPv6 addresses
In `nixStable` (2.3.7 to be precise) it's possible to connect to stores
using an IPv6 address:

  nix ping-store --store ssh://root@2001:db8::1

This is also useful for `nixops(1)` where you could specify an IPv6
address in `deployment.targetHost`.

However, this behavior is broken on `nixUnstable` and fails with the
following error:

  $ nix store ping --store ssh://root@2001:db8::1
  don't know how to open Nix store 'ssh://root@2001:db8::1'

This happened because `openStore` from `libstore` uses the `parseURL`
function from `libfetchers` which expects a valid URL as defined in
RFC2732. However, this is unsupported by `ssh(1)`:

  $ nix store ping --store 'ssh://root@[2001:db8::1]'
  cannot connect to 'root@[2001:db8::1]'

This patch now allows both ways of specifying a store (`root@2001:db8::1`) and
also `root@[2001:db8::1]` since the latter one is useful to pass query
parameters to the remote store.

In order to achieve this, the following changes were made:

* The URL regex from `url-parts.hh` now allows an IPv6 address in the
  form `2001:db8::1` and also `[2001:db8::1]`.

* In `libstore`, a new function named `extractConnStr` ensures that a
  proper URL is passed to e.g. `ssh(1)`:

  * If a URL looks like either `[2001:db8::1]` or `root@[2001:db8::1]`,
    the brackets will be removed using a regex. No additional validation
    is done here as only strings parsed by `parseURL` are expected.

  * In any other case, the string will be left untouched.

* The rules above only apply for `LegacySSHStore` and `SSHStore` (a.k.a
  `ssh://` and `ssh-ng://`).

Unresolved questions:

* I'm not really sure whether we want to allow both variants of IPv6
  addresses in the URL parser. However it should be noted that both seem
  to be possible according to RFC2732:

  > This document incudes an update to the generic syntax for Uniform
  > Resource Identifiers defined in RFC 2396 [URL].  It defines a syntax
  > for IPv6 addresses and allows the use of "[" and "]" within a URI
  > explicitly for this reserved purpose.

* Currently, it's not supported to specify a port number behind the
  hostname, however it seems as this is not really supported by the URL
  parser. Hence, this is probably out of scope here.
2020-12-09 12:23:29 +01:00
regnat
c87267c2a4 Store the final drv outputs in memory when building remotely
The `DerivationGoal` has a variable storing the “final” derivation
output paths that is used (amongst other things) to fill the environment
for the post build hook. However this variable wasn't set when the
build-hook is used, causing a crash when both hooks are used together.

Fix this by setting this variable (from the informations in the db) after a run
of the post build hook.
2020-12-09 10:45:12 +01:00
regnat
6758e65612 Revert "Re-query for the derivation outputs in the post-build-hook"
This reverts commit 1b1e076033.

Using `queryPartialDerivationOutputMap` assumes that the derivation
exists locally which isn't the case for remote builders.
2020-12-09 09:44:07 +01:00
Eelco Dolstra
82e5511594
Merge pull request #4325 from tweag/hide-local-store-sql-statements
Hide the sqlite statements declarations for the local store
2020-12-08 14:04:16 +01:00
regnat
c0f21f08f8 Hide the sqlite statements declarations for the local store
These have no need to be in the public interface and it causes spurious
rebuilds each time one wants to add or remove a new statement.
2020-12-08 13:29:13 +01:00
regnat
1b1e076033 Re-query for the derivation outputs in the post-build-hook
We can't assume that the runtime state knows about them as they might have
been built remotely, in which case we must query the db again to get
them.
2020-12-08 11:11:02 +01:00
Matthew Bauer
aa07502009 Always default to cache.nixos.org even when different nix store dir
Since 0744f7f, it is now useful to have cache.nixos.org in substituers
even if /nix/store is not the Nix Store Dir. This can always be
overridden via configuration, though.
2020-12-06 23:04:42 -06:00
Eelco Dolstra
1d1a85eb0a
Merge pull request #4318 from matthewbauer/add-slash-to-trusted-binary-cache
Canonicalize binary caches with ‘/’ when one is missing
2020-12-05 10:17:17 +01:00
Matthew Bauer
b9a00fd15b Canonicalize binary caches with ‘/’ when one is missing
This checks if there is a trusted substituter with a slash, so
trusting https://cache.nixos.org also implies https://cache.nixos.org/
is trusted.
2020-12-04 22:17:19 -06:00
Matthew Bauer
3c9b7029ba Use com.apple.oahd.plist for rosetta 2 detection 2020-12-04 13:26:53 -06:00
Stéphan Kochen
e20a3ec756 Fix compatibility with newer AWS SDKs
Tested against AWS SDK 1.8.99. Fixes #3201.
2020-12-04 19:36:09 +01:00
Eelco Dolstra
f337aa7099 Split 'nix store add-to-store' into 'add-path' and 'add-file'
This makes it consistent with 'nix hash <path|file>'.
2020-12-04 00:59:24 +01:00
Eelco Dolstra
fa8dad10ed Typo 2020-12-03 23:26:23 +01:00
Eelco Dolstra
ea2062a2d9 Move most store-related commands to 'nix store' 2020-12-03 23:22:22 +01:00
Eelco Dolstra
a1cd805cba Add 'nix nar dump-path'
This only differs from 'nix store dump-path' in that the path doesn't
need to be a store path.
2020-12-03 22:52:01 +01:00
Eelco Dolstra
af373c2ece Add deprecated aliases for renamed commands 2020-12-03 22:45:44 +01:00
Matthew Bauer
4b9acf4e21 Use posix_spawn_setbinpref_np to advise which architecture to run
When running universal binaries like /bin/bash, Darwin XNU will choose
which architecture of the binary to use based on "binary preferences".
This change sets that to the current platform for aarch64 and x86_64
builds. In addition it now uses posix_spawn instead of the usual
execve. Note, that this does not prevent the other architecture from
being run, just advises which to use.

Unfortunately, posix_spawnattr_setbinpref_np does not appear to be
inherited by child processes in x86_64 Rosetta 2 translations, meaning
that this will not always work as expected.

For example:

  {
    arm = derivation {
      name = "test";
      system = "aarch64-darwin";
      builder = "/bin/bash";
      args = [ "-e" (builtins.toFile "test" ''
        set -x
        /usr/sbin/sysctl sysctl.proc_translated
        /usr/sbin/sysctl sysctl.proc_native
        [ "$(/usr/bin/arch)" = arm64 ]
        /usr/bin/touch $out
      '') ];
    };
    rosetta = derivation {
      name = "test";
      system = "x86_64-darwin";
      builder = "/bin/bash";
      args = [ "-e" (builtins.toFile "test" ''
        set -x
        /usr/sbin/sysctl sysctl.proc_translated
        /usr/sbin/sysctl sysctl.proc_native
        [ "$(/usr/bin/arch)" = i386 ]
        echo It works!
        /usr/bin/touch $out
      '') ];
    };
  }

`arm' fails on x86_64-compiled Nix, but `arm' and `rosetta' succeed on
aarch64-compiled Nix. I suspect there is a way to fix this since:

  $ /usr/bin/arch -arch x86_64 /bin/bash \
    -c '/usr/bin/arch -arch arm64e /bin/bash -c /usr/bin/arch'
  arm64

seems to work correctly. We may need to wait for Apple to update
system_cmds in opensource.apple.com to find out how though.
2020-12-03 15:41:59 -06:00
Matthew Bauer
9b1824ecbd Add extraPlatforms for Rosetta 2 macOS
macOS systems with ARM64 can utilize a translation layer at
/Library/Apple/usr/libexec/oah to run x86_64 binaries. This change
makes Nix recognize that and it to "extra-platforms". Note that there
are two cases here since Nix could be built for either x86_64 or
aarch64. In either case, we can switch to the other architecture.
Unfortunately there is not a good way to prevent aarch64 binaries from
being run in x86_64 contexts or vice versa - programs can always
execute programs for the other architecture.
2020-12-03 15:41:43 -06:00
Eelco Dolstra
0c15ae5d4b Add FIXME 2020-12-03 20:31:45 +01:00
Eelco Dolstra
ef583303f0 Move NAR-related commands to 'nix nar' 2020-12-03 18:09:02 +01:00
Eelco Dolstra
79c1967ded Introduce 'nix store' command 2020-12-03 18:07:13 +01:00
Eelco Dolstra
5781f45c46 Allow registering subcommands of subcommands 2020-12-03 17:55:55 +01:00
Eelco Dolstra
b2d6c6161e Move 'nix hash-*' and 'nix to-*' to 'nix hash'
From the 'nix' UX review.
2020-12-03 17:55:55 +01:00
Eelco Dolstra
8ad2c9c4b9 Remove 'dist' target
We're not producing source tarballs anymore so this has been
bitrotting.
2020-12-03 16:17:58 +01:00
Eelco Dolstra
1b0ca3866b nix add-to-store: Move markdown docs into a separate file 2020-12-03 16:17:58 +01:00
Eelco Dolstra
c3c858ac6d Make doc() return arbitrary Markdown rather than the contents of the "Description" section
Thus we can return the examples section (and any other sections) from
doc() and don't need examples() anymore.
2020-12-03 16:17:58 +01:00
Eelco Dolstra
0bd060f23a
Merge pull request #4308 from tweag/properly-test-early-cutoff
Properly test the early cutoff for CA derivations
2020-12-03 14:45:29 +01:00
Eelco Dolstra
4f25644a13
Merge pull request #4304 from NixOS/separate-manpages
Separate manpages for 'nix' subcommands
2020-12-03 13:38:29 +01:00
regnat
0afab668fa Don't fail early when -j0 is passed
If the build closure contains some CA derivations, then we can't know
ahead-of-time that we won't build anything as early-cutoff might come-in
at a laster stage
2020-12-03 13:24:36 +01:00
Eelco Dolstra
df552a2645 nix eval: Add option to write a directory
This is useful for generating the nix manpages, but it may have other
applications (like generating configuration files without a Nix store).
2020-12-02 23:23:23 +01:00
Eelco Dolstra
148608ba6d Add 'nix help' 2020-12-02 23:23:23 +01:00
Greg Hale
d8fc1bb7b0 fix tokens documentation 2020-12-02 10:15:18 -05:00
regnat
0d9e1af695 Remove an unknown pragma gcc warning 2020-12-02 14:33:20 +01:00
regnat
a8a96dbaf8 Add forgotten override annotation 2020-12-02 14:23:38 +01:00
Eelco Dolstra
1b79b5b983 read(): Use char * instead of unsigned char *
This gets rid of some pointless casts.
2020-12-02 14:17:27 +01:00
Eelco Dolstra
faa31f4084 Sink: Use std::string_view 2020-12-02 14:17:27 +01:00
Eelco Dolstra
aa68486112 writeFull/writeFile: Use std::string_view 2020-12-02 14:17:27 +01:00
Eelco Dolstra
e5cf501c77
Merge pull request #4284 from tweag/fixed-output-depending-on-ca
Allow fixed-output derivations to depend on (floating) content-addressed ones
2020-12-01 20:25:41 +01:00
regnat
438977731c shut up clang warnings
- Fix some class/struct discrepancies
- Explicit the overloading of `run` in the `Cmd*` classes
- Ignore a warning in the generated lexer
2020-12-01 15:04:03 +01:00
Eelco Dolstra
88798613ee replaceStrings(): Use std::string_view 2020-12-01 13:45:43 +01:00
Eelco Dolstra
c0d1354b7d Macro hygiene 2020-12-01 13:45:06 +01:00
Eelco Dolstra
3b7e00ce22 Move primeCache() to Worker::run()
We need the missing path info to communicate the worker's remaining
goals to the progress bar.
2020-12-01 13:44:48 +01:00
Eelco Dolstra
5927624473 Lower verbosity for 'Failed to find a machine' message 2020-12-01 13:43:36 +01:00
Eelco Dolstra
e224c16d28 Macro hygiene 2020-12-01 13:43:33 +01:00
regnat
9bd8184f1f Allow fixed-output derivations to depend on (floating) content-addressed ones
Fix an overlook of https://github.com/NixOS/nix/pull/4056
2020-11-27 15:39:24 +01:00
Eelco Dolstra
05d9442f68 builtins.fetchGit: Fix shortRev attribute for dirty trees 2020-11-26 21:45:28 +01:00
Dominique Martinet
1fd13d67e8 archive: disable preallocate-contents by default
using fallocate() to preallocate files space does more harm than good:
 - breaks compression on btrfs
 - has been called "not the right thing to do" by xfs developers
(because delayed allocation that most filesystems implement leads to smarter
allocation than what the filesystem needs to do if we upfront fallocate files)
2020-11-26 14:26:57 +01:00
Eelco Dolstra
8252a44e96 Move to separate file 2020-11-26 13:16:36 +01:00
Eelco Dolstra
9a586e34ac Record trusted/untrusted settings in ~/.local/share/nix 2020-11-26 13:11:07 +01:00
Eelco Dolstra
0287f83057 Ask for confirmation before allowing flake Nix configuration settings 2020-11-26 12:37:23 +01:00
regnat
13c557fe82 fix the hash rewriting for ca-derivations 2020-11-25 11:33:00 +01:00
Lily Ballard
437189e446 Escape filename given to nix-shell in shebang mode
This prevents spaces or other metacharacters from causing nix-shell to
execute the wrong path.

Fixes #4229.
2020-11-24 15:08:37 -08:00
Luke Granger-Brown
226116f482 fetchMercurial: set HGPLAIN when invoking hg
Without setting HGPLAIN, the user's environment leaks into
hg invocations, which means that the output may not be in the
expected format.

HGPLAIN is the Mercurial-recommended solution for this in that
it's intended for uses by scripts and programs which are looking
to parse Mercurial's output in a consistent manner.
2020-11-23 16:12:33 +00:00
Wil Taylor
07603890d2 Removed reference check from bundler command 2020-11-23 21:19:40 +10:00
Wil Taylor
c3bad73e27 Added switch 2020-11-21 14:28:49 +10:00
Eelco Dolstra
4dcb183af3 AttrCursor::getStringWithContext(): Force re-evaluation if the cached context is not valid
Fixes #4236.
2020-11-19 20:59:36 +01:00
Eelco Dolstra
0327580e54 Fix assertion failure in LockFile::LockFile()
Fixes #4241.
2020-11-19 20:31:30 +01:00
John Ericson
2113ae2d85 Make drv hash modulo memo table thread-safe
Let's get one step closer to the daemon not needing to fork.
2020-11-19 16:50:06 +00:00
Eelco Dolstra
4dbd05e933
Merge pull request #4189 from edolstra/flake-config
Allow nix.conf options to be set in flake.nix
2020-11-17 15:39:14 +01:00
Eelco Dolstra
f4e790cc85
Merge pull request #4182 from mkenigs/fix-1930
Print built derivations as json for build
2020-11-17 14:59:49 +01:00
Eelco Dolstra
df5c69a94e
Merge pull request #4180 from Ma27/ssh-ng-substitute
Allow substituting paths when building remotely using `ssh-ng://`
2020-11-17 14:01:04 +01:00
Eelco Dolstra
e6b7c7b79c Cleanup 2020-11-17 13:58:55 +01:00
Eelco Dolstra
bccff827dc Fix deadlock in IFD through the daemon
Fixes #4235.
2020-11-17 13:50:36 +01:00
regnat
7de21f6664 Make the sql debug statements more useful
Print the expanded sql query (with the variables bound to their value) rather
than the original one in case of error
2020-11-17 10:05:45 +01:00
Eelco Dolstra
ef84c780bb
filterANSIEscapes(): Handle UTF-8 characters 2020-11-16 16:41:53 +01:00
Eelco Dolstra
0d6419ad87
Merge pull request #4262 from jbaum98/master
Fix deadlock in nix-store when max-connections=1
2020-11-16 10:20:36 +01:00
Jake Waksbaum
01db455733 Fix deadlock in nix-store when max-connections=1
This fixes a bug I encountered where `nix-store -qR` will deadlock when
the `--include-outputs` flag is passed and `max-connections=1`.

The deadlock occurs because `RemoteStore::queryDerivationOutputs` takes
the only connection from the connection pool and uses it to check the
daemon version. If the version is new enough, it calls
`Store::queryDerivationOutputs`, which eventually calls
`RemoteStore::queryPartialDerivationOutputMap`, where we take another
connection from the connection pool to check the version again. Because
we still haven't released the connection from the caller, this waits for
a connection to be available, causing a deadlock.

This diff solves the issue by using `getProtocol` to check the protocol
version in the caller `RemoteStore::queryDerivationOutputs`, which
immediately frees the connection back to the pool before returning the
protocol version. That way we've already freed the connection by the
time we call `RemoteStore::queryPartialDerivationOutputMap`.
2020-11-16 02:35:50 -05:00
Robert Hensing
ac5081d280 nix-build: Fix #4197 output order regression 2020-11-13 17:49:27 +01:00
Christian Höppner
c4c3c15c19 Fix default nix-path
The default nix-path values for nixpkgs and root channels were
incorrect.
2020-11-12 15:46:08 +00:00
Ben Burdette
b327de9c2d change message 2020-11-11 11:09:59 -07:00
Ben Burdette
8895fa70a4 pare down the error message 2020-11-11 11:05:21 -07:00
Ben Burdette
3edfe6090e missing argument error 2020-11-11 09:29:32 -07:00
Matthew Kenigsberg
8abb80a478 Print built derivations as json for build
Add --json option to nix build to allow machine readable output on
stdout with all built derivations

Fixes #1930
2020-11-11 10:27:02 -06:00
Ben Burdette
7d9037035e usage example location 2020-11-11 09:21:26 -07:00
Eelco Dolstra
4badb6943f Fix use of dirty Git/Mercurial inputs with chroot stores
Fixes:

  $ nix build --store /tmp/nix /home/eelco/Dev/patchelf#hydraJobs.build.x86_64-linux
  warning: Git tree '/home/eelco/Dev/patchelf' is dirty
  error: --- RestrictedPathError ------------------------------------------------------------------------------------------- nix
  access to path '/tmp/nix/nix/store/xmkvfmffk7xfnazykb5kx999aika8an4-source/flake.nix' is forbidden in restricted mode
  (use '--show-trace' to show detailed location information)
2020-11-10 23:22:45 +01:00
Eelco Dolstra
cdc840d60b
Merge pull request #4242 from wizeman/fix-stack-overflow
Fix stack overflow introduced in #4206
2020-11-10 11:51:00 +01:00
Eelco Dolstra
3f680c1dcc
Merge pull request #4233 from Kha/master
nix develop: Preserve stdin with `-c`
2020-11-10 10:51:48 +01:00
Ricardo M. Correia
108a2dab7e Fix stack overflow introduced in #4206 2020-11-10 04:25:24 +01:00
Ben Burdette
d8ef423a18 error message formatting 2020-11-09 19:16:50 -07:00
Ben Burdette
9f2b25ce55 remove unused ftn; reformat line breaks 2020-11-09 17:17:47 -07:00
Ben Burdette
6c2933a8d7 add position 2020-11-09 17:04:52 -07:00
Ben Burdette
107c91f5fe auto-call error 2020-11-09 16:48:35 -07:00
Sebastian Ullrich
fb7735e4cf nix develop: Preserve stdin with -c 2020-11-09 22:43:14 +01:00
Eelco Dolstra
b87f84cf55
Fix appending to Setting<StringSet>
Fixes: warning: unknown setting 'extra-sandbox-paths'
2020-11-09 15:04:34 +01:00
Maximilian Bosch
3a63fc6cd5
Allow substituting paths when building remotely using ssh-ng://
Until now, it was not possible to substitute missing paths from e.g.
`https://cache.nixos.org` on a remote server when building on it using
the new `ssh-ng` protocol.

This is because every store implementation except legacy `ssh://`
ignores the substitution flag passed to `Store::queryValidPaths` while
the `legacy-ssh-store` substitutes the remote store using
`cmdQueryValidPaths` when the remote store is opened with `nix-store
--serve`.

This patch slightly modifies the daemon protocol to allow passing an
integer value suggesting whether to substitute missing paths during
`wopQueryValidPaths`. To implement this on the daemon-side, the
substitution logic from `nix-store --serve` has been moved into a
protected method named `Store::substitutePaths` which gets currently
called from `LocalStore::queryValidPaths` and `Store::queryValidPaths`
if `maybeSubstitute` is `true`.

Fixes #2770
2020-11-05 20:12:37 +01:00
Eelco Dolstra
387f824cab
Merge pull request #4206 from hercules-ci/fix-coroutine-gc
Fix memory corruption caused by GC-invisible coroutine stacks
2020-11-05 10:18:31 +01:00
Eelco Dolstra
5e6eabe155 Fix error message 'assertion failed at' 2020-11-03 14:45:24 +01:00
Eelco Dolstra
e8c379555f LocalStore: Get rid of recursive_mutex 2020-11-03 14:45:24 +01:00
Eelco Dolstra
797a52e31d Add FIXME 2020-11-03 14:45:24 +01:00
Eelco Dolstra
550e11f077 nix repl: Fix handling of multi-line expressions 2020-11-02 19:07:37 +01:00
Eelco Dolstra
7cf874c17d Don't use readDerivation() in addValidPath()
readDerivation() requires a valid path.

Fixes #4210.
2020-11-02 18:46:44 +01:00
Eelco Dolstra
ab2ef851b6
Merge pull request #4207 from hercules-ci/fix-RemoteStore-filterSource-deadlock
Fix RemoteStore pool deadlock in filterSource etc
2020-11-02 14:46:10 +01:00
Eelco Dolstra
db5424bf09 Don't send eval-related settings to the daemon 2020-11-02 13:57:58 +01:00
Robert Hensing
e8a45d07bc Restore RestrictedStore.addToStoreFromDump implementation
It was accidentally removed in commit ca30abb3fb
2020-10-31 23:56:03 +01:00
Robert Hensing
b43c13a916 BoehmGCStackAllocator: increase stack size to 8MB
The default stack size was not based on the normal stack size and
was too small.
2020-10-30 23:18:26 +01:00
Robert Hensing
2192cac634 Fix RemoteStore pool deadlock in filterSource etc 2020-10-30 21:47:34 +01:00
Robert Hensing
c4d903ddb0 Fix memory corruption caused by GC-invisible coroutine stacks
Crucially this introduces BoehmGCStackAllocator, but it also
adds a bunch of wiring to avoid making libutil depend on bdw-gc.

Part of the solutions for #4178, #4200
2020-10-30 21:21:59 +01:00
Eelco Dolstra
dc5696b84f Fix test 2020-10-30 12:00:53 +01:00
Eelco Dolstra
7f56cf67ba Fix assertion failure in tab completion for --option 2020-10-29 18:26:35 +01:00
Eelco Dolstra
ff4dea63c9 Generalize extra-* settings
This removes the extra-substituters and extra-sandbox-paths settings
and instead makes every array setting extensible by setting
"extra-<name> = <value>" in the configuration file or passing
"--<name> <value>" on the command line.
2020-10-29 18:17:39 +01:00
Eelco Dolstra
bb8e837e4c
Merge pull request #4199 from stefanjaax/replaceWantedBySpecified
Alter "wanted:" to "specified:" in hash mismatch output
2020-10-29 07:38:38 +01:00
Eelco Dolstra
662e67f8de
Merge pull request #4198 from mkenigs/capitalize-JSON
Capitalize JSON for consistency
2020-10-29 07:34:34 +01:00
stev
869c0321ff Alter "wanted:" to "specified:" in hash mismatch output
This makes it even clearer which of the two hashes was specified in the
nix files. Some may think that "wanted" and "got" is obvious, but:
"got" could mean "got in nix file" and "wanted" could mean "want to see in nix file".
2020-10-29 00:33:14 +01:00
Matthew Kenigsberg
6a4bf535d8 Capitalize JSON for consistency 2020-10-28 17:54:28 -05:00
Eelco Dolstra
a5019f0508 Consistency 2020-10-28 20:45:57 +01:00
Eelco Dolstra
02a1facbdc
Merge pull request #4056 from tweag/non-ca-depending-on-ca
Allow non-CA derivations to depend on CA ones
2020-10-27 17:38:29 +01:00
regnat
bc081bcd81 Inline unkownHashes
See https://github.com/NixOS/nix/pull/4056#discussion_r493661632
2020-10-27 07:29:25 +01:00
regnat
c092fa4702 Allow non-CA derivations to depend on CA derivations 2020-10-27 07:29:23 +01:00
Eelco Dolstra
343239fc8a Allow nix.conf options to be set in flake.nix
This makes it possible to have per-project configuration in flake.nix,
e.g. binary caches and other stuff:

  nixConfig.bash-prompt-suffix = "ngi# ";
  nixConfig.substituters = [ "https://cache.ngi0.nixos.org/" ];
2020-10-26 20:45:39 +01:00
Eelco Dolstra
731edf0d9b isTrivial(): Support trivial lists 2020-10-26 20:37:11 +01:00
Eelco Dolstra
14aecbb288 BaseSetting<StringMap>::set(): Don't append to previous value 2020-10-26 20:36:46 +01:00
Eelco Dolstra
b875b8f45c Remove edition field 2020-10-26 17:59:36 +01:00
Eelco Dolstra
9d5e9ef0da Move Explicit 2020-10-26 17:01:20 +01:00
Eelco Dolstra
dc7d1322ef Make the prompt used in development shells configurable 2020-10-26 14:24:25 +01:00
Eelco Dolstra
750ce500c2 Fix clang build 2020-10-22 13:40:30 +02:00
Eelco Dolstra
f9438fb64a nix develop: Add --redirect flag to redirect dependencies
This is primarily useful if you're hacking simultaneously on a package
and one of its dependencies. E.g. if you're hacking on Hydra and Nix,
you would start a dev shell for Nix, and then a dev shell for Hydra as
follows:

  $ nix develop \
    --redirect .#hydraJobs.build.x86_64-linux.nix ~/Dev/nix/outputs/out \
    --redirect .#hydraJobs.build.x86_64-linux.nix.dev ~/Dev/nix/outputs/dev

(This assumes hydraJobs.build.x86_64-linux has a passthru.nix
attribute. You can also use a store path.)

This causes all references in the environment to those store paths to
be rewritten to ~/Dev/nix/outputs/{out,dev}. Note: unfortunately, you
may need to set LD_LIBRARY_PATH=~/Dev/nix/outputs/out/lib because
Nixpkgs' ld-wrapper only adds -rpath entries for -L flags that point
to the Nix store.
2020-10-22 13:40:30 +02:00
Eelco Dolstra
e556a1beb7 nix develop: Handle 'declare -ax' in bash output
Fixes 'nix develop nixpkgs#qpdfview'.
2020-10-21 17:54:21 +02:00
Christian Kampka
461cf2b856
Add NIX_CONFIG env var for applying nix.conf overrides 2020-10-21 13:41:26 +02:00
Matthew Kenigsberg
f6aaac2b59 Make bash non-interactive for nix develop --phase
Fix #3975: Currently if Ctrl-C is pressed during a phase, the interactive subshell
is not exited. Removing --rcfile when --phase is present makes bash
non-interactive
2020-10-20 12:00:30 -05:00
Eelco Dolstra
9635fb77bd
Merge pull request #4080 from kquick/kwq/flake-int-doc
Add some internal documentation for flake support objects.
2020-10-19 11:29:12 +02:00
Eelco Dolstra
fda835b231
Merge pull request #4143 from obsidiansystems/typed-goal-maps
Properly type the derivation and substitution goal maps
2020-10-18 18:12:21 +02:00
Robert Hensing
bd9eb5c743 DerivationGoal: only retry if output closure incomplete is only problem 2020-10-18 14:26:37 +02:00
Robert Hensing
94f1e4a441 Typo 2020-10-18 14:26:37 +02:00
John Ericson
7ed46c1574 Explain that upcast_goal is still a static cast 2020-10-17 21:50:12 +00:00
John Ericson
57d0432b39 Just use auto in two places. 2020-10-17 21:47:52 +00:00
John Ericson
e6f8ae56d8 tab -> space 2020-10-17 21:45:31 +00:00
John Ericson
619d262c97 Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-17 21:44:27 +00:00
John Ericson
64be1c15c2 Add missing include for MAX_PATH
And remove one that we didn't actually need to add
2020-10-15 19:05:17 +00:00
John Ericson
fccef6a7fa Merge remote-tracking branch 'upstream/master' into fix-and-ci-static-builds 2020-10-15 18:55:03 +00:00
John Ericson
0fefc2a439 Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-14 20:49:01 +00:00
Eelco Dolstra
11882d7c7c Create /etc/passwd *after* figuring out the sandbox uid/gid
Fixes build failures like

  # nix log /nix/store/gjaa0psfcmqvw7ivggsncx9w364p3s8s-sshd.conf-validated.drv
  No user exists for uid 30012
2020-10-14 12:20:58 +02:00
John Ericson
55592b253f Add some more docs 2020-10-13 18:04:24 +00:00
John Ericson
13804f126e Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-13 18:02:32 +00:00
Eelco Dolstra
2653801939 Merge branch 'split_build_cc' of https://github.com/obsidiansystems/nix 2020-10-13 15:36:55 +02:00
John Ericson
1b8ebe92dc Merge remote-tracking branch 'obsidian/split_build_cc' into typed-goal-maps 2020-10-12 20:47:22 +00:00
John Ericson
a73959e6be Merge remote-tracking branch 'upstream/master' into split_build_cc 2020-10-12 17:20:46 +00:00
John Ericson
d334fd4882 Merge branches 'split_build_hh-0', 'split_build_hh-1', 'split_build_hh-2', 'split_build_hh-3' and 'split_build_hh-4' into split_build_cc 2020-10-12 17:20:12 +00:00
John Ericson
542972f029 Trim worker.hh 2020-10-12 17:16:48 +00:00
John Ericson
0e2306204a Rename to hand-hold git (worker.hh) 2020-10-12 17:16:48 +00:00
John Ericson
3ffa3546bd Trim substitution-goal.hh 2020-10-12 17:16:25 +00:00
John Ericson
e77a2344d5 Rename to hand-hold git (substitution-goal.hh) 2020-10-12 17:16:25 +00:00
John Ericson
d585b4c54f Trim hook-instance.hh 2020-10-12 17:16:13 +00:00
John Ericson
10b749a156 Rename to hand-hold git (hook-instance.hh) 2020-10-12 17:16:13 +00:00
John Ericson
8067d32f2a Trim goal.hh 2020-10-12 17:16:00 +00:00
John Ericson
0d0e345cdc Rename to hand-hold git (goal.hh) 2020-10-12 17:16:00 +00:00
John Ericson
2ce726947a Trim derivation-goal.hh 2020-10-12 17:15:32 +00:00
John Ericson
4eb8c69853 Rename to hand-hold git (derivation-goal.hh) 2020-10-12 17:15:32 +00:00
John Ericson
f7099965bf Change .cc files to use split build headers 2020-10-12 17:08:52 +00:00
John Ericson
5a97621d6d Prepare for build/*.hh headers 2020-10-12 17:07:51 +00:00
volth
eee18f88dd
Handle amount of disk space saved by hard linking being negative
Fixes bogus messages like "currently hard linking saves 17592186044416.00 MiB".
2020-10-12 16:06:38 +00:00
John Ericson
15fdb7cc6b Split out uds-remote-store.{cc.hh} 2020-10-11 17:37:05 +00:00
John Ericson
38e3897162 Copy {uds-,}remote-store.{cc,hh}
This prepares for the splitting that happens in the next commit.
2020-10-11 17:18:26 +00:00
John Ericson
5c74a6147b Properly type the derivation and substitution goal maps
As a bonus, Worker::removeGoal is less inefficient.
2020-10-11 17:07:14 +00:00
Eelco Dolstra
b0fbf3a653
Merge pull request #4135 from obsidiansystems/split-local-fs-store-header
Split out `local-fs-store.hh`
2020-10-11 18:49:40 +02:00
John Ericson
8cc510fb79 Merge branches 'build-1', 'build-2', 'build-3', 'build-4', 'build-5', 'build-6', 'build-7' and 'build-8' into split_build_cc 2020-10-11 16:44:34 +00:00
John Ericson
bcb67e1ed8 Trim lock.cc 2020-10-11 16:44:19 +00:00
John Ericson
e0be04129b Rename to hand-hold git (lock.cc) 2020-10-11 16:44:14 +00:00
John Ericson
dbc588651c Trim lock.hh 2020-10-11 16:44:07 +00:00
John Ericson
d0004bfcab Rename to hand-hold git (lock.hh) 2020-10-11 16:43:59 +00:00
John Ericson
eed53ed87a Trim build/worker.cc 2020-10-11 16:43:51 +00:00
John Ericson
904e315dae Rename to hand-hold git (build/worker.cc) 2020-10-11 16:43:42 +00:00
John Ericson
d24ffe0eb1 Trim build/substitution-goal.cc 2020-10-11 16:43:24 +00:00
John Ericson
3633b3572b Rename to hand-hold git (build/substitution-goal.cc) 2020-10-11 16:43:18 +00:00
John Ericson
4bdff7d1b0 Trim build/local-store-build.cc 2020-10-11 16:43:12 +00:00
John Ericson
dc5225cde5 Rename to hand-hold git (build/local-store-build.cc) 2020-10-11 16:43:05 +00:00
John Ericson
159054f730 Trim build/hook-instance.cc 2020-10-11 16:42:35 +00:00
John Ericson
f0b8987299 Rename to hand-hold git (build/hook-instance.cc) 2020-10-11 16:42:08 +00:00
John Ericson
819fe848ac Trim build/goal.cc 2020-10-11 16:41:58 +00:00
John Ericson
184bfc301e Rename to hand-hold git (build/goal.cc) 2020-10-11 16:41:18 +00:00
John Ericson
3bab1c5bb0 Trim build/derivation-goal.cc 2020-10-11 16:41:11 +00:00
John Ericson
9629290eda Rename to hand-hold git (build/derivation-goal.cc) 2020-10-11 16:40:52 +00:00
John Ericson
a4f0fecb03 Trim build.hh 2020-10-11 16:40:34 +00:00
John Ericson
fc72cb0760 Rename to hand-hold git (build.hh) 2020-10-11 16:40:14 +00:00
John Ericson
428536fd75 Prepare for build/* files 2020-10-11 16:39:08 +00:00
John Ericson
aef44cbaa9 Split out commonChildInit 2020-10-11 16:38:46 +00:00
John Ericson
6cc1541782 Split out local-fs-store.hh
This matches the already-existing `local-fs-store.cc`.
2020-10-09 20:18:08 +00:00
Eelco Dolstra
725488b892 nix develop: Unset $HOSTNAME
This is set to "localhost" by stdenv which is probably not what you
want.
2020-10-09 22:03:18 +02:00
Eelco Dolstra
44349064f7 nix develop: Source ~/.bashrc
Fixes #4104.
2020-10-09 22:02:00 +02:00
Eelco Dolstra
59bd6e87a4 Completions::add(): Guard against newlines 2020-10-09 21:55:59 +02:00
Eelco Dolstra
ea4b2b985f
Merge pull request #4128 from tweag/extended-completions
Add a zsh completion script
2020-10-09 21:53:12 +02:00
John Ericson
39de73550d Merge remote-tracking branch 'upstream/master' into fix-and-ci-static-builds 2020-10-09 18:26:47 +00:00
Eelco Dolstra
e845d19ae3 Remove Lazy
This fixes a crash during startup when compiling Nix as a single
compilation unit.
2020-10-09 17:54:59 +02:00
Eelco Dolstra
87157b2bd3 writeFile(): Add error context to writeFull() failure
Issue #4092.
2020-10-09 16:02:53 +02:00
Eelco Dolstra
636ec17139
Remove stray DerivationOutputsAndPaths type 2020-10-09 15:41:24 +02:00
regnat
04e5d0e704 Add a description in the completion outputs
Make nix output completions in the form `completion\tdescription`.
This can't be used by bash (afaik), but other shells like zsh or fish
can display it along the completion choices
2020-10-09 09:39:51 +02:00
Eelco Dolstra
97ffc1e013 Dynamically disable user namespaces if CLONE_NEWUSER fails
This makes builds work inside nixos-enter.

Fixes #3145.
2020-10-07 22:46:01 +02:00
Eelco Dolstra
6aa64627c8 Support user namespaces being disabled
If max_user_namespaces is set to 0, then don't run the build in a user
namespace.

Fixes #4092.
2020-10-07 22:02:36 +02:00
Eelco Dolstra
f66bbd8c7b Doh 2020-10-07 21:25:06 +02:00
Eelco Dolstra
e705c24294 Tweak error messages 2020-10-07 17:28:43 +02:00
Eelco Dolstra
c43e882f54 Serialize exceptions from the daemon to the client 2020-10-07 17:13:54 +02:00
Eelco Dolstra
be149acfda Serialize exceptions from the sandbox process to the parent
Fixes #4118.
2020-10-07 16:34:03 +02:00
Eelco Dolstra
27ca87c46a Formatting 2020-10-07 16:33:19 +02:00
John Ericson
57d960dcd1 Remove generic std::optional<T> suppport from worker proto
See comment for rational; I think it's good to leave a comment lest
anyone is tempted to add such a sum-type instance again.

Fixes #4113
2020-10-07 12:50:37 +00:00
Maximilian Bosch
59f2dd8e8d
libfetchers/github: allow slashes in refs
Refs #4061
2020-10-06 20:08:51 +02:00
Eelco Dolstra
d761485010 Prevent a deadlock when user namespace setup fails
Observed on Centos 7 when user namespaces are disabled:
DerivationGoal::startBuilder() throws an exception, ~DerivationGoal()
waits for the child process to exit, but the child process hangs
forever in drainFD(userNamespaceSync.readSide.get()) in
DerivationGoal::runChild(). Not sure why the SIGKILL doesn't get
through.

Issue #4092.
2020-10-06 18:57:23 +02:00
Eelco Dolstra
ad143c5b3b Shut up some clang warnings 2020-10-06 14:52:30 +02:00
Eelco Dolstra
85c8be6286 Remove static variable name clashes
This was useful for an experiment with building Nix as a single
compilation unit. It's not very useful otherwise but also doesn't
hurt...
2020-10-06 13:49:20 +02:00
Eelco Dolstra
0419cd2695 Remove unneeded -lboost_* flags 2020-10-06 13:34:58 +02:00
Eelco Dolstra
636455c471 Remove 'using namespace fetchers' 2020-10-06 11:16:32 +02:00
Eelco Dolstra
6691256e79 Factor out common showBytes() 2020-10-06 10:40:49 +02:00
Eelco Dolstra
d0bb544128 Add missing #pragma once 2020-10-06 10:40:07 +02:00
Eelco Dolstra
88a667e49e
Fix s3:// store
Fixes https://github.com/NixOS/nixos-org-configurations/issues/123.
2020-10-05 17:53:30 +02:00
Eelco Dolstra
51c299213b
Merge pull request #3935 from obsidiansystems/binary-cache-addToStoreFromDump
Get rid of Hash::dummy from BinaryCacheStore
2020-10-05 14:41:28 +02:00
Eelco Dolstra
f3aba88737
Merge pull request #3895 from obsidiansystems/templated-daemon-protocol
More templated STL support for the daemon protocol
2020-10-05 14:40:27 +02:00
Fabian Möller
d5d196b0a1
Fix profile update in nix command 2020-10-02 12:10:31 +02:00
DavHau
f3280004e2 add more examples to --help of nix run 2020-10-01 11:34:13 +07:00
Eelco Dolstra
20a1e20d91 Style 2020-09-30 13:35:56 +02:00
Eelco Dolstra
274357eb6a Simplify 2020-09-30 12:09:18 +02:00
Eelco Dolstra
002ce8449d Merge branch 'access-tokens' of https://github.com/kquick/nix 2020-09-30 11:35:15 +02:00
John Ericson
69afaeace3 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-30 00:42:28 +00:00
John Ericson
b759701652 nix::worker_proto -> worker_proto 2020-09-30 00:41:18 +00:00
John Ericson
45a0ed82f0 Revert "Use template structs instead of phantoms"
This reverts commit 9ab07e99f5.
2020-09-30 00:39:06 +00:00
Kevin Quick
5e7838512e
Remove github-access-token in favor of access-token. 2020-09-29 16:26:34 -07:00
Eelco Dolstra
76f000712e Merge branch 'kwq/path-dir-query' of https://github.com/kquick/nix 2020-09-29 23:42:43 +02:00
Eelco Dolstra
64e9b3c83b nix registry list: Show 'dir' attribute
Issue #4050.
2020-09-29 23:33:16 +02:00
Kevin Quick
66c3959e8c
Merge branch 'master' into access-tokens 2020-09-29 08:32:06 -07:00
Eelco Dolstra
5999978a05 Make Headers an optional argument 2020-09-29 13:05:19 +02:00
Eelco Dolstra
de86abbf3f Cleanup 2020-09-29 12:55:06 +02:00
Eelco Dolstra
cebd2fc35d Merge branch 'github-api-token' of https://github.com/imalsogreg/nix 2020-09-29 12:17:00 +02:00
John Ericson
00135e13f4 Clarify comment a bit 2020-09-28 18:19:10 +00:00
Kevin Quick
887be7b6f2
Switch comment format from '// ...' to '/* ... */' for consistency. 2020-09-28 09:37:26 -07:00
Kevin Quick
128c98ab09
Clarification in the description of the FlakeInput. 2020-09-28 09:34:23 -07:00
Kevin Quick
5ae164b7cf
Update description of FlakeRef, incorporating suggestion. 2020-09-28 09:23:05 -07:00
Kevin Quick
bcb3da3b6b
Fix spelling error. 2020-09-28 08:58:14 -07:00
John Ericson
80e335bb58 Use drvPath2 and give it a better name 2020-09-28 15:43:56 +00:00
John Ericson
10202bbf29 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-28 15:39:11 +00:00
John Ericson
6c31297d80
Update src/libstore/binary-cache-store.cc
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-09-28 11:32:58 -04:00
Eelco Dolstra
649c465873
Merge pull request #4064 from serokell/balsoft/fix-max-jobs
Fix max-jobs option
2020-09-28 10:42:02 +02:00
Eelco Dolstra
5df687c316
Merge pull request #4077 from Ma27/left-whitespace-log
libmain/progress-bar: don't trim whitespace on the left
2020-09-28 10:40:00 +02:00
Kevin Quick
5885b0cfd8
Miscellaneous spelling fixes in comments. (#4071) 2020-09-27 20:04:06 +00:00
Kevin Quick
bd5328814f
Add some internal documentation for flake support objects. 2020-09-26 14:32:58 -07:00
Maximilian Bosch
a76fb07314
libmain/progress-bar: don't trim whitespace on the left
When running `nix build -L` it can be fairly hard to read the output if
the build program intentionally renders whitespace on the left. A
typical example is `g++` displaying compilation errors.

With this patch, the whitespace on the left is retained to make the log
more readable:

```
foo> no configure script, doing nothing
foo> building
foo> foobar.cc: In function 'int main()':
foo> foobar.cc:5:5: error: 'wrong_func' was not declared in this scope
foo>     5 |     wrong_func(1);
foo>       |     ^~~~~~~~~~
error: --- Error ------------------------------------------------------------------------------------- nix
error: --- Error --- nix-daemon
builder for '/nix/store/i1q76cw6cyh91raaqg5p5isd1l2x6rx2-foo-1.0.drv' failed with exit code 1
```
2020-09-26 17:38:11 +02:00
John Ericson
25fffdda86
Remove redundant nar hash and size setting
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2020-09-26 10:17:30 -04:00
John Ericson
1832436526 Fix up BinaryCacheStore::addToStore taking a path 2020-09-26 04:56:29 +00:00
John Ericson
5db83dd771 BinaryCacheStore::addTextToStore include CA field 2020-09-26 03:21:36 +00:00
John Ericson
c40c832f19 Merge remote-tracking branch 'obsidian/master' into binary-cache-addToStoreFromDump 2020-09-26 00:21:45 +00:00
Kevin Quick
cb186f1e75
Use "?dir=..." portion of "registry add" local path specification.
The registry targets generally follow a URL formatting schema with
support for a query parameter of "?dir=subpath" to specify a sub-path
location below the URL root.

Alternatively, an absolute path can be specified.  This specification
mode accepts the query parameter but ignores/drops it.  It would
probably be better to either (a) disallow the query parameter for the
path form, or (b) recognize the query parameter and add to the path.

This patch implements (b) for consistency, and to make it easier for
tooling that might switch between a remote git reference and a local
path reference.

See also issue #4050.
2020-09-25 09:36:18 -07:00
Gregory Hale
faa5607f54 Merge remote-tracking branch 'origin/master' into github-api-token 2020-09-25 12:10:58 -04:00
John Ericson
cfe791a638 stdout_ -> cout
Better to get creative than just sprinkle arbitrary underscores.
2020-09-25 11:30:04 -04:00
Eelco Dolstra
7d81582488
Merge pull request #3953 from obsidiansystems/basic-derivation-goal-outputs
Deduplicate basic derivation goals too
2020-09-25 17:21:19 +02:00
Kevin Quick
5a35cc29bf
Re-add support for github-access-token, but mark as deprecated. 2020-09-25 08:09:56 -07:00
Kevin Quick
ef2a14be19
Fix reference to older name for access-tokens config value. 2020-09-25 08:08:27 -07:00
Eelco Dolstra
cbb9862cd9
Merge pull request #3626 from W95Psp/master
Make `functionArgs` primitive accept primops (fix #3624)
2020-09-25 15:14:18 +02:00
Eelco Dolstra
7b2ae472ff
expectArg(): Respect the 'optional' flag 2020-09-25 10:27:40 +02:00
Kevin Quick
8fba2a8b54
Update to use access-tokens configuration for github/gitlab access.
This change provides support for using access tokens with other
instances of GitHub and GitLab beyond just github.com and
gitlab.com (especially company-specific or foundation-specific
instances).

This change also provides the ability to specify the type of access
token being used, where different types may have different handling,
based on the forge type.
2020-09-24 22:49:44 -07:00