Commit graph

9969 commits

Author SHA1 Message Date
Eelco Dolstra
253571e4ec
Merge pull request #4342 from tweag/fix-remote-build-hook
fix remote build hook
2020-12-09 12:40:00 +01:00
Eelco Dolstra
eb458ad1b2
Merge pull request #4319 from Ma27/store-v6-addr
libstore/openStore: fix stores with IPv6 addresses
2020-12-09 12:38:04 +01:00
Maximilian Bosch
93a8a005de
libstore/openStore: fix stores with IPv6 addresses
In `nixStable` (2.3.7 to be precise) it's possible to connect to stores
using an IPv6 address:

  nix ping-store --store ssh://root@2001:db8::1

This is also useful for `nixops(1)` where you could specify an IPv6
address in `deployment.targetHost`.

However, this behavior is broken on `nixUnstable` and fails with the
following error:

  $ nix store ping --store ssh://root@2001:db8::1
  don't know how to open Nix store 'ssh://root@2001:db8::1'

This happened because `openStore` from `libstore` uses the `parseURL`
function from `libfetchers` which expects a valid URL as defined in
RFC2732. However, this is unsupported by `ssh(1)`:

  $ nix store ping --store 'ssh://root@[2001:db8::1]'
  cannot connect to 'root@[2001:db8::1]'

This patch now allows both ways of specifying a store (`root@2001:db8::1`) and
also `root@[2001:db8::1]` since the latter one is useful to pass query
parameters to the remote store.

In order to achieve this, the following changes were made:

* The URL regex from `url-parts.hh` now allows an IPv6 address in the
  form `2001:db8::1` and also `[2001:db8::1]`.

* In `libstore`, a new function named `extractConnStr` ensures that a
  proper URL is passed to e.g. `ssh(1)`:

  * If a URL looks like either `[2001:db8::1]` or `root@[2001:db8::1]`,
    the brackets will be removed using a regex. No additional validation
    is done here as only strings parsed by `parseURL` are expected.

  * In any other case, the string will be left untouched.

* The rules above only apply for `LegacySSHStore` and `SSHStore` (a.k.a
  `ssh://` and `ssh-ng://`).

Unresolved questions:

* I'm not really sure whether we want to allow both variants of IPv6
  addresses in the URL parser. However it should be noted that both seem
  to be possible according to RFC2732:

  > This document incudes an update to the generic syntax for Uniform
  > Resource Identifiers defined in RFC 2396 [URL].  It defines a syntax
  > for IPv6 addresses and allows the use of "[" and "]" within a URI
  > explicitly for this reserved purpose.

* Currently, it's not supported to specify a port number behind the
  hostname, however it seems as this is not really supported by the URL
  parser. Hence, this is probably out of scope here.
2020-12-09 12:23:29 +01:00
regnat
c87267c2a4 Store the final drv outputs in memory when building remotely
The `DerivationGoal` has a variable storing the “final” derivation
output paths that is used (amongst other things) to fill the environment
for the post build hook. However this variable wasn't set when the
build-hook is used, causing a crash when both hooks are used together.

Fix this by setting this variable (from the informations in the db) after a run
of the post build hook.
2020-12-09 10:45:12 +01:00
regnat
ee7c94fa1b Test the post-build-hook with remote builders
Regression test for #4245
2020-12-09 10:45:12 +01:00
regnat
6758e65612 Revert "Re-query for the derivation outputs in the post-build-hook"
This reverts commit 1b1e076033.

Using `queryPartialDerivationOutputMap` assumes that the derivation
exists locally which isn't the case for remote builders.
2020-12-09 09:44:07 +01:00
Eelco Dolstra
82e5511594
Merge pull request #4325 from tweag/hide-local-store-sql-statements
Hide the sqlite statements declarations for the local store
2020-12-08 14:04:16 +01:00
regnat
c0f21f08f8 Hide the sqlite statements declarations for the local store
These have no need to be in the public interface and it causes spurious
rebuilds each time one wants to add or remove a new statement.
2020-12-08 13:29:13 +01:00
Rok Garbas
ae77f21474
Switch away from classification as Tier1-3
to classification to a more descriptive classification.
2020-12-08 11:59:23 +01:00
Eelco Dolstra
500161b970
Merge pull request #4326 from tweag/fix-post-build-hook-and-remote-builders
Re-query for the derivation outputs in the post-build-hook
2020-12-08 11:47:47 +01:00
regnat
1b1e076033 Re-query for the derivation outputs in the post-build-hook
We can't assume that the runtime state knows about them as they might have
been built remotely, in which case we must query the db again to get
them.
2020-12-08 11:11:02 +01:00
Rok Garbas
d948b10c3a
Merge remote-tracking branch 'origin/master' into cli-guideline 2020-12-08 10:25:25 +01:00
Rok Garbas
0d7714b0d7
forgot to add the files 2020-12-08 10:25:03 +01:00
Eelco Dolstra
97dc44f3d6
Merge pull request #4305 from matthewbauer/rosetta2-check
Check for rosetta 2 support before installing x86_64-darwin Nix
2020-12-07 15:59:56 +01:00
Rok Garbas
24db5b125f
Merge remote-tracking branch 'origin/master' into cli-guideline 2020-12-07 15:13:36 +01:00
Eelco Dolstra
8a06edbf7e
Merge pull request #4321 from matthewbauer/always-default-cache-nixos-org
Always default to cache.nixos.org even when different nix store dir
2020-12-07 11:05:07 +01:00
Matthew Bauer
aa07502009 Always default to cache.nixos.org even when different nix store dir
Since 0744f7f, it is now useful to have cache.nixos.org in substituers
even if /nix/store is not the Nix Store Dir. This can always be
overridden via configuration, though.
2020-12-06 23:04:42 -06:00
Eelco Dolstra
1d1a85eb0a
Merge pull request #4318 from matthewbauer/add-slash-to-trusted-binary-cache
Canonicalize binary caches with ‘/’ when one is missing
2020-12-05 10:17:17 +01:00
Eelco Dolstra
a5d85d07fa
Merge pull request #4316 from stephank/aws-sdk-compat
Fix compatibility with newer AWS SDKs
2020-12-05 10:16:19 +01:00
Matthew Bauer
b9a00fd15b Canonicalize binary caches with ‘/’ when one is missing
This checks if there is a trusted substituter with a slash, so
trusting https://cache.nixos.org also implies https://cache.nixos.org/
is trusted.
2020-12-04 22:17:19 -06:00
Matthew Bauer
692549c542 Use com.apple.oahd.plist for rosetta 2 detection 2020-12-04 13:28:09 -06:00
Matthew Bauer
3c9b7029ba Use com.apple.oahd.plist for rosetta 2 detection 2020-12-04 13:26:53 -06:00
Stéphan Kochen
e20a3ec756 Fix compatibility with newer AWS SDKs
Tested against AWS SDK 1.8.99. Fixes #3201.
2020-12-04 19:36:09 +01:00
Eelco Dolstra
45645b6f3b
Merge pull request #4314 from tweag/less-noisy-make-doc
Make `make install` less noisy
2020-12-04 15:02:37 +01:00
regnat
5f66edf245 Make make install less noisy
Remove the printing and useless output of a couple of commands when running `make install`
2020-12-04 14:50:47 +01:00
Eelco Dolstra
d94239c979
Merge pull request #4311 from matthewbauer/static-nix-support
Include static "nix" binary in Hydra build products
2020-12-04 12:33:21 +01:00
Matthew Bauer
be09af8002 Include static "nix" binary in Hydra build products
This allows users to get Nix from Hydra via a stable url like
https://hydra.nixos.org/build/132078238/download/1/nix
2020-12-03 18:05:18 -06:00
Eelco Dolstra
f337aa7099 Split 'nix store add-to-store' into 'add-path' and 'add-file'
This makes it consistent with 'nix hash <path|file>'.
2020-12-04 00:59:24 +01:00
Eelco Dolstra
8df58eae4c
Merge pull request #3858 from edolstra/group-commands
Group 'nix' subcommands
2020-12-03 23:55:28 +01:00
Eelco Dolstra
fa8dad10ed Typo 2020-12-03 23:26:23 +01:00
Eelco Dolstra
ea2062a2d9 Move most store-related commands to 'nix store' 2020-12-03 23:22:22 +01:00
Eelco Dolstra
a1cd805cba Add 'nix nar dump-path'
This only differs from 'nix store dump-path' in that the path doesn't
need to be a store path.
2020-12-03 22:52:01 +01:00
Eelco Dolstra
af373c2ece Add deprecated aliases for renamed commands 2020-12-03 22:45:44 +01:00
Matthew Bauer
4b9acf4e21 Use posix_spawn_setbinpref_np to advise which architecture to run
When running universal binaries like /bin/bash, Darwin XNU will choose
which architecture of the binary to use based on "binary preferences".
This change sets that to the current platform for aarch64 and x86_64
builds. In addition it now uses posix_spawn instead of the usual
execve. Note, that this does not prevent the other architecture from
being run, just advises which to use.

Unfortunately, posix_spawnattr_setbinpref_np does not appear to be
inherited by child processes in x86_64 Rosetta 2 translations, meaning
that this will not always work as expected.

For example:

  {
    arm = derivation {
      name = "test";
      system = "aarch64-darwin";
      builder = "/bin/bash";
      args = [ "-e" (builtins.toFile "test" ''
        set -x
        /usr/sbin/sysctl sysctl.proc_translated
        /usr/sbin/sysctl sysctl.proc_native
        [ "$(/usr/bin/arch)" = arm64 ]
        /usr/bin/touch $out
      '') ];
    };
    rosetta = derivation {
      name = "test";
      system = "x86_64-darwin";
      builder = "/bin/bash";
      args = [ "-e" (builtins.toFile "test" ''
        set -x
        /usr/sbin/sysctl sysctl.proc_translated
        /usr/sbin/sysctl sysctl.proc_native
        [ "$(/usr/bin/arch)" = i386 ]
        echo It works!
        /usr/bin/touch $out
      '') ];
    };
  }

`arm' fails on x86_64-compiled Nix, but `arm' and `rosetta' succeed on
aarch64-compiled Nix. I suspect there is a way to fix this since:

  $ /usr/bin/arch -arch x86_64 /bin/bash \
    -c '/usr/bin/arch -arch arm64e /bin/bash -c /usr/bin/arch'
  arm64

seems to work correctly. We may need to wait for Apple to update
system_cmds in opensource.apple.com to find out how though.
2020-12-03 15:41:59 -06:00
Matthew Bauer
9b1824ecbd Add extraPlatforms for Rosetta 2 macOS
macOS systems with ARM64 can utilize a translation layer at
/Library/Apple/usr/libexec/oah to run x86_64 binaries. This change
makes Nix recognize that and it to "extra-platforms". Note that there
are two cases here since Nix could be built for either x86_64 or
aarch64. In either case, we can switch to the other architecture.
Unfortunately there is not a good way to prevent aarch64 binaries from
being run in x86_64 contexts or vice versa - programs can always
execute programs for the other architecture.
2020-12-03 15:41:43 -06:00
Eelco Dolstra
0c15ae5d4b Add FIXME 2020-12-03 20:31:45 +01:00
Eelco Dolstra
ef583303f0 Move NAR-related commands to 'nix nar' 2020-12-03 18:09:02 +01:00
Eelco Dolstra
79c1967ded Introduce 'nix store' command 2020-12-03 18:07:13 +01:00
Eelco Dolstra
5781f45c46 Allow registering subcommands of subcommands 2020-12-03 17:55:55 +01:00
Eelco Dolstra
b2d6c6161e Move 'nix hash-*' and 'nix to-*' to 'nix hash'
From the 'nix' UX review.
2020-12-03 17:55:55 +01:00
Eelco Dolstra
8ad2c9c4b9 Remove 'dist' target
We're not producing source tarballs anymore so this has been
bitrotting.
2020-12-03 16:17:58 +01:00
Eelco Dolstra
1b0ca3866b nix add-to-store: Move markdown docs into a separate file 2020-12-03 16:17:58 +01:00
Eelco Dolstra
c3c858ac6d Make doc() return arbitrary Markdown rather than the contents of the "Description" section
Thus we can return the examples section (and any other sections) from
doc() and don't need examples() anymore.
2020-12-03 16:17:58 +01:00
Eelco Dolstra
0bd060f23a
Merge pull request #4308 from tweag/properly-test-early-cutoff
Properly test the early cutoff for CA derivations
2020-12-03 14:45:29 +01:00
Eelco Dolstra
4f25644a13
Merge pull request #4304 from NixOS/separate-manpages
Separate manpages for 'nix' subcommands
2020-12-03 13:38:29 +01:00
regnat
8ad72b1f1c Properly test early cutoff with CA derivations
Build things with a different seed each time to make sure that it works
despite the different drvs
2020-12-03 13:31:07 +01:00
regnat
0afab668fa Don't fail early when -j0 is passed
If the build closure contains some CA derivations, then we can't know
ahead-of-time that we won't build anything as early-cutoff might come-in
at a laster stage
2020-12-03 13:24:36 +01:00
Eelco Dolstra
7cb341ceb5
Merge pull request #4307 from matthewbauer/update-config-guess
Update config.guess & config.sub for proper arm64 macOS detection
2020-12-03 10:46:06 +01:00
Matthew Bauer
94f359525e Update config.guess for proper arm64 macOS detection
This fixes results for arm64 macOS so config.guess now reports:

  aarch64-apple-darwin20.1.0

instead of

  arm-apple-darwin20.1.0
2020-12-02 19:14:34 -06:00
Matthew Bauer
addf9f4ede Call it aarch64-darwin instead of arm64-darwin
gnu-config standardized on aarch64 for machine name so host_cpu part
of $system will always be aarch64. That means system will be
aarch64-darwin too.

uname however could report either “aarch64” (if gnu coreutils) or
“arm64” (if apple’s uname). We should support both for compatiblity
here.
2020-12-02 19:05:02 -06:00