Commit graph

9761 commits

Author SHA1 Message Date
regnat
bab1cda0e6 Use the hash modulo in the derivation outputs
Rather than storing the derivation outputs as `drvPath!outputName` internally,
store them as `drvHashModulo!outputName` (or `outputHash!outputName` for
fixed-output derivations).

This makes the storage slightly more opaque, but enables an earlier
cutoff in cases where a fixed-output dependency changes (but keeps the
same output hash) − same as what we already do for input-addressed
derivations.
2020-12-11 21:17:23 +01:00
regnat
8914e01e37 Store the realisations as JSON in the binary cache
Fix #4332
2020-12-11 21:05:09 +01:00
regnat
3ac9d74eb1 Rework the db schema for derivation outputs
Add a new table for tracking the derivation output mappings.

We used to hijack the `DerivationOutputs` table for that, but (despite its
name), it isn't a really good fit:

- Its entries depend on the drv being a valid path, making it play badly with
  garbage collection and preventing us to copy a drv output without copying
  the whole drv closure too;
- It dosen't guaranty that the output path exists;

By using a different table, we can experiment with a different schema better
suited for tracking the output mappings of CA derivations.
(incidentally, this also fixes #4138)
2020-12-11 20:41:32 +01:00
regnat
58cdab64ac Store metadata about drv outputs realisations
For each known realisation, store:
- its output
- its output path

This comes with a set of needed changes:

- New `realisations` module declaring the types needed for describing
  these mappings
- New `Store::registerDrvOutput` method registering all the needed informations
  about a derivation output (also replaces `LocalStore::linkDeriverToPath`)
- new `Store::queryRealisation` method to retrieve the informations for a
  derivations

This introcudes some redundancy on the remote-store side between
`wopQueryDerivationOutputMap` and `wopQueryRealisation`.
However we might need to keep both (regardless of backwards compat)
because we sometimes need to get some infos for all the outputs of a
derivation (where `wopQueryDerivationOutputMap` is handy), but all the
stores can't implement it − because listing all the outputs of a
derivation isn't really possible for binary caches where the server
doesn't allow to list a directory.
2020-12-11 20:41:32 +01:00
Eelco Dolstra
9c143c411b
Merge pull request #4350 from NixOS/ca/fix-build-with-nix-command
Fix the `nix` command with CA derivations
2020-12-11 15:01:49 +01:00
regnat
eb45308109 Fix the nix command with CA derivations
Prevents a crash because most `nix` subcommands assumed that derivations
know their output path, which isn't the case for CA derivations
2020-12-11 10:28:09 +01:00
Eelco Dolstra
c6a1bcd0ec nix store make-content-addressable: Show rewritten path 2020-12-10 17:11:56 +01:00
Eelco Dolstra
a8f533b664 Add lvlNotice log level
This is like syslog's LOG_NOTICE: "normal, but significant,
condition".
2020-12-10 16:41:24 +01:00
Eelco Dolstra
29fbc3cf31
Merge pull request #4343 from tweag/fix-osx-ci
Use no substituers by default in the tests
2020-12-09 20:38:06 +01:00
regnat
5286310e59 Use no substituers by default in the tests
Otherwise https://cache.nixos.org is chosen by default, causing the OSX
testsuite to hang inside the sandbox.

(In a way, this is probably rugging an actual bug under the carpet as
Nix should be able to gracefully timeout in such a case, but that's
beyond mac OSX-fu)
2020-12-09 14:53:45 +01:00
Eelco Dolstra
253571e4ec
Merge pull request #4342 from tweag/fix-remote-build-hook
fix remote build hook
2020-12-09 12:40:00 +01:00
Eelco Dolstra
eb458ad1b2
Merge pull request #4319 from Ma27/store-v6-addr
libstore/openStore: fix stores with IPv6 addresses
2020-12-09 12:38:04 +01:00
Maximilian Bosch
93a8a005de
libstore/openStore: fix stores with IPv6 addresses
In `nixStable` (2.3.7 to be precise) it's possible to connect to stores
using an IPv6 address:

  nix ping-store --store ssh://root@2001:db8::1

This is also useful for `nixops(1)` where you could specify an IPv6
address in `deployment.targetHost`.

However, this behavior is broken on `nixUnstable` and fails with the
following error:

  $ nix store ping --store ssh://root@2001:db8::1
  don't know how to open Nix store 'ssh://root@2001:db8::1'

This happened because `openStore` from `libstore` uses the `parseURL`
function from `libfetchers` which expects a valid URL as defined in
RFC2732. However, this is unsupported by `ssh(1)`:

  $ nix store ping --store 'ssh://root@[2001:db8::1]'
  cannot connect to 'root@[2001:db8::1]'

This patch now allows both ways of specifying a store (`root@2001:db8::1`) and
also `root@[2001:db8::1]` since the latter one is useful to pass query
parameters to the remote store.

In order to achieve this, the following changes were made:

* The URL regex from `url-parts.hh` now allows an IPv6 address in the
  form `2001:db8::1` and also `[2001:db8::1]`.

* In `libstore`, a new function named `extractConnStr` ensures that a
  proper URL is passed to e.g. `ssh(1)`:

  * If a URL looks like either `[2001:db8::1]` or `root@[2001:db8::1]`,
    the brackets will be removed using a regex. No additional validation
    is done here as only strings parsed by `parseURL` are expected.

  * In any other case, the string will be left untouched.

* The rules above only apply for `LegacySSHStore` and `SSHStore` (a.k.a
  `ssh://` and `ssh-ng://`).

Unresolved questions:

* I'm not really sure whether we want to allow both variants of IPv6
  addresses in the URL parser. However it should be noted that both seem
  to be possible according to RFC2732:

  > This document incudes an update to the generic syntax for Uniform
  > Resource Identifiers defined in RFC 2396 [URL].  It defines a syntax
  > for IPv6 addresses and allows the use of "[" and "]" within a URI
  > explicitly for this reserved purpose.

* Currently, it's not supported to specify a port number behind the
  hostname, however it seems as this is not really supported by the URL
  parser. Hence, this is probably out of scope here.
2020-12-09 12:23:29 +01:00
regnat
c87267c2a4 Store the final drv outputs in memory when building remotely
The `DerivationGoal` has a variable storing the “final” derivation
output paths that is used (amongst other things) to fill the environment
for the post build hook. However this variable wasn't set when the
build-hook is used, causing a crash when both hooks are used together.

Fix this by setting this variable (from the informations in the db) after a run
of the post build hook.
2020-12-09 10:45:12 +01:00
regnat
ee7c94fa1b Test the post-build-hook with remote builders
Regression test for #4245
2020-12-09 10:45:12 +01:00
regnat
6758e65612 Revert "Re-query for the derivation outputs in the post-build-hook"
This reverts commit 1b1e076033.

Using `queryPartialDerivationOutputMap` assumes that the derivation
exists locally which isn't the case for remote builders.
2020-12-09 09:44:07 +01:00
Eelco Dolstra
82e5511594
Merge pull request #4325 from tweag/hide-local-store-sql-statements
Hide the sqlite statements declarations for the local store
2020-12-08 14:04:16 +01:00
regnat
c0f21f08f8 Hide the sqlite statements declarations for the local store
These have no need to be in the public interface and it causes spurious
rebuilds each time one wants to add or remove a new statement.
2020-12-08 13:29:13 +01:00
Eelco Dolstra
500161b970
Merge pull request #4326 from tweag/fix-post-build-hook-and-remote-builders
Re-query for the derivation outputs in the post-build-hook
2020-12-08 11:47:47 +01:00
regnat
1b1e076033 Re-query for the derivation outputs in the post-build-hook
We can't assume that the runtime state knows about them as they might have
been built remotely, in which case we must query the db again to get
them.
2020-12-08 11:11:02 +01:00
Eelco Dolstra
97dc44f3d6
Merge pull request #4305 from matthewbauer/rosetta2-check
Check for rosetta 2 support before installing x86_64-darwin Nix
2020-12-07 15:59:56 +01:00
Eelco Dolstra
8a06edbf7e
Merge pull request #4321 from matthewbauer/always-default-cache-nixos-org
Always default to cache.nixos.org even when different nix store dir
2020-12-07 11:05:07 +01:00
Matthew Bauer
aa07502009 Always default to cache.nixos.org even when different nix store dir
Since 0744f7f, it is now useful to have cache.nixos.org in substituers
even if /nix/store is not the Nix Store Dir. This can always be
overridden via configuration, though.
2020-12-06 23:04:42 -06:00
Eelco Dolstra
1d1a85eb0a
Merge pull request #4318 from matthewbauer/add-slash-to-trusted-binary-cache
Canonicalize binary caches with ‘/’ when one is missing
2020-12-05 10:17:17 +01:00
Eelco Dolstra
a5d85d07fa
Merge pull request #4316 from stephank/aws-sdk-compat
Fix compatibility with newer AWS SDKs
2020-12-05 10:16:19 +01:00
Matthew Bauer
b9a00fd15b Canonicalize binary caches with ‘/’ when one is missing
This checks if there is a trusted substituter with a slash, so
trusting https://cache.nixos.org also implies https://cache.nixos.org/
is trusted.
2020-12-04 22:17:19 -06:00
Matthew Bauer
692549c542 Use com.apple.oahd.plist for rosetta 2 detection 2020-12-04 13:28:09 -06:00
Stéphan Kochen
e20a3ec756 Fix compatibility with newer AWS SDKs
Tested against AWS SDK 1.8.99. Fixes #3201.
2020-12-04 19:36:09 +01:00
Eelco Dolstra
45645b6f3b
Merge pull request #4314 from tweag/less-noisy-make-doc
Make `make install` less noisy
2020-12-04 15:02:37 +01:00
regnat
5f66edf245 Make make install less noisy
Remove the printing and useless output of a couple of commands when running `make install`
2020-12-04 14:50:47 +01:00
Eelco Dolstra
d94239c979
Merge pull request #4311 from matthewbauer/static-nix-support
Include static "nix" binary in Hydra build products
2020-12-04 12:33:21 +01:00
Matthew Bauer
be09af8002 Include static "nix" binary in Hydra build products
This allows users to get Nix from Hydra via a stable url like
https://hydra.nixos.org/build/132078238/download/1/nix
2020-12-03 18:05:18 -06:00
Eelco Dolstra
f337aa7099 Split 'nix store add-to-store' into 'add-path' and 'add-file'
This makes it consistent with 'nix hash <path|file>'.
2020-12-04 00:59:24 +01:00
Eelco Dolstra
8df58eae4c
Merge pull request #3858 from edolstra/group-commands
Group 'nix' subcommands
2020-12-03 23:55:28 +01:00
Eelco Dolstra
fa8dad10ed Typo 2020-12-03 23:26:23 +01:00
Eelco Dolstra
ea2062a2d9 Move most store-related commands to 'nix store' 2020-12-03 23:22:22 +01:00
Eelco Dolstra
a1cd805cba Add 'nix nar dump-path'
This only differs from 'nix store dump-path' in that the path doesn't
need to be a store path.
2020-12-03 22:52:01 +01:00
Eelco Dolstra
af373c2ece Add deprecated aliases for renamed commands 2020-12-03 22:45:44 +01:00
Eelco Dolstra
0c15ae5d4b Add FIXME 2020-12-03 20:31:45 +01:00
Eelco Dolstra
ef583303f0 Move NAR-related commands to 'nix nar' 2020-12-03 18:09:02 +01:00
Eelco Dolstra
79c1967ded Introduce 'nix store' command 2020-12-03 18:07:13 +01:00
Eelco Dolstra
5781f45c46 Allow registering subcommands of subcommands 2020-12-03 17:55:55 +01:00
Eelco Dolstra
b2d6c6161e Move 'nix hash-*' and 'nix to-*' to 'nix hash'
From the 'nix' UX review.
2020-12-03 17:55:55 +01:00
Eelco Dolstra
8ad2c9c4b9 Remove 'dist' target
We're not producing source tarballs anymore so this has been
bitrotting.
2020-12-03 16:17:58 +01:00
Eelco Dolstra
1b0ca3866b nix add-to-store: Move markdown docs into a separate file 2020-12-03 16:17:58 +01:00
Eelco Dolstra
c3c858ac6d Make doc() return arbitrary Markdown rather than the contents of the "Description" section
Thus we can return the examples section (and any other sections) from
doc() and don't need examples() anymore.
2020-12-03 16:17:58 +01:00
Eelco Dolstra
0bd060f23a
Merge pull request #4308 from tweag/properly-test-early-cutoff
Properly test the early cutoff for CA derivations
2020-12-03 14:45:29 +01:00
Eelco Dolstra
4f25644a13
Merge pull request #4304 from NixOS/separate-manpages
Separate manpages for 'nix' subcommands
2020-12-03 13:38:29 +01:00
regnat
8ad72b1f1c Properly test early cutoff with CA derivations
Build things with a different seed each time to make sure that it works
despite the different drvs
2020-12-03 13:31:07 +01:00
regnat
0afab668fa Don't fail early when -j0 is passed
If the build closure contains some CA derivations, then we can't know
ahead-of-time that we won't build anything as early-cutoff might come-in
at a laster stage
2020-12-03 13:24:36 +01:00