From edfcc8256ee232736e335d6cc315f98f6f40d1f3 Mon Sep 17 00:00:00 2001 From: "Travis A. Everett" Date: Sat, 11 Jun 2022 13:30:51 -0500 Subject: [PATCH 01/93] doc: add install test info to hacking.md --- doc/manual/src/contributing/hacking.md | 64 +++++++++++++++++++++++++- 1 file changed, 63 insertions(+), 1 deletion(-) diff --git a/doc/manual/src/contributing/hacking.md b/doc/manual/src/contributing/hacking.md index 59ce5cac7..9a371afa7 100644 --- a/doc/manual/src/contributing/hacking.md +++ b/doc/manual/src/contributing/hacking.md @@ -83,7 +83,7 @@ by: $ nix develop ``` -## Testing +## Testing Nix Nix comes with three different flavors of tests: unit, functional and integration. @@ -108,3 +108,65 @@ These tests include everything that needs to interact with external services or Because these tests are expensive and require more than what the standard github-actions setup provides, they only run on the master branch (on ). You can run them manually with `nix build .#hydraJobs.tests.{testName}` or `nix-build -A hydraJobs.tests.{testName}` + +## Testing the install scripts + +Testing the install scripts has traditionally been tedious, but you can now do this much more easily via the GitHub Actions CI runs (at least for platforms that Github Actions supports). + +If you've already pushed to a fork of Nix on GitHub before, you may have noticed that the CI workflows in your fork list skipped "installer" and "installer_test" jobs. Once your Nix fork is set up correctly, pushing to it will also run these jobs. +- The `installer` job will generate installers for these platforms: x86_64-linux, armv6l-linux, armv7l-linux, x86_64-darwin. While this installer is in your Cachix cache, you can use it for manual testing on any of these platforms. +- the `installer_test` job will try to use this installer and run a trivial Nix command on `ubuntu-latest` and `macos-latest`. + +### One-time setup +1. Have a GitHub account with a fork of the Nix repo. +2. At cachix.org: + - Create or log in to an account. + - Create a Cachix cache using the format `-nix-install-tests`. + - Navigate to the new cache > Settings > Auth Tokens. + - Generate a new cachix auth token and copy the generated value. +4. At github.com: + - Navigate to your Nix fork > Settings > Secrets > Actions > New repository secret. + - Name the secret `CACHIX_AUTH_TOKEN` + - Paste the copied value of the Cachix cache auth token. + +### Using the CI-generated installer for manual testing + +After the CI run completes, you can check the output to extract the installer url: +1. Click into the detailed view of the CI run. +2. Click into any `installer_test` run (the URL you're here to extract will be the same in all of them). +3. Click into the `Run cachix/install-nix-action@v...` step and click the detail triangle next to the first log line (it will also be `Run cachix/install-nix-action@v...`) +4. Copy the install_url +5. To generate an install command, plug this install_url and your github username into this template: + + ```console + sh <(curl -L ) --tarball-url-prefix https://-nix-install-tests.cachix.org/serve + ``` + + From a5be5e01200a12cc34d0e3a2e3f964d5c95208b9 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 4 Aug 2022 14:07:06 -0700 Subject: [PATCH 02/93] doc/manual: define {local,remote} store, binary cache, substituter Nix veterans intuitively know what the following terms mean. They are used in several places in the nix documentation, but never defined: - local store - remote store - binary cache - substituter In particular, I found the last two terms to be confusingly similar. Let's give definitions for them. --- doc/manual/src/SUMMARY.md.in | 1 + .../src/package-management/terminology.md | 27 +++++++++++++++++++ 2 files changed, 28 insertions(+) create mode 100644 doc/manual/src/package-management/terminology.md diff --git a/doc/manual/src/SUMMARY.md.in b/doc/manual/src/SUMMARY.md.in index a47d39f31..f8da2247b 100644 --- a/doc/manual/src/SUMMARY.md.in +++ b/doc/manual/src/SUMMARY.md.in @@ -22,6 +22,7 @@ - [Garbage Collector Roots](package-management/garbage-collector-roots.md) - [Channels](package-management/channels.md) - [Sharing Packages Between Machines](package-management/sharing-packages.md) + - [Terminology](package-management/terminology.md) - [Serving a Nix store via HTTP](package-management/binary-cache-substituter.md) - [Copying Closures via SSH](package-management/copy-closure.md) - [Serving a Nix store via SSH](package-management/ssh-substituter.md) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md new file mode 100644 index 000000000..28e2a1f0b --- /dev/null +++ b/doc/manual/src/package-management/terminology.md @@ -0,0 +1,27 @@ +# Terminology + +A *local store* exists on the local filesystem of the machine where +Nix is invoked. The `/nix/store` directory is one example of a +local store. You can use other local stores by passing the +`--store` flag to `nix`. + +A *remote store* is a store which exists anywhere other than the +local filesystem. One example is the `/nix/store` directory on +another machine, accessed via `ssh` or served by the `nix-serve` +Perl script. + +A *binary cache* is a remote store which is not the local store of +any machine. Examples of binary caches include S3 buckets and the +[NixOS binary cache](https://cache.nixos.org). Binary caches use a +disk layout that is different from local stores; in particular, they +keep metadata and signatures in `.narinfo` files rather than in +`/nix/var/nix/db`. + +A *substituter* is a store other than `/nix/store` from which nix will +copy the realisation of a derivation instead of building it. Nix will +not copy a realisation from a remote store unless one of the following +is true: + +- the realisation is signed by one of the `trusted-public-key`s +- the substituter is in the `trusted-substituters` list +- the `no-require-sigs` option has been set to disable signature checking From 56d4fc194ba90ee4e559a07a895f6bf4a61ef462 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 4 Aug 2022 14:14:24 -0700 Subject: [PATCH 03/93] fourth trust condition: FODs --- doc/manual/src/package-management/terminology.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md index 28e2a1f0b..1722a9fc8 100644 --- a/doc/manual/src/package-management/terminology.md +++ b/doc/manual/src/package-management/terminology.md @@ -25,3 +25,5 @@ is true: - the realisation is signed by one of the `trusted-public-key`s - the substituter is in the `trusted-substituters` list - the `no-require-sigs` option has been set to disable signature checking +- the derivation is a fixed-output derivation + From 8f44d24c525160b2ddef5e18a4af4ce667e23e9f Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 4 Aug 2022 14:19:25 -0700 Subject: [PATCH 04/93] !fixup whitespace --- doc/manual/src/package-management/terminology.md | 1 - 1 file changed, 1 deletion(-) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md index 1722a9fc8..d800bafc1 100644 --- a/doc/manual/src/package-management/terminology.md +++ b/doc/manual/src/package-management/terminology.md @@ -26,4 +26,3 @@ is true: - the substituter is in the `trusted-substituters` list - the `no-require-sigs` option has been set to disable signature checking - the derivation is a fixed-output derivation - From 62674659ed7b7fc6a2c884f52df2474e344400f8 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 4 Aug 2022 14:21:17 -0700 Subject: [PATCH 05/93] !fixup capitalize Nix --- doc/manual/src/package-management/terminology.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md index d800bafc1..8dc2ede33 100644 --- a/doc/manual/src/package-management/terminology.md +++ b/doc/manual/src/package-management/terminology.md @@ -17,7 +17,7 @@ disk layout that is different from local stores; in particular, they keep metadata and signatures in `.narinfo` files rather than in `/nix/var/nix/db`. -A *substituter* is a store other than `/nix/store` from which nix will +A *substituter* is a store other than `/nix/store` from which Nix will copy the realisation of a derivation instead of building it. Nix will not copy a realisation from a remote store unless one of the following is true: From 1b97f3872ed70d9ad5d19d27dc56b42ba4d26382 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 4 Aug 2022 14:22:14 -0700 Subject: [PATCH 06/93] !fixup: transposed characters --- doc/manual/src/package-management/terminology.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md index 8dc2ede33..6ad0f6833 100644 --- a/doc/manual/src/package-management/terminology.md +++ b/doc/manual/src/package-management/terminology.md @@ -22,7 +22,7 @@ copy the realisation of a derivation instead of building it. Nix will not copy a realisation from a remote store unless one of the following is true: -- the realisation is signed by one of the `trusted-public-key`s +- the realisation is signed by one of the `trusted-public-keys` - the substituter is in the `trusted-substituters` list - the `no-require-sigs` option has been set to disable signature checking - the derivation is a fixed-output derivation From aae771cad26a3803ef0a0855c782823d22949cf3 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Fri, 5 Aug 2022 10:12:46 -0700 Subject: [PATCH 07/93] !implement https://github.com/NixOS/nix/pull/6870#discussion_r938912244 --- doc/manual/src/package-management/terminology.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md index 6ad0f6833..241bb6c5a 100644 --- a/doc/manual/src/package-management/terminology.md +++ b/doc/manual/src/package-management/terminology.md @@ -18,11 +18,10 @@ keep metadata and signatures in `.narinfo` files rather than in `/nix/var/nix/db`. A *substituter* is a store other than `/nix/store` from which Nix will -copy the realisation of a derivation instead of building it. Nix will -not copy a realisation from a remote store unless one of the following -is true: +copy a store path instead of building it. Nix will not copy a store +path from a remote store unless one of the following is true: -- the realisation is signed by one of the `trusted-public-keys` +- the store object is signed by one of the `trusted-public-keys` - the substituter is in the `trusted-substituters` list - the `no-require-sigs` option has been set to disable signature checking -- the derivation is a fixed-output derivation +- the store object is the realisation of a fixed-output derivation From 2eb74c918dc7dc04ed36b3fdcd95406007d97690 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Fri, 5 Aug 2022 10:13:41 -0700 Subject: [PATCH 08/93] derivations do not need to be signed --- doc/manual/src/package-management/terminology.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md index 241bb6c5a..4b9e68de9 100644 --- a/doc/manual/src/package-management/terminology.md +++ b/doc/manual/src/package-management/terminology.md @@ -24,4 +24,5 @@ path from a remote store unless one of the following is true: - the store object is signed by one of the `trusted-public-keys` - the substituter is in the `trusted-substituters` list - the `no-require-sigs` option has been set to disable signature checking +- the store object is a derivation - the store object is the realisation of a fixed-output derivation From 66a93a76b9842ac18188b91f5a30c4ac4f2b6118 Mon Sep 17 00:00:00 2001 From: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com> Date: Fri, 5 Aug 2022 17:15:37 +0000 Subject: [PATCH 09/93] Update doc/manual/src/package-management/terminology.md Co-authored-by: Attila Gulyas --- .../src/package-management/terminology.md | 24 ++++++++++++------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md index 4b9e68de9..564667f93 100644 --- a/doc/manual/src/package-management/terminology.md +++ b/doc/manual/src/package-management/terminology.md @@ -1,14 +1,22 @@ # Terminology -A *local store* exists on the local filesystem of the machine where -Nix is invoked. The `/nix/store` directory is one example of a -local store. You can use other local stores by passing the -`--store` flag to `nix`. +From the perspective of the location where Nix is +invoked1, the Nix store can be referred to +as a "_local_" or a "_remote_" one: -A *remote store* is a store which exists anywhere other than the -local filesystem. One example is the `/nix/store` directory on -another machine, accessed via `ssh` or served by the `nix-serve` -Perl script. +\[1]: Where "invoking Nix" means an executing a Nix core +action/operation on a Nix store. For example, using any CLI +commands from the `NixOS/nix` implementation. + ++ A *local store* exists on the local filesystem of + the machine where Nix is invoked. You can use other + local stores by passing the `--store` flag to the + `nix` command. + ++ A *remote store* exists anywhere other than the + local filesystem. One example is the `/nix/store` + directory on another machine, accessed via `ssh` or + served by the `nix-serve` Perl script. A *binary cache* is a remote store which is not the local store of any machine. Examples of binary caches include S3 buckets and the From d5506aa71200425b65cc1777077478f5ff8d2aff Mon Sep 17 00:00:00 2001 From: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com> Date: Fri, 5 Aug 2022 17:19:52 +0000 Subject: [PATCH 10/93] Update doc/manual/src/package-management/terminology.md Co-authored-by: Attila Gulyas --- doc/manual/src/package-management/terminology.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md index 564667f93..b3e9ea040 100644 --- a/doc/manual/src/package-management/terminology.md +++ b/doc/manual/src/package-management/terminology.md @@ -18,12 +18,10 @@ commands from the `NixOS/nix` implementation. directory on another machine, accessed via `ssh` or served by the `nix-serve` Perl script. -A *binary cache* is a remote store which is not the local store of -any machine. Examples of binary caches include S3 buckets and the -[NixOS binary cache](https://cache.nixos.org). Binary caches use a -disk layout that is different from local stores; in particular, they -keep metadata and signatures in `.narinfo` files rather than in -`/nix/var/nix/db`. +A *binary cache* is a specialized Nix store whose metadata and +signatures are kept in `.narinfo` files rather than in the Nix +database. Examples of binary caches include S3 buckets and the +[NixOS binary cache](https://cache.nixos.org). A *substituter* is a store other than `/nix/store` from which Nix will copy a store path instead of building it. Nix will not copy a store From 4de95f7f565df71d8ebddb7434e2b0feb49a833b Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Fri, 5 Aug 2022 10:33:48 -0700 Subject: [PATCH 11/93] gesture at explanation of why binary caches exist --- doc/manual/src/package-management/terminology.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md index b3e9ea040..493f5f03e 100644 --- a/doc/manual/src/package-management/terminology.md +++ b/doc/manual/src/package-management/terminology.md @@ -18,10 +18,12 @@ commands from the `NixOS/nix` implementation. directory on another machine, accessed via `ssh` or served by the `nix-serve` Perl script. -A *binary cache* is a specialized Nix store whose metadata and -signatures are kept in `.narinfo` files rather than in the Nix -database. Examples of binary caches include S3 buckets and the -[NixOS binary cache](https://cache.nixos.org). +A *binary cache* is a Nix store which uses a different format: its +metadata and signatures are kept in `.narinfo` files rather than in a +Nix database. This different format simplifies serving store objects +over the network, but cannot host builds. Examples of binary caches +include S3 buckets and the [NixOS binary +cache](https://cache.nixos.org). A *substituter* is a store other than `/nix/store` from which Nix will copy a store path instead of building it. Nix will not copy a store From 1d3b92e80ca1564bf2c5ee207df707e215188633 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Fri, 5 Aug 2022 10:39:43 -0700 Subject: [PATCH 12/93] move package-management/terminology into glossary.md --- doc/manual/src/SUMMARY.md.in | 1 - doc/manual/src/glossary.md | 37 +++++++++++++++++++ .../src/package-management/terminology.md | 36 ------------------ 3 files changed, 37 insertions(+), 37 deletions(-) delete mode 100644 doc/manual/src/package-management/terminology.md diff --git a/doc/manual/src/SUMMARY.md.in b/doc/manual/src/SUMMARY.md.in index f8da2247b..a47d39f31 100644 --- a/doc/manual/src/SUMMARY.md.in +++ b/doc/manual/src/SUMMARY.md.in @@ -22,7 +22,6 @@ - [Garbage Collector Roots](package-management/garbage-collector-roots.md) - [Channels](package-management/channels.md) - [Sharing Packages Between Machines](package-management/sharing-packages.md) - - [Terminology](package-management/terminology.md) - [Serving a Nix store via HTTP](package-management/binary-cache-substituter.md) - [Copying Closures via SSH](package-management/copy-closure.md) - [Serving a Nix store via SSH](package-management/ssh-substituter.md) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index aa0ac78cb..f4c51588d 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -11,6 +11,32 @@ The location in the file system where store objects live. Typically `/nix/store`. + From the perspective of the location where Nix is + invoked1, the Nix store can be referred to + as a "_local_" or a "_remote_" one: + + \[1]: Where "invoking Nix" means an executing a Nix core + action/operation on a Nix store. For example, using any CLI + commands from the `NixOS/nix` implementation. + + + A *local store* exists on the local filesystem of + the machine where Nix is invoked. You can use other + local stores by passing the `--store` flag to the + `nix` command. + + + A *remote store* exists anywhere other than the + local filesystem. One example is the `/nix/store` + directory on another machine, accessed via `ssh` or + served by the `nix-serve` Perl script. + + - [binary cache]{#binary-cache}\ + A *binary cache* is a Nix store which uses a different format: its + metadata and signatures are kept in `.narinfo` files rather than in a + Nix database. This different format simplifies serving store objects + over the network, but cannot host builds. Examples of binary caches + include S3 buckets and the [NixOS binary + cache](https://cache.nixos.org). + - [store path]{#gloss-store-path}\ The location in the file system of a store object, i.e., an immediate child of the Nix store directory. @@ -29,6 +55,17 @@ store object by downloading a pre-built version of the store object from some server. + - [substituter]{#gloss-substituter}\ + A *substituter* is a store other than `/nix/store` from which Nix will + copy a store path instead of building it. Nix will not copy a store + path from a remote store unless one of the following is true: + + - the store object is signed by one of the `trusted-public-keys` + - the substituter is in the `trusted-substituters` list + - the `no-require-sigs` option has been set to disable signature checking + - the store object is a derivation + - the store object is the realisation of a fixed-output derivation + - [purity]{#gloss-purity}\ The assumption that equal Nix derivations when run always produce the same output. This cannot be guaranteed in general (e.g., a diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md deleted file mode 100644 index 493f5f03e..000000000 --- a/doc/manual/src/package-management/terminology.md +++ /dev/null @@ -1,36 +0,0 @@ -# Terminology - -From the perspective of the location where Nix is -invoked1, the Nix store can be referred to -as a "_local_" or a "_remote_" one: - -\[1]: Where "invoking Nix" means an executing a Nix core -action/operation on a Nix store. For example, using any CLI -commands from the `NixOS/nix` implementation. - -+ A *local store* exists on the local filesystem of - the machine where Nix is invoked. You can use other - local stores by passing the `--store` flag to the - `nix` command. - -+ A *remote store* exists anywhere other than the - local filesystem. One example is the `/nix/store` - directory on another machine, accessed via `ssh` or - served by the `nix-serve` Perl script. - -A *binary cache* is a Nix store which uses a different format: its -metadata and signatures are kept in `.narinfo` files rather than in a -Nix database. This different format simplifies serving store objects -over the network, but cannot host builds. Examples of binary caches -include S3 buckets and the [NixOS binary -cache](https://cache.nixos.org). - -A *substituter* is a store other than `/nix/store` from which Nix will -copy a store path instead of building it. Nix will not copy a store -path from a remote store unless one of the following is true: - -- the store object is signed by one of the `trusted-public-keys` -- the substituter is in the `trusted-substituters` list -- the `no-require-sigs` option has been set to disable signature checking -- the store object is a derivation -- the store object is the realisation of a fixed-output derivation From b5d85f0922e0f4a9585a281f6d938ec67cd07349 Mon Sep 17 00:00:00 2001 From: "Travis A. Everett" Date: Fri, 5 Aug 2022 13:49:18 -0500 Subject: [PATCH 13/93] Apply suggestions from code review Co-authored-by: Valentin Gagarin --- doc/manual/src/contributing/hacking.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/manual/src/contributing/hacking.md b/doc/manual/src/contributing/hacking.md index 9a371afa7..86c6522f2 100644 --- a/doc/manual/src/contributing/hacking.md +++ b/doc/manual/src/contributing/hacking.md @@ -113,11 +113,12 @@ You can run them manually with `nix build .#hydraJobs.tests.{testName}` or `nix- Testing the install scripts has traditionally been tedious, but you can now do this much more easily via the GitHub Actions CI runs (at least for platforms that Github Actions supports). -If you've already pushed to a fork of Nix on GitHub before, you may have noticed that the CI workflows in your fork list skipped "installer" and "installer_test" jobs. Once your Nix fork is set up correctly, pushing to it will also run these jobs. -- The `installer` job will generate installers for these platforms: x86_64-linux, armv6l-linux, armv7l-linux, x86_64-darwin. While this installer is in your Cachix cache, you can use it for manual testing on any of these platforms. +If you've already pushed to a fork of Nix on GitHub before, you may have noticed that the CI workflows in your fork list skipped `installer` and `installer_test` jobs. Once your Nix fork is set up correctly, pushing to it will also run these jobs. +- The `installer` job will generate installers for these platforms: `x86_64-linux`, `armv6l-linux`, `armv7l-linux`, `x86_64-darwin`. While this installer is in your Cachix cache, you can use it for manual testing on any of these platforms. - the `installer_test` job will try to use this installer and run a trivial Nix command on `ubuntu-latest` and `macos-latest`. ### One-time setup + 1. Have a GitHub account with a fork of the Nix repo. 2. At cachix.org: - Create or log in to an account. From 9b7bd2dd1fc83f6df449fce3967a95cb098ca4b2 Mon Sep 17 00:00:00 2001 From: "Travis A. Everett" Date: Mon, 8 Aug 2022 10:04:27 -0500 Subject: [PATCH 14/93] Apply suggestions from code review Co-authored-by: Valentin Gagarin --- doc/manual/src/contributing/hacking.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/doc/manual/src/contributing/hacking.md b/doc/manual/src/contributing/hacking.md index 86c6522f2..d8a8c8591 100644 --- a/doc/manual/src/contributing/hacking.md +++ b/doc/manual/src/contributing/hacking.md @@ -114,7 +114,13 @@ You can run them manually with `nix build .#hydraJobs.tests.{testName}` or `nix- Testing the install scripts has traditionally been tedious, but you can now do this much more easily via the GitHub Actions CI runs (at least for platforms that Github Actions supports). If you've already pushed to a fork of Nix on GitHub before, you may have noticed that the CI workflows in your fork list skipped `installer` and `installer_test` jobs. Once your Nix fork is set up correctly, pushing to it will also run these jobs. -- The `installer` job will generate installers for these platforms: `x86_64-linux`, `armv6l-linux`, `armv7l-linux`, `x86_64-darwin`. While this installer is in your Cachix cache, you can use it for manual testing on any of these platforms. +- The `installer` job will generate installers for these platforms: + - `x86_64-linux` + - `armv6l-linux` + - `armv7l-linux` + - `x86_64-darwin`. + + While this installer is in your Cachix cache, you can use it for manual testing on any of these platforms. - the `installer_test` job will try to use this installer and run a trivial Nix command on `ubuntu-latest` and `macos-latest`. ### One-time setup From bac1e1bf8c359b5e6831c3974a05bdce867775a5 Mon Sep 17 00:00:00 2001 From: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com> Date: Mon, 8 Aug 2022 17:52:31 +0000 Subject: [PATCH 15/93] Update doc/manual/src/glossary.md Co-authored-by: Valentin Gagarin --- doc/manual/src/glossary.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index f4c51588d..77de58965 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -12,13 +12,9 @@ `/nix/store`. From the perspective of the location where Nix is - invoked1, the Nix store can be referred to + invoked, the Nix store can be referred to as a "_local_" or a "_remote_" one: - \[1]: Where "invoking Nix" means an executing a Nix core - action/operation on a Nix store. For example, using any CLI - commands from the `NixOS/nix` implementation. - + A *local store* exists on the local filesystem of the machine where Nix is invoked. You can use other local stores by passing the `--store` flag to the From 520587b9a0327194371146f4dd25c9227a2c79e2 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 13:38:07 -0700 Subject: [PATCH 16/93] glossary: local store: clarify --- doc/manual/src/glossary.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 77de58965..d653a2ae4 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -15,10 +15,10 @@ invoked, the Nix store can be referred to as a "_local_" or a "_remote_" one: - + A *local store* exists on the local filesystem of + + A *local store* exists on the filesystem of the machine where Nix is invoked. You can use other local stores by passing the `--store` flag to the - `nix` command. + `nix` command. Local stores can be used for building derivations. + A *remote store* exists anywhere other than the local filesystem. One example is the `/nix/store` From 2812682ebee9d4419ba89690177b31564ce5ba77 Mon Sep 17 00:00:00 2001 From: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com> Date: Thu, 1 Sep 2022 20:40:39 +0000 Subject: [PATCH 17/93] Update doc/manual/src/glossary.md Co-authored-by: John Ericson --- doc/manual/src/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index d653a2ae4..f072f35e1 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -52,7 +52,7 @@ from some server. - [substituter]{#gloss-substituter}\ - A *substituter* is a store other than `/nix/store` from which Nix will + A *substituter* is an additional store from which Nix will copy a store path instead of building it. Nix will not copy a store path from a remote store unless one of the following is true: From 9cb84121435e8ca6a51950b9d96a3d3be47c809e Mon Sep 17 00:00:00 2001 From: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com> Date: Thu, 1 Sep 2022 20:41:04 +0000 Subject: [PATCH 18/93] Update doc/manual/src/glossary.md Co-authored-by: John Ericson --- doc/manual/src/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index f072f35e1..91865c807 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -53,7 +53,7 @@ - [substituter]{#gloss-substituter}\ A *substituter* is an additional store from which Nix will - copy a store path instead of building it. Nix will not copy a store + copy store objects it doesn't have. Nix will not copy a store path from a remote store unless one of the following is true: - the store object is signed by one of the `trusted-public-keys` From 41153f30bd5ca1bd9fa10d18da7a6b5b78a94087 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 13:54:09 -0700 Subject: [PATCH 19/93] glossary: substituter: merge output-addressed cases --- doc/manual/src/glossary.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 91865c807..608beb8f8 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -59,8 +59,9 @@ - the store object is signed by one of the `trusted-public-keys` - the substituter is in the `trusted-substituters` list - the `no-require-sigs` option has been set to disable signature checking - - the store object is a derivation - - the store object is the realisation of a fixed-output derivation + - the store object is *output-addressed*; this includes + derivations, the outputs of content-addressed derivations, and + the outputs of fixed-output derivations. - [purity]{#gloss-purity}\ The assumption that equal Nix derivations when run always produce From 1f56b5d77247d89a15a2c16ba2f5d1d672c835e8 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 14:09:06 -0700 Subject: [PATCH 20/93] doc/manual: un-inline definitions from `substitute` --- doc/manual/src/glossary.md | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 608beb8f8..8dff4646c 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -7,6 +7,14 @@ translated into low-level *store derivations* (implicitly by `nix-env` and `nix-build`, or explicitly by `nix-instantiate`). + - [content-addressed derivation]{#gloss-fixed-output-derivation} + FIXME + + - [fixed-output derivation]{#gloss-fixed-output-derivation} + A derivation which includes the `__outHash` attribute; the output + of such derivations must exactly match the hash. All fixed-output + derivations are [content-addressed derivations](#gloss-content-addressed-derivation). + - [store]{#gloss-store}\ The location in the file system where store objects live. Typically `/nix/store`. @@ -44,6 +52,16 @@ derivation outputs (objects produced by running a build action), or derivations (files describing a build action). + - [input-addressed store object]{#gloss-input-addressed-store-object}\ + Store objects produced by building a + non-[content-addressed](#gloss-content-addressed-derivation) + derivation. + + - [output-addressed store object]{#gloss-output-addressed-store-object}\ + A store object whose store path hashes its content. This + includes derivations and the outputs of + [content-addressed derivations](#gloss-content-addressed-derivation) + - [substitute]{#gloss-substitute}\ A substitute is a command invocation stored in the Nix database that describes how to build a store object, bypassing the normal build @@ -59,9 +77,7 @@ - the store object is signed by one of the `trusted-public-keys` - the substituter is in the `trusted-substituters` list - the `no-require-sigs` option has been set to disable signature checking - - the store object is *output-addressed*; this includes - derivations, the outputs of content-addressed derivations, and - the outputs of fixed-output derivations. + - the store object is [output-addressed](#gloss-output-addressed-store-object) - [purity]{#gloss-purity}\ The assumption that equal Nix derivations when run always produce From 0a98d564b3d8d195c023429e5f7faf63e20b5d93 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 14:41:27 -0700 Subject: [PATCH 21/93] glossary: resolve FIXME in #gloss-fixed-output-derivation --- doc/manual/src/glossary.md | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 8dff4646c..473aef03c 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -7,13 +7,14 @@ translated into low-level *store derivations* (implicitly by `nix-env` and `nix-build`, or explicitly by `nix-instantiate`). - - [content-addressed derivation]{#gloss-fixed-output-derivation} - FIXME + - [content-addressed derivation]{#gloss-content-addressed-derivation}\ + A derivation which has the + [`__contentAddressed`](language/advanced-attributes.md#contentAddressed) + attribute set to `true`. - - [fixed-output derivation]{#gloss-fixed-output-derivation} - A derivation which includes the `__outHash` attribute; the output - of such derivations must exactly match the hash. All fixed-output - derivations are [content-addressed derivations](#gloss-content-addressed-derivation). + - [fixed-output derivation]{#gloss-fixed-output-derivation}\ + A derivation which includes the `__outputHash` attribute; the output + of such derivations must exactly match the hash. - [store]{#gloss-store}\ The location in the file system where store objects live. Typically @@ -59,8 +60,10 @@ - [output-addressed store object]{#gloss-output-addressed-store-object}\ A store object whose store path hashes its content. This - includes derivations and the outputs of - [content-addressed derivations](#gloss-content-addressed-derivation) + includes derivations, the outputs of + [content-addressed derivations](#gloss-content-addressed-derivation), + and the outputs of + [fixed-output derivations](#gloss-fixed-output-derivation). - [substitute]{#gloss-substitute}\ A substitute is a command invocation stored in the Nix database that From 1b2b8c39fd64871b3df261c05ed001fcd1057a6c Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 14:47:17 -0700 Subject: [PATCH 22/93] fix link to language/advanced-attributes.md#adv-attr-contentAddressed --- doc/manual/src/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 473aef03c..bf1d0cf05 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -9,7 +9,7 @@ - [content-addressed derivation]{#gloss-content-addressed-derivation}\ A derivation which has the - [`__contentAddressed`](language/advanced-attributes.md#contentAddressed) + [`__contentAddressed`](language/advanced-attributes.md#adv-attr-contentAddressed) attribute set to `true`. - [fixed-output derivation]{#gloss-fixed-output-derivation}\ From def4fb9a0f73046efbf9fdb4f1e35898fb27ca34 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 14:47:33 -0700 Subject: [PATCH 23/93] __outputHash: add link --- doc/manual/src/glossary.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index bf1d0cf05..d61cfc823 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -13,8 +13,8 @@ attribute set to `true`. - [fixed-output derivation]{#gloss-fixed-output-derivation}\ - A derivation which includes the `__outputHash` attribute; the output - of such derivations must exactly match the hash. + A derivation which includes the + [`__outputHash`](language/advanced-attributes.md#adv-attr-outputHash) attribute. - [store]{#gloss-store}\ The location in the file system where store objects live. Typically From 8139bbe2ba767458cba4158627ee3d58f4a35d7d Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 15:01:07 -0700 Subject: [PATCH 24/93] implement https://github.com/NixOS/nix/pull/6870#pullrequestreview-1093700220 --- doc/manual/src/glossary.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index d61cfc823..b30633833 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -74,8 +74,8 @@ - [substituter]{#gloss-substituter}\ A *substituter* is an additional store from which Nix will - copy store objects it doesn't have. Nix will not copy a store - path from a remote store unless one of the following is true: + copy store objects it doesn't have. Nix will copy a store + path from a remote store only if one of the following is true: - the store object is signed by one of the `trusted-public-keys` - the substituter is in the `trusted-substituters` list From 57f12df5e4cde436566d3c4f0226d329e6eedf1a Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 15:09:10 -0700 Subject: [PATCH 25/93] input-addressed store object: include FODOs --- doc/manual/src/glossary.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index b30633833..89a9b94a1 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -54,8 +54,9 @@ derivations (files describing a build action). - [input-addressed store object]{#gloss-input-addressed-store-object}\ - Store objects produced by building a - non-[content-addressed](#gloss-content-addressed-derivation) + A store object produced by building a + non-[content-addressed](#gloss-content-addressed-derivation), + non-[fixed-output](#gloss-fixed-output-derivation), derivation. - [output-addressed store object]{#gloss-output-addressed-store-object}\ From d5e064d8162e377556dc9daba99868085561a080 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 17:46:31 -0700 Subject: [PATCH 26/93] glossary: fix broken link --- doc/manual/src/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 89a9b94a1..1aebdaa67 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -9,7 +9,7 @@ - [content-addressed derivation]{#gloss-content-addressed-derivation}\ A derivation which has the - [`__contentAddressed`](language/advanced-attributes.md#adv-attr-contentAddressed) + [`__contentAddressed`](language/advanced-attributes.md#adv-attr-__contentAddressed) attribute set to `true`. - [fixed-output derivation]{#gloss-fixed-output-derivation}\ From 887e922be29d37d377ef766c7fe7a2103f43ca21 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 17:47:13 -0700 Subject: [PATCH 27/93] glossary: outputHash, not __outputHash --- doc/manual/src/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 1aebdaa67..73e209103 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -14,7 +14,7 @@ - [fixed-output derivation]{#gloss-fixed-output-derivation}\ A derivation which includes the - [`__outputHash`](language/advanced-attributes.md#adv-attr-outputHash) attribute. + [`outputHash`](language/advanced-attributes.md#adv-attr-outputHash) attribute. - [store]{#gloss-store}\ The location in the file system where store objects live. Typically From f6c750e8b2f299e9876fd8f2578f9093682f6d7f Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 17:48:34 -0700 Subject: [PATCH 28/93] glossary: remove extraneous comma --- doc/manual/src/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 73e209103..7ba595ba0 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -56,7 +56,7 @@ - [input-addressed store object]{#gloss-input-addressed-store-object}\ A store object produced by building a non-[content-addressed](#gloss-content-addressed-derivation), - non-[fixed-output](#gloss-fixed-output-derivation), + non-[fixed-output](#gloss-fixed-output-derivation) derivation. - [output-addressed store object]{#gloss-output-addressed-store-object}\ From 59dc8346ca53f49ccdbbd6709b12a479376d1464 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 17:51:56 -0700 Subject: [PATCH 29/93] move substituter signature-checking conditions to configuration file documentation --- doc/manual/src/glossary.md | 9 ++------- src/libstore/globals.hh | 8 ++++++++ 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 7ba595ba0..6bf041e7c 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -75,13 +75,8 @@ - [substituter]{#gloss-substituter}\ A *substituter* is an additional store from which Nix will - copy store objects it doesn't have. Nix will copy a store - path from a remote store only if one of the following is true: - - - the store object is signed by one of the `trusted-public-keys` - - the substituter is in the `trusted-substituters` list - - the `no-require-sigs` option has been set to disable signature checking - - the store object is [output-addressed](#gloss-output-addressed-store-object) + copy store objects it doesn't have. For details, see the + [`substituters` option](command-ref/conf-file.html#conf-substituters). - [purity]{#gloss-purity}\ The assumption that equal Nix derivations when run always produce diff --git a/src/libstore/globals.hh b/src/libstore/globals.hh index d7f351166..a659036e2 100644 --- a/src/libstore/globals.hh +++ b/src/libstore/globals.hh @@ -613,6 +613,14 @@ public: are tried based on their Priority value, which each substituter can set independently. Lower value means higher priority. The default is `https://cache.nixos.org`, with a Priority of 40. + + Nix will copy a store path from a remote store only if one + of the following is true: + + - the store object is signed by one of the `trusted-public-keys` + - the substituter is in the `trusted-substituters` list + - the `no-require-sigs` option has been set to disable signature checking + - the store object is [output-addressed](#gloss-output-addressed-store-object) )", {"binary-caches"}}; From 1ab913467ef8e9ff946e64bd31841775d743b2d6 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 17:54:23 -0700 Subject: [PATCH 30/93] linkify mention of other options --- src/libstore/globals.hh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/libstore/globals.hh b/src/libstore/globals.hh index a659036e2..a4db3bf08 100644 --- a/src/libstore/globals.hh +++ b/src/libstore/globals.hh @@ -617,10 +617,10 @@ public: Nix will copy a store path from a remote store only if one of the following is true: - - the store object is signed by one of the `trusted-public-keys` - - the substituter is in the `trusted-substituters` list - - the `no-require-sigs` option has been set to disable signature checking - - the store object is [output-addressed](#gloss-output-addressed-store-object) + - the store object is signed by one of the [`trusted-public-keys`](#conf-trusted-public-keys) + - the substituter is in the [`trusted-substituters`](#conf-trusted-substituters) list + - the [`require-sigs`](#conf-require-sigs) option has been set to `false` + - the store object is [output-addressed](glossary.md#gloss-output-addressed-store-object) )", {"binary-caches"}}; From e6f5352e71a1811eb2eb3bfb989e109de590c7a7 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 18:27:00 -0700 Subject: [PATCH 31/93] #binary-cache -> #gloss-binary-cache --- doc/manual/src/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 6bf041e7c..a34b8a60c 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -34,7 +34,7 @@ directory on another machine, accessed via `ssh` or served by the `nix-serve` Perl script. - - [binary cache]{#binary-cache}\ + - [binary cache]{#gloss-binary-cache}\ A *binary cache* is a Nix store which uses a different format: its metadata and signatures are kept in `.narinfo` files rather than in a Nix database. This different format simplifies serving store objects From e90f2fcfc71ca997f254c86f8ed12fc143374752 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 18:28:05 -0700 Subject: [PATCH 32/93] glossary: add entry for `chroot store` (used 11 times in nix) --- doc/manual/src/glossary.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index a34b8a60c..70a0eb994 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -34,6 +34,9 @@ directory on another machine, accessed via `ssh` or served by the `nix-serve` Perl script. + - [chroot store]{#gloss-chroot-store}\ + A local store whose canonical path is anything other than `/nix/store`. + - [binary cache]{#gloss-binary-cache}\ A *binary cache* is a Nix store which uses a different format: its metadata and signatures are kept in `.narinfo` files rather than in a From 27be54ca533933db8c3e0cde4b213abf10dd5237 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Tue, 6 Sep 2022 18:27:39 +0200 Subject: [PATCH 33/93] nix develop: Ignore stdenv's $SHELL Stdenv sets this to a bash that doesn't have readline/completion support, so running 'nix (develop|shell)' inside a 'nix develop' gives you a crippled shell. So let's just ignore the derivation's $SHELL. This could break interactive use of build phases that use $SHELL, but they appear to be fairly rare. --- src/nix/develop.cc | 1 + 1 file changed, 1 insertion(+) diff --git a/src/nix/develop.cc b/src/nix/develop.cc index ba7ba7c25..4de109754 100644 --- a/src/nix/develop.cc +++ b/src/nix/develop.cc @@ -246,6 +246,7 @@ struct Common : InstallableCommand, MixProfile "NIX_LOG_FD", "NIX_REMOTE", "PPID", + "SHELL", "SHELLOPTS", "SSL_CERT_FILE", // FIXME: only want to ignore /no-cert-file.crt "TEMP", From ece12a97d9c7e0024ebddb9e5eb0c919a9efb694 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Tue, 6 Sep 2022 19:20:31 +0200 Subject: [PATCH 34/93] lockfile -> lock file for consistency --- src/libexpr/flake/flake.cc | 4 ++-- src/libexpr/flake/lockfile.cc | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/libexpr/flake/flake.cc b/src/libexpr/flake/flake.cc index 105e76bc6..119c556ac 100644 --- a/src/libexpr/flake/flake.cc +++ b/src/libexpr/flake/flake.cc @@ -483,12 +483,12 @@ LockedFlake lockFlake( } else if (auto follows = std::get_if<1>(&i.second)) { if (! trustLock) { // It is possible that the flake has changed, - // so we must confirm all the follows that are in the lockfile are also in the flake. + // so we must confirm all the follows that are in the lock file are also in the flake. auto overridePath(inputPath); overridePath.push_back(i.first); auto o = overrides.find(overridePath); // If the override disappeared, we have to refetch the flake, - // since some of the inputs may not be present in the lockfile. + // since some of the inputs may not be present in the lock file. if (o == overrides.end()) { mustRefetch = true; // There's no point populating the rest of the fake inputs, diff --git a/src/libexpr/flake/lockfile.cc b/src/libexpr/flake/lockfile.cc index 60b52d578..629d2e669 100644 --- a/src/libexpr/flake/lockfile.cc +++ b/src/libexpr/flake/lockfile.cc @@ -36,7 +36,7 @@ LockedNode::LockedNode(const nlohmann::json & json) , isFlake(json.find("flake") != json.end() ? (bool) json["flake"] : true) { if (!lockedRef.input.isLocked()) - throw Error("lockfile contains mutable lock '%s'", + throw Error("lock file contains mutable lock '%s'", fetchers::attrsToJSON(lockedRef.input.toAttrs())); } From fae3b4fe8abc2b307a583e396a24d7899bb21451 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Wed, 14 Sep 2022 15:40:43 +0200 Subject: [PATCH 35/93] Add an installer test This runs the installer in a QEMU VM. Unlike the old installer test that ran inside a declaratively built RedHat/Debian image, this uses an image from Vagrant. --- flake.nix | 5 + tests/installer/default.nix | 136 +++++++++++++++++++++++++++ tests/installer/vagrant_insecure_key | 27 ++++++ 3 files changed, 168 insertions(+) create mode 100644 tests/installer/default.nix create mode 100644 tests/installer/vagrant_insecure_key diff --git a/flake.nix b/flake.nix index cdb81179a..ec64719d1 100644 --- a/flake.nix +++ b/flake.nix @@ -546,6 +546,11 @@ # againstLatestStable = testNixVersions pkgs pkgs.nix pkgs.nixStable; } "touch $out"); + installerTests = import ./tests/installer { + binaryTarballs = self.hydraJobs.binaryTarball; + inherit nixpkgsFor; + }; + }; checks = forAllSystems (system: { diff --git a/tests/installer/default.nix b/tests/installer/default.nix new file mode 100644 index 000000000..39911aeb2 --- /dev/null +++ b/tests/installer/default.nix @@ -0,0 +1,136 @@ +{ binaryTarballs +, nixpkgsFor +}: + +let + + installScripts = { + install-default = { + script = '' + set -eux + + tar -xf ./nix.tar.xz + mv ./nix-* nix + ./nix/install --no-channel-add + ''; + }; + + install-force-no-daemon = { + script = '' + set -eux + + tar -xf ./nix.tar.xz + mv ./nix-* nix + ./nix/install --no-daemon + ''; + }; + + install-force-daemon = { + script = '' + set -eux + + tar -xf ./nix.tar.xz + mv ./nix-* nix + ./nix/install --daemon + ''; + }; + }; + + images = { + + "ubuntu-14-04" = { + image = import { + url = https://app.vagrantup.com/ubuntu/boxes/trusty64/versions/20190514.0.0/providers/virtualbox.box; + hash = "sha256-iUUXyRY8iW7DGirb0zwGgf1fRbLA7wimTJKgP7l/OQ8="; + }; + rootDisk = "box-disk1.vmdk"; + system = "x86_64-linux"; + }; + + "ubuntu-16-04" = { + image = import { + url = https://app.vagrantup.com/ubuntu/boxes/xenial64/versions/20211001.0.0/providers/virtualbox.box; + hash = "sha256-JCc0wd9vaSzCU8coByVtb/oDTAXYBPnORwEShS4oj4U="; + }; + rootDisk = "ubuntu-xenial-16.04-cloudimg.vmdk"; + system = "x86_64-linux"; + }; + + "ubuntu-22-10" = { + image = import { + url = https://app.vagrantup.com/ubuntu/boxes/kinetic64/versions/20220910.0.0/providers/virtualbox.box; + hash = "sha256-/IXr+Apyx2dqX6Gj4SoNtQ/5v1eKKopwzFgozAq6GFY="; + }; + rootDisk = "ubuntu-kinetic-22.10-cloudimg.vmdk"; + system = "x86_64-linux"; + }; + + }; + + makeTest = imageName: testName: + let image = images.${imageName}; in + with nixpkgsFor.${image.system}; + runCommand + "installer-test-${imageName}-${testName}" + { buildInputs = [ qemu_kvm openssh ]; + image = image.image; + installScript = installScripts.${testName}.script; + binaryTarball = binaryTarballs.${system}; + } + '' + echo "Unpacking Vagrant box..." + tar xvf $image + + qemu-img create -b ./${image.rootDisk} -F vmdk -f qcow2 ./disk.qcow2 + + echo "Starting qemu..." + qemu-kvm -m 4096 -nographic \ + -drive id=disk1,file=./disk.qcow2,if=virtio \ + -netdev user,id=net0,restrict=yes,hostfwd=tcp::20022-:22 -device virtio-net-pci,netdev=net0 & + qemu_pid=$! + trap "kill $qemu_pid" EXIT + + if ! [ -e ./vagrant_insecure_key ]; then + cp ${./vagrant_insecure_key} vagrant_insecure_key + fi + + chmod 0400 ./vagrant_insecure_key + + ssh_opts="-o StrictHostKeyChecking=no -o PubkeyAcceptedKeyTypes=+ssh-rsa -i ./vagrant_insecure_key" + ssh="ssh -p 20022 -q $ssh_opts vagrant@localhost" + + echo "Waiting for SSH..." + for ((i = 0; i < 120; i++)); do + echo "[ssh] Trying to connect..." + if $ssh -- true; then + echo "[ssh] Connected!" + break + fi + if ! kill -0 $qemu_pid; then + echo "qemu died unexpectedly" + exit 1 + fi + sleep 1 + done + + echo "Copying installer..." + scp -P 20022 $ssh_opts $binaryTarball/nix-*.tar.xz vagrant@localhost:nix.tar.xz + + echo "Running installer..." + $ssh "$installScript" + + echo "Testing Nix installation..." + # FIXME: should update ~/.bashrc. + $ssh "source ~/.profile; nix-env --version" + + echo "Done!" + touch $out + ''; + +in + +{ + ubuntu-14-04.install-default = makeTest "ubuntu-14-04" "install-default"; + #ubuntu-16-04.install-default = makeTest "ubuntu-16-04" "install-default"; + #ubuntu-22-10.install-default = makeTest "ubuntu-22-10" "install-default"; +} diff --git a/tests/installer/vagrant_insecure_key b/tests/installer/vagrant_insecure_key new file mode 100644 index 000000000..7d6a08390 --- /dev/null +++ b/tests/installer/vagrant_insecure_key @@ -0,0 +1,27 @@ +-----BEGIN RSA PRIVATE KEY----- +MIIEogIBAAKCAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzI +w+niNltGEFHzD8+v1I2YJ6oXevct1YeS0o9HZyN1Q9qgCgzUFtdOKLv6IedplqoP +kcmF0aYet2PkEDo3MlTBckFXPITAMzF8dJSIFo9D8HfdOV0IAdx4O7PtixWKn5y2 +hMNG0zQPyUecp4pzC6kivAIhyfHilFR61RGL+GPXQ2MWZWFYbAGjyiYJnAmCP3NO +Td0jMZEnDkbUvxhMmBYSdETk1rRgm+R4LOzFUGaHqHDLKLX+FIPKcF96hrucXzcW +yLbIbEgE98OHlnVYCzRdK8jlqm8tehUc9c9WhQIBIwKCAQEA4iqWPJXtzZA68mKd +ELs4jJsdyky+ewdZeNds5tjcnHU5zUYE25K+ffJED9qUWICcLZDc81TGWjHyAqD1 +Bw7XpgUwFgeUJwUlzQurAv+/ySnxiwuaGJfhFM1CaQHzfXphgVml+fZUvnJUTvzf +TK2Lg6EdbUE9TarUlBf/xPfuEhMSlIE5keb/Zz3/LUlRg8yDqz5w+QWVJ4utnKnK +iqwZN0mwpwU7YSyJhlT4YV1F3n4YjLswM5wJs2oqm0jssQu/BT0tyEXNDYBLEF4A +sClaWuSJ2kjq7KhrrYXzagqhnSei9ODYFShJu8UWVec3Ihb5ZXlzO6vdNQ1J9Xsf +4m+2ywKBgQD6qFxx/Rv9CNN96l/4rb14HKirC2o/orApiHmHDsURs5rUKDx0f9iP +cXN7S1uePXuJRK/5hsubaOCx3Owd2u9gD6Oq0CsMkE4CUSiJcYrMANtx54cGH7Rk +EjFZxK8xAv1ldELEyxrFqkbE4BKd8QOt414qjvTGyAK+OLD3M2QdCQKBgQDtx8pN +CAxR7yhHbIWT1AH66+XWN8bXq7l3RO/ukeaci98JfkbkxURZhtxV/HHuvUhnPLdX +3TwygPBYZFNo4pzVEhzWoTtnEtrFueKxyc3+LjZpuo+mBlQ6ORtfgkr9gBVphXZG +YEzkCD3lVdl8L4cw9BVpKrJCs1c5taGjDgdInQKBgHm/fVvv96bJxc9x1tffXAcj +3OVdUN0UgXNCSaf/3A/phbeBQe9xS+3mpc4r6qvx+iy69mNBeNZ0xOitIjpjBo2+ +dBEjSBwLk5q5tJqHmy/jKMJL4n9ROlx93XS+njxgibTvU6Fp9w+NOFD/HvxB3Tcz +6+jJF85D5BNAG3DBMKBjAoGBAOAxZvgsKN+JuENXsST7F89Tck2iTcQIT8g5rwWC +P9Vt74yboe2kDT531w8+egz7nAmRBKNM751U/95P9t88EDacDI/Z2OwnuFQHCPDF +llYOUI+SpLJ6/vURRbHSnnn8a/XG+nzedGH5JGqEJNQsz+xT2axM0/W/CRknmGaJ +kda/AoGANWrLCz708y7VYgAtW2Uf1DPOIYMdvo6fxIB5i9ZfISgcJ/bbCUkFrhoH ++vq/5CIWxCPp0f85R4qxxQ5ihxJ0YDQT9Jpx4TMss4PSavPaBH3RXow5Ohe+bYoQ +NE5OgEXk2wVfZczCZpigBKbKZHNYcelXtTt/nP3rsCuGcM4h53s= +-----END RSA PRIVATE KEY----- From 0a8e666dd6d18ede4b5cd648e19d5950ee19f095 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Wed, 14 Sep 2022 18:40:16 +0200 Subject: [PATCH 36/93] Add Fedora 36 --- tests/installer/default.nix | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index 39911aeb2..ab3ef62f0 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -65,6 +65,16 @@ let system = "x86_64-linux"; }; + + "fedora-36" = { + image = import { + url = https://app.vagrantup.com/generic/boxes/fedora36/versions/4.1.12/providers/libvirt.box; + hash = "sha256-rxPgnDnFkTDwvdqn2CV3ZUo3re9AdPtSZ9SvOHNvaks="; + }; + rootDisk = "box.img"; + system = "x86_64-linux"; + }; + }; makeTest = imageName: testName: @@ -78,10 +88,12 @@ let binaryTarball = binaryTarballs.${system}; } '' - echo "Unpacking Vagrant box..." + echo "Unpacking Vagrant box $image..." tar xvf $image - qemu-img create -b ./${image.rootDisk} -F vmdk -f qcow2 ./disk.qcow2 + image_type=$(qemu-img info ${image.rootDisk} | sed 's/file format: \(.*\)/\1/; t; d') + + qemu-img create -b ./${image.rootDisk} -F "$image_type" -f qcow2 ./disk.qcow2 echo "Starting qemu..." qemu-kvm -m 4096 -nographic \ @@ -121,7 +133,7 @@ let echo "Testing Nix installation..." # FIXME: should update ~/.bashrc. - $ssh "source ~/.profile; nix-env --version" + $ssh "source ~/.bash_profile || source ~/.bash_login || source ~/.profile || true; nix-env --version" echo "Done!" touch $out @@ -133,4 +145,5 @@ in ubuntu-14-04.install-default = makeTest "ubuntu-14-04" "install-default"; #ubuntu-16-04.install-default = makeTest "ubuntu-16-04" "install-default"; #ubuntu-22-10.install-default = makeTest "ubuntu-22-10" "install-default"; + fedora-36.install-default = makeTest "fedora-36" "install-default"; } From 906c947ee8f2478d27e5eda649f44716e952d8a6 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Wed, 14 Sep 2022 18:53:30 +0200 Subject: [PATCH 37/93] Enable daemon installation test on Fedora --- tests/installer/default.nix | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index ab3ef62f0..8b6bde73f 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -7,8 +7,6 @@ let installScripts = { install-default = { script = '' - set -eux - tar -xf ./nix.tar.xz mv ./nix-* nix ./nix/install --no-channel-add @@ -17,8 +15,6 @@ let install-force-no-daemon = { script = '' - set -eux - tar -xf ./nix.tar.xz mv ./nix-* nix ./nix/install --no-daemon @@ -27,15 +23,15 @@ let install-force-daemon = { script = '' - set -eux - tar -xf ./nix.tar.xz mv ./nix-* nix - ./nix/install --daemon + ./nix/install --daemon --no-channel-add ''; }; }; + disableSELinux = "sudo setenforce 0"; + images = { "ubuntu-14-04" = { @@ -65,7 +61,6 @@ let system = "x86_64-linux"; }; - "fedora-36" = { image = import { url = https://app.vagrantup.com/generic/boxes/fedora36/versions/4.1.12/providers/libvirt.box; @@ -73,6 +68,7 @@ let }; rootDisk = "box.img"; system = "x86_64-linux"; + postBoot = disableSELinux; }; }; @@ -84,6 +80,7 @@ let "installer-test-${imageName}-${testName}" { buildInputs = [ qemu_kvm openssh ]; image = image.image; + postBoot = image.postBoot or ""; installScript = installScripts.${testName}.script; binaryTarball = binaryTarballs.${system}; } @@ -125,15 +122,25 @@ let sleep 1 done + if [[ -n $postBoot ]]; then + echo "Running post-boot commands..." + $ssh "set -ex; $postBoot" + fi + echo "Copying installer..." scp -P 20022 $ssh_opts $binaryTarball/nix-*.tar.xz vagrant@localhost:nix.tar.xz echo "Running installer..." - $ssh "$installScript" + $ssh "set -eux; $installScript" echo "Testing Nix installation..." # FIXME: should update ~/.bashrc. - $ssh "source ~/.bash_profile || source ~/.bash_login || source ~/.profile || true; nix-env --version" + $ssh " + set -ex + source ~/.bash_profile || source ~/.bash_login || source ~/.profile || true + nix-env --version + nix --extra-experimental-features nix-command store ping + " echo "Done!" touch $out @@ -146,4 +153,5 @@ in #ubuntu-16-04.install-default = makeTest "ubuntu-16-04" "install-default"; #ubuntu-22-10.install-default = makeTest "ubuntu-22-10" "install-default"; fedora-36.install-default = makeTest "fedora-36" "install-default"; + fedora-36.install-force-daemon = makeTest "fedora-36" "install-force-daemon"; } From cc6e31231547fc64c89c6682316f2bab03db6879 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Wed, 14 Sep 2022 19:44:41 +0200 Subject: [PATCH 38/93] Get Ubuntu 22.10 to work --- tests/installer/default.nix | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index 8b6bde73f..9b2a34bdb 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -85,6 +85,8 @@ let binaryTarball = binaryTarballs.${system}; } '' + shopt -s nullglob + echo "Unpacking Vagrant box $image..." tar xvf $image @@ -92,10 +94,19 @@ let qemu-img create -b ./${image.rootDisk} -F "$image_type" -f qcow2 ./disk.qcow2 + extra_qemu_opts= + + # Add the config disk, required by the Ubuntu images. + config_drive=$(echo *configdrive.vmdk || true) + if [[ -n $config_drive ]]; then + extra_qemu_opts+=" -drive id=disk2,file=$config_drive,if=virtio" + fi + echo "Starting qemu..." qemu-kvm -m 4096 -nographic \ -drive id=disk1,file=./disk.qcow2,if=virtio \ - -netdev user,id=net0,restrict=yes,hostfwd=tcp::20022-:22 -device virtio-net-pci,netdev=net0 & + -netdev user,id=net0,restrict=yes,hostfwd=tcp::20022-:22 -device virtio-net-pci,netdev=net0 \ + $extra_qemu_opts & qemu_pid=$! trap "kill $qemu_pid" EXIT @@ -137,7 +148,13 @@ let # FIXME: should update ~/.bashrc. $ssh " set -ex - source ~/.bash_profile || source ~/.bash_login || source ~/.profile || true + + # FIXME: get rid of this; ideally ssh should just work. + source ~/.bash_profile || true + source ~/.bash_login || true + source ~/.profile || true + source /etc/bashrc || true + nix-env --version nix --extra-experimental-features nix-command store ping " @@ -151,7 +168,8 @@ in { ubuntu-14-04.install-default = makeTest "ubuntu-14-04" "install-default"; #ubuntu-16-04.install-default = makeTest "ubuntu-16-04" "install-default"; - #ubuntu-22-10.install-default = makeTest "ubuntu-22-10" "install-default"; + ubuntu-22-10.install-default = makeTest "ubuntu-22-10" "install-default"; + ubuntu-22-10.install-force-daemon = makeTest "ubuntu-22-10" "install-force-daemon"; fedora-36.install-default = makeTest "fedora-36" "install-default"; fedora-36.install-force-daemon = makeTest "fedora-36" "install-force-daemon"; } From 02af02854d41b390957300bac778139bc1c6b5c2 Mon Sep 17 00:00:00 2001 From: Matthew Kenigsberg Date: Wed, 14 Sep 2022 15:35:56 -0600 Subject: [PATCH 39/93] dockerImage: fix root shell Currently root's shell is set to a path that does not exist; this change sets it to the correct path to bash --- docker.nix | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker.nix b/docker.nix index e95caf274..bb2b4e7ff 100644 --- a/docker.nix +++ b/docker.nix @@ -33,7 +33,7 @@ let root = { uid = 0; - shell = "/bin/bash"; + shell = "${pkgs.bashInteractive}/bin/bash"; home = "/root"; gid = 0; }; From fe958a682d293dec5f27d0c161833b453370d755 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 Sep 2022 11:42:10 +0200 Subject: [PATCH 40/93] Test building --- tests/installer/default.nix | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index 9b2a34bdb..d0707018f 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -146,7 +146,7 @@ let echo "Testing Nix installation..." # FIXME: should update ~/.bashrc. - $ssh " + $ssh < \$out"]; }') + [[ \$(cat \$out) = foobar ]] + EOF echo "Done!" touch $out From 29aaec1e593f1837a73779f243ed0ec4220f7ea8 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 Sep 2022 13:15:26 +0200 Subject: [PATCH 41/93] Make cross product of images and tests --- tests/installer/default.nix | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index d0707018f..72ad764cc 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -168,11 +168,9 @@ let in -{ - ubuntu-14-04.install-default = makeTest "ubuntu-14-04" "install-default"; - #ubuntu-16-04.install-default = makeTest "ubuntu-16-04" "install-default"; - ubuntu-22-10.install-default = makeTest "ubuntu-22-10" "install-default"; - ubuntu-22-10.install-force-daemon = makeTest "ubuntu-22-10" "install-force-daemon"; - fedora-36.install-default = makeTest "fedora-36" "install-default"; - fedora-36.install-force-daemon = makeTest "fedora-36" "install-force-daemon"; -} +builtins.mapAttrs (imageName: image: + { ${image.system} = builtins.mapAttrs (testName: test: + makeTest imageName testName + ) installScripts; + } +) images From 5c8cdb9b60e0e8d24458a15577e4be3aaa16b600 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 Sep 2022 13:19:46 +0200 Subject: [PATCH 42/93] Add Ubuntu 22.04 LTS --- tests/installer/default.nix | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index 72ad764cc..17b0fd4ea 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -52,6 +52,15 @@ let system = "x86_64-linux"; }; + "ubuntu-22-04" = { + image = import { + url = https://app.vagrantup.com/generic/boxes/ubuntu2204/versions/4.1.12/providers/libvirt.box; + hash = "sha256-HNll0Qikw/xGIcogni5lz01vUv+R3o8xowP2EtqjuUQ="; + }; + rootDisk = "box.img"; + system = "x86_64-linux"; + }; + "ubuntu-22-10" = { image = import { url = https://app.vagrantup.com/ubuntu/boxes/kinetic64/versions/20220910.0.0/providers/virtualbox.box; From ef714aa8a566bbdb30919ffd45b8a1fd8e2bc484 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 Sep 2022 13:25:26 +0200 Subject: [PATCH 43/93] Remove pre-release Ubuntu 22.10 --- tests/installer/default.nix | 9 --------- 1 file changed, 9 deletions(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index 17b0fd4ea..a2cbbcbb8 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -61,15 +61,6 @@ let system = "x86_64-linux"; }; - "ubuntu-22-10" = { - image = import { - url = https://app.vagrantup.com/ubuntu/boxes/kinetic64/versions/20220910.0.0/providers/virtualbox.box; - hash = "sha256-/IXr+Apyx2dqX6Gj4SoNtQ/5v1eKKopwzFgozAq6GFY="; - }; - rootDisk = "ubuntu-kinetic-22.10-cloudimg.vmdk"; - system = "x86_64-linux"; - }; - "fedora-36" = { image = import { url = https://app.vagrantup.com/generic/boxes/fedora36/versions/4.1.12/providers/libvirt.box; From 503f31e2a0de3192ccc572cc17a6dd02863ebec8 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 Sep 2022 13:28:03 +0200 Subject: [PATCH 44/93] Use libvirt image --- tests/installer/default.nix | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index a2cbbcbb8..39e3c8d26 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -45,10 +45,10 @@ let "ubuntu-16-04" = { image = import { - url = https://app.vagrantup.com/ubuntu/boxes/xenial64/versions/20211001.0.0/providers/virtualbox.box; - hash = "sha256-JCc0wd9vaSzCU8coByVtb/oDTAXYBPnORwEShS4oj4U="; + url = https://app.vagrantup.com/generic/boxes/ubuntu1604/versions/4.1.12/providers/libvirt.box; + hash = "sha256-lO4oYQR2tCh5auxAYe6bPOgEqOgv3Y3GC1QM1tEEEU8="; }; - rootDisk = "ubuntu-xenial-16.04-cloudimg.vmdk"; + rootDisk = "box.img"; system = "x86_64-linux"; }; From a96ad2ab25ea054b9d1c473ce2f692bd1f83402b Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 Sep 2022 14:51:10 +0200 Subject: [PATCH 45/93] Add RHEL 7/8 --- tests/installer/default.nix | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index 39e3c8d26..d31e2a949 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -71,6 +71,38 @@ let postBoot = disableSELinux; }; + # Currently fails with 'error while loading shared libraries: + # libsodium.so.23: cannot stat shared object: Invalid argument'. + /* + "rhel-6" = { + image = import { + url = https://app.vagrantup.com/generic/boxes/rhel6/versions/4.1.12/providers/libvirt.box; + hash = "sha256-QwzbvRoRRGqUCQptM7X/InRWFSP2sqwRt2HaaO6zBGM="; + }; + rootDisk = "box.img"; + system = "x86_64-linux"; + }; + */ + + "rhel-7" = { + image = import { + url = https://app.vagrantup.com/generic/boxes/rhel7/versions/4.1.12/providers/libvirt.box; + hash = "sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="; + }; + rootDisk = "box.img"; + system = "x86_64-linux"; + }; + + "rhel-8" = { + image = import { + url = https://app.vagrantup.com/generic/boxes/rhel8/versions/4.1.12/providers/libvirt.box; + hash = "sha256-zFOPjSputy1dPgrQRixBXmlyN88cAKjJ21VvjSWUCUY="; + }; + rootDisk = "box.img"; + system = "x86_64-linux"; + postBoot = disableSELinux; + }; + }; makeTest = imageName: testName: @@ -116,7 +148,7 @@ let chmod 0400 ./vagrant_insecure_key - ssh_opts="-o StrictHostKeyChecking=no -o PubkeyAcceptedKeyTypes=+ssh-rsa -i ./vagrant_insecure_key" + ssh_opts="-o StrictHostKeyChecking=no -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedKeyTypes=+ssh-rsa -i ./vagrant_insecure_key" ssh="ssh -p 20022 -q $ssh_opts vagrant@localhost" echo "Waiting for SSH..." From ba04b5b1d74c285e12fc3d24524cb8f30f108767 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 Sep 2022 14:51:44 +0200 Subject: [PATCH 46/93] Disable Ubuntu 14.04 --- tests/installer/default.nix | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index d31e2a949..a3a7f85f9 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -34,6 +34,7 @@ let images = { + /* "ubuntu-14-04" = { image = import { url = https://app.vagrantup.com/ubuntu/boxes/trusty64/versions/20190514.0.0/providers/virtualbox.box; @@ -42,6 +43,7 @@ let rootDisk = "box-disk1.vmdk"; system = "x86_64-linux"; }; + */ "ubuntu-16-04" = { image = import { From 3dd313a7c20772ef34af4a43fb3673df1e7d00cf Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 Sep 2022 15:50:52 +0200 Subject: [PATCH 47/93] Add RHEL 9 --- tests/installer/default.nix | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index a3a7f85f9..eab103562 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -105,6 +105,17 @@ let postBoot = disableSELinux; }; + "rhel-9" = { + image = import { + url = https://app.vagrantup.com/generic/boxes/rhel9/versions/4.1.12/providers/libvirt.box; + hash = "sha256-vL/FbB3kK1rcSaR627nWmScYGKGk4seSmAdq6N5diMg="; + }; + rootDisk = "box.img"; + system = "x86_64-linux"; + postBoot = disableSELinux; + extraQemuOpts = "-cpu Westmere-v2"; + }; + }; makeTest = imageName: testName: @@ -128,7 +139,7 @@ let qemu-img create -b ./${image.rootDisk} -F "$image_type" -f qcow2 ./disk.qcow2 - extra_qemu_opts= + extra_qemu_opts="${image.extraQemuOpts}" # Add the config disk, required by the Ubuntu images. config_drive=$(echo *configdrive.vmdk || true) From 0d4bf9c4d836f8e8570ad1d39245a5835ef4aaf1 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 Sep 2022 15:56:46 +0200 Subject: [PATCH 48/93] Fix evaluation --- tests/installer/default.nix | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index eab103562..c118937a6 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -139,7 +139,7 @@ let qemu-img create -b ./${image.rootDisk} -F "$image_type" -f qcow2 ./disk.qcow2 - extra_qemu_opts="${image.extraQemuOpts}" + extra_qemu_opts="${image.extraQemuOpts or ""}" # Add the config disk, required by the Ubuntu images. config_drive=$(echo *configdrive.vmdk || true) From 47fa1087c8864654f6cfae84e25ac6db318ed1d4 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 16 Sep 2022 09:36:20 +0200 Subject: [PATCH 49/93] Update doc/manual/src/contributing/hacking.md --- doc/manual/src/contributing/hacking.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/manual/src/contributing/hacking.md b/doc/manual/src/contributing/hacking.md index d8a8c8591..628744bf2 100644 --- a/doc/manual/src/contributing/hacking.md +++ b/doc/manual/src/contributing/hacking.md @@ -142,8 +142,8 @@ After the CI run completes, you can check the output to extract the installer ur 1. Click into the detailed view of the CI run. 2. Click into any `installer_test` run (the URL you're here to extract will be the same in all of them). 3. Click into the `Run cachix/install-nix-action@v...` step and click the detail triangle next to the first log line (it will also be `Run cachix/install-nix-action@v...`) -4. Copy the install_url -5. To generate an install command, plug this install_url and your github username into this template: +4. Copy the value of `install_url` +5. To generate an install command, plug this `install_url` and your GitHub username into this template: ```console sh <(curl -L ) --tarball-url-prefix https://-nix-install-tests.cachix.org/serve From 0a4bd9fe88807cbae51b8b8b51c4897a76991d20 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 16 Sep 2022 09:36:30 +0200 Subject: [PATCH 50/93] Update doc/manual/src/contributing/hacking.md --- doc/manual/src/contributing/hacking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/contributing/hacking.md b/doc/manual/src/contributing/hacking.md index 628744bf2..9e4e679e6 100644 --- a/doc/manual/src/contributing/hacking.md +++ b/doc/manual/src/contributing/hacking.md @@ -138,7 +138,7 @@ If you've already pushed to a fork of Nix on GitHub before, you may have noticed ### Using the CI-generated installer for manual testing -After the CI run completes, you can check the output to extract the installer url: +After the CI run completes, you can check the output to extract the installer URL: 1. Click into the detailed view of the CI run. 2. Click into any `installer_test` run (the URL you're here to extract will be the same in all of them). 3. Click into the `Run cachix/install-nix-action@v...` step and click the detail triangle next to the first log line (it will also be `Run cachix/install-nix-action@v...`) From 1ae974120a24f70eba12e073dbba4b7bac73eedf Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 16 Sep 2022 09:36:37 +0200 Subject: [PATCH 51/93] Update doc/manual/src/contributing/hacking.md --- doc/manual/src/contributing/hacking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/contributing/hacking.md b/doc/manual/src/contributing/hacking.md index 9e4e679e6..7f3905d38 100644 --- a/doc/manual/src/contributing/hacking.md +++ b/doc/manual/src/contributing/hacking.md @@ -121,7 +121,7 @@ If you've already pushed to a fork of Nix on GitHub before, you may have noticed - `x86_64-darwin`. While this installer is in your Cachix cache, you can use it for manual testing on any of these platforms. -- the `installer_test` job will try to use this installer and run a trivial Nix command on `ubuntu-latest` and `macos-latest`. +- The `installer_test` job will try to use this installer and run a trivial Nix command on `ubuntu-latest` and `macos-latest`. ### One-time setup From dc8c0b173c2cc5bd0fe4273f741fda5591ba4133 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 16 Sep 2022 09:36:55 +0200 Subject: [PATCH 52/93] Update doc/manual/src/contributing/hacking.md --- doc/manual/src/contributing/hacking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/contributing/hacking.md b/doc/manual/src/contributing/hacking.md index 7f3905d38..f4aeda871 100644 --- a/doc/manual/src/contributing/hacking.md +++ b/doc/manual/src/contributing/hacking.md @@ -130,7 +130,7 @@ If you've already pushed to a fork of Nix on GitHub before, you may have noticed - Create or log in to an account. - Create a Cachix cache using the format `-nix-install-tests`. - Navigate to the new cache > Settings > Auth Tokens. - - Generate a new cachix auth token and copy the generated value. + - Generate a new Cachix auth token and copy the generated value. 4. At github.com: - Navigate to your Nix fork > Settings > Secrets > Actions > New repository secret. - Name the secret `CACHIX_AUTH_TOKEN` From 4bd52bf6c4c88e2f8a9b703d75c3db5ad062353c Mon Sep 17 00:00:00 2001 From: "Travis A. Everett" Date: Sat, 17 Sep 2022 13:20:11 -0500 Subject: [PATCH 53/93] Apply suggestions from code review Co-authored-by: Valentin Gagarin --- doc/manual/src/contributing/hacking.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/doc/manual/src/contributing/hacking.md b/doc/manual/src/contributing/hacking.md index f4aeda871..5fad34763 100644 --- a/doc/manual/src/contributing/hacking.md +++ b/doc/manual/src/contributing/hacking.md @@ -85,8 +85,6 @@ $ nix develop ## Testing Nix -Nix comes with three different flavors of tests: unit, functional and integration. - ### Unit-tests The unit-tests for each Nix library (`libexpr`, `libstore`, etc..) are defined From 84bdb0e3ade70be722087b95beb7f460e0d3da8d Mon Sep 17 00:00:00 2001 From: "Travis A. Everett" Date: Sun, 18 Sep 2022 12:58:28 -0500 Subject: [PATCH 54/93] address review feedback Mainly: - Try to triangulate between narrative that framed this as a new/easy process and the need for a reference that will not quickly grow stale. - Fix a ~continuity issue where the text was talking about "your Cachix cache" before saying that you'd need to make a Cachix cache to enable the installer tests. - Adopt suggestion on titling, and nest subtitles in the installer test section. --- doc/manual/src/contributing/hacking.md | 28 +++++++++++++------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/doc/manual/src/contributing/hacking.md b/doc/manual/src/contributing/hacking.md index 5fad34763..f67660ab2 100644 --- a/doc/manual/src/contributing/hacking.md +++ b/doc/manual/src/contributing/hacking.md @@ -83,7 +83,7 @@ by: $ nix develop ``` -## Testing Nix +## Running tests ### Unit-tests @@ -107,21 +107,21 @@ Because these tests are expensive and require more than what the standard github You can run them manually with `nix build .#hydraJobs.tests.{testName}` or `nix-build -A hydraJobs.tests.{testName}` -## Testing the install scripts +### Installer tests -Testing the install scripts has traditionally been tedious, but you can now do this much more easily via the GitHub Actions CI runs (at least for platforms that Github Actions supports). +With just a little one-time setup, the Nix repository's GitHub Actions continuous integration (CI) workflow can easily test the installer each time you push to a branch. -If you've already pushed to a fork of Nix on GitHub before, you may have noticed that the CI workflows in your fork list skipped `installer` and `installer_test` jobs. Once your Nix fork is set up correctly, pushing to it will also run these jobs. -- The `installer` job will generate installers for these platforms: +Creating a Cachix cache for your installer tests and adding its authorization token to GitHub enables [two installer-specific jobs in the CI workflow](https://github.com/NixOS/nix/blob/88a45d6149c0e304f6eb2efcc2d7a4d0d569f8af/.github/workflows/ci.yml#L50-L91): + +- The `installer` job generates installers for the platforms below and uploads them to your Cachix cache: - `x86_64-linux` - `armv6l-linux` - `armv7l-linux` - - `x86_64-darwin`. - - While this installer is in your Cachix cache, you can use it for manual testing on any of these platforms. -- The `installer_test` job will try to use this installer and run a trivial Nix command on `ubuntu-latest` and `macos-latest`. + - `x86_64-darwin` -### One-time setup +- The `installer_test` job (which runs on `ubuntu-latest` and `macos-latest`) will try to install Nix with the cached installer and run a trivial Nix command. + +#### One-time setup 1. Have a GitHub account with a fork of the Nix repo. 2. At cachix.org: @@ -129,12 +129,12 @@ If you've already pushed to a fork of Nix on GitHub before, you may have noticed - Create a Cachix cache using the format `-nix-install-tests`. - Navigate to the new cache > Settings > Auth Tokens. - Generate a new Cachix auth token and copy the generated value. -4. At github.com: +3. At github.com: - Navigate to your Nix fork > Settings > Secrets > Actions > New repository secret. - - Name the secret `CACHIX_AUTH_TOKEN` + - Name the secret `CACHIX_AUTH_TOKEN`. - Paste the copied value of the Cachix cache auth token. -### Using the CI-generated installer for manual testing +#### Using the CI-generated installer for manual testing After the CI run completes, you can check the output to extract the installer URL: 1. Click into the detailed view of the CI run. @@ -147,7 +147,7 @@ After the CI run completes, you can check the output to extract the installer UR sh <(curl -L ) --tarball-url-prefix https://-nix-install-tests.cachix.org/serve ``` -