Commit graph

1820 commits

Author SHA1 Message Date
Ben Radford
6a8de4c9dc Avoid enumerating entire overlay store dir upfront.
As an optimisation for LocalStore, we read all the store directory entries into
a set. Checking for membership of this set is much faster than a stat syscall.
However for LocalOverlayStore, the lower store directory is expected to contain
a vast number of entries and reading them all can take a very long time.

So instead of enumerating them all upfront, we call pathExists as needed. This
means making stat syscalls for each store path, but the upper layer is expected
to be relatively small compared to the lower store so that should be okay.
2023-08-01 12:48:02 +01:00
John Ericson
621bdbdb31 Merge branch 'overlayfs-store' into delete 2023-07-26 12:01:08 -04:00
Ben Radford
ca1a108dad Update tests/overlay-local-store/remount.sh
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-07-26 16:01:20 +01:00
Ben Radford
6da05c0a11 Rename test to delete-duplicate. 2023-07-26 16:01:20 +01:00
Ben Radford
ed14286924 Invoke remount-hook program when necessary. 2023-07-26 16:01:20 +01:00
Ben Radford
11c493f8fa Avoid creating whiteout for duplicate store paths. 2023-07-26 16:01:20 +01:00
Ben Radford
cc6f8aa91a Test that delete works for duplicate file edge case. 2023-07-26 16:01:20 +01:00
John Ericson
b0877ad3c9 Give test a more specific name 2023-07-26 09:50:31 -04:00
John Ericson
19c43c5d78 Write test for deleting objects referenced from below
Currently fails, as expected.
2023-07-25 18:09:02 -04:00
John Ericson
73a55e4005 Merge remote-tracking branch 'layered-store/overlayfs-store' into gc 2023-07-25 09:48:06 -04:00
Ben Radford
497464f494
Extend verify test to check that repair is supported. 2023-07-25 13:33:37 +01:00
John Ericson
3731208dc1 Adopt GC test for local-overlay store
Doesn't yet pass. Fixes are needed.
2023-07-24 18:14:12 -04:00
John Ericson
2cabf85b53 Merge remote-tracking branch 'upstream/master' into overlayfs-store 2023-07-24 15:39:36 -04:00
Naïm Favier
570a1a3ad7
parser: merge nested dynamic attributes
Fixes https://github.com/NixOS/nix/issues/7115
2023-07-21 17:14:03 +02:00
Ben Radford
2fc00ec19f
Fix unbound variable error in optimise test. 2023-07-20 11:27:41 +01:00
Ben Radford
2c66a093e0
Define storeBRoot variable distinct from storeB URI. 2023-07-20 11:03:41 +01:00
Ben Radford
878c84d5ee
Fix errors about NIX_STORE_DIR being unset. 2023-07-20 10:42:40 +01:00
Ben Radford
9769a0ae7d
Ensure all overlay tests use new tmpfs store paths. 2023-07-20 10:42:40 +01:00
Ben Radford
7fda19e2f1
Mount tmpfs first to ensure overlayfs works consistently. 2023-07-20 10:42:40 +01:00
Ben Radford
44f855d14e
Missing addTextToStore function. 2023-07-20 10:42:39 +01:00
Ben Radford
d1c77b201a
Explicitly exec shell to fix ENOENT errors. 2023-07-20 10:42:39 +01:00
Ben Radford
8ddbcb736a
Implement overlay store deduplication. 2023-07-20 10:42:39 +01:00
Ben Radford
a9510f9502
Implement test for store path deduplication. 2023-07-20 10:42:39 +01:00
Ben Radford
614efc1240
Add test for store optimise path deduplication. 2023-07-20 10:42:39 +01:00
Ben Radford
d5cd74a401
Override verifyStore to always pass NoRepair for LocalOverlayStore. 2023-07-20 10:42:39 +01:00
Ben Radford
58085e4eff
Have verify test exercise check-contents too. 2023-07-20 10:42:38 +01:00
Ben Radford
0ccf6382af
Add test for verifying overlay store. 2023-07-20 10:42:38 +01:00
Ben Radford
a33ee5c843
Paths added to lower store are accessible via overlay. 2023-07-20 10:42:38 +01:00
Ben Radford
f66b65a30a
Revert "Skip build-remote-trustless unless sandbox is supported."
This reverts commit fad0dd4afb.
2023-07-20 10:42:38 +01:00
Ben Radford
37598a13e8
Revert "Check _NIX_TEST_NO_SANDBOX when setting _canUseSandbox."
This reverts commit 7ed0ab2dab.
2023-07-20 10:42:38 +01:00
John Ericson
453c4be93c
Merge pull request #8680 from NixLayeredStore/test-groups
Introduce notion of a test group, use for CA tests
2023-07-19 11:17:57 -04:00
Ben Radford
21b9e15d25
Merge branch 'master' into overlayfs-store 2023-07-19 15:00:47 +01:00
Robert Hensing
0e3a7e34a0
Merge pull request #8506 from corngood/ssh-master
Pass NIX_SSHOPTS when checking for an ssh master connection.
2023-07-18 15:47:57 +02:00
John Ericson
259e328de8 Introduce notion of a test group, use for CA tests
Grouping our tests should make it easier to understand the intent than
one long poorly-arranged list. It also is convenient for running just
the tests for a specific component when working on that component.

We need at least one test group so this isn't dead code; I decided to
collect the tests for the `ca-derivations` and `dynamic-derivations`
experimental features in groups. Do
```bash
make ca.test-group -jN
```
and
```bash
make dyn-drv.test-group -jN
```
to try running just them.

I originally did this as part of #8397 for being able to just the local
overlay store alone. I am PRing it separately now so we can separate
general infra from new features.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-18 09:31:13 -04:00
Théophane Hufschmitt
a8d5bb5e7e
Merge pull request #8342 from NixLayeredStore/best-effort-supplementary-groups
Best effort supplementary groups
2023-07-17 20:58:17 +02:00
John Ericson
0f7242ff87 Test nested sandboxing, and make nicer error
We were bedeviled by sandboxing issues when working on the layered
store. The problem ended up being that when we have nested nix builds,
and the inner store is inside the build dir (e.g. store is
`/build/nix-test/$name/store`, build dir is `/build`) bind mounts
clobber each other and store paths cannot be found.

After thoroughly cleaning up `local-derivation-goal.cc`, we might be
able to make that work. But that is a lot of work. For now, we just fail
earlier with a proper error message.

Finally, test this: nested sandboxing without the problematic store dir
should work, and with should fail with the expected error message.

Co-authored-by: Dylan Green <67574902+cidkidnix@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-07-14 09:40:24 -04:00
cidkidnix
83cfa82e52 Add unset to NIX_STORE_DIR for local-overlay tests 2023-07-13 14:39:46 -05:00
cidkidnix
c26fc96fed Merge remote-tracking branch 'layered/test-groups' into overlayfs-store 2023-07-13 14:36:49 -05:00
cidkidnix
04d5aa02e6 Merge branch 'best-effort-supplementary-groups' into overlayfs-store 2023-07-13 14:33:52 -05:00
cidkidnix
adb28d4a26 move unset NIX_STORE_DIR in supplementary-groups.sh
to inside the unshare
2023-07-13 14:23:24 -05:00
cidkidnix
1a13757880 Add comment regarding the unset of NIX_STORE_DIR
in build-remote.sh and supplementary-groups.sh
2023-07-13 14:18:12 -05:00
John Ericson
84c4e6f0ac Revert "Skip build-remote-trustless unless sandbox is supported."
This reverts commit 41412dc4ae.
2023-07-13 15:06:50 -04:00
John Ericson
9e64f24340 Revert "Check _NIX_TEST_NO_SANDBOX when setting _canUseSandbox."
This reverts commit c1d39de1fb.
2023-07-13 15:06:34 -04:00
John Ericson
ddc0a2050b Merge remote-tracking branch 'upstream/master' into best-effort-supplementary-groups 2023-07-13 15:06:08 -04:00
John Ericson
e072e18475 Fix race condition in the language tests
When we pipe to `>(...)` like that, we unfortunately don't wait for the
process to finish. Better to just substitute the file.

Also, use the "unified" diff output that people (including myself) are
more familiar with, thanks to Git.
2023-07-13 08:09:03 -04:00
Mathnerd314
c70484454f Expanded test suite
* Lang now verifies errors and parse output

* Some new miscellaneous tests

* Easy way to update the tests

* Document workflow in manual

* Use `!` not `~` as separater char for sed

  It is confusing to use `~` when we are talking about paths and home
  directories!

* Test test suite itself (`test/lang-test/infra.sh`)

Additionally, run shellcheck on `tests/lang.sh` to help ensure it is
correct, now that is is more complex.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-11 21:43:09 -04:00
Ben Radford
41412dc4ae
Skip build-remote-trustless unless sandbox is supported. 2023-07-11 12:52:59 +01:00
Ben Radford
c1d39de1fb
Check _NIX_TEST_NO_SANDBOX when setting _canUseSandbox. 2023-07-11 12:08:33 +01:00
Ben Radford
07dabcc90e
Always attempt setgroups but allow failure to be ignored. 2023-07-11 10:44:05 +01:00
Ben Radford
25b20b4ad2
Merge remote-tracking branch 'origin/master' into best-effort-supplementary-groups 2023-07-11 09:38:34 +01:00