Commit graph

4552 commits

Author SHA1 Message Date
John Ericson
9113b4252b
Merge pull request #8760 from iFreilicht/fix-json-load-assertion-errors
Fix derivation load assertion errors
2023-08-06 17:07:43 -07:00
Felix Uhl
3fefc2b284 Fix derivation load assertion errors
When loading a derivation from a JSON, malformed input would trigger
cryptic "assertion failed" errors. Simply replacing calls to `operator []`
with calls to `.at()` was not enough, as this would cause json.execptions
to be printed verbatim.

Display nice error messages instead and give some indication where the
error happened.

*Before:*

```
$ echo 4 | nix derivation add
error: [json.exception.type_error.305] cannot use operator[] with a string argument with number

$ nix derivation show nixpkgs#hello | nix derivation add
Assertion failed: (it != m_value.object->end()), function operator[], file /nix/store/8h9pxgq1776ns6qi5arx08ifgnhmgl22-nlohmann_json-3.11.2/include/nlohmann/json.hpp, line 2135.

$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: [json.exception.type_error.302] type must be string, but is object

$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error: [json.exception.type_error.302] type must be object, but is string

```

*After:*

```
$ echo 4 | nix derivation add
error: Expected JSON of derivation to be of type 'object', but it is of type 'number'

$ nix derivation show nixpkgs#hello | nix derivation add
error: Expected JSON object to contain key 'name' but it doesn't

$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: Expected JSON value to be of type 'string' but it is of type 'number'

$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error:
       … while reading key 'outputs'

       error: Expected JSON value to be of type 'object' but it is of type 'string'
```
2023-08-05 01:34:30 +02:00
John Ericson
d137002e94 Add API docs for all overridden local overlay methods
These docs explain the implementation relative to the local store
originals. The original declaration of virtual methods can still be
consulted for proper interface-level documentation.
2023-08-03 17:28:18 -04:00
John Ericson
7ad16c9d12 Add some docs for the local overlay store 2023-08-03 16:10:32 -04:00
John Ericson
4d99e407fd Remove FIXME on why something doesn't work
I now know it is due to
https://github.com/llvm/llvm-project/issues/64108. The workaround is
just fine and already in use in this codebase.
2023-08-03 12:12:54 -04:00
John Ericson
4f5b01f5cd Start to document the local-overlay store 2023-08-03 12:10:13 -04:00
John Ericson
4b9a621812 Guard the local overlay store behind an experimental feature 2023-08-02 20:30:42 -04:00
John Ericson
3fc838c8a8 Merge remote-tracking branch 'layered-store/experimental-stores' into overlayfs-store 2023-08-02 19:03:34 -04:00
John Ericson
3b592c880a Add infra for experimental store implemenations
This is analogous to that for experimental settings and flags that we
have also added as of late.
2023-08-02 15:46:38 -04:00
John Ericson
6b297e5895 Make verifyAllValidPaths more functional
return map rather than mutate one passed in by reference
2023-08-02 14:38:22 -04:00
John Ericson
73c9fc7ab1 Merge remote-tracking branch 'upstream/master' into overlayfs-store 2023-08-02 14:14:11 -04:00
John Ericson
66550878df Add comment explaining the use of readDirectory(realStoreDir) 2023-08-02 12:46:07 -04:00
John Ericson
770d50e49c local-store verifying: Rename store to something more clear
It is not a `Store` but a `StorePathSet`.
2023-08-02 12:40:04 -04:00
Ben Radford
8225b7a011
Merge branch 'master' into overlayfs-store 2023-08-01 12:52:53 +01:00
Ben Radford
c712369ec5 Document remount-hook store parameter. 2023-08-01 12:48:02 +01:00
Ben Radford
c409a753db Fix new lines in comment. 2023-08-01 12:48:02 +01:00
Ben Radford
1255866e16 Update src/libstore/local-overlay-store.hh
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-08-01 12:48:02 +01:00
Ben Radford
6a8de4c9dc Avoid enumerating entire overlay store dir upfront.
As an optimisation for LocalStore, we read all the store directory entries into
a set. Checking for membership of this set is much faster than a stat syscall.
However for LocalOverlayStore, the lower store directory is expected to contain
a vast number of entries and reading them all can take a very long time.

So instead of enumerating them all upfront, we call pathExists as needed. This
means making stat syscalls for each store path, but the upper layer is expected
to be relatively small compared to the lower store so that should be okay.
2023-08-01 12:48:02 +01:00
Ben Radford
50ce8d15eb Preparatory refactor of LocalStore::verifyStore. 2023-08-01 12:48:02 +01:00
John Ericson
6525265f46 LocalStore::verifyPath: Try to clarify data flow with more scopes
It was initially unclear to me which of these are temporary state for
the verify paths computation, and which of these are the results of that
computation to be used in the rest of the function. Now, it is clear,
and enforced.
2023-07-31 12:44:27 -04:00
John Ericson
2a5f5fbb17 LocalStore::verifyPath: Use StorePathSet for store local var
We don't care about non-store-paths in there (things like `.links`, are,
in fact, allowed). So let's just skip them up front and be more strongly
typed.
2023-07-31 12:44:18 -04:00
Ben Radford
c9a87ce7ca Refactor verifyPath to take StorePath instead of Path.
This way we avoid having to convert from Path to StorePath and vice versa in
the body of verifyPath.
2023-07-31 11:17:55 -04:00
John Ericson
621bdbdb31 Merge branch 'overlayfs-store' into delete 2023-07-26 12:01:08 -04:00
Ben Radford
c2d54496a0 Forgot to check flag and early out. 2023-07-26 16:01:20 +01:00
Ben Radford
3a9fe1a085 Made remountRequired atomic to avoid concurrency issues. 2023-07-26 16:01:20 +01:00
Ben Radford
5744a500d6 Use debug instead of writing directly to stderr. 2023-07-26 16:01:20 +01:00
Ben Radford
ed14286924 Invoke remount-hook program when necessary. 2023-07-26 16:01:20 +01:00
Ben Radford
33ebae75ca Reuse deletion logic for optimiseStore and rename method. 2023-07-26 16:01:20 +01:00
Ben Radford
11c493f8fa Avoid creating whiteout for duplicate store paths. 2023-07-26 16:01:20 +01:00
Ben Radford
d9688ba708 Add new remount-hook store parameter. 2023-07-26 16:01:20 +01:00
John Ericson
07b34edc44 Fix deletion test
Lower layer references are ignored for deleting just in the upper layer.
2023-07-25 18:09:02 -04:00
John Ericson
9ef0a9e8aa Fix hard linking issue causing overlay fs copy-ups 2023-07-25 10:28:11 -04:00
John Ericson
73a55e4005 Merge remote-tracking branch 'layered-store/overlayfs-store' into gc 2023-07-25 09:48:06 -04:00
Ben Radford
497464f494
Extend verify test to check that repair is supported. 2023-07-25 13:33:37 +01:00
John Ericson
0e595a52a3 Remove trailing whitespace 2023-07-24 15:39:59 -04:00
John Ericson
2cabf85b53 Merge remote-tracking branch 'upstream/master' into overlayfs-store 2023-07-24 15:39:36 -04:00
John Ericson
60d8dd7aea Clean up store hierarchy with IndirectRootStore
See the API doc comments for details.
2023-07-24 09:19:44 -04:00
John Ericson
13269ba93b Make RemoteStore::ConnectionHandle part of class and expose
Will need to do subclass-specific implementations in the next commit.
This isn't because there will be multiple variations of the daemon
protocol (whew!) but because different clients pick and choose different
parts to use.
2023-07-24 09:17:01 -04:00
John Ericson
0a30b07277 Move Store::Params typedef to StoreConfig::Params
This is because `StoreConfig` also uses it.
2023-07-24 09:03:44 -04:00
John Ericson
fe1fbdb5a1
Merge pull request #8724 from obsidiansystems/queryPartialDerivationOutputMap-evalStore
Give `queryPartialDerivationOutputMap` an `evalStore` parameter
2023-07-21 08:53:18 -04:00
Eelco Dolstra
7ac24d9525
Merge pull request #8650 from obsidiansystems/content-address-simpler
Simplify `ContentAddress`
2023-07-21 13:46:53 +02:00
John Ericson
6bc98c7fba Give queryPartialDerivationOutputMap an evalStore parameter
This makes it more useful. In general, the derivation will be in one
store, and the realisation info is in another.

This also helps us avoid duplication. See how `resolveDerivedPath` is
now simpler because it uses `queryPartialDerivationOutputMap`. In #8369
we get more flavors of derived path, and need more code to resolve them
all, and this problem only gets worse.

The fact that we need a new method to deal with the multiple dispatch is
unfortunate, but this generally relates to the fact that `Store` is a
sub-par interface, too bulky/unwieldy and conflating separate concerns.
Solving that is out of scope of this PR.

This is part of the RFC 92 work. See tracking issue #6316
2023-07-20 15:59:52 -04:00
John Ericson
f62543fe1c Remove unneeded copy
It appeared in 8eb73a8724 (by me!) without
justification.
2023-07-20 15:42:06 -04:00
John Ericson
7b30293d38 Tighten #includes: DerivedPath doesn't care about Realisation 2023-07-20 15:42:06 -04:00
Ben Radford
8ddbcb736a
Implement overlay store deduplication. 2023-07-20 10:42:39 +01:00
Ben Radford
614efc1240
Add test for store optimise path deduplication. 2023-07-20 10:42:39 +01:00
Ben Radford
d5cd74a401
Override verifyStore to always pass NoRepair for LocalOverlayStore. 2023-07-20 10:42:39 +01:00
Ben Radford
21b9e15d25
Merge branch 'master' into overlayfs-store 2023-07-19 15:00:47 +01:00
Robert Hensing
32494cbb29
Merge pull request #7973 from fricklerhandwerk/remove-channels
remove the Channels section
2023-07-19 14:02:26 +02:00
Valentin Gagarin
2fa90e5824 add more details on CA derivations 2023-07-19 13:59:18 +02:00
Valentin Gagarin
5f37ebcf83 document all special system features and their behavior 2023-07-19 13:34:03 +02:00
Robert Hensing
eca07b2a47
Merge pull request #8315 from fricklerhandwerk/doc-system
add information on the system type string
2023-07-19 12:07:23 +02:00
Valentin Gagarin
493ddf617f reformat system-features setting documentation 2023-07-19 11:07:47 +02:00
Valentin Gagarin
feb01b22ed add links to store API documentation 2023-07-19 11:07:47 +02:00
Valentin Gagarin
6c3cd429a6 fix broken links 2023-07-19 11:01:48 +02:00
Valentin Gagarin
0779005f49 expand on the extra-platforms option 2023-07-19 10:57:37 +02:00
Valentin Gagarin
fcadac0a02 mention extra-platforms 2023-07-19 10:38:42 +02:00
Valentin Gagarin
aba32def73 fix wording
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-07-19 10:38:30 +02:00
Valentin Gagarin
32de11923e add cross-links 2023-07-19 10:37:40 +02:00
Valentin Gagarin
3763c7bb5e shorten system setting description 2023-07-19 10:37:40 +02:00
Valentin Gagarin
c8a42039ea move docs of the current system to the system setting
add information what happens when Nix itself is cross-compiled
2023-07-19 10:37:40 +02:00
Valentin Gagarin
0751c1bfc6 one line per sentence for easier review 2023-07-19 10:37:40 +02:00
Robert Hensing
0e3a7e34a0
Merge pull request #8506 from corngood/ssh-master
Pass NIX_SSHOPTS when checking for an ssh master connection.
2023-07-18 15:47:57 +02:00
Théophane Hufschmitt
a8d5bb5e7e
Merge pull request #8342 from NixLayeredStore/best-effort-supplementary-groups
Best effort supplementary groups
2023-07-17 20:58:17 +02:00
John Ericson
0f7242ff87 Test nested sandboxing, and make nicer error
We were bedeviled by sandboxing issues when working on the layered
store. The problem ended up being that when we have nested nix builds,
and the inner store is inside the build dir (e.g. store is
`/build/nix-test/$name/store`, build dir is `/build`) bind mounts
clobber each other and store paths cannot be found.

After thoroughly cleaning up `local-derivation-goal.cc`, we might be
able to make that work. But that is a lot of work. For now, we just fail
earlier with a proper error message.

Finally, test this: nested sandboxing without the problematic store dir
should work, and with should fail with the expected error message.

Co-authored-by: Dylan Green <67574902+cidkidnix@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-07-14 09:40:24 -04:00
cidkidnix
04d5aa02e6 Merge branch 'best-effort-supplementary-groups' into overlayfs-store 2023-07-13 14:33:52 -05:00
John Ericson
caabc4f648 Feature gate DownstreamPlaceholder::unknownCaOutput
This is a part of CA derivations that we forgot to put behind the
experimental feature.

This was caught by @fricklerhandwerk in
https://github.com/NixOS/nix/pull/8369#discussion_r1258133719
2023-07-13 07:56:33 -04:00
John Ericson
2c3fb0eb33 Move BuiltPath to its own header/C++ file in libcmd
It is less important, and used less widely, than `DerivedPath`.
2023-07-12 23:01:10 -04:00
Ben Radford
a2acd23466
Update src/libstore/globals.hh
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-12 12:33:05 +01:00
Ben Radford
0309f6b5b8
Update src/libstore/globals.hh
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-12 12:32:57 +01:00
Ben Radford
b8e8dfc3e8
Say a bit about default value in setting description. 2023-07-11 11:25:49 +01:00
Ben Radford
a193ec4052
Default should depend on whether we are root. 2023-07-11 11:14:13 +01:00
Ben Radford
2b4c59dd99
Be clearer about the security implications. 2023-07-11 11:09:25 +01:00
Ben Radford
0caf28f238
Update description for require-drop-supplementary-groups. 2023-07-11 10:57:14 +01:00
Ben Radford
07dabcc90e
Always attempt setgroups but allow failure to be ignored. 2023-07-11 10:44:05 +01:00
Ben Radford
25b20b4ad2
Merge remote-tracking branch 'origin/master' into best-effort-supplementary-groups 2023-07-11 09:38:34 +01:00
John Ericson
2add230939 Fix build 2023-07-09 21:53:06 -04:00
John Ericson
28398e6d02 Merge remote-tracking branch 'upstream/master' into overlayfs-store 2023-07-09 20:30:23 -04:00
John Ericson
903700c5e1 Simplify ContentAddress
Whereas `ContentAddressWithReferences` is a sum type complex because different
varieties support different notions of reference, and
`ContentAddressMethod` is a nested enum to support that,
`ContentAddress` can be a simple pair of a method and hash.

`ContentAddress` does not need to be a sum type on the outside because
the choice of method doesn't effect what type of hashes we can use.

Co-Authored-By: Cale Gibbard <cgibbard@gmail.com>
2023-07-07 07:30:01 -04:00
Robert Hensing
55888633dd makeContentAddressed: Add single path helper 2023-06-30 18:22:47 +02:00
Yueh-Shun Li
eebfe989a5 linkOrCopy: Fallback upon cross-device link error (EXDEV)
Fix building derivations in local chroot store on OpenAFS,
where hard linking accross directories causes cross-device link error
(EXDEV).
2023-06-30 21:12:26 +08:00
Jean-François Roche
80c9259756 Allow to sign path as unprivileged user
User can now sign path as unprivileged/allowed user

refs #1708
2023-06-27 18:31:31 +02:00
Robert Hensing
fd4f03b8fd
Merge pull request #8519 from fricklerhandwerk/reword-trusted-users
reword documentation on trusted users and substituters
2023-06-23 13:08:46 +02:00
John Ericson
97df060588 Better document build failure exit codes
- Improved API docs from comment

- Exit codes are for `nix-build`, not just `nix-store --release`

- Make note in tests so the magic numbers are not surprising

Picking up where #8387 left off.
2023-06-22 14:29:45 -04:00
John Ericson
48fe0ed554
Merge pull request #8374 from obsidiansystems/improve-path-setting
Split `OptionalPathSetting` from `PathSetting`
2023-06-21 15:40:43 -04:00
John Ericson
9c0473120f Merge remote-tracking branch 'upstream/master' into overlayfs-store 2023-06-21 10:15:13 -04:00
Eelco Dolstra
b1352700c4
Merge pull request #8552 from edolstra/fix-eagain
GC server: Clear O_NONBLOCK on the right file descriptor
2023-06-20 12:12:53 +02:00
Ben Radford
6ae35534b7
Support opening local store with database on read-only filesystem (#8356)
Previously it was not possible to open a local store when its database is on a read-only filesystem. Obviously a store on a read-only filesystem cannot be modified, but it would still be useful to be able to query it.

This change adds a new read-only setting to LocalStore. When set to true, Nix will skip operations that fail when the database is on a read-only filesystem (acquiring big-lock, schema migration, etc), and the store database will be opened in immutable mode.

Co-authored-by: Ben Radford <benradf@users.noreply.github.com>
Co-authored-by: cidkidnix <cidkidnix@protonmail.com>
Co-authored-by: Dylan Green <67574902+cidkidnix@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-20 11:34:09 +02:00
Eelco Dolstra
a6a75ecad8 GC server: Clear O_NONBLOCK on the right file descriptor
The bug fix in 6d30f9e6fe erroneously
cleared O_NONBLOCK on the server rather than client FD (leaving both
in an incorrect state).

Fixes #8551.
2023-06-20 11:19:14 +02:00
Eelco Dolstra
c5fdbdae32 LocalStore::addTempRoot(): Handle ENOENT
If the garbage collector has acquired the global GC lock, but hasn't
created the GC socket yet, then a client attempting to connect would
get ENOENT. Note that this only happens when the GC runs for the first
time on a machine. Subsequently clients will get ECONNREFUSED which
was already handled.

Fixes #7370.
2023-06-20 11:14:55 +02:00
John Ericson
3859cf6b21 Remove unused #include from local-derivation-goal.cc
These were never needed for this file, and date back to before this was
split from `derivation-goal.cc`.
2023-06-19 12:18:04 -04:00
John Ericson
9f69b7dee9 Create worker_proto::{Read,Write}Conn
Pass this around instead of `Source &` and `Sink &` directly. This will
give us something to put the protocol version on once the time comes.

To do this ergonomically, we need to expose `RemoteStore::Connection`,
so do that too. Give it some more API docs while we are at it.
2023-06-19 12:08:23 -04:00
John Ericson
4e8b495ad7 Likewise namespace and enum struct-ify ServeCommand
The motivation is exactly the same as for the last commit. In addition,
this anticipates us formally defining separate serialisers for the serve
protocol.
2023-06-19 12:08:23 -04:00
John Ericson
95eae0c002 Put worker protocol items inside a WorkerProto struct
See API docs on that struct for why. The pasing as as template argument
doesn't yet happen in that commit, but will instead happen in later
commit.

Also make `WorkerOp` (now `Op`) and enum struct. This led us to catch
that two operations were not handled!

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-06-19 12:08:23 -04:00
John Ericson
469d06f9bc Split out worker protocol template definitions from declarations
This is generally a fine practice: Putting implementations in headers
makes them harder to read and slows compilation. Unfortunately it is
necessary for templates, but we can ameliorate that by putting them in a
separate header. Only files which need to instantiate those templates
will need to include the header with the implementation; the rest can
just include the declaration.

This is now documenting in the contributing guide.

Also, it just happens that these polymorphic serializers are the
protocol agnostic ones. (Worker and serve protocol have the same logic
for these container types.) This means by doing this general template
cleanup, we are also getting a head start on better indicating which
code is protocol-specific and which code is shared between protocols.
2023-06-19 11:45:59 -04:00
Ben Radford
71f3bad749
Merge branch 'read-only-local-store' into overlayfs-store 2023-06-19 16:12:04 +01:00
Ben Radford
4e72b8483e
Update src/libstore/sqlite.hh
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-06-19 16:01:43 +01:00
Ben Radford
0b3fa995e5
Merge branch 'master' into read-only-local-store 2023-06-19 14:09:20 +01:00
Ben Radford
ba492a98ba
Update src/libstore/local-store.hh
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-19 14:07:31 +01:00
Ben Radford
ef40448b1c
Remove redundant description on experimental flag. 2023-06-19 13:52:32 +01:00
Ben Radford
feb8d552ae
Update src/libstore/local-store.hh
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-19 13:22:41 +01:00
John Ericson
c404623a1d
Clean up a few things related to profiles (#8526)
- Greatly expand API docs

- Clean up code in misc ways

  - Instead of a complicated single loop on generations, do different
    operations in successive subsequent steps.

  - Avoid `ref` in one place where `&` is fine

  - Just return path instead of mutating an argument in `makeName`

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-19 04:04:59 +00:00
John Ericson
d2ce2e89b1 Split OptionalPathSetting from PathSetting
Rather than doing `allowEmpty` as boolean, have separate types and use
`std::optional`. This makes it harder to forget the possibility of an
empty path.

The `build-hook` setting was categorized as a `PathSetting`, but
actually it was split into arguments. No good! Now, it is
`Setting<Strings>` which actually reflects what it means and how it is
used.

Because of the subtyping, we now also have support for
`Setting<std::optional<String>>` in general. I imagine this can be used
to clean up many more settings also.
2023-06-18 23:31:18 -04:00
Eelco Dolstra
e503eadafc
Merge pull request #8477 from edolstra/tarball-flake-redirects
Tarball flake improvements
2023-06-16 18:03:50 +02:00
Valentin Gagarin
f695a74751
Update src/libstore/globals.hh 2023-06-16 17:58:01 +02:00
Valentin Gagarin
1a8ca85d48
use "store URLs" consistently 2023-06-16 14:34:11 +02:00
Eelco Dolstra
713836112c
Merge pull request #8517 from hercules-ci/fix-build-hook-error-for-lib-users
Fix build hook error for libstore library users
2023-06-16 13:20:50 +02:00
Valentin Gagarin
2ceacce484
Update src/libstore/globals.hh 2023-06-15 15:57:54 +02:00
Ben Radford
abb3bb7133
Merge branch 'master' into read-only-local-store 2023-06-15 13:42:14 +01:00
Ben Radford
04d8f202a7
Merge branch 'read-only-local-store' into overlayfs-store 2023-06-15 13:37:57 +01:00
Ben Radford
f5d83a8029
One line per sentence in markdown docs. 2023-06-15 13:36:28 +01:00
Ben Radford
f2fe9822c1
Comment explaining what schema version 0 means. 2023-06-15 13:34:42 +01:00
Ben Radford
4642b60afe
Update src/libstore/local-store.hh
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-15 13:33:26 +01:00
Ben Radford
a7b1b92d81
Update src/libstore/local-store.hh
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-15 13:32:56 +01:00
Ben Radford
984b01924a
Update src/libstore/local-store.cc
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-15 13:32:35 +01:00
Ben Radford
78e2f931d0
Update src/libstore/local-store.cc
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-15 13:32:16 +01:00
Robert Hensing
d2696cdd1e Fix build hook error for libstore library users
A library shouldn't require changes to the caller's argument handling,
especially if it doesn't have to, and indeed we don't have to.

This changes the lookup order to prioritize the hardcoded path to nix
if it exists. The static executable still finds itself through /proc
and the like.
2023-06-15 14:32:00 +02:00
Valentin Gagarin
b7d47e1d22
fix wording 2023-06-15 04:58:07 +02:00
Valentin Gagarin
e09b40e0d0 reword documentation on trusted users and substituters
this is to make it slightly easier to scan over
2023-06-15 02:19:13 +02:00
Valentin Gagarin
a1cf16563f
Fixup description of substituters (#8291)
Introduce what substituters "are" in the configuration option entry.
Remove arbitrary line breaks for easier editing in the future.
Link glossary some more.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-06-14 20:49:58 +00:00
John Ericson
61a3e1f2e2
Merge pull request #4282 from tweag/fix-ca-hash-rewriting
fix the hash rewriting for ca-derivations
2023-06-14 18:25:00 +02:00
John Ericson
32147c8c7b Merge remote-tracking branch 'upstream/master' into overlayfs-store 2023-06-14 10:29:55 -04:00
Eelco Dolstra
1ad3328c5e Allow tarball URLs to redirect to a lockable immutable URL
Previously, for tarball flakes, we recorded the original URL of the
tarball flake, rather than the URL to which it ultimately
redirects. Thus, a flake URL like
http://example.org/patchelf-latest.tar that redirects to
http://example.org/patchelf-<revision>.tar was not really usable. We
couldn't record the redirected URL, because sites like GitHub redirect
to CDN URLs that we can't rely on to be stable.

So now we use the redirected URL only if the server returns the
`x-nix-is-immutable` or `x-amz-meta-nix-is-immutable` headers in its
response.
2023-06-13 14:17:45 +02:00
David McFarland
d5e1eb20a2 Pass common ssh options in isMasterRunning 2023-06-13 00:55:34 -03:00
Eelco Dolstra
87c66f6b0f Don't include uds-remote-store.md from a header file
Closes #8484.
2023-06-12 11:10:55 +02:00
Eelco Dolstra
381a32981b
Merge branch 'master' into angerman/mac-fix-recursive-nix 2023-06-09 13:06:47 +02:00
Ben Radford
ee1241da86 Remove unnecessary overrides of add methods. 2023-06-06 12:21:17 +01:00
Ben Radford
8a9baa0a30 More sensible to have deleteGCPath in LocalStore. 2023-06-06 12:21:17 +01:00
Ben Radford
a48acfd684 Skip deletion of lower paths for overlay store GC. 2023-06-06 12:21:17 +01:00
Ben Radford
98edbb9686 Factor out GC path deletion so it can be overridden. 2023-06-06 12:21:17 +01:00
Eelco Dolstra
b56319a139
Merge pull request #8391 from aneeshusa/remove-wrong-default-value-in-docs-for-hashed-mirrors
Remove old default from docs for `hashed-mirrors`
2023-06-06 12:54:28 +02:00
Eelco Dolstra
864ab1a7ac
Merge pull request #8438 from zhaofengli/armv5tel-linux32
Also set the PER_LINUX32 personality flag on armv5tel-linux
2023-06-06 12:47:16 +02:00
Ben Radford
b852bdb3f8
Merge branch 'read-only-local-store' into overlayfs-store 2023-06-06 11:09:26 +01:00
Ben Radford
c47f744e05
Also skip makeStoreWritable when read-only=true. 2023-06-06 11:07:55 +01:00
Ben Radford
e484278e9b
Merge branch 'master' into read-only-local-store 2023-06-06 11:07:36 +01:00
Alexander Bantyev
a15b2c01c0
Document manual migration for use-xdg-base-directories (#8044)
* Document manual migration for use-xdg-base-directories

As there's currently no automatic migration for use-xdg-base-directories
option, add instructions for manual migration to the option's
description.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-02 15:38:17 +00:00
Zhaofeng Li
48632e6139 Also set the PER_LINUX32 personality flag on armv5tel-linux 2023-06-02 03:04:13 -06:00
Eelco Dolstra
61ddfa154b
Merge pull request #8399 from tweag/fix-chrooted-stores-error-path
Properly report build errors on chrooted stores
2023-05-27 17:55:57 +02:00
Yorick van Pelt
2c462486fe
create pathAccessible, use it to infer default dirs 2023-05-26 15:36:47 +02:00
Yorick van Pelt
b7cde90c6b
Call getDefaultSSLCertFile() only when none is specified
This does pathExists on various paths, which crashes on EPERM in the
macOS sandbox.
2023-05-26 15:36:45 +02:00
Théophane Hufschmitt
d16a1994fb Properly report build errors on chrooted stores
When encountering a build error, Nix moves the output paths out of the
chroot into their final location (for “easier debugging of build
failures”). However this was broken for chroot stores as it was moving
it to the _logical_ location, not the _physical_ one.

Fix it by moving to the physical (_real_) location.

Fix https://github.com/NixOS/nix/issues/8395
2023-05-25 16:38:29 +02:00
Ben Radford
8e4b15e6a0
Merge branch 'master' into read-only-local-store 2023-05-25 10:06:24 +01:00
Aneesh Agrawal
c694f1a2f3 Remove old default from docs for hashed-mirrors
The `hashed-mirrors` option did use to have this default value,
but it was removed and re-added with an empty default value.
As the autogenerated docs show the (actual) default values from code,
remove this incorrect reference from the docs.

I was updating my nix.conf settings after a few years and noticed this.
2023-05-24 11:05:40 -04:00
Théophane Hufschmitt
34e1b464f0 Normalize the hash-rewriting process when building derivations 2023-05-24 14:11:50 +02:00
Théophane Hufschmitt
a917fb0d53 Use a RewritingSink in derivation goal
Possibly this will make it stream
2023-05-24 14:11:50 +02:00
Théophane Hufschmitt
3ebe1341ab Make RewritingSink accept a map of rewrites
Giving it the same semantics as `rewriteStrings`.
Also add some tests for it
2023-05-24 14:11:50 +02:00
oxalica
303858afad Fix typo in error message of too long store path 2023-05-24 01:24:28 +08:00
Ben Radford
ff12cf3b94
Merge branch 'read-only-local-store' into overlayfs-store 2023-05-23 09:52:41 +01:00
Bernardo Meurer
bf693319f6
feat: add always-allow-substitutes
This adds a new configuration option to Nix, `always-allow-substitutes`,
whose effect is simple: it causes the `allowSubstitutes` attribute in
derivations to be ignored, and for substituters to always be used.

This is extremely valuable for users of Nix in CI, where usually
`nix-build-uncached` is used. There, derivations which disallow
substitutes cause headaches as the inputs for building already-cached
derivations need to be fetched to spuriously rebuild some simple text
file.

This option should be a good middle-ground, since it doesn't imply
rebuilding the world, such as the approach I took in
https://github.com/NixOS/nixpkgs/pull/221048
2023-05-22 19:42:29 +01:00
Théophane Hufschmitt
494a09c6df
Merge pull request #8377 from layus/fix-ssl-cert-mount
Make mounting ssl cert file optional
2023-05-22 14:06:57 +02:00
Ben Radford
32404292a3
Merge branch 'master' into read-only-local-store 2023-05-22 12:08:02 +01:00
Ben Radford
7251800086
Put read-only setting behind an experimental flag. 2023-05-22 11:38:37 +01:00
Théophane Hufschmitt
673fe85976
Merge pull request #8365 from obsidiansystems/proto-structs
Revert "Revert "Use template structs instead of phantoms""
2023-05-22 09:34:34 +02:00
Guillaume Maudoux
b14fea6fff Shortcircuit for empty caFile 2023-05-19 23:30:35 +02:00
Guillaume Maudoux
36b7e30c11 Make mounting ssl cert file optional 2023-05-19 22:47:40 +02:00
Eelco Dolstra
34381d5747
Merge pull request #8215 from obsidiansystems/general-repair-path
Support `repairPath` on most stores.
2023-05-19 13:39:48 +02:00
Eelco Dolstra
e31d9b910d
Merge pull request #7312 from layus/fixed-output-system-cert
Allow system certs access to fixed-output derivations
2023-05-19 13:05:16 +02:00
Ben Radford
8ffeb1c4e5
Throw error instead of silently skipping CA migration. 2023-05-18 13:51:21 +01:00
Ben Radford
d55e38b98a
Check earlier whether schema migration is required. 2023-05-18 13:51:21 +01:00
Ben Radford
fe174d72a2
Fix spelling of 'immutable' in documentation. 2023-05-18 13:51:21 +01:00
Robert Hensing
b1c34c0ee8
Merge pull request #8366 from obsidiansystems/worker-proto-forward-decl-types
Don't use `store-api.hh` in `worker-protocol.hh`
2023-05-18 14:33:52 +02:00
Dylan Green
60c0149721 Apply suggestions from code review
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-05-18 10:31:49 +01:00
cidkidnix
85a2453052 Add tests for read-only local store
Make sure we don't go down the path of making temproots when doing
operations on a read-only store
2023-05-18 10:31:49 +01:00
John Ericson
9923403d90 Don't use store-api.hh in worker-protocol.hh
Using abstract types like can help cut down on compilation time, both
from scratch, and especially incremental builds during development. The
idea is that `worker-protocol.hh` can declare all the (de)serializers, but
only again abstract types; when code needs to use some (de)serializers, it can
include headers just for the data types it needs to (de)serialize.

`store-api.hh` in particular is a bit of a sledgehammer, and the data
types we want to serialize have their own headers.
2023-05-18 00:20:24 -04:00
John Ericson
cb5052d98f Revert "Revert "Use template structs instead of phantoms""
This is the more typically way to do [Argument-dependent
lookup](https://en.cppreference.com/w/cpp/language/adl)-leveraging
generic serializers in C++. It makes the relationship between the `read`
and `write` methods more clear and rigorous, and also looks more
familiar to users coming from other languages that do not have C++'s
libertine ad-hoc overloading.

I am returning to this because during the review in
https://github.com/NixOS/nix/pull/6223, it came up as something that
would make the code easier to read --- easier today hopefully already,
but definitely easier if we were have multiple codified protocols with
code sharing between them as that PR seeks to accomplish.

If I recall correctly, the main criticism of this the first time around
(in 2020) was that having to specify the type when writing, e.g.
`WorkerProto<MyType>::write`, was too verbose and cumbersome. This is
now addressed with the `workerProtoWrite` wrapper function.

This method is also the way `nlohmann::json`, which we have used for a
number of years now, does its serializers, for what its worth.

This reverts commit 45a0ed82f0. That
commit in turn reverted 9ab07e99f5.
2023-05-17 22:44:47 -04:00
John Ericson
b9e5ce4a27 Upgrade downstreamPlaceholder to a type with methods
This gets us ready for dynamic derivation dependencies (part of RFC 92).
2023-05-17 17:41:16 -04:00
John Ericson
e7c1113a37 Add test for downstreamPlaceholder
This is good in general, but in particular ensures when we heavily
refactor it in the next commit there is less likelihood for an
unintentional change in behavior to sneak in.
2023-05-17 17:31:33 -04:00
John Ericson
32dc77ba5d
Merge pull request #8349 from tweag/fix-control-master
Fix ControlMaster behaviour
2023-05-17 12:17:09 -04:00
John Ericson
904878d6d2 Move worker_proto defs out of remote-store.cc to own file
These items are not templates, and they declared in
`worker-protocol.hh`; therefore they should live in a
`worker-protocol.cc`.

Anything else needlessly diverges from convention. After all, it is not
like this code is only used in `remote-store.cc`; it is also used in
`daemon.cc`. There is no good reason to place it with the client
implementation or the server implementation when it used equally by
both.
2023-05-17 10:36:03 -04:00
Ben Radford
5966b76c97
Document the new read-only local store setting. 2023-05-17 09:04:52 +01:00
Ben Radford
b1a7b26eef
Rename ReadOnly to Immutable and clarify its purpose. 2023-05-17 09:04:48 +01:00
Ben Radford
aa376f4ab1
Need to open database using immutable parameter.
This requires switching on SQLITE_OPEN_URI because there is no open flag to
make the database immutable. Without immutable, sqlite will still attempt to
create journal and wal files, even when the database is opened read-only.

https://www.sqlite.org/c3ref/open.html

The immutable parameter is a boolean query parameter that indicates that the
database file is stored on read-only media. When immutable is set, SQLite
assumes that the database file cannot be changed, even by a process with higher
privilege, and so the database is opened read-only and all locking and change
detection is disabled.
2023-05-17 08:55:04 +01:00
Ben Radford
78fdd6f24e
Open sqlite database according to new modes. 2023-05-17 08:55:04 +01:00
Ben Radford
afed9ccfad
Add enum for intended sqlite database open modes. 2023-05-17 08:55:03 +01:00
Ben Radford
7f443e0428
Do not check for write access to database when read-only. 2023-05-17 08:55:03 +01:00
Ben Radford
c22936ca6a
Do not attempt to migrate to CA schema when read-only. 2023-05-17 08:55:03 +01:00
Ben Radford
50bbdc65c8
Do not attempt to acquire big-lock when read-only. 2023-05-17 08:55:03 +01:00
Ben Radford
79583c2d38
Do not attempt to chmod per-user dir when read-only. 2023-05-17 08:55:03 +01:00
Ben Radford
de359da09a
Add read-only setting to LocalStoreConfig. 2023-05-17 08:55:03 +01:00
Ben Radford
4d69bd034a
More detailed explanation of check-mount setting. 2023-05-16 17:27:49 +01:00
Alexander Bantyev
61cdb0b057
Fix ControlMaster behaviour 2023-05-16 18:50:09 +04:00
John Ericson
3496a5ee86 Merge branch 'best-effort-supplementary-groups' into overlayfs-store 2023-05-15 17:46:15 -04:00
John Ericson
746c6aae3f Merge remote-tracking branch 'upstream/master' into best-effort-supplementary-groups 2023-05-15 16:50:11 -04:00
John Ericson
72bb9604f3 Merge remote-tracking branch 'upstream/master' into overlayfs-store 2023-05-15 16:34:19 -04:00
John Ericson
2524a21186
Update src/libstore/build/local-derivation-goal.cc
Co-authored-by: Guillaume Girol <symphorien@users.noreply.github.com>
2023-05-15 12:38:39 -04:00
Ben Radford
0df37edb1c Make upper-layer a PathSetting instead of a Setting<string>. 2023-05-15 17:34:31 +01:00
Ben Radford
b0989cb10b Support percent encoded URIs for lower store. 2023-05-15 17:34:31 +01:00
Ben Radford
b5591ece4c Check that overlay store directory is mounted correctly.
Nix does not manage the overlayfs mount point itself, but the correct
functioning of the overlay store does depend on this mount point being set up
correctly. Rather than just assume this is the case, check that the lowerdir
and upperdir options are what we expect them to be. This check is on by
default, but can be disabled if needed.
2023-05-15 17:34:31 +01:00
John Ericson
f8a6a9e473
Merge pull request #8341 from obsidiansystems/dedup-gen-hh
Dedup some markdown -> C++ big literal stuff in build system
2023-05-15 11:44:35 -04:00
Robert Hensing
41591b33a9
Merge pull request #8340 from NixOS/delete-commited-build-artifacts
Delete commited build artifacts
2023-05-15 17:30:20 +02:00
John Ericson
98afd6ff76 Delete commited build artifacts
They were improperly added in 8a93b5a551.

They were not `.gitignore`d because they were stale in that commit --
build artifacts no longer used that name by then and so `.gitignore` was
updated accordingly.
2023-05-15 10:50:33 -04:00
John Ericson
bbd7d5de09 Fix some bounds in rapid check instances
`inRange` is exclusive not inclusive:
https://github.com/emil-e/rapidcheck/blob/master/doc/generators.md#usage

Furthermore, use `std::variant_size_v` so we use the right number
automatically.

Finally, make the `switch` assert the discriminant is in bounds as
expected.
2023-05-15 10:41:44 -04:00
John Ericson
f7c896cfd8 Dedup some markdown -> C++ big literal stuff in build system
This pattern rule was unwisely hidden in `src/libstore/local.mk`. Now it
is properly in `mk/` and we reuse it for the profile docs too.
2023-05-15 10:38:11 -04:00
John Ericson
914672dc4f
Merge pull request #8141 from tweag/user-files-doc
Document user files of nix
2023-05-15 07:11:47 -04:00
Matej Urbas
c66a7af0c6 max-substitution-jobs release note entry 2023-05-14 09:51:11 +01:00
Eelco Dolstra
643b8d2126
Merge pull request #8299 from urbas/max-substitution-jobs
`max-substitution-jobs` setting
2023-05-12 12:24:27 +02:00
John Ericson
53a1354acf
Merge pull request #3959 from obsidiansystems/ca-drv-exotic
Derivations can output "text-hashed" data
2023-05-10 10:41:59 -04:00
John Ericson
4173743a3c Implement more queries 2023-05-09 17:20:58 -04:00
John Ericson
5059be53b1 Fix recursive ingestion from lower store 2023-05-09 16:42:28 -04:00
John Ericson
6a3a87a714 Improve error message for self reference with text hashing
The `ContentAddressWithReferences` method is made total, with error
handling now squarely the caller's job. This is better.
2023-05-09 14:44:08 -04:00
John Ericson
e514b3939a Add name to some error messages 2023-05-09 13:24:53 -04:00
John Ericson
753fc1661d Cleanups to content address types 2023-05-09 13:05:38 -04:00
John Ericson
d3c125e5a8
Apply suggestions from code review
Thanks!

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-05-09 12:45:51 -04:00
John Ericson
6513f4fe92 Fix bug, newInfo -> newInfo0
It appears we were checking a variable in the process of definining it.
2023-05-09 12:31:36 -04:00
John Ericson
35dcbe1c21 Fix spurious change
Didn't mean to use the private name that shouldn't be exposed.
2023-05-09 12:19:03 -04:00
John Ericson
e7c3399ed2 Specialize LocalOverlayStore::queryPathFromHashPart
With test
2023-05-09 10:40:10 -04:00
John Ericson
ddaf2750b5 Specialize more methods, fix tests 2023-05-09 10:22:38 -04:00
John Ericson
b3d320c594 Convert more methods
Fixed one test, broke another
2023-05-08 18:50:16 -04:00
John Ericson
59a8099038 Fix LocalOverlayStore::queryPathInfoUncached, FIXME in test 2023-05-08 17:37:40 -04:00
John Ericson
5406256d78 Specialize LocalOverlayStore::queryPathInfoUncached 2023-05-08 17:30:17 -04:00
John Ericson
31e98ed0a0 Specialize LocalOverlayStore::registerDrvOutput 2023-05-08 16:48:55 -04:00
John Ericson
9c9f5f0d12 Merge branch 'best-effort-supplementary-groups' into overlayfs-store 2023-05-08 14:47:46 -04:00
Matej Urbas
13185133bc introduces Goal::jobCategory 2023-05-08 19:45:46 +01:00
John Ericson
6d1aa523de Create escape hatch for supplementary group sandboxing woes
There is no obvious good solution for this that has occured to anyone.
2023-05-08 14:41:47 -04:00
Matej Urbas
1ea1e378de removes MaxSubstitutionJobsSetting 2023-05-08 19:21:57 +01:00
John Ericson
d5fe828d76 Merge remote-tracking branch 'upstream/master' into overlayfs-store 2023-05-08 10:20:18 -04:00
John Ericson
f0a176e2f1 Init local overlay store 2023-05-08 10:20:06 -04:00
John Ericson
278c94d607 Rename a few things in new tests
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-05-08 08:01:58 -04:00
Yorick van Pelt
12685ef45f
CA: rewrite hashes for all outputs, not just the wanted ones 2023-05-08 12:58:59 +02:00
Yorick van Pelt
2ca2c80c4e
libstore: also pass unwanted outputs to the post-build-hook 2023-05-08 12:58:59 +02:00
Matej Urbas
613bc699bb max-substitution-jobs setting 2023-05-07 20:22:18 +01:00
Alexander Bantyev
8a93b5a551 Document user files of nix 2023-04-26 15:38:19 +02:00
Moritz Angermann
0e18254aa8
Fix shutdown behavior and resource management for recursive-nix on macOS
Previously, we relied on the `shutdown()` function to terminate `accept()`
calls on a listening socket. However, this approach did not work on macOS as
the waiting `accept()` call is not considered a connected socket, resulting in
an `ENOTCONN` error. Instead, we now close the listening socket to terminate
the `accept()` call.

Additionally, we fixed a resource management issue where we set the
`daemonSocket` variable to -1, triggering resource cleanup and causing the
`stopDaemon` function to be called twice. This resulted in errors as the socket
was already closed by the time the second `stopDaemon` call was made. Instead of
setting `daemonSocket` to -1, we now release the socket using the `release()`
method on a unique pointer. This properly transfers ownership and allows for
correct resource cleanup.

These changes ensure proper behavior and resource management for the
recursive-nix feature on macOS.
2023-04-25 09:39:05 +08:00
John Ericson
8eeaf591db
Add more docs to TextIngestionMethod
Thanks so much!

Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com>
2023-04-21 01:30:55 -04:00
John Ericson
7103c6da70 Remove references from fixed output derivation ab syntax
In other words, use a plain `ContentAddress` not
`ContentAddressWithReferences` for `DerivationOutput::CAFixed`.

Supporting fixed output derivations with (fixed) references would be a
cool feature, but it is out of scope at this moment.
2023-04-19 15:00:04 -04:00
John Ericson
aba8a8a83a Add a few more content addressing methods
Good to round out the library interface.
2023-04-19 14:13:30 -04:00
John Ericson
20decfd302 Gate dynamic-derivations with drv fromJSON too
Don't want `nix derivation add` to be a way to sneak by experimental
feature checks!
2023-04-19 12:44:38 -04:00
John Ericson
76baaeb341 Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2023-04-19 11:32:14 -04:00
Théophane Hufschmitt
d3e2394e91
Merge pull request #8233 from wentasah/narinfo-corrupt-reason
Make "NAR info file is corrupt" messages more informative
2023-04-19 06:57:08 +02:00
Michal Sojka
d30d2dc861 Make "NAR info file is corrupt" messages more informative
Recently, I encountered the "NAR info file 'xxxx' is corrupt" error
with my binary cache. The message is not helpful in determining, which
kind of corruption happened. The file, fetched with curl, looked
reasonably.

This commit adds more information to the error message, which should
allow debugging and hopefully fixing the problem.
2023-04-18 14:10:49 +02:00
John Ericson
668377f217 TextHashMethod -> TextIngestionMethod, gate with XP feature
I suppose we can use `dynamic-derivations` for the few things we neeed.
2023-04-17 19:02:45 -04:00
John Ericson
f56c4a5bdf Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2023-04-17 18:10:12 -04:00
John Ericson
aa74c7b0bc Gate experimental features in DerivationOutput::fromJSON
This is an entry point for outside data, so we need to check enabled
experimental features here.
2023-04-17 17:36:12 -04:00
Robert Hensing
64ee02890c
Merge pull request #8230 from obsidiansystems/daemon-trust-override
Experimentally allow forcing `nix-daemon` trust; use this to test
2023-04-17 19:43:41 +02:00
John Ericson
d41e1bed5e Experimentally allow forcing nix-daemon trust; use this to test
We finally test the status quo of remote build trust in a number of
ways. We create a new experimental feature on `nix-daemon` to do so.

PR #3921, which improves the situation with trustless remote building,
will build upon these changes. This code / tests was pull out of there
to make this, so everything is easier to review, and in particular we
test before and after so the new behavior in that PR is readily apparent
from the testsuite diff alone.
2023-04-17 13:06:21 -04:00
John Ericson
2c8475600d Fix some issues with experimental config settings
Issues:

1. Features gated on disabled experimental settings should warn and be
   ignored, not silently succeed.

2. Experimental settings in the same config "batch" (file or env var)
   as the enabling of the experimental feature should work.

3. For (2), the order should not matter.

These are analogous to the issues @roberth caught with my changes for
arg handling, but they are instead for config handling.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-17 12:41:04 -04:00
Robert Hensing
3f9589f17e
Merge pull request #6312 from obsidiansystems/keyed-build-result
Shuffle `BuildResult` data definition, make state machine clearer, introduce `SingleDrvOutputs`
2023-04-17 18:08:05 +02:00
John Ericson
2eb493ca51 Fix DerivationOutput::fromJSON 2023-04-17 10:28:54 -04:00
John Ericson
e12efa3654 Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2023-04-17 10:16:57 -04:00
Robert Hensing
e641de085b
Merge pull request #3746 from obsidiansystems/path-info
Introduce `StoreReferences` and `ContentAddressWithReferences`
2023-04-17 15:49:48 +02:00
John Ericson
537e8719f2
Explain various .self = false,
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-17 09:15:11 -04:00
John Ericson
72ffa7fedb
Merge pull request #7732 from hercules-ci/make-initLibStore-viable-alternative
Make `initLibStore` a viable alternative
2023-04-17 08:04:41 -04:00
Robert Hensing
9af9c260fc
Merge pull request #8193 from hercules-ci/dry-strings
Deduplicate string literal rendering, fix 4909
2023-04-17 11:19:40 +02:00
John Ericson
9800c1e807 Mark experimental configuration settings programmatically
Fix #8162

The test is changed to compare `nlohmann::json` values, not strings of dumped
JSON, which allows us to format things more nicely.
2023-04-16 10:58:04 -04:00
Robert Hensing
9c74df5bb4
Format
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-04-15 20:56:51 +02:00
John Ericson
24866b71c4 Introduce SingleDrvOutputs
In many cases we are dealing with a collection of realisations, they are
all outputs of the same derivation. In that case, we don't need
"derivation hashes modulos" to be part of our map key, because the
output names alone will be unique. Those hashes are still part of the
realisation proper, so we aren't loosing any information, we're just
"normalizing our schema" by narrowing the "primary key".

Besides making our data model a bit "tighter" this allows us to avoid a
double `for` loop in `DerivationGoal::waiteeDone`. The inner `for` loop
was previously just to select the output we cared about without knowing
its hash. Now we can just select the output by name directly.

Note that neither protocol is changed as part of this: we are still
transferring `DrvOutputs` over the wire for `BuildResult`s. I would only
consider revising this once #6223 is merged, and we can mention protocol
versions inside factored-out serialization logic. Until then it is
better not change anything because it would come a the cost of code
reuse.
2023-04-15 12:51:19 -04:00
John Ericson
0f2b5146c7 Make restarting state machines explicit
If my memory is correct, @edolstra objected to modifying `wantedOutputs`
upon falling back to doing a build (as we did before), because we should
only modify it in response to new requests --- *actual* wants --- and
not because we are "incidentally" building all the outptus beyond what
may have been requested.

That's a fair point, and the alternative is to replace the boolean soup
with proper enums: Instead of modifying `wantedOuputs` som more, we'll
modify `needsRestart` to indicate we are passed the need.
2023-04-15 11:01:31 -04:00
John Ericson
37fca662b0 Make KeyedBuildResult, BuildResult like before, and fix bug another way
In https://github.com/NixOS/nix/pull/6311#discussion_r834863823, I
realized since derivation goals' wanted outputs can "grow" due to
overlapping dependencies (See `DerivationGoal::addWantedOutputs`, called
by `Worker::makeDerivationGoalCommon`), the previous bug fix had an
unfortunate side effect of causing more pointless rebuilds.

In paticular, we have this situation:

1. Goal made from `DerivedPath::Built { foo, {a} }`.

2. Goal gives on on substituting, starts building.

3. Goal made from `DerivedPath::Built { foo, {b} }`, in fact is just
   modified original goal.

4. Though the goal had gotten as far as building, so all outputs were
   going to be produced, `addWantedOutputs` no longer knows that and so
   the goal is flagged to be restarted.

This might sound far-fetched with input-addressed drvs, where we usually
basically have all our goals "planned out" before we start doing
anything, but with CA derivation goals and especially RFC 92, where *drv
resolution* means goals are created after some building is completed, it
is more likely to happen.

So the first thing to do was restore the clearing of `wantedOutputs` we
used to do, and then filter the outputs in `buildPathsWithResults` to
only get the ones we care about.

But fix also has its own side effect in that the `DerivedPath` in the
`BuildResult` in `DerivationGoal` cannot be trusted; it is merely the
*first* `DerivedPath` for which this goal was originally created.

To remedy this, I made `BuildResult` be like it was before, and instead
made `KeyedBuildResult` be a subclass wit the path. Only
`buildPathsWithResults` returns `KeyedBuildResult`s, everything else
just becomes like it was before, where the "key" is unambiguous from
context.

I think separating the "primary key" field(s) from the other fields is
good practical in general anyways. (I would like to do the same thing
for `ValidPathInfo`.) Among other things, it allows constructions like
`std::map<Key, ThingWithKey>` where doesn't contain duplicate keys and
just precludes the possibility of those duplicate keys being out of
sync.

We might leverage the above someday to overload `buildPathsWithResults`
to take a *set* of return a *map* per the above.

-----

Unfortunately, we need to avoid C++20 strictness on designated
initializers.

(BTW
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2287r1.html
this offers some new syntax for this use-case. Hopefully this will be
adopted and we can eventually use it.)

No having that yet, maybe it would be better to not make
`KeyedBuildResult` a subclass to just avoid this.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-15 11:01:31 -04:00
John Ericson
9df7f3f537 Introduce Worker::makeGoal
This takes a `DerivedPath` so the caller doesn't need to care about
which sort of goal does what.
2023-04-15 11:01:31 -04:00
Robert Hensing
ab228d73db
Merge pull request #8214 from raphaelr/installable-derived-path-warn
Display valid installable in `InstallableDerivedPath::parse` warning
2023-04-15 14:15:02 +02:00
John Ericson
ee420ac64e Legacy vs non-legacy to_string/parse for DerivedPath
As requested by @roberth, it is good to call out the specific instances
we care about, which is `!` for the RPC protocols, and `^` for humans.

This doesn't take advantage of parametricity as much, but since the
human and computer interfaces are good to decouple anyways (we don't
care if they drift further apart over time in the slightest) some
separation and slight duplication is fine.

Also, unit test both round trips.
2023-04-14 20:45:11 -04:00
John Ericson
a6f85e052c Support repairPath on most stores.
More progress on issue #5729

The method trivially generalizes to be store-implementation-agnostic, in
fact.

However, we force it to continue to be unimplemented with `RemoteStore`
and `LegacySSHStore` because the implementation we'd get via the
generalization is probably not the one users expect. This keeps our
hands untied to do it right going forward.

For more about the tension between the scheduler logic being
store-type-agnostic and remote stores doing their own scheduling, see
issues #5025 and #5056.
2023-04-14 08:29:06 -04:00
Robert Hensing
f5ab38a688
Merge pull request #8217 from obsidiansystems/push-get-fs-accessor-unsupported-down
Push `getFSAccessor` `unsupported(...)` down `Store` class hierarchy
2023-04-14 14:23:23 +02:00
Raphael Robatsch
9e8f209036 Display valid installable in InstallableDerivedPath::parse warning
The warning message should produce an installable name that can be
passed to `nix build`, `nix path-info`, etc. again. Since the CLI
expects that the .drv path and the output names are separated by
a caret, the warning message must also separate the .drv path and output
names with a caret.

However, `DerivedPath::Built.to_string()` uses an exclamation point as
the separator instead. This commit adds a `separator` argument to the
to_string method.

This changes the warning message from:
If this command is now failing try again with '/nix/store/foo.drv!*'
to:
If this command is now failing try again with '/nix/store/foo.drv^*'
2023-04-14 10:32:03 +02:00
John Ericson
ee97f107e8 Push getFSAccessor unsupported(...) down Store class hierarchy
More progress on issue #5729.

Instead of having it by the default method in `Store` itself, have it be
the implementation in `DummyStore` and `LegacySSHStore`. Then just the
implementations which fail to provide the method pay the "penalty" of
dealing with the icky `unimplemented` function for non-compliance.

Combined with my other recent PRs, this finally makes `Store` have no
`unsupported` calls!
2023-04-13 13:39:44 -04:00
Ben Radford
de3df3009b
Move warning timing logic into handleSQLiteBusy. 2023-04-11 16:14:16 +01:00
Ben Radford
7c56e84213
Warn after a second of being busy instead of immediately.
Getting the occasional SQLITE_BUSY is expected when the database is being
accessed concurrently. The retry will likely succeed so it is pointless to warn
immediately. Instead we track how long each retrySQLite block has been running,
and only begin warning after a second has elapsed (and then every 10 seconds
subsequently).
2023-04-11 11:15:34 +01:00
Robert Hensing
4e0804c920 Deduplicate string literal rendering, fix 4909 2023-04-09 22:42:20 +02:00
John Ericson
fd21f9d76e Merge remote-tracking branch 'upstream/master' into path-info 2023-04-07 20:39:04 -04:00
Robert Hensing
ddebeb934a libstore: Remove lockCPU dead code
Left over from 9747ea84b, https://github.com/NixOS/nix/pull/5821
2023-04-07 17:50:40 +02:00
Robert Hensing
1c0b680ef9 libstore: Remove lockCPU dead code
Left over from 9747ea84b, https://github.com/NixOS/nix/pull/5821
2023-04-07 17:50:40 +02:00
Robert Hensing
781d3dceb3 Move initLibUtil() from initNix to initLibStore
libutil is a dependency of libstore, so it should always be
initialized as such.
libutil is also a dependency of libmain. Being explicit about this
dependency might be good, but not worth the slight code complexity
until the library structure gets more advanced.

Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
2023-04-07 16:24:18 +02:00
Robert Hensing
52d6ce6515 Move macOS TMPDIR hack from initNix to initLibStore
This code is bad. We shouldn't unset variables in programs whose
children may need them. Fixing one issue at a time, so postponing.
See https://github.com/NixOS/nix/issues/7731

Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
2023-04-07 16:24:18 +02:00
Robert Hensing
e706ffa007 Move preloadNSS() from initNix to initLibStore
It is required for the sandbox, which is a libstore responsibility;
not just libmain.

Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
2023-04-07 16:24:18 +02:00
Robert Hensing
a58be39476 Move sodium_init() to initLibStore()
Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
2023-04-07 16:24:18 +02:00
Robert Hensing
a692c43729 Move loadConfFile() to initLibStore
Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.

Using libstore without loading the config file is risky, as sqlite
may then be misconfigured. See https://github.com/cachix/cachix/issues/475
2023-04-07 16:24:18 +02:00
John Ericson
0746951be1
Finish converting existing comments for internal API docs (#8146)
* Finish converting existing comments for internal API docs

99% of this was just reformatting existing comments. Only two exceptions:

- Expanded upon `BuildResult::status` compat note

- Split up file-level `symbol-table.hh` doc comments to get
  per-definition docs

Also fixed a few whitespace goofs, turning leading tabs to spaces and
removing trailing spaces.

Picking up from #8133

* Fix two things from comments

* Use triple-backtick not indent for `dumpPath`

* Convert GNU-style `\`..'` quotes to markdown style in API docs

This will render correctly.
2023-04-07 13:55:28 +00:00
John Ericson
4e9f32f993 Liberate checkDerivationOutputs from LocalStore
Make it instead a method on `Derivation` that can work with any store.
We will need this for a CLI command to create a derivation.
2023-04-07 08:34:58 -04:00
John Ericson
b200784cec Include the name in the JSON for derivations
This is non-breaking change in the to-JSON direction. This *is* a
breaking change in the from-JSON direction, but we don't care, as that
is brand new in this PR.

`nix show-derivation --help` currently has the sole public documentation
of this format, it is updated accordingly.
2023-04-07 08:34:58 -04:00
John Ericson
fe9cbe838c Create Derivation::fromJSON
And test, of course
2023-04-07 08:34:58 -04:00
matthewcroughan
9207f94582 Add Store::isTrustedClient()
This function returns true or false depending on whether the Nix client
is trusted or not. Mostly relevant when speaking to a remote store with
a daemon.

We include this information in `nix ping store` and `nix doctor`

Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
2023-04-06 19:59:57 -04:00
Robert Hensing
9470ee877d Allow open switch-enum in 5 places 2023-04-03 18:45:20 +02:00
Robert Hensing
3dac4c7874 Add explicit case statements where -Wswitch-enum would report them 2023-04-03 18:17:32 +02:00
Robert Hensing
62cacc371f Fix BuildResult.toString() for NoSubstituters 2023-04-03 18:17:30 +02:00
John Ericson
9383520b75 Move querySubstitutablePathInfos from LocalStore to Store
The code is not local-store-specific, so we should share it with all
stores. More uniform behavior is better, and a less store-specific
functionality is more maintainable.

This fixes a FIXME added in f73d911628 by @edolstra himself.
2023-04-02 20:32:01 -04:00
John Ericson
5d56e2daf7 Add comparison methods for content addresses 2023-04-01 16:52:23 -04:00
John Ericson
eeecfacb43 Merge branch 'path-info' into ca-drv-exotic 2023-04-01 16:40:32 -04:00
John Ericson
1f8e1edba9 Merge commit 'a6d00a7bfb18e7ec461ac1d54203cc628aca5c66' into ca-drv-exotic 2023-04-01 15:15:38 -04:00
John Ericson
f7f44f7c96 Merge commit 'aa99005004bccc9be506a2a2f162f78bad4bcb41' into ca-drv-exotic 2023-04-01 15:15:32 -04:00
John Ericson
2ef99cd104
Merge pull request #8122 from bjornfor/use-nix-store-l-unless-experimental-enabled
Don't recommend 'nix log' unless experimental feature is enabled
2023-04-01 14:50:16 -04:00
John Ericson
f4ab297b31 Ensure all headers have #pragma once and are in API docs
`///@file` makes them show up in the internal API dos. A tiny few were
missing `#pragma once`.
2023-03-31 23:19:44 -04:00
John Ericson
abd5e7dec0 Extend internal API docs, part 2
Picking up from #8111.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-03-31 23:01:40 -04:00
Patrick Jackson
2b905d1d35 Replace unnecessary Sync<uint64_t> with std::atomic<uint64_t> 2023-03-31 17:56:25 -07:00
Théophane Hufschmitt
e32ca3cf16
Merge pull request #8018 from tweag/ssh-password-prompt
SSH: don't erase password prompt if it is displayed
2023-03-31 12:06:10 +02:00
Patrick Jackson
804180ad52 Only lock once 2023-03-30 18:06:27 -07:00
Patrick Jackson
309753ebb5 Fix data race in copyPaths 2023-03-30 16:59:34 -07:00
John Ericson
c51d554c93 Use "raw pattern" for content address types
We weren't because this ancient PR predated it!

This is actually a new version of the pattern which addresses some
issues identified in #7479.
2023-03-30 17:12:49 -04:00
John Ericson
a6d00a7bfb Fix warning 2023-03-30 16:29:13 -04:00
John Ericson
aa99005004 Merge remote-tracking branch 'upstream/master' into path-info
Also improve content-address.hh API docs.
2023-03-30 16:28:53 -04:00
John Ericson
8cab89a94b Convert a bunch of comments in headers to Doxygen documentation
The internal API docs now contain more useful information.

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-03-29 23:22:07 -04:00
Bjørn Forsman
74d94b2a13 Don't recommend 'nix log' unless experimental feature is enabled
This fixes the issue that `nix-build`, without experimental feature
'nix-command' enabled, recommends the experimental CLI `nix log` to view
build logs. Now it'll recommend the stable `nix-store -l` CLI instead.

Fixes https://github.com/NixOS/nix/issues/8118
2023-03-28 21:02:08 +02:00
Eelco Dolstra
237587bc0a
Merge pull request #8084 from edolstra/store-docs
Auto-generate store documentation
2023-03-27 15:46:18 +02:00
John Ericson
1d539aa287
Merge pull request #8073 from tweag/fix-root-channels-location
Fix root channels location
2023-03-27 09:37:53 -04:00
Eelco Dolstra
5a0f5b5c34
Merge pull request #8062 from edolstra/ssl-cert-file
Add a setting for configuring the SSL certificates file
2023-03-27 15:37:15 +02:00
John Ericson
570829d67e
Merge pull request #7609 from obsidiansystems/hide-experimental-settings
Hide experimental settings
2023-03-27 09:19:29 -04:00
Eelco Dolstra
a519436e9d ssl-cert-file: Use lists 2023-03-27 14:08:39 +02:00
Théophane Hufschmitt
128994509f Make some comments in profiles.hh doxygen-enabled
These are proper documentation of the API, so they deserve to be here
2023-03-27 10:02:10 +02:00
Eelco Dolstra
168b6021c5 Tweaks 2023-03-23 15:32:59 +01:00
Eelco Dolstra
b79df9dedc Register LocalStore to ensure it's included in the manual 2023-03-23 15:23:13 +01:00
Eelco Dolstra
161f4b0dea Document store URLs 2023-03-23 10:38:48 +01:00
Théophane Hufschmitt
fb67c1a1fb Factor out the generation of the profile/channel directory
Make sure that all the code paths use the same one, and that the
backwards-compatibility measures are probably in place when needed
2023-03-23 10:22:42 +01:00
Eelco Dolstra
abc449bc30
Update src/libstore/local-store.md
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-03-23 10:12:57 +01:00
Eelco Dolstra
80f0b8d307 Fix SSHStore 2023-03-23 09:35:35 +01:00
Eelco Dolstra
b134546f08 Fix clang build 2023-03-23 09:11:15 +01:00
Eelco Dolstra
5691bac202 Improve store setting descriptions / Markdown formatting 2023-03-22 14:23:36 +01:00
Alexander Bantyev
5291a82cd9
SSHMaster: pause logger to show password prompt
Pause logger before starting SSH connections, and resume it after the
connection is established, so that SSH password prompts are not erased
by the logger's updates.
2023-03-22 09:45:08 +04:00
Eelco Dolstra
233b063b08 Move store docs to 'nix help-stores'
Why not 'nix help stores'? Well, 'nix help <arg>' already means 'show
help on the "arg" subcommand'.
2023-03-21 14:37:09 +01:00
Eelco Dolstra
9eb53bbf17 Support per-store Markdown documentation 2023-03-21 14:03:40 +01:00
Eelco Dolstra
8d6d59cb1b nix store --help: Include store type documentation 2023-03-21 14:03:40 +01:00
Eelco Dolstra
515662ad70 Cleanup 2023-03-20 18:06:08 +01:00
Eelco Dolstra
16db8dc96f Open slave pseudoterminal before CLONE_NEWUSER
Otherwise, when running as root and user namespaces are enabled,
opening the slave fails with EPERM.

Fixes "opening pseudoterminal slave: Permission denied" followed by a
hang (https://hydra.nixos.org/build/213104244), and "error: getting
sandbox mount namespace: No such file or directory" (#8072), which
happens when the child fails very quickly and consequently reading
/proc/<child>/ns fails.
2023-03-20 17:58:36 +01:00
John Ericson
296831f641 Move enabled experimental feature to libutil struct
This is needed in subsequent commits to allow the settings and CLI args
infrastructure itself to read this setting.
2023-03-20 11:05:22 -04:00
Eelco Dolstra
e53e5c38d4 Add a setting for configuring the SSL certificates file
This provides a platform-independent way to configure the SSL
certificates file in the Nix daemon. Previously we provided
instructions for overriding the environment variable in launchd, but
that obviously doesn't work with systemd. Now we can just tell users
to add

  ssl-cert-file = /etc/ssl/my-certificate-bundle.crt

to their nix.conf.
2023-03-17 18:32:18 +01:00
Eelco Dolstra
7f46ebcf90
Merge pull request #8049 from edolstra/unexpected-eof
Fix "unexpected EOF" errors on macOS
2023-03-16 16:13:42 +01:00
Linus Heckemann
af4cbdafe7 LocalDerivationGoal: set NIX_ATTRS_*_FILE correctly for sandboxed builds 2023-03-16 15:26:52 +01:00
Eelco Dolstra
5eb8bfd0f1 Remove a variable in LocalDerivationGoal 2023-03-15 10:59:10 +01:00
Eelco Dolstra
6029c763c2 Change builderOut from Pipe to AutoCloseFD 2023-03-15 10:59:10 +01:00
Eelco Dolstra
c536e00c9d Move pseudoterminal slave open to child
Hopefully this fixes "unexpected EOF" failures on macOS
(#3137, #3605, #7242, #7702).

The problem appears to be that under some circumstances, macOS
discards the output written to the slave side of the
pseudoterminal. Hence the parent never sees the "sandbox initialized"
message from the child, even though it succeeded. The conditions are:

* The child finishes very quickly. That's why this bug is likely to
  trigger in nix-env tests, since that uses a builtin builder. Adding
  a short sleep before the child exits makes the problem go away.

* The parent has closed its duplicate of the slave file
  descriptor. This shouldn't matter, since the child has a duplicate
  as well, but it does. E.g. moving the close to the bottom of
  startBuilder() makes the problem go away. However, that's not a
  solution because it would make Nix hang if the child dies before
  sending the "sandbox initialized" message.

* The system is under high load. E.g. "make installcheck -j16" makes
  the issue pretty reproducible, while it's very rare under "make
  installcheck -j1".

As a fix/workaround, we now open the pseudoterminal slave in the
child, rather than the parent. This removes the second condition
(i.e. the parent no longer needs to close the slave fd) and I haven't
been able to reproduce the "unexpected EOF" with this.
2023-03-15 10:58:37 +01:00
Eelco Dolstra
19326ac297 Simplify commonChildInit() 2023-03-14 18:58:12 +01:00
Théophane Hufschmitt
99af56cd0d
Merge pull request #8015 from tweag/progress-during-nix-copy
Display progress when running copyPaths (nix copy)
2023-03-10 16:23:22 +01:00
Eelco Dolstra
4f3a4b732c Make findRuntimeRoots() more resilient to disappearing processes
I saw this random failure in https://hydra.nixos.org/build/211811692:

  error: opening /proc/15307/fd: No such process

while running nix-collect-garbage in a readfile-context.sh. This is
because we're not handling ESRCH errors reading /proc/<pid>/fd. So
just move the read inside the try/catch where we do handle it.
2023-03-09 16:44:51 +01:00
Alexander Bantyev
4bef2016a1
Display progress when running copyPaths (nix copy)
`nix copy` operations did not show progress. This is quite confusing.

Add a `progressSink` which displays the progress during `copyPaths`,
pretty much copied from `copyStorePath`.

Fixes https://github.com/NixOS/nix/issues/8000
2023-03-09 16:03:48 +04:00
Eelco Dolstra
7bfed34367 Fix crash/hang with CA derivations
The curl download can outlive DrvOutputSubstitutionGoal (if some other
error occurs), so at shutdown setting the promise to an exception will
fail because 'this' is no longer valid in the callback. This can
manifest itself as a segfault, "corrupted double-linked list" or hang.
2023-03-08 11:09:15 +01:00
Eelco Dolstra
0507462c06
Merge pull request #7918 from zimbatm/fix-empty-nix-store-env
treat empty NIX_STORE_DIR env vars as unset
2023-03-03 13:49:56 +01:00
Eelco Dolstra
f0908f592c
Merge pull request #7942 from edolstra/remove-format
Remove FormatOrString and remaining uses of format()
2023-03-03 10:22:11 +01:00
Eelco Dolstra
29abc8e764 Remove FormatOrString and remaining uses of format() 2023-03-02 15:57:54 +01:00
Théophane Hufschmitt
1a825b6919 Log the decompressed body in case of http errors
Don't show the users the raw (possibly compressed) error message as
everyone isn't able to decompress brotli in their brain.
2023-03-02 14:59:15 +01:00
Eelco Dolstra
767974f411
Merge pull request #7924 from mkenigs/valid
Always set valid in path-info --json output
2023-03-02 09:58:20 +01:00
Félix Baylac Jacqué
25300c0ecd
Treat empty env var paths as unset
We make sure the env var paths are actually set (ie. not "") before
sending them to the canonicalization function. If we forget to do so,
the user will end up facing a puzzled failed assertion internal error.

We issue a non-failing warning as a stop-gap measure. We could want to
revisit this to issue a detailed failing error message in the future.
2023-03-01 20:50:07 +01:00
John Ericson
b7f01a82a9 Remove dead code RemoteStore::sameMachine
It has been dead code since 9747ea84b4.
2023-03-01 11:10:30 -05:00
Valentin Gagarin
306e5c5ce5
Merge pull request #7788 from bobvanderlinden/pr-improve-nix-profile-install-error
Improve error on conflict for nix profile install
2023-03-01 11:48:43 +01:00
Matthew Kenigsberg
f86f2b973f Always set valid in path-info --json output
Currently the valid key is only present when the path is invalid, which
makes checking path validity more complex than it should be. With this
change, the valid key can always be used to check if a path is valid
2023-02-28 16:04:41 -07:00
John Ericson
5abd643c6d Merge branch 'path-info' into ca-drv-exotic 2023-02-28 12:46:00 -05:00
John Ericson
d381248ec0 No inheritance for TextInfo and FixedOutputInfo 2023-02-28 12:14:11 -05:00
John Ericson
85bb865d20 Revert "Remove some designated initializers"
This reverts commit ee9eb83a84.
2023-02-28 11:57:20 -05:00
John Ericson
123b11ff83 Clarify store path grammar and improve comment on makeType 2023-02-28 11:49:13 -05:00
John Ericson
d12f57c2c0 Merge remote-tracking branch 'upstream/master' into path-info 2023-02-28 11:34:34 -05:00
John Ericson
c36b584f8e Fix typo in the method name 2023-02-28 11:34:18 -05:00
Eelco Dolstra
a4a5d828e2
Merge pull request #7793 from layus/interrupt_downloads
Check interrupts even when download stalled
2023-02-28 13:29:29 +01:00
Bob van der Linden
3113b13df9
buildenv: throw BuildEnvFileConflictError with more context
At the moment an Error is thrown that only holds an error message
regarding `nix-env` and `nix profile`. These tools make use of
builtins.buildEnv, but buildEnv is also used in other places. These
places are unrelated to Nix profiles, so the error shouldn't mention
these tools.

This generic error is now BuildEnvFileConflictError, which holds more
contextual information about the files that were conflicting while
building the environment.
2023-02-27 21:39:34 +01:00
Valentin Gagarin
fd0e21475c add information on the build-hook setting
add a warning that you probably never want to change this.
2023-02-27 16:27:56 +01:00
Théophane Hufschmitt
532c70f531
Merge pull request #7856 from yorickvP/fix-nsswitch
Wait with making /etc unwritable until after build env setup
2023-02-21 09:39:10 +01:00
John Ericson
208c8d326d Derivation::toJSON: fix bug!
When I moved this code from the binary to libnixstore #7863, I forgot to
display the environment variables!
2023-02-20 17:38:57 -05:00
John Ericson
7998686c00 Test toJSON of DerivationOutput and Derivation 2023-02-19 11:12:12 -05:00
John Ericson
cd583362ec Move Derivation toJSON logic to libnixstore 2023-02-19 10:06:40 -05:00
Yorick van Pelt
bbba49b3e4
Wait with making /etc unwritable until after build env setup
This fixes /etc/nsswitch.conf
2023-02-17 16:34:45 +01:00
Yorick van Pelt
49fd72a903
Make /etc writability conditional on uid-range feature 2023-02-14 13:55:41 +01:00
Yorick van Pelt
db41f74af3
Don't allow writing to /etc 2023-02-14 12:03:34 +01:00
Eelco Dolstra
c205d10c66
Merge pull request #7616 from hercules-ci/fix-3898
Fix foreign key error inserting into NARs #3898
2023-02-13 13:02:19 +01:00
Eelco Dolstra
b3d29e80e0
Merge pull request #7805 from edolstra/c++2a
Fix building with GCC 9
2023-02-10 20:41:29 +01:00
Eelco Dolstra
67451d8ed7
Merge pull request #7802 from edolstra/fix-7783
Fix PID namespace support check
2023-02-10 20:41:13 +01:00
Eelco Dolstra
5978ceb271 Fix building with GCC 9
Nixpkgs on aarch64-linux is currently stuck on GCC 9
(https://github.com/NixOS/nixpkgs/issues/208412) and using gcc11Stdenv
doesn't work either.

So use c++2a instead of c++20 for now. Unfortunately this means we
can't use some C++20 features for now (like std::span).
2023-02-10 18:38:57 +01:00
Théophane Hufschmitt
9ebbe35817
Merge pull request #5588 from tweag/balsoft/xdg
Follow XDG Base Directory standard
2023-02-10 18:05:50 +01:00
Alexander Bantyev
2384d36083
A setting to follow XDG Base Directory standard
XDG Base Directory is a standard for locations for storing various
files. Nix has a few files which seem to fit in the standard, but
currently use a custom location directly in the user's ~, polluting
it:

- ~/.nix-profile
- ~/.nix-defexpr
- ~/.nix-channels

This commit adds a config option (use-xdg-base-directories) to follow
the XDG spec and instead use the following locations:

- $XDG_STATE_HOME/nix/profile
- $XDG_STATE_HOME/nix/defexpr
- $XDG_STATE_HOME/nix/channels

If $XDG_STATE_HOME is not set, it is assumed to be ~/.local/state.

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
Co-authored-by: Tim Fenney <kodekata@gmail.com>
Co-authored-by: pasqui23 <pasqui23@users.noreply.github.com>
Co-authored-by: Artturin <Artturin@artturin.com>
Co-authored-by: John Ericson <Ericson2314@Yahoo.com>
2023-02-10 20:14:06 +04:00
Eelco Dolstra
3e6e34cdf5 LocalDerivationGoal::startBuilder(): Use startProcess() to clone 2023-02-10 14:44:25 +01:00
Eelco Dolstra
f094ba7386 Simplify the PID namespace check: just try to mount /proc
Fixes #7783.
2023-02-10 14:38:14 +01:00
Guillaume Maudoux
e6ad8e8440 nit: cleaner diff 2023-02-10 00:57:56 +01:00
Guillaume Maudoux
aa18404ecb Flush data when download ends 2023-02-10 00:54:29 +01:00
Eelco Dolstra
e46429f674
Merge pull request #7712 from Mic92/advertise-compressions
advertise transport encoding in http transfers to
2023-02-09 17:15:25 +01:00
Guillaume Maudoux
78fea899e0 Check interupts even when download stalled 2023-02-09 13:56:50 +01:00
Robert Hensing
19b495a48a NarInfoDiskCache: Also test id consistency with updated fields
And clarify test
2023-02-07 23:34:36 +01:00
Robert Hensing
fb94d5cabd NarInfoDiskCache: Keep BinaryCache.id stable and improve test
Fixes #3898

The entire `BinaryCaches` row used to get replaced after it became
stale according to the `timestamp` column. In a concurrent scenario,
this leads to foreign key conflicts as different instances of the
in-process `state.caches` cache now differ, with the consequence that
the older process still tries to use the `id` number of the old record.

Furthermore, this phenomenon appears to have caused the cache for
actual narinfos to be erased about every week, while the default
ttl for narinfos was supposed to be 30 days.
2023-02-07 23:34:36 +01:00
Robert Hensing
2ceece3ef3 NarInfoDiskCache: Prepare reproducer for #3898 2023-02-07 23:34:36 +01:00
Robert Hensing
79f62d2dda NarInfoDiskCacheImpl: Make dbPath a parameter
This allows testing with a clean database.
2023-02-07 23:34:36 +01:00
Robert Hensing
29f0b196f4 NarInfoDiskCache: Rename cacheExists -> upToDateCacheExists
This is slightly more accurate considering that an outdated record
may exist in the persistent cache. Possibly-outdated records are
quite relevant as they may be foreign keys to more recent information
that we want to keep, but we will not return them here.
2023-02-07 23:34:36 +01:00
Robert Hensing
8a0ef5d58e sqlite.cc: Add SQL tracing
Set environment variable NIX_DEBUG_SQLITE_TRACES=1 to log all sql statements.
2023-02-07 23:34:36 +01:00
Eelco Dolstra
c5c0617d6f Mention --no-sandbox if sandboxing is unsupported 2023-02-07 22:59:46 +01:00
Eelco Dolstra
bc1d9fd8b5 Check whether we can use PID namespaces
In unprivileged podman containers, /proc is not fully visible (there
are other filesystems mounted on subdirectories of /proc). Therefore
we can't mount a new /proc in the sandbox that matches the PID
namespace of the sandbox. So this commit automatically disables
sandboxing if /proc is not fully visible.
2023-02-07 22:51:53 +01:00
Eelco Dolstra
fb2f7f5dcc Fix auto-uid-allocation in Docker containers
This didn't work because sandboxing doesn't work in Docker. However,
the sandboxing check is done lazily - after clone(CLONE_NEWNS) fails,
we retry with sandboxing disabled. But at that point, we've already
done UID allocation under the assumption that user namespaces are
enabled.

So let's get rid of the "goto fallback" logic and just detect early
whether user / mount namespaces are enabled.

This commit also gets rid of a compatibility hack for some ancient
Linux kernels (<2.13).
2023-02-07 22:51:53 +01:00
Jörg Thalheim
f20d3726dd advertise transport encoding in http transfers to
tl;dr: With this 1 line change I was able to get a speedup of 1.5x on 1Gbit/s
wan connections by enabling zstd compression in nginx.

Also nix already supported all common compression format for http
transfer, webservers usually only enable them if they are advertised
through the Accept-Encoding header.

This pull requests makes nix advertises content compression support for
zstd, br, gzip and deflate.

It's particular useful to add transparent compression for binary caches
that serve packages from the host nix store in particular nix-serve,
nix-serve-ng and harmonia.

I tried so far gzip, brotli and zstd, whereas only zstd was able to bring
me performance improvements for 1Gbit/s WAN connections.

The following nginx configuration was used in combination with the
[zstd module](https://github.com/tokers/zstd-nginx-module) and
[harmonia](https://github.com/nix-community/harmonia/)

```nix
{
  services.nginx.virtualHosts."cache.yourhost.com" = {
    locations."/".extraConfig = ''
      proxy_pass http://127.0.0.1:5000;
      proxy_set_header Host $host;
      proxy_redirect http:// https://;
      proxy_http_version 1.1;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;

      zstd on;
      zstd_types application/x-nix-archive;
    '';
  };
}
```

For testing I unpacked a linux kernel tarball to the nix store using
this command `nix-prefetch-url --unpack https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.1.8.tar.gz`.

Before:

```console
$ nix build && rm -rf /tmp/hello  && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwbb88sw36fvah-linux-6.1.8.tar.gz'
warning: Git tree '/scratch/joerg/nix' is dirty

real    0m18,375s
user    0m2,889s
sys     0m1,558s
```

After:

```console
$ nix build && rm -rf /tmp/hello  && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwb
b88sw36fvah-linux-6.1.8.tar.gz'

real    0m11,884s
user    0m4,130s
sys     0m1,439s
```

Signed-off-by: Jörg Thalheim <joerg@thalheim.io>

Update src/libstore/filetransfer.cc

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-02-03 12:33:38 +00:00
Eelco Dolstra
dbe0748f97
Merge pull request #7739 from obsidiansystems/user-settings
Move `trustedUsers` and `allowedUsers` to separate config struct
2023-02-03 11:55:37 +01:00
John Ericson
a47e055e09 Move trustedUsers and allowedUsers to separate config struct
These settings are not needed for libstore at all, they are just used by
the nix daemon *command* for authorization on unix domain sockets. My
moving them to a new configuration struct just in that file, we avoid
them leaking anywhere else.

Also, it is good to break up the mammoth `Settings` struct in general.
Issue #5638 tracks this.

The message is not changed because I do not want to regress in
convenience to the user. Just saying "this connection is not trusted"
doesn't tell them out to fix the issue. The ideal thing to do would be
to somehow parameterize `processCommand` on how the error should be
displayed, so different sorts of connections can display different
information to the user based on how authentication is performed for the
connection in question. This, however, is a good bit more work, so it is
left for the future.

This came up with me thinking about the tcp:// store (#5265). The larger
project is not TCP *per se*, but the idea that it should be possible for
something else to manage access control to services like the Nix Daemon,
and those services simply trust or trust the incoming connection as they
are told. This is a more capability-oriented way of thinking about trust
than "every server implements its own auth separately" as we are used to today.

Its very great that libstore itself already implements just this model,
and so via this refactor I basically want to "enshrine" that so it
continues to be the case.
2023-02-02 14:17:24 -05:00
John Ericson
479c011784 Get rid of the authHook parameter on processConnection
This is (morally) dead code.

As @edolstra pointed out in
https://github.com/NixOS/nix/pull/5226#discussion_r1073470813, this is
no longer needed.

I created this in 8d4162ff9e, so it is
fitting that I now destroy it :).
2023-02-02 12:02:03 -05:00
John Ericson
87ca46263d Merge branch 'master' into path-info 2023-02-02 09:55:07 -05:00
Shea Levy
895c525d04
daemon: Warn on old clients passing unexpected plugin-files.
The setting itself was already ignored due to exception trying to set pluginFiles.
2023-02-02 06:03:45 -05:00
Shea Levy
92edc38369
Don't send plugin-files to the daemon.
This is radically unsafe and the daemon has already loaded its plugins
anyway.

Fixes cachix/devenv#276
2023-02-01 20:05:56 -05:00
Eelco Dolstra
57a4258426 Remove an unused capture 2023-02-01 20:27:35 +01:00
John Ericson
ee9eb83a84 Remove some designated initializers
With the switch to C++20, the rules became more strict, and we can no
longer initialize base classes. Make them comments instead.

(BTW
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2287r1.html
this offers some new syntax for this use-case. Hopefully this will be
adopted and we can eventually use it.)
2023-02-01 11:25:56 -05:00
John Ericson
59d3175649 Put back TODO
I don't think the `narHash` is in need of documentation more than the
other undocumented fields, but regardless this change has nothing to do
with that field and so we should leave the comment as is.
2023-02-01 10:09:25 -05:00
John Ericson
db759b1bc2 Undo style change
`&` without space before is far more common on this codebase than I
thought, so it is not worth changing just this one file. Maybe we will
adopt a formatter someday but until then this is fine.
2023-02-01 10:07:54 -05:00
John Ericson
0983a0bd30 Shrink diff in one place 2023-02-01 10:04:28 -05:00
John Ericson
43414738a0 Merge remote-tracking branch 'upstream/master' into path-info 2023-02-01 10:02:30 -05:00
Théophane Hufschmitt
518da6c6a3
Merge pull request #7716 from obsidiansystems/small-storePath-cleanups
Separate `path.hh` from `content-address.hh`
2023-02-01 16:00:28 +01:00
Eelco Dolstra
b55a946d8d
Merge pull request #7717 from obsidiansystems/delete-dead-code
Delete dead code
2023-02-01 15:57:04 +01:00
Eelco Dolstra
14b0b9ea5a
Merge pull request #7203 from graham33/feature/cpp20
Proposal: Use C++20
2023-02-01 15:41:04 +01:00
John Ericson
b6c98752f6 Merge remote-tracking branch 'upstream/master' into path-info 2023-01-30 18:04:54 -05:00
Robert Hensing
c9b9260f34
Merge pull request #7713 from obsidiansystems/more-rapid-check
Add more property tests
2023-01-30 18:54:53 +01:00
John Ericson
e21aa43212 Delete dead code
The references set seems to have been unused since `LegacySSHStore`
references were first created in
caa5793b4a.

The method decls never were upstream, and accidentally added by me in
062533f7cd (probably due to `git rerere`).
Sorry!

This reduces the diff from #3746.
2023-01-30 11:29:01 -05:00
John Ericson
560142fec0 Make per-variant Arbitrary impls too
This is a nice idea that @roberth requested. If we could factor our a
generic `std::variant` impl as a follow-up it would be even better!
2023-01-30 10:56:00 -05:00
John Ericson
02e745ba5b Separate path.hh from content-address.hh
It is good to separate concerns; `StorePath` (in general) has nothing to
do with `ContentAddress` anyways.

This reduces the diff from #3746.
2023-01-30 10:14:03 -05:00
John Ericson
974a983351 Shrink diff in two places
Stuff crept in there.
2023-01-30 09:59:55 -05:00
John Ericson
adb3608034 Merge branch 'small-storePath-cleanups' into path-info 2023-01-30 09:46:43 -05:00
John Ericson
f3e272ba02 Avoid some StorePath <-> Path round trips
Avoid needless work and throwing away invariants.

These conversions date back to when `StorePath` was in Rust and there
were issues with it missing utility methods.
2023-01-30 09:37:57 -05:00
Eelco Dolstra
c79b1582a7
Merge pull request #5226 from NixOS/client-side-profiles
Move the default profiles to the user’s home
2023-01-30 12:21:47 +01:00
Théophane Hufschmitt
4aaf0ee52e
Merge branch 'master' into referenceablePaths 2023-01-30 10:31:00 +01:00
John Ericson
ecd3e4ebd7 More property tests
Also put proper comparison methods on `DerivedPath` and
`NixStringContextElem`, which is needed for the tests but good in
general.
2023-01-29 17:09:59 -05:00
John Ericson
ec0c0efec6 Allow unit test infra to be reused across libs' tests
This allows using Arbitrary "instances" defined in libstore-tests in
libexpr-tests, something we will leverage in a moment.
2023-01-29 13:52:57 -05:00
Solène Rapenne
6b2729c81e improve documentation about substituters and trusted users
Co-authored-by: Théophane Hufschmitt <theophane.hufschmitt@tweag.io>
2023-01-26 09:56:44 +01:00
Solène Rapenne
64951d9125 Update src/libstore/daemon.cc
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-01-26 09:34:25 +01:00
Solène Rapenne
a96156c58f warnings: enhance the case of untrusted substituter for untrusted user 2023-01-26 09:34:25 +01:00
John Ericson
e68e8e3cee Merge branch 'path-info' into ca-drv-exotic 2023-01-23 16:54:45 -05:00
John Ericson
4540e7b940 Don't add StorePathDescriptor for now
We don't need it yet, we can add it back later.
2023-01-23 12:58:27 -05:00
John Ericson
c67e0cc58c Merge remote-tracking branch 'upstream/master' into path-info 2023-01-23 11:47:20 -05:00
John Ericson
018e2571aa Test store paths, with property tests
The property test in fact found a bug: we were excluding numbers!
2023-01-23 07:05:50 -05:00
John Ericson
685395332d Better-scope Store forward declarations 2023-01-23 07:05:50 -05:00
John Ericson
7fe308c2f8 Add rapidcheck dependency for testing
Property tests are great!

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2023-01-23 07:05:50 -05:00
Eelco Dolstra
75c89c3e5e Add test for OutputsSpec::Names
From @Ericson2314.
2023-01-18 16:34:37 +01:00
Eelco Dolstra
95cfd50d25 OutputSpec: Allow all valid output names
Fixes #7624.
2023-01-18 14:14:29 +01:00
John Ericson
3965b0f75f Try again to fix aarch64-linux build failure
f419ab48e6 was on the right track, but
there are a few more missing `raw()` calls to fix.
2023-01-17 09:14:17 -05:00
Théophane Hufschmitt
6bdf4edb77 Keep the default profile the same
It's used as the “system” profile in a bunch of places, so better not
touch it. Besides, it doesn't hurt to keep it since it's owned by root
any way, so it doesn't have the `chown` problem that the user profiles
had and that led to wanting to move them on the client-side.
2023-01-17 14:17:28 +01:00
Théophane Hufschmitt
c80621dbac Don't try to migrate existing profiles
Doing so would be more dangerous than useful, better leave them as-is if
they already exist
2023-01-17 14:17:28 +01:00
Théophane Hufschmitt
0601050755 Migrate the old profiles to the new location
Make sure that we don’t just create the new profiles directory, but that
we also migrate every existing profile to it.
2023-01-17 14:17:28 +01:00
Théophane Hufschmitt
a5919f4754 Move the default profiles to the user’s home
Rather than using `/nix/var/nix/{profiles,gcroots}/per-user/`, put the user
profiles and gcroots under `$XDG_DATA_DIR/nix/{profiles,gcroots}`.

This means that the daemon no longer needs to manage these paths itself
(they are fully handled client-side). In particular, it doesn’t have to
`chown` them anymore (removing one need for root).

This does change the layout of the gc-roots created by nix-env, and is
likely to break some stuff, so I’m not sure how to properly handle that.
2023-01-17 14:17:28 +01:00
John Ericson
f419ab48e6 Try to fix build failure
Failure: https://hydra.nixos.org/build/205357257/nixlog/1

The problem seems to be trying to `std::visit` a derived class of
`std::variant`. Per
https://stackoverflow.com/questions/63616709/incomplete-type-stdvariant-used-in-nested-name-specifier
certain C++ standard library implementations allow this, but others do
not.

The solution is simply to call the `raw` method, which upcasts the
reference back to the `std::variant`.
2023-01-15 15:16:14 -05:00
John Ericson
7c82213813 Merge branch 'path-info' into ca-drv-exotic 2023-01-14 17:09:58 -05:00
John Ericson
b3d91239ae Make ValidPathInfo have plain StorePathSet references like before
This change can wait for another PR.
2023-01-14 16:42:03 -05:00
John Ericson
056cc1c1b9 Merge remote-tracking branch 'upstream/master' into path-info 2023-01-14 14:27:28 -05:00
John Ericson
a416476217 Move ValidPathInfo defintions to path-info.cc
Originally there was no `path-info.*`, then there was `path-info.hh`,
then there was `path-info.cc`, but only for new things. Moving this
stuff over makes everything consistent.
2023-01-13 15:39:19 -05:00
John Ericson
2e7be46e73 Move new ValidPathInfo methods to path-info.cc
We'll move the old ones separately, so as not to clutter the diff.
2023-01-13 15:06:07 -05:00
Robert Hensing
fec527bba1
Merge pull request #7597 from tweag/move-implem-bit-to-implem-file
Move the `getBuildLog` implementation to its own implementation file
2023-01-13 20:16:33 +01:00
Robert Hensing
d21f54958e
Merge pull request #6815 from obsidiansystems/better-wanted-outputs
`OutputSpec` for `DerivationGoal` and `DerivedPath`, today's `OutputSpec` -> `ExtendedOutputSpec`
2023-01-13 16:03:12 +01:00
Théophane Hufschmitt
b8a0e9a9b8 Move the getBuildLog implementation to its own implementation file
Keep the header minimal and clean
2023-01-13 11:05:44 +01:00
Théophane Hufschmitt
bdeb6de889
Merge pull request #7430 from tweag/ca/fix-nix-log
Ca/fix nix log
2023-01-13 11:00:56 +01:00
John Ericson
d8512653d4 Write more (extended) output spec tests 2023-01-12 22:05:55 -05:00
John Ericson
d29eb08563 Assert on construction that OutputsSpec::Names is non-empty 2023-01-12 20:52:29 -05:00
John Ericson
e947aa5401 Unit test OuputsSpec::{union_, isSubsetOf} 2023-01-12 20:33:50 -05:00
John Ericson
31875bcfb7 Split OutputsSpec::merge into OuputsSpec::{union_, isSubsetOf}
Additionally get rid of the evil time we made an empty
`OutputSpec::Names()`.
2023-01-12 20:20:27 -05:00
Valentin Gagarin
48b2a3a0d0 remove unncessary cast 2023-01-12 13:23:32 +01:00
John Ericson
0faf5326bd Improve tests for OutputsSpec 2023-01-11 19:09:21 -05:00
John Ericson
5ba6e5d0d9 Remove default constructor from OutputsSpec
This forces us to be explicit.

It also requires to rework how `from_json` works. A `JSON_IMPL` is added
to assist with this.
2023-01-11 19:08:19 -05:00
John Ericson
114a6e2b09 Make it hard to construct an empty OutputsSpec::Names
This should be a non-empty set, and so we don't want people doing this
by accident. We remove the zero-0 constructor with a little inheritance
trickery.
2023-01-11 19:08:19 -05:00
John Ericson
ce2f91d356 Split OutputsSpec and ExtendedOutputsSpec, use the former more
`DerivedPath::Built` and `DerivationGoal` were previously using a
regular set with the convention that the empty set means all outputs.
But it is easy to forget about this rule when processing those sets.
Using `OutputSpec` forces us to get it right.
2023-01-11 18:57:18 -05:00
John Ericson
a7c0cff07f Rename OutputPath -> ExtendedOutputPath
Do this prior to making a new more limitted `OutputPath` we will use in
more places.
2023-01-11 18:55:29 -05:00
John Ericson
a8f45b5e5a Improve OutputsSpec slightly
A few little changes preparing for the rest.
2023-01-11 18:54:50 -05:00
Théophane Hufschmitt
a3ba80357d
Merge pull request #7543 from obsidiansystems/typed-string-context
Parse string context elements properly
2023-01-11 07:09:37 +01:00
Robert Hensing
34a1e0d29b doc/manual: Introduce @docroot@ as a stable base for includable snippets
This way the links are clearly within the manual (ie not absolute paths),
while allowing snippets to reference the documentation root reliably,
regardless of at which base url they're included.
2023-01-10 22:30:41 +01:00
John Ericson
da64f026dd Make clear that StorePathWithOutputs is a deprecated type
- Add a comment

- Put `OutputsSpec` in a different header (First part of #6815)

- Make a few stray uses of it in new code use `DerivedPath` instead.
2023-01-10 11:27:19 -05:00
John Ericson
81727f85cb Merge branch 'path-info' into ca-drv-exotic 2023-01-06 15:45:34 -05:00
John Ericson
46e942ff9e Do big rename to clean up code
- `PathReferences` -> `References`

- `PathReferences<StorePath>` -> `StoreReference`

- `references` -> `others`

- `hasSelfReference` -> `self`

And get rid of silly subclassing
2023-01-06 15:36:05 -05:00
John Ericson
848b0832b5 Merge branch 'path-info' into ca-drv-exotic 2023-01-06 12:39:14 -05:00
John Ericson
9cfa78e58a Optimize ValidPathInfo construction a bit better 2023-01-06 12:26:15 -05:00
John Ericson
6a168254ce Use named field initialization for references 2023-01-06 12:24:20 -05:00
John Ericson
8623143921 Make formatting consistent 2023-01-06 11:18:14 -05:00
John Ericson
989b8065b4 Merge branch 'path-info' into ca-drv-exotic 2023-01-06 10:56:22 -05:00
John Ericson
e9fc1e4fdb Merge remote-tracking branch 'upstream/master' into path-info 2023-01-06 10:35:20 -05:00
Théophane Hufschmitt
8d88c3b347
Merge pull request #7307 from hercules-ci/derivation-goal-improve-comment
libstore/derivation-goal: Elaborate a TODO for performance concern
2023-01-06 13:07:57 +01:00
Eelco Dolstra
420ccecc1e
Merge pull request #7557 from NixOS/fix-7529
On macOS with auto-uid-allocation and sandboxing, use the correct gid
2023-01-06 12:35:55 +01:00
Eelco Dolstra
3a98107170
Merge pull request #7542 from edolstra/gc-deadlock
Fix deadlock between auto-GC and addTempRoot()
2023-01-05 17:08:23 +01:00
Eelco Dolstra
0fe2b222d5
Merge pull request #7539 from tweag/fix-nix-why-depends--derivation
Fix `nix why-depends --derivation`
2023-01-05 15:32:04 +01:00
Eelco Dolstra
4e84b532ed On macOS with auto-uid-allocation and sandboxing, use the correct gid
macOS doesn't have user namespacing, so the gid of the builder needs
to be nixbld. The logic got "has sandboxing enabled" confused with
"has user namespaces".

Fixes #7529.
2023-01-05 04:58:55 -08:00
Eelco Dolstra
6991e558dd Move macOS sandbox files to sr/libstore/build 2023-01-04 04:50:45 -08:00
Eelco Dolstra
609a7dc059 Include macOS sandbox files in the Nix binary
This basically reverts 6e5165b773.
It fixes errors like

  sandbox-exec: <internal init prelude>:292:47: unable to open sandbox-minimal.sb: not found

when trying to run a development Nix installed in a user's home
directory.

Also, we're trying to minimize the number of installed files
to make it possible to deploy Nix as a single statically-linked
binary.
2023-01-04 04:36:07 -08:00
Naïm Favier
3c968191f1
move unsafeDiscardReferences out of outputChecks
It's not a check.
2023-01-03 18:53:01 +01:00
Naïm Favier
15f7fa59be
unsafeDiscardReferences
Adds a new boolean structured attribute
`outputChecks.<output>.unsafeDiscardReferences` which disables scanning
an output for runtime references.

    __structuredAttrs = true;
    outputChecks.out.unsafeDiscardReferences = true;

This is useful when creating filesystem images containing their own embedded Nix
store: they are self-contained blobs of data with no runtime dependencies.

Setting this attribute requires the experimental feature
`discard-references` to be enabled.
2023-01-03 17:19:16 +01:00
Eelco Dolstra
28d5b5cd45 Fix deadlock between auto-GC and addTempRoot()
Previously addTempRoot() acquired the LocalStore state lock and waited
for the garbage collector to reply. If the garbage collector is in the
same process (as it the case with auto-GC), this would deadlock as
soon as the garbage collector thread needs the LocalStore state lock.

So now addTempRoot() uses separate Syncs for the state that it
needs. As long at the auto-GC thread doesn't call addTempRoot() (which
it shouldn't), it shouldn't deadlock.

Fixes #3224.
2023-01-03 15:20:21 +01:00
Eelco Dolstra
224b56f10e Move creation of the temp roots file into its own function
This also moves the file handle into its own Sync object so we're not
holding the _state while acquiring the file lock. There was no real
deadlock risk here since locking a newly created file cannot block,
but it's still a bit nicer.
2023-01-03 14:51:23 +01:00
Eelco Dolstra
ae31b5f50f
Merge pull request #7497 from rski/master
src/libstore: Print the reason opening the DB failed
2023-01-03 12:44:14 +01:00
Théophane Hufschmitt
8cac451fce Fix why-depends for CA derivations (again)
This has the same goal as b13fd4c58e81b2b2b0d72caa5ce80de861622610,but
achieves it in a different way in order to not break
`nix why-depends --derivation`.
2023-01-02 17:42:22 +01:00
Théophane Hufschmitt
fb8fc6fda6
Merge pull request #7478 from hercules-ci/make-sure-initNix-called
libstore: Make sure that initNix has been called
2023-01-02 14:12:49 +01:00
Théophane Hufschmitt
e8a3e58171
Merge pull request #7521 from ncfavier/migration-deadlock
Release shared lock before acquiring exclusive lock
2023-01-02 11:08:43 +01:00
Steven Shaw
84b0893725
Fix error message 2023-01-01 12:37:43 +10:00
Eelco Dolstra
8c52f8ea9d
Merge pull request #7524 from ncfavier/sandbox-paths-closure
doc: sandbox-paths computes closures
2022-12-29 19:45:51 +01:00
Naïm Favier
d5d2f50ebb
doc: sandbox-paths computes closures 2022-12-28 17:09:20 +01:00
Naïm Favier
81c3f99b36
Release shared lock before acquiring exclusive lock
In principle, this should avoid deadlocks where two instances of Nix are
holding a shared lock on big-lock and are both waiting to get an
exclusive lock.

However, it seems like `flock(2)` is supposed to do this automatically,
so it's not clear whether this is actually where the problem comes from.
2022-12-27 15:58:14 +01:00
Robert Hensing
aba6eb348e libstore: Make sure that initNix has been called
Prevent bugs like https://github.com/cachix/cachix/pull/477
2022-12-24 14:39:30 +01:00
Eelco Dolstra
c164d304f3 nix develop: Set personality
This makes 'nix develop' set the Linux personality in the same way
that the actual build does, allowing a command like 'nix develop
nix#devShells.i686-linux.default' on x86_64-linux to work correctly.
2022-12-23 16:33:55 +01:00
rski
d034ed1891 src/libstore: Print the reason opening the DB failed
Without this, the error is lost, and it makes for a hard to debug
situation. Also remove some of the busyness inside the sqlite_open_v2
args.

The errcode returned is not the extended one. The only way to make open
return an extended code, would be to add SQLITE_OPEN_EXRESCODE to the
flags. In the future it might be worth making this change,
which would also simplify the existing SQLiteError code.
2022-12-23 02:55:51 +02:00
mupdt
bc8ab21c5a [PDT] TDE-3114: prevent a race-condition when creating the S3 cache 2022-12-21 04:50:40 -05:00
squalus
5987fb7459 Add fsync-store-paths option
- Add recursiveSync function to flush a directory tree to disk

- Add AutoCloseFD::startFsync to initiate an asynchronous fsync
  without waiting for the result

- Initiate an asynchronous fsync while extracting NAR files

- Implement the fsync-store-paths option in LocalStore
2022-12-20 12:03:35 -08:00
Robert Hensing
c965f35de7 Improve sqlite error messages
They did not include the detailed error message, losing essential
information for troubleshooting.

Example message:

    warning: creating statement 'insert or rplace into NARs(cache, hashPart, namePart, url, compression, fileHash, fileSize, narHash, narSize, refs, deriver, sigs, ca, timestamp, present) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1)': at offset 10: SQL logic error, near "rplace": syntax error (in '/tmp/nix-shell.grQ6f7/nix-test/tests/binary-cache/test-home/.cache/nix/binary-cache-v6.sqlite')

It's not the best example; more important information will be in
the message for e.g. a constraint violation.

I don't see why this specific error is printed as a warning, but
that's for another commit.
2022-12-17 14:51:37 +01:00
Taeer Bar-Yam
e5eb05c599 getBuildLog: factor out resolving derivations 2022-12-15 15:58:54 -05:00
Eelco Dolstra
0687e16c4a Fix a crash in DerivedPath::Built::toJSON() with impure derivations
The use of 'nullptr' here didn't result in a null JSON value, but in a
nullptr being cast to a string, which aborts.
2022-12-15 16:02:27 +01:00
Naïm Favier
1f3c0a3c1d
Allow disabling build users by unsetting build-users-group
Unsetting `build-users-group` (without `auto-allocate-uids` enabled)
gives the following error:

```
src/libstore/lock.cc:25: static std::unique_ptr<nix::UserLock> nix::SimpleUserLock::acquire(): Assertion `settings.buildUsersGroup != ""' failed.
```

Fix the logic in `useBuildUsers` and document the default value
for `build-users-group`.
2022-12-14 00:40:30 +01:00
John Ericson
5273cf4c97 Merge remote-tracking branch 'upstream/master' into indexed-store-path-outputs 2022-12-12 17:40:49 -05:00
John Ericson
dabb03b8d0 Merge remote-tracking branch 'upstream/master' into indexed-store-path-outputs 2022-12-12 17:36:02 -05:00
John Ericson
dc075dcdd0
Apply suggestions from code review
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2022-12-12 16:26:10 -05:00
Eelco Dolstra
ae5f62a894 Move isUri() and resolveUri() out of filetransfer.cc
These are purely related to NIX_PATH / -I command line parsing, so put
them in libexpr.
2022-12-12 14:05:35 +01:00
Eelco Dolstra
7396844676
Merge pull request #7421 from edolstra/lazy-trees-trivial-changes
Trivial changes from the lazy-trees branch
2022-12-12 13:52:56 +01:00
John Ericson
1879c7c95e
Merge branch 'master' into indexed-store-path-outputs 2022-12-12 07:33:36 -05:00
Théophane Hufschmitt
2affb19c92
Merge pull request #7409 from tweag/fix-6383
check the store for input before failing (hopefully fix #6383)
2022-12-09 06:33:30 +01:00
Taeer Bar-Yam
3b27181ee5 fix missing function after rebase 2022-12-08 16:59:21 -05:00
regnat
04b113f6cb Fix nix log with CA derivations
Fix #6209

When trying to run `nix log <installable>`, try first to resolve the derivation pointed to
by `<installable>` as it is the resolved one that holds the build log.

This has a couple of shortcomings:
1. It’s expensive as it requires re-reading the derivation
2. It’s brittle because if the derivation doesn’t exist anymore or can’t
   be resolved (which is the case if any one of its build inputs is missing),
   then we can’t access the log anymore

However, I don’t think we can do better (at least not right now).
The alternatives I see are:
1. Copy the build log for the un-resolved derivation. But that means a
   lot of duplication
2. Store the results of the resolving in the db. Which might be the best
   long-term solution, but leads to a whole new class of potential
   issues.
2022-12-08 16:03:20 -05:00
Eelco Dolstra
703d863a48 Trivial changes from the lazy-trees branch 2022-12-07 14:06:34 +01:00
Linus Heckemann
8e0946e8df Remove repeat and enforce-determinism options
These only functioned if a very narrow combination of conditions held:

- The result path does not yet exist (--check did not result in
  repeated builds), AND
- The result path is not available from any configured substituters, AND
- No remote builders that can build the path are available.

If any of these do not hold, a derivation would be built 0 or 1 times
regardless of the repeat option. Thus, remove it to avoid confusion.
2022-12-07 11:36:48 +01:00
Taeer Bar-Yam
1c8de7d3d0 improve style 2022-12-06 11:25:38 -05:00
Eelco Dolstra
54906bc93c
Merge pull request #7382 from fricklerhandwerk/doc-automatic-uid
move documentation on `auto-allocate-uids` to options docs
2022-12-06 11:31:34 +01:00
Eelco Dolstra
484578d3f9
Tweak option descriptions 2022-12-06 10:30:36 +01:00
Taeer Bar-Yam
8c7661da09 check the store for input before failing (hopefully fix #6383) 2022-12-05 23:22:38 -05:00
Taeer Bar-Yam
e4f9f3bf24 check the store for input before failing (hopefully fix #6700) 2022-12-05 11:27:47 -05:00
Eelco Dolstra
1e6a5d1ff6 Clean up cgroup handling in getMaxCPU()
Also, don't assume in LocalDerivationGoal that cgroups are mounted on
/sys/fs/cgroup.
2022-12-02 12:59:13 +01:00
Eelco Dolstra
1211e59a03 Move cgroup.{cc,hh} to libutil 2022-12-02 12:38:03 +01:00
Valentin Gagarin
0ea62670ed move documentation on auto-allocate-uids to options docs
this is where it belongs and can be found together with the other
options.
2022-12-01 04:40:02 +01:00
Eelco Dolstra
fbc53e97ed
Merge pull request #3600 from NixOS/auto-uid-allocation
Automatic UID allocation
2022-11-29 14:01:42 +01:00
Eelco Dolstra
4f762e2b02 Restore ownership of / for non-uid-range builds 2022-11-29 13:10:53 +01:00
Eelco Dolstra
67bcb99700 Add a setting for enabling cgroups 2022-11-28 21:54:02 +01:00
Eelco Dolstra
ff12d1c1a1 Check that auto-allocated UIDs don't clash with existing accounts 2022-11-28 20:49:17 +01:00
Eelco Dolstra
dbf78a7ada
Merge pull request #7313 from yorickvP/nlohmann-everywhere
Replace src/libutil/json.cc with nlohmann
2022-11-28 15:03:48 +01:00
Eelco Dolstra
5b798f6cae Fix random client failures during GC server shutdown
We need to close the GC server socket before shutting down the active
GC client connections, otherwise a client may (re)connect and get
ECONNRESET. But also handle ECONNRESET for resilience.

Fixes random failures like

  GC socket disconnected
  connecting to '/tmp/nix-shell.y07M0H/nix-test/default/var/nix/gc-socket/socket'
  sending GC root '/tmp/nix-shell.y07M0H/nix-test/default/store/kb5yzija0f1x5xkqkgclrdzldxj6nnc6-non-blocking'
  reading GC root from client: error: unexpected EOF reading a line
  1 store paths deleted, 0.00 MiB freed
  error: reading from file: Connection reset by peer

in gc-non-blocking.sh.
2022-11-27 12:57:18 +01:00
John Ericson
26534f141c
Merge branch 'master' into indexed-store-path-outputs 2022-11-25 08:14:32 -05:00
Eelco Dolstra
6292d5616e Merge remote-tracking branch 'origin/master' into auto-uid-allocation 2022-11-23 11:16:09 +01:00
Eelco Dolstra
05d0892443
Merge pull request #7328 from edolstra/nix-build-stats
nix build --json: Include build statistics
2022-11-22 14:41:15 +01:00
Eelco Dolstra
3d23b9d032 SimpleUserLock::getSupplementaryGIDs(): Filter out main gid
This avoids having the user's gid in the supplementary group list as
well.
2022-11-22 10:26:17 +01:00
Eelco Dolstra
b37c2d84b6 Always call setgroups()
We shouldn't skip this if the supplementary group list is empty,
because then the sandbox won't drop the supplementary groups of the
parent (like "root").
2022-11-22 10:26:17 +01:00
Eelco Dolstra
02c02ee7c3
Merge pull request #6456 from amjoseph-nixpkgs/seccomp-mips
local-derivation-goal.cc: enable seccomp filters for mips{32,64}
2022-11-21 23:03:00 +01:00
Eelco Dolstra
c776dfbb35
Use hex for startId
Co-authored-by: Linus Heckemann <git@sphalerite.org>
2022-11-21 18:46:55 +01:00
Eelco Dolstra
9d17ce07e8 AutoUserLock: If sandboxing is disabled, use the build users group
We have to use a gid that has write access to the Nix store.
2022-11-21 12:55:49 +01:00
Eelco Dolstra
f0baa5c128 nix build --json: Include build statistics
Example:

  # nix build -L --extra-experimental-features cgroups --impure --expr 'with import <nixpkgs> {}; runCommand "foo" {} "dd if=/dev/urandom bs=1M count=1024 | md5sum; mkdir $out"' --json
  [
    {
      "cpuSystem": 1.911431,
      "cpuUser": 1.214249,
      "drvPath": "/nix/store/xzdqz67xba18hljhycp0hwfigzrs2z69-foo.drv",
      "outputs": {
        "out": "/nix/store/rh9mc9l2gkpq8kn2sgzndr6ll7ffjh6l-foo"
      },
      "startTime": 1669024076,
      "stopTime": 1669024079
    }
  ]
2022-11-21 12:06:01 +01:00
Eelco Dolstra
e7a5b76844 Rename derivedPathsWithHintsToJSON -> builtPathsToJSON 2022-11-21 11:56:20 +01:00
Eelco Dolstra
82d5cf2a76 Fix macOS build 2022-11-21 11:45:41 +01:00
Eelco Dolstra
653b32a78f Merge remote-tracking branch 'origin/master' into auto-uid-allocation 2022-11-21 11:33:23 +01:00
Eelco Dolstra
ec45f4b82e Fix indentation 2022-11-21 11:12:45 +01:00
Eelco Dolstra
300753d594 nix build --json: Include build statistics
Example:

  # nix build -L --extra-experimental-features cgroups --impure --expr 'with import <nixpkgs> {}; runCommand "foo" {} "dd if=/dev/urandom bs=1M count=1024 | md5sum; mkdir $out"' --json
  [
    {
      "cpuSystem": 1.911431,
      "cpuUser": 1.214249,
      "drvPath": "/nix/store/xzdqz67xba18hljhycp0hwfigzrs2z69-foo.drv",
      "outputs": {
        "out": "/nix/store/rh9mc9l2gkpq8kn2sgzndr6ll7ffjh6l-foo"
      },
      "startTime": 1669024076,
      "stopTime": 1669024079
    }
  ]
2022-11-21 10:49:01 +01:00
Eelco Dolstra
f538ee4342 Rename derivedPathsWithHintsToJSON -> builtPathsToJSON 2022-11-21 09:38:08 +01:00
Eelco Dolstra
e6b71f84a0 Use cgroup.kill to quickly kill cgroups 2022-11-18 16:59:36 +01:00
Eelco Dolstra
fa68eb367e Get CPU stats from the cgroup 2022-11-18 13:40:59 +01:00
Eelco Dolstra
128910ba23 Separate cgroup support from auto-uid-allocation
The new experimental feature 'cgroups' enables the use of cgroups for
all builds. This allows better containment and enables setting
resource limits and getting some build stats.
2022-11-18 10:39:28 +01:00
Eelco Dolstra
f423d4425f Fix segfault in unprivileged mode 2022-11-17 11:56:45 +01:00
Yorick van Pelt
09f00dd4d0
Replace src/libutil/json.cc with nlohmann json generation 2022-11-16 16:50:50 +01:00
Guillaume Maudoux
a7d2a3d087 Allow system certs access to fixed-output derivations 2022-11-16 15:23:59 +01:00
Robert Hensing
bcd298d39b libstore/derivation-goal: Elaborate a TODO for performance concern 2022-11-15 17:57:40 +01:00
Théophane Hufschmitt
4bf70b74a7
Merge pull request #7294 from tobim/support-aws-sdk-1.10
libstore: link to aws-crt-cpp
2022-11-15 16:51:09 +01:00
Théophane Hufschmitt
3ade5f5d60
Merge pull request #7283 from hercules-ci/issue-6572
Fix #6572 `requires non-existent output`
2022-11-15 16:24:24 +01:00
Robert Hensing
7e162c69fe derivation-goal: Fix requires non-existing output error
It occurred when a output of the dependency was already available,
so it didn't need rebuilding and didn't get added to the
inputDrvOutputs.
This process-related info wasn't suitable for the purpose of finding
the actual input paths for the builder. It is better to do this in
absolute terms by querying the store.
2022-11-14 17:52:55 +01:00
Théophane Hufschmitt
8b4352d79b Merge remote-tracking branch 'nixos/master' into readFile-scan-references 2022-11-14 15:00:05 +01:00
Tobias Mayer
07f2cb1e8f
libstore: link to aws-crt-cpp
This change is needed to support aws-sdk-cpp 1.10 and newer.

I opted not to make this dependent on the sdk version because
the crt dependency has been in the interface of the older
sdk as well, and it was only coincidence that libstore didn't
make use of any privately defined symbols directly.
2022-11-12 14:34:23 +01:00
Naïm Favier
e7ed9ae0c7
Restrict readFile context to references that appear in the string
When calling `builtins.readFile` on a store path, the references of that
path are currently added to the resulting string's context.

This change makes those references the *possible* context of the string,
but filters them to keep only the references whose hash actually appears
in the string, similarly to what is done for determining the runtime
references of a path.
2022-11-11 13:04:34 +01:00
Eelco Dolstra
6c6eff8ac4 Remove the SystemdCgroup feature 2022-11-10 17:24:12 +01:00
Eelco Dolstra
05d258667d Fix build on macOS 2022-11-08 08:00:29 -08:00
Eelco Dolstra
2fde7e0108 Split auto UID allocation from cgroups
Cgroups are now only used for derivations that require the uid-range
range feature. This allows auto UID allocation even on systems that
don't have cgroups (like macOS).

Also, make things work on modern systems that use cgroups v2 (where
there is a single hierarchy and no "systemd" controller).
2022-11-08 16:03:42 +01:00
Eelco Dolstra
b95faccf03 Merge remote-tracking branch 'origin/master' into auto-uid-allocation 2022-11-03 17:43:40 +01:00
John Ericson
13f2a6f38d
Merge branch 'master' into indexed-store-path-outputs 2022-10-28 23:22:18 +01:00
Théophane Hufschmitt
f8d0193383 Pass the right argv when calling the build hook
Call it as `['nix', '__build-remote', ... ]` rather than the previous
`["__build-remote", "nix __build-remote", ... ]` which seemed to have
been most likely unintended
2022-10-27 11:53:04 +02:00
Austin Kiekintveld
8e7804273c Defer to SSH config files for ForwardAgent option
Currently, Nix passes `-a` when it runs commands on a remote machine via
SSH, which disables agent forwarding. This causes issues when the
`ForwardAgent` option is set in SSH config files, as the command line
operation always overrides those.

In particular, this causes issues if the command being run is `sudo`
and the remote machine is configured with the equivalent of NixOS's
`security.pam.enableSSHAgentAuth` option. Not allowing SSH agent
forwarding can cause authentication to fail unexpectedly.

This can currently be worked around by setting `NIX_SSHOPTS="-A"`, but
we should defer to the options in the SSH config files to be least
surprising for users.
2022-10-22 19:51:22 -05:00
Graham Bennett
4563e80363 Fix C++20 warnings 2022-10-22 15:16:46 +01:00
Graham Bennett
c5fd34a14e Build with C++20 2022-10-22 14:24:25 +01:00
Eelco Dolstra
e136d57f26
Implement BinaryCacheStore::queryPathFromHashPart() 2022-10-18 17:48:09 +02:00
Eelco Dolstra
3093bd3a85
Merge pull request #7168 from NixOS/rosetta-test
Improve Rosetta detection
2022-10-14 17:35:57 +02:00
Eelco Dolstra
285277a61a Remove useless debug statements
We haven't parsed the '-v' command line flags yet when this code executes,
so we can't actually get debug output here.
2022-10-14 00:35:33 -07:00
Eelco Dolstra
ddd5503950 Use /usr/bin/true 2022-10-14 00:34:31 -07:00
Eelco Dolstra
0359d6d123 Fix error display if execve() in the builder fails
After we've send "\2\n" to the parent, we can't send a serialized
exception anymore. It will show up garbled like

  $ nix-build --store /tmp/nix --expr 'derivation { name = "foo"; system = "x86_64-linux"; builder = "/foo/bar"; }'
  this derivation will be built:
    /nix/store/xmdip0z5x1zqpp6gnxld3vqng7zbpapp-foo.drv
  building '/nix/store/xmdip0z5x1zqpp6gnxld3vqng7zbpapp-foo.drv'...

  ErrorErrorEexecuting '/foo/bar': No such file or directory
  error: builder for '/nix/store/xmdip0z5x1zqpp6gnxld3vqng7zbpapp-foo.drv' failed with exit code 1
2022-10-13 21:35:16 +02:00
Eelco Dolstra
96eb5ef156 Improve Rosetta detection
Turns out that one of those *.plist files can exist even if Rosetta is
not installed. So let's just try to run an x86_64-darwin binary
directly.
2022-10-13 11:46:16 -07:00
Steam Deck User
a86916eb72 Make warning about chroot store location more accurate
While trying to use an alternate directory for my Nix installation, I
noticed that nix's output didn't reflect the updated state
directory. This patch corrects that and now prints the warning before
attempting to create the directory (if the directory creation fails,
it wouldn't have been obvious why nix was attempting to create the
directory in the first place).

With this patch, I now get the following warning:

    warning: '/home/deck/.var/app/org.nixos.nix/var/nix' does not
    exist, so Nix will use '/home/deck/.local/share/nix/root' as a
    chroot store
2022-10-12 12:12:12 -07:00
Valentin Gagarin
927234cfb2
Merge pull request #6870 from amjoseph-nixpkgs/pr/doc/explain-local-remote-binary-substituter 2022-10-05 09:01:42 +02:00
Eelco Dolstra
89ca75c9f9
Merge pull request #7080 from squalus/nar-close-file
archive: check close errors when extracting nars
2022-09-30 12:58:28 +02:00
Théophane Hufschmitt
db29ddd113
Merge pull request #7078 from obsidiansystems/trustworthy-signature
"valid signature" -> "signature by a trusted key"
2022-09-24 12:52:35 +02:00
Théophane Hufschmitt
74276cb354
Merge pull request #7079 from matthewbauer/allow-untrusted-settings
Allow pass max-silent-time and build-poll-interval to daemon untrusted
2022-09-24 12:47:49 +02:00
Eelco Dolstra
c13007f012
Merge pull request #7059 from NixOS/remove-useless-ca-file-message
Remove a useless debug message in filetransfer.cc
2022-09-24 10:39:23 +02:00
John Ericson
60e23c8bae
Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Rune K. Svendsen <runesvend@gmail.com>
2022-09-23 13:57:57 -04:00
squalus
223f8dace0 archive: check close errors when extracting nars 2022-09-22 12:50:32 -07:00
Matthew Bauer
6e049ae607 Allow pass max-silent-time and build-poll-interval to daemon untrusted
These settings seem harmless, they control the same polling
functionality that timeout does, but with different behavior. Should
be safe for untrusted users to pass in.
2022-09-22 13:59:16 -05:00
John Ericson
a2a8cb10ac Dodge "trusted" vs "trustworthy" by being explicit
Hopefully this is best!
2022-09-22 14:37:52 -04:00
John Ericson
752f967c0f "valid signature" -> "trustworthy signature"
I just had a colleague get confused by the previous phrase for good
reason. "valid" sounds like an *objective* criterion, e.g. and *invalid
signature* would be one that would be trusted by no one, e.g. because it
misformatted or something.

What is actually going is that there might be a signature which is
perfectly valid to *someone else*, but not to the user, because they
don't trust the corresponding public key. This is a *subjective*
criterion, because it depends on the arbitrary and personal choice of
which public keys to trust.

I therefore think "trustworthy" is a better adjective to use. Whether
something is worthy of trust is clearly subjective, and then "trust"
within that word nicely evokes `trusted-public-keys` and friends.
2022-09-22 10:49:31 -04:00
squalus
1b595026e1 Improve durability of schema version file writes
- call close explicitly in writeFile to prevent the close exception
  from being ignored
- fsync after writing schema file to flush data to disk
- fsync schema file parent to flush metadata to disk

https://github.com/NixOS/nix/issues/7064
2022-09-19 20:13:30 -07:00
Théophane Hufschmitt
0f977bf91e
Remove a useless debug message in filetransfer.cc
Remove the `verify TLS: Nix CA file = 'blah'` message that Nix used to print when fetching anything as it's both useless (`libcurl` prints the same info in its logs) and misleading (gives the impression that a new TLS connection is being established which might not be the case because of multiplexing. See #7011 )
2022-09-19 08:42:43 +02:00
Théophane Hufschmitt
0f64bf445a
Merge pull request #6994 from agbrooks/master
Prevent tempdir from being GC-ed before addToStoreFromDump completes
2022-09-13 09:23:16 +02:00
Andrew Brooks
565d888e0f Address PR feedback on #6694 2022-09-12 11:33:23 -05:00
Andrew Brooks
84fe75a12a Keep created temp dirs inside store, but protect from GC
Implements the approach suggested by feedback on PR #6994, where
tempdir paths are created in the store (now with an exclusive lock).

As part of this work, the currently-broken and unused
`createTempDirInStore` function is updated to create an exclusive lock
on the temp directory in the store.

The GC now makes a non-blocking attempt to lock any store directories
that "look like" the temp directories created by this function, and if
it can't acquire one, ignores the directory.
2022-09-06 17:48:00 -05:00
Andrew Brooks
1f041ac54f Prevent tempdir from being GC-ed before addToStoreFromDump has renamed it
This fixes issue 6823 by placing the tempdir used in LocalStore::addToStoreFromDump
outside the Nix store, where automatic GC is no longer a concern.
2022-09-02 18:32:35 -05:00
Matthew Bauer
4894e567fb Don’t readDerivation if impure derivations feature is disabled
readDerivation is pretty slow, and while it may not be significant for
some use cases, on things like ghc-nix where we have thousands of
derivations is really slows things down.

So, this just doesn’t do the impure derivation check if the impure
derivation experimental feature is disabled. Perhaps we could cache
the result of isPure() and keep the check, but this is a quick fix to
for the slowdown introduced with impure derivations features in 2.8.0.
2022-09-02 11:46:34 -05:00
Adam Joseph
1ab913467e linkify mention of other options 2022-09-01 18:03:35 -07:00
Adam Joseph
59dc8346ca move substituter signature-checking conditions to configuration file documentation 2022-09-01 17:51:56 -07:00
Eelco Dolstra
7918adbb62
Merge pull request #6954 from winterqt/darwin-sandbox-trustd
fix(libstore): allow access to trustd on macOS
2022-08-26 11:45:00 +02:00
Winter
8e5659423e fix(libstore): allow access to trustd on macOS 2022-08-24 13:09:44 -04:00
Eelco Dolstra
56d97d4b4d Remove redundant Finally 2022-08-24 14:49:58 +02:00
Eelco Dolstra
8d906b1f3b Fix macOS build 2022-08-24 14:11:03 +02:00
Eelco Dolstra
f0358ed465 Fix a hang in nix-copy-ssh.sh
This hang for some reason didn't trigger in the Nix build, but did
running 'make installcheck' interactively. What happened:

* Store::addMultipleToStore() calls a SinkToSource object to copy a
  path, which in turn calls LegacySSHStore::narFromPath(), which
  acquires a connection.

* The SinkToSource object is not destroyed after the last bytes has
  been read, so the coroutine's stack is still alive and its
  destructors are not run. So the connection is not released.

* Then when the next path is copied, because max-connections = 1,
  LegacySSHStore::narFromPath() hangs forever waiting for a connection
  to be released.

The fix is to make sure that the source object is destroyed when we're
done with it.
2022-08-23 14:19:53 +02:00
Eelco Dolstra
f865048332 Indentation 2022-08-22 15:32:53 +02:00
Eelco Dolstra
4c96761c2b Merge remote-tracking branch 'origin/master' into parallel-nix-copy 2022-08-22 15:29:10 +02:00
Eelco Dolstra
8d84634e26
Merge pull request #6926 from rapenne-s/download_limit
add a nix.conf option to set a download speed limit
2022-08-22 14:57:19 +02:00
Eelco Dolstra
c21b1a7e67
Spelling 2022-08-22 14:14:14 +02:00
Solène Rapenne
caad87e6db
Better documentation wording
Co-authored-by: Anderson Torres <torres.anderson.85@protonmail.com>
2022-08-20 18:21:36 +02:00
Solene Rapenne
0d2bf7acf9 add a nix.conf option to set a download speed limit 2022-08-19 12:40:22 +02:00
pennae
7d934f7880 don't read outputs into memory for output rewriting
RewritingSink can handle being fed input where a reference crosses a
chunk boundary. we don't need to load the whole source into memory, and
in fact *not* loading the whole source lets nix build FODs that do not
fit into memory (eg fetchurl'ing data files larger than system memory).
2022-08-19 11:26:26 +02:00
Eelco Dolstra
53e7b7e8ac Remove warnLargeDump()
This message was unhelpful (#1184) and probably misleading since
memory is O(1) in most cases now.
2022-08-17 11:32:01 +02:00
Théophane Hufschmitt
8f3fdef1e0
Merge pull request #6850 from NinjaTrappeur/nin/build-check
Fix Nix build --check flag
2022-08-10 17:44:06 +02:00
Théophane Hufschmitt
90f9680733 Only use renameFile where needed
In most places the fallback to copying isn’t needed and can actually be
bad, so we’d rather not transparently fallback
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt
d71d9e9fbf moveFile -> renameFile
`move` tends to have this `mv` connotation of “I will copy it for you if
needs be”
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt
c2de0a232c Create a wrapper around stdlib’s rename
Directly takes some c++ strings, and gently throws an exception on error
(rather than having to inline this logic everywhere)
2022-08-03 10:27:25 +02:00
Félix Baylac-Jacqué
1467a98d4c
derivation-goal.cc: remove bmCheck custom return branch on buildDone
Once a derivation goal has been completed, we check whether or not
this goal was meant to be repeated to check its output.

An early return branch was preventing the worker to reach that repeat
code branch, hence breaking the --check command (#2619).

It seems like this early return branch is an artifact of a passed
refactoring. As far as I can tell, buildDone's main branch also
cleanup the tmp directory before returning.
2022-08-01 11:39:19 +02:00
Théophane Hufschmitt
2805439335
Merge pull request #6814 from amjoseph-nixpkgs/pr/sandbox-error-messages
local-derivation-goal.cc: improve error messages when sandboxing fails
2022-07-22 13:27:52 +02:00
Théophane Hufschmitt
7ed91d6c6a
Merge branch 'master' into parallel-nix-copy 2022-07-20 10:05:34 +02:00
Alex Wied
722de8ddcc libstore/globals.cc: Move cgroup detection to libutil 2022-07-19 16:25:53 -04:00
Alex Wied
1af5d798a4 libstore/globals.cc: Automatically set cores based on cgroup CPU limit
By default, Nix sets the "cores" setting to the number of CPUs which are
physically present on the machine. If cgroups are used to limit the CPU
and memory consumption of a large Nix build, the OOM killer may be
invoked.

For example, consider a GitLab CI pipeline which builds a large software
package. The GitLab runner spawns a container whose CPU is limited to 4
cores and whose memory is limited to 16 GiB. If the underlying machine
has 64 cores, Nix will invoke the build with -j64. In many cases, that
level of parallelism will invoke the OOM killer and the build will
completely fail.

This change sets the default value of "cores" to be
ceil(cpu_quota / cpu_period), with a fallback to
std:🧵:hardware_concurrency() if cgroups v2 is not detected.
2022-07-19 16:03:58 -04:00
Théophane Hufschmitt
56f6f3725f Don't ultimately trust the signed paths
Like the old implem did (and like you'd want it to be anyways)
2022-07-19 19:46:00 +02:00
Adam Joseph
36e1383b6b local-derivation-goal.cc: save global errno to the stack before performing tests which might clobber it 2022-07-19 03:53:20 -07:00
Adam Joseph
99fcc91f67 as requested by @thufschmitt https://github.com/NixOS/nix/pull/6814#discussion_r924275777 2022-07-19 03:33:12 -07:00
Adam Joseph
5f51539f88 change warn() to notice() 2022-07-19 03:30:52 -07:00
Adam Joseph
c8c6203c2c local-derivation-goal.cc: detect unprivileged_userns_clone failure mode
The workaround for "Some distros patch Linux" mentioned in
local-derivation-goal.cc will not help in the `--option
sandbox-fallback false` case.  To provide the user more helpful
guidance on how to get the sandbox working, let's check to see if the
`/proc` node created by the aforementioned patch is present and
configured in a way that will cause us problems.  If so, give the user
a suggestion for how to troubleshoot the problem.
2022-07-17 01:27:22 -07:00
Adam Joseph
6fc56318bf local-derivation-goal.cc: add comment re: CLONE_NEWUSER
local-derivation-goal.cc contains a comment stating that "Some distros
patch Linux to not allow unprivileged user namespaces."  Let's give a
pointer to a common version of this patch for those who want more
details about this failure mode.
2022-07-17 01:23:32 -07:00
Adam Joseph
8d35f387dc local-derivation-goal.cc: warn if failing and /proc/self/ns/user missing
This commit causes nix to `warn()` if sandbox setup has failed and
`/proc/self/ns/user` does not exist.  This is usually a sign that the
kernel was compiled without `CONFIG_USER_NS=y`, which is required for
sandboxing.
2022-07-16 19:37:27 -07:00
Adam Joseph
90830b1074 local-derivation-goal.cc: warn if failing due to max_user_namespaces==0
This commit uses `warn()` to notify the user if sandbox setup fails
with errno==EPERM and /proc/sys/user/max_user_namespaces is missing or
zero, since that is at least part of the reason why sandbox setup
failed.

Note that `echo -n 0 > /proc/sys/user/max_user_namespaces` or
equivalent at boot time has been the recommended mitigation for
several Linux LPE vulnerabilities over the past few years.  Many users
have applied this mitigation and then forgotten that they have done
so.
2022-07-16 19:30:53 -07:00
Adam Joseph
8ea3a911aa local-derivation-goal.cc: improve error messages when sandboxing fails
The failure modes for nix's sandboxing setup are pretty complicated.
When nix is unable to set up the sandbox, let's provide more detail
about what went wrong.  Specifically:

* Make sure the error message includes the word "sandbox" so the user
  knows that the failure was related to sandboxing.

* If `--option sandbox-fallback false` was provided, and removing it
  would have allowed further attempts to make progress, let the user
  know.
2022-07-16 14:56:24 -07:00
Alex Wied
b88fb50e21 fix(libstore): allow Nix to access all Rosetta 2 paths on MacOS
Fixes: #5884
2022-07-15 12:10:56 -07:00
Eelco Dolstra
3bcd7a5474 Disable auto-chroot if $NIX_STATE_DIR is set
Issue #6732.
2022-07-15 12:32:29 +02:00
John Ericson
8735f55dec Fix bug, test more, document more 2022-07-14 20:23:43 -04:00
John Ericson
6cafe308c9 Merge remote-tracking branch 'upstream/master' into indexed-store-path-outputs 2022-07-14 16:15:37 -04:00
Eelco Dolstra
ff49c75502 Disable auto-chroot if $NIX_STORE_DIR is set
Fixes #6732.
2022-07-14 17:47:09 +02:00
Eelco Dolstra
6cab528461 Don't fail if we can't create ~/.local/share/nix/root
https://hydra.nixos.org/build/182135943
2022-06-29 12:16:51 +02:00
Théophane Hufschmitt
56cf96a1b9
Merge pull request #6706 from lheckemann/cache-info-cache-invalidation
nar-info-disk-cache: refresh nix-cache-info weekly
2022-06-29 07:56:27 +02:00
Eelco Dolstra
586fa707fc
Merge pull request #6714 from edolstra/auto-chroot-store
Automatically use a chroot store if /nix doesn't exist
2022-06-25 00:03:35 +02:00