Commit graph

3781 commits

Author SHA1 Message Date
John Ericson
fe9cbe838c Create Derivation::fromJSON
And test, of course
2023-04-07 08:34:58 -04:00
matthewcroughan
9207f94582 Add Store::isTrustedClient()
This function returns true or false depending on whether the Nix client
is trusted or not. Mostly relevant when speaking to a remote store with
a daemon.

We include this information in `nix ping store` and `nix doctor`

Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
2023-04-06 19:59:57 -04:00
Robert Hensing
9470ee877d Allow open switch-enum in 5 places 2023-04-03 18:45:20 +02:00
Robert Hensing
3dac4c7874 Add explicit case statements where -Wswitch-enum would report them 2023-04-03 18:17:32 +02:00
Robert Hensing
62cacc371f Fix BuildResult.toString() for NoSubstituters 2023-04-03 18:17:30 +02:00
John Ericson
9383520b75 Move querySubstitutablePathInfos from LocalStore to Store
The code is not local-store-specific, so we should share it with all
stores. More uniform behavior is better, and a less store-specific
functionality is more maintainable.

This fixes a FIXME added in f73d911628 by @edolstra himself.
2023-04-02 20:32:01 -04:00
John Ericson
5d56e2daf7 Add comparison methods for content addresses 2023-04-01 16:52:23 -04:00
John Ericson
eeecfacb43 Merge branch 'path-info' into ca-drv-exotic 2023-04-01 16:40:32 -04:00
John Ericson
1f8e1edba9 Merge commit 'a6d00a7bfb18e7ec461ac1d54203cc628aca5c66' into ca-drv-exotic 2023-04-01 15:15:38 -04:00
John Ericson
f7f44f7c96 Merge commit 'aa99005004bccc9be506a2a2f162f78bad4bcb41' into ca-drv-exotic 2023-04-01 15:15:32 -04:00
John Ericson
2ef99cd104
Merge pull request #8122 from bjornfor/use-nix-store-l-unless-experimental-enabled
Don't recommend 'nix log' unless experimental feature is enabled
2023-04-01 14:50:16 -04:00
John Ericson
f4ab297b31 Ensure all headers have #pragma once and are in API docs
`///@file` makes them show up in the internal API dos. A tiny few were
missing `#pragma once`.
2023-03-31 23:19:44 -04:00
John Ericson
abd5e7dec0 Extend internal API docs, part 2
Picking up from #8111.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-03-31 23:01:40 -04:00
Patrick Jackson
2b905d1d35 Replace unnecessary Sync<uint64_t> with std::atomic<uint64_t> 2023-03-31 17:56:25 -07:00
Théophane Hufschmitt
e32ca3cf16
Merge pull request #8018 from tweag/ssh-password-prompt
SSH: don't erase password prompt if it is displayed
2023-03-31 12:06:10 +02:00
Patrick Jackson
804180ad52 Only lock once 2023-03-30 18:06:27 -07:00
Patrick Jackson
309753ebb5 Fix data race in copyPaths 2023-03-30 16:59:34 -07:00
John Ericson
c51d554c93 Use "raw pattern" for content address types
We weren't because this ancient PR predated it!

This is actually a new version of the pattern which addresses some
issues identified in #7479.
2023-03-30 17:12:49 -04:00
John Ericson
a6d00a7bfb Fix warning 2023-03-30 16:29:13 -04:00
John Ericson
aa99005004 Merge remote-tracking branch 'upstream/master' into path-info
Also improve content-address.hh API docs.
2023-03-30 16:28:53 -04:00
John Ericson
8cab89a94b Convert a bunch of comments in headers to Doxygen documentation
The internal API docs now contain more useful information.

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-03-29 23:22:07 -04:00
Bjørn Forsman
74d94b2a13 Don't recommend 'nix log' unless experimental feature is enabled
This fixes the issue that `nix-build`, without experimental feature
'nix-command' enabled, recommends the experimental CLI `nix log` to view
build logs. Now it'll recommend the stable `nix-store -l` CLI instead.

Fixes https://github.com/NixOS/nix/issues/8118
2023-03-28 21:02:08 +02:00
Eelco Dolstra
237587bc0a
Merge pull request #8084 from edolstra/store-docs
Auto-generate store documentation
2023-03-27 15:46:18 +02:00
John Ericson
1d539aa287
Merge pull request #8073 from tweag/fix-root-channels-location
Fix root channels location
2023-03-27 09:37:53 -04:00
Eelco Dolstra
5a0f5b5c34
Merge pull request #8062 from edolstra/ssl-cert-file
Add a setting for configuring the SSL certificates file
2023-03-27 15:37:15 +02:00
John Ericson
570829d67e
Merge pull request #7609 from obsidiansystems/hide-experimental-settings
Hide experimental settings
2023-03-27 09:19:29 -04:00
Eelco Dolstra
a519436e9d ssl-cert-file: Use lists 2023-03-27 14:08:39 +02:00
Théophane Hufschmitt
128994509f Make some comments in profiles.hh doxygen-enabled
These are proper documentation of the API, so they deserve to be here
2023-03-27 10:02:10 +02:00
Eelco Dolstra
168b6021c5 Tweaks 2023-03-23 15:32:59 +01:00
Eelco Dolstra
b79df9dedc Register LocalStore to ensure it's included in the manual 2023-03-23 15:23:13 +01:00
Eelco Dolstra
161f4b0dea Document store URLs 2023-03-23 10:38:48 +01:00
Théophane Hufschmitt
fb67c1a1fb Factor out the generation of the profile/channel directory
Make sure that all the code paths use the same one, and that the
backwards-compatibility measures are probably in place when needed
2023-03-23 10:22:42 +01:00
Eelco Dolstra
abc449bc30
Update src/libstore/local-store.md
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-03-23 10:12:57 +01:00
Eelco Dolstra
80f0b8d307 Fix SSHStore 2023-03-23 09:35:35 +01:00
Eelco Dolstra
b134546f08 Fix clang build 2023-03-23 09:11:15 +01:00
Eelco Dolstra
5691bac202 Improve store setting descriptions / Markdown formatting 2023-03-22 14:23:36 +01:00
Alexander Bantyev
5291a82cd9
SSHMaster: pause logger to show password prompt
Pause logger before starting SSH connections, and resume it after the
connection is established, so that SSH password prompts are not erased
by the logger's updates.
2023-03-22 09:45:08 +04:00
Eelco Dolstra
233b063b08 Move store docs to 'nix help-stores'
Why not 'nix help stores'? Well, 'nix help <arg>' already means 'show
help on the "arg" subcommand'.
2023-03-21 14:37:09 +01:00
Eelco Dolstra
9eb53bbf17 Support per-store Markdown documentation 2023-03-21 14:03:40 +01:00
Eelco Dolstra
8d6d59cb1b nix store --help: Include store type documentation 2023-03-21 14:03:40 +01:00
Eelco Dolstra
515662ad70 Cleanup 2023-03-20 18:06:08 +01:00
Eelco Dolstra
16db8dc96f Open slave pseudoterminal before CLONE_NEWUSER
Otherwise, when running as root and user namespaces are enabled,
opening the slave fails with EPERM.

Fixes "opening pseudoterminal slave: Permission denied" followed by a
hang (https://hydra.nixos.org/build/213104244), and "error: getting
sandbox mount namespace: No such file or directory" (#8072), which
happens when the child fails very quickly and consequently reading
/proc/<child>/ns fails.
2023-03-20 17:58:36 +01:00
John Ericson
296831f641 Move enabled experimental feature to libutil struct
This is needed in subsequent commits to allow the settings and CLI args
infrastructure itself to read this setting.
2023-03-20 11:05:22 -04:00
Eelco Dolstra
e53e5c38d4 Add a setting for configuring the SSL certificates file
This provides a platform-independent way to configure the SSL
certificates file in the Nix daemon. Previously we provided
instructions for overriding the environment variable in launchd, but
that obviously doesn't work with systemd. Now we can just tell users
to add

  ssl-cert-file = /etc/ssl/my-certificate-bundle.crt

to their nix.conf.
2023-03-17 18:32:18 +01:00
Eelco Dolstra
7f46ebcf90
Merge pull request #8049 from edolstra/unexpected-eof
Fix "unexpected EOF" errors on macOS
2023-03-16 16:13:42 +01:00
Linus Heckemann
af4cbdafe7 LocalDerivationGoal: set NIX_ATTRS_*_FILE correctly for sandboxed builds 2023-03-16 15:26:52 +01:00
Eelco Dolstra
5eb8bfd0f1 Remove a variable in LocalDerivationGoal 2023-03-15 10:59:10 +01:00
Eelco Dolstra
6029c763c2 Change builderOut from Pipe to AutoCloseFD 2023-03-15 10:59:10 +01:00
Eelco Dolstra
c536e00c9d Move pseudoterminal slave open to child
Hopefully this fixes "unexpected EOF" failures on macOS
(#3137, #3605, #7242, #7702).

The problem appears to be that under some circumstances, macOS
discards the output written to the slave side of the
pseudoterminal. Hence the parent never sees the "sandbox initialized"
message from the child, even though it succeeded. The conditions are:

* The child finishes very quickly. That's why this bug is likely to
  trigger in nix-env tests, since that uses a builtin builder. Adding
  a short sleep before the child exits makes the problem go away.

* The parent has closed its duplicate of the slave file
  descriptor. This shouldn't matter, since the child has a duplicate
  as well, but it does. E.g. moving the close to the bottom of
  startBuilder() makes the problem go away. However, that's not a
  solution because it would make Nix hang if the child dies before
  sending the "sandbox initialized" message.

* The system is under high load. E.g. "make installcheck -j16" makes
  the issue pretty reproducible, while it's very rare under "make
  installcheck -j1".

As a fix/workaround, we now open the pseudoterminal slave in the
child, rather than the parent. This removes the second condition
(i.e. the parent no longer needs to close the slave fd) and I haven't
been able to reproduce the "unexpected EOF" with this.
2023-03-15 10:58:37 +01:00
Eelco Dolstra
19326ac297 Simplify commonChildInit() 2023-03-14 18:58:12 +01:00
Théophane Hufschmitt
99af56cd0d
Merge pull request #8015 from tweag/progress-during-nix-copy
Display progress when running copyPaths (nix copy)
2023-03-10 16:23:22 +01:00
Eelco Dolstra
4f3a4b732c Make findRuntimeRoots() more resilient to disappearing processes
I saw this random failure in https://hydra.nixos.org/build/211811692:

  error: opening /proc/15307/fd: No such process

while running nix-collect-garbage in a readfile-context.sh. This is
because we're not handling ESRCH errors reading /proc/<pid>/fd. So
just move the read inside the try/catch where we do handle it.
2023-03-09 16:44:51 +01:00
Alexander Bantyev
4bef2016a1
Display progress when running copyPaths (nix copy)
`nix copy` operations did not show progress. This is quite confusing.

Add a `progressSink` which displays the progress during `copyPaths`,
pretty much copied from `copyStorePath`.

Fixes https://github.com/NixOS/nix/issues/8000
2023-03-09 16:03:48 +04:00
Eelco Dolstra
7bfed34367 Fix crash/hang with CA derivations
The curl download can outlive DrvOutputSubstitutionGoal (if some other
error occurs), so at shutdown setting the promise to an exception will
fail because 'this' is no longer valid in the callback. This can
manifest itself as a segfault, "corrupted double-linked list" or hang.
2023-03-08 11:09:15 +01:00
Eelco Dolstra
0507462c06
Merge pull request #7918 from zimbatm/fix-empty-nix-store-env
treat empty NIX_STORE_DIR env vars as unset
2023-03-03 13:49:56 +01:00
Eelco Dolstra
f0908f592c
Merge pull request #7942 from edolstra/remove-format
Remove FormatOrString and remaining uses of format()
2023-03-03 10:22:11 +01:00
Eelco Dolstra
29abc8e764 Remove FormatOrString and remaining uses of format() 2023-03-02 15:57:54 +01:00
Théophane Hufschmitt
1a825b6919 Log the decompressed body in case of http errors
Don't show the users the raw (possibly compressed) error message as
everyone isn't able to decompress brotli in their brain.
2023-03-02 14:59:15 +01:00
Eelco Dolstra
767974f411
Merge pull request #7924 from mkenigs/valid
Always set valid in path-info --json output
2023-03-02 09:58:20 +01:00
Félix Baylac Jacqué
25300c0ecd
Treat empty env var paths as unset
We make sure the env var paths are actually set (ie. not "") before
sending them to the canonicalization function. If we forget to do so,
the user will end up facing a puzzled failed assertion internal error.

We issue a non-failing warning as a stop-gap measure. We could want to
revisit this to issue a detailed failing error message in the future.
2023-03-01 20:50:07 +01:00
John Ericson
b7f01a82a9 Remove dead code RemoteStore::sameMachine
It has been dead code since 9747ea84b4.
2023-03-01 11:10:30 -05:00
Valentin Gagarin
306e5c5ce5
Merge pull request #7788 from bobvanderlinden/pr-improve-nix-profile-install-error
Improve error on conflict for nix profile install
2023-03-01 11:48:43 +01:00
Matthew Kenigsberg
f86f2b973f Always set valid in path-info --json output
Currently the valid key is only present when the path is invalid, which
makes checking path validity more complex than it should be. With this
change, the valid key can always be used to check if a path is valid
2023-02-28 16:04:41 -07:00
John Ericson
5abd643c6d Merge branch 'path-info' into ca-drv-exotic 2023-02-28 12:46:00 -05:00
John Ericson
d381248ec0 No inheritance for TextInfo and FixedOutputInfo 2023-02-28 12:14:11 -05:00
John Ericson
85bb865d20 Revert "Remove some designated initializers"
This reverts commit ee9eb83a84.
2023-02-28 11:57:20 -05:00
John Ericson
123b11ff83 Clarify store path grammar and improve comment on makeType 2023-02-28 11:49:13 -05:00
John Ericson
d12f57c2c0 Merge remote-tracking branch 'upstream/master' into path-info 2023-02-28 11:34:34 -05:00
John Ericson
c36b584f8e Fix typo in the method name 2023-02-28 11:34:18 -05:00
Eelco Dolstra
a4a5d828e2
Merge pull request #7793 from layus/interrupt_downloads
Check interrupts even when download stalled
2023-02-28 13:29:29 +01:00
Bob van der Linden
3113b13df9
buildenv: throw BuildEnvFileConflictError with more context
At the moment an Error is thrown that only holds an error message
regarding `nix-env` and `nix profile`. These tools make use of
builtins.buildEnv, but buildEnv is also used in other places. These
places are unrelated to Nix profiles, so the error shouldn't mention
these tools.

This generic error is now BuildEnvFileConflictError, which holds more
contextual information about the files that were conflicting while
building the environment.
2023-02-27 21:39:34 +01:00
Valentin Gagarin
fd0e21475c add information on the build-hook setting
add a warning that you probably never want to change this.
2023-02-27 16:27:56 +01:00
Théophane Hufschmitt
532c70f531
Merge pull request #7856 from yorickvP/fix-nsswitch
Wait with making /etc unwritable until after build env setup
2023-02-21 09:39:10 +01:00
John Ericson
208c8d326d Derivation::toJSON: fix bug!
When I moved this code from the binary to libnixstore #7863, I forgot to
display the environment variables!
2023-02-20 17:38:57 -05:00
John Ericson
7998686c00 Test toJSON of DerivationOutput and Derivation 2023-02-19 11:12:12 -05:00
John Ericson
cd583362ec Move Derivation toJSON logic to libnixstore 2023-02-19 10:06:40 -05:00
Yorick van Pelt
bbba49b3e4
Wait with making /etc unwritable until after build env setup
This fixes /etc/nsswitch.conf
2023-02-17 16:34:45 +01:00
Yorick van Pelt
49fd72a903
Make /etc writability conditional on uid-range feature 2023-02-14 13:55:41 +01:00
Yorick van Pelt
db41f74af3
Don't allow writing to /etc 2023-02-14 12:03:34 +01:00
Eelco Dolstra
c205d10c66
Merge pull request #7616 from hercules-ci/fix-3898
Fix foreign key error inserting into NARs #3898
2023-02-13 13:02:19 +01:00
Eelco Dolstra
b3d29e80e0
Merge pull request #7805 from edolstra/c++2a
Fix building with GCC 9
2023-02-10 20:41:29 +01:00
Eelco Dolstra
67451d8ed7
Merge pull request #7802 from edolstra/fix-7783
Fix PID namespace support check
2023-02-10 20:41:13 +01:00
Eelco Dolstra
5978ceb271 Fix building with GCC 9
Nixpkgs on aarch64-linux is currently stuck on GCC 9
(https://github.com/NixOS/nixpkgs/issues/208412) and using gcc11Stdenv
doesn't work either.

So use c++2a instead of c++20 for now. Unfortunately this means we
can't use some C++20 features for now (like std::span).
2023-02-10 18:38:57 +01:00
Théophane Hufschmitt
9ebbe35817
Merge pull request #5588 from tweag/balsoft/xdg
Follow XDG Base Directory standard
2023-02-10 18:05:50 +01:00
Alexander Bantyev
2384d36083
A setting to follow XDG Base Directory standard
XDG Base Directory is a standard for locations for storing various
files. Nix has a few files which seem to fit in the standard, but
currently use a custom location directly in the user's ~, polluting
it:

- ~/.nix-profile
- ~/.nix-defexpr
- ~/.nix-channels

This commit adds a config option (use-xdg-base-directories) to follow
the XDG spec and instead use the following locations:

- $XDG_STATE_HOME/nix/profile
- $XDG_STATE_HOME/nix/defexpr
- $XDG_STATE_HOME/nix/channels

If $XDG_STATE_HOME is not set, it is assumed to be ~/.local/state.

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
Co-authored-by: Tim Fenney <kodekata@gmail.com>
Co-authored-by: pasqui23 <pasqui23@users.noreply.github.com>
Co-authored-by: Artturin <Artturin@artturin.com>
Co-authored-by: John Ericson <Ericson2314@Yahoo.com>
2023-02-10 20:14:06 +04:00
Eelco Dolstra
3e6e34cdf5 LocalDerivationGoal::startBuilder(): Use startProcess() to clone 2023-02-10 14:44:25 +01:00
Eelco Dolstra
f094ba7386 Simplify the PID namespace check: just try to mount /proc
Fixes #7783.
2023-02-10 14:38:14 +01:00
Guillaume Maudoux
e6ad8e8440 nit: cleaner diff 2023-02-10 00:57:56 +01:00
Guillaume Maudoux
aa18404ecb Flush data when download ends 2023-02-10 00:54:29 +01:00
Eelco Dolstra
e46429f674
Merge pull request #7712 from Mic92/advertise-compressions
advertise transport encoding in http transfers to
2023-02-09 17:15:25 +01:00
Guillaume Maudoux
78fea899e0 Check interupts even when download stalled 2023-02-09 13:56:50 +01:00
Robert Hensing
19b495a48a NarInfoDiskCache: Also test id consistency with updated fields
And clarify test
2023-02-07 23:34:36 +01:00
Robert Hensing
fb94d5cabd NarInfoDiskCache: Keep BinaryCache.id stable and improve test
Fixes #3898

The entire `BinaryCaches` row used to get replaced after it became
stale according to the `timestamp` column. In a concurrent scenario,
this leads to foreign key conflicts as different instances of the
in-process `state.caches` cache now differ, with the consequence that
the older process still tries to use the `id` number of the old record.

Furthermore, this phenomenon appears to have caused the cache for
actual narinfos to be erased about every week, while the default
ttl for narinfos was supposed to be 30 days.
2023-02-07 23:34:36 +01:00
Robert Hensing
2ceece3ef3 NarInfoDiskCache: Prepare reproducer for #3898 2023-02-07 23:34:36 +01:00
Robert Hensing
79f62d2dda NarInfoDiskCacheImpl: Make dbPath a parameter
This allows testing with a clean database.
2023-02-07 23:34:36 +01:00
Robert Hensing
29f0b196f4 NarInfoDiskCache: Rename cacheExists -> upToDateCacheExists
This is slightly more accurate considering that an outdated record
may exist in the persistent cache. Possibly-outdated records are
quite relevant as they may be foreign keys to more recent information
that we want to keep, but we will not return them here.
2023-02-07 23:34:36 +01:00
Robert Hensing
8a0ef5d58e sqlite.cc: Add SQL tracing
Set environment variable NIX_DEBUG_SQLITE_TRACES=1 to log all sql statements.
2023-02-07 23:34:36 +01:00
Eelco Dolstra
c5c0617d6f Mention --no-sandbox if sandboxing is unsupported 2023-02-07 22:59:46 +01:00
Eelco Dolstra
bc1d9fd8b5 Check whether we can use PID namespaces
In unprivileged podman containers, /proc is not fully visible (there
are other filesystems mounted on subdirectories of /proc). Therefore
we can't mount a new /proc in the sandbox that matches the PID
namespace of the sandbox. So this commit automatically disables
sandboxing if /proc is not fully visible.
2023-02-07 22:51:53 +01:00
Eelco Dolstra
fb2f7f5dcc Fix auto-uid-allocation in Docker containers
This didn't work because sandboxing doesn't work in Docker. However,
the sandboxing check is done lazily - after clone(CLONE_NEWNS) fails,
we retry with sandboxing disabled. But at that point, we've already
done UID allocation under the assumption that user namespaces are
enabled.

So let's get rid of the "goto fallback" logic and just detect early
whether user / mount namespaces are enabled.

This commit also gets rid of a compatibility hack for some ancient
Linux kernels (<2.13).
2023-02-07 22:51:53 +01:00