Fixes#3898
The entire `BinaryCaches` row used to get replaced after it became
stale according to the `timestamp` column. In a concurrent scenario,
this leads to foreign key conflicts as different instances of the
in-process `state.caches` cache now differ, with the consequence that
the older process still tries to use the `id` number of the old record.
Furthermore, this phenomenon appears to have caused the cache for
actual narinfos to be erased about every week, while the default
ttl for narinfos was supposed to be 30 days.
This is slightly more accurate considering that an outdated record
may exist in the persistent cache. Possibly-outdated records are
quite relevant as they may be foreign keys to more recent information
that we want to keep, but we will not return them here.
In unprivileged podman containers, /proc is not fully visible (there
are other filesystems mounted on subdirectories of /proc). Therefore
we can't mount a new /proc in the sandbox that matches the PID
namespace of the sandbox. So this commit automatically disables
sandboxing if /proc is not fully visible.
This didn't work because sandboxing doesn't work in Docker. However,
the sandboxing check is done lazily - after clone(CLONE_NEWNS) fails,
we retry with sandboxing disabled. But at that point, we've already
done UID allocation under the assumption that user namespaces are
enabled.
So let's get rid of the "goto fallback" logic and just detect early
whether user / mount namespaces are enabled.
This commit also gets rid of a compatibility hack for some ancient
Linux kernels (<2.13).
tl;dr: With this 1 line change I was able to get a speedup of 1.5x on 1Gbit/s
wan connections by enabling zstd compression in nginx.
Also nix already supported all common compression format for http
transfer, webservers usually only enable them if they are advertised
through the Accept-Encoding header.
This pull requests makes nix advertises content compression support for
zstd, br, gzip and deflate.
It's particular useful to add transparent compression for binary caches
that serve packages from the host nix store in particular nix-serve,
nix-serve-ng and harmonia.
I tried so far gzip, brotli and zstd, whereas only zstd was able to bring
me performance improvements for 1Gbit/s WAN connections.
The following nginx configuration was used in combination with the
[zstd module](https://github.com/tokers/zstd-nginx-module) and
[harmonia](https://github.com/nix-community/harmonia/)
```nix
{
services.nginx.virtualHosts."cache.yourhost.com" = {
locations."/".extraConfig = ''
proxy_pass http://127.0.0.1:5000;
proxy_set_header Host $host;
proxy_redirect http:// https://;
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
zstd on;
zstd_types application/x-nix-archive;
'';
};
}
```
For testing I unpacked a linux kernel tarball to the nix store using
this command `nix-prefetch-url --unpack https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.1.8.tar.gz`.
Before:
```console
$ nix build && rm -rf /tmp/hello && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwbb88sw36fvah-linux-6.1.8.tar.gz'
warning: Git tree '/scratch/joerg/nix' is dirty
real 0m18,375s
user 0m2,889s
sys 0m1,558s
```
After:
```console
$ nix build && rm -rf /tmp/hello && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwb
b88sw36fvah-linux-6.1.8.tar.gz'
real 0m11,884s
user 0m4,130s
sys 0m1,439s
```
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
Update src/libstore/filetransfer.cc
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
These settings are not needed for libstore at all, they are just used by
the nix daemon *command* for authorization on unix domain sockets. My
moving them to a new configuration struct just in that file, we avoid
them leaking anywhere else.
Also, it is good to break up the mammoth `Settings` struct in general.
Issue #5638 tracks this.
The message is not changed because I do not want to regress in
convenience to the user. Just saying "this connection is not trusted"
doesn't tell them out to fix the issue. The ideal thing to do would be
to somehow parameterize `processCommand` on how the error should be
displayed, so different sorts of connections can display different
information to the user based on how authentication is performed for the
connection in question. This, however, is a good bit more work, so it is
left for the future.
This came up with me thinking about the tcp:// store (#5265). The larger
project is not TCP *per se*, but the idea that it should be possible for
something else to manage access control to services like the Nix Daemon,
and those services simply trust or trust the incoming connection as they
are told. This is a more capability-oriented way of thinking about trust
than "every server implements its own auth separately" as we are used to today.
Its very great that libstore itself already implements just this model,
and so via this refactor I basically want to "enshrine" that so it
continues to be the case.
The references set seems to have been unused since `LegacySSHStore`
references were first created in
caa5793b4a.
The method decls never were upstream, and accidentally added by me in
062533f7cd (probably due to `git rerere`).
Sorry!
This reduces the diff from #3746.
Avoid needless work and throwing away invariants.
These conversions date back to when `StorePath` was in Rust and there
were issues with it missing utility methods.
It's used as the “system” profile in a bunch of places, so better not
touch it. Besides, it doesn't hurt to keep it since it's owned by root
any way, so it doesn't have the `chown` problem that the user profiles
had and that led to wanting to move them on the client-side.
Rather than using `/nix/var/nix/{profiles,gcroots}/per-user/`, put the user
profiles and gcroots under `$XDG_DATA_DIR/nix/{profiles,gcroots}`.
This means that the daemon no longer needs to manage these paths itself
(they are fully handled client-side). In particular, it doesn’t have to
`chown` them anymore (removing one need for root).
This does change the layout of the gc-roots created by nix-env, and is
likely to break some stuff, so I’m not sure how to properly handle that.
Originally there was no `path-info.*`, then there was `path-info.hh`,
then there was `path-info.cc`, but only for new things. Moving this
stuff over makes everything consistent.
This should be a non-empty set, and so we don't want people doing this
by accident. We remove the zero-0 constructor with a little inheritance
trickery.
`DerivedPath::Built` and `DerivationGoal` were previously using a
regular set with the convention that the empty set means all outputs.
But it is easy to forget about this rule when processing those sets.
Using `OutputSpec` forces us to get it right.
This way the links are clearly within the manual (ie not absolute paths),
while allowing snippets to reference the documentation root reliably,
regardless of at which base url they're included.
macOS doesn't have user namespacing, so the gid of the builder needs
to be nixbld. The logic got "has sandboxing enabled" confused with
"has user namespaces".
Fixes#7529.
This basically reverts 6e5165b773.
It fixes errors like
sandbox-exec: <internal init prelude>:292:47: unable to open sandbox-minimal.sb: not found
when trying to run a development Nix installed in a user's home
directory.
Also, we're trying to minimize the number of installed files
to make it possible to deploy Nix as a single statically-linked
binary.
Adds a new boolean structured attribute
`outputChecks.<output>.unsafeDiscardReferences` which disables scanning
an output for runtime references.
__structuredAttrs = true;
outputChecks.out.unsafeDiscardReferences = true;
This is useful when creating filesystem images containing their own embedded Nix
store: they are self-contained blobs of data with no runtime dependencies.
Setting this attribute requires the experimental feature
`discard-references` to be enabled.
Previously addTempRoot() acquired the LocalStore state lock and waited
for the garbage collector to reply. If the garbage collector is in the
same process (as it the case with auto-GC), this would deadlock as
soon as the garbage collector thread needs the LocalStore state lock.
So now addTempRoot() uses separate Syncs for the state that it
needs. As long at the auto-GC thread doesn't call addTempRoot() (which
it shouldn't), it shouldn't deadlock.
Fixes#3224.
This also moves the file handle into its own Sync object so we're not
holding the _state while acquiring the file lock. There was no real
deadlock risk here since locking a newly created file cannot block,
but it's still a bit nicer.
This has the same goal as b13fd4c58e81b2b2b0d72caa5ce80de861622610,but
achieves it in a different way in order to not break
`nix why-depends --derivation`.
In principle, this should avoid deadlocks where two instances of Nix are
holding a shared lock on big-lock and are both waiting to get an
exclusive lock.
However, it seems like `flock(2)` is supposed to do this automatically,
so it's not clear whether this is actually where the problem comes from.
This makes 'nix develop' set the Linux personality in the same way
that the actual build does, allowing a command like 'nix develop
nix#devShells.i686-linux.default' on x86_64-linux to work correctly.
Without this, the error is lost, and it makes for a hard to debug
situation. Also remove some of the busyness inside the sqlite_open_v2
args.
The errcode returned is not the extended one. The only way to make open
return an extended code, would be to add SQLITE_OPEN_EXRESCODE to the
flags. In the future it might be worth making this change,
which would also simplify the existing SQLiteError code.
They did not include the detailed error message, losing essential
information for troubleshooting.
Example message:
warning: creating statement 'insert or rplace into NARs(cache, hashPart, namePart, url, compression, fileHash, fileSize, narHash, narSize, refs, deriver, sigs, ca, timestamp, present) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1)': at offset 10: SQL logic error, near "rplace": syntax error (in '/tmp/nix-shell.grQ6f7/nix-test/tests/binary-cache/test-home/.cache/nix/binary-cache-v6.sqlite')
It's not the best example; more important information will be in
the message for e.g. a constraint violation.
I don't see why this specific error is printed as a warning, but
that's for another commit.
Unsetting `build-users-group` (without `auto-allocate-uids` enabled)
gives the following error:
```
src/libstore/lock.cc:25: static std::unique_ptr<nix::UserLock> nix::SimpleUserLock::acquire(): Assertion `settings.buildUsersGroup != ""' failed.
```
Fix the logic in `useBuildUsers` and document the default value
for `build-users-group`.
Fix#6209
When trying to run `nix log <installable>`, try first to resolve the derivation pointed to
by `<installable>` as it is the resolved one that holds the build log.
This has a couple of shortcomings:
1. It’s expensive as it requires re-reading the derivation
2. It’s brittle because if the derivation doesn’t exist anymore or can’t
be resolved (which is the case if any one of its build inputs is missing),
then we can’t access the log anymore
However, I don’t think we can do better (at least not right now).
The alternatives I see are:
1. Copy the build log for the un-resolved derivation. But that means a
lot of duplication
2. Store the results of the resolving in the db. Which might be the best
long-term solution, but leads to a whole new class of potential
issues.
