Commit graph

435 commits

Author SHA1 Message Date
Eelco Dolstra
fa99ef6a87 getMaxCPU(): Lower verbosity level for ignored exceptions
Fixes #7268.
2022-12-02 15:03:40 +01:00
Eelco Dolstra
1e6a5d1ff6 Clean up cgroup handling in getMaxCPU()
Also, don't assume in LocalDerivationGoal that cgroups are mounted on
/sys/fs/cgroup.
2022-12-02 12:59:13 +01:00
squalus
1b595026e1 Improve durability of schema version file writes
- call close explicitly in writeFile to prevent the close exception
  from being ignored
- fsync after writing schema file to flush data to disk
- fsync schema file parent to flush metadata to disk

https://github.com/NixOS/nix/issues/7064
2022-09-19 20:13:30 -07:00
Théophane Hufschmitt
5192bb093a Merge remote-tracking branch 'upstream/master' into fix-mv-in-different-filesystems 2022-08-08 15:42:56 +02:00
Dave Nicponski
cb6794a0d9 Do not spam logs if the owned-homedir check results in a noop 2022-08-07 10:13:11 -04:00
Théophane Hufschmitt
1ba5b3e001 Make moveFile more atomic
Rather than directly copying the source to its dest, copy it first to a
temporary location, and eventually move that temporary.
That way, the move is at least atomic from the point-of-view of the destination
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt
8119390abc Move some fs-related functions to their own file
Unclutter `util.cc` a bit
2022-08-03 10:27:25 +02:00
Alex Wied
722de8ddcc libstore/globals.cc: Move cgroup detection to libutil 2022-07-19 16:25:53 -04:00
Eelco Dolstra
1e55ee2961 getSelfExe(): Support macOS 2022-06-23 01:32:46 +02:00
Eelco Dolstra
d3176ce076 Fix build-remote in nix-static
'build-remote' is now executed via /proc/self/exe so it always works.
2022-06-23 01:32:46 +02:00
Eelco Dolstra
f6cf644e5f Style 2022-06-22 15:35:52 +02:00
Dave Nicponski
ca2be509b9 Verify $HOME is owned by current user in getHome(), if it exists.
Useful because a default `sudo` on darwin doesn't clear `$HOME`, so things like `sudo nix-channel --list`
will surprisingly return the USER'S channels, rather than `root`'s.

Other counterintuitive outcomes can be seen in this PR description:
  https://github.com/NixOS/nix/pull/6622
2022-06-17 17:42:08 -04:00
Robert Hensing
c156155239 createUnixDomainSocket: listen(unix, 5 -> 100)
This solves the error

    error: cannot connect to socket at '/nix/var/nix/daemon-socket/socket': Connection refused

on build farm systems that are loaded but operating normally.

I've seen this happen on an M1 mac running a loaded hercules-ci-agent.
Hercules CI uses multiple worker processes, which may connect to
the Nix daemon around the same time. It's not unthinkable that
the Nix daemon listening process isn't scheduled until after 6
workers try to connect, especially on a system under load with
many workers.

Is the increase safe?

The number is the number of connections that the kernel will buffer
while the listening process hasn't `accept`-ed them yet.
It did not - and will not - restrict the total number of daemon
forks that a client can create.

History

The number 5 has remained unchanged since the introduction in
nix-worker with 0130ef88ea in 2006.
2022-05-26 11:24:04 +02:00
Eelco Dolstra
3e87c8e62b Move json stuff out of util.cc 2022-05-04 11:22:06 +02:00
Alain Zscheile
1385b20078
Get rid of most .at calls (#6393)
Use one of `get` or `getOr` instead which will either return a null-pointer (with a nicer error message) or a default value when the key is missing.
2022-05-04 07:44:32 +02:00
John Ericson
3c220442ff Merge remote-tracking branch 'upstream/master' into fix-url-format 2022-04-20 16:53:16 +00:00
Eelco Dolstra
51712bf012
Merge pull request #6128 from ncfavier/fix-completion
Shell completion improvements
2022-04-19 13:45:33 +02:00
John Ericson
75b62e5260 Avoid fmt when constructor already does it
There is a correctnes issue here, but #3724 will fix that. This is just
a cleanup for brevity's sake.
2022-04-19 01:44:11 +00:00
Eelco Dolstra
8b1e328d5d
Merge pull request #6348 from cole-h/fix-restoring-mount-namespace
libutil: Fix restoring mount namespace
2022-04-07 18:15:33 +02:00
Eelco Dolstra
c0ad86f681
Merge pull request #6366 from danpls/base64-reserve
libutil: Reserve memory when en/decoding base64
2022-04-05 23:20:33 +02:00
Daniel Pauls
513652d594 tokenizeString: Fix semantic mistake
`string_view::find_first_not_of(...)` and
`string_view::find_first_of(...)` return `string_view::npos` on error
not `string::npos`.
2022-04-05 22:33:03 +02:00
Daniel Pauls
1fa0393479 libutil: Reserve memory when en/decoding base64
The size of the output when encoding to and decoding from base64 is
(roughly) known so we can allocate it in advance to prevent
reallocation.
2022-04-05 21:30:50 +02:00
Cole Helbling
56009b2639 libutil: don't save cwd fd, use path instead
Saving the cwd fd didn't actually work well -- prior to this commit, the
following would happen:

    : ~/w/vc/nix ; doas outputs/out/bin/nix --experimental-features 'nix-command flakes' run nixpkgs#coreutils -- --coreutils-prog=pwd
    pwd: couldn't find directory entry in ‘../../../..’ with matching i-node
    : ~/w/vc/nix ; doas outputs/out/bin/nix --experimental-features 'nix-command flakes' develop -c pwd
    pwd: couldn't find directory entry in ‘../../../..’ with matching i-node
2022-04-04 10:28:08 -07:00
Cole Helbling
10b9c1b2b2 libutil: save cwd fd in restoreMountNamespace
This doesn't work very well (maybe I'm misunderstanding the desired
implementation):

    : ~/w/vc/nix ; doas outputs/out/bin/nix --experimental-features 'nix-command flakes' develop -c pwd
    pwd: couldn't find directory entry in ‘../../../..’ with matching i-node
2022-04-04 10:28:00 -07:00
Cole Helbling
f89b0f7846 libutil: try restoring the cwd from fdSavedCwd 2022-04-04 08:33:59 -07:00
Cole Helbling
e135d223f6 libutil: save fd to cwd instead of cwd itself 2022-04-04 08:32:45 -07:00
Cole Helbling
e5b70d47aa libutil: cleanup savedCwd logic
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2022-04-04 08:20:11 -07:00
Cole Helbling
2a45cf54e4 libutil: Properly guard self-allocating getcwd on GNU
It's a GNU extension, as pointed out by pennae.
2022-04-01 12:20:34 -07:00
Cole Helbling
7f5caaa7c0 libutil: Don't use std::filesystem
Just in case making libutil depend on std::filesystem is unacceptable,
here is the non-filesystem approach.
2022-04-01 10:24:31 -07:00
aszlig
435848cef1 libutil: Fix restoring mount namespace
I regularly pass around simple scripts by using nix-shell as the script
interpreter, eg. like this:

    #!/usr/bin/env nix-shell
    #!nix-shell -p dd_rescue coreutils bash -i bash

While this works most of the time, I recently had one occasion where it
would not and the above would result in the following:

    $ sudo ./myscript.sh
    bash: ./myscript.sh: No such file or directory

Note the "sudo" here, because this error only occurs if we're root.

The reason for the latter is because running Nix as root means that we
can directly access the store, which makes sure we use a filesystem
namespace to make the store writable. XXX - REWORD!

So when stracing the process, I stumbled on the following sequence:

    openat(AT_FDCWD, "/proc/self/ns/mnt", O_RDONLY) = 3
    unshare(CLONE_NEWNS)                            = 0
    ... later ...
    getcwd("/the/real/cwd", 4096)                   = 14
    setns(3, CLONE_NEWNS)                           = 0
    getcwd("/", 4096)                               = 2

In the whole strace output there are no calls to chdir() whatsoever, so
I decided to look into the kernel source to see what else could change
directories and found this[1]:

    /* Update the pwd and root */
    set_fs_pwd(fs, &root);
    set_fs_root(fs, &root);

The set_fs_pwd() call is roughly equivalent to a chdir() syscall and
this is called when the setns() syscall is invoked[2].

[1]: b14ffae378/fs/namespace.c (L4659)
[2]: b14ffae378/kernel/nsproxy.c (L346)
2022-04-01 09:30:52 -07:00
Eelco Dolstra
28309352d9 replaceEnv(): Pass newEnv by reference 2022-03-31 10:39:53 +02:00
Sergei Trofimovich
6b1872312f nix store gc: account for auto-optimised store
Before the change on a system with `auto-optimise-store = true`:

    $ nix store gc --verbose --max 1

deleted all the paths instead of one path (we requested 1 byte limit).

It happens because every file in `auto-optimise-store = true` has at
least 2 links: file itself and a link in /nix/store/.links/ directory.

The change conservatively assumes that any file that has one (as before)
or two links (assume auto-potimise mode) will free space.

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
2022-03-13 07:24:48 +00:00
pennae
8e2eaaaf69 make Finally more local
no need for function<> with c++17 deduction. this saves allocations and virtual
calls, but has the same semantics otherwise. not going through function has the
side effect of giving compilers more insight into the cleanup code, so we need a
few local warning disables.
2022-03-09 00:16:50 +01:00
Naïm Favier
55c6906701
Perform tilde expansion when completing flake fragments
Allows completing `nix build ~/flake#<Tab>`.
We can implement expansion for `~user` later if needed.
Not using wordexp(3) since that expands way too much.
2022-03-07 12:01:54 +01:00
Naïm Favier
5461ff532d
Make completeDir follow symlinks
Allows completing `nix why-depends /run/cur<Tab>` to /run/current-system
2022-03-07 12:01:54 +01:00
Eelco Dolstra
6097790863 Fix segfault in headerCallback()
https://hydra.nixos.org/build/168594664
2022-03-03 11:11:16 +01:00
Anders Kaseorg
b5cd3e2d5c filterANSIEscapes: Ignore BEL character
GCC is not as good at music as it seems to think it is.  Fixes #4546.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2022-03-01 15:35:42 -08:00
Eelco Dolstra
df552ff53e Remove std::string alias (for real this time)
Also use std::string_view in a few more places.
2022-02-25 16:13:02 +01:00
Eelco Dolstra
1ac2664472 Remove std::vector alias 2022-02-21 16:32:34 +01:00
Eelco Dolstra
fe9afb65bb Remove std::set alias 2022-02-21 16:28:23 +01:00
Eelco Dolstra
e2422c4582
Merge pull request #6052 from hercules-ci/issue-3294-fix-interruptCallback-deadlock
Fix deadlocked nix-daemon zombies on darwin #3294
2022-02-21 16:21:45 +01:00
Robert Hensing
ddb6740e7d triggerInterrupt: Refactor to use break 2022-02-21 15:43:43 +01:00
Sergei Trofimovich
3ec02deb20 Make sure no exceptions leave ignoreException()
I noticed that occasional Ctrl-C leaves *.lock files around.
`nix-daemon`'s journal logs contained crashes like:

    nix-daemon[30416]: terminate called after throwing an instance of 'nix::SysError'
    nix-daemon[30416]:   what():  error: writing to file: Broken pipe

And core dump backtraces pointed at `teriminate()` call from
destructors:

    ...
    _Unwind_Resume ()
    nix::ignoreException() ()
    nix::LocalDerivationGoal::~LocalDerivationGoal()
    ...

    void ignoreException()
    {
        try {
            throw;
        } catch (std::exception & e) {
            printError("error (ignored): %1%", e.what());
        }
    }

The crashes happen when client side closes early and printError() throws
an IO error.

The change wraps `ignoreException()` into blanket `try { ... } catch (...) {}`.

Closes: https://github.com/NixOS/nix/issues/6046
2022-02-07 16:20:56 +00:00
Robert Hensing
c3b942e0fc Don't hold interruptCallbacks lock during interrupt handling
This changes the representation of the interrupt callback list to
be safe to use during interrupt handling.

Holding a lock while executing arbitrary functions is something to
avoid in general, because of the risk of deadlock.

Such a deadlock occurs in https://github.com/NixOS/nix/issues/3294
where ~CurlDownloader tries to deregister its interrupt callback.

This happens during what seems to be a triggerInterrupt() by the
daemon connection's MonitorFdHup thread. This bit I can not confirm
based on the stack trace though; it's based on reading the code,
so no absolute certainty, but a smoking gun nonetheless.
2022-02-06 13:53:28 +01:00
Eelco Dolstra
cd35bbbeef Merge branch 'more-stringviews' of https://github.com/pennae/nix 2022-02-02 12:38:37 +01:00
Will Dietz
a0357abda7 canonPath: fix missing slash when resolving links
Fixes #6017
2022-01-29 16:32:27 -06:00
pennae
0d7fae6a57 convert a for more utilities to string_view 2022-01-27 17:15:43 +01:00
Eelco Dolstra
4af88a4c91
Merge pull request #5906 from pennae/primops-optimization
optimize primops and utils by caching more and copying less
2022-01-18 19:43:28 +01:00
Eelco Dolstra
3157028fc1
Merge pull request #5932 from edolstra/remove-shared-strings
Remove shared strings
2022-01-18 11:14:18 +01:00
Eelco Dolstra
d62a9390fc Get rid of std::shared_ptr<std::string> and ref<std::string>
These were needed back in the pre-C++11 era because we didn't have
move semantics. But now we do.
2022-01-18 11:12:30 +01:00