Commit graph

2124 commits

Author SHA1 Message Date
Robert Hensing
799abea0c4
Merge pull request #11330 from NixOS/packfile
libgit2, GitRepo: Write (thin) packfiles
2024-09-16 14:42:59 +02:00
Eelco Dolstra
7aa3e7e3a5 Make the NAR parser much stricter wrt field order
We really want to enforce a canonical representation since NAR
hashing/signing/deduplication depends on that.
2024-09-12 15:57:46 +02:00
Eelco Dolstra
27ec0def74 Typo 2024-09-12 15:33:55 +02:00
Eelco Dolstra
69bf9947c7 Put 'names' in the right scope 2024-09-12 15:29:54 +02:00
Eelco Dolstra
4de9587e50 Improve badArchive() 2024-09-12 15:27:55 +02:00
John Ericson
db7c868d24
Merge pull request #11378 from Mic92/nix-dir-errors
builtins.readDir: fix nix error trace on filesystem errors
2024-09-11 13:10:28 -04:00
John Ericson
193dc49097 tweak unpack channel built-in, std::filesystem::path for tarball 2024-09-11 12:29:49 -04:00
Noam Yorav-Raphael
38bfbb297c
Use envvars NIX_CACHE_HOME, NIX_CONFIG_HOME, NIX_DATA_HOME, NIX_STATE_HOME if defined (#11351) 2024-09-11 10:36:46 +00:00
Jörg Thalheim
1ca1439b1f add missing filesystem include (x86_64-darwin fix) 2024-09-10 16:45:39 +02:00
Eelco Dolstra
9c0e968843
Merge commit from fork
Fix unsafe NAR unpacking
2024-09-10 12:42:55 +02:00
Eelco Dolstra
5ca2f58798 Improve use-case-hack description slightly 2024-09-09 14:29:05 +02:00
Eelco Dolstra
5e337ee60d
Merge pull request #11412 from parkerhoyes/daemon-cgroup
Move daemon process into sub-cgroup
2024-09-09 13:49:31 +02:00
Eelco Dolstra
9fcb588dd8 RestoreSink::createDirectory(): Use append()
On macOS, `mkdir("x/')` behaves differently than `mkdir("x")` if `x` is
a dangling symlink (the formed succeed while the latter fails). So make
sure we always strip the trailing slash.
2024-09-05 22:21:53 +02:00
Eelco Dolstra
3557587381 Detect NAR directory entries that collide with another path after case-hacking
The test was made by @puckipedia.
2024-09-05 19:26:10 +02:00
Eelco Dolstra
da1ad28912 Test that nix-store --restore fails if the output already exists
This restores the behaviour from before the std::filesystem
refactorings.
2024-09-05 16:48:43 +02:00
Jörg Thalheim
5a5a010120 Revert "fix: Error on malformed URI query parameter"
This reverts commit c9f45677b5.

This now triggers on simple cases like `nix build .#nix`.
Reverting for now.
2024-09-05 15:18:16 +02:00
Jörg Thalheim
a81083d080 Revert "Update src/libutil/url.cc"
This reverts commit 9b1cefe27e.
2024-09-05 15:18:16 +02:00
Jörg Thalheim
22ba4dc78d builtins.readDir: fix nix error trace on filesystem errors
Before:

nix-env % ./src/nix/nix eval --impure --expr 'let f = builtins.readDir "/nix/store/hs3yxdq9knimwdm51gvbs4dvncz46f9d-hello-2.12.1/foo"; in f' --show-trace
error: filesystem error: directory iterator cannot open directory: No such file or directory [/nix/store/hs3yxdq9knimwdm51gvbs4dvncz46f9d-hello-2.12.1/foo]

After:

error:
       … while calling the 'readDir' builtin
         at «string»:1:9:
            1| let f = builtins.readDir "/nix/store/hs3yxdq9knimwdm51gvbs4dvncz46f9d-hello-2.12.1/foo"; in f
             |         ^

       error: reading directory '/nix/store/hs3yxdq9knimwdm51gvbs4dvncz46f9d-hello-2.12.1/foo': No such file or directory
2024-09-05 14:00:39 +02:00
Eelco Dolstra
495d32e1b8 NAR parser: Fix check for duplicate / incorrectly sorted entries
"prevName" was always empty because it was declared in the wrong scope.
2024-09-04 21:43:59 +02:00
Parker Hoyes
bd6ae2f3b9 Use getCurrentCgroup() in getMaxCPU() 2024-09-04 19:10:31 +00:00
Parker Hoyes
03484641a1 Simplify getRootCgroup()
Static local initializers are atomic in C++.
2024-09-04 18:11:16 +00:00
Parker Hoyes
46b3188045 Move daemon process into sub-cgroup
The daemon process is now moved into a new sub-cgroup called nix-daemon when the
daemon starts. This is necessary to abide by the no-processes-in-inner-nodes
rule, because the service cgroup becomes an inner node when the child cgroups
for the build are created (see LocalDerivationGoal::startBuilder()).

See #9675
2024-09-03 17:18:01 +00:00
Alyssa Ross
9cc550d652
Don't refer to public keys as secret keys in error
This constructor is used for public keys as well.
2024-08-31 16:00:11 +02:00
Robert Hensing
b89eca9aec
Merge pull request #11349 from bryanhonof/bryanhonof.check-query-for-equals
Warn on malformed URI query parameter
2024-08-28 21:56:00 +02:00
Bryan Honof
9b1cefe27e
Update src/libutil/url.cc
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-08-28 18:48:18 +02:00
Robert Hensing
d0f8a92363 Make tarball cache more interruptible 2024-08-28 17:49:22 +02:00
John Ericson
c7ec33605e Meson misc things
Meson-ify a few things, scripts, completions, etc. Should make our Meson
build complete except for docs.

Co-Authored-By: Qyriad <qyriad@qyriad.me>
Co-Authored-By: eldritch horrors <pennae@lix.systems>
2024-08-27 10:19:55 -04:00
John Ericson
a97a08411c More support for std::filepath in libnixutil
We're not replacing `Path` in exposed definitions in many cases, but
just adding alternatives. This will allow us to "top down" change `Path`
to `std::fileysystem::path`, and then we can remove the `Path`-using
utilities which will become unused.

Also add some test files which we forgot to include in the libutil unit
tests `meson.build`.

Co-Authored-By: siddhantCodes <siddhantk232@gmail.com>
2024-08-26 17:23:56 -04:00
John Ericson
dbabfc92d4 Make sure we have an execvpe on Windows too
Necessary to fix a build (that was already broken in other ways) after
PR #11021.
2024-08-26 15:45:55 -04:00
tomberek
440de80d34
Merge pull request #11021 from hercules-ci/issue-11010
Fix SSH invocation when local SHELL misbehaves
2024-08-26 10:40:51 -04:00
Robert Hensing
8bd835b7bc
Merge pull request #11278 from Mic92/close-fds
libutil: rename and optimize closeMostFDs
2024-08-26 15:51:34 +02:00
Bryan Honof
c9f45677b5
fix: Error on malformed URI query parameter
Signed-off-by: Bryan Honof <bryanhonof@gmail.com>
2024-08-23 22:04:37 +02:00
Eelco Dolstra
1facc3e35e
Merge pull request #7126 from squalus/fsync-store-paths
Add fsync-store-paths option
2024-08-22 17:45:11 +02:00
Eelco Dolstra
21a164aa03 Fix hang
Signed-off-by: Eelco Dolstra <edolstra@gmail.com>
2024-08-22 15:26:34 +02:00
Eelco Dolstra
915db74dbf
Merge pull request #11343 from DeterminateSystems/no-framedsink-threads
withFramedSink(): Don't use a thread to monitor the other side
2024-08-22 14:23:19 +02:00
Eelco Dolstra
270c8469d7 select() -> poll() for Windows compat 2024-08-21 20:54:02 +02:00
Eelco Dolstra
ce7cf4a2d3
Update src/libutil/serialise.hh
Co-authored-by: John Ericson <git@JohnEricson.me>
2024-08-21 19:50:24 +02:00
Eelco Dolstra
e049d38290 Merge remote-tracking branch 'origin/master' into fsync-store-paths 2024-08-21 16:37:21 +02:00
eldritch horrors
03b258bf97 libutil: rename and optimize closeMostFDs
this is only used to close non-stdio files in derivation sandboxes. we
may as well encode that in its name, drop the unnecessary integer set,
and use close_range to deal with the actual closing of files. not only
is this clearer, it also makes sandbox setup on linux fast by 1ms each

(cherry-picked and adapted from
c7d97802e4)

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2024-08-21 06:45:31 +02:00
John Ericson
af26fe3934
Merge pull request #11281 from siddhantk232/path-in-exec
Use `std::filesystem::path` in end executables
2024-08-20 12:35:57 -04:00
Eelco Dolstra
257470b58d Merge remote-tracking branch 'origin/master' into flip-coroutines 2024-08-20 17:19:17 +02:00
Eelco Dolstra
39daa4a0d3 withFramedSink(): Don't use a thread to monitor the other side
Since withFramedSink() is now used a lot more than in the past (for
every addToStore() variant), we were creating a lot of threads, e.g.

  nix flake show --no-eval-cache --all-systems github:NixOS/nix/afdd12be5e19c0001ff3297dea544301108d298

would create 46418 threads. While threads on Linux are cheap, this is
still substantial overhead.

So instead, just poll from FramedSink before every write whether there
are pending messages from the daemon. This could slightly increase the
latency on log messages from the daemon, but not on exceptions (which
were only synchronously checked from FramedSink anyway).

This speeds up the command above from 19.2s to 17.5s on my machine (a
9% speedup).
2024-08-19 18:15:15 +02:00
John Ericson
84ea12ad7f Fix build errors on Windows 2024-08-19 11:02:46 -04:00
Robert Hensing
c4192a6617 Add nix::execvpe 2024-08-16 15:53:30 +02:00
John Ericson
58b03ef1cd Move NIX_BIN_DIR and all logic using it to the Nix executable itself
This is because with the split packages of the Meson build, we simply
have no idea what directory the binaries will be installed in when we
build the library.

In the process of doing so, consolidate and make more sophisticated the
logic to cope with a few corner cases (e.g. `NIX_BIN_DIR` exists, but no
binaries are inside it).

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-08-12 12:29:17 -04:00
Robert Hensing
18485d2d53
Merge pull request #11188 from lf-/jade/kill-int-overflow
Ban integer overflow in the Nix language
2024-08-11 04:24:16 +02:00
John Ericson
0646b6cd61
Update comments / documentation.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-08-07 22:29:56 -05:00
John Ericson
6c861b9c51 Factor out lookupExecutable and other PATH improvments
This ended up motivating a good deal of other infra improvements in
order to get Windows right:

- `OsString` to complement `std::filesystem::path`

- env var code for working with the underlying `OsString`s

- Rename `PATHNG_LITERAL` to `OS_STR`

- `NativePathTrait` renamed to `OsPathTrait`, given a character template
  parameter until #9205 is complete.

Split `tests.cc` matching split of `util.{cc,hh}` last year.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-08-07 18:12:58 -04:00
John Ericson
9d2d4d11e6 Split tests, organize more string functions
The test split matches PR #8920, so the utility files and tests files
are once again to 1-1. The string changes continues what was started in
PR #11093.
2024-08-05 12:50:13 -04:00
Jade Lovelace
e28cb67d41 libutil: add checked arithmetic tools
This is in preparation for adding checked arithmetic to the evaluator.

Change-Id: I6e115ce8f5411feda1706624977a4dcd5efd4d13
2024-07-30 18:13:05 -07:00