Commit graph

10470 commits

Author SHA1 Message Date
John Ericson
c371070580 Use std::filesystem functions in more places
This makes for shorter and more portable code.

The only tricky part is catching exceptions: I just searched for near by
`catch (Error &)` or `catch (SysError &)` and adjusted them to `catch
(std::filesystem::filesystem_error &)` according to my human judgement.

Good for windows portability; will help @siddhantk232 with his GSOC
project.
2024-05-07 00:16:54 -04:00
ramboman
b4950404ba
Honor the same set of proxy environment variables (#10611)
Different parts of the project honor different sets of proxy environment
variables. With this commit all parts of the project will honor the same
set of proxy environment variables.

---------

Co-authored-by: Your Name <you@example.com>
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2024-05-06 15:39:22 -04:00
Eelco Dolstra
da3381d51f
Merge pull request #10465 from edolstra/remove-locked
Fetcher cache cleanups
2024-05-06 21:32:26 +02:00
Eelco Dolstra
c7216a416f
Update src/libfetchers/cache.hh
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-05-06 21:11:41 +02:00
John Ericson
1ad7b5451d
Merge pull request #10655 from edolstra/use-source-path
Use `SourcePath` in more places
2024-05-06 14:57:01 -04:00
John Ericson
5e189025ca Fix build failure with clang
A slight issue with feb1d10f60.
2024-05-06 13:39:21 -04:00
Eelco Dolstra
ef28c7329c Rename makeFSSourceAccessor -> getFSSourceAccessor()
This makes it clearer that it returns a shared accessor object.
2024-05-06 19:16:52 +02:00
Eelco Dolstra
eab2919119 Use SourcePath in more places
Now that SourcePath uses a SourceAccessor instead of an InputAccessor,
we can use it in function signatures instead of passing a
SourceAccessor and CanonPath separately.
2024-05-06 19:05:42 +02:00
Eelco Dolstra
2926ef0e90
Merge pull request #10652 from tweag/check-additionalSandboxProfile
libstore: check additionalSandboxProfile
2024-05-06 17:59:25 +02:00
Théophane Hufschmitt
9bd1191fcc libstore: check additionalSandboxProfile
Make sure that `extraSandboxProfile` is set before we check whether it's
empty or not (in the `sandbox=true` case).

Also adds a test case for this.

Co-Authored-By: Artemis Tosini <lix@artem.ist>
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
2024-05-06 17:37:08 +02:00
Eelco Dolstra
709cd44d3e Rename remaining instances of "InputAccessor" to "SourceAccessor" 2024-05-06 17:29:03 +02:00
Eelco Dolstra
ee2fa87a7e
Merge pull request #10642 from edolstra/remove-input-accessor
Merge InputAccessor into SourceAccessor
2024-05-06 15:48:25 +02:00
Charlie Moog
71c66de227
document store url trusted=true option behavior 2024-05-05 17:37:04 +00:00
Eelco Dolstra
ffc280f27a Formatting 2024-05-03 15:41:03 +02:00
Eelco Dolstra
20558e0462 Remove FSInputAccessor 2024-05-03 12:30:28 +02:00
Eelco Dolstra
ba5929c7be Merge InputAccessor into SourceAccessor
After the removal of the InputAccessor::fetchToStore() method, the
only remaining functionality in InputAccessor was `fingerprint` and
`getLastModified()`, and there is no reason to keep those in a
separate class.
2024-05-03 12:14:01 +02:00
John Ericson
037c8d771d Fix format errors
Fix formatting violations, update blacklist to reflect moved files.

PR #10556 passed CI before the new formating rules were added, and our
CI has the race condition of allowing old results, resulting in master
getting broken.
2024-05-02 21:42:28 -04:00
John Ericson
840267491e
Merge pull request #10633 from hercules-ci/fix-eval-state-baseEnv-gc-root
libexpr: Add missing GC root for `baseEnv`
2024-05-02 10:10:54 -04:00
John Ericson
1948ec390c
Merge pull request #10556 from nix-windows/uds-remote-on-windows
Enable the `unix://` store on Windows
2024-05-02 09:53:00 -04:00
Robert Hensing
f34b52b521 libexpr: Add missing GC root for baseEnv
This missing GC root wasn't much of a problem before, because the
heap would end up with a reference to the `baseEnv` pretty soon,
but when unit testing, the construction of `EvalState` doesn't
necessarily happen well before GC runs for the first time.

Found while unit testing the Rust bindings that currently reside
at https://github.com/nixops4/nixops4/tree/main/rust
2024-05-01 22:36:39 +02:00
Robert Hensing
e17aad23d6
Merge pull request #10555 from jlesquembre/jl/c-api_check-init
Add isValid to nix::Value
2024-05-01 16:33:01 +02:00
Eelco Dolstra
4d99d07bc9 Whitespace 2024-04-30 15:34:35 +02:00
Sarah Brofeldt
e5f509ef0b nix repl: hide progress bar during :edit 2024-04-29 10:03:34 +02:00
Eelco Dolstra
de51e5c335
Merge pull request #10570 from layus/shared_caches
Share evaluation caches across installables
2024-04-26 15:48:46 +02:00
John Ericson
4722b0c9e9
Merge pull request #10602 from haenoe/json-infra-tests-misc
Json infra tests misc
2024-04-25 16:53:51 -04:00
Eelco Dolstra
0353d6f79e
Merge pull request #10588 from hercules-ci/fix-nested-submodules
Fix fetchGit/fetchTree for nested submodules
2024-04-25 22:53:22 +02:00
Guillaume Maudoux
19cc50dcbf
fixup: Compute fingerprint only if needed 2024-04-25 00:47:46 +02:00
Guillaume Maudoux
1c4e392c64
Compute fingerprint only if needed
As per Eelco's review comments

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2024-04-25 00:44:47 +02:00
HaeNoe
943a877a6a
use default value in fetchers::PublicKey json deserialization 2024-04-24 18:23:04 +02:00
Guillaume Maudoux
a60a1f09b2 Reuse eval caches and related values when possible 2024-04-22 20:32:41 +02:00
Dylan Green
aa165301d1
Pathlocks Implementation for Windows (#10586)
Based on Volth's original port.

Co-authored-by: volth <volth@volth.com>
2024-04-22 15:08:10 +00:00
Robert Hensing
750bcaa330 Fix fetchGit nested submodules 2024-04-22 16:41:40 +02:00
Eelco Dolstra
cf1adbdd64
Merge pull request #10572 from NorfairKing/no-more-prev
Remove the 'prev' check entirely.
2024-04-22 10:53:31 +02:00
José Luis Lafuente
6acf02b32a
C API: source argument to nix_copy_value should be const 2024-04-21 22:46:18 +02:00
José Luis Lafuente
8d70db3251
C API: add check_value_[in,out] helper functions 2024-04-21 22:44:14 +02:00
José Luis Lafuente
ccad6e94e2
C API: add (un)initialized value checks 2024-04-21 22:44:14 +02:00
José Luis Lafuente
9d7dee4a8f
nix::Value: Use more descriptive names 2024-04-21 22:44:13 +02:00
José Luis Lafuente
5cc4af5231
Add isInitialized to nix::Value
Add a method to check if a value has been initialized. This helps avoid
segfaults when calling `type()`.
Useful in the context of the new C API.

Closes #10524
2024-04-21 22:44:13 +02:00
Robert Hensing
6fd2f42c2d
Merge pull request #10573 from RCoeurjoly/Rename_SearchPath
Rename SearchPath to LookupPath and searchPath to lookupPath
2024-04-21 17:33:46 +02:00
Robert Hensing
c6526fa33b
Merge pull request #10563 from hercules-ci/doc-glossary-base-directory
doc/glossary: Add base directory
2024-04-21 15:17:16 +02:00
Robert Hensing
73125e46fc doc/glossary: Add base directory 2024-04-21 14:24:33 +02:00
Tom Sydney Kerckhove
ad65a50a94 Remove the 'prev' check 2024-04-20 14:46:23 +02:00
Robert Hensing
e05b58b060 init: Add flag to avoid loading configuration 2024-04-20 01:45:04 +02:00
John Ericson
b973cd494f Enable the unix:// store on Windows
Windows now has some basic Unix Domain Socket support, see
https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/

Building `nix daemon` on Windows I've left for later, because the daemon
currently forks per connection but this is not an option on Windows. But
we can get the client part working right away.
2024-04-18 16:58:32 -04:00
John Ericson
3a3c205fa7 Use rand not random for creating GC root indirect links
I don't think fewer bits matters for this, and `rand` but not `random`
is available on Windows.
2024-04-18 16:57:11 -04:00
Robert Hensing
538eb2617a
Merge pull request #10537 from hercules-ci/c-api-nix-value-init-apply
C API: Add nix_init_apply
2024-04-18 20:09:24 +02:00
Robert Hensing
ad643cde58 C API: Add nix_init_apply
Thunks are relevant when initializing attrsets and lists, passing
arguments. This is an important way to produce them.
2024-04-18 19:13:38 +02:00
Eelco Dolstra
40499bca99
Merge pull request #10549 from alyssais/exportReferencesGraph
Fix exportReferencesGraph when given store subpath
2024-04-18 17:23:12 +02:00
John Ericson
fb9f4208ed
Don't include linux/ in #include
The linux dirs are conditionally added to the `-I` path.
2024-04-18 12:20:20 +02:00
Maximilian Bosch
ba68045187
libstore/local-derivation-goal: prohibit creating setuid/setgid binaries
With Linux kernel >=6.6 & glibc 2.39 a `fchmodat2(2)` is available that
isn't filtered away by the libseccomp sandbox.

Being able to use this to bypass that restriction has surprising results
for some builds such as lxc[1]:

> With kernel ≥6.6 and glibc 2.39, lxc's install phase uses fchmodat2,
> which slips through 9b88e52846/src/libstore/build/local-derivation-goal.cc (L1650-L1663).
> The fixupPhase then uses fchmodat, which fails.
> With older kernel or glibc, setting the suid bit fails in the
> install phase, which is not treated as fatal, and then the
> fixup phase does not try to set it again.

Please note that there are still ways to bypass this sandbox[2] and this is
mostly a fix for the breaking builds.

This change works by creating a syscall filter for the `fchmodat2`
syscall (number 452 on most systems). The problem is that glibc 2.39
and seccomp 2.5.5 are needed to have the correct syscall number available
via `__NR_fchmodat2` / `__SNR_fchmodat2`, but this flake is still on
nixpkgs 23.11. To have this change everywhere and not dependent on the
glibc this package is built against, I added a header
"fchmodat2-compat.hh" that sets the syscall number based on the
architecture. On most platforms its 452 according to glibc with a few
exceptions:

    $ rg --pcre2 'define __NR_fchmodat2 (?!452)'
    sysdeps/unix/sysv/linux/x86_64/x32/arch-syscall.h
    58:#define __NR_fchmodat2 1073742276

    sysdeps/unix/sysv/linux/mips/mips64/n32/arch-syscall.h
    67:#define __NR_fchmodat2 6452

    sysdeps/unix/sysv/linux/mips/mips64/n64/arch-syscall.h
    62:#define __NR_fchmodat2 5452

    sysdeps/unix/sysv/linux/mips/mips32/arch-syscall.h
    70:#define __NR_fchmodat2 4452

    sysdeps/unix/sysv/linux/alpha/arch-syscall.h
    59:#define __NR_fchmodat2 562

I tested the change by adding the diff below as patch to
`pkgs/tools/package-management/nix/common.nix` & then built a VM from
the following config using my dirty nixpkgs master:

    {
      vm = { pkgs, ... }: {
        virtualisation.writableStore = true;
        virtualisation.memorySize = 8192;
        virtualisation.diskSize = 12 * 1024;
        nix.package = pkgs.nixVersions.nix_2_21;
      };
    }

The original issue can be triggered via

    nix build -L github:nixos/nixpkgs/d6dc19adbda4fd92fe9a332327a8113eaa843894#lxc \
      --extra-experimental-features 'nix-command flakes'

however the problem disappears with this patch applied.

Closes #10424

[1] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2031073804
[2] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2030844251
2024-04-18 12:20:20 +02:00
John Ericson
9c815db366 file-descriptor.hh: Avoid some Cism for better C++isms
- `reinterpret_cast` not C-style cast
- `using` not `typedef`
2024-04-17 19:55:40 -04:00
John Ericson
6fa3656a32 Make a few commands that were Unix-only no longer
Also clean up some more linux-specific (`setPersonality`) code in
alignment with recent best practices.
2024-04-17 16:20:56 -04:00
Alyssa Ross
0774e8ba33
Fix exportReferencesGraph when given store subpath
With Nix 2.3, it was possible to pass a subpath of a store path to
exportReferencesGraph:

	with import <nixpkgs> {};

	let
	  hello = writeShellScriptBin "hello" ''
	    echo ${toString builtins.currentTime}
	  '';
	in

	writeClosure [ "${hello}/bin/hello" ]

This regressed with Nix 2.4, with a very confusing error message, that
presumably indicates it was unintentional:

	error: path '/nix/store/3gl7kgjr4pwf03f0x70dgx9ln3bhl7zc-hello/bin/hello' is not in the Nix store
2024-04-17 21:51:59 +02:00
John Ericson
21d9412ddc Improve local-overlay docs in a few ways
In response to https://discourse.nixos.org/t/super-colliding-nix-stores/28462/24
2024-04-17 15:28:19 -04:00
a-kenji
5fd28eeba4 docs: fix wiki link
Link to the official wiki.
2024-04-17 19:34:22 +02:00
John Ericson
1063aa502a Relax store path canonicalization 2024-04-17 12:26:10 -04:00
John Ericson
b529d91902 Prefix -DNIX_ paths to be windows-complient for windows
This is a hacky solution, but it will do for now.
2024-04-17 12:26:10 -04:00
John Ericson
8433027e35 Build a minimized Nix with MinGW
At this point many features are stripped out, but this works:

- Can run libnix{util,store,expr} unit tests
- Can run some Nix commands

Co-Authored-By volth <volth@volth.com>
Co-Authored-By Brian McKenna <brian@brianmckenna.org>
2024-04-17 12:26:10 -04:00
John Ericson
2248a3f545 Create no-op Window pathlocks implementation
This keeps the call sites simple, eventually this should be filled in.
2024-04-17 11:48:14 -04:00
John Ericson
05b9dac754
Fix friend struct/class mismatch warning 2024-04-17 15:47:07 +00:00
Eelco Dolstra
6a3ecdaa39 Merge remote-tracking branch 'origin/master' into finish-value 2024-04-17 16:02:44 +02:00
Eelco Dolstra
d53ad516e8
Merge pull request #10517 from edolstra/path-display
Add missing setPathDisplay() calls
2024-04-17 15:36:08 +02:00
José Luis Lafuente
c75b143b6c
C API: nix_get_string now accepts a callback to return the value 2024-04-16 22:36:39 +02:00
Eelco Dolstra
74e4bc9b1d
Merge pull request #10486 from tweag/jl/c-api_function-pointer
C API: Safer function pointer casting
2024-04-16 18:56:08 +02:00
Eelco Dolstra
6892c9803c GitInputScheme: Fix path display for workdirs and submodules 2024-04-16 16:42:47 +02:00
Eelco Dolstra
79363b2273 MountedInputAccessor, FilteringInputAccessor: Respect the path display prefix/suffix
This was causing Git paths not to be rendered correctly.
2024-04-16 16:42:41 +02:00
Eelco Dolstra
6df58a0891 MercurialInputScheme: Improve path display 2024-04-16 14:14:50 +02:00
Eelco Dolstra
fa01db9626 StorePathAccessor: Fix path display
Set the prefix to the store path to fix messages like "copying '/' to
the store" to "copying '/nix/store/bla' to the store".
2024-04-16 14:14:26 +02:00
Théophane Hufschmitt
d2a07a96ba
Merge pull request #10467 from edolstra/nix-shell-symlink
nix shell: Handle output paths that are symlinks
2024-04-16 12:31:14 +02:00
John Ericson
65cc237b3a
Merge pull request #10482 from tweag/fix-symlink-in-sandbox
Fix the access of symlinks to host files in the sandbox
2024-04-15 09:29:00 -04:00
Eelco Dolstra
cceae30aaf Combine the domain and key arguments into a single value for convenience 2024-04-15 13:03:27 +02:00
Eelco Dolstra
aad11f4496 Simplify the fetcher cache 2024-04-15 13:03:27 +02:00
Eelco Dolstra
d084c1cb41 Remove the "locked" flag from the fetcher cache
This also reworks the Mercurial fetcher (which was still using the
old cache interface) to have two distinct cache mappings:

* A ref-to-rev mapping, which is store-independent.
* A rev-to-store-path mapping.
2024-04-15 13:03:27 +02:00
José Luis Lafuente
774e7213e8
C API: Use nix_get_string_callback typedef 2024-04-15 12:05:57 +02:00
HaeNoe
e3fed2ebcf
update fetchers::PublicKey json (de)serialization 2024-04-14 22:42:48 +02:00
HaeNoe
bb939d3772
change implementation of optionalValueAt 2024-04-14 22:35:51 +02:00
José Luis Lafuente
76444a3958
C API: proper ifdef endif indentation 2024-04-14 16:18:32 +02:00
Roland Coeurjoly
62ce139e3f No need to undef now that there is no collision 2024-04-13 23:34:01 +02:00
John Ericson
eff90af498 Slight refactors in preparation for #10480
Code operating on store objects (including creating them) should, in
general, use `ContentAddressMethod` rather than `FileIngestionMethod`.

See also dfc876531f which included some
similar refactors.
2024-04-13 12:13:47 -04:00
Roland Coeurjoly
40a6a9fdb8 Rename SearchPath to LookupPath and searchPath to lookupPath 2024-04-13 17:35:15 +02:00
John Ericson
aa438b8fba
Merge pull request #10414 from edolstra/remove-downloadFile-locked
downloadFile(): Remove the "locked" (aka "immutable") flag
2024-04-12 17:23:53 -04:00
José Luis Lafuente
01bad63c72
C API: Safer function pointer casting
See https://github.com/NixOS/nix/pull/8699#discussion_r1554312181

Casting a function pointer to `void*` is undefined behavior in the C
spec, since there are platforms with different sizes for these two kinds
of pointers. A safe alternative might be `void (*callback)()`
2024-04-12 21:41:15 +02:00
John Ericson
95ae12b607 docs: Refer to the glossary with @docroot@ instead of ..
These unweildy relative paths probably predate the `@docroot@`
mechanism.
2024-04-12 11:46:47 -04:00
Valentin Gagarin
13c2005e7d
add intermediate variables and clarifying comments (#9274)
* add intermediate variables and clarifying comments

Co-authored-by: Alexander Groleau <alex@proof.construction>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-04-12 17:43:35 +02:00
Théophane Hufschmitt
acbb1523c1 Fix the access of symlinks to host files in the sandbox
https://github.com/NixOS/nix/pull/10456 fixed the addition of symlink
store paths to the sandbox, but also made it so that the hardcoded
sandbox paths (like `/etc/hosts`) were now bind-mounted without
following the possible symlinks. This made these files unreadable if
there were symlinks (because the sandbox would now contain a symlink to
an unreachable file rather than the underlying file).
In particular, this broke FOD derivations on NixOS as `/etc/hosts` is a
symlink there.

Fix that by canonicalizing all these hardcoded sandbox paths before
adding them to the sandbox.
2024-04-12 15:57:53 +02:00
John Ericson
5b9cb8b372
Merge pull request #10412 from roberth/c-string-context
C API: Add `nix_string_realise`
2024-04-11 12:07:31 -04:00
Robert Hensing
f2522d4ecd libexpr-c: Add nix_store_path_name 2024-04-11 17:39:21 +02:00
Robert Hensing
1233bcde37 libstore-c: Add nix_store_path_clone 2024-04-11 17:39:21 +02:00
Robert Hensing
48808a5320 tests/unit/libexpr: Enable nix_store_realise test, and add docs 2024-04-11 17:39:21 +02:00
Robert Hensing
ed13cf05a2 build-hook: Allow empty
Like always declining; local builds only, as can be inferred from the
docs. (Not worth spending too many words on this pretty obvious
behavior, I think. Also, plans to remove it? https://github.com/NixOS/nix/issues/1221)
2024-04-11 17:39:16 +02:00
Bouke van der Bijl
1a76ca4161 Set the origin instead of hacking in the URL resolving 2024-04-11 14:22:12 +02:00
Bouke van der Bijl
1f73de2629 git fetcher: relax absolute URL check of resolveSubmoduleUrl
This matches up the behavior with the internals of libgit2

Fixes #9979
2024-04-11 14:22:12 +02:00
Théophane Hufschmitt
da1e977bf4
Merge pull request #10456 from NixOS/fixpermdeniedbind
Fix adding symlink to the sandbox paths
2024-04-11 13:41:49 +02:00
Théophane Hufschmitt
def00d7b52
Merge pull request #10455 from tie/structured-attrs-get-env
Do not rely on $stdenv/setup to set output variables
2024-04-11 06:48:36 +02:00
Eelco Dolstra
85b9f4ef4f nix shell: Handle output paths that are symlinks
This requires moving resolveSymlinks() into SourceAccessor. Also, it
requires LocalStoreAccessor::maybeLstat() to work on parents of the
store (to avoid an error like "/nix is not in the store").

Fixes #10375.
2024-04-10 23:49:19 +02:00
Ivan Trubach
664532c533 Do not rely on $stdenv/setup to set output variables
Instead of relying on setup script to set output variables when
structured attributes are enabled, iterate over the values of an
outputs associative array.

See also
374fa3532e/pkgs/stdenv/generic/setup.sh (L23-L26)
2024-04-10 19:50:19 +03:00
Théophane Hufschmitt
a268c0de71
Merge pull request #10413 from Ma27/path-info-all-binary-cache
path-info: print correct path when using `nix path-info --store file://... --all --json`
2024-04-10 15:25:50 +02:00
Théophane Hufschmitt
ae4737294e doBind: Use our own lstat wrapper
Doesn't change much, but brings a bit more consistency to the code
2024-04-10 15:20:10 +02:00
Théophane Hufschmitt
913db9f738 Fix permission denied when building symlink derivation which points to a symlink out of the store
Bind-mounting symlinks is apparently not possible, which is why the
thing was failing.

Fortunately, symlinks are small, so we can fallback to copy them at no cost.

Fix https://github.com/NixOS/nix/issues/9579

Co-authored-by: Artturin <Artturin@artturin.com>
2024-04-10 15:19:18 +02:00
John Ericson
93d68e18e5 Make outputHashAlgo accept "nar", stay in sync
Now that we have a few things identifying content address methods by
name, we should be consistent about it.

Move up the `parseHashAlgoOpt` for tidiness too.

Discussed this change for consistency's sake as part of #8876

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2024-04-09 16:17:23 -04:00