Commit graph

9234 commits

Author SHA1 Message Date
Eelco Dolstra
919781cacc
Merge branch 'master' into valid_deriver_2 2023-09-01 13:35:05 +02:00
Eelco Dolstra
b88784278f
Merge pull request #8869 from hercules-ci/fix-issue-8838-pathExists-isDir
Fix #8838, pathExists: isDir when ends with `/ `
2023-09-01 13:15:54 +02:00
Eelco Dolstra
5b5f56a9d4
Merge pull request #8859 from edolstra/tarball-last-modified
Tarball trees: Propagate lastModified
2023-08-29 17:02:06 +02:00
Bryan Honof
736b9cede7
Port the flags of nix-daemon to nix daemon (#8788)
The new `nix daemon` command didn't accept the same flags that `nix-daemon` did.

* docs(daemon): clarify the daemon trust override flags
* fix: change declaration order
* docs: add examples of nix daemon usage
* Apply suggestions from code review

---------

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
Co-authored-by: tomberek <tomberek@users.noreply.github.com>
2023-08-28 13:43:34 +00:00
Robert Hensing
1e08e12d81 pathExists: isDir when endswith /
Fixes https://github.com/NixOS/nix/issues/8838
2023-08-25 17:18:37 +02:00
John Ericson
50f40ac4c0
Merge pull request #8829 from obsidiansystems/build-dynamic-derivations
Adapt scheduler to work with dynamic derivations
2023-08-25 11:13:15 -04:00
tomberek
b563ef38cc
Merge pull request #8819 from VertexA115/fix/deep-follow-paths
Fix follow path checking at depths greater than 2
2023-08-25 10:33:05 -04:00
John Ericson
cbd89568ca
Merge pull request #8814 from hercules-ci/exception-self-check
initLibUtil: Add exception handling self-check
2023-08-25 10:15:54 -04:00
John Ericson
5e3986f59c Adapt scheduler to work with dynamic derivations
To avoid dealing with an optional `drvPath` (because we might not know
it yet) everywhere, make an `CreateDerivationAndRealiseGoal`. This goal
just builds/substitutes the derivation file, and then kicks of a build
for that obtained derivation; in other words it does the chaining of
goals when the drv file is missing (as can already be the case) or
computed (new case).

This also means the `getDerivation` state can be removed from
`DerivationGoal`, which makes the `BasicDerivation` / in memory case and
`Derivation` / drv file file case closer together.

The map type is factored out for clarity, and because we will soon hvae
a second use for it (`Derivation` itself).

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-25 10:01:25 -04:00
John Ericson
692074f714 Use Worker::makeDerivationGoal less
We're about to split up `DerivationGoal` a bit. At that point
`makeDerivationGoal` will mean something more specific than it does
today. (Perhaps a future rename will make this clearer.)

On the other hand, the more public `Worker::makeGoal` function will
continue to work exactly as before. So by moving some call sites to use
that instead, we preemptively avoid issues in the next step.
2023-08-25 09:55:07 -04:00
John Ericson
1c4caef14b Throw MissingRealisation not plain Error in both resolveDerivedPath
Now we are consistent with the other `resolveDerivedPath`, and other
such functions.
2023-08-25 09:55:07 -04:00
John Ericson
2f5d3da806 Introduce OutputName and OutputNameView type aliases
Hopefully they make the code easier to understand!
2023-08-25 09:55:07 -04:00
Guillaume Girol
925a444b92 add nix-store --query --valid-derivers command
notably useful when nix-store --query --deriver returns a non-existing
path.

Co-authored-by: Felix Uhl <iFreilicht@users.noreply.github.com>
2023-08-24 11:37:24 +02:00
Eelco Dolstra
81045f243f Tarball trees: Propagate lastModified
This makes them behave consistently with GitHub/GitLab flakes.
2023-08-22 21:51:26 +02:00
Eelco Dolstra
3fcdea4ca0
Merge pull request #8836 from obsidiansystems/rm-addToSearchPath
Delete `EvalState::addToSearchPath`
2023-08-21 14:58:00 +02:00
Tom Bereknyei
c609be4072 doc: explain the . attrPath prefix notation 2023-08-19 17:19:52 -04:00
Tom Bereknyei
52248b1c27 feat: notation to refer to no attribute search prefix
An attrPath prefix of "." indicates no need to try default attrPath prefixes. For example 1nixpkgs#legacyPackages.x86_64-linux.ERROR` searches through

```
trying flake output attribute 'packages.x86_64-linux.legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute ''
trying flake output attribute 'legacyPackages.x86_64-linux.legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute 'legacyPackages.x86_64-linux'
trying flake output attribute 'legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute 'legacyPackages.x86_64-linux'
```

And there is no way to specify that one does not want the automatic
search behavior. Now one can specify
`nixpkgs#.legacyPackages.x86_64-linux.ERROR` to only refer to the rooted
attribute path without any default injection of attribute search path or
system.
2023-08-19 17:07:46 -04:00
Robert Hensing
10afcf06aa
Merge pull request #8812 from tweag/fix-clang-tidy
Fix some warnings/bugs found by clang-tidy
2023-08-19 16:00:12 +02:00
John Ericson
fe71faa920 Delete EvalState::addToSearchPath
This function is now trivial enough that it doesn't need to exist.

`EvalState` can still be initialized with a custom search path, but we
don't have a need to mutate the search path after it has been
constructed, and I don't see why we would need to in the future.

Fixes #8229
2023-08-18 14:04:33 -04:00
John Ericson
665ad4f7c5
Merge pull request #8839 from obsidiansystems/string-context-7479
Refactor Raw pattern, part of #7479
2023-08-18 13:47:01 -04:00
John Ericson
9121fed4b4 Fixing #7479
Types converted:

- `NixStringContextElem`
- `OutputsSpec`
- `ExtendedOutputsSpec`
- `DerivationOutput`
- `DerivationType`

Existing ones mostly conforming the pattern cleaned up:

- `ContentAddressMethod`
- `ContentAddressWithReferences`

The `DerivationGoal::derivationType` field had a bogus initialization,
now caught, so I made it `std::optional`. I think #8829 can make it
non-optional again because it will ensure we always have the derivation
when we construct a `DerivationGoal`.

See that issue (#7479) for details on the general goal.

`git grep 'Raw::Raw'` indicates the two types I didn't yet convert
`DerivedPath` and `BuiltPath` (and their `Single` variants) . This is
because @roberth and I (can't find issue right now...) plan on reworking
them somewhat, so I didn't want to churn them more just yet.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-08-18 11:44:00 -04:00
Cole Helbling
73696ec716 libutil: fix double-encoding of URLs
If you have a URL that needs to be percent-encoded, such as
`http://localhost:8181/test/+3d.tar.gz`, and try to lock that in a Nix
flake such as the following:

    {
      inputs.test = { url = "http://localhost:8181/test/+3d.tar.gz"; flake = false; };
      outputs = { test, ... }: {
        t = builtins.readFile test;
      };
    }

running `nix flake metadata` shows that the input URL has been
incorrectly double-encoded (despite the flake.lock being correctly
encoded only once):

    [...snip...]
    Inputs:
    └───test: http://localhost:8181/test/%252B3d.tar.gz?narHash=sha256-EFUdrtf6Rn0LWIJufrmg8q99aT3jGfLvd1//zaJEufY%3D

(Notice the `%252B`? That's just `%2B` but percent-encoded again)

With this patch, the double-encoding is gone; running `nix flake
metadata` will show the proper URL:

    [...snip...]
    Inputs:
    └───test: http://localhost:8181/test/%2B3d.tar.gz?narHash=sha256-EFUdrtf6Rn0LWIJufrmg8q99aT3jGfLvd1//zaJEufY%3D

---

As far as I can tell, this happens because Nix already percent-encodes
the URL and stores this as the value of `inputs.asdf.url`.

However, when Nix later tries to read this out of the eval state as a
string (via `getStrAttr`), it has to run it through `parseURL` again to
get the `ParsedURL` structure.

Now, this itself isn't a problem -- the true problem arises when using
`ParsedURL::to_string` later, which then _re-escapes the path_. It is
at this point that what would have been `%2B` (`+`) becomes `%252B`
(`%2B`).
2023-08-17 14:16:19 -07:00
Cole Helbling
1d7a57cfd9 libexpr/tests: test that parseFlakeRef doesn't percent-encode twice 2023-08-17 13:45:55 -07:00
Eelco Dolstra
db3bf180a5
Merge pull request #8833 from hercules-ci/jobcategory-doc
Document jobCategory()
2023-08-16 17:11:03 +02:00
Robert Hensing
d8079ee350 Document jobCategory() 2023-08-16 16:16:58 +02:00
Sergei Trofimovich
b74962c92b src/libexpr/search-path.cc: avoid out-of-bounds read on string_view
Without the change build with `-D_GLIBCXX_ASSERTIONS` exposes testsuite
assertion:

    $ gdb src/libexpr/tests/libnixexpr-tests
    Reading symbols from src/libexpr/tests/libnixexpr-tests...
    (gdb) break __glibcxx_assert_fail
    (gdb) run
    (gdb) bt
    in std::__glibcxx_assert_fail(char const*, int, char const*, char const*)@plt () from /mnt/archive/big/git/nix/src/libexpr/libnixexpr.so
    in std::basic_string_view<char, std::char_traits<char> >::operator[] (this=0x7fffffff56c0, __pos=4)
        at /nix/store/r74fw2j8rx5idb0w8s1s6ynwwgs0qmh9-gcc-14.0.0/include/c++/14.0.0/string_view:258
    in nix::SearchPath::Prefix::suffixIfPotentialMatch (this=0x7fffffff5780, path=...) at src/libexpr/search-path.cc:15
    in nix::SearchPathElem_suffixIfPotentialMatch_partialPrefix_Test::TestBody (this=0x555555a17540) at src/libexpr/tests/search-path.cc:62

As string sizes are usigned types `(a - b) > 0` effectively means
`a != b`. While the intention should be `a > b`.

The change fixes test suite pass.
2023-08-14 22:07:37 +01:00
Alex Zero
1ef8008ca7 Fix follow path checking at depths greater than 2
We need to recurse into the input tree to handle follows paths that
trarverse multiple inputs that may or may not be follow paths
themselves.
2023-08-14 18:55:46 +01:00
John Ericson
44c8d83831 Create outputOf primop.
In the Nix language, given a drv path, we should be able to construct
another string referencing to one of its output. We can do this today
with `(import drvPath).output`, but this only works for derivations we
already have.

With dynamic derivations, however, that doesn't work well because the
`drvPath` isn't yet built: importing it like would need to trigger IFD,
when the whole point of this feature is to do "dynamic build graph"
without IFD!

Instead, what we want to do is create a placeholder value with the right
string context to refer to the output of the as-yet unbuilt derivation.
A new primop in the language, analogous to `builtins.placeholder` can be
used to create one. This will achieve all the right properties. The
placeholder machinery also will match out the `outPath` attribute for CA
derivations works.

In 60b7121d2c we added that type of
placeholder, and the derived path and string holder changes necessary to
support it. Then in the previous commit we cleaned up the code
(inspiration finally hit me!) to deduplicate the code and expose exactly
what we need. Now, we can wire up the primop trivally!

Part of RFC 92: dynamic derivations (tracking issue #6316)

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-14 09:37:37 -04:00
John Ericson
e7c39ff00b Rework evaluator SingleDerivedPath infra
`EvalState::mkSingleDerivedPathString` previously contained its own
inverse (printing, rather than parsing) in order to validate what was
parsed. Now that is pulled out into its own separate function:
`EvalState::coerceToSingleDerivedPath`.

In additional that pulled out logic is deduplicated with
`EvalState::mkOutputString` via `EvalState::mkOutputStringRaw`, which is
itself deduplicated (and generalized) with
`DownstreamPlaceholder::mkOutputStringRaw`.

All these changes make the unit tests simpler.

(We would ideally write more unit tests for `mkSingleDerivedPathString`
`coerceToSingleDerivedPath` directly, but we cannot yet do that because
the IO in reading the store path won't work when the dummy store cannot
hold anything. Someday we'll have a proper in-memory store which will
work for this.)

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-14 08:44:50 -04:00
John Ericson
a04720e68c Rename optOutputPath to optStaticOutputPath
This choice of variable name makes it more clear what is going on.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-14 08:44:48 -04:00
Robert Hensing
584ff408a4
Merge pull request #8735 from obsidiansystems/defexpr
Factor out `nix-defexpr` path computation
2023-08-11 20:32:03 +02:00
Robert Hensing
c4dbb55ba9 initLibUtil: Add exception handling self-check 2023-08-11 17:25:42 +02:00
tomberek
010dc7958e
Merge pull request #8369 from obsidiansystems/inductive-derived-path
Make the Derived Path family of types inductive for dynamic derivations
2023-08-11 08:50:22 -05:00
Yorick van Pelt
e78e9a6bd1
SimpleLogger::log: fix unintended fallthrough 2023-08-11 12:05:45 +02:00
Yorick van Pelt
2e5096e4f0
FileTransfer::download: fix use-after-move
std::move(state->data) and data.empty() were called in a loop, and
could run with no other threads intervening. Accessing moved objects
is undefined behavior, and could cause a crash.
2023-08-11 12:00:31 +02:00
Yorick van Pelt
1ffb26311b
MultiCommand::toJSON: Fix use-after-move 2023-08-11 12:00:11 +02:00
Yorick van Pelt
b9b51f9579
Prevent overriding virtual methods that are called in a destructor
Virtual methods are no longer valid once the derived destructor has
run. This means the compiler is free to optimize them to be
non-virtual.

Found using clang-tidy
2023-08-11 11:58:33 +02:00
John Ericson
60b7121d2c Make the Derived Path family of types inductive for dynamic derivations
We want to be able to write down `foo.drv^bar.drv^baz`:
`foo.drv^bar.drv` is the dynamic derivation (since it is itself a
derivation output, `bar.drv` from `foo.drv`).

To that end, we create `Single{Derivation,BuiltPath}` types, that are
very similar except instead of having multiple outputs (in a set or
map), they have a single one. This is for everything to the left of the
rightmost `^`.

`NixStringContextElem` has an analogous change, and now can reuse
`SingleDerivedPath` at the top level. In fact, if we ever get rid of
`DrvDeep`, `NixStringContextElem` could be replaced with
`SingleDerivedPath` entirely!

Important note: some JSON formats have changed.

We already can *produce* dynamic derivations, but we can't refer to them
directly. Today, we can merely express building or example at the top
imperatively over time by building `foo.drv^bar.drv`, and then with a
second nix invocation doing `<result-from-first>^baz`, but this is not
declarative. The ethos of Nix of being able to write down the full plan
everything you want to do, and then execute than plan with a single
command, and for that we need the new inductive form of these types.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-08-10 00:08:32 -04:00
Peter Waller
4b1bd822ac Try to realise CA derivations during queryMissing
This enables nix to correctly report what will be fetched in the case
that everything is a cache hit.

Note however that if an intermediate build of something which is not
cached could still cause products to end up being substituted if the
intermediate build results in a CA path which is in the cache.

Fixes #8615.

Signed-off-by: Peter Waller <p@pwaller.net>
2023-08-09 20:57:04 +01:00
Théophane Hufschmitt
d00fe5f225
Merge pull request #8805 from tweag/fix-add-to-store-existing
[V2] Fix misread of source if path is already valid
2023-08-08 14:57:45 +02:00
Eelco Dolstra
5624777988
Merge pull request #8786 from Ma27/fix-why-depends-precise
nix/why-depends: fix output of `--precise`
2023-08-07 19:32:49 +02:00
Théophane Hufschmitt
4999f42a70
Merge pull request #8322 from tweag/stabilize-discard-references
Stabilize `discard-references`
2023-08-07 17:35:02 +02:00
Eelco Dolstra
eb1302670e
Merge pull request #8769 from edolstra/generalize-tarball-urls
Don't require .tar/.zip extension for tarball flakerefs
2023-08-07 17:02:17 +02:00
Théophane Hufschmitt
ad410abbe0 Stabilize discard-references
It has been there for a few releases now (landed in 2.14.0), doesn't
seem to cause any major issue and is wanted in a few places
(https://github.com/NixOS/nix/pull/7087#issuecomment-1544471346).
2023-08-07 16:53:37 +02:00
Théophane Hufschmitt
5df0f1755f
Merge pull request #8692 from obsidiansystems/add-another-xp-check
Feature gate `DownstreamPlaceholder::unknownCaOutput`
2023-08-07 13:11:44 +02:00
Simon Rainerson
31a6e10fe5 Fix misread of source if path is already valid
When receiving a stream of NARs through the ssh-ng protocol, an already
existing path would cause the NAR archive to not be read in the stream,
resulting in trying to parse the NAR as a ValidPathInfo. This results in
the error message:
    error: not an absolute path: 'nix-archive-1'

Fixes #6253

Usually this problem is avoided by running QueryValidPaths before
AddMultipleToStore, but can arise when two parallel nix processes gets
the same response from QueryValidPaths. This makes the problem more
prominent when running builds in parallel.
2023-08-07 10:27:40 +02:00
John Ericson
9113b4252b
Merge pull request #8760 from iFreilicht/fix-json-load-assertion-errors
Fix derivation load assertion errors
2023-08-06 17:07:43 -07:00
Felix Uhl
3fefc2b284 Fix derivation load assertion errors
When loading a derivation from a JSON, malformed input would trigger
cryptic "assertion failed" errors. Simply replacing calls to `operator []`
with calls to `.at()` was not enough, as this would cause json.execptions
to be printed verbatim.

Display nice error messages instead and give some indication where the
error happened.

*Before:*

```
$ echo 4 | nix derivation add
error: [json.exception.type_error.305] cannot use operator[] with a string argument with number

$ nix derivation show nixpkgs#hello | nix derivation add
Assertion failed: (it != m_value.object->end()), function operator[], file /nix/store/8h9pxgq1776ns6qi5arx08ifgnhmgl22-nlohmann_json-3.11.2/include/nlohmann/json.hpp, line 2135.

$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: [json.exception.type_error.302] type must be string, but is object

$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error: [json.exception.type_error.302] type must be object, but is string

```

*After:*

```
$ echo 4 | nix derivation add
error: Expected JSON of derivation to be of type 'object', but it is of type 'number'

$ nix derivation show nixpkgs#hello | nix derivation add
error: Expected JSON object to contain key 'name' but it doesn't

$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: Expected JSON value to be of type 'string' but it is of type 'number'

$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error:
       … while reading key 'outputs'

       error: Expected JSON value to be of type 'object' but it is of type 'string'
```
2023-08-05 01:34:30 +02:00
Maximilian Bosch
7c09104a94
nix/why-depends: fix output of --precise
I haven't checked when this was exactly introduced, but on Nix 2.16 I
realized that the additional lines inserted when using `--precise` are
completely separated from the tree:

    nix why-depends /nix/store/ccgr4faaxys39s091qridxg1947lggh4-evcxr-0.14.2 /nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0 --precise --extra-experimental-features nix-command
    /nix/store/ccgr4faaxys39s091qridxg1947lggh4-evcxr-0.14.2
        → /nix/store/lcf37pgp3rgww67v9x2990hbfwx96c1w-gcc-wrapper-12.2.0
            → /nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0
    └───bin/evcxr: …':'}.PATH=${PATH/':''/nix/store/lcf37pgp3rgww67v9x2990hbfwx96c1w-gcc-wrapper-12.2.0/bin'':'/':'}…
        └───bin/cpp: …k disable=SC2193.[[ "/nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0/bin/cpp" = *++ ]] &&…

This is apparently because `std::cout` is buffered and flushed in the
end whereas the rest of the output isn't. The fix is rather simple, just
use `logger->cout` as it's already the case for the rest of the code.

This way we also don't need to insert additional newlines in the `hits`
map since that's something the logger takes care of.

Also added a small test to make sure that the layout of this is somehow
tested to reduce the risk of further regressions here.
2023-08-04 23:11:08 +02:00
John Ericson
3b592c880a Add infra for experimental store implemenations
This is analogous to that for experimental settings and flags that we
have also added as of late.
2023-08-02 15:46:38 -04:00
John Ericson
3723363697
Merge pull request #8765 from NixLayeredStore/refactor-store-verify
More cleanups for `LocalStore::verifyPath`
2023-08-02 13:39:29 -04:00
John Ericson
9b908fa70a Factor out nix-defexpr path computation
Avoid duplicated code, and also avoid "on the fly" path construction
(which makes it harder to keep track of which paths we use).

The factored out code doesn't create the Nix state dir anymore, but this
is fine because other in nix-env and nix-channel does:

- nix-channel: Line 158 in this commit

- nix-env: Line 1407 in this commit
2023-08-02 12:54:48 -04:00
John Ericson
66550878df Add comment explaining the use of readDirectory(realStoreDir) 2023-08-02 12:46:07 -04:00
John Ericson
770d50e49c local-store verifying: Rename store to something more clear
It is not a `Store` but a `StorePathSet`.
2023-08-02 12:40:04 -04:00
Eelco Dolstra
d00469ebf9
Merge pull request #8762 from obsidiansystems/split-out-eval-settings
Move evaluator settings (type and global) to separate file/header
2023-08-02 16:54:02 +02:00
Eelco Dolstra
d9e7758f47 Don't require .tar/.zip extension for tarball flakerefs
Special-casing the file name is rather ugly, so we shouldn't do
that. So now any {file,http,https} URL is handled by
TarballInputScheme, except for non-flake inputs (i.e. inputs that have
the attribute `flake = false`).
2023-08-01 16:07:20 +02:00
John Ericson
6525265f46 LocalStore::verifyPath: Try to clarify data flow with more scopes
It was initially unclear to me which of these are temporary state for
the verify paths computation, and which of these are the results of that
computation to be used in the rest of the function. Now, it is clear,
and enforced.
2023-07-31 12:44:27 -04:00
John Ericson
2a5f5fbb17 LocalStore::verifyPath: Use StorePathSet for store local var
We don't care about non-store-paths in there (things like `.links`, are,
in fact, allowed). So let's just skip them up front and be more strongly
typed.
2023-07-31 12:44:18 -04:00
John Ericson
dcdd5fed74
Merge pull request #8754 from NixLayeredStore/refactor-store-verify
Refactor `verifyPath` to take `StorePath` instead of `Path`
2023-07-31 12:01:36 -04:00
Ben Radford
c9a87ce7ca Refactor verifyPath to take StorePath instead of Path.
This way we avoid having to convert from Path to StorePath and vice versa in
the body of verifyPath.
2023-07-31 11:17:55 -04:00
John Ericson
1570e80219 Move evaluator settings (type and global) to separate file/header 2023-07-31 10:14:15 -04:00
Robert Hensing
33d58a90c2 toJSON: Add attribute path to trace 2023-07-31 13:02:54 +02:00
Alex Ameen
2d1d81114d
Add parseFlakeRef and flakeRefToString builtins (#8670)
Over the last year or so I've run into several use cases where I need to
parse and/or serialize URLs for use by `builtins.fetchTree` or
`builtins.getFlake`, largely in order to produce _lockfile-like_ files
for lang2nix frameworks or tools which use `nix` internally to drive
builds.

I've gone through the painstaking process of emulating
`nix::FlakeRef::fromAttrs` and `nix::parseFlakeRef` several times with
mixed success; but these are difficult to create and even harder to
maintain if I hope to stay aligned with changes to the real
parser/serializer.

I understand why adding new `builtins` isn't something we want to do
flagrantly. I'm recommending this addition simply because I keep
encountering use cases where I need to parse/serialize these URIs in
`nix` expressions, and I want a reliable solution.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-07-25 17:43:33 +00:00
John Ericson
60d8dd7aea Clean up store hierarchy with IndirectRootStore
See the API doc comments for details.
2023-07-24 09:19:44 -04:00
John Ericson
13269ba93b Make RemoteStore::ConnectionHandle part of class and expose
Will need to do subclass-specific implementations in the next commit.
This isn't because there will be multiple variations of the daemon
protocol (whew!) but because different clients pick and choose different
parts to use.
2023-07-24 09:17:01 -04:00
John Ericson
0a30b07277 Move Store::Params typedef to StoreConfig::Params
This is because `StoreConfig` also uses it.
2023-07-24 09:03:44 -04:00
John Ericson
4685c9b55f
Merge pull request #8664 from ncfavier/merge-dynamic-attrs
parser: merge nested dynamic attributes
2023-07-23 11:03:38 -04:00
Naïm Favier
570a1a3ad7
parser: merge nested dynamic attributes
Fixes https://github.com/NixOS/nix/issues/7115
2023-07-21 17:14:03 +02:00
John Ericson
fe1fbdb5a1
Merge pull request #8724 from obsidiansystems/queryPartialDerivationOutputMap-evalStore
Give `queryPartialDerivationOutputMap` an `evalStore` parameter
2023-07-21 08:53:18 -04:00
Eelco Dolstra
7ac24d9525
Merge pull request #8650 from obsidiansystems/content-address-simpler
Simplify `ContentAddress`
2023-07-21 13:46:53 +02:00
John Ericson
6bc98c7fba Give queryPartialDerivationOutputMap an evalStore parameter
This makes it more useful. In general, the derivation will be in one
store, and the realisation info is in another.

This also helps us avoid duplication. See how `resolveDerivedPath` is
now simpler because it uses `queryPartialDerivationOutputMap`. In #8369
we get more flavors of derived path, and need more code to resolve them
all, and this problem only gets worse.

The fact that we need a new method to deal with the multiple dispatch is
unfortunate, but this generally relates to the fact that `Store` is a
sub-par interface, too bulky/unwieldy and conflating separate concerns.
Solving that is out of scope of this PR.

This is part of the RFC 92 work. See tracking issue #6316
2023-07-20 15:59:52 -04:00
John Ericson
f62543fe1c Remove unneeded copy
It appeared in 8eb73a8724 (by me!) without
justification.
2023-07-20 15:42:06 -04:00
John Ericson
7b30293d38 Tighten #includes: DerivedPath doesn't care about Realisation 2023-07-20 15:42:06 -04:00
Valentin Gagarin
0e4f6dfcf7 revert anchor prefix for builtin constants
the original change broke many pre-existing anchor links.

also change formatting of the constants listing slightly:
- the type should not be part of the anchor
- add highlight to the "impure only" note
2023-07-20 10:27:38 +02:00
Robert Hensing
32494cbb29
Merge pull request #7973 from fricklerhandwerk/remove-channels
remove the Channels section
2023-07-19 14:02:26 +02:00
Valentin Gagarin
2fa90e5824 add more details on CA derivations 2023-07-19 13:59:18 +02:00
Valentin Gagarin
5f37ebcf83 document all special system features and their behavior 2023-07-19 13:34:03 +02:00
Robert Hensing
eca07b2a47
Merge pull request #8315 from fricklerhandwerk/doc-system
add information on the system type string
2023-07-19 12:07:23 +02:00
Valentin Gagarin
493ddf617f reformat system-features setting documentation 2023-07-19 11:07:47 +02:00
Valentin Gagarin
feb01b22ed add links to store API documentation 2023-07-19 11:07:47 +02:00
Valentin Gagarin
6c3cd429a6 fix broken links 2023-07-19 11:01:48 +02:00
Valentin Gagarin
0779005f49 expand on the extra-platforms option 2023-07-19 10:57:37 +02:00
Valentin Gagarin
fcadac0a02 mention extra-platforms 2023-07-19 10:38:42 +02:00
Valentin Gagarin
aba32def73 fix wording
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-07-19 10:38:30 +02:00
Valentin Gagarin
32de11923e add cross-links 2023-07-19 10:37:40 +02:00
Valentin Gagarin
3763c7bb5e shorten system setting description 2023-07-19 10:37:40 +02:00
Valentin Gagarin
c8a42039ea move docs of the current system to the system setting
add information what happens when Nix itself is cross-compiled
2023-07-19 10:37:40 +02:00
Valentin Gagarin
0751c1bfc6 one line per sentence for easier review 2023-07-19 10:37:40 +02:00
Robert Hensing
0e3a7e34a0
Merge pull request #8506 from corngood/ssh-master
Pass NIX_SSHOPTS when checking for an ssh master connection.
2023-07-18 15:47:57 +02:00
Théophane Hufschmitt
a8d5bb5e7e
Merge pull request #8342 from NixLayeredStore/best-effort-supplementary-groups
Best effort supplementary groups
2023-07-17 20:58:17 +02:00
John Ericson
0f7242ff87 Test nested sandboxing, and make nicer error
We were bedeviled by sandboxing issues when working on the layered
store. The problem ended up being that when we have nested nix builds,
and the inner store is inside the build dir (e.g. store is
`/build/nix-test/$name/store`, build dir is `/build`) bind mounts
clobber each other and store paths cannot be found.

After thoroughly cleaning up `local-derivation-goal.cc`, we might be
able to make that work. But that is a lot of work. For now, we just fail
earlier with a proper error message.

Finally, test this: nested sandboxing without the problematic store dir
should work, and with should fail with the expected error message.

Co-authored-by: Dylan Green <67574902+cidkidnix@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-07-14 09:40:24 -04:00
John Ericson
caabc4f648 Feature gate DownstreamPlaceholder::unknownCaOutput
This is a part of CA derivations that we forgot to put behind the
experimental feature.

This was caught by @fricklerhandwerk in
https://github.com/NixOS/nix/pull/8369#discussion_r1258133719
2023-07-13 07:56:33 -04:00
John Ericson
2c3fb0eb33 Move BuiltPath to its own header/C++ file in libcmd
It is less important, and used less widely, than `DerivedPath`.
2023-07-12 23:01:10 -04:00
Ben Radford
a2acd23466
Update src/libstore/globals.hh
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-12 12:33:05 +01:00
Ben Radford
0309f6b5b8
Update src/libstore/globals.hh
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-12 12:32:57 +01:00
Ben Radford
b8e8dfc3e8
Say a bit about default value in setting description. 2023-07-11 11:25:49 +01:00
Ben Radford
a193ec4052
Default should depend on whether we are root. 2023-07-11 11:14:13 +01:00
Ben Radford
2b4c59dd99
Be clearer about the security implications. 2023-07-11 11:09:25 +01:00
Ben Radford
0caf28f238
Update description for require-drop-supplementary-groups. 2023-07-11 10:57:14 +01:00
Ben Radford
07dabcc90e
Always attempt setgroups but allow failure to be ignored. 2023-07-11 10:44:05 +01:00
Ben Radford
25b20b4ad2
Merge remote-tracking branch 'origin/master' into best-effort-supplementary-groups 2023-07-11 09:38:34 +01:00
John Ericson
4a880c3cc0
Merge pull request #8579 from obsidiansystems/findPath-cleanup-2
Further search path cleanups
2023-07-10 09:59:01 -04:00
Eelco Dolstra
fea7d3b1cd
Merge pull request #8681 from inclyc/libexpr/parser-move-noeffect
libexpr: remove std::move() for `basePath` in parser, it has no effect
2023-07-10 13:30:42 +02:00
Bader AlAttar
3fa0266e7a
Fix some grammar in installables doc (#8682) 2023-07-10 09:33:04 +00:00
Yingchi Long
3d74e7b811 libexpr: remove std::move() for basePath in parser, it has no effect 2023-07-10 12:02:29 +08:00
John Ericson
028b26a77f
Merge pull request #8370 from hercules-ci/fetchClosure-input-addressed
`fetchClosure`: input addressed and pure
2023-07-09 23:41:22 -04:00
John Ericson
be518e73ae Clean up SearchPath
- Better types

- Own header / C++ file pair

- Test factored out methods

- Pass parsed thing around more than strings

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-07-09 23:22:22 -04:00
John Ericson
87dcd09047 Clean up resolveSearchPathElem
We should use `std::optional<std::string>` not `std::pair<bool,
std::string>` for an optional string.
2023-07-09 23:13:30 -04:00
John Ericson
903700c5e1 Simplify ContentAddress
Whereas `ContentAddressWithReferences` is a sum type complex because different
varieties support different notions of reference, and
`ContentAddressMethod` is a nested enum to support that,
`ContentAddress` can be a simple pair of a method and hash.

`ContentAddress` does not need to be a sum type on the outside because
the choice of method doesn't effect what type of hashes we can use.

Co-Authored-By: Cale Gibbard <cgibbard@gmail.com>
2023-07-07 07:30:01 -04:00
Robert Hensing
b4b02d084f fetchClosure: Interleave the examples in the docs 2023-07-07 11:40:40 +02:00
Robert Hensing
537e8beb77
fetchClosure: Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-07 11:00:40 +02:00
Eelco Dolstra
6db66ebfc5
Merge pull request #8631 from iFreilicht/profile-list
Profile list improvements
2023-07-05 16:06:17 +02:00
Théophane Hufschmitt
82d6699976
Document the path flakeref format (#8640)
* Document the path flakeref format

Fix https://github.com/NixOS/nix/issues/8482

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-04 08:39:08 +00:00
Eelco Dolstra
a353412c43 nix profile list: Add --json flag
This just dumps the profile manifest to stdout.
2023-07-03 12:23:57 +02:00
Eelco Dolstra
b8e8f27159 Rename 'resolvedRef' to 'lockedRef'
'resolvedRef' was incorrect, since a resolved ref is one after
registry resolution, which may still be unlocked (e.g. 'nixpkgs' ->
'github:NixOS/nixpkgs').
2023-07-03 12:23:57 +02:00
Yingchi Long
3c90340fe6 libexpr: use thread_local to make the parser thread-safe
If we call `adjustLoc`, the global variable `prev_yylloc` is shared
between threads and racy.

Currently, nix itself does not concurrently parsing files, but this is
helpful for libexpr users. (The parser is thread-safe except this.)
2023-07-03 16:05:43 +08:00
Eelco Dolstra
87b82db881 nix profile list: Improve readability of the output 2023-07-02 16:17:09 +02:00
Robert Hensing
7b39a388b3
Merge pull request #8566 from inclyc/nixd/value-print-depth
libexpr: extend `Value::print` to allow limited depth
2023-07-01 20:08:52 +02:00
Robert Hensing
40052c7613 fetchClosure: Docs and error message improvements
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-30 18:23:42 +02:00
Robert Hensing
50de11d662 doc: Improve fetchClosure documentation 2023-06-30 18:23:24 +02:00
Robert Hensing
dc79636007 fetchClosure: Refactor: replace enableRewriting
A single variable is nice and self-contained.
2023-06-30 18:22:47 +02:00
Robert Hensing
5bdca46117 fetchClosure: Split into three cases 2023-06-30 18:22:47 +02:00
Robert Hensing
55888633dd makeContentAddressed: Add single path helper 2023-06-30 18:22:47 +02:00
Robert Hensing
8dca95386c fetchClosure: Disallow toPath for inputAddressed = true 2023-06-30 18:22:47 +02:00
Robert Hensing
508aa58e67 fetchClosure: Always check that inputAddressed matches the result 2023-06-30 18:22:47 +02:00
Robert Hensing
ea30f152b7 fetchClosure: Allow input addressed paths in pure mode
When explicitly requested by the caller, as suggested in the meeting
(https://github.com/NixOS/nix/pull/8090#issuecomment-1531139324)

> @edolstra: { toPath } vs { fromPath } is too implicit

I've opted for the `inputAddressed = true` requirement, because it
we did not agree on renaming the path attributes.

> @roberth: more explicit
> @edolstra: except for the direction; not immediately clear in which direction the rewriting happens

This is in fact the most explicit syntax and a bit redundant, which is
good, because that redundancy lets us deliver an error message that
reminds expression authors that CA provides a better experience to
their users.
2023-06-30 18:22:43 +02:00
Robert Hensing
7e5b6d2c45 fetchClosure: Refactor: rename toCA -> enableRewriting 2023-06-30 18:19:56 +02:00
Robert Hensing
0f6d596df5 fetchClosure: Factor out attribute hint 2023-06-30 18:19:56 +02:00
Yueh-Shun Li
eebfe989a5 linkOrCopy: Fallback upon cross-device link error (EXDEV)
Fix building derivations in local chroot store on OpenAFS,
where hard linking accross directories causes cross-device link error
(EXDEV).
2023-06-30 21:12:26 +08:00
Eelco Dolstra
a0c617348b
Merge pull request #8589 from jfroche/sign-paths-as-allowed-user
Allow to sign path as unprivileged user
2023-06-30 13:13:42 +02:00
Robert Hensing
1632f08ea2
Merge pull request #8600 from inclyc/libexpr/fix-leaking-in-stripIndentation
libexpr: fix leaking `es2` in stripIndentation (parser.y)
2023-06-29 11:31:53 +02:00
Yingchi Long
3468cbaf47 libexpr: fix leaking es2 in stripIndentation (parser.y) 2023-06-28 22:38:44 +08:00
Jean-François Roche
80c9259756 Allow to sign path as unprivileged user
User can now sign path as unprivileged/allowed user

refs #1708
2023-06-27 18:31:31 +02:00
John Ericson
22b278e011 Automatically document builtin constants
This is done in roughly the same way builtin functions are documented.

Also auto-link experimental features for primops, subsuming PR #8371.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-27 09:37:54 -04:00
John Ericson
e8067daf09 Generialize showType 2023-06-27 09:11:42 -04:00
Maximilian Bosch
559fd7ffe7
nix flake check: improve error message if overlay is not a lambda (#8582)
* nix flake check: improve error message if overlay is not a lambda

Suppose you have an overlay like this

    {
      inputs = { /* ... */ };
      outputs = { flake-utils, ... }: flake-utils.lib.eachDefaultSystem
        (system: {
          overlays.default = final: prev: {

          };
        });
    }

then `nix flake check` (correctly) fails because `overlays` are supposed
to have the structure `overlays.<name> = final: prev: exp`. However, the
error-message is a little bit counter-intuitive:

    error: overlay does not take an argument named 'final'

While one might guess where the error actually comes from because the
trace above says `… while checking the overlay 'overlays.x86_64-linux'`
this is still pretty confusing because it complains about an argument
not being named `final` even though that's evidently the case.

With this change, the error-message actually makes it clear what's
wrong:

    [ma27@carsten:~/Projects/nix/tmp]$ nix flake check --extra-experimental-features 'nix-command flakes' path:$(pwd)
    error:
           … while checking flake output 'overlays'

             at /nix/store/clgblnxx003hyrq8qkz5ab6kgqkck6qc-source/flake.nix:4:5:

                3|   outputs = { ... }: {
                4|     overlays.x86_64-linux.snens = final: prev: {
                 |     ^
                5|       kek = throw "snens";

           … while checking the overlay 'overlays.x86_64-linux'

             at /nix/store/clgblnxx003hyrq8qkz5ab6kgqkck6qc-source/flake.nix:4:5:

                3|   outputs = { ... }: {
                4|     overlays.x86_64-linux.snens = final: prev: {
                 |     ^
                5|       kek = throw "snens";

           error: overlay is not a lambda, but a set instead
2023-06-27 14:58:29 +02:00
Eelco Dolstra
8e770dac9f
Merge pull request #8576 from obsidiansystems/findPath-cleanup
Use a struct not `std::pair` for `SearchPathElem`
2023-06-26 12:37:06 +02:00
Michal Sojka
f11445952f Document builtins.fetchTree
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

Supersedes #6740
2023-06-26 00:59:06 +02:00
Théophane Hufschmitt
60f06a1714
Merge pull request #5385 from Enzime/add/dirty-rev
Add `dirtyRev` and `dirtyShortRev` to `fetchGit`
2023-06-24 14:55:31 +02:00
Michael Hoang
a7b49086c7 Add dirtyRev and dirtyShortRev to fetchGit
Fixes #4682
2023-06-24 14:17:25 +10:00
John Ericson
484290a9e0 Use a struct not std::pair for SearchPathElem
I got very confused trying to keep all the `first` and `second` straight
reading the code, *especially* as there is also another `(boolean,
string)` pair type also being used.

Named fields is much better.

There are other cleanups that we can do (for example, the existing
TODO), but we can do them later. Doing them now would just make this
harder to review.
2023-06-23 12:01:10 -04:00
Robert Hensing
fd4f03b8fd
Merge pull request #8519 from fricklerhandwerk/reword-trusted-users
reword documentation on trusted users and substituters
2023-06-23 13:08:46 +02:00
Eelco Dolstra
8350f06d6c
Merge pull request #8574 from inclyc/nixd/remove-unused-token
libexpr: remove unused token `ATTRPATH` in token declaration
2023-06-23 12:02:40 +02:00
Yingchi Long
9d8c4ac446 libexpr: remove unused token ATTRPATH in token declaration 2023-06-23 13:35:41 +08:00
John Ericson
97df060588 Better document build failure exit codes
- Improved API docs from comment

- Exit codes are for `nix-build`, not just `nix-store --release`

- Make note in tests so the magic numbers are not surprising

Picking up where #8387 left off.
2023-06-22 14:29:45 -04:00
Yingchi Long
1400fde144 libexpr: extend Value::print to allow limited depth 2023-06-22 18:28:30 +08:00
Yingchi Long
c48277c1c1 libexpr: add tests for nix::Value::print 2023-06-22 18:28:30 +08:00
John Ericson
48fe0ed554
Merge pull request #8374 from obsidiansystems/improve-path-setting
Split `OptionalPathSetting` from `PathSetting`
2023-06-21 15:40:43 -04:00
Valentin Gagarin
e91d19db5f
be more serious about security risks with trusted users 2023-06-20 12:23:53 +02:00
Eelco Dolstra
b1352700c4
Merge pull request #8552 from edolstra/fix-eagain
GC server: Clear O_NONBLOCK on the right file descriptor
2023-06-20 12:12:53 +02:00