Commit graph

1598 commits

Author SHA1 Message Date
John Ericson
8a416e819c Move RestoreSink to fs-sink.cc
Co-Authored-By: Matthew Bauer <mjbauer95@gmail.com>
Co-Authored-By: Carlo Nucera <carlo.nucera@protonmail.com>
2023-09-22 09:10:32 -04:00
John Ericson
9d6114313b Move ParseSink to its own header
We will soon add a new implemenation so the one for NARs in `archive.cc`
isn't the only one.

Co-Authored-By: Matthew Bauer <mjbauer95@gmail.com>
Co-Authored-By: Carlo Nucera <carlo.nucera@protonmail.com>
2023-09-22 09:10:32 -04:00
John Ericson
39ba81a4eb Improve internal API docs for two file hashing functions
Co-Authored-By: Matthew Bauer <mjbauer95@gmail.com>
Co-Authored-By: Carlo Nucera <carlo.nucera@protonmail.com>
2023-09-22 09:10:32 -04:00
Théophane Hufschmitt
f89b84919c
Merge pull request #8931 from fricklerhandwerk/nix3-config-options
do not show configuration override flags for each command
2023-09-22 14:13:51 +02:00
John Ericson
fd61799c0b
Merge pull request #9006 from fricklerhandwerk/fix-links
fix links to configuration settings
2023-09-21 11:58:07 -04:00
Valentin Gagarin
1a412a8d78 fix typo in docstring 2023-09-21 11:38:13 +02:00
Valentin Gagarin
4685260a77 fix links to configuration settings 2023-09-21 02:52:58 +02:00
maralorn
682dbcab9a
Print parent activity field in json log 2023-09-09 18:01:10 +02:00
John Ericson
7ad66cb3ef Allow dynamic derivation deps in inputDrvs
We use the same nested map representation we used for goals, again in
order to save space. We might someday want to combine with `inputDrvs`,
by doing `V = bool` instead of `V = std::set<OutputName>`, but we are
not doing that yet for sake of a smaller diff.

The ATerm format for Derivations also needs to be extended, in addition
to the in-memory format. To accomodate this, we added a new basic
versioning scheme, so old versions of Nix will get nice errors. (And
going forward, if the ATerm format changes again the errors will be even
better.)

`parsedStrings`, an internal function used as part of parsing
derivations in A-Term format, used to consume the final `]` but expect
the initial `[` to already be consumed. This made for what looked like
unbalanced brackets at callsites, which was confusing. Now it consumes
both which is hopefully less confusing.

As part of testing, we also created a unit test for the A-Term format for
regular non-experimental derivations too.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

Apply suggestions from code review

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-07 10:39:37 -04:00
Valentin Gagarin
09eb7f1ef6 do not show configuration override flags for each command
this removes a lot of noise from the web search, which precludes finding
the actual documentation.

some configuration settings have enough documentation to warrant
individual pages, so the alternative of including full setting
documentation in each command page doesn't make much sense here.

this change technically means that the command line flags to override
settings are "invisible", and not exported as JSON. this may or may not
be desirable. a more explicit approach would be adding a `hidden` field
to the flag's JSON output, but would also require adjusting
post-processing of that JSON for manual rendering.
2023-09-07 02:02:27 +02:00
Eelco Dolstra
5c95b32c46 Fix warning 'catching polymorphic type by value' 2023-09-01 14:49:49 +02:00
John Ericson
cbd89568ca
Merge pull request #8814 from hercules-ci/exception-self-check
initLibUtil: Add exception handling self-check
2023-08-25 10:15:54 -04:00
Robert Hensing
10afcf06aa
Merge pull request #8812 from tweag/fix-clang-tidy
Fix some warnings/bugs found by clang-tidy
2023-08-19 16:00:12 +02:00
John Ericson
665ad4f7c5
Merge pull request #8839 from obsidiansystems/string-context-7479
Refactor Raw pattern, part of #7479
2023-08-18 13:47:01 -04:00
John Ericson
9121fed4b4 Fixing #7479
Types converted:

- `NixStringContextElem`
- `OutputsSpec`
- `ExtendedOutputsSpec`
- `DerivationOutput`
- `DerivationType`

Existing ones mostly conforming the pattern cleaned up:

- `ContentAddressMethod`
- `ContentAddressWithReferences`

The `DerivationGoal::derivationType` field had a bogus initialization,
now caught, so I made it `std::optional`. I think #8829 can make it
non-optional again because it will ensure we always have the derivation
when we construct a `DerivationGoal`.

See that issue (#7479) for details on the general goal.

`git grep 'Raw::Raw'` indicates the two types I didn't yet convert
`DerivedPath` and `BuiltPath` (and their `Single` variants) . This is
because @roberth and I (can't find issue right now...) plan on reworking
them somewhat, so I didn't want to churn them more just yet.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-08-18 11:44:00 -04:00
Cole Helbling
73696ec716 libutil: fix double-encoding of URLs
If you have a URL that needs to be percent-encoded, such as
`http://localhost:8181/test/+3d.tar.gz`, and try to lock that in a Nix
flake such as the following:

    {
      inputs.test = { url = "http://localhost:8181/test/+3d.tar.gz"; flake = false; };
      outputs = { test, ... }: {
        t = builtins.readFile test;
      };
    }

running `nix flake metadata` shows that the input URL has been
incorrectly double-encoded (despite the flake.lock being correctly
encoded only once):

    [...snip...]
    Inputs:
    └───test: http://localhost:8181/test/%252B3d.tar.gz?narHash=sha256-EFUdrtf6Rn0LWIJufrmg8q99aT3jGfLvd1//zaJEufY%3D

(Notice the `%252B`? That's just `%2B` but percent-encoded again)

With this patch, the double-encoding is gone; running `nix flake
metadata` will show the proper URL:

    [...snip...]
    Inputs:
    └───test: http://localhost:8181/test/%2B3d.tar.gz?narHash=sha256-EFUdrtf6Rn0LWIJufrmg8q99aT3jGfLvd1//zaJEufY%3D

---

As far as I can tell, this happens because Nix already percent-encodes
the URL and stores this as the value of `inputs.asdf.url`.

However, when Nix later tries to read this out of the eval state as a
string (via `getStrAttr`), it has to run it through `parseURL` again to
get the `ParsedURL` structure.

Now, this itself isn't a problem -- the true problem arises when using
`ParsedURL::to_string` later, which then _re-escapes the path_. It is
at this point that what would have been `%2B` (`+`) becomes `%252B`
(`%2B`).
2023-08-17 14:16:19 -07:00
Robert Hensing
c4dbb55ba9 initLibUtil: Add exception handling self-check 2023-08-11 17:25:42 +02:00
Yorick van Pelt
e78e9a6bd1
SimpleLogger::log: fix unintended fallthrough 2023-08-11 12:05:45 +02:00
Yorick van Pelt
1ffb26311b
MultiCommand::toJSON: Fix use-after-move 2023-08-11 12:00:11 +02:00
John Ericson
60b7121d2c Make the Derived Path family of types inductive for dynamic derivations
We want to be able to write down `foo.drv^bar.drv^baz`:
`foo.drv^bar.drv` is the dynamic derivation (since it is itself a
derivation output, `bar.drv` from `foo.drv`).

To that end, we create `Single{Derivation,BuiltPath}` types, that are
very similar except instead of having multiple outputs (in a set or
map), they have a single one. This is for everything to the left of the
rightmost `^`.

`NixStringContextElem` has an analogous change, and now can reuse
`SingleDerivedPath` at the top level. In fact, if we ever get rid of
`DrvDeep`, `NixStringContextElem` could be replaced with
`SingleDerivedPath` entirely!

Important note: some JSON formats have changed.

We already can *produce* dynamic derivations, but we can't refer to them
directly. Today, we can merely express building or example at the top
imperatively over time by building `foo.drv^bar.drv`, and then with a
second nix invocation doing `<result-from-first>^baz`, but this is not
declarative. The ethos of Nix of being able to write down the full plan
everything you want to do, and then execute than plan with a single
command, and for that we need the new inductive form of these types.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-08-10 00:08:32 -04:00
Théophane Hufschmitt
ad410abbe0 Stabilize discard-references
It has been there for a few releases now (landed in 2.14.0), doesn't
seem to cause any major issue and is wanted in a few places
(https://github.com/NixOS/nix/pull/7087#issuecomment-1544471346).
2023-08-07 16:53:37 +02:00
Felix Uhl
3fefc2b284 Fix derivation load assertion errors
When loading a derivation from a JSON, malformed input would trigger
cryptic "assertion failed" errors. Simply replacing calls to `operator []`
with calls to `.at()` was not enough, as this would cause json.execptions
to be printed verbatim.

Display nice error messages instead and give some indication where the
error happened.

*Before:*

```
$ echo 4 | nix derivation add
error: [json.exception.type_error.305] cannot use operator[] with a string argument with number

$ nix derivation show nixpkgs#hello | nix derivation add
Assertion failed: (it != m_value.object->end()), function operator[], file /nix/store/8h9pxgq1776ns6qi5arx08ifgnhmgl22-nlohmann_json-3.11.2/include/nlohmann/json.hpp, line 2135.

$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: [json.exception.type_error.302] type must be string, but is object

$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error: [json.exception.type_error.302] type must be object, but is string

```

*After:*

```
$ echo 4 | nix derivation add
error: Expected JSON of derivation to be of type 'object', but it is of type 'number'

$ nix derivation show nixpkgs#hello | nix derivation add
error: Expected JSON object to contain key 'name' but it doesn't

$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: Expected JSON value to be of type 'string' but it is of type 'number'

$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error:
       … while reading key 'outputs'

       error: Expected JSON value to be of type 'object' but it is of type 'string'
```
2023-08-05 01:34:30 +02:00
John Ericson
48fe0ed554
Merge pull request #8374 from obsidiansystems/improve-path-setting
Split `OptionalPathSetting` from `PathSetting`
2023-06-21 15:40:43 -04:00
Ben Radford
6ae35534b7
Support opening local store with database on read-only filesystem (#8356)
Previously it was not possible to open a local store when its database is on a read-only filesystem. Obviously a store on a read-only filesystem cannot be modified, but it would still be useful to be able to query it.

This change adds a new read-only setting to LocalStore. When set to true, Nix will skip operations that fail when the database is on a read-only filesystem (acquiring big-lock, schema migration, etc), and the store database will be opened in immutable mode.

Co-authored-by: Ben Radford <benradf@users.noreply.github.com>
Co-authored-by: cidkidnix <cidkidnix@protonmail.com>
Co-authored-by: Dylan Green <67574902+cidkidnix@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-20 11:34:09 +02:00
John Ericson
469d06f9bc Split out worker protocol template definitions from declarations
This is generally a fine practice: Putting implementations in headers
makes them harder to read and slows compilation. Unfortunately it is
necessary for templates, but we can ameliorate that by putting them in a
separate header. Only files which need to instantiate those templates
will need to include the header with the implementation; the rest can
just include the declaration.

This is now documenting in the contributing guide.

Also, it just happens that these polymorphic serializers are the
protocol agnostic ones. (Worker and serve protocol have the same logic
for these container types.) This means by doing this general template
cleanup, we are also getting a head start on better indicating which
code is protocol-specific and which code is shared between protocols.
2023-06-19 11:45:59 -04:00
Eelco Dolstra
f5e620bf2b
Merge pull request #8483 from edolstra/save-root
restoreMountNamespace(): Restore the original root directory
2023-06-19 12:54:05 +02:00
John Ericson
d2ce2e89b1 Split OptionalPathSetting from PathSetting
Rather than doing `allowEmpty` as boolean, have separate types and use
`std::optional`. This makes it harder to forget the possibility of an
empty path.

The `build-hook` setting was categorized as a `PathSetting`, but
actually it was split into arguments. No good! Now, it is
`Setting<Strings>` which actually reflects what it means and how it is
used.

Because of the subtyping, we now also have support for
`Setting<std::optional<String>>` in general. I imagine this can be used
to clean up many more settings also.
2023-06-18 23:31:18 -04:00
John Ericson
c8825e9d8c Create nlohmann serializers for std::optional and use
This is somewhat tricky.
2023-06-18 23:31:10 -04:00
Théophane Hufschmitt
b2247ef4f6 Don't assume the type of string::size_type
The code accidentally conflated `std::string::size_type` and `long unsigned int`.
This was fine on 64bits machines where they are apparently the same in
practice, but not on 32bits. Fix that by using `std::string::size_type`
everywhere.
2023-06-15 21:24:14 +02:00
Daniel Asaturov
468add5aa0
Remove dead code (#8504)
`filesystem.cc` is the only place where `createSymlink()` is used with three arguments:
in the definition of `replaceSymlink()` with three parameters that _is not used at all_.

Closes #8495
2023-06-14 14:09:11 -04:00
John Ericson
61a3e1f2e2
Merge pull request #4282 from tweag/fix-ca-hash-rewriting
fix the hash rewriting for ca-derivations
2023-06-14 18:25:00 +02:00
Eelco Dolstra
e54538c461 restoreMountNamespace(): Restore the original root directory
This is necessary when we're in a chroot environment, where the
process root is not the same as the root of the mount namespace
(e.g. in nixos-enter).

Fixes #7602.
2023-06-09 16:09:29 +02:00
Andrea Bedini
3c78920f73
Parse TOML timestamps (#8120)
Currently `fromTOML` throws an exception when encountering a timestamp
since the Nix language lacks a way to represent them.

This patch changes this beaviour and makes `fromTOML` parse timestamps as
attrsets of the format

  { _type = "timestamp"; value = "1979-05-27T07:32:00Z"; }

This is guarded by an experimental feature flag to leave room for iterating on the representation.
2023-06-09 11:53:18 +02:00
Valentin Gagarin
738c0d5064
Merge pull request #8318 from fricklerhandwerk/doc-currentTime
document `builtins.currentTime`
2023-05-31 03:15:54 +02:00
Yorick van Pelt
2c462486fe
create pathAccessible, use it to infer default dirs 2023-05-26 15:36:47 +02:00
Silvan Mosberger
be48907470
ci: Always run with sandbox, even on Darwin
And fix a test failure in the sandbox due to /home
existing on Darwin but not being accessible in the sandbox since it's a
symlink to /System/Volumes/Data/home, see
https://github.com/NixOS/nix/actions/runs/4205378453/jobs/7297384658#step:6:2127:

    C++ exception with description "error: getting status of /home/schnitzel/darmstadt/pommes: Operation not permitted" thrown in the test body.

On Linux this wasn't a problem because there /home doesn't exist in the sandbox
2023-05-26 15:36:44 +02:00
Théophane Hufschmitt
3ebe1341ab Make RewritingSink accept a map of rewrites
Giving it the same semantics as `rewriteStrings`.
Also add some tests for it
2023-05-24 14:11:50 +02:00
Eelco Dolstra
3305fd0cb1
Merge pull request #8354 from KasyanDiGris/git-fetcher-ask-credentials
Ask for git credentials in fetcher
2023-05-19 13:40:43 +02:00
Konstantin Vukolov
4c4ae887b8 Add option isInteractive 2023-05-18 13:18:34 +03:00
John Ericson
b9e5ce4a27 Upgrade downstreamPlaceholder to a type with methods
This gets us ready for dynamic derivation dependencies (part of RFC 92).
2023-05-17 17:41:16 -04:00
Valentin Gagarin
6a5a8f51bb add cross-references to pure evaluation mode
use consistent wording everywhere.
add some details on the configuration option documentation.
2023-05-17 15:01:54 +02:00
John Ericson
98afd6ff76 Delete commited build artifacts
They were improperly added in 8a93b5a551.

They were not `.gitignore`d because they were stale in that commit --
build artifacts no longer used that name by then and so `.gitignore` was
updated accordingly.
2023-05-15 10:50:33 -04:00
John Ericson
914672dc4f
Merge pull request #8141 from tweag/user-files-doc
Document user files of nix
2023-05-15 07:11:47 -04:00
John Ericson
53a1354acf
Merge pull request #3959 from obsidiansystems/ca-drv-exotic
Derivations can output "text-hashed" data
2023-05-10 10:41:59 -04:00
Alexander Bantyev
8a93b5a551 Document user files of nix 2023-04-26 15:38:19 +02:00
Eelco Dolstra
880e7b8ed6 TarArchive: Remove a duplicate constant and increase the buffer size 2023-04-25 16:43:10 +02:00
Eelco Dolstra
01232358ff Merge remote-tracking branch 'origin/master' into source-path 2023-04-24 13:20:36 +02:00
John Ericson
668377f217 TextHashMethod -> TextIngestionMethod, gate with XP feature
I suppose we can use `dynamic-derivations` for the few things we neeed.
2023-04-17 19:02:45 -04:00
Robert Hensing
64ee02890c
Merge pull request #8230 from obsidiansystems/daemon-trust-override
Experimentally allow forcing `nix-daemon` trust; use this to test
2023-04-17 19:43:41 +02:00
John Ericson
d41e1bed5e Experimentally allow forcing nix-daemon trust; use this to test
We finally test the status quo of remote build trust in a number of
ways. We create a new experimental feature on `nix-daemon` to do so.

PR #3921, which improves the situation with trustless remote building,
will build upon these changes. This code / tests was pull out of there
to make this, so everything is easier to review, and in particular we
test before and after so the new behavior in that PR is readily apparent
from the testsuite diff alone.
2023-04-17 13:06:21 -04:00