Commit graph

8859 commits

Author SHA1 Message Date
Eelco Dolstra
643b8d2126
Merge pull request #8299 from urbas/max-substitution-jobs
`max-substitution-jobs` setting
2023-05-12 12:24:27 +02:00
John Ericson
a93110ab19 Fix nix print-dev-env & nix develop with drv paths
Fixes #8309

This regression was because both `CmdDevelop` and `CmdPrintDevEnv` were
switched to be `InstallableValueCommand` subclasses, but actually
neither should have been.

The `nixpkgsFlakeRef` method should indeed not be on the base
installable class, because "flake refs" and "nixpkgs" are not
installable-wide notions, but that doesn't mean these commands should
only accept installable values.
2023-05-10 11:29:45 -04:00
John Ericson
53a1354acf
Merge pull request #3959 from obsidiansystems/ca-drv-exotic
Derivations can output "text-hashed" data
2023-05-10 10:41:59 -04:00
Théophane Hufschmitt
85ff212051
Merge pull request #7721 from yorickvP/post-build-hook
Also pass unwanted outputs to post-build-hook
2023-05-10 14:30:42 +02:00
John Ericson
6a3a87a714 Improve error message for self reference with text hashing
The `ContentAddressWithReferences` method is made total, with error
handling now squarely the caller's job. This is better.
2023-05-09 14:44:08 -04:00
John Ericson
e514b3939a Add name to some error messages 2023-05-09 13:24:53 -04:00
John Ericson
753fc1661d Cleanups to content address types 2023-05-09 13:05:38 -04:00
John Ericson
d3c125e5a8
Apply suggestions from code review
Thanks!

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-05-09 12:45:51 -04:00
John Ericson
6513f4fe92 Fix bug, newInfo -> newInfo0
It appears we were checking a variable in the process of definining it.
2023-05-09 12:31:36 -04:00
John Ericson
35dcbe1c21 Fix spurious change
Didn't mean to use the private name that shouldn't be exposed.
2023-05-09 12:19:03 -04:00
Alex Ameen
82296f8113
prevent double quotation 2023-05-09 09:59:18 -05:00
Alex Ameen
b72bc4a972
libexpr: quote reserved keys when printing
This fixes a bug in commands like `nix eval' which would emit invalid attribute
sets if they contained reserved keywords such as "assert", "let", etc.

These keywords will not be quoted when printed, making them valid expressions.
All keywords recognized by the lexer are quoted except "or", which does not
require quotation.
2023-05-09 09:45:12 -05:00
Matej Urbas
13185133bc introduces Goal::jobCategory 2023-05-08 19:45:46 +01:00
Matej Urbas
1ea1e378de removes MaxSubstitutionJobsSetting 2023-05-08 19:21:57 +01:00
John Ericson
b5d9ef0a4c
Merge pull request #3921 from obsidiansystems/trustless-remote-builder-simple
Trustless remote building for input-addressed drvs
2023-05-08 10:43:37 -04:00
John Ericson
df53a7d268 Split comment, match with each variable 2023-05-08 10:08:01 -04:00
John Ericson
cd0d8e0bd5
Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-05-08 09:57:05 -04:00
John Ericson
278c94d607 Rename a few things in new tests
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-05-08 08:01:58 -04:00
Yorick van Pelt
12685ef45f
CA: rewrite hashes for all outputs, not just the wanted ones 2023-05-08 12:58:59 +02:00
Yorick van Pelt
2ca2c80c4e
libstore: also pass unwanted outputs to the post-build-hook 2023-05-08 12:58:59 +02:00
Robert Hensing
cf8effdae2
Merge pull request #8263 from frederictobiasc/improve-doc-genericclosure
Documentation: Improve builtins.genericClosure
2023-05-08 12:23:19 +02:00
Matej Urbas
613bc699bb max-substitution-jobs setting 2023-05-07 20:22:18 +01:00
figsoda
0662fd8599 Fix hostRegex to accept hosts with a - 2023-05-03 18:59:44 -04:00
Eelco Dolstra
17e6b85d05 nix: Support the --repair flag 2023-04-28 17:03:04 +02:00
Frédéric Christ
f0d2b7eef3 Doc: Improve builtins.genericClosure 2023-04-26 09:37:32 +02:00
Eelco Dolstra
87f676b3a0 Formatting 2023-04-25 16:52:02 +02:00
Eelco Dolstra
a74d397549 nix build --json: Only show non-zero startTime / stopTime 2023-04-25 16:43:47 +02:00
Eelco Dolstra
880e7b8ed6 TarArchive: Remove a duplicate constant and increase the buffer size 2023-04-25 16:43:10 +02:00
Eelco Dolstra
5d3f6dbf59 Add some more SourcePath docs 2023-04-24 13:37:51 +02:00
Eelco Dolstra
ad57cff9bc
Document tMisc
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-24 13:34:46 +02:00
Eelco Dolstra
01232358ff Merge remote-tracking branch 'origin/master' into source-path 2023-04-24 13:20:36 +02:00
John Ericson
8eeaf591db
Add more docs to TextIngestionMethod
Thanks so much!

Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com>
2023-04-21 01:30:55 -04:00
John Ericson
85f0cdc370 Use std::set<StringContextElem> not PathSet for string contexts
Motivation

`PathSet` is not correct because string contexts have other forms
(`Built` and `DrvDeep`) that are not rendered as plain store paths.
Instead of wrongly using `PathSet`, or "stringly typed" using
`StringSet`, use `std::std<StringContextElem>`.

-----

In support of this change, `NixStringContext` is now defined as
`std::std<StringContextElem>` not `std:vector<StringContextElem>`. The
old definition was just used by a `getContext` method which was only
used by the eval cache. It can be deleted altogether since the types are
now unified and the preexisting `copyContext` function already suffices.

Summarizing the previous paragraph:

Old:

  - `value/context.hh`: `NixStringContext = std::vector<StringContextElem>`
  - `value.hh`: `NixStringContext Value::getContext(...)`
  - `value.hh`: `copyContext(...)`

New:

  - `value/context.hh`: `NixStringContext = std::set<StringContextElem>`
  - `value.hh`: `copyContext(...)`
----

The string representation of string context elements no longer contains
the store dir. The diff of `src/libexpr/tests/value/context.cc` should
make clear what the new representation is, so we recommend reviewing
that file first. This was done for two reasons:

Less API churn:

`Value::mkString` and friends did not take a `Store` before. But if
`NixStringContextElem::{parse, to_string}` *do* take a store (as they
did before), then we cannot have the `Value` functions use them (in
order to work with the fully-structured `NixStringContext`) without
adding that argument.

That would have been a lot of churn of threading the store, and this
diff is already large enough, so the easier and less invasive thing to
do was simply make the element `parse` and `to_string` functions not
take the `Store` reference, and the easiest way to do that was to simply
drop the store dir.

Space usage:

Dropping the `/nix/store/` (or similar) from the internal representation
will safe space in the heap of the Nix programming being interpreted. If
the heap contains many strings with non-trivial contexts, the saving
could add up to something significant.

----

The eval cache version is bumped.

The eval cache serialization uses `NixStringContextElem::{parse,
to_string}`, and since those functions are changed per the above, that
means the on-disk representation is also changed.

This is simply done by changing the name of the used for the eval cache
from `eval-cache-v4` to eval-cache-v5`.

----

To avoid some duplication `EvalCache::mkPathString` is added to abstract
over the simple case of turning a store path to a string with just that
string in the context.

Context

This PR picks up where #7543 left off. That one introduced the fully
structured `NixStringContextElem` data type, but kept `PathSet context`
as an awkward middle ground between internal `char[][]` interpreter heap
string contexts and `NixStringContext` fully parsed string contexts.

The infelicity of `PathSet context` was specifically called out during
Nix team group review, but it was agreeing that fixing it could be left
as future work. This is that future work.

A possible follow-up step would be to get rid of the `char[][]`
evaluator heap representation, too, but it is not yet clear how to do
that. To use `NixStringContextElem` there we would need to get the STL
containers to GC pointers in the GC build, and I am not sure how to do
that.

----

PR #7543 effectively is writing the inverse of a `mkPathString`,
`mkOutputString`, and one more such function for the `DrvDeep` case. I
would like that PR to have property tests ensuring it is actually the
inverse as expected.

This PR sets things up nicely so that reworking that PR to be in that
more elegant and better tested way is possible.

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-04-21 01:05:49 -04:00
John Ericson
7103c6da70 Remove references from fixed output derivation ab syntax
In other words, use a plain `ContentAddress` not
`ContentAddressWithReferences` for `DerivationOutput::CAFixed`.

Supporting fixed output derivations with (fixed) references would be a
cool feature, but it is out of scope at this moment.
2023-04-19 15:00:04 -04:00
John Ericson
aba8a8a83a Add a few more content addressing methods
Good to round out the library interface.
2023-04-19 14:13:30 -04:00
John Ericson
20decfd302 Gate dynamic-derivations with drv fromJSON too
Don't want `nix derivation add` to be a way to sneak by experimental
feature checks!
2023-04-19 12:44:38 -04:00
John Ericson
76baaeb341 Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2023-04-19 11:32:14 -04:00
Théophane Hufschmitt
d3e2394e91
Merge pull request #8233 from wentasah/narinfo-corrupt-reason
Make "NAR info file is corrupt" messages more informative
2023-04-19 06:57:08 +02:00
Robert Hensing
28d7ffd448
Merge pull request #8220 from accelbread/whitelist-commit-lockfile-summary
Add commit-lockfile-summary to flake nixConfig whitelist
2023-04-18 16:49:33 +02:00
Robert Hensing
5cd9890e8a src/nix/flake.md: Itemize safe nixConfigs 2023-04-18 16:06:58 +02:00
Michal Sojka
d30d2dc861 Make "NAR info file is corrupt" messages more informative
Recently, I encountered the "NAR info file 'xxxx' is corrupt" error
with my binary cache. The message is not helpful in determining, which
kind of corruption happened. The file, fetched with curl, looked
reasonably.

This commit adds more information to the error message, which should
allow debugging and hopefully fixing the problem.
2023-04-18 14:10:49 +02:00
John Ericson
668377f217 TextHashMethod -> TextIngestionMethod, gate with XP feature
I suppose we can use `dynamic-derivations` for the few things we neeed.
2023-04-17 19:02:45 -04:00
John Ericson
f56c4a5bdf Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2023-04-17 18:10:12 -04:00
John Ericson
aa74c7b0bc Gate experimental features in DerivationOutput::fromJSON
This is an entry point for outside data, so we need to check enabled
experimental features here.
2023-04-17 17:36:12 -04:00
John Ericson
ab5ca608bf Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-17 13:54:48 -04:00
John Ericson
23ee2d79a9 Use buildPathsWithResults in build-remote.cc trustless path
It handles failures more correctly; I am glad we have it now!
2023-04-17 13:49:14 -04:00
John Ericson
79ba0ba37a Improve the build remote comment. 2023-04-17 13:49:14 -04:00
Robert Hensing
64ee02890c
Merge pull request #8230 from obsidiansystems/daemon-trust-override
Experimentally allow forcing `nix-daemon` trust; use this to test
2023-04-17 19:43:41 +02:00
John Ericson
d41e1bed5e Experimentally allow forcing nix-daemon trust; use this to test
We finally test the status quo of remote build trust in a number of
ways. We create a new experimental feature on `nix-daemon` to do so.

PR #3921, which improves the situation with trustless remote building,
will build upon these changes. This code / tests was pull out of there
to make this, so everything is easier to review, and in particular we
test before and after so the new behavior in that PR is readily apparent
from the testsuite diff alone.
2023-04-17 13:06:21 -04:00
John Ericson
2c8475600d Fix some issues with experimental config settings
Issues:

1. Features gated on disabled experimental settings should warn and be
   ignored, not silently succeed.

2. Experimental settings in the same config "batch" (file or env var)
   as the enabling of the experimental feature should work.

3. For (2), the order should not matter.

These are analogous to the issues @roberth caught with my changes for
arg handling, but they are instead for config handling.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-17 12:41:04 -04:00