max/nix-super
1
0
Fork 0
mirror of https://github.com/privatevoid-net/nix-super.git synced 2024-11-24 23:06:16 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
nix-super/src/libexpr/primops/fetchClosure.cc

196 lines
8.1 KiB
C++
Raw Normal View History

Add builtins.fetchClosure This allows closures to be imported at evaluation time, without requiring the user to configure substituters. E.g. builtins.fetchClosure { storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2; from = "https://cache.ngi0.nixos.org"; }
2022-03-21 12:31:01 +02:00
#include "primops.hh"
#include "store-api.hh"
fetchClosure: Allow a path to be rewritten to CA on the fly The advantage is that the resulting closure doesn't need to be signed, so you don't need to configure any binary cache keys on the client.
2022-03-22 22:14:58 +02:00
#include "make-content-addressed.hh"
fetchClosure: Only allow some "safe" store types
2022-03-22 23:47:33 +02:00
#include "url.hh"
Add builtins.fetchClosure This allows closures to be imported at evaluation time, without requiring the user to configure substituters. E.g. builtins.fetchClosure { storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2; from = "https://cache.ngi0.nixos.org"; }
2022-03-21 12:31:01 +02:00
namespace nix {
replace most Pos objects/ptrs with indexes into a position table Pos objects are somewhat wasteful as they duplicate the origin file name and input type for each object. on files that produce more than one Pos when parsed this a sizeable waste of memory (one pointer per Pos). the same goes for ptr<Pos> on 64 bit machines: parsing enough source to require 8 bytes to locate a position would need at least 8GB of input and 64GB of expression memory. it's not likely that we'll hit that any time soon, so we can use a uint32_t index to locate positions instead.
2022-03-04 20:31:59 +02:00
static void prim_fetchClosure(EvalState & state, const PosIdx pos, Value * * args, Value & v)
Add builtins.fetchClosure This allows closures to be imported at evaluation time, without requiring the user to configure substituters. E.g. builtins.fetchClosure { storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2; from = "https://cache.ngi0.nixos.org"; }
2022-03-21 12:31:01 +02:00
{
Revert "Revert "Merge pull request #6204 from layus/coerce-string"" This reverts commit 9b33ef3879a764bed4cc2404a08344c3a697a646.
2023-01-19 14:23:04 +02:00
state.forceAttrs(*args[0], pos, "while evaluating the argument passed to builtins.fetchClosure");
Add builtins.fetchClosure This allows closures to be imported at evaluation time, without requiring the user to configure substituters. E.g. builtins.fetchClosure { storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2; from = "https://cache.ngi0.nixos.org"; }
2022-03-21 12:31:01 +02:00
Rename
2022-03-21 15:34:45 +02:00
std::optional<std::string> fromStoreUrl;
std::optional<StorePath> fromPath;
fetchClosure: Refactor: rename toCA -> enableRewriting
2023-05-19 11:03:25 +03:00
bool enableRewriting = false;
fetchClosure: Allow a path to be rewritten to CA on the fly The advantage is that the resulting closure doesn't need to be signed, so you don't need to configure any binary cache keys on the client.
2022-03-22 22:14:58 +02:00
std::optional<StorePath> toPath;
fetchClosure: Allow input addressed paths in pure mode When explicitly requested by the caller, as suggested in the meeting (https://github.com/NixOS/nix/pull/8090#issuecomment-1531139324) > @edolstra: { toPath } vs { fromPath } is too implicit I've opted for the `inputAddressed = true` requirement, because it we did not agree on renaming the path attributes. > @roberth: more explicit > @edolstra: except for the direction; not immediately clear in which direction the rewriting happens This is in fact the most explicit syntax and a bit redundant, which is good, because that redundancy lets us deliver an error message that reminds expression authors that CA provides a better experience to their users.
2023-05-19 11:48:53 +03:00
bool inputAddressed = false;
Add builtins.fetchClosure This allows closures to be imported at evaluation time, without requiring the user to configure substituters. E.g. builtins.fetchClosure { storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2; from = "https://cache.ngi0.nixos.org"; }
2022-03-21 12:31:01 +02:00
for (auto & attr : *args[0]->attrs) {
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
const auto & attrName = state.symbols[attr.name];
fetchClosure: Factor out attribute hint
2023-05-19 10:49:18 +03:00
auto attrHint = [&]() -> std::string {
return "while evaluating the '" + attrName + "' attribute passed to builtins.fetchClosure";
};
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
if (attrName == "fromPath") {
Use `std::set<StringContextElem>` not `PathSet` for string contexts Motivation `PathSet` is not correct because string contexts have other forms (`Built` and `DrvDeep`) that are not rendered as plain store paths. Instead of wrongly using `PathSet`, or "stringly typed" using `StringSet`, use `std::std<StringContextElem>`. ----- In support of this change, `NixStringContext` is now defined as `std::std<StringContextElem>` not `std:vector<StringContextElem>`. The old definition was just used by a `getContext` method which was only used by the eval cache. It can be deleted altogether since the types are now unified and the preexisting `copyContext` function already suffices. Summarizing the previous paragraph: Old: - `value/context.hh`: `NixStringContext = std::vector<StringContextElem>` - `value.hh`: `NixStringContext Value::getContext(...)` - `value.hh`: `copyContext(...)` New: - `value/context.hh`: `NixStringContext = std::set<StringContextElem>` - `value.hh`: `copyContext(...)` ---- The string representation of string context elements no longer contains the store dir. The diff of `src/libexpr/tests/value/context.cc` should make clear what the new representation is, so we recommend reviewing that file first. This was done for two reasons: Less API churn: `Value::mkString` and friends did not take a `Store` before. But if `NixStringContextElem::{parse, to_string}` *do* take a store (as they did before), then we cannot have the `Value` functions use them (in order to work with the fully-structured `NixStringContext`) without adding that argument. That would have been a lot of churn of threading the store, and this diff is already large enough, so the easier and less invasive thing to do was simply make the element `parse` and `to_string` functions not take the `Store` reference, and the easiest way to do that was to simply drop the store dir. Space usage: Dropping the `/nix/store/` (or similar) from the internal representation will safe space in the heap of the Nix programming being interpreted. If the heap contains many strings with non-trivial contexts, the saving could add up to something significant. ---- The eval cache version is bumped. The eval cache serialization uses `NixStringContextElem::{parse, to_string}`, and since those functions are changed per the above, that means the on-disk representation is also changed. This is simply done by changing the name of the used for the eval cache from `eval-cache-v4` to eval-cache-v5`. ---- To avoid some duplication `EvalCache::mkPathString` is added to abstract over the simple case of turning a store path to a string with just that string in the context. Context This PR picks up where #7543 left off. That one introduced the fully structured `NixStringContextElem` data type, but kept `PathSet context` as an awkward middle ground between internal `char[][]` interpreter heap string contexts and `NixStringContext` fully parsed string contexts. The infelicity of `PathSet context` was specifically called out during Nix team group review, but it was agreeing that fixing it could be left as future work. This is that future work. A possible follow-up step would be to get rid of the `char[][]` evaluator heap representation, too, but it is not yet clear how to do that. To use `NixStringContextElem` there we would need to get the STL containers to GC pointers in the GC build, and I am not sure how to do that. ---- PR #7543 effectively is writing the inverse of a `mkPathString`, `mkOutputString`, and one more such function for the `DrvDeep` case. I would like that PR to have property tests ensuring it is actually the inverse as expected. This PR sets things up nicely so that reworking that PR to be in that more elegant and better tested way is possible. Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-01-29 03:31:10 +02:00
NixStringContext context;
fetchClosure: Factor out attribute hint
2023-05-19 10:49:18 +03:00
fromPath = state.coerceToStorePath(attr.pos, *attr.value, context, attrHint());
Add builtins.fetchClosure This allows closures to be imported at evaluation time, without requiring the user to configure substituters. E.g. builtins.fetchClosure { storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2; from = "https://cache.ngi0.nixos.org"; }
2022-03-21 12:31:01 +02:00
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
else if (attrName == "toPath") {
replace most Pos objects/ptrs with indexes into a position table Pos objects are somewhat wasteful as they duplicate the origin file name and input type for each object. on files that produce more than one Pos when parsed this a sizeable waste of memory (one pointer per Pos). the same goes for ptr<Pos> on 64 bit machines: parsing enough source to require 8 bytes to locate a position would need at least 8GB of input and 64GB of expression memory. it's not likely that we'll hit that any time soon, so we can use a uint32_t index to locate positions instead.
2022-03-04 20:31:59 +02:00
state.forceValue(*attr.value, attr.pos);
fetchClosure: Refactor: rename toCA -> enableRewriting
2023-05-19 11:03:25 +03:00
enableRewriting = true;
fetchClosure: Allow a path to be rewritten to CA on the fly The advantage is that the resulting closure doesn't need to be signed, so you don't need to configure any binary cache keys on the client.
2022-03-22 22:14:58 +02:00
if (attr.value->type() != nString || attr.value->string.s != std::string("")) {
Use `std::set<StringContextElem>` not `PathSet` for string contexts Motivation `PathSet` is not correct because string contexts have other forms (`Built` and `DrvDeep`) that are not rendered as plain store paths. Instead of wrongly using `PathSet`, or "stringly typed" using `StringSet`, use `std::std<StringContextElem>`. ----- In support of this change, `NixStringContext` is now defined as `std::std<StringContextElem>` not `std:vector<StringContextElem>`. The old definition was just used by a `getContext` method which was only used by the eval cache. It can be deleted altogether since the types are now unified and the preexisting `copyContext` function already suffices. Summarizing the previous paragraph: Old: - `value/context.hh`: `NixStringContext = std::vector<StringContextElem>` - `value.hh`: `NixStringContext Value::getContext(...)` - `value.hh`: `copyContext(...)` New: - `value/context.hh`: `NixStringContext = std::set<StringContextElem>` - `value.hh`: `copyContext(...)` ---- The string representation of string context elements no longer contains the store dir. The diff of `src/libexpr/tests/value/context.cc` should make clear what the new representation is, so we recommend reviewing that file first. This was done for two reasons: Less API churn: `Value::mkString` and friends did not take a `Store` before. But if `NixStringContextElem::{parse, to_string}` *do* take a store (as they did before), then we cannot have the `Value` functions use them (in order to work with the fully-structured `NixStringContext`) without adding that argument. That would have been a lot of churn of threading the store, and this diff is already large enough, so the easier and less invasive thing to do was simply make the element `parse` and `to_string` functions not take the `Store` reference, and the easiest way to do that was to simply drop the store dir. Space usage: Dropping the `/nix/store/` (or similar) from the internal representation will safe space in the heap of the Nix programming being interpreted. If the heap contains many strings with non-trivial contexts, the saving could add up to something significant. ---- The eval cache version is bumped. The eval cache serialization uses `NixStringContextElem::{parse, to_string}`, and since those functions are changed per the above, that means the on-disk representation is also changed. This is simply done by changing the name of the used for the eval cache from `eval-cache-v4` to eval-cache-v5`. ---- To avoid some duplication `EvalCache::mkPathString` is added to abstract over the simple case of turning a store path to a string with just that string in the context. Context This PR picks up where #7543 left off. That one introduced the fully structured `NixStringContextElem` data type, but kept `PathSet context` as an awkward middle ground between internal `char[][]` interpreter heap string contexts and `NixStringContext` fully parsed string contexts. The infelicity of `PathSet context` was specifically called out during Nix team group review, but it was agreeing that fixing it could be left as future work. This is that future work. A possible follow-up step would be to get rid of the `char[][]` evaluator heap representation, too, but it is not yet clear how to do that. To use `NixStringContextElem` there we would need to get the STL containers to GC pointers in the GC build, and I am not sure how to do that. ---- PR #7543 effectively is writing the inverse of a `mkPathString`, `mkOutputString`, and one more such function for the `DrvDeep` case. I would like that PR to have property tests ensuring it is actually the inverse as expected. This PR sets things up nicely so that reworking that PR to be in that more elegant and better tested way is possible. Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-01-29 03:31:10 +02:00
NixStringContext context;
fetchClosure: Factor out attribute hint
2023-05-19 10:49:18 +03:00
toPath = state.coerceToStorePath(attr.pos, *attr.value, context, attrHint());
fetchClosure: Allow a path to be rewritten to CA on the fly The advantage is that the resulting closure doesn't need to be signed, so you don't need to configure any binary cache keys on the client.
2022-03-22 22:14:58 +02:00
}
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
else if (attrName == "fromStore")
Revert "Revert "Merge pull request #6204 from layus/coerce-string"" This reverts commit 9b33ef3879a764bed4cc2404a08344c3a697a646.
2023-01-19 14:23:04 +02:00
fromStoreUrl = state.forceStringNoCtx(*attr.value, attr.pos,
fetchClosure: Factor out attribute hint
2023-05-19 10:49:18 +03:00
attrHint());
Add builtins.fetchClosure This allows closures to be imported at evaluation time, without requiring the user to configure substituters. E.g. builtins.fetchClosure { storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2; from = "https://cache.ngi0.nixos.org"; }
2022-03-21 12:31:01 +02:00
fetchClosure: Allow input addressed paths in pure mode When explicitly requested by the caller, as suggested in the meeting (https://github.com/NixOS/nix/pull/8090#issuecomment-1531139324) > @edolstra: { toPath } vs { fromPath } is too implicit I've opted for the `inputAddressed = true` requirement, because it we did not agree on renaming the path attributes. > @roberth: more explicit > @edolstra: except for the direction; not immediately clear in which direction the rewriting happens This is in fact the most explicit syntax and a bit redundant, which is good, because that redundancy lets us deliver an error message that reminds expression authors that CA provides a better experience to their users.
2023-05-19 11:48:53 +03:00
else if (attrName == "inputAddressed")
inputAddressed = state.forceBool(*attr.value, attr.pos, attrHint());
Add builtins.fetchClosure This allows closures to be imported at evaluation time, without requiring the user to configure substituters. E.g. builtins.fetchClosure { storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2; from = "https://cache.ngi0.nixos.org"; }
2022-03-21 12:31:01 +02:00
else
throw Error({
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
.msg = hintfmt("attribute '%s' isn't supported in call to 'fetchClosure'", attrName),
replace most Pos objects/ptrs with indexes into a position table Pos objects are somewhat wasteful as they duplicate the origin file name and input type for each object. on files that produce more than one Pos when parsed this a sizeable waste of memory (one pointer per Pos). the same goes for ptr<Pos> on 64 bit machines: parsing enough source to require 8 bytes to locate a position would need at least 8GB of input and 64GB of expression memory. it's not likely that we'll hit that any time soon, so we can use a uint32_t index to locate positions instead.
2022-03-04 20:31:59 +02:00
.errPos = state.positions[pos]
Add builtins.fetchClosure This allows closures to be imported at evaluation time, without requiring the user to configure substituters. E.g. builtins.fetchClosure { storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2; from = "https://cache.ngi0.nixos.org"; }
2022-03-21 12:31:01 +02:00
});
}
Rename
2022-03-21 15:34:45 +02:00
if (!fromPath)
Add builtins.fetchClosure This allows closures to be imported at evaluation time, without requiring the user to configure substituters. E.g. builtins.fetchClosure { storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2; from = "https://cache.ngi0.nixos.org"; }
2022-03-21 12:31:01 +02:00
throw Error({
Rename
2022-03-21 15:34:45 +02:00
.msg = hintfmt("attribute '%s' is missing in call to 'fetchClosure'", "fromPath"),
replace most Pos objects/ptrs with indexes into a position table Pos objects are somewhat wasteful as they duplicate the origin file name and input type for each object. on files that produce more than one Pos when parsed this a sizeable waste of memory (one pointer per Pos). the same goes for ptr<Pos> on 64 bit machines: parsing enough source to require 8 bytes to locate a position would need at least 8GB of input and 64GB of expression memory. it's not likely that we'll hit that any time soon, so we can use a uint32_t index to locate positions instead.
2022-03-04 20:31:59 +02:00
.errPos = state.positions[pos]
Add builtins.fetchClosure This allows closures to be imported at evaluation time, without requiring the user to configure substituters. E.g. builtins.fetchClosure { storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2; from = "https://cache.ngi0.nixos.org"; }
2022-03-21 12:31:01 +02:00
});
fetchClosure: Allow input addressed paths in pure mode When explicitly requested by the caller, as suggested in the meeting (https://github.com/NixOS/nix/pull/8090#issuecomment-1531139324) > @edolstra: { toPath } vs { fromPath } is too implicit I've opted for the `inputAddressed = true` requirement, because it we did not agree on renaming the path attributes. > @roberth: more explicit > @edolstra: except for the direction; not immediately clear in which direction the rewriting happens This is in fact the most explicit syntax and a bit redundant, which is good, because that redundancy lets us deliver an error message that reminds expression authors that CA provides a better experience to their users.
2023-05-19 11:48:53 +03:00
if (inputAddressed) {
if (toPath && toPath != fromPath)
throw Error({
.msg = hintfmt("attribute '%s' is set to true, but 'toPath' does not match 'fromPath'. 'toPath' should be equal, or should be omitted. Instead 'toPath' was '%s' and 'fromPath' was '%s'",
"inputAddressed",
state.store->printStorePath(*toPath),
state.store->printStorePath(*fromPath)),
.errPos = state.positions[pos]
});
assert(!enableRewriting);
}
Rename
2022-03-21 15:34:45 +02:00
if (!fromStoreUrl)
Add builtins.fetchClosure This allows closures to be imported at evaluation time, without requiring the user to configure substituters. E.g. builtins.fetchClosure { storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2; from = "https://cache.ngi0.nixos.org"; }
2022-03-21 12:31:01 +02:00
throw Error({
Rename
2022-03-21 15:34:45 +02:00
.msg = hintfmt("attribute '%s' is missing in call to 'fetchClosure'", "fromStore"),
replace most Pos objects/ptrs with indexes into a position table Pos objects are somewhat wasteful as they duplicate the origin file name and input type for each object. on files that produce more than one Pos when parsed this a sizeable waste of memory (one pointer per Pos). the same goes for ptr<Pos> on 64 bit machines: parsing enough source to require 8 bytes to locate a position would need at least 8GB of input and 64GB of expression memory. it's not likely that we'll hit that any time soon, so we can use a uint32_t index to locate positions instead.
2022-03-04 20:31:59 +02:00
.errPos = state.positions[pos]
Add builtins.fetchClosure This allows closures to be imported at evaluation time, without requiring the user to configure substituters. E.g. builtins.fetchClosure { storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2; from = "https://cache.ngi0.nixos.org"; }
2022-03-21 12:31:01 +02:00
});
fetchClosure: Only allow some "safe" store types
2022-03-22 23:47:33 +02:00
auto parsedURL = parseURL(*fromStoreUrl);
Add a test for fetchClosure and 'nix store make-content-addressed'
2022-03-23 00:19:21 +02:00
if (parsedURL.scheme != "http" &&
parsedURL.scheme != "https" &&
!(getEnv("_NIX_IN_TEST").has_value() && parsedURL.scheme == "file"))
fetchClosure: Only allow some "safe" store types
2022-03-22 23:47:33 +02:00
throw Error({
.msg = hintfmt("'fetchClosure' only supports http:// and https:// stores"),
replace most Pos objects/ptrs with indexes into a position table Pos objects are somewhat wasteful as they duplicate the origin file name and input type for each object. on files that produce more than one Pos when parsed this a sizeable waste of memory (one pointer per Pos). the same goes for ptr<Pos> on 64 bit machines: parsing enough source to require 8 bytes to locate a position would need at least 8GB of input and 64GB of expression memory. it's not likely that we'll hit that any time soon, so we can use a uint32_t index to locate positions instead.
2022-03-04 20:31:59 +02:00
.errPos = state.positions[pos]
fetchClosure: Only allow some "safe" store types
2022-03-22 23:47:33 +02:00
});
fetchClosure: Don't allow URL query parameters Allowing this is a potential security hole, since it allows the user to specify parameters like 'local-nar-cache'.
2022-04-06 12:52:51 +03:00
if (!parsedURL.query.empty())
throw Error({
.msg = hintfmt("'fetchClosure' does not support URL query parameters (in '%s')", *fromStoreUrl),
replace most Pos objects/ptrs with indexes into a position table Pos objects are somewhat wasteful as they duplicate the origin file name and input type for each object. on files that produce more than one Pos when parsed this a sizeable waste of memory (one pointer per Pos). the same goes for ptr<Pos> on 64 bit machines: parsing enough source to require 8 bytes to locate a position would need at least 8GB of input and 64GB of expression memory. it's not likely that we'll hit that any time soon, so we can use a uint32_t index to locate positions instead.
2022-03-04 20:31:59 +02:00
.errPos = state.positions[pos]
fetchClosure: Don't allow URL query parameters Allowing this is a potential security hole, since it allows the user to specify parameters like 'local-nar-cache'.
2022-04-06 12:52:51 +03:00
});
fetchClosure: Only allow some "safe" store types
2022-03-22 23:47:33 +02:00
auto fromStore = openStore(parsedURL.to_string());
Add builtins.fetchClosure This allows closures to be imported at evaluation time, without requiring the user to configure substituters. E.g. builtins.fetchClosure { storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2; from = "https://cache.ngi0.nixos.org"; }
2022-03-21 12:31:01 +02:00
fetchClosure: Refactor: rename toCA -> enableRewriting
2023-05-19 11:03:25 +03:00
if (enableRewriting) {
fetchClosure: Skip makeContentAddressed() if toPath is already valid
2022-03-22 23:01:20 +02:00
if (!toPath || !state.store->isValidPath(*toPath)) {
auto remappings = makeContentAddressed(*fromStore, *state.store, { *fromPath });
auto i = remappings.find(*fromPath);
assert(i != remappings.end());
if (toPath && *toPath != i->second)
throw Error({
.msg = hintfmt("rewriting '%s' to content-addressed form yielded '%s', while '%s' was expected",
state.store->printStorePath(*fromPath),
state.store->printStorePath(i->second),
state.store->printStorePath(*toPath)),
replace most Pos objects/ptrs with indexes into a position table Pos objects are somewhat wasteful as they duplicate the origin file name and input type for each object. on files that produce more than one Pos when parsed this a sizeable waste of memory (one pointer per Pos). the same goes for ptr<Pos> on 64 bit machines: parsing enough source to require 8 bytes to locate a position would need at least 8GB of input and 64GB of expression memory. it's not likely that we'll hit that any time soon, so we can use a uint32_t index to locate positions instead.
2022-03-04 20:31:59 +02:00
.errPos = state.positions[pos]
fetchClosure: Skip makeContentAddressed() if toPath is already valid
2022-03-22 23:01:20 +02:00
});
if (!toPath)
throw Error({
.msg = hintfmt(
"rewriting '%s' to content-addressed form yielded '%s'; "
"please set this in the 'toPath' attribute passed to 'fetchClosure'",
state.store->printStorePath(*fromPath),
state.store->printStorePath(i->second)),
replace most Pos objects/ptrs with indexes into a position table Pos objects are somewhat wasteful as they duplicate the origin file name and input type for each object. on files that produce more than one Pos when parsed this a sizeable waste of memory (one pointer per Pos). the same goes for ptr<Pos> on 64 bit machines: parsing enough source to require 8 bytes to locate a position would need at least 8GB of input and 64GB of expression memory. it's not likely that we'll hit that any time soon, so we can use a uint32_t index to locate positions instead.
2022-03-04 20:31:59 +02:00
.errPos = state.positions[pos]
fetchClosure: Skip makeContentAddressed() if toPath is already valid
2022-03-22 23:01:20 +02:00
});
}
fetchClosure: Allow a path to be rewritten to CA on the fly The advantage is that the resulting closure doesn't need to be signed, so you don't need to configure any binary cache keys on the client.
2022-03-22 22:14:58 +02:00
} else {
fetchClosure: Don't allow URL query parameters Allowing this is a potential security hole, since it allows the user to specify parameters like 'local-nar-cache'.
2022-04-06 12:52:51 +03:00
if (!state.store->isValidPath(*fromPath))
copyClosure(*fromStore, *state.store, RealisedPath::Set { *fromPath });
fetchClosure: Allow a path to be rewritten to CA on the fly The advantage is that the resulting closure doesn't need to be signed, so you don't need to configure any binary cache keys on the client.
2022-03-22 22:14:58 +02:00
toPath = fromPath;
}
Add builtins.fetchClosure This allows closures to be imported at evaluation time, without requiring the user to configure substituters. E.g. builtins.fetchClosure { storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2; from = "https://cache.ngi0.nixos.org"; }
2022-03-21 12:31:01 +02:00
fetchClosure: Allow input addressed paths in pure mode When explicitly requested by the caller, as suggested in the meeting (https://github.com/NixOS/nix/pull/8090#issuecomment-1531139324) > @edolstra: { toPath } vs { fromPath } is too implicit I've opted for the `inputAddressed = true` requirement, because it we did not agree on renaming the path attributes. > @roberth: more explicit > @edolstra: except for the direction; not immediately clear in which direction the rewriting happens This is in fact the most explicit syntax and a bit redundant, which is good, because that redundancy lets us deliver an error message that reminds expression authors that CA provides a better experience to their users.
2023-05-19 11:48:53 +03:00
/* We want input addressing to be explicit, to inform readers and to give
expression authors an opportunity to improve their user experience. */
if (!inputAddressed) {
fetchClosure: Allow a path to be rewritten to CA on the fly The advantage is that the resulting closure doesn't need to be signed, so you don't need to configure any binary cache keys on the client.
2022-03-22 22:14:58 +02:00
auto info = state.store->queryPathInfo(*toPath);
fetchClosure: Allow input addressed paths in pure mode When explicitly requested by the caller, as suggested in the meeting (https://github.com/NixOS/nix/pull/8090#issuecomment-1531139324) > @edolstra: { toPath } vs { fromPath } is too implicit I've opted for the `inputAddressed = true` requirement, because it we did not agree on renaming the path attributes. > @roberth: more explicit > @edolstra: except for the direction; not immediately clear in which direction the rewriting happens This is in fact the most explicit syntax and a bit redundant, which is good, because that redundancy lets us deliver an error message that reminds expression authors that CA provides a better experience to their users.
2023-05-19 11:48:53 +03:00
if (!info->isContentAddressed(*state.store)) {
if (enableRewriting) {
throw Error({
// Ideally we'd compute the path for them, but this error message is unlikely to occur in practice, so we keep it simple.
.msg = hintfmt("Rewriting was requested, but 'toPath' is not content addressed. This is impossible. Please change 'toPath' to the correct path, or to a non-existing path, and try again",
state.store->printStorePath(*toPath)),
.errPos = state.positions[pos]
});
} else {
// We just checked toPath, but we report fromPath, because that's what the user certainly passed.
assert (toPath == fromPath);
throw Error({
.msg = hintfmt("The 'fromPath' value '%s' is input addressed, but input addressing was not requested. If you do intend to return an input addressed store path, add 'inputAddressed = true;' to the 'fetchClosure' arguments. Note that content addressing does not require users to configure a trusted binary cache public key on their systems, and is therefore preferred.",
state.store->printStorePath(*fromPath)),
.errPos = state.positions[pos]
});
}
}
fetchClosure: Require a CA path in pure mode
2022-03-21 15:00:54 +02:00
}
Add builtins.fetchClosure This allows closures to be imported at evaluation time, without requiring the user to configure substituters. E.g. builtins.fetchClosure { storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2; from = "https://cache.ngi0.nixos.org"; }
2022-03-21 12:31:01 +02:00
Use `std::set<StringContextElem>` not `PathSet` for string contexts Motivation `PathSet` is not correct because string contexts have other forms (`Built` and `DrvDeep`) that are not rendered as plain store paths. Instead of wrongly using `PathSet`, or "stringly typed" using `StringSet`, use `std::std<StringContextElem>`. ----- In support of this change, `NixStringContext` is now defined as `std::std<StringContextElem>` not `std:vector<StringContextElem>`. The old definition was just used by a `getContext` method which was only used by the eval cache. It can be deleted altogether since the types are now unified and the preexisting `copyContext` function already suffices. Summarizing the previous paragraph: Old: - `value/context.hh`: `NixStringContext = std::vector<StringContextElem>` - `value.hh`: `NixStringContext Value::getContext(...)` - `value.hh`: `copyContext(...)` New: - `value/context.hh`: `NixStringContext = std::set<StringContextElem>` - `value.hh`: `copyContext(...)` ---- The string representation of string context elements no longer contains the store dir. The diff of `src/libexpr/tests/value/context.cc` should make clear what the new representation is, so we recommend reviewing that file first. This was done for two reasons: Less API churn: `Value::mkString` and friends did not take a `Store` before. But if `NixStringContextElem::{parse, to_string}` *do* take a store (as they did before), then we cannot have the `Value` functions use them (in order to work with the fully-structured `NixStringContext`) without adding that argument. That would have been a lot of churn of threading the store, and this diff is already large enough, so the easier and less invasive thing to do was simply make the element `parse` and `to_string` functions not take the `Store` reference, and the easiest way to do that was to simply drop the store dir. Space usage: Dropping the `/nix/store/` (or similar) from the internal representation will safe space in the heap of the Nix programming being interpreted. If the heap contains many strings with non-trivial contexts, the saving could add up to something significant. ---- The eval cache version is bumped. The eval cache serialization uses `NixStringContextElem::{parse, to_string}`, and since those functions are changed per the above, that means the on-disk representation is also changed. This is simply done by changing the name of the used for the eval cache from `eval-cache-v4` to eval-cache-v5`. ---- To avoid some duplication `EvalCache::mkPathString` is added to abstract over the simple case of turning a store path to a string with just that string in the context. Context This PR picks up where #7543 left off. That one introduced the fully structured `NixStringContextElem` data type, but kept `PathSet context` as an awkward middle ground between internal `char[][]` interpreter heap string contexts and `NixStringContext` fully parsed string contexts. The infelicity of `PathSet context` was specifically called out during Nix team group review, but it was agreeing that fixing it could be left as future work. This is that future work. A possible follow-up step would be to get rid of the `char[][]` evaluator heap representation, too, but it is not yet clear how to do that. To use `NixStringContextElem` there we would need to get the STL containers to GC pointers in the GC build, and I am not sure how to do that. ---- PR #7543 effectively is writing the inverse of a `mkPathString`, `mkOutputString`, and one more such function for the `DrvDeep` case. I would like that PR to have property tests ensuring it is actually the inverse as expected. This PR sets things up nicely so that reworking that PR to be in that more elegant and better tested way is possible. Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-01-29 03:31:10 +02:00
state.mkStorePathString(*toPath, v);
Add builtins.fetchClosure This allows closures to be imported at evaluation time, without requiring the user to configure substituters. E.g. builtins.fetchClosure { storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2; from = "https://cache.ngi0.nixos.org"; }
2022-03-21 12:31:01 +02:00
}
static RegisterPrimOp primop_fetchClosure({
.name = "__fetchClosure",
.args = {"args"},
.doc = R"(
Document fetchClosure
2022-03-23 00:31:48 +02:00
Fetch a Nix store closure from a binary cache, rewriting it into
content-addressed form. For example,
```nix
builtins.fetchClosure {
fromStore = "https://cache.nixos.org";
fromPath = /nix/store/r2jd6ygnmirm2g803mksqqjm4y39yi6i-git-2.33.1;
toPath = /nix/store/ldbhlwhh39wha58rm61bkiiwm6j7211j-git-2.33.1;
}
```
fetches `/nix/store/r2jd...` from the specified binary cache,
and rewrites it into the content-addressed store path
`/nix/store/ldbh...`.
If `fromPath` is already content-addressed, or if you are
fetchClosure: Allow input addressed paths in pure mode When explicitly requested by the caller, as suggested in the meeting (https://github.com/NixOS/nix/pull/8090#issuecomment-1531139324) > @edolstra: { toPath } vs { fromPath } is too implicit I've opted for the `inputAddressed = true` requirement, because it we did not agree on renaming the path attributes. > @roberth: more explicit > @edolstra: except for the direction; not immediately clear in which direction the rewriting happens This is in fact the most explicit syntax and a bit redundant, which is good, because that redundancy lets us deliver an error message that reminds expression authors that CA provides a better experience to their users.
2023-05-19 11:48:53 +03:00
allowing input addressing (`inputAddressed = true;`), then `toPath` may be
Document fetchClosure
2022-03-23 00:31:48 +02:00
omitted.
To find out the correct value for `toPath` given a `fromPath`,
you can use `nix store make-content-addressed`:
```console
nix store make-content-addressed: Support --from / --to
2022-03-23 00:40:04 +02:00
# nix store make-content-addressed --from https://cache.nixos.org /nix/store/r2jd6ygnmirm2g803mksqqjm4y39yi6i-git-2.33.1
Document fetchClosure
2022-03-23 00:31:48 +02:00
rewrote '/nix/store/r2jd6ygnmirm2g803mksqqjm4y39yi6i-git-2.33.1' to '/nix/store/ldbhlwhh39wha58rm61bkiiwm6j7211j-git-2.33.1'
```
This function is similar to `builtins.storePath` in that it
allows you to use a previously built store path in a Nix
expression. However, it is more reproducible because it requires
specifying a binary cache from which the path can be fetched.
fetchClosure: Allow input addressed paths in pure mode When explicitly requested by the caller, as suggested in the meeting (https://github.com/NixOS/nix/pull/8090#issuecomment-1531139324) > @edolstra: { toPath } vs { fromPath } is too implicit I've opted for the `inputAddressed = true` requirement, because it we did not agree on renaming the path attributes. > @roberth: more explicit > @edolstra: except for the direction; not immediately clear in which direction the rewriting happens This is in fact the most explicit syntax and a bit redundant, which is good, because that redundancy lets us deliver an error message that reminds expression authors that CA provides a better experience to their users.
2023-05-19 11:48:53 +03:00
Also, the default requirement of a content-addressed final store path
avoids the need for users to configure [`trusted-public-keys`](@docroot@/command-ref/conf-file.md#conf-trusted-public-keys).
This function is only available if you enable the experimental
feature `fetch-closure`.
Add builtins.fetchClosure This allows closures to be imported at evaluation time, without requiring the user to configure substituters. E.g. builtins.fetchClosure { storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2; from = "https://cache.ngi0.nixos.org"; }
2022-03-21 12:31:01 +02:00
)",
.fun = prim_fetchClosure,
Only provide builtin.{getFlake,fetchClosure} is the corresponding experimental feature is enabled This allows writing fallback code like if builtins ? fetchClosure then builtins.fetchClose { ... } else builtins.storePath ...
2022-03-25 15:04:18 +02:00
.experimentalFeature = Xp::FetchClosure,
Add builtins.fetchClosure This allows closures to be imported at evaluation time, without requiring the user to configure substituters. E.g. builtins.fetchClosure { storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2; from = "https://cache.ngi0.nixos.org"; }
2022-03-21 12:31:01 +02:00
});
}
Reference in a new issue Copy permalink