max/nix-super
1
0
Fork 0
mirror of https://github.com/privatevoid-net/nix-super.git synced 2024-11-24 23:06:16 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
nix-super/src/libexpr/nixexpr.cc

629 lines
17 KiB
C++
Raw Normal View History

* "Fix expression" -> "Nix expression". * More refactoring.
2003-11-18 14:06:07 +02:00
#include "nixexpr.hh"
* Renamed `normalise.cc' -> `build.cc', `storeexprs.cc' -> `derivations.cc', etc. * Store the SHA-256 content hash of store paths in the database after they have been built/added. This is so that we can check whether the store has been messed with (a la `rpm --verify'). * When registering path validity, verify that the closure property holds.
2005-01-19 18:39:47 +02:00
#include "derivations.hh"
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
#include "eval.hh"
#include "symbol-table.hh"
* Use a proper namespace. * Optimise header file usage a bit. * Compile the parser as C++.
2006-09-05 00:06:23 +03:00
#include "util.hh"
libexpr/value/print.* -> libexpr/print.* Generalizes the file to sensibly allow printing any part of the language syntax.
2023-04-16 14:10:45 +03:00
#include "print.hh"
* Don't use ATmake / ATmatch anymore, nor the ATMatcher class. Instead we generate data bindings (build and match functions) for the constructors specified in `constructors.def'. In particular this removes the conversions between AFuns and strings, and Nix expression evaluation now seems 3 to 4 times faster.
2004-10-27 01:54:26 +03:00
* GCC 4.3.0 (Fedora 9) compatibility fixes. Reported by Gour and Armijn Hemel.
2008-05-21 14:17:31 +03:00
#include <cstdlib>
* Don't use ATmake / ATmatch anymore, nor the ATMatcher class. Instead we generate data bindings (build and match functions) for the constructors specified in `constructors.def'. In particular this removes the conversions between AFuns and strings, and Nix expression evaluation now seems 3 to 4 times faster.
2004-10-27 01:54:26 +03:00
* Use a proper namespace. * Optimise header file usage a bit. * Compile the parser as C++.
2006-09-05 00:06:23 +03:00
namespace nix {
slim down parser.y most EvalState and Expr members defined here could be elsewhere, where they'd be easier to maintain (not being embedded in a file with arcane syntax) and *somewhat* more faithfully placed according to the path of the file they're defined in.
2024-01-15 17:52:18 +02:00
unsigned long Expr::nrExprs = 0;
use singleton expr to generate black hole errors this also reduces forceValue code size and removes the need for hideInDiagnostics. coopting thunk forcing like this has the additional benefit of clarifying how these errors can happen in the first place.
2023-12-11 17:23:08 +02:00
ExprBlackHole eBlackHole;
libexpr: Move identifier-like printing to print.cc
2023-04-16 15:07:35 +03:00
// FIXME: remove, because *symbols* are abstract and do not have a single
// textual representation; see printIdentifier()
rename SymbolIdx -> Symbol, Symbol -> SymbolStr after #6218 `Symbol` no longer confers a uniqueness invariant on the string it wraps, it is now possible to create multiple symbols that compare equal but whose string contents have different addresses. this guarantee is now only provided by `SymbolIdx`, leaving `Symbol` only as a string wrapper that knows about the intricacies of how symbols need to be formatted for output. this change renames `SymbolIdx` to `Symbol` to restore the previous semantics of `Symbol` to that name. we also keep the wrapper type and rename it to `SymbolStr` instead of returning plain strings from lookups into the symbol table because symbols are formatted for output in many places. theoretically we do not need `SymbolStr`, only a function that formats a string for output as a symbol, but having to wrap every symbol that appears in a message into eg `formatSymbol()` is error-prone and inconvient.
2022-04-22 22:45:39 +03:00
std::ostream & operator <<(std::ostream & str, const SymbolStr & symbol)
Improve printing of ASTs
2014-10-20 09:44:32 +03:00
{
rename SymbolIdx -> Symbol, Symbol -> SymbolStr after #6218 `Symbol` no longer confers a uniqueness invariant on the string it wraps, it is now possible to create multiple symbols that compare equal but whose string contents have different addresses. this guarantee is now only provided by `SymbolIdx`, leaving `Symbol` only as a string wrapper that knows about the intricacies of how symbols need to be formatted for output. this change renames `SymbolIdx` to `Symbol` to restore the previous semantics of `Symbol` to that name. we also keep the wrapper type and rename it to `SymbolStr` instead of returning plain strings from lookups into the symbol table because symbols are formatted for output in many places. theoretically we do not need `SymbolStr`, only a function that formats a string for output as a symbol, but having to wrap every symbol that appears in a message into eg `formatSymbol()` is error-prone and inconvient.
2022-04-22 22:45:39 +03:00
std::string_view s = symbol;
libexpr: Move identifier-like printing to print.cc
2023-04-16 15:07:35 +03:00
return printIdentifier(str, s);
Improve printing of ASTs
2014-10-20 09:44:32 +03:00
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
void Expr::show(const SymbolTable & symbols, std::ostream & str) const
* Indented strings.
2010-04-13 01:03:27 +03:00
{
abort();
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
void ExprInt::show(const SymbolTable & symbols, std::ostream & str) const
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
{
Expr: remove redundant int and float fields
2023-09-12 14:45:45 +03:00
str << v.integer;
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
void ExprFloat::show(const SymbolTable & symbols, std::ostream & str) const
First hit at providing support for floats in the language.
2016-01-05 01:40:40 +02:00
{
Expr: remove redundant int and float fields
2023-09-12 14:45:45 +03:00
str << v.fpoint;
First hit at providing support for floats in the language.
2016-01-05 01:40:40 +02:00
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
void ExprString::show(const SymbolTable & symbols, std::ostream & str) const
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
{
printLiteral: Do not overload
2023-04-16 13:56:31 +03:00
printLiteralString(str, s);
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
void ExprPath::show(const SymbolTable & symbols, std::ostream & str) const
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
{
str << s;
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
void ExprVar::show(const SymbolTable & symbols, std::ostream & str) const
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
{
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
str << symbols[name];
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
void ExprSelect::show(const SymbolTable & symbols, std::ostream & str) const
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
{
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
str << "(";
e->show(symbols, str);
str << ")." << showAttrPath(symbols, attrPath);
if (def) {
str << " or (";
def->show(symbols, str);
str << ")";
}
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
void ExprOpHasAttr::show(const SymbolTable & symbols, std::ostream & str) const
* More missing constructs.
2010-04-13 00:21:24 +03:00
{
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
str << "((";
e->show(symbols, str);
str << ") ? " << showAttrPath(symbols, attrPath) << ")";
* More missing constructs.
2010-04-13 00:21:24 +03:00
}
use the same bindings print for ExprAttrs and ExprLet this also has the effect of sorting let bindings lexicographically rather than by symbol creation order as was previously done, giving a better canonicalization in the process.
2024-01-27 17:33:34 +02:00
void ExprAttrs::showBindings(const SymbolTable & symbols, std::ostream & str) const
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
{
ExprAttrs::show(): Show attrs in sorted order Issue #6160.
2022-02-25 10:21:04 +02:00
typedef const decltype(attrs)::value_type * Attr;
std::vector<Attr> sorted;
for (auto & i : attrs) sorted.push_back(&i);
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
std::sort(sorted.begin(), sorted.end(), [&](Attr a, Attr b) {
std::string_view sa = symbols[a->first], sb = symbols[b->first];
return sa < sb;
});
group inherit by source during Expr::show for plain inherits this is really just a stylistic choice, but for inherit-from it actually fixes an exponential size increase problem during expr printing (as may happen during assertion failure reporting, on during duplicate attr detection in the parser)
2024-01-27 17:33:34 +02:00
std::vector<Symbol> inherits;
evaluate inherit (from) exprs only once per directive desugaring inherit-from to syntactic duplication of the source expr also duplicates side effects of the source expr (such as trace calls) and expensive computations (such as derivationStrict).
2024-01-27 17:33:34 +02:00
std::map<ExprInheritFrom *, std::vector<Symbol>> inheritsFrom;
ExprAttrs::show(): Show attrs in sorted order Issue #6160.
2022-02-25 10:21:04 +02:00
for (auto & i : sorted) {
group inherit by source during Expr::show for plain inherits this is really just a stylistic choice, but for inherit-from it actually fixes an exponential size increase problem during expr printing (as may happen during assertion failure reporting, on during duplicate attr detection in the parser)
2024-01-27 17:33:34 +02:00
switch (i->second.kind) {
case AttrDef::Kind::Plain:
break;
case AttrDef::Kind::Inherited:
inherits.push_back(i->first);
break;
case AttrDef::Kind::InheritedFrom: {
auto & select = dynamic_cast<ExprSelect &>(*i->second.e);
evaluate inherit (from) exprs only once per directive desugaring inherit-from to syntactic duplication of the source expr also duplicates side effects of the source expr (such as trace calls) and expensive computations (such as derivationStrict).
2024-01-27 17:33:34 +02:00
auto & from = dynamic_cast<ExprInheritFrom &>(*select.e);
inheritsFrom[&from].push_back(i->first);
group inherit by source during Expr::show for plain inherits this is really just a stylistic choice, but for inherit-from it actually fixes an exponential size increase problem during expr printing (as may happen during assertion failure reporting, on during duplicate attr detection in the parser)
2024-01-27 17:33:34 +02:00
break;
}
}
}
if (!inherits.empty()) {
str << "inherit";
for (auto sym : inherits) str << " " << symbols[sym];
str << "; ";
}
for (const auto & [from, syms] : inheritsFrom) {
str << "inherit (";
evaluate inherit (from) exprs only once per directive desugaring inherit-from to syntactic duplication of the source expr also duplicates side effects of the source expr (such as trace calls) and expensive computations (such as derivationStrict).
2024-01-27 17:33:34 +02:00
(*inheritFromExprs)[from->displ]->show(symbols, str);
group inherit by source during Expr::show for plain inherits this is really just a stylistic choice, but for inherit-from it actually fixes an exponential size increase problem during expr printing (as may happen during assertion failure reporting, on during duplicate attr detection in the parser)
2024-01-27 17:33:34 +02:00
str << ")";
for (auto sym : syms) str << " " << symbols[sym];
str << "; ";
}
for (auto & i : sorted) {
if (i->second.kind == AttrDef::Kind::Plain) {
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
str << symbols[i->first] << " = ";
i->second.e->show(symbols, str);
str << "; ";
}
}
for (auto & i : dynamicAttrs) {
str << "\"${";
i.nameExpr->show(symbols, str);
str << "}\" = ";
i.valueExpr->show(symbols, str);
str << "; ";
ExprAttrs::show(): Show attrs in sorted order Issue #6160.
2022-02-25 10:21:04 +02:00
}
use the same bindings print for ExprAttrs and ExprLet this also has the effect of sorting let bindings lexicographically rather than by symbol creation order as was previously done, giving a better canonicalization in the process.
2024-01-27 17:33:34 +02:00
}
void ExprAttrs::show(const SymbolTable & symbols, std::ostream & str) const
{
if (recursive) str << "rec ";
str << "{ ";
showBindings(symbols, str);
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
str << "}";
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
void ExprList::show(const SymbolTable & symbols, std::ostream & str) const
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
{
str << "[ ";
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
for (auto & i : elems) {
str << "(";
i->show(symbols, str);
str << ") ";
}
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
str << "]";
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
void ExprLambda::show(const SymbolTable & symbols, std::ostream & str) const
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
{
str << "(";
libexpr: remove matchAttrs boolean from ExprLambda The boolean is only used to determine if the formals are set to a non-null pointer in all our cases. We can get rid of that allocation and instead just compare the pointer value with NULL. Saving up to sizeof(bool) + platform specific alignment per ExprLambda instace. Probably not a lot of memory but perhaps a few kilobyte with nixpkgs? This also gets rid of a potential issue with dereferencing formals based on the value of the boolean that didn't have to be aligned with the formals pointer but was in all our cases.
2021-10-06 18:08:08 +03:00
if (hasFormals()) {
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
str << "{ ";
bool first = true;
normalize formal order on ExprLambda::show we already normalize attr order to lexicographic, doing the same for formals makes sense. doubly so because the order of formals would otherwise depend on the context of the expression, which is not quite as useful as one might expect.
2024-01-29 07:19:23 +02:00
// the natural Symbol ordering is by creation time, which can lead to the
// same expression being printed in two different ways depending on its
// context. always use lexicographic ordering to avoid this.
for (auto & i : formals->lexicographicOrder(symbols)) {
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
if (first) first = false; else str << ", ";
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
str << symbols[i.name];
if (i.def) {
str << " ? ";
i.def->show(symbols, str);
}
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
}
Improve printing of ASTs
2014-10-20 09:44:32 +03:00
if (formals->ellipsis) {
if (!first) str << ", ";
str << "...";
}
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
str << " }";
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
if (arg) str << " @ ";
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
if (arg) str << symbols[arg];
str << ": ";
body->show(symbols, str);
str << ")";
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
void ExprCall::show(const SymbolTable & symbols, std::ostream & str) const
Optimize primop calls We now parse function applications as a vector of arguments rather than as a chain of binary applications, e.g. 'substring 1 2 "foo"' is parsed as ExprCall { .fun = <substring>, .args = [ <1>, <2>, <"foo"> ] } rather than ExprApp (ExprApp (ExprApp <substring> <1>) <2>) <"foo"> This allows primops to be called immediately (if enough arguments are supplied) without having to allocate intermediate tPrimOpApp values. On $ nix-instantiate --dry-run '<nixpkgs/nixos/release-combined.nix>' -A nixos.tests.simple.x86_64-linux this gives a substantial performance improvement: user CPU time: median = 0.9209 mean = 0.9218 stddev = 0.0073 min = 0.9086 max = 0.9340 [rejected, p=0.00000, Δ=-0.21433±0.00677] elapsed time: median = 1.0585 mean = 1.0584 stddev = 0.0024 min = 1.0523 max = 1.0623 [rejected, p=0.00000, Δ=-0.20594±0.00236] because it reduces the number of tPrimOpApp allocations from 551990 to 42534 (i.e. only small minority of primop calls are partially applied) which in turn reduces time spent in the garbage collector.
2020-02-24 02:32:01 +02:00
{
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
str << '(';
fun->show(symbols, str);
Optimize primop calls We now parse function applications as a vector of arguments rather than as a chain of binary applications, e.g. 'substring 1 2 "foo"' is parsed as ExprCall { .fun = <substring>, .args = [ <1>, <2>, <"foo"> ] } rather than ExprApp (ExprApp (ExprApp <substring> <1>) <2>) <"foo"> This allows primops to be called immediately (if enough arguments are supplied) without having to allocate intermediate tPrimOpApp values. On $ nix-instantiate --dry-run '<nixpkgs/nixos/release-combined.nix>' -A nixos.tests.simple.x86_64-linux this gives a substantial performance improvement: user CPU time: median = 0.9209 mean = 0.9218 stddev = 0.0073 min = 0.9086 max = 0.9340 [rejected, p=0.00000, Δ=-0.21433±0.00677] elapsed time: median = 1.0585 mean = 1.0584 stddev = 0.0024 min = 1.0523 max = 1.0623 [rejected, p=0.00000, Δ=-0.20594±0.00236] because it reduces the number of tPrimOpApp allocations from 551990 to 42534 (i.e. only small minority of primop calls are partially applied) which in turn reduces time spent in the garbage collector.
2020-02-24 02:32:01 +02:00
for (auto e : args) {
str << ' ';
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
e->show(symbols, str);
Optimize primop calls We now parse function applications as a vector of arguments rather than as a chain of binary applications, e.g. 'substring 1 2 "foo"' is parsed as ExprCall { .fun = <substring>, .args = [ <1>, <2>, <"foo"> ] } rather than ExprApp (ExprApp (ExprApp <substring> <1>) <2>) <"foo"> This allows primops to be called immediately (if enough arguments are supplied) without having to allocate intermediate tPrimOpApp values. On $ nix-instantiate --dry-run '<nixpkgs/nixos/release-combined.nix>' -A nixos.tests.simple.x86_64-linux this gives a substantial performance improvement: user CPU time: median = 0.9209 mean = 0.9218 stddev = 0.0073 min = 0.9086 max = 0.9340 [rejected, p=0.00000, Δ=-0.21433±0.00677] elapsed time: median = 1.0585 mean = 1.0584 stddev = 0.0024 min = 1.0523 max = 1.0623 [rejected, p=0.00000, Δ=-0.20594±0.00236] because it reduces the number of tPrimOpApp allocations from 551990 to 42534 (i.e. only small minority of primop calls are partially applied) which in turn reduces time spent in the garbage collector.
2020-02-24 02:32:01 +02:00
}
str << ')';
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
void ExprLet::show(const SymbolTable & symbols, std::ostream & str) const
* Evaluate lets directly (i.e. without desugaring to `rec { attrs...; <let-body> = e; }.<let-body>). This prevents the unnecessary allocation of an attribute set.
2010-04-13 16:42:25 +03:00
{
Improve printing of ASTs
2014-10-20 09:44:32 +03:00
str << "(let ";
use the same bindings print for ExprAttrs and ExprLet this also has the effect of sorting let bindings lexicographically rather than by symbol creation order as was previously done, giving a better canonicalization in the process.
2024-01-27 17:33:34 +02:00
attrs->showBindings(symbols, str);
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
str << "in ";
body->show(symbols, str);
str << ")";
* Evaluate lets directly (i.e. without desugaring to `rec { attrs...; <let-body> = e; }.<let-body>). This prevents the unnecessary allocation of an attribute set.
2010-04-13 16:42:25 +03:00
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
void ExprWith::show(const SymbolTable & symbols, std::ostream & str) const
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
{
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
str << "(with ";
attrs->show(symbols, str);
str << "; ";
body->show(symbols, str);
str << ")";
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
void ExprIf::show(const SymbolTable & symbols, std::ostream & str) const
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
{
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
str << "(if ";
cond->show(symbols, str);
str << " then ";
then->show(symbols, str);
str << " else ";
else_->show(symbols, str);
str << ")";
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
void ExprAssert::show(const SymbolTable & symbols, std::ostream & str) const
* More missing constructs.
2010-04-13 00:21:24 +03:00
{
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
str << "assert ";
cond->show(symbols, str);
str << "; ";
body->show(symbols, str);
* More missing constructs.
2010-04-13 00:21:24 +03:00
}
* Don't use ATerms for the abstract syntax trees anymore. Not finished yet.
2010-04-12 21:30:11 +03:00
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
void ExprOpNot::show(const SymbolTable & symbols, std::ostream & str) const
* More missing constructs.
2010-04-13 00:21:24 +03:00
{
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
str << "(! ";
e->show(symbols, str);
str << ")";
* More missing constructs.
2010-04-13 00:21:24 +03:00
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
void ExprConcatStrings::show(const SymbolTable & symbols, std::ostream & str) const
* More missing constructs.
2010-04-13 00:21:24 +03:00
{
bool first = true;
Improve printing of ASTs
2014-10-20 09:44:32 +03:00
str << "(";
OCD: foreach -> C++11 ranged for
2015-07-17 20:24:28 +03:00
for (auto & i : *es) {
* More missing constructs.
2010-04-13 00:21:24 +03:00
if (first) first = false; else str << " + ";
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
i.second->show(symbols, str);
* More missing constructs.
2010-04-13 00:21:24 +03:00
}
Improve printing of ASTs
2014-10-20 09:44:32 +03:00
str << ")";
* More missing constructs.
2010-04-13 00:21:24 +03:00
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
void ExprPos::show(const SymbolTable & symbols, std::ostream & str) const
Add a symbol __curPos that expands to the current source location I.e. an attribute set { file = <string>; line = <int>; column = <int>; }.
2013-11-18 21:14:54 +02:00
{
str << "__curPos";
}
* More missing constructs.
2010-04-13 00:21:24 +03:00
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
std::string showAttrPath(const SymbolTable & symbols, const AttrPath & attrPath)
* In the ‘?’ operator, allow attribute paths. For instance, you can write ‘attrs ? a.b’ to test whether ‘attrs’ has an attribute ‘a’ containing an attribute ‘b’. This is more convenient than ‘attrs ? a && attrs.a ? b’. Slight change in the semantics: it's no longer an error if the left-hand side of ‘?’ is not an attribute set. In that case it just returns false. So, ‘null ? foo’ no longer throws an error.
2011-07-06 13:58:17 +03:00
{
Don't use any syntactic sugar for dynamic attrs This doesn't change any functionality but moves some behavior out of the parser and into the evaluator in order to simplify the code. Signed-off-by: Shea Levy <shea@shealevy.com>
2014-01-01 01:56:26 +02:00
std::ostringstream out;
bool first = true;
Improve error message
2015-03-06 15:24:08 +02:00
for (auto & i : attrPath) {
if (!first) out << '.'; else first = false;
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
if (i.symbol)
out << symbols[i.symbol];
else {
out << "\"${";
i.expr->show(symbols, out);
out << "}\"";
}
* In the ‘?’ operator, allow attribute paths. For instance, you can write ‘attrs ? a.b’ to test whether ‘attrs’ has an attribute ‘a’ containing an attribute ‘b’. This is more convenient than ‘attrs ? a && attrs.a ? b’. Slight change in the semantics: it's no longer an error if the left-hand side of ‘?’ is not an attribute set. In that case it just returns false. So, ‘null ? foo’ no longer throws an error.
2011-07-06 13:58:17 +03:00
}
Don't use any syntactic sugar for dynamic attrs This doesn't change any functionality but moves some behavior out of the parser and into the evaluator in order to simplify the code. Signed-off-by: Shea Levy <shea@shealevy.com>
2014-01-01 01:56:26 +02:00
return out.str();
* In the ‘?’ operator, allow attribute paths. For instance, you can write ‘attrs ? a.b’ to test whether ‘attrs’ has an attribute ‘a’ containing an attribute ‘b’. This is more convenient than ‘attrs ? a && attrs.a ? b’. Slight change in the semantics: it's no longer an error if the left-hand side of ‘?’ is not an attribute set. In that case it just returns false. So, ‘null ? foo’ no longer throws an error.
2011-07-06 13:58:17 +03:00
}
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
/* Computing levels/displacements for variables. */
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
void Expr::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
{
abort();
}
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
void ExprInt::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
{
remove redundant 'debugMode' flag
2022-05-23 06:45:24 +03:00
if (es.debugRepl)
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
es.exprEnvs.insert(std::make_pair(this, env));
}
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
void ExprFloat::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
First hit at providing support for floats in the language.
2016-01-05 01:40:40 +02:00
{
remove redundant 'debugMode' flag
2022-05-23 06:45:24 +03:00
if (es.debugRepl)
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
es.exprEnvs.insert(std::make_pair(this, env));
First hit at providing support for floats in the language.
2016-01-05 01:40:40 +02:00
}
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
void ExprString::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
{
remove redundant 'debugMode' flag
2022-05-23 06:45:24 +03:00
if (es.debugRepl)
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
es.exprEnvs.insert(std::make_pair(this, env));
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
}
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
void ExprPath::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
{
remove redundant 'debugMode' flag
2022-05-23 06:45:24 +03:00
if (es.debugRepl)
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
es.exprEnvs.insert(std::make_pair(this, env));
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
}
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
void ExprVar::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
{
remove redundant 'debugMode' flag
2022-05-23 06:45:24 +03:00
if (es.debugRepl)
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
es.exprEnvs.insert(std::make_pair(this, env));
storing staticenv bindings
2021-10-02 22:47:36 +03:00
reduce the size of Env by one pointer since `up` and `values` are both pointer-aligned the type field will also be pointer-aligned, wasting 48 bits of space on most machines. we can get away with removing the type field altogether by encoding some information into the `with` expr that created the env to begin with, reducing the GC load for the absolutely massive amount of single-entry envs we create for lambdas. this reduces memory usage of system eval by quite a bit (reducing heap size of our system eval from 8.4GB to 8.23GB) and gives similar savings in eval time. running `nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` before: Time (mean ± σ): 5.576 s ± 0.003 s [User: 5.197 s, System: 0.378 s] Range (min … max): 5.572 s … 5.581 s 10 runs after: Time (mean ± σ): 5.408 s ± 0.002 s [User: 5.019 s, System: 0.388 s] Range (min … max): 5.405 s … 5.411 s 10 runs
2023-12-22 19:19:53 +02:00
fromWith = nullptr;
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
/* Check whether the variable appears in the environment. If so,
set its level and displacement. */
more code cleanup
2021-10-22 23:49:58 +03:00
const StaticEnv * curEnv;
Add level / displacement types
2020-02-24 15:33:01 +02:00
Level level;
more code cleanup
2021-10-22 23:49:58 +03:00
int withLevel = -1;
for (curEnv = env.get(), level = 0; curEnv; curEnv = curEnv->up, level++) {
if (curEnv->isWith) {
if (withLevel == -1) withLevel = level;
} else {
StaticEnv: Use std::vector instead of std::map
2020-02-21 19:31:16 +02:00
auto i = curEnv->find(name);
more code cleanup
2021-10-22 23:49:58 +03:00
if (i != curEnv->vars.end()) {
this->level = level;
displ = i->second;
return;
}
}
ok was unconditoinally throwing on any With var
2021-09-23 22:02:39 +03:00
}
more code cleanup
2021-10-22 23:49:58 +03:00
/* Otherwise, the variable must be obtained from the nearest
enclosing `with'. If there is no `with', then we can issue an
"undefined variable" error now. */
line endings
2022-04-29 20:24:54 +03:00
if (withLevel == -1)
libexpr: Support structured error classes While preparing PRs like #9753, I've had to change error messages in dozens of code paths. It would be nice if instead of EvalError("expected 'boolean' but found '%1%'", showType(v)) we could write TypeError(v, "boolean") or similar. Then, changing the error message could be a mechanical refactor with the compiler pointing out places the constructor needs to be changed, rather than the error-prone process of grepping through the codebase. Structured errors would also help prevent the "same" error from having multiple slightly different messages, and could be a first step towards error codes / an error index. This PR reworks the exception infrastructure in `libexpr` to support exception types with different constructor signatures than `BaseError`. Actually refactoring the exceptions to use structured data will come in a future PR (this one is big enough already, as it has to touch every exception in `libexpr`). The core design is in `eval-error.hh`. Generally, errors like this: state.error("'%s' is not a string", getAttrPathStr()) .debugThrow<TypeError>() are transformed like this: state.error<TypeError>("'%s' is not a string", getAttrPathStr()) .debugThrow() The type annotation has moved from `ErrorBuilder::debugThrow` to `EvalState::error`.
2024-01-23 03:08:29 +02:00
es.error<UndefinedVarError>(
"undefined variable '%1%'",
es.symbols[name]
).atPos(pos).debugThrow();
reduce the size of Env by one pointer since `up` and `values` are both pointer-aligned the type field will also be pointer-aligned, wasting 48 bits of space on most machines. we can get away with removing the type field altogether by encoding some information into the `with` expr that created the env to begin with, reducing the GC load for the absolutely massive amount of single-entry envs we create for lambdas. this reduces memory usage of system eval by quite a bit (reducing heap size of our system eval from 8.4GB to 8.23GB) and gives similar savings in eval time. running `nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` before: Time (mean ± σ): 5.576 s ± 0.003 s [User: 5.197 s, System: 0.378 s] Range (min … max): 5.572 s … 5.581 s 10 runs after: Time (mean ± σ): 5.408 s ± 0.002 s [User: 5.019 s, System: 0.388 s] Range (min … max): 5.405 s … 5.411 s 10 runs
2023-12-22 19:19:53 +02:00
for (auto * e = env.get(); e && !fromWith; e = e->up)
fromWith = e->isWith;
more code cleanup
2021-10-22 23:49:58 +03:00
this->level = withLevel;
* Refactoring: combine functions that take an attribute set and functions that take a single argument (plain lambdas) into one AST node (Function) that contains a Pattern node describing the arguments. Current patterns are single lazy arguments (VarPat) and matching against an attribute set (AttrsPat). This refactoring allows other kinds of patterns to be added easily, such as Haskell-style @-patterns, or list pattern matching.
2008-08-14 13:04:22 +03:00
}
evaluate inherit (from) exprs only once per directive desugaring inherit-from to syntactic duplication of the source expr also duplicates side effects of the source expr (such as trace calls) and expensive computations (such as derivationStrict).
2024-01-27 17:33:34 +02:00
void ExprInheritFrom::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
{
if (es.debugRepl)
es.exprEnvs.insert(std::make_pair(this, env));
}
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
void ExprSelect::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
{
remove redundant 'debugMode' flag
2022-05-23 06:45:24 +03:00
if (es.debugRepl)
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
es.exprEnvs.insert(std::make_pair(this, env));
storing staticenv bindings
2021-10-02 22:47:36 +03:00
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
e->bindVars(es, env);
if (def) def->bindVars(es, env);
OCD: foreach -> C++11 ranged for
2015-07-17 20:24:28 +03:00
for (auto & i : attrPath)
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
if (!i.symbol)
i.expr->bindVars(es, env);
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
}
* Refactoring: combine functions that take an attribute set and functions that take a single argument (plain lambdas) into one AST node (Function) that contains a Pattern node describing the arguments. Current patterns are single lazy arguments (VarPat) and matching against an attribute set (AttrsPat). This refactoring allows other kinds of patterns to be added easily, such as Haskell-style @-patterns, or list pattern matching.
2008-08-14 13:04:22 +03:00
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
void ExprOpHasAttr::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
* Verify that all variables in a Nix expression are defined.
2004-02-03 16:45:34 +02:00
{
remove redundant 'debugMode' flag
2022-05-23 06:45:24 +03:00
if (es.debugRepl)
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
es.exprEnvs.insert(std::make_pair(this, env));
storing staticenv bindings
2021-10-02 22:47:36 +03:00
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
e->bindVars(es, env);
OCD: foreach -> C++11 ranged for
2015-07-17 20:24:28 +03:00
for (auto & i : attrPath)
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
if (!i.symbol)
i.expr->bindVars(es, env);
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
}
evaluate inherit (from) exprs only once per directive desugaring inherit-from to syntactic duplication of the source expr also duplicates side effects of the source expr (such as trace calls) and expensive computations (such as derivationStrict).
2024-01-27 17:33:34 +02:00
std::shared_ptr<const StaticEnv> ExprAttrs::bindInheritSources(
EvalState & es, const std::shared_ptr<const StaticEnv> & env)
{
if (!inheritFromExprs)
return nullptr;
// the inherit (from) source values are inserted into an env of its own, which
// does not introduce any variable names.
// analysis must see an empty env, or an env that contains only entries with
// otherwise unused names to not interfere with regular names. the parser
// has already filled all exprs that access this env with appropriate level
// and displacement, and nothing else is allowed to access it. ideally we'd
// not even *have* an expr that grabs anything from this env since it's fully
// invisible, but the evaluator does not allow for this yet.
auto inner = std::make_shared<StaticEnv>(nullptr, env.get(), 0);
for (auto from : *inheritFromExprs)
from->bindVars(es, env);
return inner;
}
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
void ExprAttrs::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
{
remove redundant 'debugMode' flag
2022-05-23 06:45:24 +03:00
if (es.debugRepl)
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
es.exprEnvs.insert(std::make_pair(this, env));
storing staticenv bindings
2021-10-02 22:47:36 +03:00
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
if (recursive) {
add ExprAttrs::AttrDef::chooseByKind in place of inherited() — not quite useful yet since we don't distinguish plain and inheritFrom attr kinds so far.
2024-01-27 17:33:34 +02:00
auto newEnv = [&] () -> std::shared_ptr<const StaticEnv> {
auto newEnv = std::make_shared<StaticEnv>(nullptr, env.get(), attrs.size());
Fix whitespace
2013-09-02 17:29:15 +03:00
add ExprAttrs::AttrDef::chooseByKind in place of inherited() — not quite useful yet since we don't distinguish plain and inheritFrom attr kinds so far.
2024-01-27 17:33:34 +02:00
Displacement displ = 0;
for (auto & i : attrs)
newEnv->vars.emplace_back(i.first, i.second.displ = displ++);
return newEnv;
}();
StaticEnv: Use std::vector instead of std::map
2020-02-21 19:31:16 +02:00
// No need to sort newEnv since attrs is in sorted order.
Fix whitespace
2013-09-02 17:29:15 +03:00
evaluate inherit (from) exprs only once per directive desugaring inherit-from to syntactic duplication of the source expr also duplicates side effects of the source expr (such as trace calls) and expensive computations (such as derivationStrict).
2024-01-27 17:33:34 +02:00
auto inheritFromEnv = bindInheritSources(es, newEnv);
OCD: foreach -> C++11 ranged for
2015-07-17 20:24:28 +03:00
for (auto & i : attrs)
evaluate inherit (from) exprs only once per directive desugaring inherit-from to syntactic duplication of the source expr also duplicates side effects of the source expr (such as trace calls) and expensive computations (such as derivationStrict).
2024-01-27 17:33:34 +02:00
i.second.e->bindVars(es, i.second.chooseByKind(newEnv, env, inheritFromEnv));
add cout debugging
2021-09-16 01:16:53 +03:00
for (auto & i : dynamicAttrs) {
Merge branch 'master' into debug-merge-master
2022-04-28 21:32:57 +03:00
i.nameExpr->bindVars(es, newEnv);
i.valueExpr->bindVars(es, newEnv);
add cout debugging
2021-09-16 01:16:53 +03:00
}
* New language feature: with expressions. The expression `with E1; E2' evaluates to E2 with all bindings in the attribute set E1 substituted. E.g., with {x = 123;}; x evaluates to 123. That is, the attribute set E1 is in scope in E2. This is particularly useful when importing files containing lots definitions. E.g., instead of let { inherit (import ./foo.nix) a b c d e f; body = ... a ... f ...; } we can now say with import ./foo.nix; ... a ... f ... I.e., we don't have to say what variables should be brought into scope.
2004-10-25 19:54:56 +03:00
}
add cout debugging
2021-09-16 01:16:53 +03:00
else {
evaluate inherit (from) exprs only once per directive desugaring inherit-from to syntactic duplication of the source expr also duplicates side effects of the source expr (such as trace calls) and expensive computations (such as derivationStrict).
2024-01-27 17:33:34 +02:00
auto inheritFromEnv = bindInheritSources(es, env);
OCD: foreach -> C++11 ranged for
2015-07-17 20:24:28 +03:00
for (auto & i : attrs)
evaluate inherit (from) exprs only once per directive desugaring inherit-from to syntactic duplication of the source expr also duplicates side effects of the source expr (such as trace calls) and expensive computations (such as derivationStrict).
2024-01-27 17:33:34 +02:00
i.second.e->bindVars(es, i.second.chooseByKind(env, env, inheritFromEnv));
Dynamic attrs This adds new syntax for attribute names: * attrs."${name}" => getAttr name attrs * attrs ? "${name}" => isAttrs attrs && hasAttr attrs name * attrs."${name}" or def => if attrs ? "${name}" then attrs."${name}" else def * { "${name}" = value; } => listToAttrs [{ inherit name value; }] Of course, it's a bit more complicated than that. The attribute chains can be arbitrarily long and contain combinations of static and dynamic parts (e.g. attrs."${foo}".bar."${baz}" or qux), which is relatively straightforward for the getAttrs/hasAttrs cases but is more complex for the listToAttrs case due to rules about duplicate attribute definitions. For attribute sets with dynamic attribute names, duplicate static attributes are detected at parse time while duplicate dynamic attributes are detected when the attribute set is forced. So, for example, { a = null; a.b = null; "${"c"}" = true; } will be a parse-time error, while { a = {}; "${"a"}".b = null; c = true; } will be an eval-time error (technically that case could theoretically be detected at parse time, but the general case would require full evaluation). Moreover, duplicate dynamic attributes are not allowed even in cases where they would be with static attributes ({ a.b.d = true; a.b.c = false; } is legal, but { a."${"b"}".d = true; a."${"b"}".c = false; } is not). This restriction might be relaxed in the future in cases where the static variant would not be an error, but it is not obvious that that is desirable. Finally, recursive attribute sets with dynamic attributes have the static attributes in scope but not the dynamic ones. So rec { a = true; "${"b"}" = a; } is equivalent to { a = true; b = true; } but rec { "${"a"}" = true; b = a; } would be an error or use a from the surrounding scope if it exists. Note that the getAttr, getAttr or default, and hasAttr are all implemented purely in the parser as syntactic sugar, while attribute sets with dynamic attribute names required changes to the AST to be implemented cleanly. This is an alternative solution to and closes #167 Signed-off-by: Shea Levy <shea@shealevy.com>
2013-09-21 06:25:30 +03:00
add cout debugging
2021-09-16 01:16:53 +03:00
for (auto & i : dynamicAttrs) {
Merge branch 'master' into debug-merge-master
2022-04-28 21:32:57 +03:00
i.nameExpr->bindVars(es, env);
i.valueExpr->bindVars(es, env);
add cout debugging
2021-09-16 01:16:53 +03:00
}
Dynamic attrs This adds new syntax for attribute names: * attrs."${name}" => getAttr name attrs * attrs ? "${name}" => isAttrs attrs && hasAttr attrs name * attrs."${name}" or def => if attrs ? "${name}" then attrs."${name}" else def * { "${name}" = value; } => listToAttrs [{ inherit name value; }] Of course, it's a bit more complicated than that. The attribute chains can be arbitrarily long and contain combinations of static and dynamic parts (e.g. attrs."${foo}".bar."${baz}" or qux), which is relatively straightforward for the getAttrs/hasAttrs cases but is more complex for the listToAttrs case due to rules about duplicate attribute definitions. For attribute sets with dynamic attribute names, duplicate static attributes are detected at parse time while duplicate dynamic attributes are detected when the attribute set is forced. So, for example, { a = null; a.b = null; "${"c"}" = true; } will be a parse-time error, while { a = {}; "${"a"}".b = null; c = true; } will be an eval-time error (technically that case could theoretically be detected at parse time, but the general case would require full evaluation). Moreover, duplicate dynamic attributes are not allowed even in cases where they would be with static attributes ({ a.b.d = true; a.b.c = false; } is legal, but { a."${"b"}".d = true; a."${"b"}".c = false; } is not). This restriction might be relaxed in the future in cases where the static variant would not be an error, but it is not obvious that that is desirable. Finally, recursive attribute sets with dynamic attributes have the static attributes in scope but not the dynamic ones. So rec { a = true; "${"b"}" = a; } is equivalent to { a = true; b = true; } but rec { "${"a"}" = true; b = a; } would be an error or use a from the surrounding scope if it exists. Note that the getAttr, getAttr or default, and hasAttr are all implemented purely in the parser as syntactic sugar, while attribute sets with dynamic attribute names required changes to the AST to be implemented cleanly. This is an alternative solution to and closes #167 Signed-off-by: Shea Levy <shea@shealevy.com>
2013-09-21 06:25:30 +03:00
}
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
}
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
void ExprList::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
{
remove redundant 'debugMode' flag
2022-05-23 06:45:24 +03:00
if (es.debugRepl)
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
es.exprEnvs.insert(std::make_pair(this, env));
storing staticenv bindings
2021-10-02 22:47:36 +03:00
OCD: foreach -> C++11 ranged for
2015-07-17 20:24:28 +03:00
for (auto & i : elems)
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
i->bindVars(es, env);
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
}
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
void ExprLambda::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
{
remove redundant 'debugMode' flag
2022-05-23 06:45:24 +03:00
if (es.debugRepl)
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
es.exprEnvs.insert(std::make_pair(this, env));
storing staticenv bindings
2021-10-02 22:47:36 +03:00
Style fixes In particular, use std::make_shared and enumerate(). Also renamed some fields to fit naming conventions.
2022-05-05 13:29:14 +03:00
auto newEnv = std::make_shared<StaticEnv>(
reduce the size of Env by one pointer since `up` and `values` are both pointer-aligned the type field will also be pointer-aligned, wasting 48 bits of space on most machines. we can get away with removing the type field altogether by encoding some information into the `with` expr that created the env to begin with, reducing the GC load for the absolutely massive amount of single-entry envs we create for lambdas. this reduces memory usage of system eval by quite a bit (reducing heap size of our system eval from 8.4GB to 8.23GB) and gives similar savings in eval time. running `nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` before: Time (mean ± σ): 5.576 s ± 0.003 s [User: 5.197 s, System: 0.378 s] Range (min … max): 5.572 s … 5.581 s 10 runs after: Time (mean ± σ): 5.408 s ± 0.002 s [User: 5.019 s, System: 0.388 s] Range (min … max): 5.405 s … 5.411 s 10 runs
2023-12-22 19:19:53 +02:00
nullptr, env.get(),
Style fixes In particular, use std::make_shared and enumerate(). Also renamed some fields to fit naming conventions.
2022-05-05 13:29:14 +03:00
(hasFormals() ? formals->formals.size() : 0) +
(!arg ? 0 : 1));
Fix whitespace
2013-09-02 17:29:15 +03:00
Add level / displacement types
2020-02-24 15:33:01 +02:00
Displacement displ = 0;
Fix whitespace
2013-09-02 17:29:15 +03:00
incorporate PosIdx changes, symbol changes.
2022-04-29 19:02:17 +03:00
if (arg) newEnv->vars.emplace_back(arg, displ++);
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
libexpr: remove matchAttrs boolean from ExprLambda The boolean is only used to determine if the formals are set to a non-null pointer in all our cases. We can get rid of that allocation and instead just compare the pointer value with NULL. Saving up to sizeof(bool) + platform specific alignment per ExprLambda instace. Probably not a lot of memory but perhaps a few kilobyte with nixpkgs? This also gets rid of a potential issue with dereferencing formals based on the value of the boolean that didn't have to be aligned with the formals pointer but was in all our cases.
2021-10-06 18:08:08 +03:00
if (hasFormals()) {
OCD: foreach -> C++11 ranged for
2015-07-17 20:24:28 +03:00
for (auto & i : formals->formals)
fix merge issues
2021-11-30 23:15:02 +02:00
newEnv->vars.emplace_back(i.name, displ++);
StaticEnv: Use std::vector instead of std::map
2020-02-21 19:31:16 +02:00
fix merge issues
2021-11-30 23:15:02 +02:00
newEnv->sort();
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
OCD: foreach -> C++11 ranged for
2015-07-17 20:24:28 +03:00
for (auto & i : formals->formals)
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
if (i.def) i.def->bindVars(es, newEnv);
* Verify that all variables in a Nix expression are defined.
2004-02-03 16:45:34 +02:00
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
body->bindVars(es, newEnv);
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
}
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
void ExprCall::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
{
remove redundant 'debugMode' flag
2022-05-23 06:45:24 +03:00
if (es.debugRepl)
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
es.exprEnvs.insert(std::make_pair(this, env));
storing staticenv bindings
2021-10-02 22:47:36 +03:00
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
fun->bindVars(es, env);
Optimize primop calls We now parse function applications as a vector of arguments rather than as a chain of binary applications, e.g. 'substring 1 2 "foo"' is parsed as ExprCall { .fun = <substring>, .args = [ <1>, <2>, <"foo"> ] } rather than ExprApp (ExprApp (ExprApp <substring> <1>) <2>) <"foo"> This allows primops to be called immediately (if enough arguments are supplied) without having to allocate intermediate tPrimOpApp values. On $ nix-instantiate --dry-run '<nixpkgs/nixos/release-combined.nix>' -A nixos.tests.simple.x86_64-linux this gives a substantial performance improvement: user CPU time: median = 0.9209 mean = 0.9218 stddev = 0.0073 min = 0.9086 max = 0.9340 [rejected, p=0.00000, Δ=-0.21433±0.00677] elapsed time: median = 1.0585 mean = 1.0584 stddev = 0.0024 min = 1.0523 max = 1.0623 [rejected, p=0.00000, Δ=-0.20594±0.00236] because it reduces the number of tPrimOpApp allocations from 551990 to 42534 (i.e. only small minority of primop calls are partially applied) which in turn reduces time spent in the garbage collector.
2020-02-24 02:32:01 +02:00
for (auto e : args)
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
e->bindVars(es, env);
Optimize primop calls We now parse function applications as a vector of arguments rather than as a chain of binary applications, e.g. 'substring 1 2 "foo"' is parsed as ExprCall { .fun = <substring>, .args = [ <1>, <2>, <"foo"> ] } rather than ExprApp (ExprApp (ExprApp <substring> <1>) <2>) <"foo"> This allows primops to be called immediately (if enough arguments are supplied) without having to allocate intermediate tPrimOpApp values. On $ nix-instantiate --dry-run '<nixpkgs/nixos/release-combined.nix>' -A nixos.tests.simple.x86_64-linux this gives a substantial performance improvement: user CPU time: median = 0.9209 mean = 0.9218 stddev = 0.0073 min = 0.9086 max = 0.9340 [rejected, p=0.00000, Δ=-0.21433±0.00677] elapsed time: median = 1.0585 mean = 1.0584 stddev = 0.0024 min = 1.0523 max = 1.0623 [rejected, p=0.00000, Δ=-0.20594±0.00236] because it reduces the number of tPrimOpApp allocations from 551990 to 42534 (i.e. only small minority of primop calls are partially applied) which in turn reduces time spent in the garbage collector.
2020-02-24 02:32:01 +02:00
}
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
void ExprLet::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
{
add ExprAttrs::AttrDef::chooseByKind in place of inherited() — not quite useful yet since we don't distinguish plain and inheritFrom attr kinds so far.
2024-01-27 17:33:34 +02:00
auto newEnv = [&] () -> std::shared_ptr<const StaticEnv> {
auto newEnv = std::make_shared<StaticEnv>(nullptr, env.get(), attrs->attrs.size());
Fix whitespace
2013-09-02 17:29:15 +03:00
add ExprAttrs::AttrDef::chooseByKind in place of inherited() — not quite useful yet since we don't distinguish plain and inheritFrom attr kinds so far.
2024-01-27 17:33:34 +02:00
Displacement displ = 0;
for (auto & i : attrs->attrs)
newEnv->vars.emplace_back(i.first, i.second.displ = displ++);
return newEnv;
}();
StaticEnv: Use std::vector instead of std::map
2020-02-21 19:31:16 +02:00
// No need to sort newEnv since attrs->attrs is in sorted order.
Fix whitespace
2013-09-02 17:29:15 +03:00
evaluate inherit (from) exprs only once per directive desugaring inherit-from to syntactic duplication of the source expr also duplicates side effects of the source expr (such as trace calls) and expensive computations (such as derivationStrict).
2024-01-27 17:33:34 +02:00
auto inheritFromEnv = attrs->bindInheritSources(es, newEnv);
OCD: foreach -> C++11 ranged for
2015-07-17 20:24:28 +03:00
for (auto & i : attrs->attrs)
evaluate inherit (from) exprs only once per directive desugaring inherit-from to syntactic duplication of the source expr also duplicates side effects of the source expr (such as trace calls) and expensive computations (such as derivationStrict).
2024-01-27 17:33:34 +02:00
i.second.e->bindVars(es, i.second.chooseByKind(newEnv, env, inheritFromEnv));
Fix whitespace
2013-09-02 17:29:15 +03:00
Expose locals from `let` expressions to the debugger
2024-02-03 05:31:18 +02:00
if (es.debugRepl)
es.exprEnvs.insert(std::make_pair(this, newEnv));
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
body->bindVars(es, newEnv);
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
}
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
void ExprWith::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
{
remove redundant 'debugMode' flag
2022-05-23 06:45:24 +03:00
if (es.debugRepl)
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
es.exprEnvs.insert(std::make_pair(this, env));
storing staticenv bindings
2021-10-02 22:47:36 +03:00
reduce the size of Env by one pointer since `up` and `values` are both pointer-aligned the type field will also be pointer-aligned, wasting 48 bits of space on most machines. we can get away with removing the type field altogether by encoding some information into the `with` expr that created the env to begin with, reducing the GC load for the absolutely massive amount of single-entry envs we create for lambdas. this reduces memory usage of system eval by quite a bit (reducing heap size of our system eval from 8.4GB to 8.23GB) and gives similar savings in eval time. running `nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` before: Time (mean ± σ): 5.576 s ± 0.003 s [User: 5.197 s, System: 0.378 s] Range (min … max): 5.572 s … 5.581 s 10 runs after: Time (mean ± σ): 5.408 s ± 0.002 s [User: 5.019 s, System: 0.388 s] Range (min … max): 5.405 s … 5.411 s 10 runs
2023-12-22 19:19:53 +02:00
parentWith = nullptr;
for (auto * e = env.get(); e && !parentWith; e = e->up)
parentWith = e->isWith;
* Implemented withs.
2010-04-14 18:01:04 +03:00
/* Does this `with' have an enclosing `with'? If so, record its
* Simplify the implementation of `with'. This gives a 7% speedup in evaluating the NixOS system configuration.
2010-04-22 18:08:09 +03:00
level so that `lookupVar' can look up variables in the previous
`with' if this one doesn't contain the desired attribute. */
* Implemented withs.
2010-04-14 18:01:04 +03:00
const StaticEnv * curEnv;
Add level / displacement types
2020-02-24 15:33:01 +02:00
Level level;
* Simplify the implementation of `with'. This gives a 7% speedup in evaluating the NixOS system configuration.
2010-04-22 18:08:09 +03:00
prevWith = 0;
shared_ptr for StaticEnv
2021-09-14 19:49:22 +03:00
for (curEnv = env.get(), level = 1; curEnv; curEnv = curEnv->up, level++)
* Implemented withs.
2010-04-14 18:01:04 +03:00
if (curEnv->isWith) {
prevWith = level;
break;
}
Fix whitespace
2013-09-02 17:29:15 +03:00
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
attrs->bindVars(es, env);
reduce the size of Env by one pointer since `up` and `values` are both pointer-aligned the type field will also be pointer-aligned, wasting 48 bits of space on most machines. we can get away with removing the type field altogether by encoding some information into the `with` expr that created the env to begin with, reducing the GC load for the absolutely massive amount of single-entry envs we create for lambdas. this reduces memory usage of system eval by quite a bit (reducing heap size of our system eval from 8.4GB to 8.23GB) and gives similar savings in eval time. running `nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` before: Time (mean ± σ): 5.576 s ± 0.003 s [User: 5.197 s, System: 0.378 s] Range (min … max): 5.572 s … 5.581 s 10 runs after: Time (mean ± σ): 5.408 s ± 0.002 s [User: 5.019 s, System: 0.388 s] Range (min … max): 5.405 s … 5.411 s 10 runs
2023-12-22 19:19:53 +02:00
auto newEnv = std::make_shared<StaticEnv>(this, env.get());
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
body->bindVars(es, newEnv);
* Memoise checkVarDefs since internally produced terms (i.e., not the result of parsing) can have very heavy sharing, causing exponential complexity if we naively recurse into them. ATerms are graphs, not trees!
2005-11-04 17:17:05 +02:00
}
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
void ExprIf::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
{
remove redundant 'debugMode' flag
2022-05-23 06:45:24 +03:00
if (es.debugRepl)
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
es.exprEnvs.insert(std::make_pair(this, env));
storing staticenv bindings
2021-10-02 22:47:36 +03:00
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
cond->bindVars(es, env);
then->bindVars(es, env);
else_->bindVars(es, env);
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
}
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
void ExprAssert::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
{
remove redundant 'debugMode' flag
2022-05-23 06:45:24 +03:00
if (es.debugRepl)
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
es.exprEnvs.insert(std::make_pair(this, env));
storing staticenv bindings
2021-10-02 22:47:36 +03:00
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
cond->bindVars(es, env);
body->bindVars(es, env);
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
}
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
void ExprOpNot::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
{
remove redundant 'debugMode' flag
2022-05-23 06:45:24 +03:00
if (es.debugRepl)
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
es.exprEnvs.insert(std::make_pair(this, env));
storing staticenv bindings
2021-10-02 22:47:36 +03:00
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
e->bindVars(es, env);
* After parsing, compute level/displacement pairs for each variable use site, allowing environments to be stores as vectors of values rather than maps. This should speed up evaluation and reduce the number of allocations.
2010-04-14 17:42:32 +03:00
}
* Memoise checkVarDefs since internally produced terms (i.e., not the result of parsing) can have very heavy sharing, causing exponential complexity if we naively recurse into them. ATerms are graphs, not trees!
2005-11-04 17:17:05 +02:00
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
void ExprConcatStrings::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
* Memoise checkVarDefs since internally produced terms (i.e., not the result of parsing) can have very heavy sharing, causing exponential complexity if we naively recurse into them. ATerms are graphs, not trees!
2005-11-04 17:17:05 +02:00
{
remove redundant 'debugMode' flag
2022-05-23 06:45:24 +03:00
if (es.debugRepl)
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
es.exprEnvs.insert(std::make_pair(this, env));
storing staticenv bindings
2021-10-02 22:47:36 +03:00
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
for (auto & i : *this->es)
i.second->bindVars(es, env);
* Verify that all variables in a Nix expression are defined.
2004-02-03 16:45:34 +02:00
}
* Big cleanup of the semantics of paths, strings, contexts, string concatenation and string coercion. This was a big mess (see e.g. NIX-67). Contexts are now folded into strings, so that they don't cause evaluation errors when they're not expected. The semantics of paths has been clarified (see nixexpr-ast.def). toString() and coerceToString() have been merged. Semantic change: paths are now copied to the store when they're in a concatenation (and in most other situations - that's the formalisation of the meaning of a path). So "foo " + ./bla evaluates to "foo /nix/store/hash...-bla", not "foo /path/to/current-dir/bla". This prevents accidental impurities, and is more consistent with the treatment of derivation outputs, e.g., `"foo " + bla' where `bla' is a derivation. (Here `bla' would be replaced by the output path of `bla'.)
2006-10-16 18:55:34 +03:00
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
void ExprPos::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
Add a symbol __curPos that expands to the current source location I.e. an attribute set { file = <string>; line = <int>; column = <int>; }.
2013-11-18 21:14:54 +02:00
{
remove redundant 'debugMode' flag
2022-05-23 06:45:24 +03:00
if (es.debugRepl)
use an expr->StaticEnv table in evalState
2022-05-19 19:48:10 +03:00
es.exprEnvs.insert(std::make_pair(this, env));
Add a symbol __curPos that expands to the current source location I.e. an attribute set { file = <string>; line = <int>; column = <int>; }.
2013-11-18 21:14:54 +02:00
}
* Big cleanup of the semantics of paths, strings, contexts, string concatenation and string coercion. This was a big mess (see e.g. NIX-67). Contexts are now folded into strings, so that they don't cause evaluation errors when they're not expected. The semantics of paths has been clarified (see nixexpr-ast.def). toString() and coerceToString() have been merged. Semantic change: paths are now copied to the store when they're in a concatenation (and in most other situations - that's the formalisation of the meaning of a path). So "foo " + ./bla evaluates to "foo /nix/store/hash...-bla", not "foo /path/to/current-dir/bla". This prevents accidental impurities, and is more consistent with the treatment of derivation outputs, e.g., `"foo " + bla' where `bla' is a derivation. (Here `bla' would be replaced by the output path of `bla'.)
2006-10-16 18:55:34 +03:00
Show function names in error messages Functions in Nix are anonymous, but if they're assigned to a variable/attribute, we can use the variable/attribute name in error messages, e.g. while evaluating `concatMapStrings' at `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/strings.nix:18:25': ...
2013-05-16 20:08:02 +03:00
/* Storing function names. */
rename SymbolIdx -> Symbol, Symbol -> SymbolStr after #6218 `Symbol` no longer confers a uniqueness invariant on the string it wraps, it is now possible to create multiple symbols that compare equal but whose string contents have different addresses. this guarantee is now only provided by `SymbolIdx`, leaving `Symbol` only as a string wrapper that knows about the intricacies of how symbols need to be formatted for output. this change renames `SymbolIdx` to `Symbol` to restore the previous semantics of `Symbol` to that name. we also keep the wrapper type and rename it to `SymbolStr` instead of returning plain strings from lookups into the symbol table because symbols are formatted for output in many places. theoretically we do not need `SymbolStr`, only a function that formats a string for output as a symbol, but having to wrap every symbol that appears in a message into eg `formatSymbol()` is error-prone and inconvient.
2022-04-22 22:45:39 +03:00
void Expr::setName(Symbol name)
Show function names in error messages Functions in Nix are anonymous, but if they're assigned to a variable/attribute, we can use the variable/attribute name in error messages, e.g. while evaluating `concatMapStrings' at `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/strings.nix:18:25': ...
2013-05-16 20:08:02 +03:00
{
}
rename SymbolIdx -> Symbol, Symbol -> SymbolStr after #6218 `Symbol` no longer confers a uniqueness invariant on the string it wraps, it is now possible to create multiple symbols that compare equal but whose string contents have different addresses. this guarantee is now only provided by `SymbolIdx`, leaving `Symbol` only as a string wrapper that knows about the intricacies of how symbols need to be formatted for output. this change renames `SymbolIdx` to `Symbol` to restore the previous semantics of `Symbol` to that name. we also keep the wrapper type and rename it to `SymbolStr` instead of returning plain strings from lookups into the symbol table because symbols are formatted for output in many places. theoretically we do not need `SymbolStr`, only a function that formats a string for output as a symbol, but having to wrap every symbol that appears in a message into eg `formatSymbol()` is error-prone and inconvient.
2022-04-22 22:45:39 +03:00
void ExprLambda::setName(Symbol name)
Show function names in error messages Functions in Nix are anonymous, but if they're assigned to a variable/attribute, we can use the variable/attribute name in error messages, e.g. while evaluating `concatMapStrings' at `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/strings.nix:18:25': ...
2013-05-16 20:08:02 +03:00
{
this->name = name;
body->setName(name);
}
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
std::string ExprLambda::showNamePos(const EvalState & state) const
Show function names in error messages Functions in Nix are anonymous, but if they're assigned to a variable/attribute, we can use the variable/attribute name in error messages, e.g. while evaluating `concatMapStrings' at `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/strings.nix:18:25': ...
2013-05-16 20:08:02 +03:00
{
store Symbols in a table as well, like positions this slightly increases the amount of memory used for any given symbol, but this increase is more than made up for if the symbol is referenced more than once in the EvalState that holds it. on average every symbol should be referenced at least twice (once to introduce a binding, once to use it), so we expect no increase in memory on average. symbol tables are limited to 2³² entries like position tables, and similar arguments apply to why overflow is not likely: 2³² symbols would require as many string instances (at 24 bytes each) and map entries (at 24 bytes or more each, assuming that the map holds on average at most one item per bucket as the docs say). a full symbol table would require at least 192GB of memory just for symbols, which is well out of reach. (an ofborg eval of nixpks today creates less than a million symbols!)
2022-03-05 15:40:24 +02:00
std::string id(name
? concatStrings("'", state.symbols[name], "'")
: "anonymous function");
return fmt("%1% at %2%", id, state.positions[pos]);
Show function names in error messages Functions in Nix are anonymous, but if they're assigned to a variable/attribute, we can use the variable/attribute name in error messages, e.g. while evaluating `concatMapStrings' at `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/strings.nix:18:25': ...
2013-05-16 20:08:02 +03:00
}
printStats(): Print the size of the symbol table in bytes
2013-10-08 16:34:57 +03:00
use byte indexed locations for PosIdx we now keep not a table of all positions, but a table of all origins and their sizes. position indices are now direct pointers into the virtual concatenation of all parsed contents. this slightly reduces memory usage and time spent in the parser, at the cost of not being able to report positions if the total input size exceeds 4GiB. this limit is not unique to nix though, rustc and clang also limit their input to 4GiB (although at least clang refuses to process inputs that are larger, we will not). this new 4GiB limit probably will not cause any problems for quite a while, all of nixpkgs together is less than 100MiB in size and already needs over 700MiB of memory and multiple seconds just to parse. 4GiB worth of input will easily take multiple minutes and over 30GiB of memory without even evaluating anything. if problems *do* arise we can probably recover the old table-based system by adding some tracking to Pos::Origin (or increasing the size of PosIdx outright), but for time being this looks like more complexity than it's worth. since we now need to read the entire input again to determine the line/column of a position we'll make unsafeGetAttrPos slightly lazy: mostly the set it returns is only used to determine the file of origin of an attribute, not its exact location. the thunks do not add measurable runtime overhead. notably this change is necessary to allow changing the parser since apparently nothing supports nix's very idiosyncratic line ending choice of "anything goes", making it very hard to calculate line/column positions in the parser (while byte offsets are very easy).
2024-01-29 07:19:23 +02:00
/* Position table. */
Pos PosTable::operator[](PosIdx p) const
{
auto origin = resolve(p);
if (!origin)
return {};
const auto offset = origin->offsetOf(p);
Pos result{0, 0, origin->origin};
auto lines = this->lines.lock();
auto linesForInput = (*lines)[origin->offset];
if (linesForInput.empty()) {
auto source = result.getSource().value_or("");
const char * begin = source.data();
for (Pos::LinesIterator it(source), end; it != end; it++)
linesForInput.push_back(it->data() - begin);
if (linesForInput.empty())
linesForInput.push_back(0);
}
// as above: the first line starts at byte 0 and is always present
auto lineStartOffset = std::prev(
std::upper_bound(linesForInput.begin(), linesForInput.end(), offset));
result.line = 1 + (lineStartOffset - linesForInput.begin());
result.column = 1 + (offset - *lineStartOffset);
return result;
}
printStats(): Print the size of the symbol table in bytes
2013-10-08 16:34:57 +03:00
/* Symbol table. */
size_t SymbolTable::totalSize() const
{
size_t n = 0;
rename SymbolIdx -> Symbol, Symbol -> SymbolStr after #6218 `Symbol` no longer confers a uniqueness invariant on the string it wraps, it is now possible to create multiple symbols that compare equal but whose string contents have different addresses. this guarantee is now only provided by `SymbolIdx`, leaving `Symbol` only as a string wrapper that knows about the intricacies of how symbols need to be formatted for output. this change renames `SymbolIdx` to `Symbol` to restore the previous semantics of `Symbol` to that name. we also keep the wrapper type and rename it to `SymbolStr` instead of returning plain strings from lookups into the symbol table because symbols are formatted for output in many places. theoretically we do not need `SymbolStr`, only a function that formats a string for output as a symbol, but having to wrap every symbol that appears in a message into eg `formatSymbol()` is error-prone and inconvient.
2022-04-22 22:45:39 +03:00
dump([&] (const std::string & s) { n += s.size(); });
printStats(): Print the size of the symbol table in bytes
2013-10-08 16:34:57 +03:00
return n;
}
* Use a proper namespace. * Optimise header file usage a bit. * Compile the parser as C++.
2006-09-05 00:06:23 +03:00
}
Reference in a new issue Copy permalink