Commit graph

2643 commits

Author SHA1 Message Date
Eelco Dolstra
31ebc6028b Fix symlink handling
This restores the symlink handling behaviour prior to
94812cca98.

Fixes #9298.
2023-11-16 16:45:14 +01:00
John Ericson
6c8f4ef350 Allow installing unit tests
Closes #9343

See that issue for motivation.

Installing these is disabled by default, but we enable it (and the
additional output we want isntall these too so as not to clutter the
existing ones) to use in cross builds and dev shells.
2023-11-16 09:55:42 -05:00
Robert Hensing
4e27f1947a libexpr: Reduce nonRecursiveStackReservation
128 is still beyond the point where the allocation overhead is
insignificant, but we don't anticipate to overflow for these
use cases, so it's fine.
2023-11-16 15:38:21 +01:00
Robert Hensing
a96be29db5 removeAttrs: increase stack reservation to 64 2023-11-16 15:38:21 +01:00
Robert Hensing
1b9813e4e6 primops: Name stack reservation limits 2023-11-16 15:38:21 +01:00
Robert Hensing
898c47384f primops: Err on the side of less stack usage
Try to stay away from stack overflows.

These small vectors use stack space. Most instances will not need
to allocate because in general most things are small, and large
things are worth heap allocating.

16 * 3 * word = 384 bytes is still quite a bit, but these functions
tend not to be part of deep recursions.
2023-11-16 15:38:21 +01:00
Robert Hensing
91114a6fa4 ExprCall::eval: Heap allocate at arity 5+ 2023-11-16 15:38:21 +01:00
Robert Hensing
206ece0f41 builtins.{any,all}: Use constant errorCtx
Clang warned that the expanded code used to have a buffer overflow.
Very strange, but also very avoidable.
2023-11-16 15:38:21 +01:00
Robert Hensing
12c91a823e maxPrimOpArity: 64 -> 8
This makes stack usage significantly more compact, allowing larger
amounts of data to be processed on the same stack.

PrimOp functions with more than 8 positional (curried) arguments
should use an attrset instead.
2023-11-16 15:38:21 +01:00
Robert Hensing
0daccb1121 libexpr: Check primop arity earlier 2023-11-16 12:28:32 +01:00
Théophane Hufschmitt
ba3cb4a049 Remove all the occurences of VLAs
There's generally no strict reason for using them, and they are somewhat
fishy, so let's avoid them.
2023-11-16 12:27:30 +01:00
Théophane Hufschmitt
5196613e82 Use boost small vectors instead of VLAs in the primops
VLAs are a dangerous feature, and their usage triggers an undefined
behavior since theire size can be zero in some cases.
So replace them with `boost::small_vector`s which fit the same goal but
are safer.

It's also incidentally consistently 1% faster on the benchmarks.
2023-11-16 12:27:30 +01:00
John Ericson
f0adb72c23 Mark fetchTree as unstable again
As discussed in our last meeting, we need a bit more time, but we are
"time boxing" the work left to do to ensure there is not unbounded
delay.

Rather than putting it back underneath `flakes`, though, put it
underneath its own `fetch-tree` experimental feature (which `flakes`
includes/implies). This signals our commitment to the plan to stabilize
it first without waiting to go through the rest of Flakes, and also will
give users a "release candidate" when we get closer to stabilization.

This reverts commit 4112dd1fc9.
2023-11-08 23:23:56 -05:00
Mel Zuser
61d6fe059e
Fix boost::bad_format_string exception in builtins.addErrorContext (#9291)
* Fix boost::bad_format_string exception in builtins.addErrorContext

The message passed to addTrace was incorrectly being used as a format
string and this this would cause an exception when the string contained
a '%', which can be hit in places where arbitrary file paths are
interpolated.

* add test
2023-11-06 19:13:40 +00:00
John Ericson
ac89bb064a Split up util.{hh,cc}
All OS and IO operations should be moved out, leaving only some misc
portable pure functions.

This is useful to avoid copious CPP when doing things like Windows and
Emscripten ports.

Newly exposed functions to break cycles:

 - `restoreSignals`
 - `updateWindowSize`
2023-11-05 12:20:02 -05:00
John Ericson
dde1d86338 Restrict some code to StoreDirConfig
- part of eval cache
 - part of derivations
 - derived path
 - store path with outputs
 - serializers
2023-11-04 19:05:36 -04:00
BootRhetoric
098f0615c9
fetchGit and flake: add publicKeys list input
This adds publicKeys as an optional fetcher input attribute to flakes
and builtins.fetchGit to provide a nix interface for the json-encoded
`publicKeys` attribute of the git fetcher.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-11-03 20:23:31 +01:00
Eelco Dolstra
cdb27c1519 SourceAccessor: Change the main interface from lstat() to maybeLstat() 2023-11-01 15:26:07 +01:00
Felix Uhl
c7dcdb8325 Overhaul nix flake update and lock commands
Closes #5110
2023-10-31 15:33:57 +01:00
Felix Uhl
1fd0867389 Fix missing output when creating lockfile 2023-10-31 15:33:50 +01:00
Eelco Dolstra
fa6bc33604
Merge pull request #9239 from edolstra/putFile
Input: Replace markFileChanged() by putFile()
2023-10-31 14:49:42 +01:00
Eelco Dolstra
05316d401f Cleanup 2023-10-30 17:03:06 +01:00
Silvan Mosberger
46028ff764
doc: Fix fetchGit default name (#9241) 2023-10-26 07:05:48 +02:00
Eelco Dolstra
15c430f389 Remove unused LockFile::write() 2023-10-25 18:44:09 +02:00
Eelco Dolstra
95d657c8b3 Input: Replace markFileChanged() by putFile()
Committing a lock file using markFileChanged() required the input to
be writable by the caller in the local filesystem (using the path
returned by getSourcePath()). putFile() abstracts over this.
2023-10-25 18:30:29 +02:00
Silvan Mosberger
f269911641
Document builtins.substring negative length behavior (#9226) 2023-10-24 11:22:02 +02:00
John Ericson
765436e300 Add builtins.addDrvOutputDependencies
End goal: make `(mkDerivation x).drvPath` behave like a non-DrvDeep
context.

Problem: users won't be able to recover the DrvDeep behavior when
nixpkgs makes this change.

Solution: add this primop.

The new primop is fairly simple, and is supposed to complement other
existing ones (`builtins.storePath`, `builtins.outputOf`) so there are
simple ways to construct strings with every type of string context
element.

(It allows nothing we couldn't already do with `builtins.getContext` and `builtins.appendContext`, which is also true of those other two primops.)

This was originally in #8595, but then it was proposed to land some doc
changes separately. So now the code changes proper is just moved to
this, and the doc will be done in that.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.nore
github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io
2023-10-23 12:49:14 -04:00
Eelco Dolstra
955bbe53c5
Merge pull request #9177 from edolstra/input-accessors
Backport FSInputAccessor and MemoryInputAccessor from lazy-trees
2023-10-23 11:42:04 +02:00
Eelco Dolstra
935c9981de Remove fetchers::Tree and move tarball-related stuff into its own header 2023-10-20 19:56:52 +02:00
Eelco Dolstra
bcf5c31950 Add future FIXME 2023-10-20 16:58:33 +02:00
Eelco Dolstra
173abec0bc coerceToPath(): Handle __toString, add tests 2023-10-20 13:06:44 +02:00
Eelco Dolstra
5be7705ddf Remove stuff we don't need yet 2023-10-19 19:20:21 +02:00
Johannes Kirschbauer
9bc7b4f463
doc: generic closure supported key types (#9183)
* doc: generic closure supported key types

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-10-19 12:39:41 +00:00
Eelco Dolstra
06c57899e3 Remove FIXME 2023-10-19 14:22:05 +02:00
Eelco Dolstra
f16af08e83 Fix macOS compilation 2023-10-19 14:20:50 +02:00
Eelco Dolstra
12214fef09 InputAccessor::fetchToStore(): Support arbitrary ingestion methods 2023-10-19 14:19:10 +02:00
Yueh-Shun Li
5088e6563a primops: add builtins.convertHash
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-10-19 00:58:56 +08:00
Yueh-Shun Li
e026f3e1ae treewide: Reference HashFormat members with scope
Base* -> HashFormat::Base*
2023-10-19 00:56:41 +08:00
Yueh-Shun Li
aff177d860 Elaborate the "unknown hash algorithm" error
List the allowed hash formats
2023-10-19 00:38:32 +08:00
Eelco Dolstra
df73c6eb8c Introduce MemoryInputAccessor and use it for corepkgs
MemoryInputAccessor is an in-memory virtual filesystem that returns
files like <nix/fetchurl.nix>. This removes the need for special hacks
to handle those files.
2023-10-18 17:38:11 +02:00
Eelco Dolstra
ea38605d11 Introduce FSInputAccessor and use it
Backported from the lazy-trees branch. Note that this doesn't yet use
the access control features of FSInputAccessor.
2023-10-18 17:37:32 +02:00
Vladimir Kryachko
311e2ad024 Address review comments 2023-10-18 10:37:06 -04:00
Vladimir Kryachko
a2f0ba6a6d Fix transitive input locking.
Fixes reproducibility issue described in #9143

Fixes #9143
2023-10-18 10:31:36 -04:00
Eelco Dolstra
201c115c3e
Merge pull request #9151 from edolstra/stabilize-fetchTree
Stabilize fetchTree
2023-10-18 10:54:08 +02:00
Théophane Hufschmitt
c1a1766c46
Merge pull request #9169 from vkryachko/follow_cycle
Detect cycles in flake follows.
2023-10-18 07:34:03 +02:00
Eelco Dolstra
f62b5500ff fetchTree: Require the flakes experimental feature for the URL syntax 2023-10-17 14:52:34 +02:00
Jacek Galowicz
54b350d517 Drop some moves that would happen anyway but forbid NRVO where appicable 2023-10-16 21:48:35 +01:00
Jacek Galowicz
abf7df2b37 Fix moves that accidentally copy anyway 2023-10-16 21:48:35 +01:00
Vladimir Kryachko
d6066c90f8 Don't convert InputPaths to strings prematurely. 2023-10-16 15:47:28 -04:00
Vladimir Kryachko
b3fd7db63f Detect cycles in flake follows.
This change results in an error thrown as opposed to segfaulting due to
stack overflow.

Fixes #9144
2023-10-16 13:00:49 -04:00
Eelco Dolstra
4112dd1fc9 Mark fetchTree as stable 2023-10-13 16:45:08 +02:00
Eelco Dolstra
8eb4f735dc fetchTree: Only use the registry if flakes are enabled 2023-10-13 14:34:32 +02:00
Eelco Dolstra
4ce7a53a9c Update fetchTree docs 2023-10-13 14:34:32 +02:00
Eelco Dolstra
856fe13533 fetchTree cleanup
Two changes:

* The (probably unintentional) hack to handle paths as tarballs has
  been removed. This is almost certainly not what users expect and is
  inconsistent with flakeref handling everywhere else.

* The hack to support scp-style Git URLs has been moved to the Git
  fetcher, so it's now supported not just by fetchTree but by flake
  inputs.
2023-10-13 14:34:23 +02:00
Michal Sojka
db0d94560b Document builtins.fetchTree
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

Supersedes #6740
2023-10-13 14:24:10 +02:00
Valentin Gagarin
7642894a4e reword documentation on lookup path resolution 2023-10-10 00:49:27 +02:00
John Ericson
838be5e4a0
Merge pull request #9114 from fricklerhandwerk/lookup-path
introduce lookup paths as a distinct language construct
2023-10-09 11:28:40 -04:00
Robert Hensing
aeea49609b
Merge pull request #8895 from hercules-ci/gc-before-stats
eval: Run a full GC before printing stats
2023-10-09 17:14:45 +02:00
John Ericson
3953901796
Merge pull request #9112 from fricklerhandwerk/restrict-eval-nix-path
describe the effect of `restrict-eval` in a more focused manner
2023-10-09 11:07:21 -04:00
John Ericson
a7a9386c96
Merge pull request #9111 from fricklerhandwerk/impure-constants
document that pure-eval also disables `builtins.nixPath`
2023-10-09 11:04:47 -04:00
John Ericson
217d863f7a Merge branch 'master' into lookup-path 2023-10-09 10:54:54 -04:00
Robert Hensing
c32084a12c printStats -> maybePrintStats 2023-10-09 16:34:35 +02:00
Valentin Gagarin
f00a5eb11b introduce lookup paths as a distinct language construct
so far they did not really have a name, and were at best referred to as
"angle bracket syntax".
2023-10-07 04:44:09 +02:00
Valentin Gagarin
6305801626 reword and reformat description of builtins.import 2023-10-07 03:57:08 +02:00
Valentin Gagarin
2fe1ccf797 describe the effect of restrict-eval in a more focused manner 2023-10-07 03:20:20 +02:00
Valentin Gagarin
a86a3e5e59 document that pure-eval also disables builtins.nixPath 2023-10-07 03:12:25 +02:00
Valentin Gagarin
a67cee965a expand on interpolated expressions 2023-10-07 02:49:55 +02:00
Robert Hensing
f95364a803 eval: Run a full GC before printing stats
This makes the numbers more deterministic, especially when it comes
to the final heap size.
2023-10-06 00:21:05 +02:00
Robert Hensing
f8a3893e8d pathExists: isDir when endswith /. 2023-09-30 02:35:26 +01:00
John Ericson
b912f3a937 Move flakeIdRegex{,S} from libutil to flakeref.{cc,hh
It isn't used, and doesn't belong in `libutil`.
2023-09-28 20:55:41 -04:00
John Ericson
cede94dbf7 builtins.fetchTree: Mark experimental the new way
This helps ensure uniform docs/error message.
2023-09-28 20:51:25 -04:00
John Ericson
b7e712f9fd
Merge pull request #8509 from wentasah/fetch-tree-doc
Document fetchTree
2023-09-28 15:13:53 -04:00
Robert Hensing
13a9090ffc
Merge pull request #9047 from flox/tomberek.string_refactor
string Value refactor
2023-09-28 02:58:57 +01:00
Robert Hensing
16a6ea7249
Merge pull request #9049 from inclyc/users/inclyc/move-path
libexpr: construct ExprPath by move ctor, not copy cotr
2023-09-27 22:30:44 +01:00
Tom Bereknyei
399ef84420 refactor: use string accessors
Create context, string_view, and c_str, accessors throughout in order to
better support improvements to the underlying string representation.
2023-09-27 00:33:01 -04:00
Yingchi Long
5b902ce9d6 libexpr: construct ExprPath by move ctor, not copy cotr 2023-09-26 23:30:32 +08:00
Robert Hensing
57202969d0
Merge pull request #9040 from waalge/waalge/tail-docstr
fix docstring
2023-09-26 15:32:00 +01:00
Valentin Gagarin
b17f200b11
Document "Import From Derivation" (#7332)
* document "Import From Derivation"

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-09-26 01:49:03 +00:00
Tom Bereknyei
7e24dc606b fix(tests): fix assumption that string.s is a char* 2023-09-25 21:37:32 -04:00
Théophane Hufschmitt
9a78d87bc0
Merge pull request #6614 from RasmusRendal/spaces
Implement support for percent encoded filepaths for flakerefs
2023-09-26 02:27:09 +02:00
Robert Hensing
b19bd4f348
Merge pull request #8970 from hercules-ci/eval-stuff
Expr: remove redundant fields, add nrExprs
2023-09-25 19:49:22 +02:00
Robert Hensing
bd24176ac5
libexpr/nixexpr.hh: Remove redundant inline
This is redundant since definitions in C++ record are implicitly inline-ed.

Co-authored-by: Yingchi Long <i@lyc.dev>
2023-09-25 17:51:17 +01:00
waalge
70b5e6050c
fix docstring 2023-09-25 13:39:11 +00:00
Yingchi Long
e4b83fbfe2 libexpr: const rvalue reference -> value for nix::Expr nodes 2023-09-24 14:54:41 +08:00
Théophane Hufschmitt
e8113747e1 Split the parseFlakeRefWithFragment function
Was starting to be very complex and hard to follow.
Now the different cases should be easier to understand.
2023-09-22 10:06:43 +02:00
Théophane Hufschmitt
50e61f579c Allow special characters in flake paths
Support using nix flakes in paths with spaces or abitrary unicode characters.
This introduces the convention that the path part of the URL should be
percent-encoded when dealing with `path:` urls and not when using
filepaths (following the convention of firefox).

Co-authored-by: Rendal <rasmus@rend.al>
2023-09-22 10:06:43 +02:00
Eelco Dolstra
126e2645f2 Disable rapidcheck tests in the coverage run
https://hydra.nixos.org/build/233688539
2023-09-19 16:04:00 +02:00
Robert Hensing
bf8deb4991 Expr: remove redundant int and float fields 2023-09-12 13:45:45 +02:00
Robert Hensing
3720e811fa libexpr: Add nrExprs to NIX_SHOW_STATS 2023-09-12 13:21:55 +02:00
Matthew Kenigsberg
07545add53
Drop dead code
localPath is unused
2023-09-10 12:18:03 +02:00
Robert Hensing
5f55c33917
Merge pull request #8944 from fricklerhandwerk/fix-anchor
fix invalid anchor link
2023-09-07 17:58:16 +02:00
John Ericson
7ad66cb3ef Allow dynamic derivation deps in inputDrvs
We use the same nested map representation we used for goals, again in
order to save space. We might someday want to combine with `inputDrvs`,
by doing `V = bool` instead of `V = std::set<OutputName>`, but we are
not doing that yet for sake of a smaller diff.

The ATerm format for Derivations also needs to be extended, in addition
to the in-memory format. To accomodate this, we added a new basic
versioning scheme, so old versions of Nix will get nice errors. (And
going forward, if the ATerm format changes again the errors will be even
better.)

`parsedStrings`, an internal function used as part of parsing
derivations in A-Term format, used to consume the final `]` but expect
the initial `[` to already be consumed. This made for what looked like
unbalanced brackets at callsites, which was confusing. Now it consumes
both which is hopefully less confusing.

As part of testing, we also created a unit test for the A-Term format for
regular non-experimental derivations too.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

Apply suggestions from code review

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-07 10:39:37 -04:00
Valentin Gagarin
829d4d3e03 fix invalid anchor link 2023-09-07 15:13:22 +02:00
Eelco Dolstra
b88784278f
Merge pull request #8869 from hercules-ci/fix-issue-8838-pathExists-isDir
Fix #8838, pathExists: isDir when ends with `/ `
2023-09-01 13:15:54 +02:00
Robert Hensing
1e08e12d81 pathExists: isDir when endswith /
Fixes https://github.com/NixOS/nix/issues/8838
2023-08-25 17:18:37 +02:00
John Ericson
50f40ac4c0
Merge pull request #8829 from obsidiansystems/build-dynamic-derivations
Adapt scheduler to work with dynamic derivations
2023-08-25 11:13:15 -04:00
tomberek
b563ef38cc
Merge pull request #8819 from VertexA115/fix/deep-follow-paths
Fix follow path checking at depths greater than 2
2023-08-25 10:33:05 -04:00
John Ericson
2f5d3da806 Introduce OutputName and OutputNameView type aliases
Hopefully they make the code easier to understand!
2023-08-25 09:55:07 -04:00
John Ericson
fe71faa920 Delete EvalState::addToSearchPath
This function is now trivial enough that it doesn't need to exist.

`EvalState` can still be initialized with a custom search path, but we
don't have a need to mutate the search path after it has been
constructed, and I don't see why we would need to in the future.

Fixes #8229
2023-08-18 14:04:33 -04:00
John Ericson
665ad4f7c5
Merge pull request #8839 from obsidiansystems/string-context-7479
Refactor Raw pattern, part of #7479
2023-08-18 13:47:01 -04:00