Commit graph

2684 commits

Author SHA1 Message Date
Robert Hensing
fcf09813c6
Merge pull request #6236 from obsidiansystems/store-dir-config
Factor out `StoreDirConfig`
2023-12-01 15:38:14 +01:00
Robert Hensing
188c803ddb
Merge pull request #9508 from infinisil/revert-7762-boost-regex
Revert "Switch from std::regex to boost::regex"
2023-12-01 02:08:58 +01:00
Silvan Mosberger
333ea684b0 Add boost::regex regression test 2023-12-01 01:39:52 +01:00
Valentin Gagarin
4781e7fa70 Document each store type on its own page
This makes for more useful manual table of contents, that displays the
information at a glance.

The `nix help-stores` command is kept as-is, even though it will show up
in the manual with the same information as these pages due to the way it
is written as a "`--help`-style" command. Deciding what to do with that
command is left for a later PR.

This change also lists all store types at the top of the respective overview page.

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems
2023-12-01 01:27:52 +01:00
Silvan Mosberger
908a011a4a
Revert "Switch from std::regex to boost::regex" 2023-12-01 00:50:20 +01:00
Eelco Dolstra
8cafc754d8 Move access control from FSInputAccessor to FilteringInputAccessor 2023-11-30 21:54:53 +01:00
Eelco Dolstra
43d9fb6cf1 Remove InputAccessor::root() 2023-11-30 16:44:54 +01:00
Eelco Dolstra
305939655a Remove superfluous use of hasAccessControl() 2023-11-30 16:28:33 +01:00
Eelco Dolstra
ea95327e72 Move restricted/pure-eval access control out of the evaluator and into the accessor 2023-11-30 16:16:17 +01:00
Eelco Dolstra
cb7f25869d
Merge pull request #9430 from hercules-ci/remove-vlas
Fix stack overflow in `filter`
2023-11-30 15:31:44 +01:00
John Ericson
52e0911302 Use buildprefix in a few more places
`installcheck` doesn't yet work, but the rest of the build can now
happen mostly inside a separate build directory.

Progress on #9342

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-11-29 19:49:07 -05:00
Robert Hensing
f0180487a0
Merge pull request #9462 from trofi/eval.o-dependency
libexpr: add missing dependency on 'flake/call-flake.nix.gen.hh'
2023-11-27 17:25:21 +01:00
Robert Hensing
68c48756fe libexpr/local.mk: Make eval compile deps regular
Dependency is now entirely through the eval.cc rule.
All gen.hh deps are now there.
2023-11-27 15:52:24 +01:00
Sergei Trofimovich
75134b7513 libexpr: add missing dependency on 'flake/call-flake.nix.gen.hh'
Without the change build for `eval.o` fails occasionally as:

    $ make src/libexpr/eval.o
      GEN    Makefile.config
      GEN    src/libexpr/primops/derivation.nix.gen.hh
      GEN    src/libexpr/fetchurl.nix.gen.hh
      GEN    src/libexpr/parser-tab.cc
      GEN    src/libexpr/lexer-tab.cc
    src/libexpr/lexer.l:314: warning, -s option given but default rule can be matched
      CXX    src/libexpr/eval.o
    src/libexpr/eval.cc:519:18: fatal error: flake/call-flake.nix.gen.hh: No such file or directory
      519 |         #include "flake/call-flake.nix.gen.hh"
          |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    compilation terminated.
    make: *** [mk/patterns.mk:3: src/libexpr/eval.o] Error 1

Noticed in https://github.com/NixOS/nixpkgs/pull/269439
2023-11-27 08:56:24 +00:00
Yorick van Pelt
d2f5e263e3
Switch from std::regex to boost::regex 2023-11-25 15:14:18 +01:00
Théophane Hufschmitt
7f626dba33
Merge pull request #9398 from Qyriad/fixes/flake-not-found
flakes: bare minimum fix the error message for untracked flake.nix
2023-11-24 15:33:08 +01:00
Philipp Schuster
5be0e6b314 doc: primops: fix typo 2023-11-24 10:50:01 +01:00
Philipp Schuster
2ce8c9650b
doc: primops: add more info for foldl (#9254)
* doc: primops: add more info for foldl

From the existing doc it is not obvious whether the first or the
second argument is the accumulator. This is however relevant to
know, as for certain scenarios, this might change the behavior.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-11-23 21:02:20 +00:00
Théophane Hufschmitt
bf13943206
Merge pull request #9173 from vkryachko/transitive-input-overrides
Fix transitive input locking.
2023-11-23 08:54:45 +01:00
John Ericson
46131567da Add missing -lrapidcheck fixing build with shared lib
https://github.com/NixOS/nixpkgs/pull/269064 makes rapidcheck be build
as a shared lib, but that broke Nix because the `-lrapidcheck` was
missing. This fixes that (and doesn't break Nix what the library is a
static archive as today).
2023-11-21 23:19:25 -05:00
Robert Hensing
4e790efade Use boost::container::small_vector in place of VLAs 2023-11-21 20:52:27 +01:00
Eelco Dolstra
a0162d5732 Improve SourceAccessor path display
Backported from lazy-trees. This allows SourceAccessors to show the
origin of the accessor. E.g. we now get

  copying '«git+https://github.com/blender/blender.git?ref=refs/heads/main&rev=4edc1389337dd3679ff66969c332d2aff52e1992»/' to the store

instead of

  copying '/' to the store
2023-11-20 18:54:36 +01:00
Théophane Hufschmitt
cfc6494d57
Merge pull request #9390 from DavHau/git-shallow-docs
fetchTree: clarify docs for shallow flag
2023-11-20 15:15:25 +01:00
Eelco Dolstra
1d6abec993 Revert use of boost::container::small_vector in the evaluator
It caused random crashes (https://hydra.nixos.org/build/241514506,
https://hydra.nixos.org/build/241443330) because the heap allocation
done by small_vector in the not-small case is not scanned for GC
roots.
2023-11-20 12:35:35 +01:00
Qyriad
19993398a1 flakes: check for flake.nix before complaining that lstat on it fails
getFlake currently calls lstat (via isLink via canonPath) before it
performs the sanity check that a flake.nix exists in the first place.
This commit moves the check to before path canonicalization, so that
failed symlink check operations don't throw before the check does.
2023-11-20 03:37:02 -07:00
tomberek
fb68699456
Merge pull request #9370 from hercules-ci/add-value-types
refactor: Add `Value` types, use `std::span` for list iteration
2023-11-20 01:32:32 -05:00
nicoo
d5928085d5 builtins.concatMap: Fix typo in error message 2023-11-19 19:57:07 +01:00
DavHau
796a7eb92d fetchTree: clarify docs for shallow flag 2023-11-19 20:32:23 +07:00
Silvan Mosberger
70ddf298e0 doc: Add link to filterSource from path 2023-11-19 04:09:14 +01:00
Robert Hensing
4a539ac3ea Fix buildNoGc
Fixes https://hydra.nixos.org/build/241067941/nixlog/1

    src/libexpr/eval.cc:1776:54: error: variable 'boost::container::small_vector<nix::Value*, 4> vArgs' has initializer but incomplete type
2023-11-17 17:38:08 +01:00
Eelco Dolstra
3a7f024a9c
Merge pull request #9363 from edolstra/symlink-regression
Fix symlink handling
2023-11-17 14:11:50 +01:00
Robert Hensing
260c614762 Value: use std::span, change use of const
**`Value` and `const`**

These two deserve some explanation. We'll get to lists later.

Values can normally be thought of as immutable, except they are
are also the vehicle for call by need, which must be implemented
using mutation.

This circumstance makes a `const Value` a rather useless thing:

 - If it's a thunk, you can't evaluate it, except by copying, but
   that would not be call by need.

 - If it's not a thunk, you know the type, so the method that
   acquired it for you should have returned something more specific,
   such as a `const Bindings &` (which actually does make sense
   because that's an immutable span of pointers to mutable `Value`s.

 - If you don't care about the type yet, you might establish the
   convention that `const Value` means `deepSeq`-ed data, but
   this is hardly useful and not actually as safe as you would
   supposedly want to trust it to be - just convention.

**Lists**

`std::span` is a tuple of pointer and size - just what we need.

We don't return them as `const Value`, because considering the
first bullet point we discussed before, we'd have to force all
the list values, which isn't what we want.

So what we end up with is a nice representation of a list in
weak head normal form: the spine is immutable, but the
items may need some evaluation later.
2023-11-17 10:19:03 +01:00
Robert Hensing
7055c65285 Value: extract Value::Lambda 2023-11-17 10:19:03 +01:00
Robert Hensing
6af1d9f7b9 Value: extract Value::FunctionApplicationThunk 2023-11-17 10:19:03 +01:00
Robert Hensing
b55203e874 Value: extract Value::ClosureThunk 2023-11-17 10:19:03 +01:00
Robert Hensing
d8ff5cfe8e Value: extract Value::Path 2023-11-17 10:19:03 +01:00
Robert Hensing
2eb59c34b5 Value: extract Value::StringWithContext 2023-11-17 10:19:03 +01:00
Théophane Hufschmitt
fabae98ab4
Merge pull request #9360 from obsidiansystems/install-unit-tests
Allow installing unit tests
2023-11-16 21:25:21 +01:00
Théophane Hufschmitt
ac4431e9d0
Merge pull request #7348 from thufschmitt/dont-use-vlas
Remove the usage of VLAs in the code
2023-11-16 19:05:37 +01:00
Eelco Dolstra
96d67620d5 Fix a broken generated header file dependency
https://hydra.nixos.org/build/240882042
2023-11-16 17:12:06 +01:00
Eelco Dolstra
31ebc6028b Fix symlink handling
This restores the symlink handling behaviour prior to
94812cca98.

Fixes #9298.
2023-11-16 16:45:14 +01:00
John Ericson
6c8f4ef350 Allow installing unit tests
Closes #9343

See that issue for motivation.

Installing these is disabled by default, but we enable it (and the
additional output we want isntall these too so as not to clutter the
existing ones) to use in cross builds and dev shells.
2023-11-16 09:55:42 -05:00
Robert Hensing
4e27f1947a libexpr: Reduce nonRecursiveStackReservation
128 is still beyond the point where the allocation overhead is
insignificant, but we don't anticipate to overflow for these
use cases, so it's fine.
2023-11-16 15:38:21 +01:00
Robert Hensing
a96be29db5 removeAttrs: increase stack reservation to 64 2023-11-16 15:38:21 +01:00
Robert Hensing
1b9813e4e6 primops: Name stack reservation limits 2023-11-16 15:38:21 +01:00
Robert Hensing
898c47384f primops: Err on the side of less stack usage
Try to stay away from stack overflows.

These small vectors use stack space. Most instances will not need
to allocate because in general most things are small, and large
things are worth heap allocating.

16 * 3 * word = 384 bytes is still quite a bit, but these functions
tend not to be part of deep recursions.
2023-11-16 15:38:21 +01:00
Robert Hensing
91114a6fa4 ExprCall::eval: Heap allocate at arity 5+ 2023-11-16 15:38:21 +01:00
Robert Hensing
206ece0f41 builtins.{any,all}: Use constant errorCtx
Clang warned that the expanded code used to have a buffer overflow.
Very strange, but also very avoidable.
2023-11-16 15:38:21 +01:00
Robert Hensing
12c91a823e maxPrimOpArity: 64 -> 8
This makes stack usage significantly more compact, allowing larger
amounts of data to be processed on the same stack.

PrimOp functions with more than 8 positional (curried) arguments
should use an attrset instead.
2023-11-16 15:38:21 +01:00
Robert Hensing
0daccb1121 libexpr: Check primop arity earlier 2023-11-16 12:28:32 +01:00
Théophane Hufschmitt
ba3cb4a049 Remove all the occurences of VLAs
There's generally no strict reason for using them, and they are somewhat
fishy, so let's avoid them.
2023-11-16 12:27:30 +01:00
Théophane Hufschmitt
5196613e82 Use boost small vectors instead of VLAs in the primops
VLAs are a dangerous feature, and their usage triggers an undefined
behavior since theire size can be zero in some cases.
So replace them with `boost::small_vector`s which fit the same goal but
are safer.

It's also incidentally consistently 1% faster on the benchmarks.
2023-11-16 12:27:30 +01:00
John Ericson
f0adb72c23 Mark fetchTree as unstable again
As discussed in our last meeting, we need a bit more time, but we are
"time boxing" the work left to do to ensure there is not unbounded
delay.

Rather than putting it back underneath `flakes`, though, put it
underneath its own `fetch-tree` experimental feature (which `flakes`
includes/implies). This signals our commitment to the plan to stabilize
it first without waiting to go through the rest of Flakes, and also will
give users a "release candidate" when we get closer to stabilization.

This reverts commit 4112dd1fc9.
2023-11-08 23:23:56 -05:00
Bob van der Linden
9c0a09f09f allow ^ in URLs
Users may select specific outputs using the ^output syntax or selecting
any output using ^*.

URL parsing currently doesn't support these kinds of output references:
parsing will fail.

Currently `queryRegex` was reused for URL fragments, which didn't
include support for ^. Now queryRegex has been split from fragmentRegex,
where only the fragmentRegex supports ^.
2023-11-06 21:21:20 +01:00
Mel Zuser
61d6fe059e
Fix boost::bad_format_string exception in builtins.addErrorContext (#9291)
* Fix boost::bad_format_string exception in builtins.addErrorContext

The message passed to addTrace was incorrectly being used as a format
string and this this would cause an exception when the string contained
a '%', which can be hit in places where arbitrary file paths are
interpolated.

* add test
2023-11-06 19:13:40 +00:00
John Ericson
ac89bb064a Split up util.{hh,cc}
All OS and IO operations should be moved out, leaving only some misc
portable pure functions.

This is useful to avoid copious CPP when doing things like Windows and
Emscripten ports.

Newly exposed functions to break cycles:

 - `restoreSignals`
 - `updateWindowSize`
2023-11-05 12:20:02 -05:00
John Ericson
dde1d86338 Restrict some code to StoreDirConfig
- part of eval cache
 - part of derivations
 - derived path
 - store path with outputs
 - serializers
2023-11-04 19:05:36 -04:00
BootRhetoric
098f0615c9
fetchGit and flake: add publicKeys list input
This adds publicKeys as an optional fetcher input attribute to flakes
and builtins.fetchGit to provide a nix interface for the json-encoded
`publicKeys` attribute of the git fetcher.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-11-03 20:23:31 +01:00
Eelco Dolstra
cdb27c1519 SourceAccessor: Change the main interface from lstat() to maybeLstat() 2023-11-01 15:26:07 +01:00
Felix Uhl
c7dcdb8325 Overhaul nix flake update and lock commands
Closes #5110
2023-10-31 15:33:57 +01:00
Felix Uhl
1fd0867389 Fix missing output when creating lockfile 2023-10-31 15:33:50 +01:00
Eelco Dolstra
fa6bc33604
Merge pull request #9239 from edolstra/putFile
Input: Replace markFileChanged() by putFile()
2023-10-31 14:49:42 +01:00
Eelco Dolstra
05316d401f Cleanup 2023-10-30 17:03:06 +01:00
Silvan Mosberger
46028ff764
doc: Fix fetchGit default name (#9241) 2023-10-26 07:05:48 +02:00
Eelco Dolstra
15c430f389 Remove unused LockFile::write() 2023-10-25 18:44:09 +02:00
Eelco Dolstra
95d657c8b3 Input: Replace markFileChanged() by putFile()
Committing a lock file using markFileChanged() required the input to
be writable by the caller in the local filesystem (using the path
returned by getSourcePath()). putFile() abstracts over this.
2023-10-25 18:30:29 +02:00
Silvan Mosberger
f269911641
Document builtins.substring negative length behavior (#9226) 2023-10-24 11:22:02 +02:00
John Ericson
765436e300 Add builtins.addDrvOutputDependencies
End goal: make `(mkDerivation x).drvPath` behave like a non-DrvDeep
context.

Problem: users won't be able to recover the DrvDeep behavior when
nixpkgs makes this change.

Solution: add this primop.

The new primop is fairly simple, and is supposed to complement other
existing ones (`builtins.storePath`, `builtins.outputOf`) so there are
simple ways to construct strings with every type of string context
element.

(It allows nothing we couldn't already do with `builtins.getContext` and `builtins.appendContext`, which is also true of those other two primops.)

This was originally in #8595, but then it was proposed to land some doc
changes separately. So now the code changes proper is just moved to
this, and the doc will be done in that.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.nore
github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io
2023-10-23 12:49:14 -04:00
Eelco Dolstra
955bbe53c5
Merge pull request #9177 from edolstra/input-accessors
Backport FSInputAccessor and MemoryInputAccessor from lazy-trees
2023-10-23 11:42:04 +02:00
Eelco Dolstra
935c9981de Remove fetchers::Tree and move tarball-related stuff into its own header 2023-10-20 19:56:52 +02:00
Eelco Dolstra
bcf5c31950 Add future FIXME 2023-10-20 16:58:33 +02:00
Eelco Dolstra
173abec0bc coerceToPath(): Handle __toString, add tests 2023-10-20 13:06:44 +02:00
Eelco Dolstra
5be7705ddf Remove stuff we don't need yet 2023-10-19 19:20:21 +02:00
Johannes Kirschbauer
9bc7b4f463
doc: generic closure supported key types (#9183)
* doc: generic closure supported key types

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-10-19 12:39:41 +00:00
Eelco Dolstra
06c57899e3 Remove FIXME 2023-10-19 14:22:05 +02:00
Eelco Dolstra
f16af08e83 Fix macOS compilation 2023-10-19 14:20:50 +02:00
Eelco Dolstra
12214fef09 InputAccessor::fetchToStore(): Support arbitrary ingestion methods 2023-10-19 14:19:10 +02:00
Yueh-Shun Li
5088e6563a primops: add builtins.convertHash
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-10-19 00:58:56 +08:00
Yueh-Shun Li
e026f3e1ae treewide: Reference HashFormat members with scope
Base* -> HashFormat::Base*
2023-10-19 00:56:41 +08:00
Yueh-Shun Li
aff177d860 Elaborate the "unknown hash algorithm" error
List the allowed hash formats
2023-10-19 00:38:32 +08:00
Eelco Dolstra
df73c6eb8c Introduce MemoryInputAccessor and use it for corepkgs
MemoryInputAccessor is an in-memory virtual filesystem that returns
files like <nix/fetchurl.nix>. This removes the need for special hacks
to handle those files.
2023-10-18 17:38:11 +02:00
Eelco Dolstra
ea38605d11 Introduce FSInputAccessor and use it
Backported from the lazy-trees branch. Note that this doesn't yet use
the access control features of FSInputAccessor.
2023-10-18 17:37:32 +02:00
Vladimir Kryachko
311e2ad024 Address review comments 2023-10-18 10:37:06 -04:00
Vladimir Kryachko
a2f0ba6a6d Fix transitive input locking.
Fixes reproducibility issue described in #9143

Fixes #9143
2023-10-18 10:31:36 -04:00
Eelco Dolstra
201c115c3e
Merge pull request #9151 from edolstra/stabilize-fetchTree
Stabilize fetchTree
2023-10-18 10:54:08 +02:00
Théophane Hufschmitt
c1a1766c46
Merge pull request #9169 from vkryachko/follow_cycle
Detect cycles in flake follows.
2023-10-18 07:34:03 +02:00
Eelco Dolstra
f62b5500ff fetchTree: Require the flakes experimental feature for the URL syntax 2023-10-17 14:52:34 +02:00
Jacek Galowicz
54b350d517 Drop some moves that would happen anyway but forbid NRVO where appicable 2023-10-16 21:48:35 +01:00
Jacek Galowicz
abf7df2b37 Fix moves that accidentally copy anyway 2023-10-16 21:48:35 +01:00
Vladimir Kryachko
d6066c90f8 Don't convert InputPaths to strings prematurely. 2023-10-16 15:47:28 -04:00
Vladimir Kryachko
b3fd7db63f Detect cycles in flake follows.
This change results in an error thrown as opposed to segfaulting due to
stack overflow.

Fixes #9144
2023-10-16 13:00:49 -04:00
Eelco Dolstra
4112dd1fc9 Mark fetchTree as stable 2023-10-13 16:45:08 +02:00
Eelco Dolstra
8eb4f735dc fetchTree: Only use the registry if flakes are enabled 2023-10-13 14:34:32 +02:00
Eelco Dolstra
4ce7a53a9c Update fetchTree docs 2023-10-13 14:34:32 +02:00
Eelco Dolstra
856fe13533 fetchTree cleanup
Two changes:

* The (probably unintentional) hack to handle paths as tarballs has
  been removed. This is almost certainly not what users expect and is
  inconsistent with flakeref handling everywhere else.

* The hack to support scp-style Git URLs has been moved to the Git
  fetcher, so it's now supported not just by fetchTree but by flake
  inputs.
2023-10-13 14:34:23 +02:00
Michal Sojka
db0d94560b Document builtins.fetchTree
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

Supersedes #6740
2023-10-13 14:24:10 +02:00
Valentin Gagarin
7642894a4e reword documentation on lookup path resolution 2023-10-10 00:49:27 +02:00
John Ericson
838be5e4a0
Merge pull request #9114 from fricklerhandwerk/lookup-path
introduce lookup paths as a distinct language construct
2023-10-09 11:28:40 -04:00
Robert Hensing
aeea49609b
Merge pull request #8895 from hercules-ci/gc-before-stats
eval: Run a full GC before printing stats
2023-10-09 17:14:45 +02:00
John Ericson
3953901796
Merge pull request #9112 from fricklerhandwerk/restrict-eval-nix-path
describe the effect of `restrict-eval` in a more focused manner
2023-10-09 11:07:21 -04:00