Commit graph

9647 commits

Author SHA1 Message Date
John Ericson
5417990e31 Create ServeProto::BuildOptions and a serializer for it
More tests, and more serializers for Hydra reuse.
2023-12-09 11:35:13 -05:00
Robert Hensing
c8458bd731
Merge pull request #9555 from 9999years/positions-in-errors
Pass positions when evaluating
2023-12-09 03:55:58 +01:00
Robert Hensing
7cdc8786d9
Merge pull request #9568 from hercules-ci/revert-9553
Revert 9553
2023-12-09 03:49:53 +01:00
Robert Hensing
9b7b7a7561 Revert "Print the value in error: cannot coerce messages (#9553)"
This reverts commit f0ac2a35d5.

The request from the sibling PR, which also applies here, was not addressed.
https://github.com/NixOS/nix/pull/9554#issuecomment-1845095735
2023-12-09 02:13:32 +01:00
John Ericson
ce4ca574d2 Clarify SourceAccessor methods should never implicitly follow symlinks
The code has already been fixed (yay!) so what is left of this commit is
just updating the API docs.

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2023-12-08 20:00:47 -05:00
Robert Hensing
d4f6b1d38b
Merge pull request #9497 from edolstra/move-access-control
Move restricted/pure-eval access control out of the evaluator and into the accessor
2023-12-08 22:21:50 +01:00
John Ericson
762af72728
Merge pull request #9564 from NixOS/fix-clang-build
Avoid `std::strstream`, fix the clang build
2023-12-08 15:07:36 -05:00
Théophane Hufschmitt
36ca6adc60
Merge pull request #9563 from obsidiansystems/tryResolve-evalStore
Give `Derivation::tryResolve` an `evalStore` argument
2023-12-08 19:21:35 +01:00
John Ericson
f9ee1bedcf Avoid std::strstream, fix the clang build
According https://en.cppreference.com/w/cpp/io/strstream, it has been
deprecated since C++98! The Clang + Linux build systems to not have it
at all, or at least be hiding it.

We can just use `std::stringstream` instead, I think.
2023-12-08 13:18:52 -05:00
Rebecca Turner
f0ac2a35d5
Print the value in error: cannot coerce messages (#9553)
* Print the value in `error: cannot coerce` messages

This extends the `error: cannot coerce a TYPE to a string` message
to print the value that could not be coerced. This helps with debugging
by making it easier to track down where the value is being produced
from, especially in errors with deep or unhelpful stack traces.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-12-08 16:36:57 +00:00
John Ericson
96dd757b0c Give Derivation::tryResolve an evalStore argument
This is needed for building CA deriations with a src store / dest store
split. In particular it is needed for Hydra.

https://github.com/NixOS/hydra/issues/838 currently puts realizations,
and thus build outputs, in the local store, but it should not.
2023-12-08 10:01:05 -05:00
Rebecca Turner
0b80935c22
Pass positions when evaluating
This includes position information in more places, making debugging
easier.

Before:

```
$ nix-instantiate --show-trace --eval tests/functional/lang/eval-fail-using-set-as-attr-name.nix
error:
       … while evaluating an attribute name

         at «none»:0: (source not available)

       error: value is a set while a string was expected
```

After:

```
error:
       … while evaluating an attribute name

         at /pwd/lang/eval-fail-using-set-as-attr-name.nix:5:10:

            4| in
            5|   attr.${key}
             |          ^
            6|

       error: value is a set while a string was expected
```
2023-12-07 10:27:21 -08:00
John Ericson
a5521b7d94 Factor out ServeProto::Serialiser<UnkeyedValidPathInfo> and test
In the process, partially undo e89b5bd0bf
in that the ancient < 2.4 version is now supported again by the
serializer again. `LegacySSHStore`, instead of also asserting that the
version is at least 4, just checks that `narHash` is set.

This allows us to better test the serializer in isolation for both
versions (< 4 and >= 4).
2023-12-07 11:34:18 -05:00
Bryan Honof
bf00d5ecef
fix(libutil/tarfile): add option to libarchive so it behaves correctly with AppleDouble files
AppleDouble files were extracted differently on macOS machines than on other
UNIX's.
Setting `archive_read_set_format_option(this->archive, NULL ,"mac-ext",NULL)`
fixes this problem, since it just ignores the AppleDouble file and treats it as
a normal one.
This was a problem since it caused source archives to be different between macOS
and Linux.

Ref: nixos/nix#9290
2023-12-07 11:35:15 +01:00
Peter Kolloch
d38ec12855 Update src/libexpr/primops.cc
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-12-06 23:43:42 +01:00
Peter Kolloch
8afeaf05c4 Add docs/rl-notes for nix hash convert / builtins.convertHash
https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch
7ff876b92b Add deprecation notice for old nix hash conversion subcommands.
(But not yet nix-hash since `nix hash` is still hidden behind a feature flag.)

https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch
fc6f29053a Renamed HashFormat::Base32 to HashFormat::Nix32
...and also adjusted parsing accordingly.

Also added CLI completion for HashFormats.

https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch
837b889c41 Further HashType renaming + using mkHashAlgoOptFlag for new conversion
https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch
5334c9c792 HashType: Rename to HashAlgorithm
To be consistent with CLI, nix API
and many other references.

As part of this, we also converted it to a scoped enum.

https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch
0c2d5f7673 nix hash convert: s/--type/--algo/ + more functional tests
https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch
6bbd900d4f nix hash convert: added
This deviated from the proposal! See comments on the issue.

https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:34 +01:00
Peter Kolloch
156ea78d74 CmdHashBase: doc comment 2023-12-06 23:41:07 +01:00
Peter Kolloch
e7abf60a0c hash.cc/hash.h: Minor C++ improvements 2023-12-06 23:41:07 +01:00
Théophane Hufschmitt
7fff625e39
Improve the error message for “multicommands” commands (#9510)
* Factor out the default `MultiCommand` behavior

All the `MultiCommand`s had (nearly) the same behavior when called
without a subcommand.
Factor out this behavior into the `NixMultiCommand` class.

* Display the list of available subcommands when none is specified

Whenever a user runs a command that excepts a subcommand, add the list
of available subcommands to the error message.

* Print the multi-command lists as Markdown lists

This takes more screen real estate, but is also much more readable than
a comma-separated list
2023-12-06 13:13:45 +00:00
Eelco Dolstra
2bd8322500
Update src/libfetchers/filtering-input-accessor.hh
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-12-06 14:08:40 +01:00
Eelco Dolstra
57246c4c38 PosixSourceAccessor: Cache lstat() calls
Since we're doing a lot of them in assertNoSymlinks().
2023-12-06 13:55:54 +01:00
Eelco Dolstra
504e4fc457 CanonPath: Support std::hash 2023-12-06 13:45:59 +01:00
Eelco Dolstra
83c067c0fa PosixSourceAccessor: Don't follow any symlinks
All path components must not be symlinks now (so the user needs to
call `resolveSymlinks()` when needed).
2023-12-05 23:02:59 +01:00
Valentin Gagarin
3c310bde2e reword description for the fetch-tree experimental feature
without knowing a lot of context, it's not clear who "we" are in that
text. I'm also strongly opposed to adding procedural notes into
a reference manual; it just won't age well.

this change leaves a factual description of the experimental feature and
its purpose.
2023-12-04 16:42:32 +01:00
Valentin Gagarin
368fdb482d reword description of the builders-use-substitutes setting 2023-12-02 03:07:27 +01:00
Valentin Gagarin
24b781773f fix random docs errors
remove link to the contributing guide from user documentation.
it doesn't help here, and the target at first glance shows redundant
information.
2023-12-02 03:02:59 +01:00
Valentin Gagarin
51adfb9b27 reword documentation on settings and attributes related to substitution
- add links
- be more concise
- clarify the distinction between `preferLocalBuild` and `allowSubstitutes`
2023-12-02 02:56:25 +01:00
John Ericson
91b6833686 Move tests to separate directories, and document
Today, with the tests inside a `tests` intermingled with the
corresponding library's source code, we have a few problems:

- We have to be careful that wildcards don't end up with tests being
  built as part of Nix proper, or test headers being installed as part
  of Nix proper.

- Tests in libraries but not executables is not right:

  - It means each executable runs the previous unit tests again, because
    it needs the libraries.

  - It doesn't work right on Windows, which doesn't want you to load a
    DLL just for the side global variable . It could be made to work
    with the dlopen equivalent, but that's gross!

This reorg solves these problems.

There is a remaining problem which is that sibbling headers (like
`hash.hh` the test header vs `hash.hh` the main `libnixutil` header) end
up shadowing each other. This PR doesn't solve that. That is left as
future work for a future PR.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-12-01 10:48:58 -05:00
Robert Hensing
77adb55ae4
Merge pull request #9511 from obsidiansystems/add-missing-includes
Add two missing `#include "nar-info.hh"`
2023-12-01 16:43:56 +01:00
John Ericson
d59bdbe4fd Add two missing #include "nar-info.hh"
GitHub's racy CI caused this oversight to sneak through.
2023-12-01 10:20:19 -05:00
John Ericson
ac4d2e7b85
Merge pull request #9478 from tweag/nix-config-check
Rename `nix doctor` to `nix config check`
2023-12-01 10:19:19 -05:00
Théophane Hufschmitt
dfa219d03b
Merge pull request #9494 from sellout/nix-run-execv
Don’t use `execvp` when we know the path
2023-12-01 16:07:16 +01:00
Robert Hensing
fcf09813c6
Merge pull request #6236 from obsidiansystems/store-dir-config
Factor out `StoreDirConfig`
2023-12-01 15:38:14 +01:00
Théophane Hufschmitt
ea2dd16623 Use a proper enum rather than a boolean in runProgramInStore
Makes the call-site much easier to understand.
2023-12-01 15:35:21 +01:00
John Ericson
82359eba6b
Merge pull request #9233 from bouk/bouk/apply-config-inner
config: add included files into parsedContents before applying
2023-12-01 08:23:32 -05:00
Robert Hensing
188c803ddb
Merge pull request #9508 from infinisil/revert-7762-boost-regex
Revert "Switch from std::regex to boost::regex"
2023-12-01 02:08:58 +01:00
Silvan Mosberger
333ea684b0 Add boost::regex regression test 2023-12-01 01:39:52 +01:00
Valentin Gagarin
4781e7fa70 Document each store type on its own page
This makes for more useful manual table of contents, that displays the
information at a glance.

The `nix help-stores` command is kept as-is, even though it will show up
in the manual with the same information as these pages due to the way it
is written as a "`--help`-style" command. Deciding what to do with that
command is left for a later PR.

This change also lists all store types at the top of the respective overview page.

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems
2023-12-01 01:27:52 +01:00
Silvan Mosberger
908a011a4a
Revert "Switch from std::regex to boost::regex" 2023-12-01 00:50:20 +01:00
Valentin Gagarin
39de819eda rename debugging helper environment variable 2023-11-30 23:23:32 +01:00
Eelco Dolstra
8cafc754d8 Move access control from FSInputAccessor to FilteringInputAccessor 2023-11-30 21:54:53 +01:00
Valentin Gagarin
44d21f6ef9 keep generated documentation in a separate directory
- helps navigating the code as it highlights which files are generated
- makes it less error prone when working incrementally
  (although this should be just fixed by building out of tree)
2023-11-30 20:39:24 +01:00
Eelco Dolstra
43d9fb6cf1 Remove InputAccessor::root() 2023-11-30 16:44:54 +01:00
Eelco Dolstra
305939655a Remove superfluous use of hasAccessControl() 2023-11-30 16:28:33 +01:00
Eelco Dolstra
ea95327e72 Move restricted/pure-eval access control out of the evaluator and into the accessor 2023-11-30 16:16:17 +01:00
Eelco Dolstra
3bebaefcd0
Merge pull request #9476 from alois31/restore-progress-bar
nix repl: Only hide the progress bar while waiting for user input
2023-11-30 16:07:28 +01:00
Eelco Dolstra
cb7f25869d
Merge pull request #9430 from hercules-ci/remove-vlas
Fix stack overflow in `filter`
2023-11-30 15:31:44 +01:00
Eelco Dolstra
b6a3fde6b7
Merge pull request #9465 from obsidiansystems/build-dir
Use `buildprefix` in a few more places
2023-11-30 15:28:37 +01:00
Eelco Dolstra
eac0a62052
Merge pull request #9491 from obsidiansystems/fix-freebsd
Improve ACL clearing support (fixing FreeBSD build)
2023-11-30 15:27:24 +01:00
Greg Pfeil
743232bf04
Don’t use execvp when we know the path 2023-11-30 00:17:25 -07:00
John Ericson
a7115a47ef Improve ACL clearing support (fixing FreeBSD build)
The problem was that f880469173 forgot
that the `#include <sys/xattr.h>` was guarded by an `#ifdef __linux__`.

However, the build failure was only on FreeBSD --- turns out other
platforms have this header too!

The fix therefore uses a new configure check so we properly clear ACLs
on more platforms.
2023-11-29 21:09:19 -05:00
Alex Ameen
02bd821f2e
fix: nlohmann::adl_serializer for std::optional (#9147)
This allows templates such as `NLOHMANN_DEFINE_TYPE_*` templates and other generators with things like `std::vector<std::optional<T>>`.

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2023-11-30 01:26:39 +00:00
John Ericson
52e0911302 Use buildprefix in a few more places
`installcheck` doesn't yet work, but the rest of the build can now
happen mostly inside a separate build directory.

Progress on #9342

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-11-29 19:49:07 -05:00
Eelco Dolstra
043413bb59 boost::span -> std::span 2023-11-29 12:38:46 +01:00
Eelco Dolstra
b36857ac8d Add a Git-based content-addressed tarball cache
GitArchiveInputScheme now streams tarballs into a Git repository. This
deduplicates data a lot, e.g. when you're fetching different revisions
of the Nixpkgs repo. It also warns if the tree hash returned by GitHub
doesn't match the tree hash of the imported tarball.
2023-11-29 12:37:03 +01:00
Théophane Hufschmitt
6d1605818c Rename nix doctor to nix config check
Fix #7672
2023-11-28 14:28:19 +01:00
Théophane Hufschmitt
f300e11b05 Rename nix show-config to nix config show
Part of #7672
2023-11-28 14:28:19 +01:00
Alois Wohlschlager
20cd5eb2b3
nix repl: Only hide the progress bar while waiting for user input
In commit 0d2163c6dc, the progress bar was hidden
in nix repl because of a regression that caused it to interfere with user
input. Several users like(d) seeing the progress bar in the repl during builds.
Only hiding it while waiting for user input gives us the best of both worlds,
so do just that.
2023-11-27 19:25:15 +01:00
Robert Hensing
f0180487a0
Merge pull request #9462 from trofi/eval.o-dependency
libexpr: add missing dependency on 'flake/call-flake.nix.gen.hh'
2023-11-27 17:25:21 +01:00
Robert Hensing
68c48756fe libexpr/local.mk: Make eval compile deps regular
Dependency is now entirely through the eval.cc rule.
All gen.hh deps are now there.
2023-11-27 15:52:24 +01:00
Valentin Gagarin
384ffb4443 add deprecation warnings in documentation
this is hacky, but can serve as a stopgap until we can do it
programmatically.
2023-11-27 14:07:32 +01:00
Théophane Hufschmitt
d46230eff5
Merge pull request #7762 from yorickvP/boost-regex
Switch from std::regex to boost::regex
2023-11-27 10:59:22 +01:00
Sergei Trofimovich
75134b7513 libexpr: add missing dependency on 'flake/call-flake.nix.gen.hh'
Without the change build for `eval.o` fails occasionally as:

    $ make src/libexpr/eval.o
      GEN    Makefile.config
      GEN    src/libexpr/primops/derivation.nix.gen.hh
      GEN    src/libexpr/fetchurl.nix.gen.hh
      GEN    src/libexpr/parser-tab.cc
      GEN    src/libexpr/lexer-tab.cc
    src/libexpr/lexer.l:314: warning, -s option given but default rule can be matched
      CXX    src/libexpr/eval.o
    src/libexpr/eval.cc:519:18: fatal error: flake/call-flake.nix.gen.hh: No such file or directory
      519 |         #include "flake/call-flake.nix.gen.hh"
          |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    compilation terminated.
    make: *** [mk/patterns.mk:3: src/libexpr/eval.o] Error 1

Noticed in https://github.com/NixOS/nixpkgs/pull/269439
2023-11-27 08:56:24 +00:00
Moritz Angermann
f56401a114 nix flake update add deprecation warnings.
This builds on #8817, to add additional UX help for people with existing
muscle memory (or shell history) with --update-input and tries to gently
guide them towards the newly evolved CLI UI.

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2023-11-27 09:08:15 +01:00
Yorick van Pelt
d2f5e263e3
Switch from std::regex to boost::regex 2023-11-25 15:14:18 +01:00
John Ericson
dda0e34ecf
Merge pull request #9443 from ivan770/reproducibility
doc: fix machine-specific capabilities leaking
2023-11-24 22:59:38 -05:00
ivan770
54b6847655 doc: fix machine-specific capabilities leaking 2023-11-24 11:17:35 -05:00
Théophane Hufschmitt
7f626dba33
Merge pull request #9398 from Qyriad/fixes/flake-not-found
flakes: bare minimum fix the error message for untracked flake.nix
2023-11-24 15:33:08 +01:00
Robert Hensing
8a52325d7d
Merge pull request #9444 from phip1611/foldl-doc
doc: primops: fix typo
2023-11-24 13:36:18 +01:00
Théophane Hufschmitt
6a94755b12 Allow user input in git commit
We occasionnally commit to git repositories (like with `nix flake update --commit-lock-file`).
This shells out to `git commit`, which might wait for user input (for a signing key passphrase for instance).
Disable the progress bar while this is running to make sure that the
user can enter it.
2023-11-24 11:45:37 +01:00
Philipp Schuster
5be0e6b314 doc: primops: fix typo 2023-11-24 10:50:01 +01:00
Philipp Schuster
2ce8c9650b
doc: primops: add more info for foldl (#9254)
* doc: primops: add more info for foldl

From the existing doc it is not obvious whether the first or the
second argument is the accumulator. This is however relevant to
know, as for certain scenarios, this might change the behavior.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-11-23 21:02:20 +00:00
Théophane Hufschmitt
bf13943206
Merge pull request #9173 from vkryachko/transitive-input-overrides
Fix transitive input locking.
2023-11-23 08:54:45 +01:00
Robert Hensing
030fb86dd1
Merge pull request #9394 from NixOS/separate-file-canonicalise
Put `canonicaliseTimestampAndPermissions` in its own header/file
2023-11-22 18:27:52 +01:00
Eelco Dolstra
38844943d0
Merge pull request #9413 from edolstra/fetchToStore-cache
Persistently cache `InputAccessor::fetchToStore()`
2023-11-22 12:19:25 +01:00
Eelco Dolstra
b1ab592f28 Use the StorePath-based cache interface 2023-11-22 11:44:02 +01:00
r-vdp
5292f36426
Fix compile warning due to unused variable binding.
We still need the check, since we don't have narinfo for locally built store paths.
2023-11-22 11:34:53 +01:00
Eelco Dolstra
61b76f5f34
Apply suggestion
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-11-22 11:26:12 +01:00
John Ericson
46131567da Add missing -lrapidcheck fixing build with shared lib
https://github.com/NixOS/nixpkgs/pull/269064 makes rapidcheck be build
as a shared lib, but that broke Nix because the `-lrapidcheck` was
missing. This fixes that (and doesn't break Nix what the library is a
static archive as today).
2023-11-21 23:19:25 -05:00
Robert Hensing
4e790efade Use boost::container::small_vector in place of VLAs 2023-11-21 20:52:27 +01:00
Matej Urbas
06b8902562 MountedSSHStore: stores on shared filesystems 2023-11-21 13:34:01 -05:00
Matej Urbas
226b0f3956 Extend the worker protocol with wopAddPermRoot 2023-11-21 13:26:55 -05:00
Matej Urbas
9796ebd7ef Add --process-ops flag to nix-daemon 2023-11-21 13:18:19 -05:00
John Ericson
949f5841f8 Add the MountedSSHStore experimental feature
It will be implemented in the subsequent commits of this PR.
2023-11-21 13:16:46 -05:00
John Ericson
f880469173 Put canonicaliseTimestampAndPermissions in its own header/file
It is not inherently tied to `LocalStore`, it could probably even go in
`libnixutil`. Functions not attached to `LocalStore` should not be
declared in `local-store.hh`.

I am moving it to facilitate experimenting for #9344. If
canonicalisation should be done client-side in client-side builds, there
wouldn't be a `LocalStore` at all so having to include that header to
get this freestanding function is cumbersome and wrong.

Perhaps canonicalisation should still be done server-side for security
reasons --- I don't mean to make that judgement call now --- but even if
so, this freestanding function still isn't connected to `LocalStore` so
while less urgent it is still better to move out of this header.
2023-11-21 12:57:59 -05:00
Robert Hensing
9cd69e1c39
Merge pull request #9411 from edolstra/path-display
Improve SourceAccessor path display
2023-11-21 18:12:01 +01:00
Lin Jian
4d8decbd13 doc: fix number of template attributes
This number is not updated when welcomeText is added[1][2].

[1]: f3a2940e70
[2]: https://github.com/NixOS/nix/pull/6103
2023-11-21 16:52:25 +01:00
Eelco Dolstra
c185e45ed7
Merge pull request #9410 from edolstra/shallow
Some shallow git fetching improvements
2023-11-21 12:58:01 +01:00
John Ericson
516e7ddc41
Merge pull request #9257 from Artturin/nixenvjsondrvpath
`nix-env --query`: fix `--json` ignoring `--drv-path`
2023-11-20 16:32:35 -05:00
Eelco Dolstra
99d5204baa Persistently cache InputAccessor::fetchToStore()
This avoids repeated copying of the same source tree between Nix
invocations. It requires the accessor to have a "fingerprint" (e.g. a
Git revision) that uniquely determines its contents.
2023-11-20 20:04:37 +01:00
Eelco Dolstra
a0162d5732 Improve SourceAccessor path display
Backported from lazy-trees. This allows SourceAccessors to show the
origin of the accessor. E.g. we now get

  copying '«git+https://github.com/blender/blender.git?ref=refs/heads/main&rev=4edc1389337dd3679ff66969c332d2aff52e1992»/' to the store

instead of

  copying '/' to the store
2023-11-20 18:54:36 +01:00
Eelco Dolstra
071f14a0bb Don't do shallow fetches over ssh 2023-11-20 17:25:40 +01:00
Eelco Dolstra
e4066c0444 Fetch specific Git revisions
This is more efficient, and necessary when using shallow=1 with a rev.
2023-11-20 17:25:40 +01:00
Eelco Dolstra
f450c8773c
Merge pull request #9408 from roblabla/fix-bad-format-string
Fix bad_format_string error when builder stdout contains %
2023-11-20 17:22:40 +01:00