Commit graph

9978 commits

Author SHA1 Message Date
DavHau
bc00fa4647 fetchTree/fetchGit: re-enable shallow fetching
Add several tests for git fetching:
- shallow-cache-separation: can fetch the same repo shallowly and non-shallowly
- shallow-ignore-ref: ensure that ref gets ignored when shallow=true is set
- ssh-shallow: can fetch a git repo via ssh using shallow=1
2024-01-19 20:30:47 +07:00
Robert Hensing
d762caff46
Merge pull request #9806 from hercules-ci/fix-git-auth
Fix git auth
2024-01-19 13:47:52 +01:00
Théophane Hufschmitt
28d7db249a
Remove a nonsensical shorthand flag in nix store add
`-n` was an alias for `--mode`, but that seems to just be a copy-paste error as it doesn't make sense.
`--mode` probably doesn't need a shorthand flag at all, so remove it.

Noticed in https://github.com/NixOS/nix/pull/9809#issuecomment-1899890555
2024-01-19 10:10:00 +01:00
John Ericson
a34ec0bd12 Include store path exact spec in the docs
This is niche, but deserves to be in the manual because it is describing
behavior visible to the outside world, not mere implementation details.
2024-01-18 22:19:14 -05:00
Robert Hensing
8d422c2fef Revert libgit2 fetching
libgit2 is not capable of using git-credentials helpers yet.
This prevents private repositories from being used.

Based on code that was replaced in https://github.com/NixOS/nix/pull/9240
(Introduce libgit2); hence:

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2024-01-18 22:29:26 +01:00
Robert Hensing
76a50b3a69 doc: GitRepoImpl::path 2024-01-18 22:29:26 +01:00
Eelco Dolstra
dca0a80240 copyStorePath(): Bail out early if the store path already exists
In rare cases (e.g. when using allowSubstitutes = false), it's
possible that we simultaneously have a DerivationGoal *and* a
SubstitutionGoal building the same path. So if a DerivationGoal
already built the path while the SubstitutionGoal was waiting for a
download slot, it saves us a superfluous download to exit early.
2024-01-18 17:19:15 +01:00
Eelco Dolstra
a18d8d688a LocalStore::addToStore(): Ignore exceptions from parseDump()
In the "discard" case (i.e. when the store path already exists
locally), when we call parseDump() from a Finally and it throws an
exception (e.g. if the download of the NAR fails), Nix crashes:

   terminate called after throwing an instance of 'nix::SubstituteGone'
     what():  error: file 'nar/06br3254rx4gz4cvjzxlv028jrx80zg5i4jr62vjmn416dqihgr7.nar.xz' does not exist in binary cache 'http://localhost'
   Aborted (core dumped)
2024-01-18 17:19:15 +01:00
Eelco Dolstra
ab786e22f1 Show what goal is waiting for a build slot 2024-01-18 17:19:15 +01:00
Théophane Hufschmitt
e652322a61
Merge pull request #9799 from obsidiansystems/push-add-to-store-from-dump-unsupported-down
Push `addToStoreFromDump` `unsupported(...)` down `Store` class hierarchy
2024-01-18 16:54:44 +01:00
Eelco Dolstra
32706b14a7
Merge pull request #9798 from edolstra/remote-store-eof
Print a more helpful message if the daemon crashes
2024-01-18 15:47:59 +01:00
Eelco Dolstra
1fe8f54bd3 Use BackedStringView 2024-01-18 15:27:57 +01:00
Théophane Hufschmitt
28674247ec
Merge pull request #9787 from obsidiansystems/bind-proc-syserror
`bind`: give same treatment as `connect` in #8544, dedup
2024-01-18 09:34:15 +01:00
John Ericson
574db83504 Push addToStoreFromDump unsupported(...) down Store class hierarchy
Instead of having it be the default method in `Store` itself, have it be
the implementation in `DummyStore` and `LegacySSHStore`. Then just the
implementations which fail to provide the method pay the "penalty" of
dealing with the icky `unimplemented` function for non-compliance.

Picks up where #8217. Getting close to no `unsupported` in the `Store`
interface itself!

More progress on issue #5729.
2024-01-17 23:46:03 -05:00
John Ericson
3016e67c21 bind: give same treatment as connect in #8544, dedup
It is good to propagate the underlying error so whether or not we use a
process to deal with path length issues is not observable.

Also, as these wrapper functions got more and more complex, the code
duplication got worse and worse. The new `bindConnectProcHelper`
function deduplicates them.
2024-01-17 23:35:08 -05:00
Eelco Dolstra
a3cf27ca47 Print a more helpful message if the daemon crashes
Instead of

   error: unexpected end-of-file

you now get

   error: Nix daemon disconnected unexpectedly (maybe it crashed?)
2024-01-17 22:54:02 +01:00
Eelco Dolstra
39ab50f9ee Store::buildPaths(): Fix display of store paths
This was broken in 7ac39ff05c.
2024-01-17 21:41:37 +01:00
Robert Hensing
d0a284284b refactor: Extract simply, awkwardly Store::queryPathInfoFromClientCache
This is useful for determining quickly which substituters to query.

An alternative would be for users to invoke the narinfo cache db directly,
so why do we need this change?

 - It is easier to use. I believe Nix itself should also use it.

 - This way, the narinfo cache db remains an implementation detail.

 - Callers get to use the in-memory cache as well.
2024-01-17 17:54:19 +01:00
Robert Hensing
2a3c5e6b8b
Merge pull request #9480 from NixOS/libfetchers-git-exportIgnore
libfetchers/git: Support export-ignore
2024-01-16 23:03:46 +01:00
Robert Hensing
01271f2996
Merge pull request #9786 from hercules-ci/package
DerivationInfo -> PackageInfo
2024-01-16 22:43:14 +01:00
Jacek Galowicz
8ae3aeec94 Don't use std::make_unique right before release 2024-01-16 17:00:46 +00:00
Jacek Galowicz
9d9f42cc38 Remove C-style casts 2024-01-16 17:00:46 +00:00
Jacek Galowicz
d11d7849f7 Use ChildWrapperFunction type and make casts more explicit 2024-01-16 17:00:46 +00:00
Jacek Galowicz
c924147c9d Drop parentheses from thunks 2024-01-16 17:00:46 +00:00
Jacek Galowicz
1885d579db Improve String Handling 2024-01-16 17:00:46 +00:00
Jacek Galowicz
0bc66e529f Use npos member variables instead of full type 2024-01-16 15:17:57 +00:00
Robert Hensing
ea6aa5ffd8 Package{,Info}: comments 2024-01-16 15:44:02 +01:00
Robert Hensing
65255edc9b DerivationInfo -> PackageInfo
This does not yet resolve the coupling between packages and
derivations, but it makes the code more consistent with the
terminology, and it accentuates places where the coupling is
obvious, such as

         auto drvPath = packageInfo.queryDrvPath();
         if (!drvPath)
             throw Error("'%s' is not a derivation", what());

... which isn't wrong, and in my opinion, doesn't even look
wrong, because it just reflects the current logic.
However, I do like that we can now start to see in the code that
this coupling is perhaps a bit arbitrary.
After this rename, we can bring the DerivingPath concept into type
and start to lift this limitation.
2024-01-16 15:28:03 +01:00
Eelco Dolstra
d005bade7f connect(): Propagate errno from the child process
This is necessary on macOS since addTempRoot() relies on errno.
2024-01-16 15:26:17 +01:00
Eelco Dolstra
0b1d93d2ba Sleep a bit between attempts to connect to the root server 2024-01-16 15:26:17 +01:00
Eelco Dolstra
302625e83b Merge remote-tracking branch 'origin/master' into handle-missing-gc-socket 2024-01-16 13:18:58 +01:00
Robert Hensing
b2deff1947
Merge pull request #9747 from awakesecurity/mz/fix-quadratic-splitString
Fix performance of builtins.substring for empty substrings
2024-01-16 12:18:59 +01:00
pennae
09a1128d9e don't repeatedly look up ast internal symbols
these symbols are used a *lot*, so it makes sense to cache them. this
mostly increases clarity of the code (however clear one may wish to call
the parser desugaring here), but it also provides a small performance
benefit.
2024-01-15 16:52:18 +01:00
pennae
b596cc9e79 decouple parser and EvalState
there's no reason the parser itself should be doing semantic analysis
like bindVars. split this bit apart (retaining the previous name in
EvalState) and have the parser really do *only* parsing, decoupled from
EvalState.
2024-01-15 16:52:18 +01:00
pennae
e1aa585964 slim down parser.y
most EvalState and Expr members defined here could be elsewhere, where
they'd be easier to maintain (not being embedded in a file with arcane
syntax) and *somewhat* more faithfully placed according to the path of
the file they're defined in.
2024-01-15 16:52:18 +01:00
pennae
835a6c7bcf rename ParserState::{makeCurPos -> at}
most instances of this being used do not refer to the "current"
position, sometimes not even to one reasonably close by. it could also
be called `makePos` instead, but `at` seems clear in context.
2024-01-15 16:52:18 +01:00
pennae
0076056164 move ParseData to own header, rename to ParserState
ParserState better describes what this struct really is. the parser
really does modify its state (most notably position and symbol tables),
so calling it that rather than obliquely "data" (which implies being
input only) makes sense.
2024-01-15 16:52:18 +01:00
pennae
1b09b80afa make parser utility functions members of ParseData
all of them need access to parser state in some way. make them members
to allow this without fussing so much.
2024-01-15 16:52:18 +01:00
pennae
e8d9de967f simplify parse error reporting
since nix doesn't use the bison `error` terminal anywhere any invocation
of yyerror will immediately cause a failure. since we're *already*
leaking tons of memory whatever little bit bison allocates internally
doesn't much matter any more, and we'll be replacing the parser soon anyway.

coincidentally this now also matches the error behavior of URIs when
they are disabled or ~/ paths in pure eval mode, duplicate attr
detection etc.
2024-01-15 16:52:18 +01:00
pennae
f07388bf98 remove ParserFormals
this is a proper subset of Formals anyway, so let's just use those and
avoid the extra allocations and moves.
2024-01-15 16:52:18 +01:00
John Ericson
beed00c04e absPath: just take a std::string_view
1. Slightly more efficient

2. Easier to call

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2024-01-15 08:21:54 -05:00
Robert Hensing
268c49264a
Merge pull request #9763 from L-as/avoid-unnecessary-copy
Avoid unnecessary copy of goal log
2024-01-15 08:32:03 +01:00
Las Safin
f61d951909
Avoid unnecessary copy of goal log
The data was (accidentally?) copied into a std::string,
even though the string is immediately converted into a std::string_view.
The code has been changed to construct a std::string_view directly,
such that one copy less happens.
2024-01-13 19:32:37 +00:00
John Ericson
e739a5002d Avoid Windows macros in the parser and lexer
`FLOAT`, `INT`, and `IN` are identifers taken by macros.

The name `IN_KW` is chosen to match `OR_KW`, which is presumably named
that way for the same reason of dodging macros.
2024-01-12 19:51:36 -05:00
Robert Hensing
15f7bdaf27 CanonPath: Add rel_c_str()
Defensively because isRoot() is also defensive.
2024-01-12 22:55:37 +01:00
John Ericson
c58da62a06
Merge pull request #9737 from obsidiansystems/sys-error-split
Separate `SystemError` from `SysError`
2024-01-12 12:41:36 -05:00
John Ericson
6208ca7209 Separate SystemError from SysError
Most of this is a `catch SysError` -> `catch SystemError` sed. This
is a rather pure-churn change I would like to get out of the way. **The
intersting part is `src/libutil/error.hh`.**

On Unix, we will only throw the `SysError` concrete class, which has
the same constructors that `SystemError` used to have.

On Windows, we will throw `WinError` *and* `SysError`. `WinError`
(which will be created in a later PR), will use a `DWORD` instead of
`int` error value, and `GetLastError()`, which is the Windows equivalent
of the `errno` machinery. Windows will *also* use `SysError` because
Window's "libc" (MSVCRT) implements the POSIX interface, and we use it
too.

As the docs describe, while we *throw* one of the 3 choices above (2
concrete classes or the alias), we should always *catch* `SystemError`.
This ensures no matter how the implementation changes for Windows (e.g.
between `SysError` and `WinError`) the catching logic stays the same
and stays correct.

Co-Authored-By volth <volth@volth.com>
Co-Authored-By Eugene Butler <eugene@eugene4.com>
2024-01-12 12:00:33 -05:00
Mel Zuser
25c889baac Fix performance of builtins.substring for empty substrings
When returning a 0-length substring, avoid calling coerceToString,
since it returns a string_view with the string's length, which is
expensive to compute for large strings.
2024-01-12 08:37:04 -08:00
Robert Hensing
274d887fee fetchTree/git: Cache export-ignore filter 2024-01-12 17:18:56 +01:00
Robert Hensing
d80c582b78 libfetchers: Add CachingFilteringInputAccessor
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2024-01-12 17:16:59 +01:00
Eelco Dolstra
52f949bbf5
Merge pull request #9656 from edolstra/nix-profile-stable-names
Make profile element names stable
2024-01-12 16:59:36 +01:00
Robert Hensing
f68ad5acbb fetchTree/git: Don't expose exportIgnore attr 2024-01-12 16:05:36 +01:00
Robert Hensing
469cf263c7 Format 2024-01-12 15:31:53 +01:00
Robert Hensing
692e9197bc fetchTree: Disallow combination of submodules and exportIgnore for now 2024-01-12 15:31:53 +01:00
Robert Hensing
99bd12f0b1 fetchGit/fetchTree: Improve exportIgnore, submodule interaction
Also fingerprint and some preparatory improvements.

Testing is still not up to scratch because lots of logic is duplicated
between the workdir and commit cases.
2024-01-12 15:31:53 +01:00
Robert Hensing
1bbe837184 fetchTree: Add isFetchGit exportIgnore 2024-01-12 15:31:53 +01:00
Robert Hensing
7774eff10e libfetchers/git: Move workdir accessor into GitRepo::getAccessor 2024-01-12 15:31:53 +01:00
Robert Hensing
8024b954d7 fetchTree: Recommend against exportIgnore 2024-01-12 15:31:53 +01:00
Robert Hensing
467c62a96e GitRepoImpl: Move exportIgnore into a filtering accessor 2024-01-12 15:31:53 +01:00
Robert Hensing
cd5e752fa7 GitRepoImpl::getSubmodules: Access getSubmoduleRev without cast
This will be needed because the accessor will be wrapped, and therefore
not be an instance of GitInputAccessor anymore.
2024-01-12 15:31:53 +01:00
Robert Hensing
f6b1d15580 MakeNotAllowedError: Touch up doc 2024-01-12 15:31:53 +01:00
Robert Hensing
1c6bb609af fetchTree: allow larger output attrsets
Intentionally dumb change ahead of architectural improvements.
2024-01-12 15:31:53 +01:00
Robert Hensing
ce6d58a97c git fetcher: Add exportIgnore parameter
Enabled for fetchGit, which historically had this behavior,
among other behaviors we do not want in fetchGit.

fetchTree disables this parameter by default. It can choose the
simpler behavior, as it is still experimental.

I am not confident that the filtering implementation is future
proof. It should reuse a source filtering wrapper, which I believe
Eelco has already written, but not merged yet.
2024-01-12 15:31:52 +01:00
Eelco Dolstra
0d55d660d5
Merge pull request #8043 from bobvanderlinden/pr-shell-env
nix: develop: use SHELL from rc script
2024-01-12 13:42:42 +01:00
Eelco Dolstra
e21b3cf9db Merge remote-tracking branch 'origin/master' into nix-profile-stable-names 2024-01-12 13:36:27 +01:00
Eelco Dolstra
7c6f093abc .data() -> .c_str() to be on the safe side 2024-01-12 13:00:53 +01:00
Eelco Dolstra
66bd1b0298 Merge remote-tracking branch 'origin/master' into pr-shell-env 2024-01-12 12:56:26 +01:00
Eelco Dolstra
3e23759834 gc-non-blocking.sh: Add explanation
Also name the _NIX_TEST_GC_SYNC environment variables logically.
2024-01-12 12:38:55 +01:00
Eelco Dolstra
5703c31325 Merge remote-tracking branch 'origin/master' into handle-missing-gc-socket 2024-01-12 12:26:25 +01:00
Rebecca Turner
df84dd4d8d
Restore ambiguous value printer for nix-instantiate
The Nix team has requested that this output format remain unchanged.
I've added a warning to the man page explaining that `nix-instantiate
--eval` output will not parse correctly in many situations.
2024-01-11 16:34:36 -08:00
Rebecca Turner
0fa08b4516
Unify and refactor value printing
Previously, there were two mostly-identical value printers -- one in
`libexpr/eval.cc` (which didn't force values) and one in
`libcmd/repl.cc` (which did force values and also printed ANSI color
codes).

This PR unifies both of these printers into `print.cc` and provides a
`PrintOptions` struct for controlling the output, which allows for
toggling whether values are forced, whether repeated values are tracked,
and whether ANSI color codes are displayed.

Additionally, `PrintOptions` allows tuning the maximum number of
attributes, list items, and bytes in a string that will be displayed;
this makes it ideal for contexts where printing too much output (e.g.
all of Nixpkgs) is distracting. (As requested by @roberth in
https://github.com/NixOS/nix/pull/9554#issuecomment-1845095735)

Please read the tests for example output.

Future work:
- It would be nice to provide this function as a builtin, perhaps
  `builtins.toStringDebug` -- a printing function that never fails would
  be useful when debugging Nix code.
- It would be nice to support customizing `PrintOptions` members on the
  command line, e.g. `--option to-string-max-attrs 1000`.
2024-01-11 16:34:36 -08:00
Rebecca Turner
c9125603a5
Unindent print.hh declarations 2024-01-11 16:07:01 -08:00
John Ericson
84502674f8
Merge pull request #9736 from obsidiansystems/mingw-makefiles
Some small Makefile improvements to prepare for Windows support
2024-01-11 10:58:55 -05:00
John Ericson
113499d16f
Merge pull request #9634 from 9999years/combine-abstract-pos-and-pos
Combine `AbstractPos`, `PosAdapter`, and `Pos`
2024-01-11 10:47:27 -05:00
John Ericson
423484ad26 Only link with -pthread on Unix
We don't want this with MinGW.
2024-01-10 20:38:39 -05:00
John Ericson
f9e5eb5f0a Make indentation in makesfiles consistent
Tab (as required) for rules, two spaces for `if`...`endif`.
2024-01-10 20:26:34 -05:00
John Ericson
2cea88dbc8 Improve build sytem support for readline instead of editline
Changes:

- CPP variable is now `USE_READLINE` not `READLINE`

- `configure.ac` supports with new CLI flag

- `package.nix` supports with new configuration option

- `flake.nix` CIs this (along with no markdown)

Remove old Ubuntu 16.04 stop-gap too, as that is now quite old.

Motivation:

- editline does not build for Windows, but readline *should*. (I am
  still working on this in Nixpkgs at this time, however. So there will
  be a follow-up Nix PR removing the windows-only skipping of the
  readline library once I am done.)

- Per
  https://salsa.debian.org/debian/nix/-/blob/master/debian/rules?ref_type=heads#L27
  and #2551, Debian builds Nix with readline. Now we better support and
  CI that build configuration.

This is picking up where #2551 left off, ensuring we test a few more
things not merely have CPP for them.

Co-authored-by: Weijia Wang <9713184+wegank@users.noreply.github.com>
2024-01-09 16:56:55 -05:00
Eelco Dolstra
b91c935c2f
Merge pull request #9648 from cole-h/nix-shell-ordering
nix shell: reflect command line order in PATH order
2024-01-09 19:00:33 +01:00
John Ericson
fa2a27d720
Merge pull request #9710 from wegank/disable-lowdown
Make lowdown optional
2024-01-08 15:13:45 -05:00
Weijia Wang
bbd0a959e1 Make lowdown optional
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2024-01-08 20:37:42 +01:00
Rebecca Turner
4feb7d9f71
Combine AbstractPos, PosAdapter, and Pos
Also move `SourcePath` into `libutil`.

These changes allow `error.hh` and `error.cc` to access source path and
position information, which we can use to produce better error messages
(for example, we could consider omitting filenames when two or more
consecutive stack frames originate from the same file).
2024-01-08 10:59:41 -08:00
John Ericson
ff6de4a9ee
Merge pull request #9662 from shlevy/flat-fixed-references-assert
Improve error message for fixed-outputs with references.
2024-01-08 10:46:37 -05:00
Eelco Dolstra
faf87b51f7
Show why GC socket connection was refused
Co-authored-by: John Ericson <git@JohnEricson.me>
2024-01-08 14:14:36 +01:00
Shea Levy
eeb2f083c5
Improve error message for fixed-outputs with references.
This codepath is possible, e.g. with a dockerTools.pullImage of an image with a Nix store.
2024-01-07 07:32:31 -05:00
Eelco Dolstra
359990dfdc
Merge pull request #9324 from 9999years/fix-8854-take-2
Don't attempt to `git add` ignored files
2024-01-05 14:57:32 +01:00
Eelco Dolstra
dedbbbb451
Merge pull request #9670 from DavHau/log-lines
saner default for log-lines: change to 25
2024-01-05 14:39:50 +01:00
Eelco Dolstra
965cfe9688
Merge pull request #9687 from edolstra/withFramedSink-ctrl-c-hang
withFramedSink(): Receive interrupts on the stderr thread
2024-01-04 17:05:18 +01:00
John Ericson
12bb8cdd38 Signer infrastructure: Prep for #9076
This sets up infrastructure in libutil to allow for signing other than
by a secret key in memory. #9076 uses this to implement remote signing.

(Split from that PR to allow reviewing in smaller chunks.)

Co-Authored-By: Raito Bezarius <masterancpp@gmail.com>
2024-01-03 16:13:55 -05:00
Eelco Dolstra
295a2ff8bd Make some more threads receive interrupts
Shouldn't hurt to do this. In particular, this should speed up
shutting down the PathSubstitutionGoal thread if it's copying from a
remote store.
2024-01-03 19:30:02 +01:00
Eelco Dolstra
24e70489e5 withFramedSink(): Receive interrupts on the stderr thread
Otherwise Nix deadlocks when Ctrl-C is received in withFramedSink():
the parent thread will wait forever for the stderr thread to shut
down.

Fixes the hang reported in https://github.com/NixOS/nix/issues/7245#issuecomment-1770560923.
2024-01-03 19:14:50 +01:00
Eelco Dolstra
315aade89d
Merge pull request #9681 from edolstra/eval-optimisations
Optimize empty list constants
2024-01-03 10:43:01 +01:00
Eelco Dolstra
484881f302 Move empty list constant 2024-01-03 10:23:27 +01:00
John Ericson
2b20f36f95 Fix NetBSD build
There was still a mistake after my earlier
a7115a47ef and
e13fc0bbdb. This finally gets it right.
2024-01-02 12:33:51 -05:00
Eelco Dolstra
3f796514b3 Optimize empty list constants
This avoids a Value allocation for empty list constants. During a `nix
search nixpkgs`, about 82% of all thunked lists are empty, so this
removes about 3 million Value allocations.

Performance comparison on `nix search github:NixOS/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 --no-eval-cache`:

maximum RSS:        median = 3845432.0000  mean = 3845432.0000  stddev =      0.0000  min = 3845432.0000  max = 3845432.0000  [rejected?, p=0.00000, Δ=-70084.00000±0.00000]
soft page faults:   median = 965395.0000  mean = 965394.6667  stddev =      1.1181  min = 965392.0000  max = 965396.0000  [rejected?, p=0.00000, Δ=-17929.77778±38.59610]
system CPU time:    median =      1.8029  mean =      1.7702  stddev =      0.0621  min =      1.6749  max =      1.8417  [rejected, p=0.00064, Δ=-0.12873±0.09905]
user CPU time:      median =     14.1022  mean =     14.0633  stddev =      0.1869  min =     13.8118  max =     14.3190  [not rejected, p=0.03006, Δ=-0.18248±0.24928]
elapsed time:       median =     15.8205  mean =     15.8618  stddev =      0.2312  min =     15.5033  max =     16.1670  [not rejected, p=0.00558, Δ=-0.28963±0.29434]
2024-01-02 12:49:11 +01:00
Eelco Dolstra
3f834f5c64
Merge pull request #9657 from edolstra/fix-getNameFromURL
getNameFromURL(): Support uppercase characters in attribute names
2024-01-02 12:48:17 +01:00
Robert Hensing
83f5622545
Merge pull request #9658 from pennae/env-diet
reduce the size of Env by one pointer
2023-12-31 13:57:16 +01:00
Robert Hensing
3511430902
Merge pull request #9673 from pennae/drv-parse-opts
optimize derivation parsing
2023-12-31 13:49:03 +01:00
pennae
1fe66852ff reduce the size of Env by one pointer
since `up` and `values` are both pointer-aligned the type field will
also be pointer-aligned, wasting 48 bits of space on most machines. we
can get away with removing the type field altogether by encoding some
information into the `with` expr that created the env to begin with,
reducing the GC load for the absolutely massive amount of single-entry
envs we create for lambdas. this reduces memory usage of system eval by
quite a bit (reducing heap size of our system eval from 8.4GB to 8.23GB)
and gives similar savings in eval time.

running `nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'`

before:

  Time (mean ± σ):      5.576 s ±  0.003 s    [User: 5.197 s, System: 0.378 s]
  Range (min … max):    5.572 s …  5.581 s    10 runs

after:

  Time (mean ± σ):      5.408 s ±  0.002 s    [User: 5.019 s, System: 0.388 s]
  Range (min … max):    5.405 s …  5.411 s    10 runs
2023-12-30 18:55:13 +01:00
pennae
c62686a95b reduce copies during drv parsing
many paths need not be heap-allocated, and derivation env name/valye
pairs can be moved into the map.

before:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.883 s ±  0.016 s    [User: 5.250 s, System: 1.424 s]
  Range (min … max):    6.860 s …  6.905 s    10 runs

after:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.868 s ±  0.027 s    [User: 5.194 s, System: 1.466 s]
  Range (min … max):    6.828 s …  6.913 s    10 runs
2023-12-30 18:44:15 +01:00
pennae
02c64abf1e use translation table for drv string parsing
the table is very small compared to cache sizes and a single indexed
load is faster than three comparisons.

before:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.907 s ±  0.012 s    [User: 5.272 s, System: 1.429 s]
  Range (min … max):    6.893 s …  6.926 s    10 runs

after:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.883 s ±  0.016 s    [User: 5.250 s, System: 1.424 s]
  Range (min … max):    6.860 s …  6.905 s    10 runs
2023-12-30 18:44:15 +01:00
pennae
79d3d412ca optimize derivation string parsing
a bunch of derivation strings contain no escape sequences. we can
optimize for this fact by first scanning for the end of a derivation
string and simply returning the contents unmodified if no escape
sequences were found. to make this even more efficient we can also use
BackedStringViews to avoid copies, avoiding heap allocations for
transient data.

before:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.952 s ±  0.015 s    [User: 5.294 s, System: 1.452 s]
  Range (min … max):    6.926 s …  6.974 s    10 runs

after:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.907 s ±  0.012 s    [User: 5.272 s, System: 1.429 s]
  Range (min … max):    6.893 s …  6.926 s    10 runs
2023-12-30 18:44:10 +01:00
Rebecca Turner
7434caca05
Fix segfault on infinite recursion in some cases
This fixes a segfault on infinite function call recursion (rather than
infinite thunk recursion) by tracking the function call depth in
`EvalState`.

Additionally, to avoid printing extremely long stack traces, stack
frames are now deduplicated, with a `(19997 duplicate traces omitted)`
message. This should only really be triggered in infinite recursion
scenarios.

Before:

    $ nix-instantiate --eval --expr '(x: x x) (x: x x)'
    Segmentation fault: 11

After:

    $ nix-instantiate --eval --expr '(x: x x) (x: x x)'
    error: stack overflow

           at «string»:1:14:
                1| (x: x x) (x: x x)
                 |              ^

    $ nix-instantiate --eval --expr '(x: x x) (x: x x)' --show-trace
    error:
           … from call site
             at «string»:1:1:
                1| (x: x x) (x: x x)
                 | ^

           … while calling anonymous lambda
             at «string»:1:2:
                1| (x: x x) (x: x x)
                 |  ^

           … from call site
             at «string»:1:5:
                1| (x: x x) (x: x x)
                 |     ^

           … while calling anonymous lambda
             at «string»:1:11:
                1| (x: x x) (x: x x)
                 |           ^

           … from call site
             at «string»:1:14:
                1| (x: x x) (x: x x)
                 |              ^

           (19997 duplicate traces omitted)

           error: stack overflow
           at «string»:1:14:
                1| (x: x x) (x: x x)
                 |              ^
2023-12-29 22:16:44 -08:00
pennae
2cfc4ace35 malloc/memset even less
more buffers that can be uninitialized and on the stack. small
difference, but still worth doing.

before:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.963 s ±  0.011 s    [User: 5.330 s, System: 1.421 s]
  Range (min … max):    6.943 s …  6.974 s    10 runs

after:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.952 s ±  0.015 s    [User: 5.294 s, System: 1.452 s]
  Range (min … max):    6.926 s …  6.974 s    10 runs
2023-12-30 00:45:11 +01:00
pennae
99a691c8a1 don't use istreams in hot paths
istream sentry objects are very expensive for single-character
operations, and since we don't configure exception masks for the
istreams used here they don't even do anything. all we need is
end-of-string checks and an advancing position in an immutable memory
buffer, both of which can be had for much cheaper than istreams allow.

the effect of this change is most apparent on empty stores.

before:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      7.167 s ±  0.013 s    [User: 5.528 s, System: 1.431 s]
  Range (min … max):    7.147 s …  7.182 s    10 runs

after:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.963 s ±  0.011 s    [User: 5.330 s, System: 1.421 s]
  Range (min … max):    6.943 s …  6.974 s    10 runs
2023-12-30 00:45:10 +01:00
DavHau
b6313f64f7 saner default for log-lines: change to 25
This seems to be a much saner default. 10 lines are just not enough in so many cases.
2023-12-27 19:57:27 +07:00
Brian Le
e2399fc949 Change "dervation" typos to "derivation" 2023-12-26 17:12:28 -05:00
Shea Levy
c3942ef85f
Build IFD in the build store when using eval-store.
Previously, IFDs would be built within the eval store, even though one
is typically using `--eval-store` precisely to *avoid* local builds.

Because the resulting Nix expression must be copied back to the eval
store in order to be imported, this requires the eval store to trust
the build store's signatures.
2023-12-23 21:33:56 -05:00
Robert Hensing
ee439734e9
Merge pull request #9582 from pennae/misc-opts
a packet of small optimizations
2023-12-22 17:00:59 +01:00
Eelco Dolstra
cf02b3335c Merge remote-tracking branch 'origin/master' into tarball-cache 2023-12-22 16:45:01 +01:00
Eelco Dolstra
936a364226 getNameFromURL(): Support uppercase characters in attribute names
In particular, this makes it handle 'legacyPackages' correctly.
2023-12-22 16:35:58 +01:00
Eelco Dolstra
a748e88bf4 nix profile: Remove check for "name" attribute in manifests
AFAIK, we've never emitted this attribute.
2023-12-22 16:27:31 +01:00
John Ericson
d65d56fa77
Merge pull request #9655 from iFreilicht/move-flake-url-name-util
Move url-name utility to libexpr/flake
2023-12-22 10:26:42 -05:00
Eelco Dolstra
6268a45b65 nix profile: Make profile element names stable
The profile manifest is now an object keyed on the name returned by
getNameFromURL() at installation time, instead of an array. This
ensures that the names of profile elements don't change when other
elements are added/removed.
2023-12-22 16:21:30 +01:00
Eelco Dolstra
3187bc9ac3 nix profile: Remove indices 2023-12-22 16:21:30 +01:00
Robert Hensing
d2016c6b59
Merge pull request #9621 from blaggacao/fix/too-restrictive-branch-regex-master
fix: valid branch name
2023-12-22 16:02:25 +01:00
Felix Uhl
26d7b0c793 Move url-name utility to libexpr/flake 2023-12-22 09:33:02 +01:00
Robert Hensing
4f47152209
libutil/url-parts.hh: Fix regex
Regex syntax is awful.
2023-12-21 23:11:25 +01:00
Felix Uhl
397cf4e285 nix search: Disallow empty regex
Fixes #4739
Fixes #3553 in spirit IMO
2023-12-21 22:13:43 +01:00
Robert Hensing
8c4ea12f11
libutil/url-parts.hh: comment 2023-12-21 21:03:06 +01:00
Cole Helbling
f4454aac9f nix shell: reflect command line order in PATH order
Prior to this change, Nix would prepend every installable to the PATH
list in order to ensure that installables appeared before the current
PATH from the ambient environment.

With this change, all the installables are still prepended to the PATH,
but in the same order as they appear on the command line. This means
that the first of two packages that expose an executable `hello` would
appear in the PATH first, and thus be executed first.

See the test in the prior commit for a more concrete example.
2023-12-21 10:56:19 -08:00
Cole Helbling
5ed1884875 libcmd: Installable::toStorePaths -> Installable::toStorePathSet 2023-12-21 10:23:07 -08:00
Eelco Dolstra
14508ade28 Typo 2023-12-21 16:25:55 +01:00
Eelco Dolstra
d77a39a314 Fix indent 2023-12-21 16:22:34 +01:00
Eelco Dolstra
9d9d9ff0de Merge remote-tracking branch 'origin/master' into profile-names-instead-of-index 2023-12-21 16:21:26 +01:00
Silvan Mosberger
ea454d8687 Undeprecate isNull
There's no good reason to deprecate it:
- For consistency reasons it should continue to exist, such that all
primitive types have a corresponding `builtins.is*` primop.
- There's no implementation cost to continuing to have this function
- It costs users time to try to migrate away from it, e.g.
  https://github.com/NixOS/nixpkgs/pull/219747 and https://github.com/NixOS/nixpkgs/pull/275548
- Using it can give easier-to-read code like `all isNull list`

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-12-20 17:55:19 +01:00
pennae
2b0e95e7aa use singleton expr to generate black hole errors
this also reduces forceValue code size and removes the need for
hideInDiagnostics. coopting thunk forcing like this has the additional
benefit of clarifying how these errors can happen in the first place.
2023-12-19 19:32:16 +01:00
pennae
f9db4de0f3 force-inline forceValue
forceValue is extremely hot. interestingly adding likeliness annotations
to the branches does not seem to make a difference.

before:

  Time (mean ± σ):      4.224 s ±  0.005 s    [User: 3.711 s, System: 0.512 s]
  Range (min … max):    4.218 s …  4.234 s    10 runs

after:

  Time (mean ± σ):      4.140 s ±  0.009 s    [User: 3.647 s, System: 0.492 s]
  Range (min … max):    4.130 s …  4.152 s    10 runs
2023-12-19 19:32:16 +01:00
pennae
69ed4aee61 remove lazy-pos forceValue
almost all uses of this are interactive, except for deepSeq. deepSeq is
going to be expensive and rare enough to not care much about, and
Value::determinePos should usually be cheap enough to not be too much of
a burden in any case.
2023-12-19 19:32:16 +01:00
pennae
f9aee2f2c4 don't malloc/memset posix accessor buffer
it's relatively small and fits on the stack nicely, and we don't need it
initialized either.
2023-12-19 19:32:16 +01:00
pennae
b78e77b34c use custom location type in the parser
~1% parser speedup from not using TLS indirections, less on system eval.
this could have also gone in flex yyextra data, but that's significantly
slower for some reason (albeit still faster than thread locals).

before:

  Time (mean ± σ):      4.231 s ±  0.004 s    [User: 3.725 s, System: 0.504 s]
  Range (min … max):    4.226 s …  4.240 s    10 runs

after:

  Time (mean ± σ):      4.224 s ±  0.005 s    [User: 3.711 s, System: 0.512 s]
  Range (min … max):    4.218 s …  4.234 s    10 runs
2023-12-19 19:32:16 +01:00
pennae
2e0321912a use aligned flex tables
~2% speedup on parsing without eval, less (but still significant) on
system eval. having flex generate faster parsers leads to very strange
misparses. maybe re2c is worth investigating.

before:

  Time (mean ± σ):      4.260 s ±  0.003 s    [User: 3.754 s, System: 0.505 s]
  Range (min … max):    4.257 s …  4.266 s    10 runs

after:

  Time (mean ± σ):      4.231 s ±  0.004 s    [User: 3.725 s, System: 0.504 s]
  Range (min … max):    4.226 s …  4.240 s    10 runs
2023-12-19 19:32:16 +01:00
pennae
cc4038d541 use std::tie() for macro-generated operators
as written the comparisons generate copies, even though it looks as
though they shouldn't.

before:

  Time (mean ± σ):      4.396 s ±  0.002 s    [User: 3.894 s, System: 0.501 s]
  Range (min … max):    4.393 s …  4.399 s    10 runs

after:

  Time (mean ± σ):      4.260 s ±  0.003 s    [User: 3.754 s, System: 0.505 s]
  Range (min … max):    4.257 s …  4.266 s    10 runs
2023-12-19 19:32:16 +01:00
pennae
74c134914c compare string values with strcmp
string_view()ification calls strlen() first, which we don't need here.
2023-12-19 19:32:16 +01:00
pennae
78353deb02 encode black holes as tApp values
checking for isBlackhole in the forceValue hot path is rather more
expensive than necessary, and with a little bit of trickery we can move
such handling into the isApp case. small performance benefit, but under
some circumstances we've seen 2% improvement as well.

〉 nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'

before:

  Time (mean ± σ):      4.429 s ±  0.002 s    [User: 3.929 s, System: 0.500 s]
  Range (min … max):    4.427 s …  4.433 s    10 runs

after:

  Time (mean ± σ):      4.396 s ±  0.002 s    [User: 3.894 s, System: 0.501 s]
  Range (min … max):    4.393 s …  4.399 s    10 runs
2023-12-19 19:32:16 +01:00
pennae
0218e4e6c3 memset less in addToStoreFromDump
resizing a std::string clears the newly added bytes, which is not
necessary here and comes with a ~1.4% slowdown on our test nixos config.

〉 nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'

before:

  Time (mean ± σ):      4.486 s ±  0.003 s    [User: 3.978 s, System: 0.507 s]
  Range (min … max):    4.482 s …  4.492 s    10 runs

after:

  Time (mean ± σ):      4.429 s ±  0.002 s    [User: 3.929 s, System: 0.500 s]
  Range (min … max):    4.427 s …  4.433 s    10 runs
2023-12-19 19:32:16 +01:00
Eelco Dolstra
7cfd6c0efe
Merge pull request #9325 from NixOS/accessor-add-to-store
Content addressing and adding to store cleanup
2023-12-19 15:10:31 +01:00
Andrew Marshall
7526b7ded6 Allow access to /dev/stderr in Darwin sandbox
We allow /dev/stdout, so why not this? Since it is process-local,
anyway, should not be possible to escape sandbox using it.
2023-12-18 19:33:20 -05:00
John Ericson
ed26b186fb Remove now-redundant text-hashing store methods
`addTextToStore` and `computeStorePathFromDump` are now redundant.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-12-18 10:44:10 -05:00
John Ericson
dfc876531f Organize content addressing, use SourceAccessor with Store::addToStore
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-12-18 10:41:54 -05:00
Théophane Hufschmitt
ebfbc5a6a5
Merge pull request #9628 from fricklerhandwerk/add-links
add cross-reference
2023-12-18 13:07:39 +01:00
Valentin Gagarin
4f95800854 add cross-reference 2023-12-18 11:41:52 +01:00
Julia Evans
a47fabff0d use params.isFetchGit instead to check if it came from fetchGit 2023-12-17 12:14:55 -05:00
Julia Evans
06bed2eacd Make fetchTree locked input error message clearer 2023-12-17 12:04:59 -05:00
David Arnold
c05d4fadd5
fix: valid branch name 2023-12-16 23:14:33 +01:00
Rebecca Turner
5cb98095ba
Remove some blank lines from stack traces
This keeps hint messages, source location information, and source code
snippets grouped together, while making stack traces shorter (so that
more stack frames can be viewed on the same terminal).

Before:

    error:
           … while evaluating the attribute 'body'

             at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:4:3:

                3|
                4|   body = x "x";
                 |   ^
                5| }

           … from call site

             at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:4:10:

                3|
                4|   body = x "x";
                 |          ^
                5| }

           … while calling 'x'

             at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:2:7:

                1| let {
                2|   x = arg: assert arg == "y"; 123;
                 |       ^
                3|

           error: assertion '(arg == "y")' failed

           at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:2:12:

                1| let {
                2|   x = arg: assert arg == "y"; 123;
                 |            ^
                3|

After:

    error:
           … while evaluating the attribute 'body'
             at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:4:3:
                3|
                4|   body = x "x";
                 |   ^
                5| }

           … from call site
             at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:4:10:
                3|
                4|   body = x "x";
                 |          ^
                5| }

           … while calling 'x'
             at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:2:7:
                1| let {
                2|   x = arg: assert arg == "y"; 123;
                 |       ^
                3|

           error: assertion '(arg == "y")' failed
           at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:2:12:
                1| let {
                2|   x = arg: assert arg == "y"; 123;
                 |            ^
                3|
2023-12-15 23:57:26 -08:00
John Ericson
e7d95a1d80 Merge remote-tracking branch 'upstream/master' into tarball-cache 2023-12-15 09:37:13 -05:00
Matthew Bauer
bcbdb09ccf Add eval-system option
`eval-system` option overrides just the value of `builtins.currentSystem`.
This is more useful than overriding `system` since you can build these
derivations on remote builders which can work on the given system.

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-12-14 19:04:00 -05:00
John Ericson
e13fc0bbdb Fix sys/xattr.h check
I wrote the `configure.ac` wrong, and so we just got no builds
supporting ACLs.

Also, it needs to be more precise because Darwin puts other stuff in
that same header, evidently.
2023-12-14 10:03:48 -05:00
Eelco Dolstra
06e106beff Disable GitHub tree hash mismatch warning 2023-12-14 13:38:10 +01:00
Eelco Dolstra
1fce12ec32 Merge remote-tracking branch 'origin/master' into tarball-cache 2023-12-14 13:31:29 +01:00
Ramses
1e3d811840
worker protocol: serialise cgroup stats in BuildResult (#9598)
By doing so, they get reported when building through the daemon via either `unix://` or `ssh-ng://`.
2023-12-13 16:37:17 -05:00
Eelco Dolstra
1b7968ed86
Merge pull request #9547 from hercules-ci/allowed-scheme-without-slash
`allowed-uris`: match whole schemes without slashes
2023-12-13 20:23:33 +01:00
Théophane Hufschmitt
b1c633c6bb
Merge pull request #9600 from SharzyL/fix_nix_copy
fix: nix copy ssh-ng:// not respecting --substitute-on-destination
2023-12-13 18:08:38 +01:00
Eelco Dolstra
103ca0bde5 Improve SourcePath display 2023-12-13 13:27:29 +01:00
Eelco Dolstra
cc3913e458 Remove unused variable 2023-12-13 13:27:23 +01:00
SharzyL
04f454f2a0
fix: nix copy ssh-ng:// not respecting --substitute-on-destination 2023-12-13 10:30:28 +08:00
tomberek
7026abfdde
Merge pull request #9523 from fricklerhandwerk/conf-reword-always-allow-substitutes
reword documentation on settings and attributes related to substitution
2023-12-12 20:09:48 -05:00
tomberek
09041071bf
Merge pull request #9525 from fricklerhandwerk/conf-reword-builders-use-substitutes
reword description of the `builders-use-substitutes` setting
2023-12-12 20:08:00 -05:00
Robert Hensing
0b87ba50c0 Revert "Add nix::isASCII*, locale-independent"
This reverts commit 79eb2920bb.

Not used at this time.
2023-12-12 17:46:34 +01:00
Robert Hensing
4eaeda6604 isValidSchemeName: Use regex
As requested by Eelco Dolstra. I think it used to be simpler.
2023-12-12 17:46:34 +01:00
Robert Hensing
2e451a663e schemeRegex -> schemeNameRegex
Scheme could be understood to include the typical `:` separator.
2023-12-12 17:25:20 +01:00
Théophane Hufschmitt
0dfa66d120
Merge pull request #9593 from B4dM4n/fix-path-like-flake-query
Fix query parsing for path-like flakes
2023-12-12 17:15:30 +01:00
John Ericson
9f39dda66c Fix building CA derivations with and eval store
I don't love the way this code looks. There are two larger problems:

- eval, build/scratch, destination stores (#5025) should have different
  types to reflect the fact that they are used for different purposes
  and those purposes correspond to different operations. It should be
  impossible to "use the wrong store" in my cases.

- Since drvs can end up in both the eval and build/scratch store, we
  should have some sort of union/layered store (not on the file sytem
  level, just conceptual level) that allows accessing both. This would
  get rid of the ugly "check both" boilerplate in this PR.

Still, it might be better to land this now / soon after minimal cleanup,
so we have a concrete idea of what problem better abstractions are
supposed to solve.
2023-12-11 12:17:36 -05:00
John Ericson
8cddda4f89
Merge pull request #9588 from obsidiansystems/queryDerivationOutputMap-evalStore
Give `Store::queryDerivationOutputMap` and `evalStore` argument
2023-12-11 11:16:18 -05:00
John Ericson
5f30c8acc7 Give Store::queryDerivationOutputMap and evalStore argument
Picking up where https://github.com/NixOS/nix/pull/9563 left off.
2023-12-11 10:39:08 -05:00
Fabian Möller
f45d2ee2b7
Fix query parsing for path-like flakes 2023-12-11 16:02:09 +01:00
Robert Hensing
da58c00ee0
Merge pull request #9586 from obsidiansystems/legacy-ssh-store-header
Create header for `LegacySSHStore`
2023-12-11 12:21:56 +01:00
Robert Hensing
a05bc9eb92 allowed-uris: Match whole schemes also when scheme is not followed by slashes 2023-12-11 12:18:04 +01:00
Robert Hensing
d3a85b6834 isValidSchemeName: Add function 2023-12-11 12:12:43 +01:00
Robert Hensing
79eb2920bb Add nix::isASCII*, locale-independent 2023-12-11 12:12:43 +01:00
Robert Hensing
1fa958dda1 isAllowedURI: Format 2023-12-11 12:12:43 +01:00
Robert Hensing
6cbba914a7 isAllowedURI: Remove incorrect note 2023-12-11 12:12:43 +01:00
Robert Hensing
91ba7b2307 isAllowedURI: Extract function and test 2023-12-11 12:12:42 +01:00
Adam Joseph
e43bb655fe libstore/daemon.cc: note trust model difference in readDerivation()s
Below the comment added by this commit is a much longer comment
followed by a trust check, both of which have confused me on at
least two occasions.  I figured it out once, forgot it, then had to
ask @Ericson2314 to explain it, at which point I understood it
again.  I think this might confuse other people too, or maybe I will
just forget it a third time.  So let's add a comment.

Farther down in the function is the following check:

```
if (!(drvType.isCA() || trusted))
  throw Error("you are not privileged to build input-addressed derivations");
```

This seems really strange at first.  A key property of Nix is that
you can compute the outpath of a derivation using the derivation
(and its references-closure) without trusting anybody!

The missing insight is that at this point in the code the builder
doesn't necessarily have the references-closure of the derivation
being built, and therefore needs to trust that the derivation's
outPath is honest.  It's incredibly easy to overlook this, because
the only difference between these two cases is which of these
identically-named functions we used:

- `readDerivation(Source,Store)`
- `Store::readDerivation()`

These functions have different trust models (except in the special
case where the first function is used on the local store).  We
should call the reader's attention to this fact.

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2023-12-10 17:47:07 -08:00
John Ericson
deadb3bfe9 Create header for LegacySSHStore
In https://github.com/NixOS/nix/pull/6134#issuecomment-1079199888,
@thuffschmitt proposed exposing `LegacySSHStore` in Nix for
deduplication with Hydra, at least temporarily. I think that is a good
idea.

Note that the diff will look bad unless one ignores whitespace! Also try
this locally:

```shell-session
git diff --ignore-all-space HEAD^:src/libstore/legacy-ssh-store.cc HEAD:src/libstore/legacy-ssh-store.cc
git diff --ignore-all-space HEAD^:src/libstore/legacy-ssh-store.cc HEAD:src/libstore/legacy-ssh-store.hh
```
2023-12-10 14:29:09 -05:00
John Ericson
b7e016ab24
Merge pull request #9572 from obsidiansystems/serve-proto-build-options
Create `ServeProto::BuildOptions` and a serializer for it
2023-12-10 12:16:02 -05:00
Valentin Gagarin
3c200da242
document fetchTree (#9258)
* document `fetchTree`

* display experimental feature note at the top

we have to enable the new `fetchTree` experimental feature to render it
at all. this was a bug introduced when adding that new feature flag.

Co-authored-by: tomberek <tomberek@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Silvan Mosberger <github@infinisil.com>
2023-12-10 05:16:32 +00:00
John Ericson
5417990e31 Create ServeProto::BuildOptions and a serializer for it
More tests, and more serializers for Hydra reuse.
2023-12-09 11:35:13 -05:00
Robert Hensing
c8458bd731
Merge pull request #9555 from 9999years/positions-in-errors
Pass positions when evaluating
2023-12-09 03:55:58 +01:00
Robert Hensing
7cdc8786d9
Merge pull request #9568 from hercules-ci/revert-9553
Revert 9553
2023-12-09 03:49:53 +01:00
Robert Hensing
9b7b7a7561 Revert "Print the value in error: cannot coerce messages (#9553)"
This reverts commit f0ac2a35d5.

The request from the sibling PR, which also applies here, was not addressed.
https://github.com/NixOS/nix/pull/9554#issuecomment-1845095735
2023-12-09 02:13:32 +01:00
John Ericson
ce4ca574d2 Clarify SourceAccessor methods should never implicitly follow symlinks
The code has already been fixed (yay!) so what is left of this commit is
just updating the API docs.

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2023-12-08 20:00:47 -05:00
Robert Hensing
d4f6b1d38b
Merge pull request #9497 from edolstra/move-access-control
Move restricted/pure-eval access control out of the evaluator and into the accessor
2023-12-08 22:21:50 +01:00
John Ericson
762af72728
Merge pull request #9564 from NixOS/fix-clang-build
Avoid `std::strstream`, fix the clang build
2023-12-08 15:07:36 -05:00
Théophane Hufschmitt
36ca6adc60
Merge pull request #9563 from obsidiansystems/tryResolve-evalStore
Give `Derivation::tryResolve` an `evalStore` argument
2023-12-08 19:21:35 +01:00
John Ericson
f9ee1bedcf Avoid std::strstream, fix the clang build
According https://en.cppreference.com/w/cpp/io/strstream, it has been
deprecated since C++98! The Clang + Linux build systems to not have it
at all, or at least be hiding it.

We can just use `std::stringstream` instead, I think.
2023-12-08 13:18:52 -05:00
Rebecca Turner
f0ac2a35d5
Print the value in error: cannot coerce messages (#9553)
* Print the value in `error: cannot coerce` messages

This extends the `error: cannot coerce a TYPE to a string` message
to print the value that could not be coerced. This helps with debugging
by making it easier to track down where the value is being produced
from, especially in errors with deep or unhelpful stack traces.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-12-08 16:36:57 +00:00
John Ericson
96dd757b0c Give Derivation::tryResolve an evalStore argument
This is needed for building CA deriations with a src store / dest store
split. In particular it is needed for Hydra.

https://github.com/NixOS/hydra/issues/838 currently puts realizations,
and thus build outputs, in the local store, but it should not.
2023-12-08 10:01:05 -05:00
Rebecca Turner
0b80935c22
Pass positions when evaluating
This includes position information in more places, making debugging
easier.

Before:

```
$ nix-instantiate --show-trace --eval tests/functional/lang/eval-fail-using-set-as-attr-name.nix
error:
       … while evaluating an attribute name

         at «none»:0: (source not available)

       error: value is a set while a string was expected
```

After:

```
error:
       … while evaluating an attribute name

         at /pwd/lang/eval-fail-using-set-as-attr-name.nix:5:10:

            4| in
            5|   attr.${key}
             |          ^
            6|

       error: value is a set while a string was expected
```
2023-12-07 10:27:21 -08:00
John Ericson
a5521b7d94 Factor out ServeProto::Serialiser<UnkeyedValidPathInfo> and test
In the process, partially undo e89b5bd0bf
in that the ancient < 2.4 version is now supported again by the
serializer again. `LegacySSHStore`, instead of also asserting that the
version is at least 4, just checks that `narHash` is set.

This allows us to better test the serializer in isolation for both
versions (< 4 and >= 4).
2023-12-07 11:34:18 -05:00
Bryan Honof
bf00d5ecef
fix(libutil/tarfile): add option to libarchive so it behaves correctly with AppleDouble files
AppleDouble files were extracted differently on macOS machines than on other
UNIX's.
Setting `archive_read_set_format_option(this->archive, NULL ,"mac-ext",NULL)`
fixes this problem, since it just ignores the AppleDouble file and treats it as
a normal one.
This was a problem since it caused source archives to be different between macOS
and Linux.

Ref: nixos/nix#9290
2023-12-07 11:35:15 +01:00
Peter Kolloch
d38ec12855 Update src/libexpr/primops.cc
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-12-06 23:43:42 +01:00
Peter Kolloch
8afeaf05c4 Add docs/rl-notes for nix hash convert / builtins.convertHash
https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch
7ff876b92b Add deprecation notice for old nix hash conversion subcommands.
(But not yet nix-hash since `nix hash` is still hidden behind a feature flag.)

https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch
fc6f29053a Renamed HashFormat::Base32 to HashFormat::Nix32
...and also adjusted parsing accordingly.

Also added CLI completion for HashFormats.

https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch
837b889c41 Further HashType renaming + using mkHashAlgoOptFlag for new conversion
https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch
5334c9c792 HashType: Rename to HashAlgorithm
To be consistent with CLI, nix API
and many other references.

As part of this, we also converted it to a scoped enum.

https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch
0c2d5f7673 nix hash convert: s/--type/--algo/ + more functional tests
https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch
6bbd900d4f nix hash convert: added
This deviated from the proposal! See comments on the issue.

https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:34 +01:00
Peter Kolloch
156ea78d74 CmdHashBase: doc comment 2023-12-06 23:41:07 +01:00
Peter Kolloch
e7abf60a0c hash.cc/hash.h: Minor C++ improvements 2023-12-06 23:41:07 +01:00
Théophane Hufschmitt
7fff625e39
Improve the error message for “multicommands” commands (#9510)
* Factor out the default `MultiCommand` behavior

All the `MultiCommand`s had (nearly) the same behavior when called
without a subcommand.
Factor out this behavior into the `NixMultiCommand` class.

* Display the list of available subcommands when none is specified

Whenever a user runs a command that excepts a subcommand, add the list
of available subcommands to the error message.

* Print the multi-command lists as Markdown lists

This takes more screen real estate, but is also much more readable than
a comma-separated list
2023-12-06 13:13:45 +00:00
Eelco Dolstra
2bd8322500
Update src/libfetchers/filtering-input-accessor.hh
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-12-06 14:08:40 +01:00
Eelco Dolstra
57246c4c38 PosixSourceAccessor: Cache lstat() calls
Since we're doing a lot of them in assertNoSymlinks().
2023-12-06 13:55:54 +01:00
Eelco Dolstra
504e4fc457 CanonPath: Support std::hash 2023-12-06 13:45:59 +01:00
Eelco Dolstra
83c067c0fa PosixSourceAccessor: Don't follow any symlinks
All path components must not be symlinks now (so the user needs to
call `resolveSymlinks()` when needed).
2023-12-05 23:02:59 +01:00
Valentin Gagarin
3c310bde2e reword description for the fetch-tree experimental feature
without knowing a lot of context, it's not clear who "we" are in that
text. I'm also strongly opposed to adding procedural notes into
a reference manual; it just won't age well.

this change leaves a factual description of the experimental feature and
its purpose.
2023-12-04 16:42:32 +01:00
Valentin Gagarin
5b281ddf50 reword description of the max-jobs setting
- remove prose for the default value, which is shown programmatically
- add note on how this relates to `cores`
- add link to mentioned derivation attribute
2023-12-02 04:06:26 +01:00
Valentin Gagarin
368fdb482d reword description of the builders-use-substitutes setting 2023-12-02 03:07:27 +01:00
Valentin Gagarin
24b781773f fix random docs errors
remove link to the contributing guide from user documentation.
it doesn't help here, and the target at first glance shows redundant
information.
2023-12-02 03:02:59 +01:00
Valentin Gagarin
51adfb9b27 reword documentation on settings and attributes related to substitution
- add links
- be more concise
- clarify the distinction between `preferLocalBuild` and `allowSubstitutes`
2023-12-02 02:56:25 +01:00
John Ericson
91b6833686 Move tests to separate directories, and document
Today, with the tests inside a `tests` intermingled with the
corresponding library's source code, we have a few problems:

- We have to be careful that wildcards don't end up with tests being
  built as part of Nix proper, or test headers being installed as part
  of Nix proper.

- Tests in libraries but not executables is not right:

  - It means each executable runs the previous unit tests again, because
    it needs the libraries.

  - It doesn't work right on Windows, which doesn't want you to load a
    DLL just for the side global variable . It could be made to work
    with the dlopen equivalent, but that's gross!

This reorg solves these problems.

There is a remaining problem which is that sibbling headers (like
`hash.hh` the test header vs `hash.hh` the main `libnixutil` header) end
up shadowing each other. This PR doesn't solve that. That is left as
future work for a future PR.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-12-01 10:48:58 -05:00
Robert Hensing
77adb55ae4
Merge pull request #9511 from obsidiansystems/add-missing-includes
Add two missing `#include "nar-info.hh"`
2023-12-01 16:43:56 +01:00
John Ericson
d59bdbe4fd Add two missing #include "nar-info.hh"
GitHub's racy CI caused this oversight to sneak through.
2023-12-01 10:20:19 -05:00
John Ericson
ac4d2e7b85
Merge pull request #9478 from tweag/nix-config-check
Rename `nix doctor` to `nix config check`
2023-12-01 10:19:19 -05:00
Théophane Hufschmitt
dfa219d03b
Merge pull request #9494 from sellout/nix-run-execv
Don’t use `execvp` when we know the path
2023-12-01 16:07:16 +01:00
Robert Hensing
fcf09813c6
Merge pull request #6236 from obsidiansystems/store-dir-config
Factor out `StoreDirConfig`
2023-12-01 15:38:14 +01:00
Théophane Hufschmitt
ea2dd16623 Use a proper enum rather than a boolean in runProgramInStore
Makes the call-site much easier to understand.
2023-12-01 15:35:21 +01:00
John Ericson
82359eba6b
Merge pull request #9233 from bouk/bouk/apply-config-inner
config: add included files into parsedContents before applying
2023-12-01 08:23:32 -05:00
Robert Hensing
188c803ddb
Merge pull request #9508 from infinisil/revert-7762-boost-regex
Revert "Switch from std::regex to boost::regex"
2023-12-01 02:08:58 +01:00
Silvan Mosberger
333ea684b0 Add boost::regex regression test 2023-12-01 01:39:52 +01:00
Valentin Gagarin
4781e7fa70 Document each store type on its own page
This makes for more useful manual table of contents, that displays the
information at a glance.

The `nix help-stores` command is kept as-is, even though it will show up
in the manual with the same information as these pages due to the way it
is written as a "`--help`-style" command. Deciding what to do with that
command is left for a later PR.

This change also lists all store types at the top of the respective overview page.

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems
2023-12-01 01:27:52 +01:00
Silvan Mosberger
908a011a4a
Revert "Switch from std::regex to boost::regex" 2023-12-01 00:50:20 +01:00
Valentin Gagarin
39de819eda rename debugging helper environment variable 2023-11-30 23:23:32 +01:00
Eelco Dolstra
8cafc754d8 Move access control from FSInputAccessor to FilteringInputAccessor 2023-11-30 21:54:53 +01:00
Valentin Gagarin
44d21f6ef9 keep generated documentation in a separate directory
- helps navigating the code as it highlights which files are generated
- makes it less error prone when working incrementally
  (although this should be just fixed by building out of tree)
2023-11-30 20:39:24 +01:00
Rebecca Turner
be30c2ea8d
Don't attempt to git add ignored files
This uses `git check-ignore` to determine if files are ignored before
attempting to add them in `putFile`.

We also add a condition to the `fetchFromWorkdir` filter to always add
the `flake.lock` file, even if it's not tracked. This is necessary to
resolve inputs.

This fixes #8854 without `git add --force`.
2023-11-30 10:26:13 -08:00
Eelco Dolstra
43d9fb6cf1 Remove InputAccessor::root() 2023-11-30 16:44:54 +01:00
Eelco Dolstra
305939655a Remove superfluous use of hasAccessControl() 2023-11-30 16:28:33 +01:00
Eelco Dolstra
ea95327e72 Move restricted/pure-eval access control out of the evaluator and into the accessor 2023-11-30 16:16:17 +01:00
Eelco Dolstra
3bebaefcd0
Merge pull request #9476 from alois31/restore-progress-bar
nix repl: Only hide the progress bar while waiting for user input
2023-11-30 16:07:28 +01:00
Eelco Dolstra
cb7f25869d
Merge pull request #9430 from hercules-ci/remove-vlas
Fix stack overflow in `filter`
2023-11-30 15:31:44 +01:00
Eelco Dolstra
b6a3fde6b7
Merge pull request #9465 from obsidiansystems/build-dir
Use `buildprefix` in a few more places
2023-11-30 15:28:37 +01:00
Eelco Dolstra
eac0a62052
Merge pull request #9491 from obsidiansystems/fix-freebsd
Improve ACL clearing support (fixing FreeBSD build)
2023-11-30 15:27:24 +01:00
Greg Pfeil
743232bf04
Don’t use execvp when we know the path 2023-11-30 00:17:25 -07:00
John Ericson
a7115a47ef Improve ACL clearing support (fixing FreeBSD build)
The problem was that f880469173 forgot
that the `#include <sys/xattr.h>` was guarded by an `#ifdef __linux__`.

However, the build failure was only on FreeBSD --- turns out other
platforms have this header too!

The fix therefore uses a new configure check so we properly clear ACLs
on more platforms.
2023-11-29 21:09:19 -05:00
Alex Ameen
02bd821f2e
fix: nlohmann::adl_serializer for std::optional (#9147)
This allows templates such as `NLOHMANN_DEFINE_TYPE_*` templates and other generators with things like `std::vector<std::optional<T>>`.

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2023-11-30 01:26:39 +00:00
John Ericson
52e0911302 Use buildprefix in a few more places
`installcheck` doesn't yet work, but the rest of the build can now
happen mostly inside a separate build directory.

Progress on #9342

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-11-29 19:49:07 -05:00
Eelco Dolstra
043413bb59 boost::span -> std::span 2023-11-29 12:38:46 +01:00
Eelco Dolstra
b36857ac8d Add a Git-based content-addressed tarball cache
GitArchiveInputScheme now streams tarballs into a Git repository. This
deduplicates data a lot, e.g. when you're fetching different revisions
of the Nixpkgs repo. It also warns if the tree hash returned by GitHub
doesn't match the tree hash of the imported tarball.
2023-11-29 12:37:03 +01:00
Théophane Hufschmitt
6d1605818c Rename nix doctor to nix config check
Fix #7672
2023-11-28 14:28:19 +01:00
Théophane Hufschmitt
f300e11b05 Rename nix show-config to nix config show
Part of #7672
2023-11-28 14:28:19 +01:00
Alois Wohlschlager
20cd5eb2b3
nix repl: Only hide the progress bar while waiting for user input
In commit 0d2163c6dc, the progress bar was hidden
in nix repl because of a regression that caused it to interfere with user
input. Several users like(d) seeing the progress bar in the repl during builds.
Only hiding it while waiting for user input gives us the best of both worlds,
so do just that.
2023-11-27 19:25:15 +01:00
Robert Hensing
f0180487a0
Merge pull request #9462 from trofi/eval.o-dependency
libexpr: add missing dependency on 'flake/call-flake.nix.gen.hh'
2023-11-27 17:25:21 +01:00
Robert Hensing
68c48756fe libexpr/local.mk: Make eval compile deps regular
Dependency is now entirely through the eval.cc rule.
All gen.hh deps are now there.
2023-11-27 15:52:24 +01:00
Valentin Gagarin
384ffb4443 add deprecation warnings in documentation
this is hacky, but can serve as a stopgap until we can do it
programmatically.
2023-11-27 14:07:32 +01:00
Théophane Hufschmitt
d46230eff5
Merge pull request #7762 from yorickvP/boost-regex
Switch from std::regex to boost::regex
2023-11-27 10:59:22 +01:00
Sergei Trofimovich
75134b7513 libexpr: add missing dependency on 'flake/call-flake.nix.gen.hh'
Without the change build for `eval.o` fails occasionally as:

    $ make src/libexpr/eval.o
      GEN    Makefile.config
      GEN    src/libexpr/primops/derivation.nix.gen.hh
      GEN    src/libexpr/fetchurl.nix.gen.hh
      GEN    src/libexpr/parser-tab.cc
      GEN    src/libexpr/lexer-tab.cc
    src/libexpr/lexer.l:314: warning, -s option given but default rule can be matched
      CXX    src/libexpr/eval.o
    src/libexpr/eval.cc:519:18: fatal error: flake/call-flake.nix.gen.hh: No such file or directory
      519 |         #include "flake/call-flake.nix.gen.hh"
          |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    compilation terminated.
    make: *** [mk/patterns.mk:3: src/libexpr/eval.o] Error 1

Noticed in https://github.com/NixOS/nixpkgs/pull/269439
2023-11-27 08:56:24 +00:00
Moritz Angermann
f56401a114 nix flake update add deprecation warnings.
This builds on #8817, to add additional UX help for people with existing
muscle memory (or shell history) with --update-input and tries to gently
guide them towards the newly evolved CLI UI.

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2023-11-27 09:08:15 +01:00
Yorick van Pelt
d2f5e263e3
Switch from std::regex to boost::regex 2023-11-25 15:14:18 +01:00