Commit graph

15851 commits

Author SHA1 Message Date
pennae
79d3d412ca optimize derivation string parsing
a bunch of derivation strings contain no escape sequences. we can
optimize for this fact by first scanning for the end of a derivation
string and simply returning the contents unmodified if no escape
sequences were found. to make this even more efficient we can also use
BackedStringViews to avoid copies, avoiding heap allocations for
transient data.

before:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.952 s ±  0.015 s    [User: 5.294 s, System: 1.452 s]
  Range (min … max):    6.926 s …  6.974 s    10 runs

after:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.907 s ±  0.012 s    [User: 5.272 s, System: 1.429 s]
  Range (min … max):    6.893 s …  6.926 s    10 runs
2023-12-30 18:44:10 +01:00
Robert Hensing
bf1b294bd8
Merge pull request #9617 from 9999years/stack-overflow-segfault
Fix segfault on infinite recursion in some cases
2023-12-30 10:40:25 +01:00
Rebecca Turner
7434caca05
Fix segfault on infinite recursion in some cases
This fixes a segfault on infinite function call recursion (rather than
infinite thunk recursion) by tracking the function call depth in
`EvalState`.

Additionally, to avoid printing extremely long stack traces, stack
frames are now deduplicated, with a `(19997 duplicate traces omitted)`
message. This should only really be triggered in infinite recursion
scenarios.

Before:

    $ nix-instantiate --eval --expr '(x: x x) (x: x x)'
    Segmentation fault: 11

After:

    $ nix-instantiate --eval --expr '(x: x x) (x: x x)'
    error: stack overflow

           at «string»:1:14:
                1| (x: x x) (x: x x)
                 |              ^

    $ nix-instantiate --eval --expr '(x: x x) (x: x x)' --show-trace
    error:
           … from call site
             at «string»:1:1:
                1| (x: x x) (x: x x)
                 | ^

           … while calling anonymous lambda
             at «string»:1:2:
                1| (x: x x) (x: x x)
                 |  ^

           … from call site
             at «string»:1:5:
                1| (x: x x) (x: x x)
                 |     ^

           … while calling anonymous lambda
             at «string»:1:11:
                1| (x: x x) (x: x x)
                 |           ^

           … from call site
             at «string»:1:14:
                1| (x: x x) (x: x x)
                 |              ^

           (19997 duplicate traces omitted)

           error: stack overflow
           at «string»:1:14:
                1| (x: x x) (x: x x)
                 |              ^
2023-12-29 22:16:44 -08:00
pennae
2cfc4ace35 malloc/memset even less
more buffers that can be uninitialized and on the stack. small
difference, but still worth doing.

before:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.963 s ±  0.011 s    [User: 5.330 s, System: 1.421 s]
  Range (min … max):    6.943 s …  6.974 s    10 runs

after:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.952 s ±  0.015 s    [User: 5.294 s, System: 1.452 s]
  Range (min … max):    6.926 s …  6.974 s    10 runs
2023-12-30 00:45:11 +01:00
pennae
99a691c8a1 don't use istreams in hot paths
istream sentry objects are very expensive for single-character
operations, and since we don't configure exception masks for the
istreams used here they don't even do anything. all we need is
end-of-string checks and an advancing position in an immutable memory
buffer, both of which can be had for much cheaper than istreams allow.

the effect of this change is most apparent on empty stores.

before:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      7.167 s ±  0.013 s    [User: 5.528 s, System: 1.431 s]
  Range (min … max):    7.147 s …  7.182 s    10 runs

after:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.963 s ±  0.011 s    [User: 5.330 s, System: 1.421 s]
  Range (min … max):    6.943 s …  6.974 s    10 runs
2023-12-30 00:45:10 +01:00
Eelco Dolstra
a21c762dab
Merge pull request #9666 from unblevable/dervation-typo
Fix "dervation" typo
2023-12-27 21:52:37 +01:00
DavHau
b6313f64f7 saner default for log-lines: change to 25
This seems to be a much saner default. 10 lines are just not enough in so many cases.
2023-12-27 19:57:27 +07:00
Brian Le
e2399fc949 Change "dervation" typos to "derivation" 2023-12-26 17:12:28 -05:00
John Ericson
e23983db82
Merge pull request #9661 from shlevy/ifd-buildStore
Build IFD in the build store when using eval-store.
2023-12-24 01:26:14 -05:00
Shea Levy
c3942ef85f
Build IFD in the build store when using eval-store.
Previously, IFDs would be built within the eval store, even though one
is typically using `--eval-store` precisely to *avoid* local builds.

Because the resulting Nix expression must be copied back to the eval
store in order to be imported, this requires the eval store to trust
the build store's signatures.
2023-12-23 21:33:56 -05:00
Shea Levy
9cb287657b
remote-store test: Break out IFD expression into a separate file 2023-12-23 17:15:09 -05:00
Robert Hensing
ee439734e9
Merge pull request #9582 from pennae/misc-opts
a packet of small optimizations
2023-12-22 17:00:59 +01:00
Eelco Dolstra
936a364226 getNameFromURL(): Support uppercase characters in attribute names
In particular, this makes it handle 'legacyPackages' correctly.
2023-12-22 16:35:58 +01:00
John Ericson
d65d56fa77
Merge pull request #9655 from iFreilicht/move-flake-url-name-util
Move url-name utility to libexpr/flake
2023-12-22 10:26:42 -05:00
Robert Hensing
d2016c6b59
Merge pull request #9621 from blaggacao/fix/too-restrictive-branch-regex-master
fix: valid branch name
2023-12-22 16:02:25 +01:00
Felix Uhl
4b41118663 Move flakeref tests to new flake/ subdirectory 2023-12-22 09:38:13 +01:00
Felix Uhl
26d7b0c793 Move url-name utility to libexpr/flake 2023-12-22 09:33:02 +01:00
Robert Hensing
4f47152209
libutil/url-parts.hh: Fix regex
Regex syntax is awful.
2023-12-21 23:11:25 +01:00
Robert Hensing
8c4ea12f11
libutil/url-parts.hh: comment 2023-12-21 21:03:06 +01:00
Cole Helbling
f4454aac9f nix shell: reflect command line order in PATH order
Prior to this change, Nix would prepend every installable to the PATH
list in order to ensure that installables appeared before the current
PATH from the ambient environment.

With this change, all the installables are still prepended to the PATH,
but in the same order as they appear on the command line. This means
that the first of two packages that expose an executable `hello` would
appear in the PATH first, and thus be executed first.

See the test in the prior commit for a more concrete example.
2023-12-21 10:56:19 -08:00
Cole Helbling
1fb43d1eee tests: add a test for command line ordering 2023-12-21 10:23:07 -08:00
Cole Helbling
5ed1884875 libcmd: Installable::toStorePaths -> Installable::toStorePathSet 2023-12-21 10:23:07 -08:00
Eelco Dolstra
17c202b3ea
Merge pull request #8678 from iFreilicht/profile-names-instead-of-index
`nix profile`: Allow referring to elements by human-readable name
2023-12-21 17:14:11 +01:00
Eelco Dolstra
942d635102 Fix release notes 2023-12-21 16:38:25 +01:00
Eelco Dolstra
14508ade28 Typo 2023-12-21 16:25:55 +01:00
Eelco Dolstra
d77a39a314 Fix indent 2023-12-21 16:22:34 +01:00
Eelco Dolstra
9d9d9ff0de Merge remote-tracking branch 'origin/master' into profile-names-instead-of-index 2023-12-21 16:21:26 +01:00
Robert Hensing
8e64cd59b0
Merge pull request #9645 from tweag/undeprecate-isNull
Undeprecate isNull
2023-12-20 18:15:49 +01:00
Silvan Mosberger
ea454d8687 Undeprecate isNull
There's no good reason to deprecate it:
- For consistency reasons it should continue to exist, such that all
primitive types have a corresponding `builtins.is*` primop.
- There's no implementation cost to continuing to have this function
- It costs users time to try to migrate away from it, e.g.
  https://github.com/NixOS/nixpkgs/pull/219747 and https://github.com/NixOS/nixpkgs/pull/275548
- Using it can give easier-to-read code like `all isNull list`

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-12-20 17:55:19 +01:00
Eelco Dolstra
75e10e42f3
Merge pull request #9631 from cole-h/fixup-check-warnings
Fix warnings when running checks
2023-12-20 17:47:30 +01:00
Eelco Dolstra
16a77767cd
Merge pull request #9635 from 9999years/document-makefile-variables
Document more `Makefile` variables
2023-12-20 17:47:07 +01:00
Théophane Hufschmitt
e3cdad4cf8
Merge pull request #9646 from obsidiansystems/fix-cross-build
Move down fallback targets in `Makefile`
2023-12-20 13:44:40 +01:00
John Ericson
26d60b837c Move down fallback targets in Makefile
This ensures `lib.mk` still defines `default` as the first target. This
fixes some builds.
2023-12-20 03:23:49 -05:00
pennae
2b0e95e7aa use singleton expr to generate black hole errors
this also reduces forceValue code size and removes the need for
hideInDiagnostics. coopting thunk forcing like this has the additional
benefit of clarifying how these errors can happen in the first place.
2023-12-19 19:32:16 +01:00
pennae
f9db4de0f3 force-inline forceValue
forceValue is extremely hot. interestingly adding likeliness annotations
to the branches does not seem to make a difference.

before:

  Time (mean ± σ):      4.224 s ±  0.005 s    [User: 3.711 s, System: 0.512 s]
  Range (min … max):    4.218 s …  4.234 s    10 runs

after:

  Time (mean ± σ):      4.140 s ±  0.009 s    [User: 3.647 s, System: 0.492 s]
  Range (min … max):    4.130 s …  4.152 s    10 runs
2023-12-19 19:32:16 +01:00
pennae
69ed4aee61 remove lazy-pos forceValue
almost all uses of this are interactive, except for deepSeq. deepSeq is
going to be expensive and rare enough to not care much about, and
Value::determinePos should usually be cheap enough to not be too much of
a burden in any case.
2023-12-19 19:32:16 +01:00
pennae
f9aee2f2c4 don't malloc/memset posix accessor buffer
it's relatively small and fits on the stack nicely, and we don't need it
initialized either.
2023-12-19 19:32:16 +01:00
pennae
b78e77b34c use custom location type in the parser
~1% parser speedup from not using TLS indirections, less on system eval.
this could have also gone in flex yyextra data, but that's significantly
slower for some reason (albeit still faster than thread locals).

before:

  Time (mean ± σ):      4.231 s ±  0.004 s    [User: 3.725 s, System: 0.504 s]
  Range (min … max):    4.226 s …  4.240 s    10 runs

after:

  Time (mean ± σ):      4.224 s ±  0.005 s    [User: 3.711 s, System: 0.512 s]
  Range (min … max):    4.218 s …  4.234 s    10 runs
2023-12-19 19:32:16 +01:00
pennae
2e0321912a use aligned flex tables
~2% speedup on parsing without eval, less (but still significant) on
system eval. having flex generate faster parsers leads to very strange
misparses. maybe re2c is worth investigating.

before:

  Time (mean ± σ):      4.260 s ±  0.003 s    [User: 3.754 s, System: 0.505 s]
  Range (min … max):    4.257 s …  4.266 s    10 runs

after:

  Time (mean ± σ):      4.231 s ±  0.004 s    [User: 3.725 s, System: 0.504 s]
  Range (min … max):    4.226 s …  4.240 s    10 runs
2023-12-19 19:32:16 +01:00
pennae
cc4038d541 use std::tie() for macro-generated operators
as written the comparisons generate copies, even though it looks as
though they shouldn't.

before:

  Time (mean ± σ):      4.396 s ±  0.002 s    [User: 3.894 s, System: 0.501 s]
  Range (min … max):    4.393 s …  4.399 s    10 runs

after:

  Time (mean ± σ):      4.260 s ±  0.003 s    [User: 3.754 s, System: 0.505 s]
  Range (min … max):    4.257 s …  4.266 s    10 runs
2023-12-19 19:32:16 +01:00
pennae
74c134914c compare string values with strcmp
string_view()ification calls strlen() first, which we don't need here.
2023-12-19 19:32:16 +01:00
pennae
78353deb02 encode black holes as tApp values
checking for isBlackhole in the forceValue hot path is rather more
expensive than necessary, and with a little bit of trickery we can move
such handling into the isApp case. small performance benefit, but under
some circumstances we've seen 2% improvement as well.

〉 nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'

before:

  Time (mean ± σ):      4.429 s ±  0.002 s    [User: 3.929 s, System: 0.500 s]
  Range (min … max):    4.427 s …  4.433 s    10 runs

after:

  Time (mean ± σ):      4.396 s ±  0.002 s    [User: 3.894 s, System: 0.501 s]
  Range (min … max):    4.393 s …  4.399 s    10 runs
2023-12-19 19:32:16 +01:00
pennae
0218e4e6c3 memset less in addToStoreFromDump
resizing a std::string clears the newly added bytes, which is not
necessary here and comes with a ~1.4% slowdown on our test nixos config.

〉 nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'

before:

  Time (mean ± σ):      4.486 s ±  0.003 s    [User: 3.978 s, System: 0.507 s]
  Range (min … max):    4.482 s …  4.492 s    10 runs

after:

  Time (mean ± σ):      4.429 s ±  0.002 s    [User: 3.929 s, System: 0.500 s]
  Range (min … max):    4.427 s …  4.433 s    10 runs
2023-12-19 19:32:16 +01:00
Eelco Dolstra
7cfd6c0efe
Merge pull request #9325 from NixOS/accessor-add-to-store
Content addressing and adding to store cleanup
2023-12-19 15:10:31 +01:00
Théophane Hufschmitt
83a32969ad
Merge pull request #9633 from cole-h/fix-sysconfdir
package: don't set sysconfdir in devShells
2023-12-19 10:14:48 +01:00
Théophane Hufschmitt
94a03a810e
Merge pull request #9637 from 9999years/fix-logging-test-on-macos
Fix `logging.sh` test on macOS
2023-12-19 10:13:08 +01:00
Théophane Hufschmitt
b077d545ad
Merge pull request #9638 from NixOS/dependabot/github_actions/zeebe-io/backport-action-2.3.0
build(deps): bump zeebe-io/backport-action from 2.2.0 to 2.3.0
2023-12-19 10:11:37 +01:00
Théophane Hufschmitt
e85fd92816
Merge pull request #9639 from awakesecurity/mz/darwin-UID-override
installer: allow overriding of NIX_FIRST_BUILD_ID on darwin
2023-12-19 10:11:24 +01:00
Théophane Hufschmitt
c6d7013583
Merge pull request #9641 from amarshall/darwin-sandbox-stderr
Allow access to /dev/stderr in Darwin sandbox
2023-12-19 10:09:38 +01:00
Andrew Marshall
7526b7ded6 Allow access to /dev/stderr in Darwin sandbox
We allow /dev/stdout, so why not this? Since it is process-local,
anyway, should not be possible to escape sandbox using it.
2023-12-18 19:33:20 -05:00