Jade Lovelace
473d2d56fc
Remove 100s of CPU time (10%) from build times (1465s -> 1302s)
...
Result's from Mic92's framework 13th Gen Intel Core i7-1360P:
Before: 3595.92s user 183.01s system 1360% cpu 4:37.74 total
After: 3486.07s user 168.93s system 1354% cpu 4:29.79 total
I saw that boost/lexical_cast was costing about 100s in CPU time on our
compiles. We can fix this trivially by doing explicit template
instantiation in exactly one place and eliminating all other includes of
it, which is a code improvement anyway by hiding the boost.
Before:
```
lix/lix2 » ClangBuildAnalyzer --analyze buildtimeold.bin
Analyzing build trace from 'buildtimeold.bin'...
**** Time summary:
Compilation (551 times):
Parsing (frontend): 1465.3 s
Codegen & opts (backend): 1110.9 s
<snip>
**** Expensive headers:
178153 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 3426 ms), included via:
40x: command.hh
5x: command-installable-value.hh
3x: installable-flake.hh
2x: <direct include>
2x: installable-attr-path.hh
176217 ms: ../src/libutil/error.hh (included 246 times, avg 716 ms), included via:
36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
12x: globals.hh config.hh experimental-features.hh
11x: file-system.hh file-descriptor.hh
6x: serialise.hh strings.hh
6x: <direct include>
6x: archive.hh serialise.hh strings.hh
...
173243 ms: ../src/libstore/store-api.hh (included 152 times, avg 1139 ms), included via:
55x: <direct include>
39x: command.hh installable-value.hh installables.hh
7x: libexpr.hh
4x: local-store.hh
4x: command-installable-value.hh installable-value.hh installables.hh
3x: binary-cache-store.hh
...
170482 ms: ../src/libutil/serialise.hh (included 201 times, avg 848 ms), included via:
37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh
14x: store-api.hh nar-info.hh hash.hh
11x: <direct include>
7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh
7x: libexpr.hh value.hh source-path.hh archive.hh
6x: fetchers.hh hash.hh
...
169397 ms: ../src/libcmd/installables.hh (included 53 times, avg 3196 ms), included via:
40x: command.hh installable-value.hh
5x: command-installable-value.hh installable-value.hh
3x: installable-flake.hh installable-value.hh
2x: <direct include>
1x: installable-derived-path.hh
1x: installable-value.hh
...
159740 ms: ../src/libutil/strings.hh (included 221 times, avg 722 ms), included via:
37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh
19x: <direct include>
14x: store-api.hh nar-info.hh hash.hh serialise.hh
11x: serialise.hh
7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh
7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh
...
156796 ms: ../src/libcmd/command.hh (included 51 times, avg 3074 ms), included via:
42x: <direct include>
7x: command-installable-value.hh
2x: installable-attr-path.hh
150392 ms: ../src/libutil/types.hh (included 251 times, avg 599 ms), included via:
36x: command.hh installable-value.hh installables.hh path.hh
11x: file-system.hh
10x: globals.hh
6x: fetchers.hh
6x: serialise.hh strings.hh error.hh
5x: archive.hh
...
133101 ms: /nix/store/644b90j1vms44nr18yw3520pzkrg4dd1-boost-1.81.0-dev/include/boost/lexical_cast.hpp (included 226 times, avg 588 ms), included via
:
37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh strings.hh
19x: file-system.hh
11x: store-api.hh nar-info.hh hash.hh serialise.hh strings.hh
7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
6x: eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
...
132887 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 507 ms), included via:
36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
16x: gtest.h
11x: file-system.hh types.hh ref.hh
10x: globals.hh types.hh ref.hh
10x: json.hpp
6x: serialise.hh
...
done in 0.6s.
```
After:
```
lix/lix2 » maintainers/buildtime_report.sh build
Processing all files and saving to '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
done in 0.6s. Run 'ClangBuildAnalyzer --analyze /home/jade/lix/lix2/maintainers/../buildtime.bin' to analyze it.
Analyzing build trace from '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
**** Time summary:
Compilation (551 times):
Parsing (frontend): 1302.1 s
Codegen & opts (backend): 956.3 s
<snip>
**** Expensive headers:
178145 ms: ../src/libutil/error.hh (included 246 times, avg 724 ms), included via:
36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
12x: globals.hh config.hh experimental-features.hh
11x: file-system.hh file-descriptor.hh
6x: <direct include>
6x: serialise.hh strings.hh
6x: fetchers.hh hash.hh serialise.hh strings.hh
...
154043 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 2962 ms), included via:
40x: command.hh
5x: command-installable-value.hh
3x: installable-flake.hh
2x: <direct include>
2x: installable-attr-path.hh
153593 ms: ../src/libstore/store-api.hh (included 152 times, avg 1010 ms), included via:
55x: <direct include>
39x: command.hh installable-value.hh installables.hh
7x: libexpr.hh
4x: local-store.hh
4x: command-installable-value.hh installable-value.hh installables.hh
3x: binary-cache-store.hh
...
149948 ms: ../src/libutil/types.hh (included 251 times, avg 597 ms), included via:
36x: command.hh installable-value.hh installables.hh path.hh
11x: file-system.hh
10x: globals.hh
6x: fetchers.hh
6x: serialise.hh strings.hh error.hh
5x: archive.hh
...
144560 ms: ../src/libcmd/installables.hh (included 53 times, avg 2727 ms), included via:
40x: command.hh installable-value.hh
5x: command-installable-value.hh installable-value.hh
3x: installable-flake.hh installable-value.hh
2x: <direct include>
1x: installable-value.hh
1x: installable-derived-path.hh
...
136585 ms: ../src/libcmd/command.hh (included 51 times, avg 2678 ms), included via:
42x: <direct include>
7x: command-installable-value.hh
2x: installable-attr-path.hh
133394 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 509 ms), included via:
36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
16x: gtest.h
11x: file-system.hh types.hh ref.hh
10x: globals.hh types.hh ref.hh
10x: json.hpp
6x: serialise.hh
...
89315 ms: ../src/libstore/derived-path.hh (included 178 times, avg 501 ms), included via:
37x: command.hh installable-value.hh installables.hh
25x: store-api.hh realisation.hh
7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh context.hh
6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh
6x: libexpr.hh value.hh context.hh
6x: shared.hh
...
87347 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/ostream (included 273 times, avg 319 ms), included via:
35x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh memory unique_ptr.h
12x: regex sstream istream
10x: file-system.hh types.hh ref.hh memory unique_ptr.h
10x: gtest.h memory unique_ptr.h
10x: globals.hh types.hh ref.hh memory unique_ptr.h
6x: fetchers.hh types.hh ref.hh memory unique_ptr.h
...
85249 ms: ../src/libutil/config.hh (included 213 times, avg 400 ms), included via:
37x: command.hh installable-value.hh installables.hh derived-path.hh
20x: globals.hh
20x: logging.hh
16x: store-api.hh logging.hh
6x: <direct include>
6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh derived-path.hh
...
done in 0.5s.
```
Adapated from 18aa3e1d57
2024-05-31 13:00:09 +02:00
Jörg Thalheim
69c159811e
add regression test for nix edit
2024-05-31 12:58:47 +02:00
Robert Hensing
138aa2b0a7
Merge pull request #10807 from hercules-ci/issue-10504-nix-env-shell
...
Add `nix env shell`
2024-05-31 12:34:03 +02:00
Robert Hensing
962475d97f
Merge pull request #10811 from Mic92/fetcher-fix
...
libfetchers: handle nonexistent refs in GitLab repos more gracefully
2024-05-31 12:24:28 +02:00
Jörg Thalheim
e1a817fb1b
fix nix edit in pure mode
...
FilteringSourceAccessor was not delegating getPhysicalPath to its inner accessor.
2024-05-31 10:39:30 +02:00
Linus Heckemann
a9031978da
libfetchers: handle nonexistent refs in GitLab repos more gracefully
...
Before:
$ nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent
fetching git input 'git+file:///home/linus/projects/lix'
fetching gitlab input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'
error: [json.exception.type_error.302] type must be string, but is null
After:
/tmp/inst/bin/nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent
warning: unknown experimental feature 'repl-flake'
error:
… while updating the lock file of flake 'git+file:///home/joerg/git/nix?ref=refs/heads/master&rev=62693c2c37c8edd92f95114eb1387b461fc671df'
… while updating the flake input 'nixpkgs'
… while fetching the input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'
error: No commits returned by GitLab API -- does the git ref really exist?
Adapted from: 3df013597d
2024-05-31 08:24:53 +02:00
J. Dekker
73f9afd716
upload-release.pl: add riscv64 to nix-fallback-paths.nix
...
This uses the x86_64-linux's cross-compiled output as we don't have a
native riscv64 builder.
Signed-off-by: J. Dekker <jdek@itanimul.li>
2024-05-30 20:11:37 +02:00
Robert Hensing
d93cc11491
Format
2024-05-30 19:41:58 +02:00
Robert Hensing
c692f6af13
nix env shell: Move from nix shell, add shorthand alias
2024-05-30 19:41:58 +02:00
Robert Hensing
98b85b2166
nix/main: Add AliasStatus::{Deprecated,AcceptedShorthand}
2024-05-30 19:41:58 +02:00
J. Dekker
0ed356f3c0
scripts/install.in: add riscv64 support to installer
...
The artifacts are already built and hosted, the install script just needs to be taught about riscv64.
Signed-off-by: J. Dekker <jdek@itanimul.li>
2024-05-30 15:03:20 +02:00
Robert Hensing
ef5c846e25
Merge pull request #10768 from obsidiansystems/legacy-ssh-expose-ssh-master-for-hydra
...
Create `CommonSSHStoreConfig::createSSHMaster`
2024-05-29 22:53:29 +02:00
Robert Hensing
1054ff0873
Merge pull request #10789 from nix-windows/windows-substitution-goal
...
More work on the scheduler for windows
2024-05-29 22:45:55 +02:00
Robert Hensing
154769544d
Merge pull request #10805 from hercules-ci/issue-10774
...
libcmd: Fix #10774
2024-05-29 22:39:00 +02:00
Robert Hensing
5df42223e2
Merge pull request #10797 from obsidiansystems/shellcheck-tests
...
Shellcheck some test scripts
2024-05-29 22:27:00 +02:00
Robert Hensing
1c70eb8eee
libcmd: Fix #10774
2024-05-29 21:42:53 +02:00
Qyriad
18ac6545fc
print type and value in "flake attr is not a derivation" errors
...
This turns errors like:
error: flake output attribute 'hydraJobs' is not a derivation or path
into errors like:
error: expected flake output attribute 'hydraJobs' to be a derivation or
path but found a set: { binaryTarball = «thunk»; build = «thunk»; etc> }
This change affects all InstallableFlake commands.
Source: 20981461d4
2024-05-29 20:51:32 +02:00
Jörg Thalheim
e2182d07d9
fixup extension of changelog entry
2024-05-29 20:51:32 +02:00
Jörg Thalheim
5786e1ae7c
docs: mention importNative/exec in allow-unsafe-native-code-during-evaluation ( #10803 )
...
* docs: mention importNative/exec in allow-unsafe-native-code-during-evaluation
Both of these still needs their own actual documentation, but they are
at least now mentioned that they exist and what they're enabled by.
Co-authored-by: Qyriad <qyriad@qyriad.me>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2024-05-29 07:50:51 +00:00
Philipp
2bd66922ee
add empty line to documentation comments after @brief field ( #10800 )
...
* add empty line to documentation comments after `@brief` field
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2024-05-28 23:05:40 +00:00
John Ericson
10f864c5ae
Ensure all functional scripts are (a) executable (b) have shebang
...
This is good for shebang, and also good for future build system
simplifications
2024-05-28 12:46:24 -04:00
John Ericson
2e12b58126
Shellcheck some test scripts
...
Progress on #10795
2024-05-28 12:32:22 -04:00
John Ericson
bcdee80a0d
More work on the scheduler for windows
...
- Get a rump derivation goal: hook instance will come later, local
derivation goal will come after that.
- Start cleaning up the channel / waiting code with an abstraction.
2024-05-28 11:39:49 -04:00
John Ericson
1e2b26734b
Merge pull request #10799 from hercules-ci/safer-tab-completion
...
Add repl completion test
2024-05-28 11:30:56 -04:00
John Ericson
aa5f013d64
Merge pull request #10794 from obsidiansystems/per-hook-excludes
...
dev shell: excludes are per hook
2024-05-28 09:56:04 -04:00
Eelco Dolstra
7b471547e6
Merge pull request #10793 from NixOS/dependabot/github_actions/zeebe-io/backport-action-3.0.2
...
Bump zeebe-io/backport-action from 2.5.0 to 3.0.2
2024-05-28 15:54:15 +02:00
John Ericson
ebc29017fc
dev shell: excludes are per hook
...
As suggested by Robert in
https://github.com/NixOS/nix/pull/10787#discussion_r1617145374
2024-05-28 09:28:36 -04:00
dependabot[bot]
d7b04d61a9
Bump zeebe-io/backport-action from 2.5.0 to 3.0.2
...
Bumps [zeebe-io/backport-action](https://github.com/zeebe-io/backport-action ) from 2.5.0 to 3.0.2.
- [Release notes](https://github.com/zeebe-io/backport-action/releases )
- [Commits](https://github.com/zeebe-io/backport-action/compare/v2.5.0...v3.0.2 )
---
updated-dependencies:
- dependency-name: zeebe-io/backport-action
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-28 13:10:48 +00:00
John Ericson
cc98fce039
Merge pull request #10787 from NixOS/shellcheck-tests
...
Start getting all shell scripts passing shellcheck
2024-05-28 09:09:37 -04:00
John Ericson
ef96a58ed7
Merge pull request #10791 from obsidiansystems/fix-format
...
Fix format
2024-05-27 22:58:04 -04:00
John Ericson
567265ae67
Start getting all shell scripts passing shellcheck
...
Like with the formatter, we are blacklisting most files by default.
Do a few files to get us started, and get a sense of what this looks
like.
2024-05-27 22:39:56 -04:00
John Ericson
1d5d748fe4
Fix format
...
39b2a399ad1d6c2316a9
2024-05-27 22:32:52 -04:00
John Ericson
d0c7da131f
Merge pull request #10678 from nix-windows/windows-substitution-goal
...
Start building the scheduler for Windows
2024-05-27 17:47:29 -04:00
John Ericson
3e9c3738d3
Create CommonSSHStoreConfig::createSSHMaster
...
By moving `host` to the config, we can do a lot further cleanups and
dedups. This anticipates a world where we always go `StoreReference` ->
`*StoreConfig` -> `Store*` rather than skipping the middle step too.
Progress on #10766
Progress on https://github.com/NixOS/hydra/issues/1164
2024-05-27 16:12:53 -04:00
John Ericson
17964441d9
Merge pull request #10781 from obsidiansystems/build-mode-parse
...
Worker proto use proper serialiser for `BuildMode`
2024-05-27 16:07:54 -04:00
John Ericson
263905da4b
Merge pull request #10785 from mjoerg/fix-typos
...
fix typos
2024-05-27 10:24:53 -04:00
John Ericson
8527f4e7fa
Merge pull request #10754 from qwqawawow/master
...
nix repl: make runNix() isInteractive is true by default
2024-05-27 10:07:15 -04:00
Martin Joerg
e7ea5591a2
fix typos
2024-05-27 15:56:52 +02:00
John Ericson
aa4a2927a7
Merge pull request #10778 from Mic92/safer-tab-completion
...
repl: do not crash when tab-completing import errors
2024-05-27 09:54:29 -04:00
Robert Hensing
ebfada36a1
Add repl completion test
2024-05-27 09:58:49 +02:00
John Ericson
f71b4da0b3
Factor our connection code for worker proto like serve proto
...
This increases test coverage, and gets the worker protocol ready to be
used by Hydra.
Why don't we just try to use the store interface in Hydra? Well, the
problem is that the store interface works on connection pools, with each
opreation getting potentially a different connection, but the way temp
roots work requires that we keep one logical "transaction" (temp root
session) using the same connection.
The longer-term solution probably is making connections themselves
implement the store interface, but that is something that builds on
this, so I feel OK that this is not churn in the wrong direction.
Fixes #9584
2024-05-27 00:43:46 -04:00
John Ericson
8ebd99c74e
Back in enum values for BuildMode serializer
...
We don't want to rely on how C assigns numbers for enums in the wire
format. Sure, this is totally determined by the ABI, but it obscures the
code and makes it harder to safely change the enum definition (should we
need to) without accidentally breaking the wire format.
2024-05-27 00:22:55 -04:00
John Ericson
eeb89c28b0
Worker proto use proper serialiser for BuildMode
...
Do this instead of an unchecked cast
I redid this to use the serialisation framework (including a unit test),
but I am keeping the reference to credit Jade for spotting the issue.
Change-Id: Icf6af7935e8f139bef36b40ad475e973aa48855c
(adapted from commit 2a7a824d83dc5fb33326b8b89625685f283a743b)
Co-Authored-By: Jade Lovelace <lix@jade.fyi>
2024-05-27 00:22:55 -04:00
John Ericson
7de033d63f
Merge pull request #10777 from Mic92/copy-path
...
libstore: remove unused copyPath function
2024-05-26 10:58:38 -04:00
Pierre Bourdon
ffe6ba69d6
repl: do not crash when tab-completing import errors
...
File not found while importing is not currently caught by the tab-completion handler.
Original bug report: https://git.lix.systems/lix-project/lix/issues/340
Fix has been adapted from https://gerrit.lix.systems/c/lix/+/1189
Example crash:
$ cat /tmp/foo.nix
{
someImport = import ./this_file_does_not_exist;
}
$ ./src/nix/nix repl --file /tmp/foo.nix
warning: unknown experimental feature 'repl-flake'
Nix 2.23.0pre20240517_dirty
Type :? for help.
Loading installable ''...
Added 1 variables.
nix-repl> someImport.<TAB>
2024-05-25 23:15:28 +02:00
Jörg Thalheim
f97da4b11c
libutil/source-accessor: custom error if source does not exist
...
This allows better error handling by catching this error in particular.
2024-05-25 23:06:57 +02:00
eldritch horrors
2de4589e46
libstore: remove unused copyPath function
...
Source: 47523944c5
2024-05-25 22:28:41 +02:00
John Ericson
5cfa75ea16
Merge pull request #10737 from poweredbypie/mingw-stackSize
...
Implement `setStackSize` on Windows
2024-05-25 09:56:02 -04:00
John Ericson
e0c94b91ee
Merge pull request #10757 from obsidiansystems/fix-4977
...
Require `drvPath` attribute to end with `.drv`
2024-05-24 12:14:59 -04:00
Eelco Dolstra
eeb4c40867
Typo
2024-05-24 16:35:06 +02:00
