max/nix-super
1
0
Fork 0
mirror of https://github.com/privatevoid-net/nix-super.git synced 2024-11-10 08:16:15 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
17476 commits 7 branches 0 tags 62 MiB
Commit graph

17476 commits

This branch
This branch
All branches
Author SHA1 Message Date
Robert Hensing
84e116379c
Merge pull request #10812 from Mic92/build-perf 
Remove 100s of CPU time (10%) from build times (1465s -> 1302s)
 2024-05-31 13:28:24 +02:00
Jade Lovelace
473d2d56fc Remove 100s of CPU time (10%) from build times (1465s -> 1302s) 
Result's from Mic92's framework 13th Gen Intel Core i7-1360P:

Before: 3595.92s user 183.01s system 1360% cpu 4:37.74 total
After: 3486.07s user 168.93s system 1354% cpu 4:29.79 total

I saw that boost/lexical_cast was costing about 100s in CPU time on our
compiles. We can fix this trivially by doing explicit template
instantiation in exactly one place and eliminating all other includes of
it, which is a code improvement anyway by hiding the boost.

Before:
```
lix/lix2 » ClangBuildAnalyzer --analyze buildtimeold.bin
Analyzing build trace from 'buildtimeold.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1465.3 s
  Codegen & opts (backend):   1110.9 s

<snip>

**** Expensive headers:
178153 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 3426 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

176217 ms: ../src/libutil/error.hh (included 246 times, avg 716 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: serialise.hh strings.hh
  6x: <direct include>
  6x: archive.hh serialise.hh strings.hh
  ...

173243 ms: ../src/libstore/store-api.hh (included 152 times, avg 1139 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

170482 ms: ../src/libutil/serialise.hh (included 201 times, avg 848 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh
  14x: store-api.hh nar-info.hh hash.hh
  11x: <direct include>
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh
  6x: fetchers.hh hash.hh
  ...

169397 ms: ../src/libcmd/installables.hh (included 53 times, avg 3196 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-derived-path.hh
  1x: installable-value.hh
  ...

159740 ms: ../src/libutil/strings.hh (included 221 times, avg 722 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh
  19x: <direct include>
  14x: store-api.hh nar-info.hh hash.hh serialise.hh
  11x: serialise.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh
  ...

156796 ms: ../src/libcmd/command.hh (included 51 times, avg 3074 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

150392 ms: ../src/libutil/types.hh (included 251 times, avg 599 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

133101 ms: /nix/store/644b90j1vms44nr18yw3520pzkrg4dd1-boost-1.81.0-dev/include/boost/lexical_cast.hpp (included 226 times, avg 588 ms), included via
:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh strings.hh
  19x: file-system.hh
  11x: store-api.hh nar-info.hh hash.hh serialise.hh strings.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  ...

132887 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 507 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

  done in 0.6s.
```

After:
```
lix/lix2 » maintainers/buildtime_report.sh build
Processing all files and saving to '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
  done in 0.6s. Run 'ClangBuildAnalyzer --analyze /home/jade/lix/lix2/maintainers/../buildtime.bin' to analyze it.
Analyzing build trace from '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1302.1 s
  Codegen & opts (backend):    956.3 s

<snip>

**** Expensive headers:
178145 ms: ../src/libutil/error.hh (included 246 times, avg 724 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: <direct include>
  6x: serialise.hh strings.hh
  6x: fetchers.hh hash.hh serialise.hh strings.hh
  ...

154043 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 2962 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

153593 ms: ../src/libstore/store-api.hh (included 152 times, avg 1010 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

149948 ms: ../src/libutil/types.hh (included 251 times, avg 597 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

144560 ms: ../src/libcmd/installables.hh (included 53 times, avg 2727 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-value.hh
  1x: installable-derived-path.hh
  ...

136585 ms: ../src/libcmd/command.hh (included 51 times, avg 2678 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

133394 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 509 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

89315 ms: ../src/libstore/derived-path.hh (included 178 times, avg 501 ms), included via:
  37x: command.hh installable-value.hh installables.hh
  25x: store-api.hh realisation.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: libexpr.hh value.hh context.hh
  6x: shared.hh
  ...

87347 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/ostream (included 273 times, avg 319 ms), included via:
  35x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh memory unique_ptr.h
  12x: regex sstream istream
  10x: file-system.hh types.hh ref.hh memory unique_ptr.h
  10x: gtest.h memory unique_ptr.h
  10x: globals.hh types.hh ref.hh memory unique_ptr.h
  6x: fetchers.hh types.hh ref.hh memory unique_ptr.h
  ...

85249 ms: ../src/libutil/config.hh (included 213 times, avg 400 ms), included via:
  37x: command.hh installable-value.hh installables.hh derived-path.hh
  20x: globals.hh
  20x: logging.hh
  16x: store-api.hh logging.hh
  6x: <direct include>
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh derived-path.hh
  ...

  done in 0.5s.
```

Adapated from 18aa3e1d57
 2024-05-31 13:00:09 +02:00
Jörg Thalheim
69c159811e add regression test for nix edit 2024-05-31 12:58:47 +02:00
Robert Hensing
138aa2b0a7
Merge pull request #10807 from hercules-ci/issue-10504-nix-env-shell 
Add `nix env shell`
 2024-05-31 12:34:03 +02:00
Robert Hensing
962475d97f
Merge pull request #10811 from Mic92/fetcher-fix 
libfetchers: handle nonexistent refs in GitLab repos more gracefully
 2024-05-31 12:24:28 +02:00
Jörg Thalheim
e1a817fb1b fix nix edit in pure mode 
FilteringSourceAccessor was not delegating getPhysicalPath to its inner accessor.
 2024-05-31 10:39:30 +02:00
Linus Heckemann
a9031978da libfetchers: handle nonexistent refs in GitLab repos more gracefully 
Before:

$ nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent
fetching git input 'git+file:///home/linus/projects/lix'
fetching gitlab input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'
error: [json.exception.type_error.302] type must be string, but is null

After:

/tmp/inst/bin/nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent

warning: unknown experimental feature 'repl-flake'
error:
       … while updating the lock file of flake 'git+file:///home/joerg/git/nix?ref=refs/heads/master&rev=62693c2c37c8edd92f95114eb1387b461fc671df'

       … while updating the flake input 'nixpkgs'

       … while fetching the input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'

       error: No commits returned by GitLab API -- does the git ref really exist?

Adapted from: 3df013597d
 2024-05-31 08:24:53 +02:00
J. Dekker
73f9afd716 upload-release.pl: add riscv64 to nix-fallback-paths.nix 
This uses the x86_64-linux's cross-compiled output as we don't have a
native riscv64 builder.

Signed-off-by: J. Dekker <jdek@itanimul.li>
 2024-05-30 20:11:37 +02:00
Robert Hensing
d93cc11491 Format 2024-05-30 19:41:58 +02:00
Robert Hensing
c692f6af13 nix env shell: Move from nix shell, add shorthand alias 2024-05-30 19:41:58 +02:00
Robert Hensing
98b85b2166 nix/main: Add AliasStatus::{Deprecated,AcceptedShorthand} 2024-05-30 19:41:58 +02:00
J. Dekker
0ed356f3c0 scripts/install.in: add riscv64 support to installer 
The artifacts are already built and hosted, the install script just needs to be taught about riscv64.

Signed-off-by: J. Dekker <jdek@itanimul.li>
 2024-05-30 15:03:20 +02:00
Robert Hensing
ef5c846e25
Merge pull request #10768 from obsidiansystems/legacy-ssh-expose-ssh-master-for-hydra 
Create `CommonSSHStoreConfig::createSSHMaster`
 2024-05-29 22:53:29 +02:00
Robert Hensing
1054ff0873
Merge pull request #10789 from nix-windows/windows-substitution-goal 
More work on the scheduler for windows
 2024-05-29 22:45:55 +02:00
Robert Hensing
154769544d
Merge pull request #10805 from hercules-ci/issue-10774 
libcmd: Fix #10774
 2024-05-29 22:39:00 +02:00
Robert Hensing
5df42223e2
Merge pull request #10797 from obsidiansystems/shellcheck-tests 
Shellcheck some test scripts
 2024-05-29 22:27:00 +02:00
Robert Hensing
1c70eb8eee libcmd: Fix #10774 2024-05-29 21:42:53 +02:00
Qyriad
18ac6545fc print type and value in "flake attr is not a derivation" errors 
This turns errors like:

error: flake output attribute 'hydraJobs' is not a derivation or path

into errors like:

error: expected flake output attribute 'hydraJobs' to be a derivation or
path but found a set: { binaryTarball = «thunk»; build = «thunk»; etc> }

This change affects all InstallableFlake commands.

Source: 20981461d4
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
 2024-05-29 20:51:32 +02:00
Jörg Thalheim
e2182d07d9 fixup extension of changelog entry 2024-05-29 20:51:32 +02:00
Jörg Thalheim
5786e1ae7c
docs: mention importNative/exec in allow-unsafe-native-code-during-evaluation (#10803) 
* docs: mention importNative/exec in allow-unsafe-native-code-during-evaluation

Both of these still needs their own actual documentation, but they are
at least now mentioned that they exist and what they're enabled by.

Co-authored-by: Qyriad <qyriad@qyriad.me>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
 2024-05-29 07:50:51 +00:00
Philipp
2bd66922ee
add empty line to documentation comments after @brief field (#10800) 
* add empty line to documentation comments after `@brief` field

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
 2024-05-28 23:05:40 +00:00
John Ericson
10f864c5ae Ensure all functional scripts are (a) executable (b) have shebang 
This is good for shebang, and also good for future build system
simplifications
 2024-05-28 12:46:24 -04:00
John Ericson
2e12b58126 Shellcheck some test scripts 
Progress on #10795
 2024-05-28 12:32:22 -04:00
John Ericson
bcdee80a0d More work on the scheduler for windows 
- Get a rump derivation goal: hook instance will come later, local
  derivation goal will come after that.

- Start cleaning up the channel / waiting code with an abstraction.
 2024-05-28 11:39:49 -04:00
John Ericson
1e2b26734b
Merge pull request #10799 from hercules-ci/safer-tab-completion 
Add repl completion test
 2024-05-28 11:30:56 -04:00
John Ericson
aa5f013d64
Merge pull request #10794 from obsidiansystems/per-hook-excludes 
dev shell: excludes are per hook
 2024-05-28 09:56:04 -04:00
Eelco Dolstra
7b471547e6
Merge pull request #10793 from NixOS/dependabot/github_actions/zeebe-io/backport-action-3.0.2 
Bump zeebe-io/backport-action from 2.5.0 to 3.0.2
 2024-05-28 15:54:15 +02:00
John Ericson
ebc29017fc dev shell: excludes are per hook 
As suggested by Robert in
https://github.com/NixOS/nix/pull/10787#discussion_r1617145374
 2024-05-28 09:28:36 -04:00
dependabot[bot]
d7b04d61a9
Bump zeebe-io/backport-action from 2.5.0 to 3.0.2 
Bumps [zeebe-io/backport-action](https://github.com/zeebe-io/backport-action) from 2.5.0 to 3.0.2.
- [Release notes](https://github.com/zeebe-io/backport-action/releases)
- [Commits](https://github.com/zeebe-io/backport-action/compare/v2.5.0...v3.0.2)

---
updated-dependencies:
- dependency-name: zeebe-io/backport-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-05-28 13:10:48 +00:00
John Ericson
cc98fce039
Merge pull request #10787 from NixOS/shellcheck-tests 
Start getting all shell scripts passing shellcheck
 2024-05-28 09:09:37 -04:00
John Ericson
ef96a58ed7
Merge pull request #10791 from obsidiansystems/fix-format 
Fix format
 2024-05-27 22:58:04 -04:00
John Ericson
567265ae67 Start getting all shell scripts passing shellcheck 
Like with the formatter, we are blacklisting most files by default.

Do a few files to get us started, and get a sense of what this looks
like.
 2024-05-27 22:39:56 -04:00
John Ericson
1d5d748fe4 Fix format 
39b2a399ad passed CI but was landed after
the formatting change in 1d6c2316a9.
 2024-05-27 22:32:52 -04:00
John Ericson
d0c7da131f
Merge pull request #10678 from nix-windows/windows-substitution-goal 
Start building the scheduler for Windows
 2024-05-27 17:47:29 -04:00
John Ericson
3e9c3738d3 Create CommonSSHStoreConfig::createSSHMaster 
By moving `host` to the config, we can do a lot further cleanups and
dedups. This anticipates a world where we always go `StoreReference` ->
`*StoreConfig` -> `Store*` rather than skipping the middle step too.

Progress on #10766

Progress on https://github.com/NixOS/hydra/issues/1164
 2024-05-27 16:12:53 -04:00
John Ericson
17964441d9
Merge pull request #10781 from obsidiansystems/build-mode-parse 
Worker proto use proper serialiser for `BuildMode`
 2024-05-27 16:07:54 -04:00
John Ericson
263905da4b
Merge pull request #10785 from mjoerg/fix-typos 
fix typos
 2024-05-27 10:24:53 -04:00
John Ericson
8527f4e7fa
Merge pull request #10754 from qwqawawow/master 
nix repl: make runNix() isInteractive is true by default
 2024-05-27 10:07:15 -04:00
Martin Joerg
e7ea5591a2 fix typos 2024-05-27 15:56:52 +02:00
John Ericson
aa4a2927a7
Merge pull request #10778 from Mic92/safer-tab-completion 
repl: do not crash when tab-completing import errors
 2024-05-27 09:54:29 -04:00
Robert Hensing
ebfada36a1 Add repl completion test 2024-05-27 09:58:49 +02:00
John Ericson
f71b4da0b3 Factor our connection code for worker proto like serve proto 
This increases test coverage, and gets the worker protocol ready to be
used by Hydra.

Why don't we just try to use the store interface in Hydra? Well, the
problem is that the store interface works on connection pools, with each
opreation getting potentially a different connection, but the way temp
roots work requires that we keep one logical "transaction" (temp root
session) using the same connection.

The longer-term solution probably is making connections themselves
implement the store interface, but that is something that builds on
this, so I feel OK that this is not churn in the wrong direction.

Fixes #9584
 2024-05-27 00:43:46 -04:00
John Ericson
8ebd99c74e Back in enum values for BuildMode serializer 
We don't want to rely on how C assigns numbers for enums in the wire
format. Sure, this is totally determined by the ABI, but it obscures the
code and makes it harder to safely change the enum definition (should we
need to) without accidentally breaking the wire format.
 2024-05-27 00:22:55 -04:00
John Ericson
eeb89c28b0 Worker proto use proper serialiser for BuildMode 
Do this instead of an unchecked cast

I redid this to use the serialisation framework (including a unit test),
but I am keeping the reference to credit Jade for spotting the issue.

Change-Id: Icf6af7935e8f139bef36b40ad475e973aa48855c
(adapted from commit 2a7a824d83dc5fb33326b8b89625685f283a743b)

Co-Authored-By: Jade Lovelace <lix@jade.fyi>
 2024-05-27 00:22:55 -04:00
John Ericson
7de033d63f
Merge pull request #10777 from Mic92/copy-path 
libstore: remove unused copyPath function
 2024-05-26 10:58:38 -04:00
Pierre Bourdon
ffe6ba69d6 repl: do not crash when tab-completing import errors 
File not found while importing is not currently caught by the tab-completion handler.

Original bug report: https://git.lix.systems/lix-project/lix/issues/340
Fix has been adapted from https://gerrit.lix.systems/c/lix/+/1189

Example crash:

$ cat /tmp/foo.nix
{
  someImport = import ./this_file_does_not_exist;
}
$ ./src/nix/nix repl --file /tmp/foo.nix
warning: unknown experimental feature 'repl-flake'
Nix 2.23.0pre20240517_dirty
Type :? for help.
Loading installable ''...
Added 1 variables.
nix-repl> someImport.<TAB>
 2024-05-25 23:15:28 +02:00
Jörg Thalheim
f97da4b11c libutil/source-accessor: custom error if source does not exist 
This allows better error handling by catching this error in particular.
 2024-05-25 23:06:57 +02:00
eldritch horrors
2de4589e46 libstore: remove unused copyPath function 
Source: 47523944c5

Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
 2024-05-25 22:28:41 +02:00
John Ericson
5cfa75ea16
Merge pull request #10737 from poweredbypie/mingw-stackSize 
Implement `setStackSize` on Windows
 2024-05-25 09:56:02 -04:00
John Ericson
e0c94b91ee
Merge pull request #10757 from obsidiansystems/fix-4977 
Require `drvPath` attribute to end with `.drv`
 2024-05-24 12:14:59 -04:00