max/nix-super
1
0
Fork 0
mirror of https://github.com/privatevoid-net/nix-super.git synced 2025-02-14 22:27:17 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
18514 commits 7 branches 0 tags 64 MiB
Commit graph

2141 commits

Author SHA1 Message Date
Jade Lovelace
473d2d56fc Remove 100s of CPU time (10%) from build times (1465s -> 1302s) 
Result's from Mic92's framework 13th Gen Intel Core i7-1360P:

Before: 3595.92s user 183.01s system 1360% cpu 4:37.74 total
After: 3486.07s user 168.93s system 1354% cpu 4:29.79 total

I saw that boost/lexical_cast was costing about 100s in CPU time on our
compiles. We can fix this trivially by doing explicit template
instantiation in exactly one place and eliminating all other includes of
it, which is a code improvement anyway by hiding the boost.

Before:
```
lix/lix2 » ClangBuildAnalyzer --analyze buildtimeold.bin
Analyzing build trace from 'buildtimeold.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1465.3 s
  Codegen & opts (backend):   1110.9 s

<snip>

**** Expensive headers:
178153 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 3426 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

176217 ms: ../src/libutil/error.hh (included 246 times, avg 716 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: serialise.hh strings.hh
  6x: <direct include>
  6x: archive.hh serialise.hh strings.hh
  ...

173243 ms: ../src/libstore/store-api.hh (included 152 times, avg 1139 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

170482 ms: ../src/libutil/serialise.hh (included 201 times, avg 848 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh
  14x: store-api.hh nar-info.hh hash.hh
  11x: <direct include>
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh
  6x: fetchers.hh hash.hh
  ...

169397 ms: ../src/libcmd/installables.hh (included 53 times, avg 3196 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-derived-path.hh
  1x: installable-value.hh
  ...

159740 ms: ../src/libutil/strings.hh (included 221 times, avg 722 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh
  19x: <direct include>
  14x: store-api.hh nar-info.hh hash.hh serialise.hh
  11x: serialise.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh
  ...

156796 ms: ../src/libcmd/command.hh (included 51 times, avg 3074 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

150392 ms: ../src/libutil/types.hh (included 251 times, avg 599 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

133101 ms: /nix/store/644b90j1vms44nr18yw3520pzkrg4dd1-boost-1.81.0-dev/include/boost/lexical_cast.hpp (included 226 times, avg 588 ms), included via
:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh strings.hh
  19x: file-system.hh
  11x: store-api.hh nar-info.hh hash.hh serialise.hh strings.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  ...

132887 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 507 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

  done in 0.6s.
```

After:
```
lix/lix2 » maintainers/buildtime_report.sh build
Processing all files and saving to '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
  done in 0.6s. Run 'ClangBuildAnalyzer --analyze /home/jade/lix/lix2/maintainers/../buildtime.bin' to analyze it.
Analyzing build trace from '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1302.1 s
  Codegen & opts (backend):    956.3 s

<snip>

**** Expensive headers:
178145 ms: ../src/libutil/error.hh (included 246 times, avg 724 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: <direct include>
  6x: serialise.hh strings.hh
  6x: fetchers.hh hash.hh serialise.hh strings.hh
  ...

154043 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 2962 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

153593 ms: ../src/libstore/store-api.hh (included 152 times, avg 1010 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

149948 ms: ../src/libutil/types.hh (included 251 times, avg 597 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

144560 ms: ../src/libcmd/installables.hh (included 53 times, avg 2727 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-value.hh
  1x: installable-derived-path.hh
  ...

136585 ms: ../src/libcmd/command.hh (included 51 times, avg 2678 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

133394 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 509 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

89315 ms: ../src/libstore/derived-path.hh (included 178 times, avg 501 ms), included via:
  37x: command.hh installable-value.hh installables.hh
  25x: store-api.hh realisation.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: libexpr.hh value.hh context.hh
  6x: shared.hh
  ...

87347 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/ostream (included 273 times, avg 319 ms), included via:
  35x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh memory unique_ptr.h
  12x: regex sstream istream
  10x: file-system.hh types.hh ref.hh memory unique_ptr.h
  10x: gtest.h memory unique_ptr.h
  10x: globals.hh types.hh ref.hh memory unique_ptr.h
  6x: fetchers.hh types.hh ref.hh memory unique_ptr.h
  ...

85249 ms: ../src/libutil/config.hh (included 213 times, avg 400 ms), included via:
  37x: command.hh installable-value.hh installables.hh derived-path.hh
  20x: globals.hh
  20x: logging.hh
  16x: store-api.hh logging.hh
  6x: <direct include>
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh derived-path.hh
  ...

  done in 0.5s.
```

Adapated from 18aa3e1d57
 2024-05-31 13:00:09 +02:00
John Ericson
bcdee80a0d More work on the scheduler for windows 
- Get a rump derivation goal: hook instance will come later, local
  derivation goal will come after that.

- Start cleaning up the channel / waiting code with an abstraction.
 2024-05-28 11:39:49 -04:00
John Ericson
1d5d748fe4 Fix format 
39b2a399ad passed CI but was landed after
the formatting change in 1d6c2316a9.
 2024-05-27 22:32:52 -04:00
John Ericson
d0c7da131f
Merge pull request #10678 from nix-windows/windows-substitution-goal 
Start building the scheduler for Windows
 2024-05-27 17:47:29 -04:00
John Ericson
aa4a2927a7
Merge pull request #10778 from Mic92/safer-tab-completion 
repl: do not crash when tab-completing import errors
 2024-05-27 09:54:29 -04:00
Jörg Thalheim
f97da4b11c libutil/source-accessor: custom error if source does not exist 
This allows better error handling by catching this error in particular.
 2024-05-25 23:06:57 +02:00
eldritch horrors
2de4589e46 libstore: remove unused copyPath function 
Source: 47523944c5

Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
 2024-05-25 22:28:41 +02:00
John Ericson
5cfa75ea16
Merge pull request #10737 from poweredbypie/mingw-stackSize 
Implement `setStackSize` on Windows
 2024-05-25 09:56:02 -04:00
PoweredByPie
5f68e6d69f Get max stack size in setStackSize to match Linux 2024-05-23 03:54:35 -07:00
John Ericson
4a19f4a866
Merge pull request #9839 from obsidiansystems/more-machine-cleanup 
Create `StoreReference` and use it in `Machine`
 2024-05-22 17:01:57 -04:00
John Ericson
dc7615dbbb tryUnshareFilesystem: Ignore ENOSYS too 
Fixes #10747
 2024-05-22 16:07:38 -04:00
John Ericson
d5fdfdc592 unshareFilesystem: Do not assume caller 2024-05-22 16:07:34 -04:00
John Ericson
b59a7a14c4 Add StoreReference::render 
This will be needed for the next step.

Also allows us to write round trip tests.
 2024-05-22 09:20:15 -04:00
John Ericson
1d6c2316a9 Slightly change formatting style 
For long expressions, one argument or parameter per line is just easier.
 2024-05-22 09:20:15 -04:00
Robert Hensing
e4be8abe42
Merge pull request #10701 from cole-h/nix-command-warn-unknown-settings 
Warn on unknown settings when the first positional is an argument
 2024-05-20 15:15:13 +02:00
John Ericson
d48bbda2e7
Update the updateWindowSize documentation 2024-05-20 08:34:49 -04:00
PoweredByPie
6a3f906382 Implement setStackSize for Windows 2024-05-18 16:19:01 -07:00
PoweredByPie
e42d00c961 Change rlim_t to size_t in setStackSize in preparation of Windows impl 2024-05-18 16:18:50 -07:00
PoweredByPie
53f0c44d6c Implement updateWindowSize for Windows 2024-05-18 16:14:20 -07:00
Qyriad
52a16b7e59 add and fix -Wdeprecated-copy 
*so* many warnings, from only two definitions

Change-Id: If2561cd500c05a1e33cce984faf9f3e42a8a95ac
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
 2024-05-17 19:25:35 +02:00
Robert Hensing
303268bb71
Merge pull request #10479 from obsidiansystems/ca-fso-docs 
Document file system object content addressing
 2024-05-15 22:52:53 +02:00
John Ericson
043135a848 Document file system object content addressing 
In addition:

- Take the opportunity to add a bunch more missing hyperlinks, too.

- Remove some glossary entries that are now subsumed by dedicated pages.
  We used to not be able to do this without breaking link fragments, but
  now we can, so pick up where we left off.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
 2024-05-15 16:28:48 -04:00
Cole Helbling
39a269657e libutil/args: warn on unknown settings after parsing all flags 2024-05-15 12:25:03 -07:00
siddhantCodes
39e8aad446 Merge branch 'master' of github.com:NixOS/nix 2024-05-13 18:43:12 +05:30
siddhantCodes
62e1ea2f4b use path for from arg in nix::copyFile 2024-05-13 16:10:21 +05:30
Eelco Dolstra
5534682166 Update src/libutil/util.hh 
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
 2024-05-13 12:14:43 +02:00
Eelco Dolstra
cf3b044b7e Make large path warnings human-readable 2024-05-13 11:52:38 +02:00
Eelco Dolstra
5314430437 Move printSize() into libutil 
Also always include the unit (i.e. "MiB" instead of "M").
 2024-05-13 11:52:38 +02:00
Eelco Dolstra
dbe1b51580 Add setting to warn about copying/hashing large paths 
This is useful for diagnosing whether an evaluation is copying large
paths to the store. Example:

   $ nix build .#packages.x86_64-linux.default --large-path-warning-threshold 1000000
   warning: copied large path '/home/eelco/Dev/nix-master/' to the store (6271792 bytes)
   warning: copied large path '«github:NixOS/nixpkgs/b550fe4b4776908ac2a861124307045f8e717c8e?narHash=sha256-7kkJQd4rZ%2BvFrzWu8sTRtta5D1kBG0LSRYAfhtmMlSo%3D»/' to the store (155263768 bytes)
   warning: copied large path '«github:libgit2/libgit2/45fd9ed7ae1a9b74b957ef4f337bc3c8b3df01b5?narHash=sha256-oX4Z3S9WtJlwvj0uH9HlYcWv%2Bx1hqp8mhXl7HsLu2f0%3D»/' to the store (22175416 bytes)
   warning: copied large path '/nix/store/z985088mcd6w23qwdlirsinnyzayagki-source' to the store (5885872 bytes)
 2024-05-13 11:52:27 +02:00
siddhantCodes
ccf94545db rename copy -> copyFile and remove old copyFile 
the old `copyFile` was just a wrapper that was calling the `copy`
function. This wrapper function is removed and the `copy` function is
renamed to `copyFile`.
 2024-05-12 19:20:17 +05:30
siddhantCodes
d3b7367c80 inline usage of nix::getFileType and remove it 2024-05-12 18:58:05 +05:30
siddhantCodes
4537663740 inline the usage of nix::renameFile 
use `std::filesystem::rename` everywhere and remove `nix::renameFile`
 2024-05-12 18:40:16 +05:30
siddhantCodes
1db7d1b840 inline the usage of nix::readDirectory 
`nix::readDirectory` is removed. `std::filesystem::directory_iterator`
is used directly in places that used this util.
 2024-05-12 17:42:18 +05:30
John Ericson
39b2a399ad Start building the scheduler for Windows 
Building derivations is a lot harder, but the downloading goals is
portable enough.

The "common channel" code is due to Volth. I wonder if there is a way we
can factor it out into separate functions / files to avoid some
within-function CPP.

Co-authored-by: volth <volth@volth.com>
 2024-05-10 20:23:59 -04:00
John Ericson
e0ff8da9d5 Build the local store on Windows 
Fixes #10558

Co-Authored-By: Eugene Butler <eugene@eugene4.com>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2024-05-10 13:05:23 -04:00
Eelco Dolstra
77a406a5a6 Fix warning 2024-05-08 21:16:53 +02:00
Eelco Dolstra
79c7d6205c Support unit prefixes in configuration settings 
E.g. you can now say `--min-free 1G`.
 2024-05-08 21:11:09 +02:00
siddhantCodes
ddea4c6deb rm isLink 
isLink util is removed in favour of std::filesystem::is_symlink
 2024-05-08 19:59:37 +05:30
Siddhant Kumar
fcbc36cf78
Use std::filesystem::path in more places (#10657) 
Progress on #9205

Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>

* Get rid of `PathNG`, just use `std::filesystem::path`
 2024-05-07 22:28:50 +00:00
John Ericson
72a0d4b022 Try to fix macOS Nixpkgs lib test failure 
Sometimes we read a directory with children we cannot stat. It's a pitty
we even try to stat at all (wasteful) in the `DT_UNKNOWN` case, but at
least this should get rid of the failure.
 2024-05-07 16:21:02 -04:00
John Ericson
a3c573950b Replace our DirEntry with std::filesystem's 2024-05-07 16:21:02 -04:00
John Ericson
c371070580 Use std::filesystem functions in more places 
This makes for shorter and more portable code.

The only tricky part is catching exceptions: I just searched for near by
`catch (Error &)` or `catch (SysError &)` and adjusted them to `catch
(std::filesystem::filesystem_error &)` according to my human judgement.

Good for windows portability; will help @siddhantk232 with his GSOC
project.
 2024-05-07 00:16:54 -04:00
Eelco Dolstra
ef28c7329c Rename makeFSSourceAccessor -> getFSSourceAccessor() 
This makes it clearer that it returns a shared accessor object.
 2024-05-06 19:16:52 +02:00
Eelco Dolstra
eab2919119 Use SourcePath in more places 
Now that SourcePath uses a SourceAccessor instead of an InputAccessor,
we can use it in function signatures instead of passing a
SourceAccessor and CanonPath separately.
 2024-05-06 19:05:42 +02:00
Eelco Dolstra
709cd44d3e Rename remaining instances of "InputAccessor" to "SourceAccessor" 2024-05-06 17:29:03 +02:00
Eelco Dolstra
20558e0462 Remove FSInputAccessor 2024-05-03 12:30:28 +02:00
Eelco Dolstra
ba5929c7be Merge InputAccessor into SourceAccessor 
After the removal of the InputAccessor::fetchToStore() method, the
only remaining functionality in InputAccessor was `fingerprint` and
`getLastModified()`, and there is no reason to keep those in a
separate class.
 2024-05-03 12:14:01 +02:00
John Ericson
037c8d771d Fix format errors 
Fix formatting violations, update blacklist to reflect moved files.

PR #10556 passed CI before the new formating rules were added, and our
CI has the race condition of allowing old results, resulting in master
getting broken.
 2024-05-02 21:42:28 -04:00
John Ericson
1948ec390c
Merge pull request #10556 from nix-windows/uds-remote-on-windows 
Enable the `unix://` store on Windows
 2024-05-02 09:53:00 -04:00
Robert Hensing
6fd2f42c2d
Merge pull request #10573 from RCoeurjoly/Rename_SearchPath 
Rename SearchPath to LookupPath and searchPath to lookupPath
 2024-04-21 17:33:46 +02:00