Jade Lovelace
473d2d56fc
Remove 100s of CPU time (10%) from build times (1465s -> 1302s)
...
Result's from Mic92's framework 13th Gen Intel Core i7-1360P:
Before: 3595.92s user 183.01s system 1360% cpu 4:37.74 total
After: 3486.07s user 168.93s system 1354% cpu 4:29.79 total
I saw that boost/lexical_cast was costing about 100s in CPU time on our
compiles. We can fix this trivially by doing explicit template
instantiation in exactly one place and eliminating all other includes of
it, which is a code improvement anyway by hiding the boost.
Before:
```
lix/lix2 » ClangBuildAnalyzer --analyze buildtimeold.bin
Analyzing build trace from 'buildtimeold.bin'...
**** Time summary:
Compilation (551 times):
Parsing (frontend): 1465.3 s
Codegen & opts (backend): 1110.9 s
<snip>
**** Expensive headers:
178153 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 3426 ms), included via:
40x: command.hh
5x: command-installable-value.hh
3x: installable-flake.hh
2x: <direct include>
2x: installable-attr-path.hh
176217 ms: ../src/libutil/error.hh (included 246 times, avg 716 ms), included via:
36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
12x: globals.hh config.hh experimental-features.hh
11x: file-system.hh file-descriptor.hh
6x: serialise.hh strings.hh
6x: <direct include>
6x: archive.hh serialise.hh strings.hh
...
173243 ms: ../src/libstore/store-api.hh (included 152 times, avg 1139 ms), included via:
55x: <direct include>
39x: command.hh installable-value.hh installables.hh
7x: libexpr.hh
4x: local-store.hh
4x: command-installable-value.hh installable-value.hh installables.hh
3x: binary-cache-store.hh
...
170482 ms: ../src/libutil/serialise.hh (included 201 times, avg 848 ms), included via:
37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh
14x: store-api.hh nar-info.hh hash.hh
11x: <direct include>
7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh
7x: libexpr.hh value.hh source-path.hh archive.hh
6x: fetchers.hh hash.hh
...
169397 ms: ../src/libcmd/installables.hh (included 53 times, avg 3196 ms), included via:
40x: command.hh installable-value.hh
5x: command-installable-value.hh installable-value.hh
3x: installable-flake.hh installable-value.hh
2x: <direct include>
1x: installable-derived-path.hh
1x: installable-value.hh
...
159740 ms: ../src/libutil/strings.hh (included 221 times, avg 722 ms), included via:
37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh
19x: <direct include>
14x: store-api.hh nar-info.hh hash.hh serialise.hh
11x: serialise.hh
7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh
7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh
...
156796 ms: ../src/libcmd/command.hh (included 51 times, avg 3074 ms), included via:
42x: <direct include>
7x: command-installable-value.hh
2x: installable-attr-path.hh
150392 ms: ../src/libutil/types.hh (included 251 times, avg 599 ms), included via:
36x: command.hh installable-value.hh installables.hh path.hh
11x: file-system.hh
10x: globals.hh
6x: fetchers.hh
6x: serialise.hh strings.hh error.hh
5x: archive.hh
...
133101 ms: /nix/store/644b90j1vms44nr18yw3520pzkrg4dd1-boost-1.81.0-dev/include/boost/lexical_cast.hpp (included 226 times, avg 588 ms), included via
:
37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh strings.hh
19x: file-system.hh
11x: store-api.hh nar-info.hh hash.hh serialise.hh strings.hh
7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
6x: eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
...
132887 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 507 ms), included via:
36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
16x: gtest.h
11x: file-system.hh types.hh ref.hh
10x: globals.hh types.hh ref.hh
10x: json.hpp
6x: serialise.hh
...
done in 0.6s.
```
After:
```
lix/lix2 » maintainers/buildtime_report.sh build
Processing all files and saving to '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
done in 0.6s. Run 'ClangBuildAnalyzer --analyze /home/jade/lix/lix2/maintainers/../buildtime.bin' to analyze it.
Analyzing build trace from '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
**** Time summary:
Compilation (551 times):
Parsing (frontend): 1302.1 s
Codegen & opts (backend): 956.3 s
<snip>
**** Expensive headers:
178145 ms: ../src/libutil/error.hh (included 246 times, avg 724 ms), included via:
36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
12x: globals.hh config.hh experimental-features.hh
11x: file-system.hh file-descriptor.hh
6x: <direct include>
6x: serialise.hh strings.hh
6x: fetchers.hh hash.hh serialise.hh strings.hh
...
154043 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 2962 ms), included via:
40x: command.hh
5x: command-installable-value.hh
3x: installable-flake.hh
2x: <direct include>
2x: installable-attr-path.hh
153593 ms: ../src/libstore/store-api.hh (included 152 times, avg 1010 ms), included via:
55x: <direct include>
39x: command.hh installable-value.hh installables.hh
7x: libexpr.hh
4x: local-store.hh
4x: command-installable-value.hh installable-value.hh installables.hh
3x: binary-cache-store.hh
...
149948 ms: ../src/libutil/types.hh (included 251 times, avg 597 ms), included via:
36x: command.hh installable-value.hh installables.hh path.hh
11x: file-system.hh
10x: globals.hh
6x: fetchers.hh
6x: serialise.hh strings.hh error.hh
5x: archive.hh
...
144560 ms: ../src/libcmd/installables.hh (included 53 times, avg 2727 ms), included via:
40x: command.hh installable-value.hh
5x: command-installable-value.hh installable-value.hh
3x: installable-flake.hh installable-value.hh
2x: <direct include>
1x: installable-value.hh
1x: installable-derived-path.hh
...
136585 ms: ../src/libcmd/command.hh (included 51 times, avg 2678 ms), included via:
42x: <direct include>
7x: command-installable-value.hh
2x: installable-attr-path.hh
133394 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 509 ms), included via:
36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
16x: gtest.h
11x: file-system.hh types.hh ref.hh
10x: globals.hh types.hh ref.hh
10x: json.hpp
6x: serialise.hh
...
89315 ms: ../src/libstore/derived-path.hh (included 178 times, avg 501 ms), included via:
37x: command.hh installable-value.hh installables.hh
25x: store-api.hh realisation.hh
7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh context.hh
6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh
6x: libexpr.hh value.hh context.hh
6x: shared.hh
...
87347 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/ostream (included 273 times, avg 319 ms), included via:
35x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh memory unique_ptr.h
12x: regex sstream istream
10x: file-system.hh types.hh ref.hh memory unique_ptr.h
10x: gtest.h memory unique_ptr.h
10x: globals.hh types.hh ref.hh memory unique_ptr.h
6x: fetchers.hh types.hh ref.hh memory unique_ptr.h
...
85249 ms: ../src/libutil/config.hh (included 213 times, avg 400 ms), included via:
37x: command.hh installable-value.hh installables.hh derived-path.hh
20x: globals.hh
20x: logging.hh
16x: store-api.hh logging.hh
6x: <direct include>
6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh derived-path.hh
...
done in 0.5s.
```
Adapated from 18aa3e1d57
2024-05-31 13:00:09 +02:00
Eelco Dolstra
eab2919119
Use SourcePath in more places
...
Now that SourcePath uses a SourceAccessor instead of an InputAccessor,
we can use it in function signatures instead of passing a
SourceAccessor and CanonPath separately.
2024-05-06 19:05:42 +02:00
John Ericson
5a365b0c89
Delete dead openFile in binary-cache-store.cc ( #10418 )
...
d64cb33e90#5111 previously deleted the dead code where this was used, but missed this.
2024-04-05 18:31:43 +00:00
John Ericson
d4ad1fcf30
Avoid creating temporary store object for git over the wire
...
Instead, serialize as NAR and send that over, then rehash sever side.
This is alorithmically simpler, but comes at the cost of a newer
parameter to `Store::addToStoreFromDump`.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2024-02-27 11:27:34 -05:00
John Ericson
201551c937
Add Git object hashing to the store layer
...
Part of RFC 133
Extracted from our old IPFS branches.
Co-Authored-By: Matthew Bauer <mjbauer95@gmail.com>
Co-Authored-By: Carlo Nucera <carlo.nucera@protonmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Florian Klink <flokli@flokli.de>
2024-02-27 11:27:34 -05:00
Eelco Dolstra
a6737b7e17
CanonPath, SourcePath: Change operator + to /
...
This is less confusing and makes it more similar to std::filesystem::path.
2024-02-05 15:17:39 +01:00
John Ericson
12bb8cdd38
Signer infrastructure: Prep for #9076
...
This sets up infrastructure in libutil to allow for signing other than
by a secret key in memory. #9076 uses this to implement remote signing.
(Split from that PR to allow reviewing in smaller chunks.)
Co-Authored-By: Raito Bezarius <masterancpp@gmail.com>
2024-01-03 16:13:55 -05:00
John Ericson
ed26b186fb
Remove now-redundant text-hashing store methods
...
`addTextToStore` and `computeStorePathFromDump` are now redundant.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-12-18 10:44:10 -05:00
John Ericson
dfc876531f
Organize content addressing, use SourceAccessor with Store::addToStore
...
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-12-18 10:41:54 -05:00
Peter Kolloch
fc6f29053a
Renamed HashFormat::Base32 to HashFormat::Nix32
...
...and also adjusted parsing accordingly.
Also added CLI completion for HashFormats.
https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch
5334c9c792
HashType: Rename to HashAlgorithm
...
To be consistent with CLI, nix API
and many other references.
As part of this, we also converted it to a scoped enum.
https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
John Ericson
ac89bb064a
Split up util.{hh,cc}
...
All OS and IO operations should be moved out, leaving only some misc
portable pure functions.
This is useful to avoid copious CPP when doing things like Windows and
Emscripten ports.
Newly exposed functions to break cycles:
- `restoreSignals`
- `updateWindowSize`
2023-11-05 12:20:02 -05:00
Eelco Dolstra
e47984ce0b
Fix whitespace
...
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-11-01 20:19:08 +01:00
Eelco Dolstra
1a902f5fa7
Merge FSAccessor into SourceAccessor
2023-11-01 17:09:28 +01:00
Eelco Dolstra
5381123879
Unify DirEntries types
2023-11-01 15:33:35 +01:00
Eelco Dolstra
8ffd1695ce
Unify FSAccessor::Type and SourceAccessor::Type
2023-11-01 14:43:20 +01:00
Eelco Dolstra
b2ac6fc040
Remove FSAccessor::Type::tMissing
...
Instead stat() now returns std::nullopt to denote that the file
doesn't exist.
2023-11-01 14:36:40 +01:00
Yueh-Shun Li
e026f3e1ae
treewide: Reference HashFormat members with scope
...
Base* -> HashFormat::Base*
2023-10-19 00:56:41 +08:00
John Ericson
903700c5e1
Simplify ContentAddress
...
Whereas `ContentAddressWithReferences` is a sum type complex because different
varieties support different notions of reference, and
`ContentAddressMethod` is a nested enum to support that,
`ContentAddress` can be a simple pair of a method and hash.
`ContentAddress` does not need to be a sum type on the outside because
the choice of method doesn't effect what type of hashes we can use.
Co-Authored-By: Cale Gibbard <cgibbard@gmail.com>
2023-07-07 07:30:01 -04:00
John Ericson
537e8719f2
Explain various .self = false,
...
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-17 09:15:11 -04:00
John Ericson
a6d00a7bfb
Fix warning
2023-03-30 16:29:13 -04:00
John Ericson
d381248ec0
No inheritance for TextInfo and FixedOutputInfo
2023-02-28 12:14:11 -05:00
John Ericson
85bb865d20
Revert "Remove some designated initializers"
...
This reverts commit ee9eb83a84
2023-02-28 11:57:20 -05:00
John Ericson
ee9eb83a84
Remove some designated initializers
...
With the switch to C++20, the rules became more strict, and we can no
longer initialize base classes. Make them comments instead.
(BTW
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2287r1.html
this offers some new syntax for this use-case. Hopefully this will be
adopted and we can eventually use it.)
2023-02-01 11:25:56 -05:00
John Ericson
43414738a0
Merge remote-tracking branch 'upstream/master' into path-info
2023-02-01 10:02:30 -05:00
Eelco Dolstra
14b0b9ea5a
Merge pull request #7203 from graham33/feature/cpp20
...
Proposal: Use C++20
2023-02-01 15:41:04 +01:00
John Ericson
4540e7b940
Don't add StorePathDescriptor for now
...
We don't need it yet, we can add it back later.
2023-01-23 12:58:27 -05:00
John Ericson
b3d91239ae
Make ValidPathInfo have plain StorePathSet references like before
...
This change can wait for another PR.
2023-01-14 16:42:03 -05:00
John Ericson
056cc1c1b9
Merge remote-tracking branch 'upstream/master' into path-info
2023-01-14 14:27:28 -05:00
Théophane Hufschmitt
bdeb6de889
Merge pull request #7430 from tweag/ca/fix-nix-log
...
Ca/fix nix log
2023-01-13 11:00:56 +01:00
John Ericson
46e942ff9e
Do big rename to clean up code
...
- `PathReferences` -> `References`
- `PathReferences<StorePath>` -> `StoreReference`
- `references` -> `others`
- `hasSelfReference` -> `self`
And get rid of silly subclassing
2023-01-06 15:36:05 -05:00
John Ericson
6a168254ce
Use named field initialization for references
2023-01-06 12:24:20 -05:00
John Ericson
e9fc1e4fdb
Merge remote-tracking branch 'upstream/master' into path-info
2023-01-06 10:35:20 -05:00
Taeer Bar-Yam
e5eb05c599
getBuildLog: factor out resolving derivations
2022-12-15 15:58:54 -05:00
regnat
04b113f6cb
Fix nix log with CA derivations
...
Fix #6209
When trying to run `nix log <installable>`, try first to resolve the derivation pointed to
by `<installable>` as it is the resolved one that holds the build log.
This has a couple of shortcomings:
1. It’s expensive as it requires re-reading the derivation
2. It’s brittle because if the derivation doesn’t exist anymore or can’t
be resolved (which is the case if any one of its build inputs is missing),
then we can’t access the log anymore
However, I don’t think we can do better (at least not right now).
The alternatives I see are:
1. Copy the build log for the un-resolved derivation. But that means a
lot of duplication
2. Store the results of the resolving in the db. Which might be the best
long-term solution, but leads to a whole new class of potential
issues.
2022-12-08 16:03:20 -05:00
Eelco Dolstra
703d863a48
Trivial changes from the lazy-trees branch
2022-12-07 14:06:34 +01:00
Yorick van Pelt
09f00dd4d0
Replace src/libutil/json.cc with nlohmann json generation
2022-11-16 16:50:50 +01:00
Graham Bennett
4563e80363
Fix C++20 warnings
2022-10-22 15:16:46 +01:00
Eelco Dolstra
e136d57f26
Implement BinaryCacheStore::queryPathFromHashPart()
2022-10-18 17:48:09 +02:00
John Ericson
8ba089597f
Merge remote-tracking branch 'upstream/master' into path-info
2022-03-10 15:48:14 +00:00
Eelco Dolstra
df552ff53e
Remove std::string alias (for real this time)
...
Also use std::string_view in a few more places.
2022-02-25 16:13:02 +01:00
Eelco Dolstra
2d6d9a28eb
addToStoreFromDump(): Take std::string_view
2022-02-16 11:02:35 +01:00
Eelco Dolstra
5b243a2b4b
BinaryCacheStore: Implement addBuildLog()
2022-01-18 16:14:01 +01:00
Eelco Dolstra
f6f0bcf11f
Doh
2022-01-18 14:06:51 +01:00
Eelco Dolstra
50be51d9a8
Doh
2022-01-18 13:50:25 +01:00
Eelco Dolstra
d62a9390fc
Get rid of std::shared_ptr<std::string> and ref<std::string>
...
These were needed back in the pre-C++11 era because we didn't have
move semantics. But now we do.
2022-01-18 11:12:30 +01:00
Eelco Dolstra
6463eaca14
Merge pull request #5472 from NixOS/async-realisation-substitution
...
async realisation substitution
2021-11-16 12:54:20 +01:00
Eelco Dolstra
67179472df
Merge pull request #5494 from tweag/balsoft/allow-references-in-addPath
...
Allow references in addPath
2021-11-09 15:57:39 +01:00
Alexander Bantyev
0b005bc9d6
addToStore, addToStoreFromDump: refactor: pass refs by const reference
...
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
2021-11-09 12:24:49 +03:00
Alexander Bantyev
9d4dcff37a
addPath: allow paths with references
...
Since 4806f2f6b0
2021-11-05 22:41:30 +03:00
