max/nix-super
1
0
Fork 0
mirror of https://github.com/privatevoid-net/nix-super.git synced 2024-11-16 03:06:17 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
17543 commits 7 branches 0 tags 64 MiB
Commit graph

186 commits

Author SHA1 Message Date
Eelco Dolstra
7f5b57d18f Merge remote-tracking branch 'origin/master' into large-path-warning 2024-06-03 15:32:27 +02:00
Jade Lovelace
473d2d56fc Remove 100s of CPU time (10%) from build times (1465s -> 1302s) 
Result's from Mic92's framework 13th Gen Intel Core i7-1360P:

Before: 3595.92s user 183.01s system 1360% cpu 4:37.74 total
After: 3486.07s user 168.93s system 1354% cpu 4:29.79 total

I saw that boost/lexical_cast was costing about 100s in CPU time on our
compiles. We can fix this trivially by doing explicit template
instantiation in exactly one place and eliminating all other includes of
it, which is a code improvement anyway by hiding the boost.

Before:
```
lix/lix2 » ClangBuildAnalyzer --analyze buildtimeold.bin
Analyzing build trace from 'buildtimeold.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1465.3 s
  Codegen & opts (backend):   1110.9 s

<snip>

**** Expensive headers:
178153 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 3426 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

176217 ms: ../src/libutil/error.hh (included 246 times, avg 716 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: serialise.hh strings.hh
  6x: <direct include>
  6x: archive.hh serialise.hh strings.hh
  ...

173243 ms: ../src/libstore/store-api.hh (included 152 times, avg 1139 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

170482 ms: ../src/libutil/serialise.hh (included 201 times, avg 848 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh
  14x: store-api.hh nar-info.hh hash.hh
  11x: <direct include>
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh
  6x: fetchers.hh hash.hh
  ...

169397 ms: ../src/libcmd/installables.hh (included 53 times, avg 3196 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-derived-path.hh
  1x: installable-value.hh
  ...

159740 ms: ../src/libutil/strings.hh (included 221 times, avg 722 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh
  19x: <direct include>
  14x: store-api.hh nar-info.hh hash.hh serialise.hh
  11x: serialise.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh
  ...

156796 ms: ../src/libcmd/command.hh (included 51 times, avg 3074 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

150392 ms: ../src/libutil/types.hh (included 251 times, avg 599 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

133101 ms: /nix/store/644b90j1vms44nr18yw3520pzkrg4dd1-boost-1.81.0-dev/include/boost/lexical_cast.hpp (included 226 times, avg 588 ms), included via
:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh strings.hh
  19x: file-system.hh
  11x: store-api.hh nar-info.hh hash.hh serialise.hh strings.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  ...

132887 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 507 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

  done in 0.6s.
```

After:
```
lix/lix2 » maintainers/buildtime_report.sh build
Processing all files and saving to '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
  done in 0.6s. Run 'ClangBuildAnalyzer --analyze /home/jade/lix/lix2/maintainers/../buildtime.bin' to analyze it.
Analyzing build trace from '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1302.1 s
  Codegen & opts (backend):    956.3 s

<snip>

**** Expensive headers:
178145 ms: ../src/libutil/error.hh (included 246 times, avg 724 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: <direct include>
  6x: serialise.hh strings.hh
  6x: fetchers.hh hash.hh serialise.hh strings.hh
  ...

154043 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 2962 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

153593 ms: ../src/libstore/store-api.hh (included 152 times, avg 1010 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

149948 ms: ../src/libutil/types.hh (included 251 times, avg 597 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

144560 ms: ../src/libcmd/installables.hh (included 53 times, avg 2727 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-value.hh
  1x: installable-derived-path.hh
  ...

136585 ms: ../src/libcmd/command.hh (included 51 times, avg 2678 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

133394 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 509 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

89315 ms: ../src/libstore/derived-path.hh (included 178 times, avg 501 ms), included via:
  37x: command.hh installable-value.hh installables.hh
  25x: store-api.hh realisation.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: libexpr.hh value.hh context.hh
  6x: shared.hh
  ...

87347 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/ostream (included 273 times, avg 319 ms), included via:
  35x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh memory unique_ptr.h
  12x: regex sstream istream
  10x: file-system.hh types.hh ref.hh memory unique_ptr.h
  10x: gtest.h memory unique_ptr.h
  10x: globals.hh types.hh ref.hh memory unique_ptr.h
  6x: fetchers.hh types.hh ref.hh memory unique_ptr.h
  ...

85249 ms: ../src/libutil/config.hh (included 213 times, avg 400 ms), included via:
  37x: command.hh installable-value.hh installables.hh derived-path.hh
  20x: globals.hh
  20x: logging.hh
  16x: store-api.hh logging.hh
  6x: <direct include>
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh derived-path.hh
  ...

  done in 0.5s.
```

Adapated from 18aa3e1d57
 2024-05-31 13:00:09 +02:00
Eelco Dolstra
dbe1b51580 Add setting to warn about copying/hashing large paths 
This is useful for diagnosing whether an evaluation is copying large
paths to the store. Example:

   $ nix build .#packages.x86_64-linux.default --large-path-warning-threshold 1000000
   warning: copied large path '/home/eelco/Dev/nix-master/' to the store (6271792 bytes)
   warning: copied large path '«github:NixOS/nixpkgs/b550fe4b4776908ac2a861124307045f8e717c8e?narHash=sha256-7kkJQd4rZ%2BvFrzWu8sTRtta5D1kBG0LSRYAfhtmMlSo%3D»/' to the store (155263768 bytes)
   warning: copied large path '«github:libgit2/libgit2/45fd9ed7ae1a9b74b957ef4f337bc3c8b3df01b5?narHash=sha256-oX4Z3S9WtJlwvj0uH9HlYcWv%2Bx1hqp8mhXl7HsLu2f0%3D»/' to the store (22175416 bytes)
   warning: copied large path '/nix/store/z985088mcd6w23qwdlirsinnyzayagki-source' to the store (5885872 bytes)
 2024-05-13 11:52:27 +02:00
Eelco Dolstra
eab2919119 Use SourcePath in more places 
Now that SourcePath uses a SourceAccessor instead of an InputAccessor,
we can use it in function signatures instead of passing a
SourceAccessor and CanonPath separately.
 2024-05-06 19:05:42 +02:00
John Ericson
5a365b0c89
Delete dead openFile in binary-cache-store.cc (#10418) 
d64cb33e90 / #5111 previously deleted the dead code where this was used, but missed this.
 2024-04-05 18:31:43 +00:00
John Ericson
d4ad1fcf30 Avoid creating temporary store object for git over the wire 
Instead, serialize as NAR and send that over, then rehash sever side.
This is alorithmically simpler, but comes at the cost of a newer
parameter to `Store::addToStoreFromDump`.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2024-02-27 11:27:34 -05:00
John Ericson
201551c937 Add Git object hashing to the store layer 
Part of RFC 133

Extracted from our old IPFS branches.

Co-Authored-By: Matthew Bauer <mjbauer95@gmail.com>
Co-Authored-By: Carlo Nucera <carlo.nucera@protonmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Florian Klink <flokli@flokli.de>
 2024-02-27 11:27:34 -05:00
Eelco Dolstra
a6737b7e17 CanonPath, SourcePath: Change operator + to / 
This is less confusing and makes it more similar to std::filesystem::path.
 2024-02-05 15:17:39 +01:00
John Ericson
12bb8cdd38 Signer infrastructure: Prep for #9076 
This sets up infrastructure in libutil to allow for signing other than
by a secret key in memory. #9076 uses this to implement remote signing.

(Split from that PR to allow reviewing in smaller chunks.)

Co-Authored-By: Raito Bezarius <masterancpp@gmail.com>
 2024-01-03 16:13:55 -05:00
John Ericson
ed26b186fb Remove now-redundant text-hashing store methods 
`addTextToStore` and `computeStorePathFromDump` are now redundant.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
 2023-12-18 10:44:10 -05:00
John Ericson
dfc876531f Organize content addressing, use SourceAccessor with Store::addToStore 
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
 2023-12-18 10:41:54 -05:00
Peter Kolloch
fc6f29053a Renamed HashFormat::Base32 to HashFormat::Nix32 
...and also adjusted parsing accordingly.

Also added CLI completion for HashFormats.

https://github.com/NixOS/nix/issues/8876
 2023-12-06 23:43:42 +01:00
Peter Kolloch
5334c9c792 HashType: Rename to HashAlgorithm 
To be consistent with CLI, nix API
and many other references.

As part of this, we also converted it to a scoped enum.

https://github.com/NixOS/nix/issues/8876
 2023-12-06 23:43:42 +01:00
John Ericson
ac89bb064a Split up util.{hh,cc} 
All OS and IO operations should be moved out, leaving only some misc
portable pure functions.

This is useful to avoid copious CPP when doing things like Windows and
Emscripten ports.

Newly exposed functions to break cycles:

 - `restoreSignals`
 - `updateWindowSize`
 2023-11-05 12:20:02 -05:00
Eelco Dolstra
e47984ce0b
Fix whitespace 
Co-authored-by: John Ericson <git@JohnEricson.me>
 2023-11-01 20:19:08 +01:00
Eelco Dolstra
1a902f5fa7 Merge FSAccessor into SourceAccessor 2023-11-01 17:09:28 +01:00
Eelco Dolstra
5381123879 Unify DirEntries types 2023-11-01 15:33:35 +01:00
Eelco Dolstra
8ffd1695ce Unify FSAccessor::Type and SourceAccessor::Type 2023-11-01 14:43:20 +01:00
Eelco Dolstra
b2ac6fc040 Remove FSAccessor::Type::tMissing 
Instead stat() now returns std::nullopt to denote that the file
doesn't exist.
 2023-11-01 14:36:40 +01:00
Yueh-Shun Li
e026f3e1ae treewide: Reference HashFormat members with scope 
Base* -> HashFormat::Base*
 2023-10-19 00:56:41 +08:00
John Ericson
903700c5e1 Simplify ContentAddress 
Whereas `ContentAddressWithReferences` is a sum type complex because different
varieties support different notions of reference, and
`ContentAddressMethod` is a nested enum to support that,
`ContentAddress` can be a simple pair of a method and hash.

`ContentAddress` does not need to be a sum type on the outside because
the choice of method doesn't effect what type of hashes we can use.

Co-Authored-By: Cale Gibbard <cgibbard@gmail.com>
 2023-07-07 07:30:01 -04:00
John Ericson
537e8719f2
Explain various .self = false, 
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
 2023-04-17 09:15:11 -04:00
John Ericson
a6d00a7bfb Fix warning 2023-03-30 16:29:13 -04:00
John Ericson
d381248ec0 No inheritance for TextInfo and FixedOutputInfo 2023-02-28 12:14:11 -05:00
John Ericson
85bb865d20 Revert "Remove some designated initializers" 
This reverts commit ee9eb83a84.
 2023-02-28 11:57:20 -05:00
John Ericson
ee9eb83a84 Remove some designated initializers 
With the switch to C++20, the rules became more strict, and we can no
longer initialize base classes. Make them comments instead.

(BTW
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2287r1.html
this offers some new syntax for this use-case. Hopefully this will be
adopted and we can eventually use it.)
 2023-02-01 11:25:56 -05:00
John Ericson
43414738a0 Merge remote-tracking branch 'upstream/master' into path-info 2023-02-01 10:02:30 -05:00
Eelco Dolstra
14b0b9ea5a
Merge pull request #7203 from graham33/feature/cpp20 
Proposal: Use C++20
 2023-02-01 15:41:04 +01:00
John Ericson
4540e7b940 Don't add StorePathDescriptor for now 
We don't need it yet, we can add it back later.
 2023-01-23 12:58:27 -05:00
John Ericson
b3d91239ae Make ValidPathInfo have plain StorePathSet references like before 
This change can wait for another PR.
 2023-01-14 16:42:03 -05:00
John Ericson
056cc1c1b9 Merge remote-tracking branch 'upstream/master' into path-info 2023-01-14 14:27:28 -05:00
Théophane Hufschmitt
bdeb6de889
Merge pull request #7430 from tweag/ca/fix-nix-log 
Ca/fix nix log
 2023-01-13 11:00:56 +01:00
John Ericson
46e942ff9e Do big rename to clean up code 
- `PathReferences` -> `References`

- `PathReferences<StorePath>` -> `StoreReference`

- `references` -> `others`

- `hasSelfReference` -> `self`

And get rid of silly subclassing
 2023-01-06 15:36:05 -05:00
John Ericson
6a168254ce Use named field initialization for references 2023-01-06 12:24:20 -05:00
John Ericson
e9fc1e4fdb Merge remote-tracking branch 'upstream/master' into path-info 2023-01-06 10:35:20 -05:00
Taeer Bar-Yam
e5eb05c599 getBuildLog: factor out resolving derivations 2022-12-15 15:58:54 -05:00
regnat
04b113f6cb Fix nix log with CA derivations 
Fix #6209

When trying to run `nix log <installable>`, try first to resolve the derivation pointed to
by `<installable>` as it is the resolved one that holds the build log.

This has a couple of shortcomings:
1. It’s expensive as it requires re-reading the derivation
2. It’s brittle because if the derivation doesn’t exist anymore or can’t
   be resolved (which is the case if any one of its build inputs is missing),
   then we can’t access the log anymore

However, I don’t think we can do better (at least not right now).
The alternatives I see are:
1. Copy the build log for the un-resolved derivation. But that means a
   lot of duplication
2. Store the results of the resolving in the db. Which might be the best
   long-term solution, but leads to a whole new class of potential
   issues.
 2022-12-08 16:03:20 -05:00
Eelco Dolstra
703d863a48 Trivial changes from the lazy-trees branch 2022-12-07 14:06:34 +01:00
Yorick van Pelt
09f00dd4d0
Replace src/libutil/json.cc with nlohmann json generation 2022-11-16 16:50:50 +01:00
Graham Bennett
4563e80363 Fix C++20 warnings 2022-10-22 15:16:46 +01:00
Eelco Dolstra
e136d57f26
Implement BinaryCacheStore::queryPathFromHashPart() 2022-10-18 17:48:09 +02:00
John Ericson
8ba089597f Merge remote-tracking branch 'upstream/master' into path-info 2022-03-10 15:48:14 +00:00
Eelco Dolstra
df552ff53e Remove std::string alias (for real this time) 
Also use std::string_view in a few more places.
 2022-02-25 16:13:02 +01:00
Eelco Dolstra
2d6d9a28eb addToStoreFromDump(): Take std::string_view 2022-02-16 11:02:35 +01:00
Eelco Dolstra
5b243a2b4b BinaryCacheStore: Implement addBuildLog() 2022-01-18 16:14:01 +01:00
Eelco Dolstra
f6f0bcf11f
Doh 2022-01-18 14:06:51 +01:00
Eelco Dolstra
50be51d9a8 Doh 2022-01-18 13:50:25 +01:00
Eelco Dolstra
d62a9390fc Get rid of std::shared_ptr<std::string> and ref<std::string> 
These were needed back in the pre-C++11 era because we didn't have
move semantics. But now we do.
 2022-01-18 11:12:30 +01:00
Eelco Dolstra
6463eaca14
Merge pull request #5472 from NixOS/async-realisation-substitution 
async realisation substitution
 2021-11-16 12:54:20 +01:00
Eelco Dolstra
67179472df
Merge pull request #5494 from tweag/balsoft/allow-references-in-addPath 
Allow references in addPath
 2021-11-09 15:57:39 +01:00