Commit graph

10547 commits

Author SHA1 Message Date
Robert Hensing
2477e4e3b8 libexpr: Use GC_set_sp_corrector instead of patch
Manually tested by printing to stderr in both branches (sp in os
stack, or not), and triggering a GC in a filterSource function,
e.g.:

    let
      generateTree = n: if n == 0 then "ha" else { left = generateTree (n - 1); right = generateTree (n - 1); };
    in
      builtins.deepSeq (generateTree 18) ...

Note that the darwin still uses the strategy of disabling GC, despite
having an implementation that compiles. The proper solution will be
enabled and tested later.
2024-06-03 16:37:39 +02:00
Eelco Dolstra
eb0d46fab6
Merge pull request #9897 from bryango/fix-submodule-subdir
libutil/url: fix git+file:./ parse error
2024-06-03 16:04:41 +02:00
Eelco Dolstra
ac3e5d22e3
Merge pull request #10028 from DavHau/fetchTree-shallow-default
fetchTree: shallow git fetching by default
2024-06-03 16:02:34 +02:00
John Ericson
4e62629a2d
Merge pull request #10833 from obsidiansystems/hash-ordering
Modernize `Hash` ordering with C++20 `<=>`
2024-06-03 09:50:04 -04:00
Eelco Dolstra
ecfad6a828
Merge pull request #10564 from edolstra/remove-forceErrors
AttrCursor: Remove forceErrors
2024-06-03 15:30:01 +02:00
John Ericson
cfc18a7739 Modernize Hash ordering with C++20 <=>
Progress on #10832

This doesn't switch to auto-deriving the fields, but by defining `<=>`
we allow deriving `<=>` in downstream types where `Hash` is used.
2024-06-03 09:24:33 -04:00
Eelco Dolstra
d16fcaee21
Merge pull request #10782 from obsidiansystems/both-connections
Factor our connection code for worker proto like serve proto
2024-06-03 15:10:38 +02:00
John Ericson
84c65135a5 ValidPathInfo JSON format should use null not omit field
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-06-03 08:21:22 -04:00
John Ericson
213a7a87b4 Decouple within-build (structured attrs) and unstable CLI path info JSON
See code comment for details.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-06-03 08:21:22 -04:00
Robert Hensing
b74a0df645
Merge pull request #10825 from tie/output-spec-assert
Fix empty outputsToInstall for InstallableAttrPath
2024-06-03 12:27:50 +02:00
Philipp Zander
25e2b1f7f7 improve note in nix_value_force documentation 2024-06-03 09:55:44 +02:00
Ivan Trubach
68090d7ff1 Fix empty outputsToInstall for InstallableAttrPath
Fixes assertion failure if outputsToInstall is empty by defaulting to the "out"
output. That is, behavior between the following commands should be consistent:

	$ nix build --no-link --json .#nothing-to-install-no-out
	error: derivation '/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nothing-to-install-no-out.drv' does not have wanted outputs 'out'

	$ nix build --no-link --file default.nix --json nothing-to-install-no-out
	error: derivation '/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nothing-to-install-no-out.drv' does not have wanted outputs 'out'

Real-world example of this issue:

	$ nix build --json .#.legacyPackages.aarch64-linux.texlive.pkgs.iwona
	error: derivation '/nix/store/dj0h6b0pnlnan5nidnhqa0bmzq4rv6sx-iwona-0.995b.drv' does not have wanted outputs 'out'

	$ git rev-parse HEAD
	eee33247cf6941daea8398c976bd2dda7962b125
	$ nix build --json --file . texlive.pkgs.iwona
	nix: src/libstore/outputs-spec.hh:46: nix::OutputsSpec::Names::Names(std::set<std::__cxx11::basic_string<char> >&&): Assertion `!empty()' failed.
	Aborted (core dumped)
2024-06-02 14:26:18 +03:00
John Ericson
e0b159549b Misc Windows fixes
1. Fix build by making the legacy SSH Storey's secret `logFD` setting
   not a setting on Windows. (It doesn't make sense to specify `void *`
   handles by integer cross-proccess, I don't think.)

2. Move some files that don't need to be Unix-only anymore back to their
   original locations.
2024-06-01 19:19:35 -04:00
Robert Hensing
802b4e403b
Merge pull request #10814 from Mic92/fix-nix-edit
Fix nix edit
2024-05-31 13:30:24 +02:00
Robert Hensing
84e116379c
Merge pull request #10812 from Mic92/build-perf
Remove 100s of CPU time (10%) from build times (1465s -> 1302s)
2024-05-31 13:28:24 +02:00
Jade Lovelace
473d2d56fc Remove 100s of CPU time (10%) from build times (1465s -> 1302s)
Result's from Mic92's framework 13th Gen Intel Core i7-1360P:

Before: 3595.92s user 183.01s system 1360% cpu 4:37.74 total
After: 3486.07s user 168.93s system 1354% cpu 4:29.79 total

I saw that boost/lexical_cast was costing about 100s in CPU time on our
compiles. We can fix this trivially by doing explicit template
instantiation in exactly one place and eliminating all other includes of
it, which is a code improvement anyway by hiding the boost.

Before:
```
lix/lix2 » ClangBuildAnalyzer --analyze buildtimeold.bin
Analyzing build trace from 'buildtimeold.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1465.3 s
  Codegen & opts (backend):   1110.9 s

<snip>

**** Expensive headers:
178153 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 3426 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

176217 ms: ../src/libutil/error.hh (included 246 times, avg 716 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: serialise.hh strings.hh
  6x: <direct include>
  6x: archive.hh serialise.hh strings.hh
  ...

173243 ms: ../src/libstore/store-api.hh (included 152 times, avg 1139 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

170482 ms: ../src/libutil/serialise.hh (included 201 times, avg 848 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh
  14x: store-api.hh nar-info.hh hash.hh
  11x: <direct include>
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh
  6x: fetchers.hh hash.hh
  ...

169397 ms: ../src/libcmd/installables.hh (included 53 times, avg 3196 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-derived-path.hh
  1x: installable-value.hh
  ...

159740 ms: ../src/libutil/strings.hh (included 221 times, avg 722 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh
  19x: <direct include>
  14x: store-api.hh nar-info.hh hash.hh serialise.hh
  11x: serialise.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh
  ...

156796 ms: ../src/libcmd/command.hh (included 51 times, avg 3074 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

150392 ms: ../src/libutil/types.hh (included 251 times, avg 599 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

133101 ms: /nix/store/644b90j1vms44nr18yw3520pzkrg4dd1-boost-1.81.0-dev/include/boost/lexical_cast.hpp (included 226 times, avg 588 ms), included via
:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh strings.hh
  19x: file-system.hh
  11x: store-api.hh nar-info.hh hash.hh serialise.hh strings.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  ...

132887 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 507 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

  done in 0.6s.
```

After:
```
lix/lix2 » maintainers/buildtime_report.sh build
Processing all files and saving to '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
  done in 0.6s. Run 'ClangBuildAnalyzer --analyze /home/jade/lix/lix2/maintainers/../buildtime.bin' to analyze it.
Analyzing build trace from '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1302.1 s
  Codegen & opts (backend):    956.3 s

<snip>

**** Expensive headers:
178145 ms: ../src/libutil/error.hh (included 246 times, avg 724 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: <direct include>
  6x: serialise.hh strings.hh
  6x: fetchers.hh hash.hh serialise.hh strings.hh
  ...

154043 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 2962 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

153593 ms: ../src/libstore/store-api.hh (included 152 times, avg 1010 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

149948 ms: ../src/libutil/types.hh (included 251 times, avg 597 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

144560 ms: ../src/libcmd/installables.hh (included 53 times, avg 2727 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-value.hh
  1x: installable-derived-path.hh
  ...

136585 ms: ../src/libcmd/command.hh (included 51 times, avg 2678 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

133394 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 509 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

89315 ms: ../src/libstore/derived-path.hh (included 178 times, avg 501 ms), included via:
  37x: command.hh installable-value.hh installables.hh
  25x: store-api.hh realisation.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: libexpr.hh value.hh context.hh
  6x: shared.hh
  ...

87347 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/ostream (included 273 times, avg 319 ms), included via:
  35x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh memory unique_ptr.h
  12x: regex sstream istream
  10x: file-system.hh types.hh ref.hh memory unique_ptr.h
  10x: gtest.h memory unique_ptr.h
  10x: globals.hh types.hh ref.hh memory unique_ptr.h
  6x: fetchers.hh types.hh ref.hh memory unique_ptr.h
  ...

85249 ms: ../src/libutil/config.hh (included 213 times, avg 400 ms), included via:
  37x: command.hh installable-value.hh installables.hh derived-path.hh
  20x: globals.hh
  20x: logging.hh
  16x: store-api.hh logging.hh
  6x: <direct include>
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh derived-path.hh
  ...

  done in 0.5s.
```

Adapated from 18aa3e1d57
2024-05-31 13:00:09 +02:00
Robert Hensing
138aa2b0a7
Merge pull request #10807 from hercules-ci/issue-10504-nix-env-shell
Add `nix env shell`
2024-05-31 12:34:03 +02:00
Jörg Thalheim
e1a817fb1b fix nix edit in pure mode
FilteringSourceAccessor was not delegating getPhysicalPath to its inner accessor.
2024-05-31 10:39:30 +02:00
Linus Heckemann
a9031978da libfetchers: handle nonexistent refs in GitLab repos more gracefully
Before:

$ nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent
fetching git input 'git+file:///home/linus/projects/lix'
fetching gitlab input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'
error: [json.exception.type_error.302] type must be string, but is null

After:

/tmp/inst/bin/nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent

warning: unknown experimental feature 'repl-flake'
error:
       … while updating the lock file of flake 'git+file:///home/joerg/git/nix?ref=refs/heads/master&rev=62693c2c37c8edd92f95114eb1387b461fc671df'

       … while updating the flake input 'nixpkgs'

       … while fetching the input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'

       error: No commits returned by GitLab API -- does the git ref really exist?

Adapted from: 3df013597d
2024-05-31 08:24:53 +02:00
Robert Hensing
d93cc11491 Format 2024-05-30 19:41:58 +02:00
Robert Hensing
c692f6af13 nix env shell: Move from nix shell, add shorthand alias 2024-05-30 19:41:58 +02:00
Robert Hensing
98b85b2166 nix/main: Add AliasStatus::{Deprecated,AcceptedShorthand} 2024-05-30 19:41:58 +02:00
Robert Hensing
ef5c846e25
Merge pull request #10768 from obsidiansystems/legacy-ssh-expose-ssh-master-for-hydra
Create `CommonSSHStoreConfig::createSSHMaster`
2024-05-29 22:53:29 +02:00
Robert Hensing
1054ff0873
Merge pull request #10789 from nix-windows/windows-substitution-goal
More work on the scheduler for windows
2024-05-29 22:45:55 +02:00
Robert Hensing
154769544d
Merge pull request #10805 from hercules-ci/issue-10774
libcmd: Fix #10774
2024-05-29 22:39:00 +02:00
Robert Hensing
1c70eb8eee libcmd: Fix #10774 2024-05-29 21:42:53 +02:00
Qyriad
18ac6545fc print type and value in "flake attr is not a derivation" errors
This turns errors like:

error: flake output attribute 'hydraJobs' is not a derivation or path

into errors like:

error: expected flake output attribute 'hydraJobs' to be a derivation or
path but found a set: { binaryTarball = «thunk»; build = «thunk»; etc> }

This change affects all InstallableFlake commands.

Source: 20981461d4
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
2024-05-29 20:51:32 +02:00
Jörg Thalheim
5786e1ae7c
docs: mention importNative/exec in allow-unsafe-native-code-during-evaluation (#10803)
* docs: mention importNative/exec in allow-unsafe-native-code-during-evaluation

Both of these still needs their own actual documentation, but they are
at least now mentioned that they exist and what they're enabled by.

Co-authored-by: Qyriad <qyriad@qyriad.me>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2024-05-29 07:50:51 +00:00
Philipp
2bd66922ee
add empty line to documentation comments after @brief field (#10800)
* add empty line to documentation comments after `@brief` field

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2024-05-28 23:05:40 +00:00
John Ericson
bcdee80a0d More work on the scheduler for windows
- Get a rump derivation goal: hook instance will come later, local
  derivation goal will come after that.

- Start cleaning up the channel / waiting code with an abstraction.
2024-05-28 11:39:49 -04:00
John Ericson
1d5d748fe4 Fix format
39b2a399ad passed CI but was landed after
the formatting change in 1d6c2316a9.
2024-05-27 22:32:52 -04:00
John Ericson
d0c7da131f
Merge pull request #10678 from nix-windows/windows-substitution-goal
Start building the scheduler for Windows
2024-05-27 17:47:29 -04:00
John Ericson
3e9c3738d3 Create CommonSSHStoreConfig::createSSHMaster
By moving `host` to the config, we can do a lot further cleanups and
dedups. This anticipates a world where we always go `StoreReference` ->
`*StoreConfig` -> `Store*` rather than skipping the middle step too.

Progress on #10766

Progress on https://github.com/NixOS/hydra/issues/1164
2024-05-27 16:12:53 -04:00
John Ericson
17964441d9
Merge pull request #10781 from obsidiansystems/build-mode-parse
Worker proto use proper serialiser for `BuildMode`
2024-05-27 16:07:54 -04:00
John Ericson
263905da4b
Merge pull request #10785 from mjoerg/fix-typos
fix typos
2024-05-27 10:24:53 -04:00
John Ericson
8527f4e7fa
Merge pull request #10754 from qwqawawow/master
nix repl: make runNix() isInteractive is true by default
2024-05-27 10:07:15 -04:00
Martin Joerg
e7ea5591a2 fix typos 2024-05-27 15:56:52 +02:00
John Ericson
aa4a2927a7
Merge pull request #10778 from Mic92/safer-tab-completion
repl: do not crash when tab-completing import errors
2024-05-27 09:54:29 -04:00
John Ericson
f71b4da0b3 Factor our connection code for worker proto like serve proto
This increases test coverage, and gets the worker protocol ready to be
used by Hydra.

Why don't we just try to use the store interface in Hydra? Well, the
problem is that the store interface works on connection pools, with each
opreation getting potentially a different connection, but the way temp
roots work requires that we keep one logical "transaction" (temp root
session) using the same connection.

The longer-term solution probably is making connections themselves
implement the store interface, but that is something that builds on
this, so I feel OK that this is not churn in the wrong direction.

Fixes #9584
2024-05-27 00:43:46 -04:00
John Ericson
8ebd99c74e Back in enum values for BuildMode serializer
We don't want to rely on how C assigns numbers for enums in the wire
format. Sure, this is totally determined by the ABI, but it obscures the
code and makes it harder to safely change the enum definition (should we
need to) without accidentally breaking the wire format.
2024-05-27 00:22:55 -04:00
John Ericson
eeb89c28b0 Worker proto use proper serialiser for BuildMode
Do this instead of an unchecked cast

I redid this to use the serialisation framework (including a unit test),
but I am keeping the reference to credit Jade for spotting the issue.

Change-Id: Icf6af7935e8f139bef36b40ad475e973aa48855c
(adapted from commit 2a7a824d83dc5fb33326b8b89625685f283a743b)

Co-Authored-By: Jade Lovelace <lix@jade.fyi>
2024-05-27 00:22:55 -04:00
Pierre Bourdon
ffe6ba69d6 repl: do not crash when tab-completing import errors
File not found while importing is not currently caught by the tab-completion handler.

Original bug report: https://git.lix.systems/lix-project/lix/issues/340
Fix has been adapted from https://gerrit.lix.systems/c/lix/+/1189

Example crash:

$ cat /tmp/foo.nix
{
  someImport = import ./this_file_does_not_exist;
}
$ ./src/nix/nix repl --file /tmp/foo.nix
warning: unknown experimental feature 'repl-flake'
Nix 2.23.0pre20240517_dirty
Type :? for help.
Loading installable ''...
Added 1 variables.
nix-repl> someImport.<TAB>
2024-05-25 23:15:28 +02:00
Jörg Thalheim
f97da4b11c libutil/source-accessor: custom error if source does not exist
This allows better error handling by catching this error in particular.
2024-05-25 23:06:57 +02:00
eldritch horrors
2de4589e46 libstore: remove unused copyPath function
Source: 47523944c5

Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
2024-05-25 22:28:41 +02:00
John Ericson
5cfa75ea16
Merge pull request #10737 from poweredbypie/mingw-stackSize
Implement `setStackSize` on Windows
2024-05-25 09:56:02 -04:00
John Ericson
e0c94b91ee
Merge pull request #10757 from obsidiansystems/fix-4977
Require `drvPath` attribute to end with `.drv`
2024-05-24 12:14:59 -04:00
Eelco Dolstra
eeb4c40867 Typo 2024-05-24 16:35:06 +02:00
Eelco Dolstra
2c88930ef2 AttrCursor: Remove forceErrors
Instead, force evaluation of the original value only if we need to
show the exception to the user.
2024-05-24 16:34:57 +02:00
Robert Hensing
c90a763273
Merge pull request #10767 from hercules-ci/fix-c-api-primop-for-strict-initializers
C API: Fix custom primops
2024-05-24 08:32:02 +02:00
PoweredByPie
0b7da099d1 Commit more stack size in some windows binaries
This way we can commit the same amount of stack size (64 MB) without a conditional.
Includes nix, libnixexpr-tests, libnixfetchers-tests, libnixstore-tests, libnixutil-tests.
2024-05-23 17:42:55 -07:00