Commit graph

482 commits

Author SHA1 Message Date
Robert Hensing
3df619339c Split ignoreException for destructors or interrupt-safe 2024-09-30 11:50:25 +02:00
John Ericson
2b6b03d8df Ensure error messages don't leak private key
Since #8766, invalid base64 is rendered in errors, but we don't actually
want to show this in the case of an invalid private keys.

Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
2024-09-23 16:36:48 -04:00
John Ericson
d0c351bf43 Revert "base64Decode: clearer error message when an invalid character is detected"
We have a safer way of doing this.

This reverts commit dc3ccf02bf.
2024-09-20 10:41:45 -04:00
John Ericson
95f2b2beab
Merge pull request #8766 from brianmcgee/fix/base64-decode-error-message
base64Decode: clearer error message when an invalid character is detected
2024-09-17 09:59:15 -04:00
John Ericson
6c861b9c51 Factor out lookupExecutable and other PATH improvments
This ended up motivating a good deal of other infra improvements in
order to get Windows right:

- `OsString` to complement `std::filesystem::path`

- env var code for working with the underlying `OsString`s

- Rename `PATHNG_LITERAL` to `OS_STR`

- `NativePathTrait` renamed to `OsPathTrait`, given a character template
  parameter until #9205 is complete.

Split `tests.cc` matching split of `util.{cc,hh}` last year.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-08-07 18:12:58 -04:00
John Ericson
9d2d4d11e6 Split tests, organize more string functions
The test split matches PR #8920, so the utility files and tests files
are once again to 1-1. The string changes continues what was started in
PR #11093.
2024-08-05 12:50:13 -04:00
Eelco Dolstra
7f5b57d18f Merge remote-tracking branch 'origin/master' into large-path-warning 2024-06-03 15:32:27 +02:00
Jade Lovelace
473d2d56fc Remove 100s of CPU time (10%) from build times (1465s -> 1302s)
Result's from Mic92's framework 13th Gen Intel Core i7-1360P:

Before: 3595.92s user 183.01s system 1360% cpu 4:37.74 total
After: 3486.07s user 168.93s system 1354% cpu 4:29.79 total

I saw that boost/lexical_cast was costing about 100s in CPU time on our
compiles. We can fix this trivially by doing explicit template
instantiation in exactly one place and eliminating all other includes of
it, which is a code improvement anyway by hiding the boost.

Before:
```
lix/lix2 » ClangBuildAnalyzer --analyze buildtimeold.bin
Analyzing build trace from 'buildtimeold.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1465.3 s
  Codegen & opts (backend):   1110.9 s

<snip>

**** Expensive headers:
178153 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 3426 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

176217 ms: ../src/libutil/error.hh (included 246 times, avg 716 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: serialise.hh strings.hh
  6x: <direct include>
  6x: archive.hh serialise.hh strings.hh
  ...

173243 ms: ../src/libstore/store-api.hh (included 152 times, avg 1139 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

170482 ms: ../src/libutil/serialise.hh (included 201 times, avg 848 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh
  14x: store-api.hh nar-info.hh hash.hh
  11x: <direct include>
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh
  6x: fetchers.hh hash.hh
  ...

169397 ms: ../src/libcmd/installables.hh (included 53 times, avg 3196 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-derived-path.hh
  1x: installable-value.hh
  ...

159740 ms: ../src/libutil/strings.hh (included 221 times, avg 722 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh
  19x: <direct include>
  14x: store-api.hh nar-info.hh hash.hh serialise.hh
  11x: serialise.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh
  ...

156796 ms: ../src/libcmd/command.hh (included 51 times, avg 3074 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

150392 ms: ../src/libutil/types.hh (included 251 times, avg 599 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

133101 ms: /nix/store/644b90j1vms44nr18yw3520pzkrg4dd1-boost-1.81.0-dev/include/boost/lexical_cast.hpp (included 226 times, avg 588 ms), included via
:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh strings.hh
  19x: file-system.hh
  11x: store-api.hh nar-info.hh hash.hh serialise.hh strings.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  ...

132887 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 507 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

  done in 0.6s.
```

After:
```
lix/lix2 » maintainers/buildtime_report.sh build
Processing all files and saving to '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
  done in 0.6s. Run 'ClangBuildAnalyzer --analyze /home/jade/lix/lix2/maintainers/../buildtime.bin' to analyze it.
Analyzing build trace from '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1302.1 s
  Codegen & opts (backend):    956.3 s

<snip>

**** Expensive headers:
178145 ms: ../src/libutil/error.hh (included 246 times, avg 724 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: <direct include>
  6x: serialise.hh strings.hh
  6x: fetchers.hh hash.hh serialise.hh strings.hh
  ...

154043 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 2962 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

153593 ms: ../src/libstore/store-api.hh (included 152 times, avg 1010 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

149948 ms: ../src/libutil/types.hh (included 251 times, avg 597 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

144560 ms: ../src/libcmd/installables.hh (included 53 times, avg 2727 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-value.hh
  1x: installable-derived-path.hh
  ...

136585 ms: ../src/libcmd/command.hh (included 51 times, avg 2678 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

133394 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 509 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

89315 ms: ../src/libstore/derived-path.hh (included 178 times, avg 501 ms), included via:
  37x: command.hh installable-value.hh installables.hh
  25x: store-api.hh realisation.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: libexpr.hh value.hh context.hh
  6x: shared.hh
  ...

87347 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/ostream (included 273 times, avg 319 ms), included via:
  35x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh memory unique_ptr.h
  12x: regex sstream istream
  10x: file-system.hh types.hh ref.hh memory unique_ptr.h
  10x: gtest.h memory unique_ptr.h
  10x: globals.hh types.hh ref.hh memory unique_ptr.h
  6x: fetchers.hh types.hh ref.hh memory unique_ptr.h
  ...

85249 ms: ../src/libutil/config.hh (included 213 times, avg 400 ms), included via:
  37x: command.hh installable-value.hh installables.hh derived-path.hh
  20x: globals.hh
  20x: logging.hh
  16x: store-api.hh logging.hh
  6x: <direct include>
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh derived-path.hh
  ...

  done in 0.5s.
```

Adapated from 18aa3e1d57
2024-05-31 13:00:09 +02:00
Eelco Dolstra
cf3b044b7e Make large path warnings human-readable 2024-05-13 11:52:38 +02:00
Eelco Dolstra
5314430437 Move printSize() into libutil
Also always include the unit (i.e. "MiB" instead of "M").
2024-05-13 11:52:38 +02:00
John Ericson
02fa20622f Start factoring out Unix assumptions
This splits files and adds new identifiers in preperation for supporting
windows, but no Windows-specific code is actually added yet.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-04-02 14:43:38 -04:00
Jade Lovelace
2625e9fb0a Ban building Nix with NDEBUG
When reviewing old PRs, I found that #9997 adds some code to ensure one
particular assert is always present. But, removing asserts isn't
something we do in our own release builds either in the flake here or in
nixpkgs, and is plainly a bad idea that increases support burden,
especially if other distros make bad choices of build flags in their Nix
packaging.

For context, the assert macro in the C standard is defined to do nothing
if NDEBUG is set.

There is no way in our build system to set -DNDEBUG without manually
adding it to CFLAGS, so this is simply a configuration we do not use.
Let's ban it at compile time.

I put this preprocessor directive in src/libutil.cc because it is not
obvious where else to put it, and it seems like the most logical file
since you are not getting a usable nix without it.
2024-03-02 11:22:46 -08:00
Jacek Galowicz
c924147c9d Drop parentheses from thunks 2024-01-16 17:00:46 +00:00
Jacek Galowicz
1885d579db Improve String Handling 2024-01-16 17:00:46 +00:00
Jacek Galowicz
0bc66e529f Use npos member variables instead of full type 2024-01-16 15:17:57 +00:00
John Ericson
6208ca7209 Separate SystemError from SysError
Most of this is a `catch SysError` -> `catch SystemError` sed. This
is a rather pure-churn change I would like to get out of the way. **The
intersting part is `src/libutil/error.hh`.**

On Unix, we will only throw the `SysError` concrete class, which has
the same constructors that `SystemError` used to have.

On Windows, we will throw `WinError` *and* `SysError`. `WinError`
(which will be created in a later PR), will use a `DWORD` instead of
`int` error value, and `GetLastError()`, which is the Windows equivalent
of the `errno` machinery. Windows will *also* use `SysError` because
Window's "libc" (MSVCRT) implements the POSIX interface, and we use it
too.

As the docs describe, while we *throw* one of the 3 choices above (2
concrete classes or the alias), we should always *catch* `SystemError`.
This ensures no matter how the implementation changes for Windows (e.g.
between `SysError` and `WinError`) the catching logic stays the same
and stays correct.

Co-Authored-By volth <volth@volth.com>
Co-Authored-By Eugene Butler <eugene@eugene4.com>
2024-01-12 12:00:33 -05:00
John Ericson
12bb8cdd38 Signer infrastructure: Prep for #9076
This sets up infrastructure in libutil to allow for signing other than
by a secret key in memory. #9076 uses this to implement remote signing.

(Split from that PR to allow reviewing in smaller chunks.)

Co-Authored-By: Raito Bezarius <masterancpp@gmail.com>
2024-01-03 16:13:55 -05:00
Robert Hensing
20ff61ab25 nix: Reserve shebang line syntax and only parse double backtick quotes
Being restrictive about syntax leaves opportunity to improve the
syntax and functionality later.
2023-11-07 13:24:01 -05:00
Tom Bereknyei
74210c12fe Shellbang support with flakes
Enables shebang usage of nix shell. All arguments with `#! nix` get
added to the nix invocation. This implementation does NOT set any
additional arguments other than placing the script path itself as the
first argument such that the interpreter can utilize it.

Example below:

```
    #!/usr/bin/env nix
    #! nix shell --quiet
    #! nix nixpkgs#bash
    #! nix nixpkgs#shellcheck
    #! nix nixpkgs#hello
    #! nix --ignore-environment --command bash
    # shellcheck shell=bash
    set -eu
    shellcheck "$0" || exit 1
    function main {
        hello
        echo 0:"$0" 1:"$1" 2:"$2"
    }
    "$@"
```

fix: include programName usage

EDIT: For posterity I've changed shellwords to shellwords2 in order
      not to interfere with other changes during a rebase.
      shellwords2 is removed in a later commit. -- roberth
2023-11-07 13:24:01 -05:00
John Ericson
ac89bb064a Split up util.{hh,cc}
All OS and IO operations should be moved out, leaving only some misc
portable pure functions.

This is useful to avoid copious CPP when doing things like Windows and
Emscripten ports.

Newly exposed functions to break cycles:

 - `restoreSignals`
 - `updateWindowSize`
2023-11-05 12:20:02 -05:00
Alex Ameen
5bc540a8ca
Respect NOCOLOR
While `nix` has always been respectful towards requests for `NO_COLOR=1`, this change asks represents a new stage of maturity for `nix` - making it also respect quests for `NOCOLOR=1`.

This ideally makes the tool more accessible to folks like me, who are exhausted by guessing whether `NO_COLOR` or `NOCOLOR` is the right environment variable to set.

<3
2023-09-27 14:49:52 -05:00
Eelco Dolstra
5c95b32c46 Fix warning 'catching polymorphic type by value' 2023-09-01 14:49:49 +02:00
Robert Hensing
c4dbb55ba9 initLibUtil: Add exception handling self-check 2023-08-11 17:25:42 +02:00
Brian McGee
dc3ccf02bf
base64Decode: clearer error message when an invalid character is detected
Output the offending string in its entirety to provide context.

Closes #8479
2023-08-01 08:55:00 +01:00
Eelco Dolstra
e54538c461 restoreMountNamespace(): Restore the original root directory
This is necessary when we're in a chroot environment, where the
process root is not the same as the root of the mount namespace
(e.g. in nixos-enter).

Fixes #7602.
2023-06-09 16:09:29 +02:00
Yorick van Pelt
2c462486fe
create pathAccessible, use it to infer default dirs 2023-05-26 15:36:47 +02:00
Konstantin Vukolov
4c4ae887b8 Add option isInteractive 2023-05-18 13:18:34 +03:00
Robert Hensing
2445afd92c Require openssl >= 1.1.1
Versions older this are sufficiently old that we don't want to support
them, and they require extra support code.
2023-04-07 17:50:40 +02:00
Robert Hensing
2196fd1146 libutil: Provide alternatives to startSignalHandlerThread
How signals should be handled depends on what kind of process Nix
is integrated into. The signal handler thread used by the stand-alone
Nix commands / processes may not work well in the context of other
runtime systems, such as those of Python, Perl, or Haskell.
2023-04-07 17:50:40 +02:00
Robert Hensing
6e0b7109ab Move OpenSSL init to initLibUtil
Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
2023-04-07 16:24:18 +02:00
Eelco Dolstra
515662ad70 Cleanup 2023-03-20 18:06:08 +01:00
Eelco Dolstra
16db8dc96f Open slave pseudoterminal before CLONE_NEWUSER
Otherwise, when running as root and user namespaces are enabled,
opening the slave fails with EPERM.

Fixes "opening pseudoterminal slave: Permission denied" followed by a
hang (https://hydra.nixos.org/build/213104244), and "error: getting
sandbox mount namespace: No such file or directory" (#8072), which
happens when the child fails very quickly and consequently reading
/proc/<child>/ns fails.
2023-03-20 17:58:36 +01:00
Eelco Dolstra
19326ac297 Simplify commonChildInit() 2023-03-14 18:58:12 +01:00
Eelco Dolstra
fcd0b0fbd5 Fix macOS warning 2023-03-13 05:31:03 -07:00
Eelco Dolstra
0507462c06
Merge pull request #7918 from zimbatm/fix-empty-nix-store-env
treat empty NIX_STORE_DIR env vars as unset
2023-03-03 13:49:56 +01:00
zimbatm
dc8820c71f
fixup: use same style as getEnv 2023-03-03 11:34:36 +01:00
zimbatm
b96d9c1687
fixup: remove warning entirely
fixes 72e1e23051 (r1124211067)
2023-03-03 11:32:06 +01:00
Jonas Chevalier
72e1e23051
Update src/libutil/util.cc
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-03-02 16:17:20 +01:00
Eelco Dolstra
29abc8e764 Remove FormatOrString and remaining uses of format() 2023-03-02 15:57:54 +01:00
Félix Baylac Jacqué
25300c0ecd
Treat empty env var paths as unset
We make sure the env var paths are actually set (ie. not "") before
sending them to the canonicalization function. If we forget to do so,
the user will end up facing a puzzled failed assertion internal error.

We issue a non-failing warning as a stop-gap measure. We could want to
revisit this to issue a detailed failing error message in the future.
2023-03-01 20:50:07 +01:00
Eelco Dolstra
67451d8ed7
Merge pull request #7802 from edolstra/fix-7783
Fix PID namespace support check
2023-02-10 20:41:13 +01:00
Alexander Bantyev
2384d36083
A setting to follow XDG Base Directory standard
XDG Base Directory is a standard for locations for storing various
files. Nix has a few files which seem to fit in the standard, but
currently use a custom location directly in the user's ~, polluting
it:

- ~/.nix-profile
- ~/.nix-defexpr
- ~/.nix-channels

This commit adds a config option (use-xdg-base-directories) to follow
the XDG spec and instead use the following locations:

- $XDG_STATE_HOME/nix/profile
- $XDG_STATE_HOME/nix/defexpr
- $XDG_STATE_HOME/nix/channels

If $XDG_STATE_HOME is not set, it is assumed to be ~/.local/state.

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
Co-authored-by: Tim Fenney <kodekata@gmail.com>
Co-authored-by: pasqui23 <pasqui23@users.noreply.github.com>
Co-authored-by: Artturin <Artturin@artturin.com>
Co-authored-by: John Ericson <Ericson2314@Yahoo.com>
2023-02-10 20:14:06 +04:00
Eelco Dolstra
c49b7472ea Fix macOS build 2023-02-10 17:08:33 +01:00
Eelco Dolstra
f094ba7386 Simplify the PID namespace check: just try to mount /proc
Fixes #7783.
2023-02-10 14:38:14 +01:00
Théophane Hufschmitt
a5919f4754 Move the default profiles to the user’s home
Rather than using `/nix/var/nix/{profiles,gcroots}/per-user/`, put the user
profiles and gcroots under `$XDG_DATA_DIR/nix/{profiles,gcroots}`.

This means that the daemon no longer needs to manage these paths itself
(they are fully handled client-side). In particular, it doesn’t have to
`chown` them anymore (removing one need for root).

This does change the layout of the gc-roots created by nix-env, and is
likely to break some stuff, so I’m not sure how to properly handle that.
2023-01-17 14:17:28 +01:00
Eelco Dolstra
703d863a48 Trivial changes from the lazy-trees branch 2022-12-07 14:06:34 +01:00
Jörg Thalheim
cccd57c022 getMaxCPU: fix cgroup path
Given this typo I am not sure if it has been tested.
2022-12-04 18:22:12 +01:00
Eelco Dolstra
fa99ef6a87 getMaxCPU(): Lower verbosity level for ignored exceptions
Fixes #7268.
2022-12-02 15:03:40 +01:00
Eelco Dolstra
1e6a5d1ff6 Clean up cgroup handling in getMaxCPU()
Also, don't assume in LocalDerivationGoal that cgroups are mounted on
/sys/fs/cgroup.
2022-12-02 12:59:13 +01:00
squalus
1b595026e1 Improve durability of schema version file writes
- call close explicitly in writeFile to prevent the close exception
  from being ignored
- fsync after writing schema file to flush data to disk
- fsync schema file parent to flush metadata to disk

https://github.com/NixOS/nix/issues/7064
2022-09-19 20:13:30 -07:00