Commit graph

7256 commits

Author SHA1 Message Date
Eelco Dolstra
f39d94a55b Merge branch 'readfile' of https://github.com/abbradar/nix 2022-01-10 16:55:28 +01:00
Eelco Dolstra
19fd6e585d 'target' points to 'source' 2022-01-10 16:52:25 +01:00
Eelco Dolstra
df11e75d0e Merge branch 'recursive-nix-fix' of https://github.com/L-as/nix 2022-01-10 16:51:59 +01:00
Nikolay Amiantov
c66865dff1 builtins.readFile: Propagate path context
Co-authored-by: Shea Levy <shea@shealevy.com>
2022-01-09 13:07:00 +03:00
Eelco Dolstra
fcddaa4b9b dump(): Use emplace() 2022-01-07 13:23:17 +01:00
Eelco Dolstra
e1a0359b59 isInDir() / isDirOrInDir(): Use std::string_view 2022-01-07 13:23:00 +01:00
Eelco Dolstra
be64fb9b51 DerivationGoal::loadDerivation(): Don't use derivationFromPath()
This causes a recursive call to ensurePath(), which is not a good
idea.
2022-01-07 13:22:34 +01:00
pennae
26a8b220eb avoid ostream sentries per json string character
we don't have to create an ostream sentry object for every character of a JSON
string we write. format a bunch of characters and flush them to the stream all
at once instead.

this doesn't affect small numbers of string characters, but larger numbers of
total JSON string characters written gain a lot. at 1MB of total string written
we gain almost 30%, at 16MB it's almost a factor of 3x. large numbers of JSON
string characters do occur naturally in a nixos system evaluation to generate
documentation (though this is now somewhat mitigated by caching the largest part
of nixos option docs).

benchmarked with

  hyperfine 'nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) {e})"' --warmup 1 -L e 1,4,256,4096,65536

before:

  Benchmark 1: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 1)"
    Time (mean ± σ):      12.5 ms ±   0.2 ms    [User: 9.2 ms, System: 4.0 ms]
    Range (min … max):    11.9 ms …  13.1 ms    223 runs

  Benchmark 2: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 4)"
    Time (mean ± σ):      12.5 ms ±   0.2 ms    [User: 9.3 ms, System: 3.8 ms]
    Range (min … max):    11.9 ms …  13.2 ms    220 runs

  Benchmark 3: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 256)"
    Time (mean ± σ):      13.2 ms ±   0.3 ms    [User: 9.8 ms, System: 4.0 ms]
    Range (min … max):    12.6 ms …  14.3 ms    205 runs

  Benchmark 4: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 4096)"
    Time (mean ± σ):      24.0 ms ±   0.4 ms    [User: 19.4 ms, System: 5.2 ms]
    Range (min … max):    22.7 ms …  25.8 ms    119 runs

  Benchmark 5: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 65536)"
    Time (mean ± σ):     196.0 ms ±   3.7 ms    [User: 171.2 ms, System: 25.8 ms]
    Range (min … max):   190.6 ms … 201.5 ms    14 runs

after:

  Benchmark 1: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 1)"
    Time (mean ± σ):      12.4 ms ±   0.3 ms    [User: 9.1 ms, System: 4.0 ms]
    Range (min … max):    11.7 ms …  13.3 ms    204 runs

  Benchmark 2: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 4)"
    Time (mean ± σ):      12.4 ms ±   0.2 ms    [User: 9.2 ms, System: 3.9 ms]
    Range (min … max):    11.8 ms …  13.0 ms    214 runs

  Benchmark 3: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 256)"
    Time (mean ± σ):      12.6 ms ±   0.2 ms    [User: 9.5 ms, System: 3.8 ms]
    Range (min … max):    12.1 ms …  13.3 ms    209 runs

  Benchmark 4: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 4096)"
    Time (mean ± σ):      15.9 ms ±   0.2 ms    [User: 11.4 ms, System: 5.1 ms]
    Range (min … max):    15.2 ms …  16.4 ms    171 runs

  Benchmark 5: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 65536)"
    Time (mean ± σ):      69.0 ms ±   0.9 ms    [User: 44.3 ms, System: 25.3 ms]
    Range (min … max):    67.2 ms …  70.9 ms    42 runs
2022-01-07 06:53:47 +01:00
Farid Zakaria
e36fdbbfd9 Check for --meta before emitting it for --json
Check that the meta flag is present when emitting JSON query information
for `nix-env`.

fixes #5882
2022-01-06 14:40:02 -08:00
Eelco Dolstra
f71d84672b Merge branch 'install-nlohmann-json-headers' of https://github.com/hercules-ci/nix 2022-01-06 22:34:03 +01:00
Eelco Dolstra
5c7d3b351f
Merge pull request #5878 from fzakaria/faridzakaria/json-outputs
Add outputs to JSON query
2022-01-06 22:33:11 +01:00
Eelco Dolstra
3ec02c6743
Merge pull request #5546 from NixOS/allow-paths-in-flake-local-options
allow paths in flake local options
2022-01-06 21:55:20 +01:00
Farid Zakaria
65257614ea Add outputs to JSON query
Emit output information when printing JSON information
and `--out-paths` is requested.

fixes #5877
2022-01-06 09:13:18 -08:00
Domen Kožar
bdc577936f
Merge pull request #5749 from GavinRay97/patch-1
Better diagnostics if no valid signature found
2022-01-06 08:39:58 +01:00
Eelco Dolstra
2b4c944823 Remove EvalState::mkAttrs() 2022-01-04 20:29:17 +01:00
Eelco Dolstra
17daec0b83 Move empty attrset optimisation 2022-01-04 19:23:11 +01:00
Eelco Dolstra
ca5baf2392 Turn mkString(Symbol) into a method 2022-01-04 19:09:40 +01:00
Eelco Dolstra
ed93aec3c3 Remove non-method mkPath() 2022-01-04 18:45:16 +01:00
Eelco Dolstra
263a8d293c Remove non-method mk<X> functions 2022-01-04 18:40:39 +01:00
Eelco Dolstra
cc08364315 Remove non-method mkString() 2022-01-04 18:24:42 +01:00
Eelco Dolstra
6d9a6d2cc3 Ensure that attrsets are sorted
Previously you had to remember to call value->attrs->sort() after
populating value->attrs. Now there is a BindingsBuilder helper that
wraps Bindings and ensures that sort() is called before you can use
it.
2022-01-04 18:00:33 +01:00
pennae
00c993f48b add zipAttrsWith primop
nixpkgs can save a good bit of eval memory with this primop. zipAttrsWith is
used quite a bit around nixpkgs (eg in the form of recursiveUpdate), but the
most costly application for this primop is in the module system. it improves
the implementation of zipAttrsWith from nixpkgs by not checking an attribute
multiple times if it occurs more than once in the input list, allocates less
values and set elements, and just avoids many a temporary object in general.

nixpkgs has a more generic version of this operation, zipAttrsWithNames, but
this version is only used once so isn't suitable for being the base of a new
primop. if it were to be used more we should add a second primop instead.
2022-01-03 21:05:53 +01:00
Eelco Dolstra
96d08fcd66
Merge pull request #5839 from tweag/balsoft/yet-another-follows-bugfix
flake.cc: computeLocks: Only verify overrides when they could change
2022-01-03 20:51:58 +01:00
Eelco Dolstra
70dfcbbb37
Merge pull request #5840 from tweag/balsoft/nix-repl-show-trace
nix repl: fix --show-trace and add the ability to set trace display
2022-01-03 20:41:59 +01:00
Sebastian Ullrich
d0c8e9254e Fix IFD with chroot store 2021-12-29 19:00:02 +01:00
Alexander Bantyev
a26351da02
Add ability to toggle show-trace from within the repl 2021-12-29 02:54:36 +03:00
Lily Foster
92c7d33ee3 Sort attrs from tables in fromTOML
This was dropped in 10a8b5d for the migration from cpptoml to toml11 but
seems to be necessary for the attrsets to work correctly.

Fixes #5833
2021-12-28 10:11:00 -05:00
Alexander Bantyev
581f774284
BaseError::calcWhat: take loggerSettings.showTrace into account
Text representation for errors should include the trace if
--show-trace is passed.
2021-12-28 15:53:21 +03:00
Alexander Bantyev
2664a216e5
flake.cc: computeLocks: Only verify overrides when they could change
When we check for disappeared overrides, we can get "false positives"
for follows and overrides which are defined in the dependencies of the
flake we are locking, since they are not parsed by
parseFlakeInputs. However, at that point we already know that the
overrides couldn't have possible been changed if the input itself
hasn't changed (since we check that oldLock->originalRef == *input.ref
for the input's parent). So, to prevent this, only perform this check
when it was possible that the flake changed (e.g. the flake we're
locking, or a new input, or the input has changed and mustRefetch ==
true).
2021-12-28 15:30:06 +03:00
Yorick van Pelt
a4ab0a74d9
Fix accidental O(n^2 * log n) performance in NixRepl::addAttrsToScope
Only sort once, after adding all of the attrs first. This reduces my
`nix repl '<nixpkgs>'` loading time from 1.07s to 103ms.

Fixes #5823
2021-12-27 13:18:55 +01:00
tomberek
f404189368
Merge branch 'master' into flake_search 2021-12-23 15:22:52 -05:00
Nicolas Mattia
b16643b6fc Use int64_t and NixFloat in fromTOML types
This makes sure that values parsed from TOML have a proper size. Using
e.g. `double` caused issues on i686 where the size of `double` (32bit)
was too small to accommodate some values.
2021-12-23 14:12:49 +01:00
regnat
dc89dfa7b3 Properly return false on builtins.pathExists /someNonAllowedPath
Follow-up from https://github.com/NixOS/nix/pull/5807 to fix https://github.com/NixOS/nix/pull/5807#issuecomment-1000135394
2021-12-23 10:49:33 +01:00
Eelco Dolstra
7feb741e00
Merge pull request #5821 from edolstra/remove-affinity
Remove CPU locking
2021-12-22 20:31:19 +01:00
Eelco Dolstra
b666a2ca8c
Merge pull request #5820 from ncfavier/completion-nospace
Don't insert spaces when completing attribute paths
2021-12-22 17:55:05 +01:00
Naïm Favier
c4a03bc4ae
Fix attr path completion after a dot 2021-12-22 16:37:58 +01:00
Naïm Favier
1da1b2b345
Don't insert spaces when completing attribute paths 2021-12-22 16:17:01 +01:00
Eelco Dolstra
9747ea84b4 Remove CPU locking
This was already accidentally disabled in ba87b08. It also no longer
appears to be beneficial, and in fact slow things down, e.g. when
evaluating a NixOS system configuration:

  elapsed time:       median =      3.8170  mean =      3.8202  stddev =      0.0195  min =      3.7894  max =      3.8600  [rejected, p=0.00000, Δ=0.36929±0.02513]
2021-12-22 15:56:25 +01:00
Eelco Dolstra
89faff93e2
Merge pull request #5807 from NixOS/5805-ca-ifd
Fix IFD with CA derivations
2021-12-21 18:47:34 +01:00
Eelco Dolstra
25cea2b737 Merge branch 'nm-toml11' of https://github.com/nmattia/nix 2021-12-21 14:18:04 +01:00
Eelco Dolstra
3228a38e1c Merge branch 'why-depends-completion' of https://github.com/ncfavier/nix 2021-12-21 11:38:07 +01:00
regnat
d90f9d4b99 Fix IFD with CA derivations
Rewrite the string taken by the IFD-like primops to contain the actual
output paths of the derivations rather than the placeholders

Fix #5805
2021-12-21 09:36:50 +01:00
regnat
cbbd21ec07 Factor out the path realisation bit of IFD 2021-12-21 09:36:19 +01:00
pennae
09b245690a bulk-allocate Value instances in the evaluator
calling GC_malloc for each value is significantly more expensive than
allocating a bunch of values at once with GC_malloc_many. "a bunch" here
is a GC block size, ie 16KiB or less.

this gives a 1.5% performance boost when evaluating our nixos system.

tested with

nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'

 # on master

  Time (mean ± σ):      3.335 s ±  0.007 s    [User: 2.774 s, System: 0.293 s]
  Range (min … max):    3.315 s …  3.347 s    50 runs

 # with this change

  Time (mean ± σ):      3.288 s ±  0.006 s    [User: 2.728 s, System: 0.292 s]
  Range (min … max):    3.274 s …  3.307 s    50 runs
2021-12-20 23:01:28 +01:00
Nicolas Mattia
9dea5768ef Clean up toml parsing code 2021-12-20 21:39:37 +01:00
Naïm Favier
1827cf5a9b
Add shell completion for why-depends
Fixes #5806
2021-12-20 19:58:30 +01:00
Nicolas Mattia
5d06836b9f Use long in fromTOML 2021-12-17 23:15:26 +01:00
Nicolas Mattia
10a8b5d3ae Update primops/fromTOML.cc to use toml11 2021-12-17 22:13:04 +01:00
Nicolas Mattia
c3f68b5db7 Replace cpptoml with toml11 2021-12-17 22:03:33 +01:00
Eelco Dolstra
6e6e998930
Merge pull request #5787 from edolstra/unshare-fs
Ignore EPERM when unsharing FS state
2021-12-17 12:00:09 +01:00
Théophane Hufschmitt
23ea1e46cc
Merge pull request #5047 from symphorien/fix-nix-channel
nix-channel: use nix-env -i --remove-all to upgrade
2021-12-17 10:05:48 +01:00
regnat
ab902521b1 Allow paths in flake local settings
Fix #5505
2021-12-17 09:31:13 +01:00
Eelco Dolstra
ec8f24ed3a Ignore EPERM when unsharing FS state
On Docker (but not podman), unshare(CLONE_FS) fails with EPERM. So
let's ignore it and hope nothing bad happens.

Attempted fix for #5777.
2021-12-16 22:02:50 +01:00
Noah Snelson
c3f9344cae Fix typo in src/nix/shell.md
Changed `If not command` to `If no command`, also specified that the
default shell launched in `nix shell` can be specified with `$SHELL`.
2021-12-15 23:40:37 -08:00
Eelco Dolstra
18e4851752
Merge pull request #5769 from NixOS/ca/get-build-stats
Add a crude tracing mechansim for the build results
2021-12-14 10:53:51 +01:00
Théophane Hufschmitt
8868da45a8
Merge pull request #5623 from yorickvP/fix-5621
flakes: fix boolean and int nixConfig values
2021-12-14 10:35:37 +01:00
Théophane Hufschmitt
99d617bcde
Merge pull request #5771 from edolstra/single-file-flake-inputs
Re-allow inputs.x.url = "/path/to/file"
2021-12-14 09:17:24 +01:00
Théophane Hufschmitt
6234e1c811
Merge pull request #5702 from baloo/baloo/hide-non-reproducible-settings
reproducibility: hide non-reproducible settings from manual
2021-12-14 09:14:25 +01:00
Théophane Hufschmitt
3fc8042f77
Merge pull request #5366 from trofi/speedup-ca-query
ca-specific-schema.sql: add index on RealisationsRefs(referrer)
2021-12-14 09:08:17 +01:00
Eelco Dolstra
95bd5da341 queryMissing(): Use a bigger thread pool
This is a temporary hack until we make queryMissing() async.

Closes #5324.
2021-12-13 21:11:37 +01:00
Eelco Dolstra
ade870764a
Merge pull request #5725 from ConnorBaker/master
Add support for .tar.gz shorthand (.tgz) to TarballInputScheme
2021-12-13 21:08:06 +01:00
Eelco Dolstra
32a62b0d25 Re-allow inputs.x.url = "/path/to/file" 2021-12-13 20:59:33 +01:00
Eelco Dolstra
46d86e06ba Simplify 2021-12-13 20:28:53 +01:00
Eelco Dolstra
9529de2eed Merge branch 'arm32-personality' of https://github.com/cleverca22/nix 2021-12-13 19:58:40 +01:00
Eelco Dolstra
6ae5f39ea0
Merge pull request #5770 from edolstra/gc-fixes
Fix macOS GC failures
2021-12-13 19:55:32 +01:00
Eelco Dolstra
e2e5f3a78f Respect /etc/hosts
https://hydra.nixos.org/build/161439329
2021-12-13 19:37:30 +01:00
Eelco Dolstra
cc6406cc59 Merge branch 'better-interpolation-error-location' of https://github.com/greedy/nix 2021-12-13 19:20:48 +01:00
Eelco Dolstra
2567b74c66 nix run: Tweak docs 2021-12-13 19:14:32 +01:00
Eelco Dolstra
775bdc0d9e Merge branch 'pname-main-program' of https://github.com/figsoda/nix 2021-12-13 19:09:36 +01:00
Eelco Dolstra
d17d46cfc2 More debug info 2021-12-13 19:03:08 +01:00
Eelco Dolstra
05fcc681ac Retry on ECONNREFUSED
https://hydra.nixos.org/build/161439235
2021-12-13 19:00:19 +01:00
Eelco Dolstra
6d30f9e6fe Explicitly make GC roots client sockets blocking
On macOS / BSD, these sockets inherit the non-blocking flag of the
server soocket, which is not what we want.

https://github.com/dotnet/runtime/issues/25069
https://bugs.python.org/issue7995

https://hydra.nixos.org/build/161439304
2021-12-13 18:59:50 +01:00
regnat
2eec2f765a Add a crude tracing mechansim for the build results
Add a `_NIX_TRACE_BUILT_OUTPUTS` environment variable that can be set to
a filename in which the result of each build will be logged.

This is intentionally crude and undocumented as it’s only meant to be a
temporary thing to assess the usefulness of CA derivations.
Any other use would need a cleaner re-implementation first.
2021-12-13 17:02:14 +01:00
regnat
55dbb7f1cc More properly track the status of CA builds
Make the build of unresolved derivations return the same status as the
resolved one, except in the case of an `AlreadyValid` in which case it
will return `ResolvesToAlreadyValid` to mean that the outputs of the unresolved
derivation weren’t known, but the resolved one is.
2021-12-13 17:02:13 +01:00
Eelco Dolstra
84497119ad
Merge pull request #5761 from yorickvP/fix-5741
extract_archive: fix "Hard-link target '...'" error
2021-12-13 13:37:40 +01:00
Yorick van Pelt
33e96820d5
EvalCommand::getEvalState: use gc tracable allocator for EvalState 2021-12-11 16:02:08 +01:00
Gavin Ray
3542d4fe16
Incorporate suggestions from @edolstra 2021-12-10 19:02:22 -05:00
Yorick van Pelt
0e0de90b35
extract_archive: fix "Hard-link target '...'" error
Fixes #5741
2021-12-10 17:03:51 +01:00
Yorick van Pelt
be1055f2cc
extract_archive: use copy_pathname instead of set_pathname.
Libarchive documentation mentions that archive_entry_set_pathname
expects us to keep the passed string alive, which we don't.
2021-12-10 17:00:32 +01:00
Eelco Dolstra
20b1290103 Ignore system.nfs4_acl
Closes #1584.
2021-12-10 15:12:55 +01:00
Eelco Dolstra
f9437b4f18 Merge branch 'master' of https://github.com/MagicRB/nix 2021-12-10 14:49:25 +01:00
regnat
55e3a1108c Properly take the last assignment in the REPL
When a variable is assigned in the REPL, make sure to remove any possible reference to the old one so that we correctly pick the new one afterwards

Fix #5706
2021-12-09 16:35:27 +01:00
Eelco Dolstra
bc57b3854e
Merge pull request #5695 from obsidiansystems/tidy-logging
Tidy up the logging
2021-12-09 15:31:08 +01:00
Gavin Ray
971382cab0
Better diagnostics if no valid signature found
I downloaded Nix tonight, and immediately broke it by accidentally removing the default binary caching.
After figuring this out, I also failed to fix it properly, due to using the wrong key for Nix's default binary cache

If the diagnostic message would have been clearer about what/where a "signature" for a "substituter" is + comes from, it probably would have saved me a few hours.
Maybe we can save other noobs the same pain?
2021-12-08 19:55:34 -05:00
Eelco Dolstra
92ff5b4254 Tweak warning 2021-12-07 22:22:02 +01:00
Eelco Dolstra
21206caf13 Merge remote-tracking branch 'origin/5712-ca-drvs-with-old-daemon' 2021-12-07 22:06:22 +01:00
Eelco Dolstra
8b5088b62f Merge branch 'balsoft/nix-repl-log' of https://github.com/tweag/nix 2021-12-07 21:33:32 +01:00
Jan Tojnar
ae21aab456 Update manual links
Fixes: https://github.com/NixOS/nixos-homepage/issues/762
2021-12-06 16:42:57 +01:00
Alexander Bantyev
33926ed1e7
nix repl: add :log
Add a :log command that shows logs for a derivation.

Closes https://github.com/NixOS/nix/issues/3504

Co-authored-by: Taeer Bar-Yam <taeer@bar-yam.me>
2021-12-06 16:52:50 +03:00
Alexander Bantyev
0a2fa2d684
RunPager: restore stdout upon pager exit
Before this change, stdout was closed after the pager exits. This is
fine for non-interactive commands where we want to exit right after
the pager exits anyways, but for interactive things (e.g. nix repl)
this breaks the output after we quit the pager.

Keep the initial stdout fd as part of RunPager, and restore it in
RunPager::~RunPager using dup2.
2021-12-06 16:51:55 +03:00
regnat
fa58bff1ab Fallback when the daemon is too old for CA derivations
Fix #5712
2021-12-06 11:45:18 +01:00
Connor Baker
52f5fa948a Add support for .tar.gz shorthand (.tgz) to TarballInputScheme 2021-12-03 22:16:02 -05:00
Tom Bereknyei
b6cc0a704d flakes: search up to git or filesystem boundary
While parsing a flakeref, upon not finding a flake.nix, search upwards
until git or filesystem boundary.
2021-12-03 10:55:30 -05:00
Silvan Mosberger
90700736c7 Introduce builtins.groupBy primop
This function is very useful in nixpkgs, but its implementation in Nix
itself is rather slow due to it requiring a lot of attribute set and
list appends.
2021-12-02 21:54:51 +01:00
Eelco Dolstra
2ff71b0213
Merge pull request #5713 from tweag/balsoft/another-follows-bugfix
Flakes: computeLocks: pass correct LockParent when reusing oldLock
2021-12-02 13:17:50 +01:00
Eelco Dolstra
77b3e4b4ee
Merge pull request #5708 from trofi/fix-build-on-gcc-11
src/libutil/util.hh: fix build on gcc-11
2021-12-02 13:16:15 +01:00
Alexander Bantyev
8cbf862e6f
Flakes: computeLocks: pass correct LockParent when reusing oldLock
Previously, when we were attempting to reuse the old lockfile
information in the computeLocks function, we have passed the parent of
the current input to the next computeLocks call. This was incorrect,
since the follows are resolved relative to the parent. This caused
issues when we tried to reuse oldLock but couldn't for some
reason (read: mustRefetch is true), in that case the follows were
resolved incorrectly.

Fix this by passing the correct parent, and adding some tests to
prevent this particular regression from happening again.

Closes https://github.com/NixOS/nix/issues/5697
2021-12-02 12:02:34 +03:00
David Arnold
7489811736
Fix parent path check boundary
- Previous to this commit the boundary was exclusive of the
  top level flake.
- This is wrong since the top level flake is still a valid
  relative reference.
- Now, the check boundary is inclusive of the top level flake.

Signed-off-by: Timothy DeHerrera <tim.deh@pm.me>
2021-12-01 17:01:48 -07:00
Sergei Trofimovich
c32a5f4d38 src/libutil/util.hh: fix build on gcc-11
Due to missing <atomic> declaration the build fails as:

    src/libutil/util.hh:350:24: error: no match for 'operator||' (operand types are 'std::atomic<bool>' and 'bool')
      350 |     if (_isInterrupted || (interruptCheck && interruptCheck()))
          |         ~~~~~~~~~~~~~~ ^~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          |         |                                 |
          |         std::atomic<bool>                 bool
2021-12-01 22:08:05 +00:00
Eelco Dolstra
782837d934 Merge branch 'nix-env-install-suggestions' of https://github.com/tweag/nix 2021-12-01 21:47:28 +01:00
Eelco Dolstra
fb662e0acf
Merge pull request #5696 from obsidiansystems/fix-5299
Fix #5299 and remove uncesssary unbounded buffer
2021-12-01 21:32:26 +01:00
Arthur Gautier
21520297da reproducibility: hide non-reproducible settings from manual
Because the manual is generated from default values which are themselves
generated from various sources (cpuid, bios settings (kvm), number of
cores). This commit hides non-reproducible settings from the manual
output.
2021-12-01 17:25:58 +01:00
Théophane Hufschmitt
f3f32f0c30
Merge pull request #5599 from fzakaria/faridzakaria/fix-progress-bar-leak
Fix heap use after free in progress-bar.cc
2021-12-01 16:28:38 +01:00
John Ericson
2f5c913d4a Fix #5299
No matter what, we need to resize the buffer to not have any scratch
space after we do the `read`. In the end of file case, `got` will be 0
from it's initial value.

Before, we forgot to resize in the EOF case with the break. Yes, we know
we didn't recieve any data in that case, but we still have the scatch
space to undo.

Co-Authored-By: Will Fancher <Will.Fancher@Obsidian.Systems>
2021-11-30 22:11:46 +00:00
John Ericson
95157b4e66 Push wopAddToStore old style stream adapters into smaller scopes
This doesn't fix the bug, but makes the code less difficult to read.
Also improve the comments, now that it is clear what part is needed in
each code path.
2021-11-30 22:10:58 +00:00
John Ericson
70a717f7a8 Nix daemon stream old wopAddToStore
No more buffering in string.
2021-11-30 21:55:53 +00:00
John Ericson
1f15441103 Tidy up the logging
Use the macros more, so we properly skip work when the log level
excludes. Also log the daemon operation number on the daemon side.
2021-11-30 20:23:13 +00:00
Yorick van Pelt
bd628cf3da
flakes: fix boolean and int nixConfig values
Some type confusion was causing ints to be pointers, and bools
to be ints. Fixes #5621
2021-11-29 15:53:22 +01:00
Eelco Dolstra
6e0cbc666b createTempFile(): Mark file as CLOEEXEC
Fixes #5674.
2021-11-29 11:20:50 +01:00
Eelco Dolstra
e1420c66a4
Merge pull request #5680 from andir/libexpr-dont-move-primop-args
Don't move the arguments of the primOp
2021-11-29 11:11:54 +01:00
Eelco Dolstra
3d36f48acb
Merge pull request #5667 from NixOS/5661-dont-complete-flakes-without-xp-feature
Don’t try to complete flakes is the feature isn’t enabled
2021-11-29 10:51:41 +01:00
Andreas Rammhold
90d8178009
Don't move the arguments of the primOp
Moving arguments of the primOp into the registration structure makes it
impossible to initialize a second EvalState with the correct primOp
registration. It will end up registering all those "RegisterPrimOp"'s
with an arity of zero on all but the 2nd instance of the EvalState.

Not moving the memory will add a tiny bit of memory overhead during the
eval since we need a copy of all the argument lists of all the primOp's.
The overhead shouldn't be too bad as it is static (based on the amonut
of registered operations) and only occurs once during the interpreter
startup.
2021-11-28 02:06:47 +01:00
Aleksey Kladov
1d5a881da5
Fix typo 2021-11-27 23:29:48 +03:00
regnat
3a0277305a Don’t try to complete flakes is the feature isn’t enabled
Fix #5661
2021-11-26 16:56:51 +01:00
Eelco Dolstra
3ef66cd23a nix repl: Reset the terminal on exceptional exits 2021-11-26 13:10:28 +01:00
Eelco Dolstra
b653fb9ccf
Merge pull request #5662 from NixOS/5259-nix-shell-with-ca-bash
Make `nix-shell` work when nixpkgs is content-addressed
2021-11-26 12:47:30 +01:00
Alex Shabalin
6d166d19a6 Catch flake-related exception type in REPL
Closes https://github.com/NixOS/nix/issues/5656
2021-11-26 11:03:48 +01:00
regnat
05081bedc1 Make nix-shell work when nixpkgs is content-addressed
Fix #5259
2021-11-26 09:56:48 +01:00
Eelco Dolstra
5fcf7f04a9
Merge pull request #5384 from baloo/baloo/dns-timeout
preloadNSS / dns timeout
2021-11-25 17:37:53 +01:00
Eelco Dolstra
d5d0621250
Merge pull request #5603 from obsidiansystems/recursive-nix-system-feature
"recursive-nix" system feature only with experimental feature
2021-11-25 17:20:58 +01:00
Eelco Dolstra
6f46434f32
Merge pull request #5648 from edolstra/list-iter
Support range-based for loop over list values
2021-11-25 17:13:25 +01:00
Eelco Dolstra
b6c8e57056 Support range-based for loop over list values 2021-11-25 16:31:39 +01:00
regnat
c47027f3a1 Fix the error when accessing a forbidden path in pure eval
If we’re in pure eval mode, then tell that in the error message rather
than (wrongly) speaking about restricted mode.

Fix https://github.com/NixOS/nix/issues/5611
2021-11-25 14:48:01 +01:00
Eelco Dolstra
1f7584d24c
Merge pull request #5644 from tweag/fix-interrupted-data-race
Fix a minor data race with _isInterrupted
2021-11-25 13:55:03 +01:00
Las Safin
8388d2c7c6
Make recursive-nix work even when not privileged
Before this, `setns` would fail when switching to the mount namespace,
since we did not have the privileges to do so when not root.

Closes #5360
2021-11-25 11:15:11 +00:00
Eelco Dolstra
d58f149140
Merge pull request #5631 from Infinisil/list-compare
Make lists be comparable
2021-11-24 15:48:05 +01:00
Alex Shabalin
2970ca18bf Fix a minor data race with _isInterrupted 2021-11-24 14:50:08 +01:00
Eelco Dolstra
884674a8e2 nix flake check: Fix markdown 2021-11-24 13:53:09 +01:00
Silvan Mosberger
09471d2680 Make lists be comparable
Makes lists comparable using lexicographic comparison.

Increments builtins.langVersion in order for this change to be
detectable
2021-11-24 13:40:46 +01:00
John Ericson
286eb81143 "recursive-nix" system feature only with experimental feature 2021-11-23 20:19:04 +00:00
Farid Zakaria
d41af23a6c Fix heap use after free in progress-bar.cc
Fix some heap-use-after-free in progress-bar.cc

These are somewhat tricky failures here due to temporary variable
creation and string_view
2021-11-23 08:34:55 -08:00
Alex Shabalin
b26cb0c9ac Fix use after free in content-address.cc
Inspired by https://github.com/NixOS/nix/pull/5599
2021-11-23 16:15:34 +01:00
Eelco Dolstra
dbfcaa607a
Merge pull request #5628 from matthewbauer/rosetta2-new-path
Set new rosetta 2 path in sandbox
2021-11-23 10:46:50 +01:00
Alex Shabalin
ab3c6a2295 Refactor: extract source file patching 2021-11-23 09:57:57 +01:00
Matthew Bauer
08b1ac3e38 Set new rosetta 2 path in sandbox
see:

https://github.com/NixOS/nix/pull/5388

and

https://github.com/NixOS/nix/pull/5251
2021-11-22 14:42:31 -06:00
Eelco Dolstra
b367f1061c
Merge pull request #5624 from rofrol/typo-single-quote
Typo: change to normal single quote
2021-11-22 21:33:21 +01:00
Alex Shabalin
0c9c7d5d44 Print worktree path instead of store path
for flake errors

Closes https://github.com/NixOS/nix/issues/5425
2021-11-22 16:28:31 +01:00
Eelco Dolstra
720ed47678
Merge pull request #5613 from tomberek/allow_realpath
add real path to allowedPaths
2021-11-22 15:10:47 +01:00
Eelco Dolstra
dcaa8dfd10
Merge pull request #5620 from alyssais/wayland
nix-shell --pure: let variables for Wayland through
2021-11-22 15:09:52 +01:00
Alex Shabalin
db2e4489a5 Unify #if linux 2021-11-22 13:57:56 +01:00
Roman Frołow
0768c08d99 Typo: change to normal singlequote 2021-11-22 13:37:38 +01:00
Alyssa Ross
664ee49e0d
nix-shell --pure: let variables for Wayland through
We let DISPLAY (X11) through, so we should let the Wayland equivalents
through as well.  Similarly, we let HOME through, so it should be okay
to allow XDG_RUNTIME_DIR (which is needed for connecting to Wayland
with WAYLAND_DISPLAY) through as well.  Otherwise graphical
applications will either fall back to X11 (if they support it), or
just not work (if they don't).
2021-11-21 16:14:19 +00:00
Tom Bereknyei
4318ba2ec5 add real path to allowedPaths 2021-11-20 00:25:36 -05:00
Alex Shabalin
86b7962807 Use warn to print a warning 2021-11-19 16:29:55 +01:00
Alex Shabalin
a18d9269a5 Fix build warnings on MacOS 2021-11-19 15:22:31 +01:00
Alex Shabalin
4b28798bfc Offer suggestions for nix-env -i
Closes https://github.com/NixOS/nix/issues/972
2021-11-18 15:34:57 +01:00
Eelco Dolstra
0961c1068a Merge branch 'accept_conf' of https://github.com/tomberek/nix 2021-11-18 12:56:09 +01:00
Eelco Dolstra
0adced4b9e
Merge pull request #5580 from ksonj/fix/non-standard-ssh
Fix detection of scp-style URIs to support non-standard SSH ports
2021-11-18 12:30:29 +01:00
Eelco Dolstra
262a3c7ce3 Simplify 2021-11-18 12:12:31 +01:00
Eelco Dolstra
acf381b061 Merge branch 'faridzakaria/fix-stack-buffer-overflow' of https://github.com/fzakaria/nix 2021-11-18 12:10:52 +01:00
Eelco Dolstra
205655e98b
Merge pull request #5586 from tweag/fix-stdout-for-editors-in-repl
Fix :e in repl
2021-11-18 11:46:44 +01:00
Eelco Dolstra
1d0c6a4b99
Merge pull request #5583 from jtojnar/patch-1
Fix XDG_CONFIG_DIRS fallback
2021-11-18 11:35:18 +01:00
Alex Shabalin
9653858ce6 Fix :e in repl
Closes https://github.com/NixOS/nix/issues/5487

Co-authored-by: Alexander Bantyev balsoft@balsoft.ru
2021-11-18 09:03:33 +01:00
Farid Zakaria
bc14465e08 Fix stack buffer overflow
Fix a stack buffer overflow found by running MemorySanitizer.
2021-11-18 04:05:25 +00:00
Jan Tojnar
e96faadcd6 Fix XDG_CONFIG_DIRS fallback
According to XDG Base Directory Specification,
it should fall back to /etc/xdg when the env var is not present.
2021-11-17 14:31:15 +01:00
Kalle Jepsen
46d2a5a10b Simplify fix by disallowing / in front of @ to match scp style 2021-11-17 13:49:10 +01:00
Eelco Dolstra
d03e89e5d1 Parse '(f x) y' the same as 'f x y'
(cherry picked from commit 5253cb4b68ad248f37b27849c0ebf3614e4f2777)
2021-11-17 09:53:57 +01:00
Kalle Jepsen
6d46b5b609 Fix detection of scp-style URIs to support non-standard SSH ports for git 2021-11-17 08:41:26 +01:00
Eelco Dolstra
d7bae52b9d Call functors with both arguments at once
This is not really useful on its own, but it does recover the
'infinite recursion' error message for '{ __functor = x: x; } 1', and
is more efficient in conjunction with #3718.

Fixes #5515.
2021-11-16 22:34:17 +01:00
Eelco Dolstra
e41cf8511f Don't hang when calling an attrset
Fixes #5565.
2021-11-16 17:44:19 +01:00
Eelco Dolstra
8c93a481af Ignore errors unsharing/restoring the mount namespace
This prevents Nix from barfing when run in a container where it
doesn't have the appropriate privileges.
2021-11-16 14:48:44 +01:00
Eelco Dolstra
ec608e3def
Merge pull request #5452 from Kha/git-extend-lock
git: extend cache dir lock over all mutating operations
2021-11-16 12:55:12 +01:00
Eelco Dolstra
6463eaca14
Merge pull request #5472 from NixOS/async-realisation-substitution
async realisation substitution
2021-11-16 12:54:20 +01:00
Eelco Dolstra
6d0aa8d175
Merge pull request #5559 from Kloenk/libfetchers-gitlab-headers
libfetchers: set free gitlab headers
2021-11-16 12:47:49 +01:00
Eelco Dolstra
9fe0343bfd Merge branch 'master' of github.com:NixOS/nix 2021-11-15 18:45:20 +01:00
Eelco Dolstra
671817a858 Simplify lockFlake() a bit 2021-11-15 18:44:27 +01:00
Eelco Dolstra
d1d223838b Merge branch 'balsoft/fix-subinputs-not-getting-updated' of https://github.com/tweag/nix 2021-11-15 18:37:13 +01:00
Domen Kožar
164179983e
Merge pull request #5428 from kreisys/add-pos-to-json-type-error
toJSON: report error position for fancier output
2021-11-15 07:57:46 -06:00
Finn Behrens
79d07d0980 libfetchers: set free gitlab headers 2021-11-14 12:23:46 +01:00
Kevin Amado
d0e9e18489
toXML: display errors position
- This change applies to builtins.toXML and inner workings
- Proof of concept:
  ```nix
  let e = builtins.toXML e; in e
  ```
- Before:
  ```
  $ nix-instantiate --eval poc.nix
  error: infinite recursion encountered
  ```
- After:
  ```
  $ nix-instantiate --eval poc.nix
  error: infinite recursion encountered

       at /data/github/kamadorueda/nix/poc.nix:1:9:

            1| let e = builtins.toXML e; in e
             |
  ```
2021-11-13 20:33:34 -05:00
Tom Bereknyei
30496af598 Adds an accept-flake-config flag 2021-11-12 09:50:07 -05:00
Eelco Dolstra
bceda30498 Typo 2021-11-12 13:41:15 +01:00
Eelco Dolstra
44aed58538
Merge pull request #5540 from samueldr/fix/key-subcommand
nix key: Fix error message and don't require flakes
2021-11-12 10:38:00 +01:00
Samuel Dionne-Riel
30e5c5c55f nix registry: Mark experimental
This is part of the flakes feature. Mark it as such.
2021-11-12 03:18:31 -05:00
Samuel Dionne-Riel
c1dea92dd6 nix key: Fix error message and don't require flakes 2021-11-11 21:31:26 -05:00
Robert Hensing
3884f7a69a Install nlohmann_json headers
These headers are included by the libexpr, libfetchers, libstore
and libutil headers.
Considering that these are vendored sources, Nix should expose them,
as it is not a good idea for reverse dependencies to rely on a
potentially different source that can go out of sync.
2021-11-11 11:05:44 +01:00
Alexander Bantyev
07bffe7998
Flakes: refetch the input when a follows disappears
When an input follows disappears, we can't just reuse the old lock
file entries since we may be missing some required ones. Refetch the
input when this happens.

Closes https://github.com/NixOS/nix/issues/5289
2021-11-11 12:05:18 +03:00
Eelco Dolstra
a218cfd6c7
Merge pull request #5528 from matthewbauer/recognize-nixosModule
Recognize singular "nixosModule" in nix flake show
2021-11-10 12:17:11 +01:00
Sergei Trofimovich
edfc5b2f12 ca-specific-schema.sql: add index on RealisationsRefs(referrer) and (outputPath)
For a typical desktop system (~2K packages) we can easily get 100K
entries in RealisationsRefs. Without indices query for RealisationsRefs
requires linear scan.

RealisationsRefs(referrer)
--------------------------

Inefficiency is seen as a 100% CPU load of nix-daemon for the following
scenario:

    $ nix edit -f . bash # add unused environment variable, like FOO="1"
    # populate RealisationsRefs, build fresh system
    $ nix build -f nixos system --arg config '{ contentAddressedByDefault = true; }'
    $ nix edit -f . bash # add unused environment variable, like FOO="2"
    $ time nix build -f nixos system --arg config '{ contentAddressedByDefault = true; }'

In this case `bash `will be rebuilt a few times and then rest of CPU
time is spent on scanning RealisationsRefs table (about 5 CPU-minutes
on my machine).

Before the change:

    $ time nix build -f nixos system ... # step 4 above
    real    34m3,613s
    user    0m5,232s
    sys     0m0,758s

Of all this time about 29.5 minutes are taken by nix-daemon's CPU time.

After the change:

    $ time nix build -f nixos system ... # step 4 above
    real    4m50,061s
    user    0m5,038s
    sys     0m0,677s

Of all this time about 1 minute is taken by nix-daemon's CPU time.
Most of the time is spent polling for non-existent realisations on
cache-nixos.org.

Realisations(outputPath)
------------------------

After running CA system for two weeks I got ~1M entries in Realisations
table. `nix-collect-garbage` became very slow (seemingly 100 path deletions
per second). It happens due to a slow cascading delete from Realisations
triggered by deletion from ValidPaths.

The fix is to add an index on primary key from ValidPaths(id) that
triggers cascading deletions.

Before the change:
    $ time nix-collect-garbage -d --max-freed 100G
    <interrupted before finish, took too long>
    real    23m32.411s
    user    17m49.679s
    sys     4m50.609s

Most of time was spent in re-scanning Realisations table on each path deletion.

After the change:
    $ time nix-collect-garbage -d --max-freed 100G

    real    8m43.226s
    user    6m16.317s
    sys     1m40.188s

Time is spent scanning sqlite indices and in kernel when unlinking directories.
2021-11-10 08:32:05 +00:00
Matthew Bauer
ceeecf2f9e Recognize singular "nixosModule" in nix flake show
This makes nixosModule appears as a "NixOS Module" like nixosModules
does.
2021-11-09 11:14:15 -06:00
Eelco Dolstra
67179472df
Merge pull request #5494 from tweag/balsoft/allow-references-in-addPath
Allow references in addPath
2021-11-09 15:57:39 +01:00
Alexander Bantyev
0b005bc9d6
addToStore, addToStoreFromDump: refactor: pass refs by const reference
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
2021-11-09 12:24:49 +03:00
Eelco Dolstra
ff2af4d64e Unshare mount namespace in main()
Doing it as a side-effect of calling LocalStore::makeStoreWritable()
is very ugly.

Also, make sure that stopping the progress bar joins the update
thread, otherwise that thread should be unshared as well.
2021-11-08 22:00:45 +01:00
Eelco Dolstra
7a71621b7c Merge branch 'fix-writable-shell' of https://github.com/yorickvP/nix 2021-11-08 21:12:51 +01:00
Eelco Dolstra
6c2af1f201
Merge pull request #5434 from timothyklim/git-url-submodules
Support building flakes from a Git repo url with submodules
2021-11-08 14:13:42 +01:00
Eelco Dolstra
736d6ab721
Merge pull request #5504 from NixOS/flake-options-and-daemon
Make the flake options work when using the daemon
2021-11-08 13:54:55 +01:00
Andreas Rammhold
8e7359db64
Remove unused "<let-body>" symbol
The requirement for the symbol has been removed since at least 7d47498.
2021-11-07 18:26:43 +01:00
Alexander Bantyev
9d4dcff37a
addPath: allow paths with references
Since 4806f2f6b0, we can't have paths with
references passed to builtins.{path,filterSource}. This prevents many cases
of those functions called on IFD outputs from working. Resolve this by
passing the references found in the original path to the added path.
2021-11-05 22:41:30 +03:00
Théophane Hufschmitt
3d6ee223d6
Merge pull request #5476 from NixOS/ca/fix-build-in-nix-repl
Fix `nix repl`’s building of CA derivations
2021-11-05 17:35:18 +01:00
regnat
1f3c3a3785 Make the flake options work when using the daemon
When setting flake-local options (with the `nixConfig` field), forward
these options to the daemon in case we’re using one.

This is necessary in particular for options like `binary-caches` or
`post-build-hook` to make sense.

Fix <343239fc8a (r44356843)>
2021-11-05 16:19:16 +01:00
regnat
1a4c9ba50b Fix nix repl’s building of CA derivations
When running a `:b` command in the repl, after building the derivations
query the store for its outputs rather than just assuming that they are
known in the derivation itself (which isn’t true for CA derivations)

Fix #5328
2021-11-05 11:04:14 +01:00
Eelco Dolstra
40925337a9 Remove maxPrimOpArity 2021-11-04 15:04:07 +01:00
Eelco Dolstra
acd6bddec7 Fix derivation primop 2021-11-04 15:04:00 +01:00
Eelco Dolstra
cbfbf71e08 Use callFunction() with an array for some calls with arity > 1 2021-11-04 15:03:57 +01:00
Eelco Dolstra
bcf4780006 Add level / displacement types 2021-11-04 15:03:45 +01:00
Eelco Dolstra
81e7c40264 Optimize primop calls
We now parse function applications as a vector of arguments rather
than as a chain of binary applications, e.g. 'substring 1 2 "foo"' is
parsed as

  ExprCall { .fun = <substring>, .args = [ <1>, <2>, <"foo"> ] }

rather than

  ExprApp (ExprApp (ExprApp <substring> <1>) <2>) <"foo">

This allows primops to be called immediately (if enough arguments are
supplied) without having to allocate intermediate tPrimOpApp values.

On

  $ nix-instantiate --dry-run '<nixpkgs/nixos/release-combined.nix>' -A nixos.tests.simple.x86_64-linux

this gives a substantial performance improvement:

  user CPU time:      median =      0.9209  mean =      0.9218  stddev =      0.0073  min =      0.9086  max =      0.9340  [rejected, p=0.00000, Δ=-0.21433±0.00677]
  elapsed time:       median =      1.0585  mean =      1.0584  stddev =      0.0024  min =      1.0523  max =      1.0623  [rejected, p=0.00000, Δ=-0.20594±0.00236]

because it reduces the number of tPrimOpApp allocations from 551990 to
42534 (i.e. only small minority of primop calls are partially
applied) which in turn reduces time spent in the garbage collector.
2021-11-04 15:03:40 +01:00
Eelco Dolstra
ab35cbd675 StaticEnv: Use std::vector instead of std::map 2021-11-04 15:03:34 +01:00
Eelco Dolstra
c4bd6a15c2 Add helper function to check whether a function arg is 'X' or '_X'
Also allow '_'.
2021-11-04 14:52:35 +01:00
Eelco Dolstra
1e7c796e66
Merge pull request #5475 from doronbehar/SQLiteWAL-vfs
libstore: Use unix-dotfile vfs if useSQLiteWAL is false
2021-11-03 21:20:27 +01:00
Christopher League
3f070cc417 In checkOverlay, accept underscored names for final/prev args.
Resolves #4416.
2021-11-03 09:25:27 -04:00