Commit graph

10958 commits

Author SHA1 Message Date
Sergei Trofimovich
edfc5b2f12 ca-specific-schema.sql: add index on RealisationsRefs(referrer) and (outputPath)
For a typical desktop system (~2K packages) we can easily get 100K
entries in RealisationsRefs. Without indices query for RealisationsRefs
requires linear scan.

RealisationsRefs(referrer)
--------------------------

Inefficiency is seen as a 100% CPU load of nix-daemon for the following
scenario:

    $ nix edit -f . bash # add unused environment variable, like FOO="1"
    # populate RealisationsRefs, build fresh system
    $ nix build -f nixos system --arg config '{ contentAddressedByDefault = true; }'
    $ nix edit -f . bash # add unused environment variable, like FOO="2"
    $ time nix build -f nixos system --arg config '{ contentAddressedByDefault = true; }'

In this case `bash `will be rebuilt a few times and then rest of CPU
time is spent on scanning RealisationsRefs table (about 5 CPU-minutes
on my machine).

Before the change:

    $ time nix build -f nixos system ... # step 4 above
    real    34m3,613s
    user    0m5,232s
    sys     0m0,758s

Of all this time about 29.5 minutes are taken by nix-daemon's CPU time.

After the change:

    $ time nix build -f nixos system ... # step 4 above
    real    4m50,061s
    user    0m5,038s
    sys     0m0,677s

Of all this time about 1 minute is taken by nix-daemon's CPU time.
Most of the time is spent polling for non-existent realisations on
cache-nixos.org.

Realisations(outputPath)
------------------------

After running CA system for two weeks I got ~1M entries in Realisations
table. `nix-collect-garbage` became very slow (seemingly 100 path deletions
per second). It happens due to a slow cascading delete from Realisations
triggered by deletion from ValidPaths.

The fix is to add an index on primary key from ValidPaths(id) that
triggers cascading deletions.

Before the change:
    $ time nix-collect-garbage -d --max-freed 100G
    <interrupted before finish, took too long>
    real    23m32.411s
    user    17m49.679s
    sys     4m50.609s

Most of time was spent in re-scanning Realisations table on each path deletion.

After the change:
    $ time nix-collect-garbage -d --max-freed 100G

    real    8m43.226s
    user    6m16.317s
    sys     1m40.188s

Time is spent scanning sqlite indices and in kernel when unlinking directories.
2021-11-10 08:32:05 +00:00
Eelco Dolstra
67179472df
Merge pull request #5494 from tweag/balsoft/allow-references-in-addPath
Allow references in addPath
2021-11-09 15:57:39 +01:00
Eelco Dolstra
884ef336c4
Merge pull request #5519 from edolstra/move-unshare
Unshare mount namespace in main()
2021-11-09 12:33:25 +01:00
Alexander Bantyev
0b005bc9d6
addToStore, addToStoreFromDump: refactor: pass refs by const reference
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
2021-11-09 12:24:49 +03:00
Eelco Dolstra
25d2316e8f
Merge pull request #5520 from NixOS/dependabot/github_actions/actions/checkout-2.4.0
Bump actions/checkout from 2.3.5 to 2.4.0
2021-11-09 09:16:42 +01:00
dependabot[bot]
732dd90428
Bump actions/checkout from 2.3.5 to 2.4.0
Bumps [actions/checkout](https://github.com/actions/checkout) from 2.3.5 to 2.4.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2.3.5...v2.4.0)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-11-08 22:00:58 +00:00
Eelco Dolstra
ff2af4d64e Unshare mount namespace in main()
Doing it as a side-effect of calling LocalStore::makeStoreWritable()
is very ugly.

Also, make sure that stopping the progress bar joins the update
thread, otherwise that thread should be unshared as well.
2021-11-08 22:00:45 +01:00
Eelco Dolstra
7a71621b7c Merge branch 'fix-writable-shell' of https://github.com/yorickvP/nix 2021-11-08 21:12:51 +01:00
Eelco Dolstra
6c2af1f201
Merge pull request #5434 from timothyklim/git-url-submodules
Support building flakes from a Git repo url with submodules
2021-11-08 14:13:42 +01:00
Eelco Dolstra
d9c9d0e0eb
Merge pull request #5500 from abathur/fix_darwin_existing_mounted_volume
darwin-install: fix already-mounted store volumes
2021-11-08 13:55:19 +01:00
Eelco Dolstra
736d6ab721
Merge pull request #5504 from NixOS/flake-options-and-daemon
Make the flake options work when using the daemon
2021-11-08 13:54:55 +01:00
Eelco Dolstra
c9ecc0948b
Merge pull request #5506 from NixOS/fix-post-hook-test-with-different-daemon
Make the post-build-hook use the daemon Nix package
2021-11-08 13:53:30 +01:00
Eelco Dolstra
0fd96eeb09
Merge pull request #5517 from Zimmi48/fix-cli-guideline-typo
Fix some typos in CLI guideline.
2021-11-08 13:52:57 +01:00
Eelco Dolstra
b5cb31e032
Merge pull request #5514 from andir/let-body-unused
Remove unused "<let-body>" symbol
2021-11-08 13:52:32 +01:00
Théo Zimmermann
d589782fb0
Fix some typos in CLI guideline. 2021-11-08 08:24:15 +01:00
Andreas Rammhold
8e7359db64
Remove unused "<let-body>" symbol
The requirement for the symbol has been removed since at least 7d47498.
2021-11-07 18:26:43 +01:00
Alexander Bantyev
9d4dcff37a
addPath: allow paths with references
Since 4806f2f6b0, we can't have paths with
references passed to builtins.{path,filterSource}. This prevents many cases
of those functions called on IFD outputs from working. Resolve this by
passing the references found in the original path to the added path.
2021-11-05 22:41:30 +03:00
Théophane Hufschmitt
3d6ee223d6
Merge pull request #5476 from NixOS/ca/fix-build-in-nix-repl
Fix `nix repl`’s building of CA derivations
2021-11-05 17:35:18 +01:00
regnat
1f3c3a3785 Make the flake options work when using the daemon
When setting flake-local options (with the `nixConfig` field), forward
these options to the daemon in case we’re using one.

This is necessary in particular for options like `binary-caches` or
`post-build-hook` to make sense.

Fix <343239fc8a (r44356843)>
2021-11-05 16:19:16 +01:00
Eelco Dolstra
a1c1b0e553
Merge pull request #5501 from edolstra/optimize-calls
Optimize primop calls
2021-11-05 12:57:19 +01:00
regnat
93eadd5803 Make the post-build-hook use the daemon Nix package
Having the `post-build-hook` use `nix` from the client package can lead
to a deadlock in case there’s a db migration to do between both, as a
`nix` command running inside the hook will run as root (and as such will
bypass the daemon), so might trigger a db migration, which will get
stuck trying to get a global lock on the DB (as the daemon that ran the
hook already has a lock on it).
2021-11-05 11:11:33 +01:00
regnat
1a4c9ba50b Fix nix repl’s building of CA derivations
When running a `:b` command in the repl, after building the derivations
query the store for its outputs rather than just assuming that they are
known in the derivation itself (which isn’t true for CA derivations)

Fix #5328
2021-11-05 11:04:14 +01:00
Eelco Dolstra
7d6017b7a9
Merge pull request #5493 from jtojnar/patch-1
docs: Correct fallback user config path
2021-11-04 20:31:47 +01:00
Travis A. Everett
abdf9f2a6e darwin-install: fix already-mounted store volumes
This adds an explicit unmount of the store volume to avoid cases
where the installer can hang in await_volume when:
- the user already has a store volume
- that volume is already mounted somewhere other than /nix
- they do not take a path through the installer that results in an
  explicit unmount (as both removing and encrypting the volume
  would do)
2021-11-04 14:09:40 -05:00
Eelco Dolstra
40925337a9 Remove maxPrimOpArity 2021-11-04 15:04:07 +01:00
Eelco Dolstra
05560f6350 Fix function-trace test case 2021-11-04 15:04:04 +01:00
Eelco Dolstra
acd6bddec7 Fix derivation primop 2021-11-04 15:04:00 +01:00
Eelco Dolstra
cbfbf71e08 Use callFunction() with an array for some calls with arity > 1 2021-11-04 15:03:57 +01:00
Eelco Dolstra
bcf4780006 Add level / displacement types 2021-11-04 15:03:45 +01:00
Eelco Dolstra
81e7c40264 Optimize primop calls
We now parse function applications as a vector of arguments rather
than as a chain of binary applications, e.g. 'substring 1 2 "foo"' is
parsed as

  ExprCall { .fun = <substring>, .args = [ <1>, <2>, <"foo"> ] }

rather than

  ExprApp (ExprApp (ExprApp <substring> <1>) <2>) <"foo">

This allows primops to be called immediately (if enough arguments are
supplied) without having to allocate intermediate tPrimOpApp values.

On

  $ nix-instantiate --dry-run '<nixpkgs/nixos/release-combined.nix>' -A nixos.tests.simple.x86_64-linux

this gives a substantial performance improvement:

  user CPU time:      median =      0.9209  mean =      0.9218  stddev =      0.0073  min =      0.9086  max =      0.9340  [rejected, p=0.00000, Δ=-0.21433±0.00677]
  elapsed time:       median =      1.0585  mean =      1.0584  stddev =      0.0024  min =      1.0523  max =      1.0623  [rejected, p=0.00000, Δ=-0.20594±0.00236]

because it reduces the number of tPrimOpApp allocations from 551990 to
42534 (i.e. only small minority of primop calls are partially
applied) which in turn reduces time spent in the garbage collector.
2021-11-04 15:03:40 +01:00
Eelco Dolstra
ab35cbd675 StaticEnv: Use std::vector instead of std::map 2021-11-04 15:03:34 +01:00
Eelco Dolstra
c4bd6a15c2 Add helper function to check whether a function arg is 'X' or '_X'
Also allow '_'.
2021-11-04 14:52:35 +01:00
Jan Tojnar
e5d4c2235f
docs: Correct fallback user config path
This is in line with XDG Base Directory Specification, where ~/.config is supposed to be used when XDG_CONFIG_HOME is unset.

It also better matches the reality, where ~/.config/nix.conf does not seem to be used.
2021-11-04 11:41:29 +01:00
Eelco Dolstra
c5fd0b46ae
Merge pull request #5486 from mohe2015/bugfixes2
Fix leaking pthread_attr_t
2021-11-04 10:44:33 +01:00
Eelco Dolstra
3f447bcd5f
Merge pull request #5488 from JanCVanB/patch-1
Reword "we"s to "I"s for consistency
2021-11-04 10:41:35 +01:00
Jan Van Bruggen
7d56174c1e
Reword "we"s to "I"s for consistency
This script uses multiple forms of the first-person POV:
1. "We" to refer to the Nix team (1e7c796e66/scripts/install-multi-user.sh (L72))
2. "We" to refer to the combination of the installation script & the user/executor (1e7c796e66/scripts/install-multi-user.sh (L710))
3. "We" to refer to the installation script alone (1e7c796e66/scripts/install-multi-user.sh (L602))
4. "I" to refer to the installation script alone (1e7c796e66/scripts/install-multi-user.sh (L200))

Since I prefer POV 4 to POV 3, this changes all instances of POV 3 to POV 4.
2021-11-03 18:42:32 -06:00
Moritz Hedtke
6f291ed718
Fix leaking pthread_attr_t
pthread_attr_destroy was not called.
2021-11-03 22:54:16 +01:00
Eelco Dolstra
1e7c796e66
Merge pull request #5475 from doronbehar/SQLiteWAL-vfs
libstore: Use unix-dotfile vfs if useSQLiteWAL is false
2021-11-03 21:20:27 +01:00
Eelco Dolstra
ae14113969
Merge pull request #5477 from league/check-overlay-args
In checkOverlay, accept underscored names for final/prev args.
2021-11-03 21:18:58 +01:00
Domen Kožar
f1c9ee0364
Merge pull request #5480 from rex4539/typos
Fix typos
2021-11-03 11:07:19 -06:00
Dimitris Apostolou
c34cc5e488
Fix typos 2021-11-03 18:11:20 +02:00
Christopher League
3f070cc417 In checkOverlay, accept underscored names for final/prev args.
Resolves #4416.
2021-11-03 09:25:27 -04:00
Doron Behar
14fcf17277 libstore: Use unix-dotfile vfs if useSQLiteWAL is false 2021-11-03 14:19:11 +02:00
Eelco Dolstra
133905b309
Merge pull request #5471 from simon04/patch-1
command-ref/nix-shell: fix --pure, --keep
2021-11-03 11:15:13 +01:00
Simon Legner
1968760f4a
command-ref/nix-shell: fix --pure, --keep 2021-11-02 22:00:08 +01:00
Eelco Dolstra
886ad0055f
Merge pull request #5459 from andersk/echo-e
installer: Do not use echo -e in #!/bin/sh script
2021-11-02 20:00:43 +01:00
Anders Kaseorg
447350fe0e installer: Do not use echo -e in #!/bin/sh script
ShellCheck correctly warns:

In scripts/install-nix-from-closure.sh line 218:
                echo -e "\nif [ -e $p ]; then . $p; fi # added by Nix installer" >> "$fn"
                     ^-- SC3037: In POSIX sh, echo flags are undefined.

In scripts/install-nix-from-closure.sh line 229:
                echo -e "\nif [ -e $p ]; then . $p; fi # added by Nix installer" >> "$fn"
                     ^-- SC3037: In POSIX sh, echo flags are undefined.

Indeed, this actually breaks on Ubuntu where /bin/sh is dash.

Fixes #5458.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2021-11-02 10:27:27 -07:00
Eelco Dolstra
647baaa151
Merge pull request #5462 from edolstra/remove-source-tarball
Remove references to building from the source tarball
2021-11-02 16:22:45 +01:00
Eelco Dolstra
b61b307bad Remove references to building from the source tarball 2021-11-02 15:29:47 +01:00
Domen Kožar
b8532c9ff1 install-nix-actionv@v14.1 2021-11-02 08:24:17 -06:00