Commit graph

11999 commits

Author SHA1 Message Date
Théophane Hufschmitt
7117053457
Merge pull request #6270 from Artturin/stdineval
nix: allow using --file - to read from stdin
2022-03-17 14:35:01 +01:00
Théophane Hufschmitt
3fc4c612fb Fix nix build --dry-run with CA derivations
Don’t try and assume that we know the output paths when we’ve just built
with `--dry-run`. Instead make `--dry-run` follow a different code path
that won’t assume the knowledge of the output paths at all.

Fix #6275
2022-03-17 11:55:19 +01:00
Théophane Hufschmitt
a0b517de57
Merge pull request #6242 from ncfavier/print-output-names
nix-env: always print output names in JSON and XML
2022-03-17 10:55:22 +01:00
Théophane Hufschmitt
c20e07763d
Add some tests for nix-env -q --json 2022-03-16 21:33:38 +01:00
Naïm Favier
8dcecc0738
nix-env: print a final newline after JSON 2022-03-16 21:26:19 +01:00
Naïm Favier
5736661922
nix-env: always print output names in JSON and XML
The current `--out-path` flag has two disadvantages when one is only
concerned with querying the names of outputs:
- it requires evaluating every output's `outPath`, which takes
  significantly more resources and runs into more failures
- it destroys the information of the order of outputs so we can't tell
  which one is the main output

This patch makes the output names always present (replacing paths with
`null` in JSON if `--out-path` isn't given), and adds an `outputName`
field.
2022-03-16 21:26:19 +01:00
Théophane Hufschmitt
fe1ad96135
Merge pull request #6267 from nomeata/patch-1
distributed-builds.md: fixing typo of the most minor sort
2022-03-16 21:18:14 +01:00
Artturin
4f8ad41d4e add tests for nix eval and nix-instantiate 2022-03-16 21:01:51 +02:00
Artturin
a5c969db49 nix: allow using --file - to read from stdin 2022-03-16 21:01:51 +02:00
Joachim Breitner
af013281c9
distributed-builds.md: fixing typo of the most minor sort 2022-03-16 12:53:38 +01:00
Eelco Dolstra
a53c1dc96d
Merge pull request #6245 from trofi/fix-nix-store-gc-limit-auto-optimise-store
nix store gc: account for auto-optimised store
2022-03-16 09:19:57 +01:00
Eelco Dolstra
7b0eb9eaf2
Merge pull request #6264 from thufschmitt/fix-32bits-tests
Fix the tests on 32bits machines
2022-03-16 09:13:23 +01:00
Théophane Hufschmitt
3cea6f569e
Fix the date in the comment of fetchPath’s test
Co-authored-by: pennae <82953136+pennae@users.noreply.github.com>
2022-03-16 08:56:01 +01:00
Théophane Hufschmitt
516a7ac4de
Merge pull request #6229 from obsidiansystems/refactor-hash-modulo
Overhaul derivation hash modulo somewhat
2022-03-15 21:23:44 +01:00
regnat
2d5c43f210 Fix the tests on 32bits machines
year 2222 is too much for a 32 bit timestamp. So replace it by something
smaller
2022-03-15 21:05:01 +01:00
Théophane Hufschmitt
ba9e69cdcd
Merge pull request #6119 from Ma27/path-mtime
libfetchers/path: set `lastModified` to path's mtime
2022-03-15 17:51:16 +01:00
Maximilian Bosch
975bade7f0
Implement simple test for path-fetcher setting a correct lastModifiedDate
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-03-15 12:55:32 +01:00
Maximilian Bosch
244baff2c7
libfetchers: remove obsolete filesystem #include 2022-03-15 12:32:11 +01:00
Maximilian Bosch
34e20c164c
libfetchers/path: set lastModified to path's mtime
When importing e.g. a local `nixpkgs` in a flake to test a change like

    {
      inputs.nixpkgs.url = path:/home/ma27/Projects/nixpkgs;
      outputs = /* ... */
    }

then the input is missing a `lastModified`-field that's e.g. used in
`nixpkgs.lib.nixosSystem`. Due to the missing `lastMoified`-field, the
mtime is set to 19700101:

    result -> /nix/store/b7dg1lmmsill2rsgyv2w7b6cnmixkvc1-nixos-system-nixos-22.05.19700101.dirty

With this change, the `path`-fetcher now sets a `lastModified` attribute
to the `mtime` just like it's the case in the `tarball`-fetcher already.
When building NixOS systems with `nixpkgs` being a `path`-input and this
patch, the output-path now looks like this:

    result -> /nix/store/ld2qf9c1s98dxmiwcaq5vn9k5ylzrm1s-nixos-system-nixos-22.05.20220217.dirty
2022-03-15 12:32:11 +01:00
Eelco Dolstra
a61809722f
Merge pull request #6244 from Artturin/nixenvbettermessage
nix-env: Add a suggestion for when there's a name collision in channels
2022-03-14 14:17:50 +01:00
Théophane Hufschmitt
0e86ebf461
Merge pull request #6220 from obsidiansystems/log-store
Factor out a `LogStore` interface
2022-03-14 10:51:01 +01:00
Eelco Dolstra
edfd94c486
Merge pull request #6248 from thkoch2001/patch-1
Add documentation= entry to systemd unit file
2022-03-14 10:34:18 +01:00
thkoch2001
e06b264f94
Add documentation= entry to systemd unit file
Closes: #6246
2022-03-13 18:42:26 +02:00
Sergei Trofimovich
6b1872312f nix store gc: account for auto-optimised store
Before the change on a system with `auto-optimise-store = true`:

    $ nix store gc --verbose --max 1

deleted all the paths instead of one path (we requested 1 byte limit).

It happens because every file in `auto-optimise-store = true` has at
least 2 links: file itself and a link in /nix/store/.links/ directory.

The change conservatively assumes that any file that has one (as before)
or two links (assume auto-potimise mode) will free space.

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
2022-03-13 07:24:48 +00:00
Artturin
cb1a76112e nix-env: Add a suggestion for when there's a name collision in channels
help new users find a solution to their problem

./result/bin/nix-env -qa hello
warning: name collision in input Nix expressions, skipping '/home/artturin/.nix-defexpr/channels_root/master'
suggestion: remove 'master' from either the root channels or the user channels
hello-2.12
hello-2.12
2022-03-13 00:24:46 +02:00
John Ericson
91adfb8894 Create some type aliases for string Contexts 2022-03-11 22:30:10 +00:00
John Ericson
0948b8e94d Reduce variants for derivation hash modulo
This changes was taken from dynamic derivation (#4628). It` somewhat
undoes the refactors I first did for floating CA derivations, as the
benefit of hindsight + requirements of dynamic derivations made me
reconsider some things.

They aren't to consequential, but I figured they might be good to land
first, before the more profound changes @thufschmitt has in the works.
2022-03-11 21:20:37 +00:00
Eelco Dolstra
d5322698a2
Merge pull request #6230 from obsidiansystems/nix-build-derived-path
Desugar `StorePathWithOutputs` in nix-build implementation
2022-03-11 22:19:56 +01:00
John Ericson
015d2ad507 Desugar StorePathWithOutputs in nix-build implementation
`DerivedPath` has replaced `StorePathWithOutputs` internally, so
shrinking the usage of `StorePathWithOutputs` to just the boundary is
good.
2022-03-11 20:50:52 +00:00
Kevin Amado
2191dab657
nix-fmt: add command 2022-03-11 10:00:19 -05:00
John Ericson
a03b1fd7f6 Deduplicate the Store downcasting with a template 2022-03-11 13:32:33 +00:00
John Ericson
678d1c2aa0 Factor out a LogStore interface
Continue progress on #5729.

Just as I hoped, this uncovered an issue: the daemon protocol is missing
a way to query build logs. This doesn't effect `unix://`, but does
effect `ssh://`. A FIXME is left for this, so we come back to it later.
2022-03-11 13:32:16 +00:00
John Ericson
89effe9d4a GcStore::resolve should print the URI 2022-03-11 13:27:38 +00:00
John Ericson
073e134de6 Rename requireGcStore to GcStore::require
I should have done this to begin with. This will be nicer once more
Store sub-interfaces exist too, to illustrate the pattern.
2022-03-11 13:27:38 +00:00
Eelco Dolstra
aee56e0f89 Merge remote-tracking branch 'origin/eval-suggestions' 2022-03-11 12:02:26 +01:00
Eelco Dolstra
31a392dfe2 Merge pull request #5865 from pennae/memory-friendliness
be more memory friendly
2022-03-11 11:52:39 +01:00
Eelco Dolstra
167766b65c Style 2022-03-11 11:19:21 +01:00
Théophane Hufschmitt
7ebd6f1093
Merge pull request #5758 from mschwaig/fix-git-workspace-dirty-detection
git fetcher: fix detection of dirty git workspaces
2022-03-11 09:35:01 +01:00
Eelco Dolstra
4d98143914 BuildResult: Remove unused drvPath field 2022-03-09 20:31:50 +01:00
Eelco Dolstra
1c1a7074da
Merge pull request #6221 from NixOS/build-paths-with-results
Add Store::buildPathsWithResults()
2022-03-09 14:37:14 +01:00
Eelco Dolstra
761242afa0 BuildResult: Use DerivedPath 2022-03-09 12:25:35 +01:00
pennae
4d629c4f7a add HAVE_BOEHMGC guards to batched allocation functions 2022-03-09 00:18:50 +01:00
pennae
8e2eaaaf69 make Finally more local
no need for function<> with c++17 deduction. this saves allocations and virtual
calls, but has the same semantics otherwise. not going through function has the
side effect of giving compilers more insight into the cleanup code, so we need a
few local warning disables.
2022-03-09 00:16:50 +01:00
pennae
47baa9d43c make Pos smaller
reduces peak hep memory use on eval of our test system from 264.4MB to 242.3MB,
possibly also a slight performance boost.

theoretically memory use could be cut down by another eight bytes per Pos on
average by turning it into a tuple containing an index into a global base
position table with row and column offsets, but that doesn't seem worth the
effort at this point.
2022-03-08 23:30:18 +01:00
pennae
c96460f352 force-inline a few much-used functions
these functions are called a whole lot, and they're all comparatively small.
always inlining them gives ~0.7% performance boost on eval.

before:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.935 s ±  0.052 s    [User: 5.852 s, System: 0.853 s]
    Range (min … max):    6.808 s …  7.026 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     329.8 ms ±   2.7 ms    [User: 299.0 ms, System: 30.8 ms]
    Range (min … max):   326.6 ms … 336.5 ms    20 runs

  Benchmark 3: nix flakes eval --raw --impure --file expr.nix
    Time (mean ± σ):      2.655 s ±  0.038 s    [User: 2.364 s, System: 0.220 s]
    Range (min … max):    2.574 s …  2.737 s    20 runs

after:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.912 s ±  0.036 s    [User: 5.823 s, System: 0.856 s]
    Range (min … max):    6.849 s …  6.980 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     325.1 ms ±   2.5 ms    [User: 293.2 ms, System: 31.8 ms]
    Range (min … max):   322.2 ms … 332.8 ms    20 runs

  Benchmark 3: nix flakes eval --raw --impure --file expr.nix
    Time (mean ± σ):      2.636 s ±  0.024 s    [User: 2.352 s, System: 0.226 s]
    Range (min … max):    2.574 s …  2.681 s    20 runs
2022-03-08 23:30:18 +01:00
pennae
60ed4e908a cache singleton Envs just like Values
vast majority of envs is this size.

before:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.946 s ±  0.041 s    [User: 5.875 s, System: 0.835 s]
    Range (min … max):    6.834 s …  7.005 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     330.3 ms ±   2.5 ms    [User: 299.2 ms, System: 30.9 ms]
    Range (min … max):   327.5 ms … 337.7 ms    20 runs

  Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.671 s ±  0.035 s    [User: 2.370 s, System: 0.232 s]
    Range (min … max):    2.597 s …  2.749 s    20 runs

after:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.935 s ±  0.052 s    [User: 5.852 s, System: 0.853 s]
    Range (min … max):    6.808 s …  7.026 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     329.8 ms ±   2.7 ms    [User: 299.0 ms, System: 30.8 ms]
    Range (min … max):   326.6 ms … 336.5 ms    20 runs

  Benchmark 3: nix flakes eval --raw --impure --file expr.nix
    Time (mean ± σ):      2.655 s ±  0.038 s    [User: 2.364 s, System: 0.220 s]
    Range (min … max):    2.574 s …  2.737 s    20 runs
2022-03-08 23:30:18 +01:00
pennae
4b2b0d3a55 remove GC_PTR_STORE_AND_DIRTY
turns out it's only necessary for MANUAL_VDB, which nix doesn't use. omitting
them gives a slight performance improvement on eval.

before:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.988 s ±  0.061 s    [User: 5.935 s, System: 0.845 s]
    Range (min … max):    6.865 s …  7.075 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     332.6 ms ±   3.9 ms    [User: 299.6 ms, System: 32.9 ms]
    Range (min … max):   328.1 ms … 339.1 ms    20 runs

  Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.681 s ±  0.049 s    [User: 2.382 s, System: 0.228 s]
    Range (min … max):    2.607 s …  2.776 s    20 runs

after:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.946 s ±  0.041 s    [User: 5.875 s, System: 0.835 s]
    Range (min … max):    6.834 s …  7.005 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     330.3 ms ±   2.5 ms    [User: 299.2 ms, System: 30.9 ms]
    Range (min … max):   327.5 ms … 337.7 ms    20 runs

  Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.671 s ±  0.035 s    [User: 2.370 s, System: 0.232 s]
    Range (min … max):    2.597 s …  2.749 s    20 runs
2022-03-08 23:30:18 +01:00
Eelco Dolstra
a4604f1928 Add Store::buildPathsWithResults()
This function is like buildPaths(), except that it returns a vector of
BuildResults containing the exact statuses and output paths of each
derivation / substitution. This is convenient for functions like
Installable::build(), because they then don't need to do another
series of calls to get the outputs of CA derivations. It's also a
precondition to impure derivations, where we *can't* query the output
of those derivations since they're not stored in the Nix database.

Note that PathSubstitutionGoal can now also return a BuildStatus.
2022-03-08 19:56:34 +01:00
regnat
f6078e474d Also display some suggestions for invalid formal arguments
```console
$ nix eval --expr '({ foo ? 1 }: foo) { fob = 2; }'
error: anonymous function at (string):1:2 called with unexpected argument 'fob'

       at «string»:1:1:

            1| ({ foo ? 1 }: foo) { fob = 2; }
             | ^
       Did you mean foo?
```

Not that because Nix will first check for _missing_ arguments before
checking for extra arguments, `({ foo }: foo) { fob = 1; }` will
complain about the missing `foo` argument (rather than extra `fob`) and
so won’t display a suggestion.
2022-03-08 16:40:22 +01:00
regnat
33b7514035 Try and make the darwin build happy 2022-03-08 16:07:17 +01:00