Commit graph

8401 commits

Author SHA1 Message Date
John Ericson
8a3b1b7ced Simplify and document store path installable parsing 2023-01-11 19:08:19 -05:00
John Ericson
ce2f91d356 Split OutputsSpec and ExtendedOutputsSpec, use the former more
`DerivedPath::Built` and `DerivationGoal` were previously using a
regular set with the convention that the empty set means all outputs.
But it is easy to forget about this rule when processing those sets.
Using `OutputSpec` forces us to get it right.
2023-01-11 18:57:18 -05:00
John Ericson
a7c0cff07f Rename OutputPath -> ExtendedOutputPath
Do this prior to making a new more limitted `OutputPath` we will use in
more places.
2023-01-11 18:55:29 -05:00
John Ericson
a8f45b5e5a Improve OutputsSpec slightly
A few little changes preparing for the rest.
2023-01-11 18:54:50 -05:00
Eelco Dolstra
9fc8d00d74 MonitorFdHup: Make it work on macOS again
It appears that on current macOS versions, our use of poll() to detect
client disconnects no longer works. As a workaround, poll() for
POLLRDNORM, since this *will* wake up when the client has
disconnected. The downside is that it also wakes up when input is
available. So just sleep for a bit in that case.  This means that on
macOS, a client disconnect may take up to a second to be detected,
but that's better than not being detected at all.

Fixes #7584.
2023-01-11 10:48:40 -08:00
Eelco Dolstra
7515617ad0 Backport getLine tests from lazy-trees 2023-01-11 13:49:39 +01:00
Théophane Hufschmitt
a3ba80357d
Merge pull request #7543 from obsidiansystems/typed-string-context
Parse string context elements properly
2023-01-11 07:09:37 +01:00
Robert Hensing
f58c301112
Merge pull request #7541 from hercules-ci/check-manual-links
Check links in the manual
2023-01-10 23:07:38 +01:00
Robert Hensing
fefa3a49ce doc/manual: Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-01-10 22:30:41 +01:00
Robert Hensing
e79f935718 doc/manual: Fix broken internal links
The targets I could find.
2023-01-10 22:30:41 +01:00
Robert Hensing
34a1e0d29b doc/manual: Introduce @docroot@ as a stable base for includable snippets
This way the links are clearly within the manual (ie not absolute paths),
while allowing snippets to reference the documentation root reliably,
regardless of at which base url they're included.
2023-01-10 22:30:41 +01:00
John Ericson
5576d5e987 Parse string context elements properly
Prior to this change, we had a bunch of ad-hoc string manipulation code
scattered around. This made it hard to figure out what data model for
string contexts is.

Now, we still store string contexts most of the time as encoded strings
--- I was wary of the performance implications of changing that --- but
whenever we parse them we do so only through the
`NixStringContextElem::parse` method, which handles all cases. This
creates a data type that is very similar to `DerivedPath` but:

 - Represents the funky `=<drvpath>` case as properly distinct from the
   others.

 - Only encodes a single output, no wildcards and no set, for the
   "built" case.

(I would like to deprecate `=<path>`, after which we are in spitting
distance of `DerivedPath` and could maybe get away with fewer types, but
that is another topic for another day.)
2023-01-10 13:10:49 -05:00
John Ericson
da64f026dd Make clear that StorePathWithOutputs is a deprecated type
- Add a comment

- Put `OutputsSpec` in a different header (First part of #6815)

- Make a few stray uses of it in new code use `DerivedPath` instead.
2023-01-10 11:27:19 -05:00
Eelco Dolstra
59cc920cc0 Add a FIXME 2023-01-10 15:20:30 +01:00
Eelco Dolstra
7f1af270dd Clean up toDerivedPaths() logic 2023-01-10 15:08:46 +01:00
Eelco Dolstra
1123c42f90
Apply suggestions from code review
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-01-10 14:57:35 +01:00
Eelco Dolstra
b4dc68f0be Show string in error message 2023-01-10 14:56:03 +01:00
Eelco Dolstra
b80e4b57da ExtraInfo -> ExtraPathInfo 2023-01-10 14:52:49 +01:00
Eelco Dolstra
8e923bf4c5 Merge remote-tracking branch 'origin/master' into fix-7417 2023-01-10 14:35:06 +01:00
Will Bush
05b13aff3d
Fix typo in example for builtin function map 2023-01-06 23:04:43 -06:00
Max
1c35324a97 Merge remote-tracking branch 'nixos/master' 2023-01-06 22:44:20 +01:00
Théophane Hufschmitt
8d88c3b347
Merge pull request #7307 from hercules-ci/derivation-goal-improve-comment
libstore/derivation-goal: Elaborate a TODO for performance concern
2023-01-06 13:07:57 +01:00
Eelco Dolstra
420ccecc1e
Merge pull request #7557 from NixOS/fix-7529
On macOS with auto-uid-allocation and sandboxing, use the correct gid
2023-01-06 12:35:55 +01:00
Naïm Favier
f1ee4ece80
Don't check NixOS modules
NixOS modules can be paths. Rather than dig further down into the layer
violation, don't check anything specific to NixOS modules.
2023-01-05 18:23:30 +01:00
Eelco Dolstra
3a98107170
Merge pull request #7542 from edolstra/gc-deadlock
Fix deadlock between auto-GC and addTempRoot()
2023-01-05 17:08:23 +01:00
Eelco Dolstra
0fe2b222d5
Merge pull request #7539 from tweag/fix-nix-why-depends--derivation
Fix `nix why-depends --derivation`
2023-01-05 15:32:04 +01:00
Eelco Dolstra
4e84b532ed On macOS with auto-uid-allocation and sandboxing, use the correct gid
macOS doesn't have user namespacing, so the gid of the builder needs
to be nixbld. The logic got "has sandboxing enabled" confused with
"has user namespaces".

Fixes #7529.
2023-01-05 04:58:55 -08:00
Eelco Dolstra
6991e558dd Move macOS sandbox files to sr/libstore/build 2023-01-04 04:50:45 -08:00
Eelco Dolstra
609a7dc059 Include macOS sandbox files in the Nix binary
This basically reverts 6e5165b773.
It fixes errors like

  sandbox-exec: <internal init prelude>:292:47: unable to open sandbox-minimal.sb: not found

when trying to run a development Nix installed in a user's home
directory.

Also, we're trying to minimize the number of installed files
to make it possible to deploy Nix as a single statically-linked
binary.
2023-01-04 04:36:07 -08:00
Eelco Dolstra
d4d1ca8b11 nix --version: Print the data directory 2023-01-03 08:30:49 -08:00
Eelco Dolstra
28d5b5cd45 Fix deadlock between auto-GC and addTempRoot()
Previously addTempRoot() acquired the LocalStore state lock and waited
for the garbage collector to reply. If the garbage collector is in the
same process (as it the case with auto-GC), this would deadlock as
soon as the garbage collector thread needs the LocalStore state lock.

So now addTempRoot() uses separate Syncs for the state that it
needs. As long at the auto-GC thread doesn't call addTempRoot() (which
it shouldn't), it shouldn't deadlock.

Fixes #3224.
2023-01-03 15:20:21 +01:00
Eelco Dolstra
224b56f10e Move creation of the temp roots file into its own function
This also moves the file handle into its own Sync object so we're not
holding the _state while acquiring the file lock. There was no real
deadlock risk here since locking a newly created file cannot block,
but it's still a bit nicer.
2023-01-03 14:51:23 +01:00
Eelco Dolstra
ae31b5f50f
Merge pull request #7497 from rski/master
src/libstore: Print the reason opening the DB failed
2023-01-03 12:44:14 +01:00
Eelco Dolstra
d33d15a48b Put the --show-trace hint in the logical place 2023-01-02 20:53:58 +01:00
Eelco Dolstra
c548e35498 Don't use state.positions[noPos]
This caused traces 'at «none»:0: (source not available)'.
2023-01-02 20:53:58 +01:00
Eelco Dolstra
6b69652385 Merge remote-tracking branch 'origin/master' into coerce-string 2023-01-02 20:53:39 +01:00
Théophane Hufschmitt
8cac451fce Fix why-depends for CA derivations (again)
This has the same goal as b13fd4c58e81b2b2b0d72caa5ce80de861622610,but
achieves it in a different way in order to not break
`nix why-depends --derivation`.
2023-01-02 17:42:22 +01:00
Théophane Hufschmitt
105d74eb81 Revert "Fix why-depends for CA derivations"
This reverts commit b13fd4c58e.
2023-01-02 15:44:04 +01:00
Théophane Hufschmitt
9af16c5f74
Merge pull request #5941 from hercules-ci/optimize-intersectAttrs
Optimize intersectAttrs performance
2023-01-02 15:22:38 +01:00
Valentin Gagarin
e0c4a95611 antiquotation -> string interpolation
as proposed by @mkaito[1] and @tazjin[2] and discussed with @edolstra
and Nix maintainers

[1]: https://github.com/NixOS/nix.dev/pull/267#issuecomment-1270076332
[2]: https://github.com/NixOS/nix.dev/pull/267#issuecomment-1270201979

Co-authored-by: John Ericson <git@JohnEricson.me>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-01-02 14:38:57 +01:00
Théophane Hufschmitt
fb8fc6fda6
Merge pull request #7478 from hercules-ci/make-sure-initNix-called
libstore: Make sure that initNix has been called
2023-01-02 14:12:49 +01:00
Valentin Gagarin
9cdf8ededb remove redundant re-definition of store derivations 2023-01-02 13:37:59 +01:00
Théophane Hufschmitt
cfd6c7fc9b
Merge pull request #7485 from fricklerhandwerk/doc-store-derivation
define "store derivation"
2023-01-02 13:26:41 +01:00
Théophane Hufschmitt
b3285c7722
Merge pull request #7351 from NaN-git/fix-mkString
cleanup eval.hh/eval.cc
2023-01-02 11:41:52 +01:00
Théophane Hufschmitt
e8a3e58171
Merge pull request #7521 from ncfavier/migration-deadlock
Release shared lock before acquiring exclusive lock
2023-01-02 11:08:43 +01:00
Steven Shaw
84b0893725
Fix error message 2023-01-01 12:37:43 +10:00
Eelco Dolstra
8c52f8ea9d
Merge pull request #7524 from ncfavier/sandbox-paths-closure
doc: sandbox-paths computes closures
2022-12-29 19:45:51 +01:00
Eelco Dolstra
8227fe819e
Merge pull request #7504 from edolstra/nix-develop-personality
nix develop: Set personality
2022-12-29 19:42:45 +01:00
Naïm Favier
d5d2f50ebb
doc: sandbox-paths computes closures 2022-12-28 17:09:20 +01:00
Naïm Favier
81c3f99b36
Release shared lock before acquiring exclusive lock
In principle, this should avoid deadlocks where two instances of Nix are
holding a shared lock on big-lock and are both waiting to get an
exclusive lock.

However, it seems like `flock(2)` is supposed to do this automatically,
so it's not clear whether this is actually where the problem comes from.
2022-12-27 15:58:14 +01:00
Max
8cc8551afd Merge remote-tracking branch 'nixos/master' 2022-12-24 16:39:35 +01:00
Robert Hensing
336908cf4c Optimize intersectAttrs performance
Always traverse the shortest set.
2022-12-24 14:51:05 +01:00
Robert Hensing
aba6eb348e libstore: Make sure that initNix has been called
Prevent bugs like https://github.com/cachix/cachix/pull/477
2022-12-24 14:39:30 +01:00
Philipp Otterbein
8af839f48c remove undefined function 2022-12-24 12:19:53 +01:00
Philipp Otterbein
a6e9d9cb2f remove function makeImmutableStringWithLen 2022-12-24 12:09:06 +01:00
Eelco Dolstra
c164d304f3 nix develop: Set personality
This makes 'nix develop' set the Linux personality in the same way
that the actual build does, allowing a command like 'nix develop
nix#devShells.i686-linux.default' on x86_64-linux to work correctly.
2022-12-23 16:33:55 +01:00
Eelco Dolstra
14f7dae3e4
Merge pull request #7503 from edolstra/fix-dirOf
Fix CanonPath::dirOf() returning a string_view of a temporary
2022-12-23 16:17:48 +01:00
Eelco Dolstra
64c60f7241 Fix CanonPath::dirOf() returning a string_view of a temporary
https://hydra.nixos.org/build/202837872
2022-12-23 15:32:54 +01:00
rski
d034ed1891 src/libstore: Print the reason opening the DB failed
Without this, the error is lost, and it makes for a hard to debug
situation. Also remove some of the busyness inside the sqlite_open_v2
args.

The errcode returned is not the extended one. The only way to make open
return an extended code, would be to add SQLITE_OPEN_EXRESCODE to the
flags. In the future it might be worth making this change,
which would also simplify the existing SQLiteError code.
2022-12-23 02:55:51 +02:00
Eelco Dolstra
c9eee5a84d
Merge pull request #7493 from mupdt/primops-storeDir-test-non-standard-path
primops `storeDir` test uses `settings.nixStore`
2022-12-22 16:18:48 +01:00
mupdt
a33e45b60b primops storeDir test uses settings.nixStore 2022-12-21 07:01:57 -05:00
Valentin Gagarin
7797661a70 link "store derivation" to glossary definition 2022-12-21 11:42:50 +01:00
Valentin Gagarin
3a66d82e1d update description of "store derivation" in installables section
a store derivation is not a store path itself, it has a store path.
2022-12-21 11:42:22 +01:00
mupdt
bc8ab21c5a [PDT] TDE-3114: prevent a race-condition when creating the S3 cache 2022-12-21 04:50:40 -05:00
Eelco Dolstra
5c97b5a398 InstallableFlake::toDerivedPaths(): Support paths and store paths
This makes 'nix build' work on paths (which will be copied to the
store) and store paths (returned as is). E.g. the following flake
output attributes can be built using 'nix build .#foo':

  foo = ./src;
  foo = self.outPath;
  foo = builtins.fetchTarball { ... };
  foo = (builtins.fetchTree { .. }).outPath;
  foo = builtins.fetchTree { .. } + "/README.md";
  foo = builtins.storePath /nix/store/...;

Note that this is potentially risky, e.g.

  foo = /.;

will cause Nix to try to copy the entire file system to the store.

What doesn't work yet:

  foo = self;
  foo = builtins.fetchTree { .. };

because we don't handle attrsets with an outPath attribute in it yet,
and

  foo = builtins.storePath /nix/store/.../README.md;

since result symlinks have to point to a store path currently (rather
than a file inside a store path).

Fixes #7417.
2022-12-20 15:11:44 +01:00
Eelco Dolstra
bda879170f EvalState::copyPathToStore(): Return a StorePath 2022-12-20 14:58:39 +01:00
Eelco Dolstra
845fc3f605 Merge toDerivations() into toDerivedPaths()
toDerivedPaths() now returns DerivedPathWithInfo, which is DerivedPath
with some attributes needed by 'nix profile' etc.

Preparation for #7417.
2022-12-20 14:24:14 +01:00
Eelco Dolstra
8332ac6a1d
Merge pull request #7451 from edolstra/abstract-pos
Introduce AbstractPos
2022-12-20 12:55:13 +01:00
Théophane Hufschmitt
000dd77d8d
Merge pull request #7473 from hercules-ci/sqlite-error
Improve sqlite error messages
2022-12-20 11:27:30 +01:00
Théophane Hufschmitt
c3d522dc51
Merge pull request #7480 from nrdxp/fix-develop-drv
fix(develop): make `nix develop` drv recreatable
2022-12-20 07:41:46 +01:00
Timothy DeHerrera
94cf0da7b2
fix(develop): make nix develop drv recreatable 2022-12-19 13:16:06 -07:00
Domen Kožar
f118e661e0
Merge pull request #7462 from rapenne-s/improve_messaging_reusing_trusted_substituter
Inform user instead of warning them when using a trusted substituter
2022-12-19 16:04:01 +00:00
Eelco Dolstra
d00bfe4833
Merge pull request #7450 from edolstra/canon-path
Add CanonPath wrapper to represent canonicalized paths
2022-12-19 16:21:11 +01:00
Max
8615547a51 fix incorrect merge 2022-12-17 23:00:58 +01:00
Robert Hensing
c965f35de7 Improve sqlite error messages
They did not include the detailed error message, losing essential
information for troubleshooting.

Example message:

    warning: creating statement 'insert or rplace into NARs(cache, hashPart, namePart, url, compression, fileHash, fileSize, narHash, narSize, refs, deriver, sigs, ca, timestamp, present) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1)': at offset 10: SQL logic error, near "rplace": syntax error (in '/tmp/nix-shell.grQ6f7/nix-test/tests/binary-cache/test-home/.cache/nix/binary-cache-v6.sqlite')

It's not the best example; more important information will be in
the message for e.g. a constraint violation.

I don't see why this specific error is printed as a warning, but
that's for another commit.
2022-12-17 14:51:37 +01:00
Max
74ecb58b74 Merge remote-tracking branch 'nixos/master' 2022-12-17 13:36:27 +01:00
Solène Rapenne
d0660c6c0b
printMsg replacement by printInfo
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-12-16 09:34:22 +01:00
Taeer Bar-Yam
e5eb05c599 getBuildLog: factor out resolving derivations 2022-12-15 15:58:54 -05:00
Eelco Dolstra
0687e16c4a Fix a crash in DerivedPath::Built::toJSON() with impure derivations
The use of 'nullptr' here didn't result in a null JSON value, but in a
nullptr being cast to a string, which aborts.
2022-12-15 16:02:27 +01:00
Solène Rapenne
98e01da0b1 warnings: switch to info level when using a saved substituter 2022-12-14 14:11:21 +01:00
Naïm Favier
1f3c0a3c1d
Allow disabling build users by unsetting build-users-group
Unsetting `build-users-group` (without `auto-allocate-uids` enabled)
gives the following error:

```
src/libstore/lock.cc:25: static std::unique_ptr<nix::UserLock> nix::SimpleUserLock::acquire(): Assertion `settings.buildUsersGroup != ""' failed.
```

Fix the logic in `useBuildUsers` and document the default value
for `build-users-group`.
2022-12-14 00:40:30 +01:00
Eelco Dolstra
46b3c026fc
Merge pull request #7455 from rapenne-s/documentation_fix
Add anchors and links to definition
2022-12-13 17:26:52 +01:00
Eelco Dolstra
e2a4e7aecd
Merge pull request #4543 from obsidiansystems/indexed-store-path-outputs
Low level `<drvPath>^<outputName>` installable syntax to match existing `<highLevelInstallable>^<outputNames>` syntax
2022-12-13 17:22:29 +01:00
Eelco Dolstra
c9b0a85b08 Restore display of source lines for stdin/string inputs 2022-12-13 16:00:44 +01:00
Eelco Dolstra
aea97f07a3 Fix compilation 2022-12-13 15:23:12 +01:00
Eelco Dolstra
1315133b50
Improve cast safety
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-12-13 12:38:33 +01:00
Solène Rapenne
09860c16ce documentation: use sections instead of list items 2022-12-13 11:47:44 +01:00
Solène Rapenne
e43b0f5b12 documentation: link flake URL term to definition 2022-12-13 11:47:38 +01:00
Solène Rapenne
ae27181f16 documentation: fix link to definition 2022-12-13 11:47:22 +01:00
Théophane Hufschmitt
6e31d27cba
Merge pull request #6741 from Mindavi/nix-no-gc
support building with --enable-gc=no
2022-12-13 10:36:59 +01:00
Eelco Dolstra
b3fdab28a2 Introduce AbstractPos
This makes the position object used in exceptions abstract, with a
method getSource() to get the source code of the file in which the
error originated. This is needed for lazy trees because source files
don't necessarily exist in the filesystem, and we don't want to make
libutil depend on the InputAccessor type in libfetcher.
2022-12-13 00:50:43 +01:00
John Ericson
f61d575810 Merge branch 'indexed-store-path-outputs' of github.com:obsidiansystems/nix into indexed-store-path-outputs 2022-12-12 17:43:10 -05:00
John Ericson
5273cf4c97 Merge remote-tracking branch 'upstream/master' into indexed-store-path-outputs 2022-12-12 17:40:49 -05:00
John Ericson
32ae715db1
Fix typos in the docs
Thanks!

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2022-12-12 17:37:45 -05:00
John Ericson
dabb03b8d0 Merge remote-tracking branch 'upstream/master' into indexed-store-path-outputs 2022-12-12 17:36:02 -05:00
John Ericson
d8c1c24c78 Adjust docs 2022-12-12 17:32:24 -05:00
Rick van Schijndel
672ee88231 support building with --enable-gc=no
Some minor changes fixing the build without boehm.
Fixes NixOS#6250
2022-12-12 23:31:30 +01:00
John Ericson
dc075dcdd0
Apply suggestions from code review
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2022-12-12 16:26:10 -05:00
Eelco Dolstra
900b854084 Add CanonPath wrapper to represent canonicalized paths 2022-12-12 19:57:32 +01:00
Florian Friesdorf
8618c6cc75 Simplify loop, feedback from @tfc and @Ericson2314 2022-12-12 18:41:00 +00:00