Commit graph

9647 commits

Author SHA1 Message Date
Eelco Dolstra
ee36a44bf2 GitInputScheme: Use libgit2
This replaces most calls to the "git" binary with libgit2.
2023-10-31 14:52:21 +01:00
Eelco Dolstra
1d0e3d84b6 Provide a InputScheme::fetch() built on top of InputScheme::getAccessor()
This is for graceful migration to lazy-trees fetchers (which are all
accessor-based). Eventually fetch() will be removed.
2023-10-31 14:52:17 +01:00
Eelco Dolstra
e1b8442fa1 Fetcher cache: Add support for caching facts not related to store paths 2023-10-31 14:51:39 +01:00
Eelco Dolstra
fa6bc33604
Merge pull request #9239 from edolstra/putFile
Input: Replace markFileChanged() by putFile()
2023-10-31 14:49:42 +01:00
Eelco Dolstra
05316d401f Cleanup 2023-10-30 17:03:06 +01:00
John Ericson
077de2968e Include fetcher input scheme info in the CLI dump
Leverages the previous commit.
2023-10-30 10:30:59 -04:00
John Ericson
8381eeda6f Systematize fetcher input attribute validation
We now have `schemeName` and `allowedAttrs` functions for this purpose.
We look up the schema with the former; we restrict the set of input
attributes with the latter.
2023-10-30 10:17:24 -04:00
John Ericson
a419b61497 Turn derivation unit tests into unit characterization tests
The brings a number of advantages, including:

- Easier to update test data if design changes (and I do think our
  derivation JSON is not yet complaint with the guidelines).

- Easier to reuse test data in other implementations, inching closer to
  compliance tests for Nix *the concept* rather than any one
  implementation.
2023-10-26 18:09:01 -04:00
Sergei Trofimovich
e69c764708 local-derivation-goal.cc: slightly clarify waiting message
Before the change builder ID exhaustion printed the following message:

    [0/1 built] waiting for UID to build '/nix/store/hiy9136x0iyib4ssh3w3r5m8pxjnad50-python3.11-breathe-4.35.0.drv'

After the change it should be:

    [0/1 built] waiting for a free build user ID for '/nix/store/hiy9136x0iyib4ssh3w3r5m8pxjnad50-python3.11-breathe-4.35.0.drv'
2023-10-26 20:54:21 +01:00
John Ericson
1dc6a65d36
Merge pull request #9238 from tfc/small-improvements2
Small improvements 2
2023-10-26 09:35:03 -04:00
Jacek Galowicz
28c39c370c Provide default value for id to fix warning 2023-10-26 09:47:40 +02:00
Jacek Galowicz
b66381e8d8 Use using instead of typedef 2023-10-26 09:47:40 +02:00
Silvan Mosberger
46028ff764
doc: Fix fetchGit default name (#9241) 2023-10-26 07:05:48 +02:00
Eelco Dolstra
622191c2b5
Merge pull request #8965 from Artturin/bindfilesinchroot
Bindmount files instead of hardlinking or copying to chroot
2023-10-25 19:10:03 +02:00
Eelco Dolstra
15c430f389 Remove unused LockFile::write() 2023-10-25 18:44:09 +02:00
Eelco Dolstra
95d657c8b3 Input: Replace markFileChanged() by putFile()
Committing a lock file using markFileChanged() required the input to
be writable by the caller in the local filesystem (using the path
returned by getSourcePath()). putFile() abstracts over this.
2023-10-25 18:30:29 +02:00
John Ericson
5ac87a75dd
Merge pull request #9229 from tfc/small-improvements
Remove warnings, small improvements
2023-10-25 10:50:46 -04:00
Jacek Galowicz
f555c98a34 Improve loop over gid container 2023-10-25 16:10:35 +02:00
Felix Uhl
7bc45c6136 docs: clarify flake types and implied defaults 2023-10-25 15:10:35 +02:00
Bouke van der Bijl
1d28d613b1 config: add included files into parsedContents before applying
Fixes #8719
2023-10-25 11:39:18 +02:00
Eelco Dolstra
3ca0dade5b
Merge pull request #9212 from trofkm/args-refactoring
Clean up `args.hh`
2023-10-25 10:33:53 +02:00
Jacek Galowicz
b113d925de Fix warning 2023-10-24 19:57:38 +02:00
Jacek Galowicz
eaced12c94 Fix signed vs. unsigned comparison warning and improve code 2023-10-24 19:57:38 +02:00
Silvan Mosberger
f269911641
Document builtins.substring negative length behavior (#9226) 2023-10-24 11:22:02 +02:00
Robert Hensing
cde3c63617
system-features: Typo
There I was, thinking all of Apple's OSes started with lower case.
2023-10-23 19:30:00 +02:00
John Ericson
a58d7f143e
Merge pull request #9216 from obsidiansystems/addDrvOutputDependencies-pre
Add `builtins.addDrvOutputDependencies`
2023-10-23 13:26:51 -04:00
Valentin Gagarin
cd680bd53d
Merge how-to section on S3 buckets into S3 store docs (#7972)
Rather than having a misc tutorial page in the grab-bag "package management" section, this information should just be part of the S3 store docs.

---------

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2023-10-23 13:22:33 -04:00
John Ericson
765436e300 Add builtins.addDrvOutputDependencies
End goal: make `(mkDerivation x).drvPath` behave like a non-DrvDeep
context.

Problem: users won't be able to recover the DrvDeep behavior when
nixpkgs makes this change.

Solution: add this primop.

The new primop is fairly simple, and is supposed to complement other
existing ones (`builtins.storePath`, `builtins.outputOf`) so there are
simple ways to construct strings with every type of string context
element.

(It allows nothing we couldn't already do with `builtins.getContext` and `builtins.appendContext`, which is also true of those other two primops.)

This was originally in #8595, but then it was proposed to land some doc
changes separately. So now the code changes proper is just moved to
this, and the doc will be done in that.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.nore
github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io
2023-10-23 12:49:14 -04:00
Kirill Trofimov
90e3ed06f8 fix: Use default destructor. 2023-10-23 18:07:57 +03:00
Kirill Trofimov
a31fc5cc86 fix: Use using instead of typedef for type aliasing.
Since C++ 11 we shouldn't use c-style `typedefs`. In addition, `using` can be templated.
2023-10-23 18:07:17 +03:00
Kirill Trofimov
b205da16ef fix: Explicitly pass lambda scope variables.
Default capture implicitly also capture *this, which would automatically be used if for example you referenced a method from the enclosing scope.
2023-10-23 18:06:15 +03:00
Kirill Trofimov
c82066cf73 fix: Declare constructor as default 2023-10-23 16:59:19 +03:00
Naïm Favier
595010b631 nix-shell: fix shebang whitespace parsing
Leading whitespace after `nix-shell` used to produce an empty argument,
while an empty argument at the end of the line was ignored.

Fix the first issue by consuming the initial whitespace before calling
shellwords; fix the second issue by returning immediately if whitespace
is found at the end of the string instead of checking for an empty
string.

Also throw if quotes aren't terminated.
2023-10-23 15:56:07 +02:00
Naïm Favier
fa9642ec45 nix-shell: support single quotes in shebangs
Single quotes are a basic feature of shell syntax that people expect to
work. They are also more convenient for writing literal code expressions
with less escaping.
2023-10-23 15:56:05 +02:00
John Ericson
8b68bbb777
Merge pull request #6223 from obsidiansystems/worker-proto-with-version
Give `nix daemon` and `nix-store --serve` protocols separate serializers with version info
2023-10-23 09:16:23 -04:00
John Ericson
b461cac21a
Overhaul completions, redo #6693 (#8131)
As I complained in
https://github.com/NixOS/nix/pull/6784#issuecomment-1421777030 (a
comment on the wrong PR, sorry again!), #6693 introduced a second
completions mechanism to fix a bug. Having two completion mechanisms
isn't so nice.

As @thufschmitt also pointed out, it was a bummer to go from `FlakeRef`
to `std::string` when collecting flake refs. Now it is `FlakeRefs`
again.

The underlying issue that sought to work around was that completion of
arguments not at the end can still benefit from the information from
latter arguments.

To fix this better, we rip out that change and simply defer all
completion processing until after all the (regular, already-complete)
arguments have been passed.

In addition, I noticed the original completion logic used some global
variables. I do not like global variables, because even if they save
lines of code, they also obfuscate the architecture of the code.

I got rid of them  moved them to a new `RootArgs` class, which now has
`parseCmdline` instead of `Args`. The idea is that we have many argument
parsers from subcommands and what-not, but only one root args that owns
the other per actual parsing invocation. The state that was global is
now part of the root args instead.

This did, admittedly, add a bunch of new code. And I do feel bad about
that. So I went and added a lot of API docs to try to at least make the
current state of things clear to the next person.

--

This is needed for RFC 134 (tracking issue #7868). It was very hard to
modularize `Installable` parsing when there were two completion
arguments. I wouldn't go as far as to say it is *easy* now, but at least
it is less hard (and the completions test finally passed).

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-10-23 15:03:11 +02:00
Eelco Dolstra
955bbe53c5
Merge pull request #9177 from edolstra/input-accessors
Backport FSInputAccessor and MemoryInputAccessor from lazy-trees
2023-10-23 11:42:04 +02:00
Eelco Dolstra
34a42f0d0a Move PosixSourceAccessor into its own file 2023-10-23 11:05:50 +02:00
Кирилл Трофимов
201a4af9a4
Clean up app.cc (#9201)
- Rename `expected` to `expectedType`

- Use early `return` and `continue` to reduce nesting
2023-10-22 22:56:46 +00:00
John Ericson
edc07588ec
Merge pull request #9191 from tfc/libutil-implementation
libutil: Small improvements
2023-10-22 10:02:03 -04:00
John Ericson
70f8b96c11 Factor out UnkeyedValidPathInfo and test
This makes the path info serialisers ideomatic again, which allows me to
test them.
2023-10-20 15:21:04 -04:00
John Ericson
596bd469cc Move ValidPathInfo serialization code to worker-protocol.{cc.hh}
It does not belong with the data type itself.

This also materializes the fact that `copyPath` does not do any version
negotiation just just hard-codes "16".

The non-standard interface of these serializers makes it harder to test,
but this is fixed in the next commit which then adds those tests.
2023-10-20 15:20:48 -04:00
John Ericson
ab822af0df Factor out serialization for BuildResult
Worker Protocol:

Note that the worker protocol already had a serialization for
`BuildResult`; this was added in
a4604f1928. It didn't have any versioning
support because at that time reusable seralizers were not away for the protocol
version. It could thus only be used for new messages also introduced in
that commit.

Now that we do support versioning in reusable serializers, we can expand
it to support all known versions and use it in many more places.

The exist test data becomes the version 1.29 tests: note that those
files' contents are unchanged. 1.28 and 1.27 tests are added to cover
the older code-paths.

The keyered build result test only has 1.29 because the keying was also
added in a4604f19284254ac98f19a13ff7c2216de7fe176; the older
serializations are always used unkeyed.

Serve Protocol:

Conversely, no attempt was made to factor out such a serializer for the
serve protocol, so our work there in this commit for that protocol
proceeds from scratch.
2023-10-20 15:19:28 -04:00
John Ericson
4372738efe Systematize the worker protocol derived path serialiser
It was some ad-hoc functions to account for versions, while the already
factored-out serializer just supported the latest version.

Now, we can fold that version-specific logic into the factored out one,
and so we do.
2023-10-20 15:19:28 -04:00
John Ericson
96c58550b8 Test more derived paths 2023-10-20 15:19:28 -04:00
Eelco Dolstra
935c9981de Remove fetchers::Tree and move tarball-related stuff into its own header 2023-10-20 19:56:52 +02:00
Eelco Dolstra
091e5b4513
Merge pull request #9198 from edolstra/remove-direct
Input: Remove 'direct' field
2023-10-20 19:49:53 +02:00
John Ericson
81ed1d56ce
Merge pull request #9197 from obsidiansystems/delete-dead-code
Delete dead code
2023-10-20 13:32:06 -04:00
Eelco Dolstra
0f7e9d0513 Input: Remove 'direct' field 2023-10-20 19:14:04 +02:00
John Ericson
862d16436b Remove the ValidPathInfo == operator
It is dead code. It was added in
8e0946e8df as part of the repeated /
enforce-determinism feature, but that was removed in
8fdd156a65.

It is not good because it skips many fields. For testing purposes we
will soon want to add a new one that doesn't skip fields, but we want to
make sure making == sensitive to those fields won't change how Nix
works. Proving in this commit that the old version is dead code achieves
that.
2023-10-20 12:39:17 -04:00
John Ericson
3e6b9f9357 Remove prevInfos as its dead code
It is unused since 8e0946e8df removed
support for the repeat and enforce-determinism options.
2023-10-20 12:39:17 -04:00
Eelco Dolstra
af302267e5 Input::hasAllInfo(): Remove 2023-10-20 17:19:36 +02:00
Eelco Dolstra
bcf5c31950 Add future FIXME 2023-10-20 16:58:33 +02:00
Eelco Dolstra
57db3be9e4 SourceAccessor::readFile(): Support reading into a sink 2023-10-20 16:36:41 +02:00
Robert Hensing
4d17c59d8d
Merge pull request #9157 from obsidiansystems/protocol-versions
Add protocol versions to `{Worker,Serve}Proto::*Conn`
2023-10-20 15:34:26 +02:00
Eelco Dolstra
7a086a32bc fetchToStore(): Handle flat ingestion method and add test 2023-10-20 13:32:15 +02:00
Eelco Dolstra
173abec0bc coerceToPath(): Handle __toString, add tests 2023-10-20 13:06:44 +02:00
Eelco Dolstra
bacceaea91 Move getLastModified(), remove setPathDisplay() 2023-10-20 12:40:46 +02:00
Eelco Dolstra
df10dc630f
Doxygen
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-10-20 12:36:18 +02:00
Robert Hensing
bb645c5d02 system-features doc: kvm is Linux-only 2023-10-20 10:21:39 +02:00
Robert Hensing
9277eb276b libstore: Add apple-virt to system features when available
I'm sure that we'll adjust the implementation over time, but this
at least discerns between an apple silicon bare metal machine and
a tart VM.
2023-10-20 10:21:39 +02:00
Jacek Galowicz
b0f4ac29d3 libutil: Use c++ style cast 2023-10-19 18:26:49 +01:00
Jacek Galowicz
55f06b6f30 libutil: Remove non-needed constructor 2023-10-19 18:26:49 +01:00
Jacek Galowicz
87c4f4a972 libutil: Move some non-template implememntations from config.hh to
config.cc
2023-10-19 18:26:49 +01:00
Eelco Dolstra
5be7705ddf Remove stuff we don't need yet 2023-10-19 19:20:21 +02:00
Eelco Dolstra
50156302c0 Deduplicate FSSourceAccessor and FSInputAccessor 2023-10-19 15:20:10 +02:00
Eelco Dolstra
9f572eb0e3 Unify the two implementations of dumpPath() 2023-10-19 15:07:56 +02:00
Eelco Dolstra
fb6a3910c4 Move most of InputAccessor into libutil 2023-10-19 14:45:26 +02:00
Johannes Kirschbauer
9bc7b4f463
doc: generic closure supported key types (#9183)
* doc: generic closure supported key types

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-10-19 12:39:41 +00:00
Eelco Dolstra
06c57899e3 Remove FIXME 2023-10-19 14:22:05 +02:00
Eelco Dolstra
f16af08e83 Fix macOS compilation 2023-10-19 14:20:50 +02:00
Eelco Dolstra
12214fef09 InputAccessor::fetchToStore(): Support arbitrary ingestion methods 2023-10-19 14:19:10 +02:00
Yueh-Shun Li
5088e6563a primops: add builtins.convertHash
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-10-19 00:58:56 +08:00
Yueh-Shun Li
6b47635180 Add helper function parseHashFormat[Opt] printHashFormat
Add hash format analogy of
parseHashTypeOpt, parseHashType, and printHashType.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-10-19 00:56:44 +08:00
Yueh-Shun Li
231b0fca6d Migrate HashFormat to scoped enumeration (enum struct) 2023-10-19 00:56:44 +08:00
Yueh-Shun Li
e026f3e1ae treewide: Reference HashFormat members with scope
Base* -> HashFormat::Base*
2023-10-19 00:56:41 +08:00
Yueh-Shun Li
5043e6cf4e Document HashFormat 2023-10-19 00:38:32 +08:00
Yueh-Shun Li
838c70f621 treewide: Rename hashBase to hashFormat
hashBase is ambiguous, since it's not about the digital bases, but about
the format of hashes. Base16, Base32 and Base64 are all character maps
for binary encoding.

Rename the enum Base to HashFormat.

Rename variables of type HashFormat from [hash]Base to hashFormat,
including CmdHashBase::hashFormat and CmdToBase::hashFormat.
2023-10-19 00:38:32 +08:00
Yueh-Shun Li
aff177d860 Elaborate the "unknown hash algorithm" error
List the allowed hash formats
2023-10-19 00:38:32 +08:00
Yueh-Shun Li
e9ddf0b400 Simplify parseHashTypeOpt
Remove redundant "else" after "return".

Use std::nullopt to increase readability.
2023-10-19 00:38:32 +08:00
Eelco Dolstra
df73c6eb8c Introduce MemoryInputAccessor and use it for corepkgs
MemoryInputAccessor is an in-memory virtual filesystem that returns
files like <nix/fetchurl.nix>. This removes the need for special hacks
to handle those files.
2023-10-18 17:38:11 +02:00
Eelco Dolstra
ea38605d11 Introduce FSInputAccessor and use it
Backported from the lazy-trees branch. Note that this doesn't yet use
the access control features of FSInputAccessor.
2023-10-18 17:37:32 +02:00
Vladimir Kryachko
311e2ad024 Address review comments 2023-10-18 10:37:06 -04:00
Vladimir Kryachko
a2f0ba6a6d Fix transitive input locking.
Fixes reproducibility issue described in #9143

Fixes #9143
2023-10-18 10:31:36 -04:00
Eelco Dolstra
201c115c3e
Merge pull request #9151 from edolstra/stabilize-fetchTree
Stabilize fetchTree
2023-10-18 10:54:08 +02:00
Théophane Hufschmitt
c1a1766c46
Merge pull request #9169 from vkryachko/follow_cycle
Detect cycles in flake follows.
2023-10-18 07:34:03 +02:00
John Ericson
9d1f42db52
Merge pull request #9150 from vicky1999/fix/8914
`nix store ping` -> `nix store info`
2023-10-17 22:52:28 -04:00
John Ericson
e36c9175f4 Add protocol versions to {Worker,Serve}Proto::*Conn
This will allow us to factor out logic, which is currently scattered
inline, into several reusable instances

The tests are also updated to support versioning. Currently all Worker
and Serve protocol tests are using the minimum version, since no
version-specific serialisers have been created yet. But in subsequent
commits when that changes, we will test individual versions to ensure
complete coverage.
2023-10-17 11:21:10 -04:00
John Ericson
ff68426095 Name the protocol version types
This makes the code clearer, and will help us replace them with proper
structs and get rid of the macros later.
2023-10-17 11:20:39 -04:00
Eelco Dolstra
3470cd68c4 Mark some fetchers as experimental 2023-10-17 14:57:29 +02:00
Eelco Dolstra
f62b5500ff fetchTree: Require the flakes experimental feature for the URL syntax 2023-10-17 14:52:34 +02:00
vicky1999
dcc5f801f4 Store info command help updates 2023-10-17 09:39:59 +05:30
Artturin
b8dfa3d53b use doBind in addDependency 2023-10-17 01:26:34 +03:00
Artturin
11e47e7dfb factor out doBind from runChild 2023-10-17 01:26:34 +03:00
Artturin
630c2545d1 remove linkOrCopy and use bindmounts for files in addDependency 2023-10-17 01:26:34 +03:00
Artturin
5649229394 Bindmount files instead of hardlinking or copying to chroot
16591eb3cc (diff-19f999107b609d37cfb22c58e7f0bc1cf76edf1180e238dd6389e03cc279b604) (2013) added support for files to doBind

This is work towards allowing users to change the location of chrootRootDir, to, for example, a tmpfs.

inspired by trofi on matrix

> It looks like build sandbox created by nix-daemon runs on the same filesystem, as /nix/store including things like /tmp which makes all small temporary files hit the disk. Is it intentional? If it is is there an easy way to redirect chroot's root to be tmpfs?

dirsInChroot -> pathsInChroot
2023-10-17 01:26:34 +03:00
Jacek Galowicz
54b350d517 Drop some moves that would happen anyway but forbid NRVO where appicable 2023-10-16 21:48:35 +01:00
Jacek Galowicz
abf7df2b37 Fix moves that accidentally copy anyway 2023-10-16 21:48:35 +01:00
Vladimir Kryachko
d6066c90f8 Don't convert InputPaths to strings prematurely. 2023-10-16 15:47:28 -04:00
Jacek Galowicz
add066cc7b Fix broken move 2023-10-16 19:32:47 +01:00
Vladimir Kryachko
b3fd7db63f Detect cycles in flake follows.
This change results in an error thrown as opposed to segfaulting due to
stack overflow.

Fixes #9144
2023-10-16 13:00:49 -04:00
John Ericson
483d99c622 Add API docs to some args-related functionality 2023-10-16 11:25:15 -04:00
Eelco Dolstra
6bf68c1e64
Merge pull request #8904 from wentasah/metadata-shows-last-modified
nix flakes metadata: Show lastModified timestamp for each input
2023-10-16 13:52:51 +02:00
John Ericson
d070d8b746
Merge pull request #9137 from obsidiansystems/serve-protocol
Introduce separate Serve protocol serialisers
2023-10-13 10:51:46 -04:00
Eelco Dolstra
4112dd1fc9 Mark fetchTree as stable 2023-10-13 16:45:08 +02:00
Robert Hensing
da2b59a088
Merge pull request #8047 from lovesegfault/always-allow-substitutes
feat: add always-allow-substitutes
2023-10-13 15:42:11 +02:00
Eelco Dolstra
8eb4f735dc fetchTree: Only use the registry if flakes are enabled 2023-10-13 14:34:32 +02:00
Eelco Dolstra
4ce7a53a9c Update fetchTree docs 2023-10-13 14:34:32 +02:00
Eelco Dolstra
856fe13533 fetchTree cleanup
Two changes:

* The (probably unintentional) hack to handle paths as tarballs has
  been removed. This is almost certainly not what users expect and is
  inconsistent with flakeref handling everywhere else.

* The hack to support scp-style Git URLs has been moved to the Git
  fetcher, so it's now supported not just by fetchTree but by flake
  inputs.
2023-10-13 14:34:23 +02:00
Michal Sojka
db0d94560b Document builtins.fetchTree
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

Supersedes #6740
2023-10-13 14:24:10 +02:00
vicky1999
5c65379b22 info store alias added to store-ping 2023-10-13 07:16:05 +05:30
vicky1999
b4b1a07f97 store info alias created 2023-10-13 06:48:35 +05:30
Ninlives
94e91566ed
Allow CLI to pass environment variables to FOD builder (#8830)
Add a new experimental `impure-env` setting that is a key-value list of
environment variables to inject into FOD derivations that specify the
corresponding `impureEnvVars`.

This allows clients to make use of this feature (without having to change the
environment of the daemon itself) and might eventually deprecate the current
behaviour (pick whatever is in the environment of the daemon) as it's more
principled and might prevent information leakage.
2023-10-11 11:58:42 +00:00
John Ericson
f7b8f8aff6 Introduce separate Serve protocol serialisers
To start, it is just a clone of the common protocol. But now that we
have the separate protocol implementations, we can add versioning
information without the versions of one protocol leaking into another.

Using the infrastructure from the previous commit, we don't have to
duplicate code for shared behavior.

Motivation: No more perverse incentives. [0] did some awkward things
because the serialisers did not store the version. I don't want anyone
making changes to be pushed towards keeping the serialization logic with
the core data types just because it's easier or the alternative is
tedious.

The actual versioning of the Worker and Serve protocol serialisers
(Common remains unversioned as the underlying mini-protocols are not
versioned) will happen in subsequent commits / PRs.

[0]: fe1f34fa60
2023-10-10 11:52:45 -04:00
Valentin Gagarin
7642894a4e reword documentation on lookup path resolution 2023-10-10 00:49:27 +02:00
John Ericson
4de54b2190 Unit test the "common protocol" too
Copy the relevant tests to ensure the new interfaces added in the last
commit are tested.

Perhaps I should try to deduplicat these tests some more. However its
not clear how to do that outside of a big ugly C++ macro.
https://github.com/google/googletest/blob/main/docs/advanced.md has some
stuff but it is cumbersome and I didn't figure it out yet.

This is done in a separate commit in order to be sure that the first
commit really didn't change any behavior; if we changed the
implementation and the tests at once, it would be harder to tell whether
or not some behavioral changes slipped in what is supposed to be a "pure
refactor".

Co-Authored-By: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-10-09 16:57:03 -04:00
John Ericson
be81764320 Factor out bits of the worker protocol to use elsewhere
This introduces some shared infrastructure for our notion of protocols.
We can then define multiple protocols in terms of that notion.
We an also express how particular protocols depend on each other.

For example, we can define a common protocol and a worker protocol,
where the second depends on the first in terms of the data types it can
read and write.

The "serve" protocol can just use the common one for now, but will
eventually need its own machinary just like the worker protocol for
version-aware serialisers
2023-10-09 16:55:12 -04:00
John Ericson
22513c91dc
Merge pull request #8942 from fricklerhandwerk/option-anchors
always show anchors on setting listings
2023-10-09 11:29:36 -04:00
John Ericson
838be5e4a0
Merge pull request #9114 from fricklerhandwerk/lookup-path
introduce lookup paths as a distinct language construct
2023-10-09 11:28:40 -04:00
Robert Hensing
aeea49609b
Merge pull request #8895 from hercules-ci/gc-before-stats
eval: Run a full GC before printing stats
2023-10-09 17:14:45 +02:00
John Ericson
3953901796
Merge pull request #9112 from fricklerhandwerk/restrict-eval-nix-path
describe the effect of `restrict-eval` in a more focused manner
2023-10-09 11:07:21 -04:00
John Ericson
a7a9386c96
Merge pull request #9111 from fricklerhandwerk/impure-constants
document that pure-eval also disables `builtins.nixPath`
2023-10-09 11:04:47 -04:00
John Ericson
217d863f7a Merge branch 'master' into lookup-path 2023-10-09 10:54:54 -04:00
Robert Hensing
c32084a12c printStats -> maybePrintStats 2023-10-09 16:34:35 +02:00
Valentin Gagarin
f00a5eb11b introduce lookup paths as a distinct language construct
so far they did not really have a name, and were at best referred to as
"angle bracket syntax".
2023-10-07 04:44:09 +02:00
Valentin Gagarin
6305801626 reword and reformat description of builtins.import 2023-10-07 03:57:08 +02:00
Valentin Gagarin
2fe1ccf797 describe the effect of restrict-eval in a more focused manner 2023-10-07 03:20:20 +02:00
Valentin Gagarin
a86a3e5e59 document that pure-eval also disables builtins.nixPath 2023-10-07 03:12:25 +02:00
Valentin Gagarin
a67cee965a expand on interpolated expressions 2023-10-07 02:49:55 +02:00
John Ericson
68c81c7375 Put functional tests in tests/functional
I think it is bad for these reasons when `tests/` contains a mix of
functional and integration tests

 - Concepts is harder to understand, the documentation makes a good
   unit vs functional vs integration distinction, but when the
   integration tests are just two subdirs within `tests/` this is not
   clear.

 - Source filtering in the `flake.nix` is more complex. We need to
   filter out some of the dirs from `tests/`, rather than simply pick
   the dirs we want and take all of them. This is a good sign the
   structure of what we are trying to do is not matching the structure
   of the files.

With this change we have a clean:
```shell-session
$ git show 'HEAD:tests'
tree HEAD:tests

functional/
installer/
nixos/
```
2023-10-06 09:05:56 -04:00
Eelco Dolstra
3dd4475826
Merge pull request #8905 from hercules-ci/no-unknown-location
Don't print unknown locations unless requested for dev purposes
2023-10-06 14:41:01 +02:00
Eelco Dolstra
62434951d9
Merge pull request #9095 from edef1c/reject-dot-paths
StorePath: reject names starting with '.'
2023-10-06 14:12:53 +02:00
Robert Hensing
f95364a803 eval: Run a full GC before printing stats
This makes the numbers more deterministic, especially when it comes
to the final heap size.
2023-10-06 00:21:05 +02:00
Valentin Gagarin
e0e47c0a68 accommodate inconsistent output from lowdown
the `term` output mode leaves inline HTML around verbatim, while `nroff`
mode (used for `man` pages) does not.

the correct solution would be to pre-render all output with a more
benign tool so we have less liabilities in our own code, but this has to
do for now.
2023-10-05 01:20:26 +02:00
edef
24bda0c7b3 StorePath: reject names starting with '.'
This has been the behaviour before Nix 2.4. It was dropped in a rewrite
in 759947bf72, allowing the creation of
store paths that aren't considered valid by older Nix versions or other
Nix tooling.

Nix 2.4 didn't ship in NixOS until 22.05, and stdenv.mkDerivation in
nixpkgs drops leading periods since April 2022, so it's unlikely anyone
is relying on the current lax behaviour.

Closes #9091.

Change-Id: I4a57bd9899e1b0dba56870ae5a1b680918a18ce9
2023-10-04 22:10:52 +00:00
Valentin Gagarin
8232711c9f fix wiring of baked-in Nix expressions 2023-10-05 00:05:08 +02:00
John Ericson
e1af175707 Enable most of the third BuildResult worker protocol test
This was somewhat of a false alarm. The problem was not that the
protocol implementation actually failed to round trip, but that two of
the fields were ignored entirely --- not serialized and deserialized at
all.

For reference, those fields were added in
fa68eb367e.
2023-10-04 18:03:50 -04:00
John Ericson
632f24166d Test the rest of the worker protocol serializers
Part of the `BuildResult` test is commented out because we have caught a
roundtrip bug! A future PR will fix the bug and uncomment that test.
2023-10-04 15:31:52 -04:00
Robert Hensing
3c042f3b0b
Merge pull request #9032 from Ma27/structured-attrs-env-vars
structured attrs: improve support / usage of NIX_ATTRS_{SH,JSON}_FILE
2023-10-04 11:57:26 +02:00
John Ericson
8440afbed7 Revert "Adapt scheduler to work with dynamic derivations"
This reverts commit 5e3986f59c. This
un-implements RFC 92 but fixes the critical bug #9052 which many people
are hitting. This is a decent stop-gap until a minimal reproduction of
that bug is found and a proper fix can be made.

Mostly fixed #9052, but I would like to leave that issue open until we
have a regression test, so I can then properly fix the bug (unbreaking
RFC 92) later.
2023-10-01 23:43:12 -04:00
Maximilian Bosch
bfdd908f7d structured attrs: improve support / usage of NIX_ATTRS_{SH,JSON}_FILE
In #4770 I implemented proper `nix-shell(1)` support for derivations
using `__structuredAttrs = true;`. Back then we decided to introduce two
new environment variables, `NIX_ATTRS_SH_FILE` for `.attrs.sh` and
`NIX_ATTRS_JSON_FILE` for `.attrs.json`. This was to avoid having to
copy these files to `$NIX_BUILD_TOP` in a `nix-shell(1)` session which
effectively meant copying these files to the project dir without
cleaning up afterwords[1].

On last NixCon I resumed hacking on `__structuredAttrs = true;` by
default for `nixpkgs` with a few other folks and getting back to it,
I identified a few problems with the how it's used in `nixpkgs`:

* A lot of builders in `nixpkgs` don't care about the env vars and
  assume that `.attrs.sh` and `.attrs.json` are in `$NIX_BUILD_TOP`.
  The sole reason why this works is that `nix-shell(1)` sources
  the contents of `.attrs.sh` and then sources `$stdenv/setup` if it
  exists. This may not be pretty, but it mostly works. One notable
  difference when using nixpkgs' stdenv as of now is however that
  `$__structuredAttrs` is set to `1` on regular builds, but set to
  an empty string in a shell session.

  Also, `.attrs.json` cannot be used in shell sessions because
  it can only be accessed by `$NIX_ATTRS_JSON_FILE` and not by
  `$NIX_BUILD_TOP/.attrs.json`.

  I considered changing Nix to be compatible with what nixpkgs
  effectively does, but then we'd have to either move $NIX_BUILD_TOP for
  shell sessions to a temporary location (and thus breaking a lot of
  assumptions) or we'd reintroduce all the problems we solved back then
  by using these two env vars.

  This is partly because I didn't document these variables back
  then (mea culpa), so I decided to drop all mentions of
  `.attrs.{json,sh}` in the  manual and only refer to `$NIX_ATTRS_SH_FILE`
  and `$NIX_ATTRS_JSON_FILE`. The same applies to all our integration tests.
  Theoretically we could deprecated using `"$NIX_BUILD_TOP"/.attrs.sh` in
  the future now.

* `nix develop` and `nix print-dev-env` don't support this environment
  variable at all even though they're supposed to be part of the replacement
  for `nix-shell` - for the drv debugging part to be precise.

  This isn't a big deal for the vast majority of derivations, i.e.
  derivations relying on nixpkgs' `stdenv` wiring things together
  properly. This is because `nix develop` effectively "clones" the
  derivation and replaces the builder with a script that dumps all of
  the environment, shell variables, functions etc, so the state of
  structured attrs being "sourced" is transmitted into the dev shell and
  most of the time you don't need to worry about `.attrs.sh` not
  existing because the shell is correctly configured and the

      if [ -e .attrs.sh ]; then source .attrs.sh; fi

  is simply omitted.

  However, this will break when having a derivation that reads e.g. from
  `.attrs.json` like

      with import <nixpkgs> {};
      runCommand "foo" { __structuredAttrs = true; foo.bar = 23; } ''
        cat $NIX_ATTRS_JSON_FILE # doesn't work because it points to /build/.attrs.json
      ''

  To work around this I employed a similar approach as it exists for
  `nix-shell`: the `NIX_ATTRS_{JSON,SH}_FILE` vars are replaced with
  temporary locations.

  The contents of `.attrs.sh` and `.attrs.json` are now written into the
  JSON by `get-env.sh`, the builder that `nix develop` injects into the
  derivation it's debugging. So finally the exact file contents are
  present and exported by `nix develop`.

  I also made `.attrs.json` a JSON string in the JSON printed by
  `get-env.sh` on purpose because then it's not necessary to serialize
  the object structure again. `nix develop` only needs the JSON
  as string because it's only written into the temporary file.

  I'm not entirely sure if it makes sense to also use a temporary
  location for `nix print-dev-env` (rather than just skipping the
  rewrite in there), but this would probably break certain cases where
  it's relied upon `$NIX_ATTRS_SH_FILE` to exist (prime example are the
  `nix print-dev-env` test-cases I wrote in this patch using
  `tests/shell.nix`, these would fail because the env var exists, but it
  cannot read from it).

[1] https://github.com/NixOS/nix/pull/4770#issuecomment-836799719
2023-10-01 13:22:48 +01:00
Robert Hensing
f8a3893e8d pathExists: isDir when endswith /. 2023-09-30 02:35:26 +01:00
John Ericson
89b3952063 Make the indirect fetcher input scheme part of the Flakes XP feature
I don't know much about it, but by the number of times "flake" appears
in the code it seems like is part of flakes, at least for now.
2023-09-28 21:35:36 -04:00
John Ericson
bfe1308d3f Add infra for InputSchemes to be experimental 2023-09-28 21:35:30 -04:00
John Ericson
c816c67eed Reword some comments/API docs to reflect libfetcher's multiple users
It's not just flakes, but also `builtins.fetchTree`. Also try to provide
some more info in general.
2023-09-28 21:10:51 -04:00
John Ericson
b912f3a937 Move flakeIdRegex{,S} from libutil to flakeref.{cc,hh
It isn't used, and doesn't belong in `libutil`.
2023-09-28 20:55:41 -04:00
John Ericson
cede94dbf7 builtins.fetchTree: Mark experimental the new way
This helps ensure uniform docs/error message.
2023-09-28 20:51:25 -04:00
John Ericson
b7e712f9fd
Merge pull request #8509 from wentasah/fetch-tree-doc
Document fetchTree
2023-09-28 15:13:53 -04:00
Andrea Bedini
add7c99c3b Include "original" and "locked" in nix flake prefetch --json 2023-09-28 12:34:06 -04:00
tomberek
976f596579
Merge branch 'master' into tomberek.absolute.attrpath.notation 2023-09-28 10:01:57 -04:00
Robert Hensing
13a9090ffc
Merge pull request #9047 from flox/tomberek.string_refactor
string Value refactor
2023-09-28 02:58:57 +01:00
Ilan Joselevich
13ed5d7106 flakes: adopt repl-flake behavior as default 2023-09-27 20:47:10 -04:00
Robert Hensing
11a3dc99b2
Merge pull request #7003 from SuperSandro2000/patch-1
Improve experimental-features error wording
2023-09-27 23:26:27 +01:00
Robert Hensing
16a6ea7249
Merge pull request #9049 from inclyc/users/inclyc/move-path
libexpr: construct ExprPath by move ctor, not copy cotr
2023-09-27 22:30:44 +01:00
Alex Ameen
5bc540a8ca
Respect NOCOLOR
While `nix` has always been respectful towards requests for `NO_COLOR=1`, this change asks represents a new stage of maturity for `nix` - making it also respect quests for `NOCOLOR=1`.

This ideally makes the tool more accessible to folks like me, who are exhausted by guessing whether `NO_COLOR` or `NOCOLOR` is the right environment variable to set.

<3
2023-09-27 14:49:52 -05:00
Tom Bereknyei
399ef84420 refactor: use string accessors
Create context, string_view, and c_str, accessors throughout in order to
better support improvements to the underlying string representation.
2023-09-27 00:33:01 -04:00
Christina Sørensen
1eeea01931
Fix repl.md duplicate typo
Seems like `legacyPackages.x86_64-linux.emacs.name` is accidentally shown twice.
2023-09-27 02:07:43 +00:00
Yingchi Long
5b902ce9d6 libexpr: construct ExprPath by move ctor, not copy cotr 2023-09-26 23:30:32 +08:00
Théophane Hufschmitt
c6faef61a6
Merge pull request #8923 from obsidiansystems/test-proto
Unit test some worker protocol serializers
2023-09-26 17:12:24 +02:00
Robert Hensing
57202969d0
Merge pull request #9040 from waalge/waalge/tail-docstr
fix docstring
2023-09-26 15:32:00 +01:00
Théophane Hufschmitt
1da1642527
Merge pull request #9041 from trofi/profiles-sign
src/libstore/profiles.cc: fix comparison of sign difference
2023-09-26 07:50:17 +02:00
Valentin Gagarin
60a155d01c
Merge pull request #8706 from fricklerhandwerk/doc-system-features
document system features
2023-09-26 04:21:31 +02:00
Valentin Gagarin
b17f200b11
Document "Import From Derivation" (#7332)
* document "Import From Derivation"

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-09-26 01:49:03 +00:00
Tom Bereknyei
7e24dc606b fix(tests): fix assumption that string.s is a char* 2023-09-25 21:37:32 -04:00
Théophane Hufschmitt
9a78d87bc0
Merge pull request #6614 from RasmusRendal/spaces
Implement support for percent encoded filepaths for flakerefs
2023-09-26 02:27:09 +02:00
Robert Hensing
b19bd4f348
Merge pull request #8970 from hercules-ci/eval-stuff
Expr: remove redundant fields, add nrExprs
2023-09-25 19:49:22 +02:00
John Ericson
1f3fc08c59
Merge pull request #8887 from obsidiansystems/bsd-cross-ci
Support cross compiling to BSD and CI it
2023-09-25 13:46:55 -04:00
Robert Hensing
bd24176ac5
libexpr/nixexpr.hh: Remove redundant inline
This is redundant since definitions in C++ record are implicitly inline-ed.

Co-authored-by: Yingchi Long <i@lyc.dev>
2023-09-25 17:51:17 +01:00
Sergei Trofimovich
ad213103d8 src/libstore/profiles.cc: fix comparison of sign difference
Detected by `gcc` as:

      CXX    src/libstore/profiles.o
    src/libstore/profiles.cc: In function 'void nix::deleteGenerationsGreaterThan(const Path&, GenerationNumber, bool)':
    src/libstore/profiles.cc:186:50: warning: comparison of integer expressions of different signedness: 'int' and 'nix::GenerationNumber' {aka 'long unsigned int'} [-Wsign-compare]
      186 |     for (auto keep = 0; i != gens.rend() && keep < max; ++i, ++keep);
          |                                             ~~~~~^~~~~
2023-09-25 17:45:57 +01:00
Robert Hensing
b21c41529d
Merge pull request #9024 from obsidiansystems/git-objects-prep
Shuffle `ParseSink` code in preparation for git hashing support
2023-09-25 16:55:11 +02:00
waalge
70b5e6050c
fix docstring 2023-09-25 13:39:11 +00:00
John Ericson
728767db03
Merge pull request #9028 from Ericson2314/nix3-config-options
Misc options rendering adjustments
2023-09-25 09:04:56 -04:00
Théophane Hufschmitt
b3433099d4
Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-09-25 09:56:49 +02:00
Yingchi Long
e4b83fbfe2 libexpr: const rvalue reference -> value for nix::Expr nodes 2023-09-24 14:54:41 +08:00
John Ericson
1d9fd3a6f8 manual / manpages: Adjust option filter filtering, move from C++ to Nix
Behavior change:

Before we only showed uption if the command-specific options were
non-empty. But that is somewhat odd since we also show common options.
Now, we do everything based on the union of both sorts of options (with
hidden-categories filtered, as before).

Implementation change:

The JSON dumping once again includes all options; the filtering of
hidden categories is done in the Nix instead. This is better separation
of "content" vs "presentation", and prepare the way for the HTML manual
vs manpages / `--help` doing different things.
2023-09-23 00:34:51 -04:00
John Ericson
f2e201fbdb Expose RestoreSink in header (fs-sink.hh)
Co-Authored-By: Matthew Bauer <mjbauer95@gmail.com>
Co-Authored-By: Carlo Nucera <carlo.nucera@protonmail.com>
2023-09-22 09:11:29 -04:00
John Ericson
8a416e819c Move RestoreSink to fs-sink.cc
Co-Authored-By: Matthew Bauer <mjbauer95@gmail.com>
Co-Authored-By: Carlo Nucera <carlo.nucera@protonmail.com>
2023-09-22 09:10:32 -04:00
John Ericson
9d6114313b Move ParseSink to its own header
We will soon add a new implemenation so the one for NARs in `archive.cc`
isn't the only one.

Co-Authored-By: Matthew Bauer <mjbauer95@gmail.com>
Co-Authored-By: Carlo Nucera <carlo.nucera@protonmail.com>
2023-09-22 09:10:32 -04:00
John Ericson
39ba81a4eb Improve internal API docs for two file hashing functions
Co-Authored-By: Matthew Bauer <mjbauer95@gmail.com>
Co-Authored-By: Carlo Nucera <carlo.nucera@protonmail.com>
2023-09-22 09:10:32 -04:00
Théophane Hufschmitt
f89b84919c
Merge pull request #8931 from fricklerhandwerk/nix3-config-options
do not show configuration override flags for each command
2023-09-22 14:13:51 +02:00
Rasmus Rendal
3411507696 Document the percent-encoding mechanism 2023-09-22 10:07:14 +02:00
Théophane Hufschmitt
e8113747e1 Split the parseFlakeRefWithFragment function
Was starting to be very complex and hard to follow.
Now the different cases should be easier to understand.
2023-09-22 10:06:43 +02:00
Théophane Hufschmitt
50e61f579c Allow special characters in flake paths
Support using nix flakes in paths with spaces or abitrary unicode characters.
This introduces the convention that the path part of the URL should be
percent-encoded when dealing with `path:` urls and not when using
filepaths (following the convention of firefox).

Co-authored-by: Rendal <rasmus@rend.al>
2023-09-22 10:06:43 +02:00
John Ericson
fd61799c0b
Merge pull request #9006 from fricklerhandwerk/fix-links
fix links to configuration settings
2023-09-21 11:58:07 -04:00
Valentin Gagarin
1a412a8d78 fix typo in docstring 2023-09-21 11:38:13 +02:00
Valentin Gagarin
4685260a77 fix links to configuration settings 2023-09-21 02:52:58 +02:00
John Ericson
7f76d7f038 Rename an identifier of ours called stdout
This is a reserved identifier on NetBSD --- it is replaced by a macro on
that platform --- and so we cannot use it.
2023-09-20 09:04:42 -04:00
Eelco Dolstra
126e2645f2 Disable rapidcheck tests in the coverage run
https://hydra.nixos.org/build/233688539
2023-09-19 16:04:00 +02:00
Théophane Hufschmitt
2a52ec4e92
Merge pull request #7661 from henrik-ch/repl-doc
improved help command listing.
2023-09-14 16:43:02 +02:00
Théophane Hufschmitt
b99fdcf8db
Merge pull request #8959 from maralorn/print-parent
Print parent activity field in json log
2023-09-13 11:41:47 +02:00
Emil Nikolov
21783cff16
docs: make the nix develop --command example unambiguous (#8952) 2023-09-12 17:15:36 +02:00
Robert Hensing
bf8deb4991 Expr: remove redundant int and float fields 2023-09-12 13:45:45 +02:00
Robert Hensing
3720e811fa libexpr: Add nrExprs to NIX_SHOW_STATS 2023-09-12 13:21:55 +02:00
Matthew Kenigsberg
07545add53
Drop dead code
localPath is unused
2023-09-10 12:18:03 +02:00
maralorn
682dbcab9a
Print parent activity field in json log 2023-09-09 18:01:10 +02:00
Graham Bennett
82ddb13098 Unlock output paths when a derivation is already built
Without this change, nix build processes will not drop the locks for derivation goals
which have already been built by another process when the current process gets
round to building them. This means the locks are held until the process
terminates.

If there are other nix build processes in a similar state, they will also try to
acquire the same locks when they try to build the same derivation, and so will
wait until the lock holder terminates (which might be a very long time if it has
a lot to build). In some pathological cases, those processes might be holding
their own locks on other derivations due to the same issue, and this can lead to
deadlock.

Resolves #6468
2023-09-09 10:11:11 +01:00
thenbe
5473e10249
fix: nix shell multiple commands example (#8950)
The `-c` flag belongs to `sh` not `nix shell`. As it stands, the command errors with:

```
$ nix shell nixpkgs#gnumake --command sh --command "cd src && make"
sh: --command: invalid option
```

https://github.com/NixOS/nix/pull/8276 was good for readability, but it missed this since that PR used a find/replace script.
2023-09-07 22:25:32 +00:00
Robert Hensing
5f55c33917
Merge pull request #8944 from fricklerhandwerk/fix-anchor
fix invalid anchor link
2023-09-07 17:58:16 +02:00
John Ericson
80d7994f52 Special-case error message to add extra information
The Derivation parser and old ATerm unfortunately leaves few ways to get
nice errors when an old version of Nix encounters a new version of the
format. The most likely scenario for this to occur is with a new client
making a derivation that the old daemon it is communicating with cannot
understand.

The extensions we just created for dynamic derivation deps will add a
version field, solving the problem going forward, but there is still the
issue of what to do about old versions of Nix up to now.

The solution here is to carefully catch the bad error from the daemon
that is likely to indicate this problem, and add some extra context to
it.

There is another "Ugly backwards compatibility hack" in
`remote-store.cc` that also works by transforming an error.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-07 10:39:37 -04:00
John Ericson
7ad66cb3ef Allow dynamic derivation deps in inputDrvs
We use the same nested map representation we used for goals, again in
order to save space. We might someday want to combine with `inputDrvs`,
by doing `V = bool` instead of `V = std::set<OutputName>`, but we are
not doing that yet for sake of a smaller diff.

The ATerm format for Derivations also needs to be extended, in addition
to the in-memory format. To accomodate this, we added a new basic
versioning scheme, so old versions of Nix will get nice errors. (And
going forward, if the ATerm format changes again the errors will be even
better.)

`parsedStrings`, an internal function used as part of parsing
derivations in A-Term format, used to consume the final `]` but expect
the initial `[` to already be consumed. This made for what looked like
unbalanced brackets at callsites, which was confusing. Now it consumes
both which is hopefully less confusing.

As part of testing, we also created a unit test for the A-Term format for
regular non-experimental derivations too.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

Apply suggestions from code review

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-07 10:39:37 -04:00
Valentin Gagarin
829d4d3e03 fix invalid anchor link 2023-09-07 15:13:22 +02:00
Valentin Gagarin
09eb7f1ef6 do not show configuration override flags for each command
this removes a lot of noise from the web search, which precludes finding
the actual documentation.

some configuration settings have enough documentation to warrant
individual pages, so the alternative of including full setting
documentation in each command page doesn't make much sense here.

this change technically means that the command line flags to override
settings are "invisible", and not exported as JSON. this may or may not
be desirable. a more explicit approach would be adding a `hidden` field
to the flag's JSON output, but would also require adjusting
post-processing of that JSON for manual rendering.
2023-09-07 02:02:27 +02:00
John Ericson
b7edc2099f Improve derivation parsing
- Don't assert: Derivation ATerms are not necessarily produced by Nix,
  and parsers should always throw graceful errors

- Improve error message from `static void except(..)`, shows both what
  we expected and what we actually got.

The intention is that we backport it, and then hopefully a few people
might get slightly better errors if they try out new experimental drv
files (for RFC 92) with an old version of Nix.
2023-09-06 11:44:06 -04:00
John Ericson
754528241a
Merge pull request #8927 from obsidiansystems/test-derivation-aterm
Test and begin documentation of the ATerm format for derivations
2023-09-06 09:24:52 -04:00
Christina Sørensen
2b3a17820f
Fix globals.hh typo 2023-09-06 04:19:40 +00:00
John Ericson
880d9cabed Test and begin documentation of the ATerm format for derivations
Wanted to do this before the last dynamic derivations PR when I
introduce a variation, to make sure I wasn't changing the old version by
mistake.
2023-09-05 11:16:39 -04:00
John Ericson
7ff43435f9 Unit test some worker protocol serializers
Continue with the characterization testing idioms begun in
c70484454f, but this time for unit tests.

Co-authored-by: Andreas Rammhold <andreas@rammhold.de>
2023-09-05 10:48:02 -04:00
Robert Hensing
477bc617bb traces: Add _NIX_DEVELOPER_SHOW_UNKNOWN_LOCATIONS 2023-09-03 14:08:03 +02:00
Robert Hensing
f1aeeea32b traces: DRY printPosMaybe 2023-09-03 14:00:55 +02:00
Robert Hensing
4c50f5d130 traces: Do not print unknown location
Solves 1/3 of the infinite recursion at unknown location meme.
See #8879 for ensuring we always have a trace (for stack overflows)
We might want to re-add this for finding missing location info
*while hacking on that problem only*.
2023-09-03 13:44:32 +02:00
Walter Franzini
3ae1489847 nix flakes metadata: Show lastModified timestamp for each input
Before this commit, the output of `nix flake metadata` in nix repo
looked like this:

    ...
    Last modified: 2023-07-09 16:00:16
    Inputs:
    ├───flake-compat: github:edolstra/flake-compat/35bb57c0c8d8b62bbfd284272c928ceb64ddbde9
    ├───lowdown-src: github:kristapsdz/lowdown/d2c2b44ff6c27b936ec27358a2653caaef8f73b8
    ├───nixpkgs: github:NixOS/nixpkgs/04a75b2eecc0acf6239acf9dd04485ff8d14f425
    └───nixpkgs-regression: github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2

This commit changes that to:

    ...
    Last modified: 2023-07-09 16:00:16
    Inputs:
    ├───flake-compat: github:edolstra/flake-compat/35bb57c0c8d8b62bbfd284272c928ceb64ddbde9 (2023-01-17 11:47:33)
    ├───lowdown-src: github:kristapsdz/lowdown/d2c2b44ff6c27b936ec27358a2653caaef8f73b8 (2021-10-06 10:00:07)
    ├───nixpkgs: github:NixOS/nixpkgs/04a75b2eecc0acf6239acf9dd04485ff8d14f425 (2022-12-08 01:04:00)
    └───nixpkgs-regression: github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2 (2022-01-24 19:20:45)
2023-09-03 12:35:29 +02:00
Eelco Dolstra
5c95b32c46 Fix warning 'catching polymorphic type by value' 2023-09-01 14:49:49 +02:00
Eelco Dolstra
919781cacc
Merge branch 'master' into valid_deriver_2 2023-09-01 13:35:05 +02:00
Eelco Dolstra
b88784278f
Merge pull request #8869 from hercules-ci/fix-issue-8838-pathExists-isDir
Fix #8838, pathExists: isDir when ends with `/ `
2023-09-01 13:15:54 +02:00
Eelco Dolstra
5b5f56a9d4
Merge pull request #8859 from edolstra/tarball-last-modified
Tarball trees: Propagate lastModified
2023-08-29 17:02:06 +02:00
Bryan Honof
736b9cede7
Port the flags of nix-daemon to nix daemon (#8788)
The new `nix daemon` command didn't accept the same flags that `nix-daemon` did.

* docs(daemon): clarify the daemon trust override flags
* fix: change declaration order
* docs: add examples of nix daemon usage
* Apply suggestions from code review

---------

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
Co-authored-by: tomberek <tomberek@users.noreply.github.com>
2023-08-28 13:43:34 +00:00
Robert Hensing
1e08e12d81 pathExists: isDir when endswith /
Fixes https://github.com/NixOS/nix/issues/8838
2023-08-25 17:18:37 +02:00
John Ericson
50f40ac4c0
Merge pull request #8829 from obsidiansystems/build-dynamic-derivations
Adapt scheduler to work with dynamic derivations
2023-08-25 11:13:15 -04:00
tomberek
b563ef38cc
Merge pull request #8819 from VertexA115/fix/deep-follow-paths
Fix follow path checking at depths greater than 2
2023-08-25 10:33:05 -04:00
John Ericson
cbd89568ca
Merge pull request #8814 from hercules-ci/exception-self-check
initLibUtil: Add exception handling self-check
2023-08-25 10:15:54 -04:00
John Ericson
5e3986f59c Adapt scheduler to work with dynamic derivations
To avoid dealing with an optional `drvPath` (because we might not know
it yet) everywhere, make an `CreateDerivationAndRealiseGoal`. This goal
just builds/substitutes the derivation file, and then kicks of a build
for that obtained derivation; in other words it does the chaining of
goals when the drv file is missing (as can already be the case) or
computed (new case).

This also means the `getDerivation` state can be removed from
`DerivationGoal`, which makes the `BasicDerivation` / in memory case and
`Derivation` / drv file file case closer together.

The map type is factored out for clarity, and because we will soon hvae
a second use for it (`Derivation` itself).

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-25 10:01:25 -04:00
John Ericson
692074f714 Use Worker::makeDerivationGoal less
We're about to split up `DerivationGoal` a bit. At that point
`makeDerivationGoal` will mean something more specific than it does
today. (Perhaps a future rename will make this clearer.)

On the other hand, the more public `Worker::makeGoal` function will
continue to work exactly as before. So by moving some call sites to use
that instead, we preemptively avoid issues in the next step.
2023-08-25 09:55:07 -04:00
John Ericson
1c4caef14b Throw MissingRealisation not plain Error in both resolveDerivedPath
Now we are consistent with the other `resolveDerivedPath`, and other
such functions.
2023-08-25 09:55:07 -04:00
John Ericson
2f5d3da806 Introduce OutputName and OutputNameView type aliases
Hopefully they make the code easier to understand!
2023-08-25 09:55:07 -04:00
Guillaume Girol
925a444b92 add nix-store --query --valid-derivers command
notably useful when nix-store --query --deriver returns a non-existing
path.

Co-authored-by: Felix Uhl <iFreilicht@users.noreply.github.com>
2023-08-24 11:37:24 +02:00
Eelco Dolstra
81045f243f Tarball trees: Propagate lastModified
This makes them behave consistently with GitHub/GitLab flakes.
2023-08-22 21:51:26 +02:00
Eelco Dolstra
3fcdea4ca0
Merge pull request #8836 from obsidiansystems/rm-addToSearchPath
Delete `EvalState::addToSearchPath`
2023-08-21 14:58:00 +02:00
Tom Bereknyei
c609be4072 doc: explain the . attrPath prefix notation 2023-08-19 17:19:52 -04:00
Tom Bereknyei
52248b1c27 feat: notation to refer to no attribute search prefix
An attrPath prefix of "." indicates no need to try default attrPath prefixes. For example 1nixpkgs#legacyPackages.x86_64-linux.ERROR` searches through

```
trying flake output attribute 'packages.x86_64-linux.legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute ''
trying flake output attribute 'legacyPackages.x86_64-linux.legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute 'legacyPackages.x86_64-linux'
trying flake output attribute 'legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute 'legacyPackages.x86_64-linux'
```

And there is no way to specify that one does not want the automatic
search behavior. Now one can specify
`nixpkgs#.legacyPackages.x86_64-linux.ERROR` to only refer to the rooted
attribute path without any default injection of attribute search path or
system.
2023-08-19 17:07:46 -04:00
Robert Hensing
10afcf06aa
Merge pull request #8812 from tweag/fix-clang-tidy
Fix some warnings/bugs found by clang-tidy
2023-08-19 16:00:12 +02:00
John Ericson
fe71faa920 Delete EvalState::addToSearchPath
This function is now trivial enough that it doesn't need to exist.

`EvalState` can still be initialized with a custom search path, but we
don't have a need to mutate the search path after it has been
constructed, and I don't see why we would need to in the future.

Fixes #8229
2023-08-18 14:04:33 -04:00
John Ericson
665ad4f7c5
Merge pull request #8839 from obsidiansystems/string-context-7479
Refactor Raw pattern, part of #7479
2023-08-18 13:47:01 -04:00
John Ericson
9121fed4b4 Fixing #7479
Types converted:

- `NixStringContextElem`
- `OutputsSpec`
- `ExtendedOutputsSpec`
- `DerivationOutput`
- `DerivationType`

Existing ones mostly conforming the pattern cleaned up:

- `ContentAddressMethod`
- `ContentAddressWithReferences`

The `DerivationGoal::derivationType` field had a bogus initialization,
now caught, so I made it `std::optional`. I think #8829 can make it
non-optional again because it will ensure we always have the derivation
when we construct a `DerivationGoal`.

See that issue (#7479) for details on the general goal.

`git grep 'Raw::Raw'` indicates the two types I didn't yet convert
`DerivedPath` and `BuiltPath` (and their `Single` variants) . This is
because @roberth and I (can't find issue right now...) plan on reworking
them somewhat, so I didn't want to churn them more just yet.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-08-18 11:44:00 -04:00
Cole Helbling
73696ec716 libutil: fix double-encoding of URLs
If you have a URL that needs to be percent-encoded, such as
`http://localhost:8181/test/+3d.tar.gz`, and try to lock that in a Nix
flake such as the following:

    {
      inputs.test = { url = "http://localhost:8181/test/+3d.tar.gz"; flake = false; };
      outputs = { test, ... }: {
        t = builtins.readFile test;
      };
    }

running `nix flake metadata` shows that the input URL has been
incorrectly double-encoded (despite the flake.lock being correctly
encoded only once):

    [...snip...]
    Inputs:
    └───test: http://localhost:8181/test/%252B3d.tar.gz?narHash=sha256-EFUdrtf6Rn0LWIJufrmg8q99aT3jGfLvd1//zaJEufY%3D

(Notice the `%252B`? That's just `%2B` but percent-encoded again)

With this patch, the double-encoding is gone; running `nix flake
metadata` will show the proper URL:

    [...snip...]
    Inputs:
    └───test: http://localhost:8181/test/%2B3d.tar.gz?narHash=sha256-EFUdrtf6Rn0LWIJufrmg8q99aT3jGfLvd1//zaJEufY%3D

---

As far as I can tell, this happens because Nix already percent-encodes
the URL and stores this as the value of `inputs.asdf.url`.

However, when Nix later tries to read this out of the eval state as a
string (via `getStrAttr`), it has to run it through `parseURL` again to
get the `ParsedURL` structure.

Now, this itself isn't a problem -- the true problem arises when using
`ParsedURL::to_string` later, which then _re-escapes the path_. It is
at this point that what would have been `%2B` (`+`) becomes `%252B`
(`%2B`).
2023-08-17 14:16:19 -07:00
Cole Helbling
1d7a57cfd9 libexpr/tests: test that parseFlakeRef doesn't percent-encode twice 2023-08-17 13:45:55 -07:00
Eelco Dolstra
db3bf180a5
Merge pull request #8833 from hercules-ci/jobcategory-doc
Document jobCategory()
2023-08-16 17:11:03 +02:00
Robert Hensing
d8079ee350 Document jobCategory() 2023-08-16 16:16:58 +02:00
Sergei Trofimovich
b74962c92b src/libexpr/search-path.cc: avoid out-of-bounds read on string_view
Without the change build with `-D_GLIBCXX_ASSERTIONS` exposes testsuite
assertion:

    $ gdb src/libexpr/tests/libnixexpr-tests
    Reading symbols from src/libexpr/tests/libnixexpr-tests...
    (gdb) break __glibcxx_assert_fail
    (gdb) run
    (gdb) bt
    in std::__glibcxx_assert_fail(char const*, int, char const*, char const*)@plt () from /mnt/archive/big/git/nix/src/libexpr/libnixexpr.so
    in std::basic_string_view<char, std::char_traits<char> >::operator[] (this=0x7fffffff56c0, __pos=4)
        at /nix/store/r74fw2j8rx5idb0w8s1s6ynwwgs0qmh9-gcc-14.0.0/include/c++/14.0.0/string_view:258
    in nix::SearchPath::Prefix::suffixIfPotentialMatch (this=0x7fffffff5780, path=...) at src/libexpr/search-path.cc:15
    in nix::SearchPathElem_suffixIfPotentialMatch_partialPrefix_Test::TestBody (this=0x555555a17540) at src/libexpr/tests/search-path.cc:62

As string sizes are usigned types `(a - b) > 0` effectively means
`a != b`. While the intention should be `a > b`.

The change fixes test suite pass.
2023-08-14 22:07:37 +01:00
Alex Zero
1ef8008ca7 Fix follow path checking at depths greater than 2
We need to recurse into the input tree to handle follows paths that
trarverse multiple inputs that may or may not be follow paths
themselves.
2023-08-14 18:55:46 +01:00
John Ericson
44c8d83831 Create outputOf primop.
In the Nix language, given a drv path, we should be able to construct
another string referencing to one of its output. We can do this today
with `(import drvPath).output`, but this only works for derivations we
already have.

With dynamic derivations, however, that doesn't work well because the
`drvPath` isn't yet built: importing it like would need to trigger IFD,
when the whole point of this feature is to do "dynamic build graph"
without IFD!

Instead, what we want to do is create a placeholder value with the right
string context to refer to the output of the as-yet unbuilt derivation.
A new primop in the language, analogous to `builtins.placeholder` can be
used to create one. This will achieve all the right properties. The
placeholder machinery also will match out the `outPath` attribute for CA
derivations works.

In 60b7121d2c we added that type of
placeholder, and the derived path and string holder changes necessary to
support it. Then in the previous commit we cleaned up the code
(inspiration finally hit me!) to deduplicate the code and expose exactly
what we need. Now, we can wire up the primop trivally!

Part of RFC 92: dynamic derivations (tracking issue #6316)

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-14 09:37:37 -04:00
John Ericson
e7c39ff00b Rework evaluator SingleDerivedPath infra
`EvalState::mkSingleDerivedPathString` previously contained its own
inverse (printing, rather than parsing) in order to validate what was
parsed. Now that is pulled out into its own separate function:
`EvalState::coerceToSingleDerivedPath`.

In additional that pulled out logic is deduplicated with
`EvalState::mkOutputString` via `EvalState::mkOutputStringRaw`, which is
itself deduplicated (and generalized) with
`DownstreamPlaceholder::mkOutputStringRaw`.

All these changes make the unit tests simpler.

(We would ideally write more unit tests for `mkSingleDerivedPathString`
`coerceToSingleDerivedPath` directly, but we cannot yet do that because
the IO in reading the store path won't work when the dummy store cannot
hold anything. Someday we'll have a proper in-memory store which will
work for this.)

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-14 08:44:50 -04:00
John Ericson
a04720e68c Rename optOutputPath to optStaticOutputPath
This choice of variable name makes it more clear what is going on.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-14 08:44:48 -04:00
Robert Hensing
584ff408a4
Merge pull request #8735 from obsidiansystems/defexpr
Factor out `nix-defexpr` path computation
2023-08-11 20:32:03 +02:00
Robert Hensing
c4dbb55ba9 initLibUtil: Add exception handling self-check 2023-08-11 17:25:42 +02:00
tomberek
010dc7958e
Merge pull request #8369 from obsidiansystems/inductive-derived-path
Make the Derived Path family of types inductive for dynamic derivations
2023-08-11 08:50:22 -05:00
Yorick van Pelt
e78e9a6bd1
SimpleLogger::log: fix unintended fallthrough 2023-08-11 12:05:45 +02:00
Yorick van Pelt
2e5096e4f0
FileTransfer::download: fix use-after-move
std::move(state->data) and data.empty() were called in a loop, and
could run with no other threads intervening. Accessing moved objects
is undefined behavior, and could cause a crash.
2023-08-11 12:00:31 +02:00
Yorick van Pelt
1ffb26311b
MultiCommand::toJSON: Fix use-after-move 2023-08-11 12:00:11 +02:00
Yorick van Pelt
b9b51f9579
Prevent overriding virtual methods that are called in a destructor
Virtual methods are no longer valid once the derived destructor has
run. This means the compiler is free to optimize them to be
non-virtual.

Found using clang-tidy
2023-08-11 11:58:33 +02:00
John Ericson
60b7121d2c Make the Derived Path family of types inductive for dynamic derivations
We want to be able to write down `foo.drv^bar.drv^baz`:
`foo.drv^bar.drv` is the dynamic derivation (since it is itself a
derivation output, `bar.drv` from `foo.drv`).

To that end, we create `Single{Derivation,BuiltPath}` types, that are
very similar except instead of having multiple outputs (in a set or
map), they have a single one. This is for everything to the left of the
rightmost `^`.

`NixStringContextElem` has an analogous change, and now can reuse
`SingleDerivedPath` at the top level. In fact, if we ever get rid of
`DrvDeep`, `NixStringContextElem` could be replaced with
`SingleDerivedPath` entirely!

Important note: some JSON formats have changed.

We already can *produce* dynamic derivations, but we can't refer to them
directly. Today, we can merely express building or example at the top
imperatively over time by building `foo.drv^bar.drv`, and then with a
second nix invocation doing `<result-from-first>^baz`, but this is not
declarative. The ethos of Nix of being able to write down the full plan
everything you want to do, and then execute than plan with a single
command, and for that we need the new inductive form of these types.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-08-10 00:08:32 -04:00
Peter Waller
4b1bd822ac Try to realise CA derivations during queryMissing
This enables nix to correctly report what will be fetched in the case
that everything is a cache hit.

Note however that if an intermediate build of something which is not
cached could still cause products to end up being substituted if the
intermediate build results in a CA path which is in the cache.

Fixes #8615.

Signed-off-by: Peter Waller <p@pwaller.net>
2023-08-09 20:57:04 +01:00
Théophane Hufschmitt
d00fe5f225
Merge pull request #8805 from tweag/fix-add-to-store-existing
[V2] Fix misread of source if path is already valid
2023-08-08 14:57:45 +02:00
Eelco Dolstra
5624777988
Merge pull request #8786 from Ma27/fix-why-depends-precise
nix/why-depends: fix output of `--precise`
2023-08-07 19:32:49 +02:00
Théophane Hufschmitt
4999f42a70
Merge pull request #8322 from tweag/stabilize-discard-references
Stabilize `discard-references`
2023-08-07 17:35:02 +02:00
Eelco Dolstra
eb1302670e
Merge pull request #8769 from edolstra/generalize-tarball-urls
Don't require .tar/.zip extension for tarball flakerefs
2023-08-07 17:02:17 +02:00
Théophane Hufschmitt
ad410abbe0 Stabilize discard-references
It has been there for a few releases now (landed in 2.14.0), doesn't
seem to cause any major issue and is wanted in a few places
(https://github.com/NixOS/nix/pull/7087#issuecomment-1544471346).
2023-08-07 16:53:37 +02:00
Théophane Hufschmitt
5df0f1755f
Merge pull request #8692 from obsidiansystems/add-another-xp-check
Feature gate `DownstreamPlaceholder::unknownCaOutput`
2023-08-07 13:11:44 +02:00
Simon Rainerson
31a6e10fe5 Fix misread of source if path is already valid
When receiving a stream of NARs through the ssh-ng protocol, an already
existing path would cause the NAR archive to not be read in the stream,
resulting in trying to parse the NAR as a ValidPathInfo. This results in
the error message:
    error: not an absolute path: 'nix-archive-1'

Fixes #6253

Usually this problem is avoided by running QueryValidPaths before
AddMultipleToStore, but can arise when two parallel nix processes gets
the same response from QueryValidPaths. This makes the problem more
prominent when running builds in parallel.
2023-08-07 10:27:40 +02:00
John Ericson
9113b4252b
Merge pull request #8760 from iFreilicht/fix-json-load-assertion-errors
Fix derivation load assertion errors
2023-08-06 17:07:43 -07:00
Felix Uhl
3fefc2b284 Fix derivation load assertion errors
When loading a derivation from a JSON, malformed input would trigger
cryptic "assertion failed" errors. Simply replacing calls to `operator []`
with calls to `.at()` was not enough, as this would cause json.execptions
to be printed verbatim.

Display nice error messages instead and give some indication where the
error happened.

*Before:*

```
$ echo 4 | nix derivation add
error: [json.exception.type_error.305] cannot use operator[] with a string argument with number

$ nix derivation show nixpkgs#hello | nix derivation add
Assertion failed: (it != m_value.object->end()), function operator[], file /nix/store/8h9pxgq1776ns6qi5arx08ifgnhmgl22-nlohmann_json-3.11.2/include/nlohmann/json.hpp, line 2135.

$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: [json.exception.type_error.302] type must be string, but is object

$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error: [json.exception.type_error.302] type must be object, but is string

```

*After:*

```
$ echo 4 | nix derivation add
error: Expected JSON of derivation to be of type 'object', but it is of type 'number'

$ nix derivation show nixpkgs#hello | nix derivation add
error: Expected JSON object to contain key 'name' but it doesn't

$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: Expected JSON value to be of type 'string' but it is of type 'number'

$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error:
       … while reading key 'outputs'

       error: Expected JSON value to be of type 'object' but it is of type 'string'
```
2023-08-05 01:34:30 +02:00
Maximilian Bosch
7c09104a94
nix/why-depends: fix output of --precise
I haven't checked when this was exactly introduced, but on Nix 2.16 I
realized that the additional lines inserted when using `--precise` are
completely separated from the tree:

    nix why-depends /nix/store/ccgr4faaxys39s091qridxg1947lggh4-evcxr-0.14.2 /nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0 --precise --extra-experimental-features nix-command
    /nix/store/ccgr4faaxys39s091qridxg1947lggh4-evcxr-0.14.2
        → /nix/store/lcf37pgp3rgww67v9x2990hbfwx96c1w-gcc-wrapper-12.2.0
            → /nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0
    └───bin/evcxr: …':'}.PATH=${PATH/':''/nix/store/lcf37pgp3rgww67v9x2990hbfwx96c1w-gcc-wrapper-12.2.0/bin'':'/':'}…
        └───bin/cpp: …k disable=SC2193.[[ "/nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0/bin/cpp" = *++ ]] &&…

This is apparently because `std::cout` is buffered and flushed in the
end whereas the rest of the output isn't. The fix is rather simple, just
use `logger->cout` as it's already the case for the rest of the code.

This way we also don't need to insert additional newlines in the `hits`
map since that's something the logger takes care of.

Also added a small test to make sure that the layout of this is somehow
tested to reduce the risk of further regressions here.
2023-08-04 23:11:08 +02:00
John Ericson
3b592c880a Add infra for experimental store implemenations
This is analogous to that for experimental settings and flags that we
have also added as of late.
2023-08-02 15:46:38 -04:00
John Ericson
3723363697
Merge pull request #8765 from NixLayeredStore/refactor-store-verify
More cleanups for `LocalStore::verifyPath`
2023-08-02 13:39:29 -04:00
John Ericson
9b908fa70a Factor out nix-defexpr path computation
Avoid duplicated code, and also avoid "on the fly" path construction
(which makes it harder to keep track of which paths we use).

The factored out code doesn't create the Nix state dir anymore, but this
is fine because other in nix-env and nix-channel does:

- nix-channel: Line 158 in this commit

- nix-env: Line 1407 in this commit
2023-08-02 12:54:48 -04:00
John Ericson
66550878df Add comment explaining the use of readDirectory(realStoreDir) 2023-08-02 12:46:07 -04:00
John Ericson
770d50e49c local-store verifying: Rename store to something more clear
It is not a `Store` but a `StorePathSet`.
2023-08-02 12:40:04 -04:00
Eelco Dolstra
d00469ebf9
Merge pull request #8762 from obsidiansystems/split-out-eval-settings
Move evaluator settings (type and global) to separate file/header
2023-08-02 16:54:02 +02:00
Eelco Dolstra
d9e7758f47 Don't require .tar/.zip extension for tarball flakerefs
Special-casing the file name is rather ugly, so we shouldn't do
that. So now any {file,http,https} URL is handled by
TarballInputScheme, except for non-flake inputs (i.e. inputs that have
the attribute `flake = false`).
2023-08-01 16:07:20 +02:00
John Ericson
6525265f46 LocalStore::verifyPath: Try to clarify data flow with more scopes
It was initially unclear to me which of these are temporary state for
the verify paths computation, and which of these are the results of that
computation to be used in the rest of the function. Now, it is clear,
and enforced.
2023-07-31 12:44:27 -04:00
John Ericson
2a5f5fbb17 LocalStore::verifyPath: Use StorePathSet for store local var
We don't care about non-store-paths in there (things like `.links`, are,
in fact, allowed). So let's just skip them up front and be more strongly
typed.
2023-07-31 12:44:18 -04:00
John Ericson
dcdd5fed74
Merge pull request #8754 from NixLayeredStore/refactor-store-verify
Refactor `verifyPath` to take `StorePath` instead of `Path`
2023-07-31 12:01:36 -04:00
Ben Radford
c9a87ce7ca Refactor verifyPath to take StorePath instead of Path.
This way we avoid having to convert from Path to StorePath and vice versa in
the body of verifyPath.
2023-07-31 11:17:55 -04:00
John Ericson
1570e80219 Move evaluator settings (type and global) to separate file/header 2023-07-31 10:14:15 -04:00
Robert Hensing
33d58a90c2 toJSON: Add attribute path to trace 2023-07-31 13:02:54 +02:00
Alex Ameen
2d1d81114d
Add parseFlakeRef and flakeRefToString builtins (#8670)
Over the last year or so I've run into several use cases where I need to
parse and/or serialize URLs for use by `builtins.fetchTree` or
`builtins.getFlake`, largely in order to produce _lockfile-like_ files
for lang2nix frameworks or tools which use `nix` internally to drive
builds.

I've gone through the painstaking process of emulating
`nix::FlakeRef::fromAttrs` and `nix::parseFlakeRef` several times with
mixed success; but these are difficult to create and even harder to
maintain if I hope to stay aligned with changes to the real
parser/serializer.

I understand why adding new `builtins` isn't something we want to do
flagrantly. I'm recommending this addition simply because I keep
encountering use cases where I need to parse/serialize these URIs in
`nix` expressions, and I want a reliable solution.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-07-25 17:43:33 +00:00
John Ericson
60d8dd7aea Clean up store hierarchy with IndirectRootStore
See the API doc comments for details.
2023-07-24 09:19:44 -04:00
John Ericson
13269ba93b Make RemoteStore::ConnectionHandle part of class and expose
Will need to do subclass-specific implementations in the next commit.
This isn't because there will be multiple variations of the daemon
protocol (whew!) but because different clients pick and choose different
parts to use.
2023-07-24 09:17:01 -04:00
John Ericson
0a30b07277 Move Store::Params typedef to StoreConfig::Params
This is because `StoreConfig` also uses it.
2023-07-24 09:03:44 -04:00
John Ericson
4685c9b55f
Merge pull request #8664 from ncfavier/merge-dynamic-attrs
parser: merge nested dynamic attributes
2023-07-23 11:03:38 -04:00
Naïm Favier
570a1a3ad7
parser: merge nested dynamic attributes
Fixes https://github.com/NixOS/nix/issues/7115
2023-07-21 17:14:03 +02:00
John Ericson
fe1fbdb5a1
Merge pull request #8724 from obsidiansystems/queryPartialDerivationOutputMap-evalStore
Give `queryPartialDerivationOutputMap` an `evalStore` parameter
2023-07-21 08:53:18 -04:00
Eelco Dolstra
7ac24d9525
Merge pull request #8650 from obsidiansystems/content-address-simpler
Simplify `ContentAddress`
2023-07-21 13:46:53 +02:00
John Ericson
6bc98c7fba Give queryPartialDerivationOutputMap an evalStore parameter
This makes it more useful. In general, the derivation will be in one
store, and the realisation info is in another.

This also helps us avoid duplication. See how `resolveDerivedPath` is
now simpler because it uses `queryPartialDerivationOutputMap`. In #8369
we get more flavors of derived path, and need more code to resolve them
all, and this problem only gets worse.

The fact that we need a new method to deal with the multiple dispatch is
unfortunate, but this generally relates to the fact that `Store` is a
sub-par interface, too bulky/unwieldy and conflating separate concerns.
Solving that is out of scope of this PR.

This is part of the RFC 92 work. See tracking issue #6316
2023-07-20 15:59:52 -04:00
John Ericson
f62543fe1c Remove unneeded copy
It appeared in 8eb73a8724 (by me!) without
justification.
2023-07-20 15:42:06 -04:00
John Ericson
7b30293d38 Tighten #includes: DerivedPath doesn't care about Realisation 2023-07-20 15:42:06 -04:00
Valentin Gagarin
0e4f6dfcf7 revert anchor prefix for builtin constants
the original change broke many pre-existing anchor links.

also change formatting of the constants listing slightly:
- the type should not be part of the anchor
- add highlight to the "impure only" note
2023-07-20 10:27:38 +02:00
Robert Hensing
32494cbb29
Merge pull request #7973 from fricklerhandwerk/remove-channels
remove the Channels section
2023-07-19 14:02:26 +02:00
Valentin Gagarin
2fa90e5824 add more details on CA derivations 2023-07-19 13:59:18 +02:00
Valentin Gagarin
5f37ebcf83 document all special system features and their behavior 2023-07-19 13:34:03 +02:00
Robert Hensing
eca07b2a47
Merge pull request #8315 from fricklerhandwerk/doc-system
add information on the system type string
2023-07-19 12:07:23 +02:00
Valentin Gagarin
493ddf617f reformat system-features setting documentation 2023-07-19 11:07:47 +02:00
Valentin Gagarin
feb01b22ed add links to store API documentation 2023-07-19 11:07:47 +02:00
Valentin Gagarin
6c3cd429a6 fix broken links 2023-07-19 11:01:48 +02:00
Valentin Gagarin
0779005f49 expand on the extra-platforms option 2023-07-19 10:57:37 +02:00
Valentin Gagarin
fcadac0a02 mention extra-platforms 2023-07-19 10:38:42 +02:00
Valentin Gagarin
aba32def73 fix wording
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-07-19 10:38:30 +02:00
Valentin Gagarin
32de11923e add cross-links 2023-07-19 10:37:40 +02:00
Valentin Gagarin
3763c7bb5e shorten system setting description 2023-07-19 10:37:40 +02:00
Valentin Gagarin
c8a42039ea move docs of the current system to the system setting
add information what happens when Nix itself is cross-compiled
2023-07-19 10:37:40 +02:00
Valentin Gagarin
0751c1bfc6 one line per sentence for easier review 2023-07-19 10:37:40 +02:00
Robert Hensing
0e3a7e34a0
Merge pull request #8506 from corngood/ssh-master
Pass NIX_SSHOPTS when checking for an ssh master connection.
2023-07-18 15:47:57 +02:00
Théophane Hufschmitt
a8d5bb5e7e
Merge pull request #8342 from NixLayeredStore/best-effort-supplementary-groups
Best effort supplementary groups
2023-07-17 20:58:17 +02:00
John Ericson
0f7242ff87 Test nested sandboxing, and make nicer error
We were bedeviled by sandboxing issues when working on the layered
store. The problem ended up being that when we have nested nix builds,
and the inner store is inside the build dir (e.g. store is
`/build/nix-test/$name/store`, build dir is `/build`) bind mounts
clobber each other and store paths cannot be found.

After thoroughly cleaning up `local-derivation-goal.cc`, we might be
able to make that work. But that is a lot of work. For now, we just fail
earlier with a proper error message.

Finally, test this: nested sandboxing without the problematic store dir
should work, and with should fail with the expected error message.

Co-authored-by: Dylan Green <67574902+cidkidnix@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-07-14 09:40:24 -04:00
John Ericson
caabc4f648 Feature gate DownstreamPlaceholder::unknownCaOutput
This is a part of CA derivations that we forgot to put behind the
experimental feature.

This was caught by @fricklerhandwerk in
https://github.com/NixOS/nix/pull/8369#discussion_r1258133719
2023-07-13 07:56:33 -04:00
John Ericson
2c3fb0eb33 Move BuiltPath to its own header/C++ file in libcmd
It is less important, and used less widely, than `DerivedPath`.
2023-07-12 23:01:10 -04:00
Ben Radford
a2acd23466
Update src/libstore/globals.hh
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-12 12:33:05 +01:00
Ben Radford
0309f6b5b8
Update src/libstore/globals.hh
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-12 12:32:57 +01:00
Ben Radford
b8e8dfc3e8
Say a bit about default value in setting description. 2023-07-11 11:25:49 +01:00
Ben Radford
a193ec4052
Default should depend on whether we are root. 2023-07-11 11:14:13 +01:00
Ben Radford
2b4c59dd99
Be clearer about the security implications. 2023-07-11 11:09:25 +01:00
Ben Radford
0caf28f238
Update description for require-drop-supplementary-groups. 2023-07-11 10:57:14 +01:00
Ben Radford
07dabcc90e
Always attempt setgroups but allow failure to be ignored. 2023-07-11 10:44:05 +01:00
Ben Radford
25b20b4ad2
Merge remote-tracking branch 'origin/master' into best-effort-supplementary-groups 2023-07-11 09:38:34 +01:00
John Ericson
4a880c3cc0
Merge pull request #8579 from obsidiansystems/findPath-cleanup-2
Further search path cleanups
2023-07-10 09:59:01 -04:00
Eelco Dolstra
fea7d3b1cd
Merge pull request #8681 from inclyc/libexpr/parser-move-noeffect
libexpr: remove std::move() for `basePath` in parser, it has no effect
2023-07-10 13:30:42 +02:00
Bader AlAttar
3fa0266e7a
Fix some grammar in installables doc (#8682) 2023-07-10 09:33:04 +00:00
Yingchi Long
3d74e7b811 libexpr: remove std::move() for basePath in parser, it has no effect 2023-07-10 12:02:29 +08:00
John Ericson
028b26a77f
Merge pull request #8370 from hercules-ci/fetchClosure-input-addressed
`fetchClosure`: input addressed and pure
2023-07-09 23:41:22 -04:00
John Ericson
be518e73ae Clean up SearchPath
- Better types

- Own header / C++ file pair

- Test factored out methods

- Pass parsed thing around more than strings

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-07-09 23:22:22 -04:00
John Ericson
87dcd09047 Clean up resolveSearchPathElem
We should use `std::optional<std::string>` not `std::pair<bool,
std::string>` for an optional string.
2023-07-09 23:13:30 -04:00
John Ericson
903700c5e1 Simplify ContentAddress
Whereas `ContentAddressWithReferences` is a sum type complex because different
varieties support different notions of reference, and
`ContentAddressMethod` is a nested enum to support that,
`ContentAddress` can be a simple pair of a method and hash.

`ContentAddress` does not need to be a sum type on the outside because
the choice of method doesn't effect what type of hashes we can use.

Co-Authored-By: Cale Gibbard <cgibbard@gmail.com>
2023-07-07 07:30:01 -04:00
Robert Hensing
b4b02d084f fetchClosure: Interleave the examples in the docs 2023-07-07 11:40:40 +02:00
Robert Hensing
537e8beb77
fetchClosure: Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-07 11:00:40 +02:00
Eelco Dolstra
6db66ebfc5
Merge pull request #8631 from iFreilicht/profile-list
Profile list improvements
2023-07-05 16:06:17 +02:00
Théophane Hufschmitt
82d6699976
Document the path flakeref format (#8640)
* Document the path flakeref format

Fix https://github.com/NixOS/nix/issues/8482

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-04 08:39:08 +00:00
Eelco Dolstra
a353412c43 nix profile list: Add --json flag
This just dumps the profile manifest to stdout.
2023-07-03 12:23:57 +02:00
Eelco Dolstra
b8e8f27159 Rename 'resolvedRef' to 'lockedRef'
'resolvedRef' was incorrect, since a resolved ref is one after
registry resolution, which may still be unlocked (e.g. 'nixpkgs' ->
'github:NixOS/nixpkgs').
2023-07-03 12:23:57 +02:00
Yingchi Long
3c90340fe6 libexpr: use thread_local to make the parser thread-safe
If we call `adjustLoc`, the global variable `prev_yylloc` is shared
between threads and racy.

Currently, nix itself does not concurrently parsing files, but this is
helpful for libexpr users. (The parser is thread-safe except this.)
2023-07-03 16:05:43 +08:00
Eelco Dolstra
87b82db881 nix profile list: Improve readability of the output 2023-07-02 16:17:09 +02:00
Robert Hensing
7b39a388b3
Merge pull request #8566 from inclyc/nixd/value-print-depth
libexpr: extend `Value::print` to allow limited depth
2023-07-01 20:08:52 +02:00
Robert Hensing
40052c7613 fetchClosure: Docs and error message improvements
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-30 18:23:42 +02:00
Robert Hensing
50de11d662 doc: Improve fetchClosure documentation 2023-06-30 18:23:24 +02:00
Robert Hensing
dc79636007 fetchClosure: Refactor: replace enableRewriting
A single variable is nice and self-contained.
2023-06-30 18:22:47 +02:00
Robert Hensing
5bdca46117 fetchClosure: Split into three cases 2023-06-30 18:22:47 +02:00
Robert Hensing
55888633dd makeContentAddressed: Add single path helper 2023-06-30 18:22:47 +02:00
Robert Hensing
8dca95386c fetchClosure: Disallow toPath for inputAddressed = true 2023-06-30 18:22:47 +02:00
Robert Hensing
508aa58e67 fetchClosure: Always check that inputAddressed matches the result 2023-06-30 18:22:47 +02:00
Robert Hensing
ea30f152b7 fetchClosure: Allow input addressed paths in pure mode
When explicitly requested by the caller, as suggested in the meeting
(https://github.com/NixOS/nix/pull/8090#issuecomment-1531139324)

> @edolstra: { toPath } vs { fromPath } is too implicit

I've opted for the `inputAddressed = true` requirement, because it
we did not agree on renaming the path attributes.

> @roberth: more explicit
> @edolstra: except for the direction; not immediately clear in which direction the rewriting happens

This is in fact the most explicit syntax and a bit redundant, which is
good, because that redundancy lets us deliver an error message that
reminds expression authors that CA provides a better experience to
their users.
2023-06-30 18:22:43 +02:00
Robert Hensing
7e5b6d2c45 fetchClosure: Refactor: rename toCA -> enableRewriting 2023-06-30 18:19:56 +02:00
Robert Hensing
0f6d596df5 fetchClosure: Factor out attribute hint 2023-06-30 18:19:56 +02:00
Yueh-Shun Li
eebfe989a5 linkOrCopy: Fallback upon cross-device link error (EXDEV)
Fix building derivations in local chroot store on OpenAFS,
where hard linking accross directories causes cross-device link error
(EXDEV).
2023-06-30 21:12:26 +08:00
Eelco Dolstra
a0c617348b
Merge pull request #8589 from jfroche/sign-paths-as-allowed-user
Allow to sign path as unprivileged user
2023-06-30 13:13:42 +02:00
Robert Hensing
1632f08ea2
Merge pull request #8600 from inclyc/libexpr/fix-leaking-in-stripIndentation
libexpr: fix leaking `es2` in stripIndentation (parser.y)
2023-06-29 11:31:53 +02:00
Yingchi Long
3468cbaf47 libexpr: fix leaking es2 in stripIndentation (parser.y) 2023-06-28 22:38:44 +08:00
Jean-François Roche
80c9259756 Allow to sign path as unprivileged user
User can now sign path as unprivileged/allowed user

refs #1708
2023-06-27 18:31:31 +02:00
John Ericson
22b278e011 Automatically document builtin constants
This is done in roughly the same way builtin functions are documented.

Also auto-link experimental features for primops, subsuming PR #8371.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-27 09:37:54 -04:00
John Ericson
e8067daf09 Generialize showType 2023-06-27 09:11:42 -04:00
Maximilian Bosch
559fd7ffe7
nix flake check: improve error message if overlay is not a lambda (#8582)
* nix flake check: improve error message if overlay is not a lambda

Suppose you have an overlay like this

    {
      inputs = { /* ... */ };
      outputs = { flake-utils, ... }: flake-utils.lib.eachDefaultSystem
        (system: {
          overlays.default = final: prev: {

          };
        });
    }

then `nix flake check` (correctly) fails because `overlays` are supposed
to have the structure `overlays.<name> = final: prev: exp`. However, the
error-message is a little bit counter-intuitive:

    error: overlay does not take an argument named 'final'

While one might guess where the error actually comes from because the
trace above says `… while checking the overlay 'overlays.x86_64-linux'`
this is still pretty confusing because it complains about an argument
not being named `final` even though that's evidently the case.

With this change, the error-message actually makes it clear what's
wrong:

    [ma27@carsten:~/Projects/nix/tmp]$ nix flake check --extra-experimental-features 'nix-command flakes' path:$(pwd)
    error:
           … while checking flake output 'overlays'

             at /nix/store/clgblnxx003hyrq8qkz5ab6kgqkck6qc-source/flake.nix:4:5:

                3|   outputs = { ... }: {
                4|     overlays.x86_64-linux.snens = final: prev: {
                 |     ^
                5|       kek = throw "snens";

           … while checking the overlay 'overlays.x86_64-linux'

             at /nix/store/clgblnxx003hyrq8qkz5ab6kgqkck6qc-source/flake.nix:4:5:

                3|   outputs = { ... }: {
                4|     overlays.x86_64-linux.snens = final: prev: {
                 |     ^
                5|       kek = throw "snens";

           error: overlay is not a lambda, but a set instead
2023-06-27 14:58:29 +02:00
Eelco Dolstra
8e770dac9f
Merge pull request #8576 from obsidiansystems/findPath-cleanup
Use a struct not `std::pair` for `SearchPathElem`
2023-06-26 12:37:06 +02:00
Michal Sojka
f11445952f Document builtins.fetchTree
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

Supersedes #6740
2023-06-26 00:59:06 +02:00