Commit graph

10974 commits

Author SHA1 Message Date
Eelco Dolstra
12214fef09 InputAccessor::fetchToStore(): Support arbitrary ingestion methods 2023-10-19 14:19:10 +02:00
Yueh-Shun Li
5088e6563a primops: add builtins.convertHash
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-10-19 00:58:56 +08:00
Yueh-Shun Li
6b47635180 Add helper function parseHashFormat[Opt] printHashFormat
Add hash format analogy of
parseHashTypeOpt, parseHashType, and printHashType.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-10-19 00:56:44 +08:00
Yueh-Shun Li
231b0fca6d Migrate HashFormat to scoped enumeration (enum struct) 2023-10-19 00:56:44 +08:00
Yueh-Shun Li
e026f3e1ae treewide: Reference HashFormat members with scope
Base* -> HashFormat::Base*
2023-10-19 00:56:41 +08:00
Yueh-Shun Li
5043e6cf4e Document HashFormat 2023-10-19 00:38:32 +08:00
Yueh-Shun Li
838c70f621 treewide: Rename hashBase to hashFormat
hashBase is ambiguous, since it's not about the digital bases, but about
the format of hashes. Base16, Base32 and Base64 are all character maps
for binary encoding.

Rename the enum Base to HashFormat.

Rename variables of type HashFormat from [hash]Base to hashFormat,
including CmdHashBase::hashFormat and CmdToBase::hashFormat.
2023-10-19 00:38:32 +08:00
Yueh-Shun Li
aff177d860 Elaborate the "unknown hash algorithm" error
List the allowed hash formats
2023-10-19 00:38:32 +08:00
Yueh-Shun Li
e9ddf0b400 Simplify parseHashTypeOpt
Remove redundant "else" after "return".

Use std::nullopt to increase readability.
2023-10-19 00:38:32 +08:00
Eelco Dolstra
df73c6eb8c Introduce MemoryInputAccessor and use it for corepkgs
MemoryInputAccessor is an in-memory virtual filesystem that returns
files like <nix/fetchurl.nix>. This removes the need for special hacks
to handle those files.
2023-10-18 17:38:11 +02:00
Eelco Dolstra
ea38605d11 Introduce FSInputAccessor and use it
Backported from the lazy-trees branch. Note that this doesn't yet use
the access control features of FSInputAccessor.
2023-10-18 17:37:32 +02:00
Vladimir Kryachko
311e2ad024 Address review comments 2023-10-18 10:37:06 -04:00
Vladimir Kryachko
a2f0ba6a6d Fix transitive input locking.
Fixes reproducibility issue described in #9143

Fixes #9143
2023-10-18 10:31:36 -04:00
Eelco Dolstra
201c115c3e
Merge pull request #9151 from edolstra/stabilize-fetchTree
Stabilize fetchTree
2023-10-18 10:54:08 +02:00
Théophane Hufschmitt
c1a1766c46
Merge pull request #9169 from vkryachko/follow_cycle
Detect cycles in flake follows.
2023-10-18 07:34:03 +02:00
John Ericson
9d1f42db52
Merge pull request #9150 from vicky1999/fix/8914
`nix store ping` -> `nix store info`
2023-10-17 22:52:28 -04:00
John Ericson
e36c9175f4 Add protocol versions to {Worker,Serve}Proto::*Conn
This will allow us to factor out logic, which is currently scattered
inline, into several reusable instances

The tests are also updated to support versioning. Currently all Worker
and Serve protocol tests are using the minimum version, since no
version-specific serialisers have been created yet. But in subsequent
commits when that changes, we will test individual versions to ensure
complete coverage.
2023-10-17 11:21:10 -04:00
John Ericson
ff68426095 Name the protocol version types
This makes the code clearer, and will help us replace them with proper
structs and get rid of the macros later.
2023-10-17 11:20:39 -04:00
Eelco Dolstra
3470cd68c4 Mark some fetchers as experimental 2023-10-17 14:57:29 +02:00
Eelco Dolstra
f62b5500ff fetchTree: Require the flakes experimental feature for the URL syntax 2023-10-17 14:52:34 +02:00
vicky1999
dcc5f801f4 Store info command help updates 2023-10-17 09:39:59 +05:30
Artturin
b8dfa3d53b use doBind in addDependency 2023-10-17 01:26:34 +03:00
Artturin
11e47e7dfb factor out doBind from runChild 2023-10-17 01:26:34 +03:00
Artturin
630c2545d1 remove linkOrCopy and use bindmounts for files in addDependency 2023-10-17 01:26:34 +03:00
Artturin
5649229394 Bindmount files instead of hardlinking or copying to chroot
16591eb3cc (diff-19f999107b609d37cfb22c58e7f0bc1cf76edf1180e238dd6389e03cc279b604) (2013) added support for files to doBind

This is work towards allowing users to change the location of chrootRootDir, to, for example, a tmpfs.

inspired by trofi on matrix

> It looks like build sandbox created by nix-daemon runs on the same filesystem, as /nix/store including things like /tmp which makes all small temporary files hit the disk. Is it intentional? If it is is there an easy way to redirect chroot's root to be tmpfs?

dirsInChroot -> pathsInChroot
2023-10-17 01:26:34 +03:00
Jacek Galowicz
54b350d517 Drop some moves that would happen anyway but forbid NRVO where appicable 2023-10-16 21:48:35 +01:00
Jacek Galowicz
abf7df2b37 Fix moves that accidentally copy anyway 2023-10-16 21:48:35 +01:00
Vladimir Kryachko
d6066c90f8 Don't convert InputPaths to strings prematurely. 2023-10-16 15:47:28 -04:00
Jacek Galowicz
add066cc7b Fix broken move 2023-10-16 19:32:47 +01:00
Vladimir Kryachko
b3fd7db63f Detect cycles in flake follows.
This change results in an error thrown as opposed to segfaulting due to
stack overflow.

Fixes #9144
2023-10-16 13:00:49 -04:00
John Ericson
483d99c622 Add API docs to some args-related functionality 2023-10-16 11:25:15 -04:00
Eelco Dolstra
6bf68c1e64
Merge pull request #8904 from wentasah/metadata-shows-last-modified
nix flakes metadata: Show lastModified timestamp for each input
2023-10-16 13:52:51 +02:00
John Ericson
d070d8b746
Merge pull request #9137 from obsidiansystems/serve-protocol
Introduce separate Serve protocol serialisers
2023-10-13 10:51:46 -04:00
Eelco Dolstra
4112dd1fc9 Mark fetchTree as stable 2023-10-13 16:45:08 +02:00
Robert Hensing
da2b59a088
Merge pull request #8047 from lovesegfault/always-allow-substitutes
feat: add always-allow-substitutes
2023-10-13 15:42:11 +02:00
Eelco Dolstra
8eb4f735dc fetchTree: Only use the registry if flakes are enabled 2023-10-13 14:34:32 +02:00
Eelco Dolstra
4ce7a53a9c Update fetchTree docs 2023-10-13 14:34:32 +02:00
Eelco Dolstra
856fe13533 fetchTree cleanup
Two changes:

* The (probably unintentional) hack to handle paths as tarballs has
  been removed. This is almost certainly not what users expect and is
  inconsistent with flakeref handling everywhere else.

* The hack to support scp-style Git URLs has been moved to the Git
  fetcher, so it's now supported not just by fetchTree but by flake
  inputs.
2023-10-13 14:34:23 +02:00
Michal Sojka
db0d94560b Document builtins.fetchTree
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

Supersedes #6740
2023-10-13 14:24:10 +02:00
vicky1999
5c65379b22 info store alias added to store-ping 2023-10-13 07:16:05 +05:30
vicky1999
b4b1a07f97 store info alias created 2023-10-13 06:48:35 +05:30
Ninlives
94e91566ed
Allow CLI to pass environment variables to FOD builder (#8830)
Add a new experimental `impure-env` setting that is a key-value list of
environment variables to inject into FOD derivations that specify the
corresponding `impureEnvVars`.

This allows clients to make use of this feature (without having to change the
environment of the daemon itself) and might eventually deprecate the current
behaviour (pick whatever is in the environment of the daemon) as it's more
principled and might prevent information leakage.
2023-10-11 11:58:42 +00:00
John Ericson
f7b8f8aff6 Introduce separate Serve protocol serialisers
To start, it is just a clone of the common protocol. But now that we
have the separate protocol implementations, we can add versioning
information without the versions of one protocol leaking into another.

Using the infrastructure from the previous commit, we don't have to
duplicate code for shared behavior.

Motivation: No more perverse incentives. [0] did some awkward things
because the serialisers did not store the version. I don't want anyone
making changes to be pushed towards keeping the serialization logic with
the core data types just because it's easier or the alternative is
tedious.

The actual versioning of the Worker and Serve protocol serialisers
(Common remains unversioned as the underlying mini-protocols are not
versioned) will happen in subsequent commits / PRs.

[0]: fe1f34fa60
2023-10-10 11:52:45 -04:00
Valentin Gagarin
7642894a4e reword documentation on lookup path resolution 2023-10-10 00:49:27 +02:00
John Ericson
4de54b2190 Unit test the "common protocol" too
Copy the relevant tests to ensure the new interfaces added in the last
commit are tested.

Perhaps I should try to deduplicat these tests some more. However its
not clear how to do that outside of a big ugly C++ macro.
https://github.com/google/googletest/blob/main/docs/advanced.md has some
stuff but it is cumbersome and I didn't figure it out yet.

This is done in a separate commit in order to be sure that the first
commit really didn't change any behavior; if we changed the
implementation and the tests at once, it would be harder to tell whether
or not some behavioral changes slipped in what is supposed to be a "pure
refactor".

Co-Authored-By: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-10-09 16:57:03 -04:00
John Ericson
be81764320 Factor out bits of the worker protocol to use elsewhere
This introduces some shared infrastructure for our notion of protocols.
We can then define multiple protocols in terms of that notion.
We an also express how particular protocols depend on each other.

For example, we can define a common protocol and a worker protocol,
where the second depends on the first in terms of the data types it can
read and write.

The "serve" protocol can just use the common one for now, but will
eventually need its own machinary just like the worker protocol for
version-aware serialisers
2023-10-09 16:55:12 -04:00
John Ericson
22513c91dc
Merge pull request #8942 from fricklerhandwerk/option-anchors
always show anchors on setting listings
2023-10-09 11:29:36 -04:00
John Ericson
838be5e4a0
Merge pull request #9114 from fricklerhandwerk/lookup-path
introduce lookup paths as a distinct language construct
2023-10-09 11:28:40 -04:00
Robert Hensing
aeea49609b
Merge pull request #8895 from hercules-ci/gc-before-stats
eval: Run a full GC before printing stats
2023-10-09 17:14:45 +02:00
John Ericson
3953901796
Merge pull request #9112 from fricklerhandwerk/restrict-eval-nix-path
describe the effect of `restrict-eval` in a more focused manner
2023-10-09 11:07:21 -04:00
John Ericson
a7a9386c96
Merge pull request #9111 from fricklerhandwerk/impure-constants
document that pure-eval also disables `builtins.nixPath`
2023-10-09 11:04:47 -04:00
John Ericson
217d863f7a Merge branch 'master' into lookup-path 2023-10-09 10:54:54 -04:00
Robert Hensing
c32084a12c printStats -> maybePrintStats 2023-10-09 16:34:35 +02:00
Valentin Gagarin
f00a5eb11b introduce lookup paths as a distinct language construct
so far they did not really have a name, and were at best referred to as
"angle bracket syntax".
2023-10-07 04:44:09 +02:00
Valentin Gagarin
6305801626 reword and reformat description of builtins.import 2023-10-07 03:57:08 +02:00
Valentin Gagarin
2fe1ccf797 describe the effect of restrict-eval in a more focused manner 2023-10-07 03:20:20 +02:00
Valentin Gagarin
a86a3e5e59 document that pure-eval also disables builtins.nixPath 2023-10-07 03:12:25 +02:00
Valentin Gagarin
a67cee965a expand on interpolated expressions 2023-10-07 02:49:55 +02:00
John Ericson
68c81c7375 Put functional tests in tests/functional
I think it is bad for these reasons when `tests/` contains a mix of
functional and integration tests

 - Concepts is harder to understand, the documentation makes a good
   unit vs functional vs integration distinction, but when the
   integration tests are just two subdirs within `tests/` this is not
   clear.

 - Source filtering in the `flake.nix` is more complex. We need to
   filter out some of the dirs from `tests/`, rather than simply pick
   the dirs we want and take all of them. This is a good sign the
   structure of what we are trying to do is not matching the structure
   of the files.

With this change we have a clean:
```shell-session
$ git show 'HEAD:tests'
tree HEAD:tests

functional/
installer/
nixos/
```
2023-10-06 09:05:56 -04:00
Eelco Dolstra
3dd4475826
Merge pull request #8905 from hercules-ci/no-unknown-location
Don't print unknown locations unless requested for dev purposes
2023-10-06 14:41:01 +02:00
Eelco Dolstra
62434951d9
Merge pull request #9095 from edef1c/reject-dot-paths
StorePath: reject names starting with '.'
2023-10-06 14:12:53 +02:00
Robert Hensing
f95364a803 eval: Run a full GC before printing stats
This makes the numbers more deterministic, especially when it comes
to the final heap size.
2023-10-06 00:21:05 +02:00
Valentin Gagarin
e0e47c0a68 accommodate inconsistent output from lowdown
the `term` output mode leaves inline HTML around verbatim, while `nroff`
mode (used for `man` pages) does not.

the correct solution would be to pre-render all output with a more
benign tool so we have less liabilities in our own code, but this has to
do for now.
2023-10-05 01:20:26 +02:00
edef
24bda0c7b3 StorePath: reject names starting with '.'
This has been the behaviour before Nix 2.4. It was dropped in a rewrite
in 759947bf72, allowing the creation of
store paths that aren't considered valid by older Nix versions or other
Nix tooling.

Nix 2.4 didn't ship in NixOS until 22.05, and stdenv.mkDerivation in
nixpkgs drops leading periods since April 2022, so it's unlikely anyone
is relying on the current lax behaviour.

Closes #9091.

Change-Id: I4a57bd9899e1b0dba56870ae5a1b680918a18ce9
2023-10-04 22:10:52 +00:00
Valentin Gagarin
8232711c9f fix wiring of baked-in Nix expressions 2023-10-05 00:05:08 +02:00
John Ericson
e1af175707 Enable most of the third BuildResult worker protocol test
This was somewhat of a false alarm. The problem was not that the
protocol implementation actually failed to round trip, but that two of
the fields were ignored entirely --- not serialized and deserialized at
all.

For reference, those fields were added in
fa68eb367e.
2023-10-04 18:03:50 -04:00
John Ericson
632f24166d Test the rest of the worker protocol serializers
Part of the `BuildResult` test is commented out because we have caught a
roundtrip bug! A future PR will fix the bug and uncomment that test.
2023-10-04 15:31:52 -04:00
Robert Hensing
3c042f3b0b
Merge pull request #9032 from Ma27/structured-attrs-env-vars
structured attrs: improve support / usage of NIX_ATTRS_{SH,JSON}_FILE
2023-10-04 11:57:26 +02:00
John Ericson
8440afbed7 Revert "Adapt scheduler to work with dynamic derivations"
This reverts commit 5e3986f59c. This
un-implements RFC 92 but fixes the critical bug #9052 which many people
are hitting. This is a decent stop-gap until a minimal reproduction of
that bug is found and a proper fix can be made.

Mostly fixed #9052, but I would like to leave that issue open until we
have a regression test, so I can then properly fix the bug (unbreaking
RFC 92) later.
2023-10-01 23:43:12 -04:00
Maximilian Bosch
bfdd908f7d structured attrs: improve support / usage of NIX_ATTRS_{SH,JSON}_FILE
In #4770 I implemented proper `nix-shell(1)` support for derivations
using `__structuredAttrs = true;`. Back then we decided to introduce two
new environment variables, `NIX_ATTRS_SH_FILE` for `.attrs.sh` and
`NIX_ATTRS_JSON_FILE` for `.attrs.json`. This was to avoid having to
copy these files to `$NIX_BUILD_TOP` in a `nix-shell(1)` session which
effectively meant copying these files to the project dir without
cleaning up afterwords[1].

On last NixCon I resumed hacking on `__structuredAttrs = true;` by
default for `nixpkgs` with a few other folks and getting back to it,
I identified a few problems with the how it's used in `nixpkgs`:

* A lot of builders in `nixpkgs` don't care about the env vars and
  assume that `.attrs.sh` and `.attrs.json` are in `$NIX_BUILD_TOP`.
  The sole reason why this works is that `nix-shell(1)` sources
  the contents of `.attrs.sh` and then sources `$stdenv/setup` if it
  exists. This may not be pretty, but it mostly works. One notable
  difference when using nixpkgs' stdenv as of now is however that
  `$__structuredAttrs` is set to `1` on regular builds, but set to
  an empty string in a shell session.

  Also, `.attrs.json` cannot be used in shell sessions because
  it can only be accessed by `$NIX_ATTRS_JSON_FILE` and not by
  `$NIX_BUILD_TOP/.attrs.json`.

  I considered changing Nix to be compatible with what nixpkgs
  effectively does, but then we'd have to either move $NIX_BUILD_TOP for
  shell sessions to a temporary location (and thus breaking a lot of
  assumptions) or we'd reintroduce all the problems we solved back then
  by using these two env vars.

  This is partly because I didn't document these variables back
  then (mea culpa), so I decided to drop all mentions of
  `.attrs.{json,sh}` in the  manual and only refer to `$NIX_ATTRS_SH_FILE`
  and `$NIX_ATTRS_JSON_FILE`. The same applies to all our integration tests.
  Theoretically we could deprecated using `"$NIX_BUILD_TOP"/.attrs.sh` in
  the future now.

* `nix develop` and `nix print-dev-env` don't support this environment
  variable at all even though they're supposed to be part of the replacement
  for `nix-shell` - for the drv debugging part to be precise.

  This isn't a big deal for the vast majority of derivations, i.e.
  derivations relying on nixpkgs' `stdenv` wiring things together
  properly. This is because `nix develop` effectively "clones" the
  derivation and replaces the builder with a script that dumps all of
  the environment, shell variables, functions etc, so the state of
  structured attrs being "sourced" is transmitted into the dev shell and
  most of the time you don't need to worry about `.attrs.sh` not
  existing because the shell is correctly configured and the

      if [ -e .attrs.sh ]; then source .attrs.sh; fi

  is simply omitted.

  However, this will break when having a derivation that reads e.g. from
  `.attrs.json` like

      with import <nixpkgs> {};
      runCommand "foo" { __structuredAttrs = true; foo.bar = 23; } ''
        cat $NIX_ATTRS_JSON_FILE # doesn't work because it points to /build/.attrs.json
      ''

  To work around this I employed a similar approach as it exists for
  `nix-shell`: the `NIX_ATTRS_{JSON,SH}_FILE` vars are replaced with
  temporary locations.

  The contents of `.attrs.sh` and `.attrs.json` are now written into the
  JSON by `get-env.sh`, the builder that `nix develop` injects into the
  derivation it's debugging. So finally the exact file contents are
  present and exported by `nix develop`.

  I also made `.attrs.json` a JSON string in the JSON printed by
  `get-env.sh` on purpose because then it's not necessary to serialize
  the object structure again. `nix develop` only needs the JSON
  as string because it's only written into the temporary file.

  I'm not entirely sure if it makes sense to also use a temporary
  location for `nix print-dev-env` (rather than just skipping the
  rewrite in there), but this would probably break certain cases where
  it's relied upon `$NIX_ATTRS_SH_FILE` to exist (prime example are the
  `nix print-dev-env` test-cases I wrote in this patch using
  `tests/shell.nix`, these would fail because the env var exists, but it
  cannot read from it).

[1] https://github.com/NixOS/nix/pull/4770#issuecomment-836799719
2023-10-01 13:22:48 +01:00
Robert Hensing
f8a3893e8d pathExists: isDir when endswith /. 2023-09-30 02:35:26 +01:00
John Ericson
89b3952063 Make the indirect fetcher input scheme part of the Flakes XP feature
I don't know much about it, but by the number of times "flake" appears
in the code it seems like is part of flakes, at least for now.
2023-09-28 21:35:36 -04:00
John Ericson
bfe1308d3f Add infra for InputSchemes to be experimental 2023-09-28 21:35:30 -04:00
John Ericson
c816c67eed Reword some comments/API docs to reflect libfetcher's multiple users
It's not just flakes, but also `builtins.fetchTree`. Also try to provide
some more info in general.
2023-09-28 21:10:51 -04:00
John Ericson
b912f3a937 Move flakeIdRegex{,S} from libutil to flakeref.{cc,hh
It isn't used, and doesn't belong in `libutil`.
2023-09-28 20:55:41 -04:00
John Ericson
cede94dbf7 builtins.fetchTree: Mark experimental the new way
This helps ensure uniform docs/error message.
2023-09-28 20:51:25 -04:00
John Ericson
b7e712f9fd
Merge pull request #8509 from wentasah/fetch-tree-doc
Document fetchTree
2023-09-28 15:13:53 -04:00
Andrea Bedini
add7c99c3b Include "original" and "locked" in nix flake prefetch --json 2023-09-28 12:34:06 -04:00
tomberek
976f596579
Merge branch 'master' into tomberek.absolute.attrpath.notation 2023-09-28 10:01:57 -04:00
Robert Hensing
13a9090ffc
Merge pull request #9047 from flox/tomberek.string_refactor
string Value refactor
2023-09-28 02:58:57 +01:00
Ilan Joselevich
13ed5d7106 flakes: adopt repl-flake behavior as default 2023-09-27 20:47:10 -04:00
Robert Hensing
11a3dc99b2
Merge pull request #7003 from SuperSandro2000/patch-1
Improve experimental-features error wording
2023-09-27 23:26:27 +01:00
Robert Hensing
16a6ea7249
Merge pull request #9049 from inclyc/users/inclyc/move-path
libexpr: construct ExprPath by move ctor, not copy cotr
2023-09-27 22:30:44 +01:00
Alex Ameen
5bc540a8ca
Respect NOCOLOR
While `nix` has always been respectful towards requests for `NO_COLOR=1`, this change asks represents a new stage of maturity for `nix` - making it also respect quests for `NOCOLOR=1`.

This ideally makes the tool more accessible to folks like me, who are exhausted by guessing whether `NO_COLOR` or `NOCOLOR` is the right environment variable to set.

<3
2023-09-27 14:49:52 -05:00
Tom Bereknyei
399ef84420 refactor: use string accessors
Create context, string_view, and c_str, accessors throughout in order to
better support improvements to the underlying string representation.
2023-09-27 00:33:01 -04:00
Christina Sørensen
1eeea01931
Fix repl.md duplicate typo
Seems like `legacyPackages.x86_64-linux.emacs.name` is accidentally shown twice.
2023-09-27 02:07:43 +00:00
Yingchi Long
5b902ce9d6 libexpr: construct ExprPath by move ctor, not copy cotr 2023-09-26 23:30:32 +08:00
Théophane Hufschmitt
c6faef61a6
Merge pull request #8923 from obsidiansystems/test-proto
Unit test some worker protocol serializers
2023-09-26 17:12:24 +02:00
Robert Hensing
57202969d0
Merge pull request #9040 from waalge/waalge/tail-docstr
fix docstring
2023-09-26 15:32:00 +01:00
Théophane Hufschmitt
1da1642527
Merge pull request #9041 from trofi/profiles-sign
src/libstore/profiles.cc: fix comparison of sign difference
2023-09-26 07:50:17 +02:00
Valentin Gagarin
60a155d01c
Merge pull request #8706 from fricklerhandwerk/doc-system-features
document system features
2023-09-26 04:21:31 +02:00
Valentin Gagarin
b17f200b11
Document "Import From Derivation" (#7332)
* document "Import From Derivation"

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-09-26 01:49:03 +00:00
Tom Bereknyei
7e24dc606b fix(tests): fix assumption that string.s is a char* 2023-09-25 21:37:32 -04:00
Théophane Hufschmitt
9a78d87bc0
Merge pull request #6614 from RasmusRendal/spaces
Implement support for percent encoded filepaths for flakerefs
2023-09-26 02:27:09 +02:00
Robert Hensing
b19bd4f348
Merge pull request #8970 from hercules-ci/eval-stuff
Expr: remove redundant fields, add nrExprs
2023-09-25 19:49:22 +02:00
John Ericson
1f3fc08c59
Merge pull request #8887 from obsidiansystems/bsd-cross-ci
Support cross compiling to BSD and CI it
2023-09-25 13:46:55 -04:00
Robert Hensing
bd24176ac5
libexpr/nixexpr.hh: Remove redundant inline
This is redundant since definitions in C++ record are implicitly inline-ed.

Co-authored-by: Yingchi Long <i@lyc.dev>
2023-09-25 17:51:17 +01:00
Sergei Trofimovich
ad213103d8 src/libstore/profiles.cc: fix comparison of sign difference
Detected by `gcc` as:

      CXX    src/libstore/profiles.o
    src/libstore/profiles.cc: In function 'void nix::deleteGenerationsGreaterThan(const Path&, GenerationNumber, bool)':
    src/libstore/profiles.cc:186:50: warning: comparison of integer expressions of different signedness: 'int' and 'nix::GenerationNumber' {aka 'long unsigned int'} [-Wsign-compare]
      186 |     for (auto keep = 0; i != gens.rend() && keep < max; ++i, ++keep);
          |                                             ~~~~~^~~~~
2023-09-25 17:45:57 +01:00
Robert Hensing
b21c41529d
Merge pull request #9024 from obsidiansystems/git-objects-prep
Shuffle `ParseSink` code in preparation for git hashing support
2023-09-25 16:55:11 +02:00
waalge
70b5e6050c
fix docstring 2023-09-25 13:39:11 +00:00
John Ericson
728767db03
Merge pull request #9028 from Ericson2314/nix3-config-options
Misc options rendering adjustments
2023-09-25 09:04:56 -04:00
Théophane Hufschmitt
b3433099d4
Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-09-25 09:56:49 +02:00
Yingchi Long
e4b83fbfe2 libexpr: const rvalue reference -> value for nix::Expr nodes 2023-09-24 14:54:41 +08:00
John Ericson
1d9fd3a6f8 manual / manpages: Adjust option filter filtering, move from C++ to Nix
Behavior change:

Before we only showed uption if the command-specific options were
non-empty. But that is somewhat odd since we also show common options.
Now, we do everything based on the union of both sorts of options (with
hidden-categories filtered, as before).

Implementation change:

The JSON dumping once again includes all options; the filtering of
hidden categories is done in the Nix instead. This is better separation
of "content" vs "presentation", and prepare the way for the HTML manual
vs manpages / `--help` doing different things.
2023-09-23 00:34:51 -04:00
John Ericson
f2e201fbdb Expose RestoreSink in header (fs-sink.hh)
Co-Authored-By: Matthew Bauer <mjbauer95@gmail.com>
Co-Authored-By: Carlo Nucera <carlo.nucera@protonmail.com>
2023-09-22 09:11:29 -04:00
John Ericson
8a416e819c Move RestoreSink to fs-sink.cc
Co-Authored-By: Matthew Bauer <mjbauer95@gmail.com>
Co-Authored-By: Carlo Nucera <carlo.nucera@protonmail.com>
2023-09-22 09:10:32 -04:00
John Ericson
9d6114313b Move ParseSink to its own header
We will soon add a new implemenation so the one for NARs in `archive.cc`
isn't the only one.

Co-Authored-By: Matthew Bauer <mjbauer95@gmail.com>
Co-Authored-By: Carlo Nucera <carlo.nucera@protonmail.com>
2023-09-22 09:10:32 -04:00
John Ericson
39ba81a4eb Improve internal API docs for two file hashing functions
Co-Authored-By: Matthew Bauer <mjbauer95@gmail.com>
Co-Authored-By: Carlo Nucera <carlo.nucera@protonmail.com>
2023-09-22 09:10:32 -04:00
Théophane Hufschmitt
f89b84919c
Merge pull request #8931 from fricklerhandwerk/nix3-config-options
do not show configuration override flags for each command
2023-09-22 14:13:51 +02:00
Rasmus Rendal
3411507696 Document the percent-encoding mechanism 2023-09-22 10:07:14 +02:00
Théophane Hufschmitt
e8113747e1 Split the parseFlakeRefWithFragment function
Was starting to be very complex and hard to follow.
Now the different cases should be easier to understand.
2023-09-22 10:06:43 +02:00
Théophane Hufschmitt
50e61f579c Allow special characters in flake paths
Support using nix flakes in paths with spaces or abitrary unicode characters.
This introduces the convention that the path part of the URL should be
percent-encoded when dealing with `path:` urls and not when using
filepaths (following the convention of firefox).

Co-authored-by: Rendal <rasmus@rend.al>
2023-09-22 10:06:43 +02:00
John Ericson
fd61799c0b
Merge pull request #9006 from fricklerhandwerk/fix-links
fix links to configuration settings
2023-09-21 11:58:07 -04:00
Valentin Gagarin
1a412a8d78 fix typo in docstring 2023-09-21 11:38:13 +02:00
Valentin Gagarin
4685260a77 fix links to configuration settings 2023-09-21 02:52:58 +02:00
John Ericson
7f76d7f038 Rename an identifier of ours called stdout
This is a reserved identifier on NetBSD --- it is replaced by a macro on
that platform --- and so we cannot use it.
2023-09-20 09:04:42 -04:00
Eelco Dolstra
126e2645f2 Disable rapidcheck tests in the coverage run
https://hydra.nixos.org/build/233688539
2023-09-19 16:04:00 +02:00
Théophane Hufschmitt
2a52ec4e92
Merge pull request #7661 from henrik-ch/repl-doc
improved help command listing.
2023-09-14 16:43:02 +02:00
Théophane Hufschmitt
b99fdcf8db
Merge pull request #8959 from maralorn/print-parent
Print parent activity field in json log
2023-09-13 11:41:47 +02:00
Emil Nikolov
21783cff16
docs: make the nix develop --command example unambiguous (#8952) 2023-09-12 17:15:36 +02:00
Robert Hensing
bf8deb4991 Expr: remove redundant int and float fields 2023-09-12 13:45:45 +02:00
Robert Hensing
3720e811fa libexpr: Add nrExprs to NIX_SHOW_STATS 2023-09-12 13:21:55 +02:00
Matthew Kenigsberg
07545add53
Drop dead code
localPath is unused
2023-09-10 12:18:03 +02:00
maralorn
682dbcab9a
Print parent activity field in json log 2023-09-09 18:01:10 +02:00
Graham Bennett
82ddb13098 Unlock output paths when a derivation is already built
Without this change, nix build processes will not drop the locks for derivation goals
which have already been built by another process when the current process gets
round to building them. This means the locks are held until the process
terminates.

If there are other nix build processes in a similar state, they will also try to
acquire the same locks when they try to build the same derivation, and so will
wait until the lock holder terminates (which might be a very long time if it has
a lot to build). In some pathological cases, those processes might be holding
their own locks on other derivations due to the same issue, and this can lead to
deadlock.

Resolves #6468
2023-09-09 10:11:11 +01:00
thenbe
5473e10249
fix: nix shell multiple commands example (#8950)
The `-c` flag belongs to `sh` not `nix shell`. As it stands, the command errors with:

```
$ nix shell nixpkgs#gnumake --command sh --command "cd src && make"
sh: --command: invalid option
```

https://github.com/NixOS/nix/pull/8276 was good for readability, but it missed this since that PR used a find/replace script.
2023-09-07 22:25:32 +00:00
Robert Hensing
5f55c33917
Merge pull request #8944 from fricklerhandwerk/fix-anchor
fix invalid anchor link
2023-09-07 17:58:16 +02:00
John Ericson
80d7994f52 Special-case error message to add extra information
The Derivation parser and old ATerm unfortunately leaves few ways to get
nice errors when an old version of Nix encounters a new version of the
format. The most likely scenario for this to occur is with a new client
making a derivation that the old daemon it is communicating with cannot
understand.

The extensions we just created for dynamic derivation deps will add a
version field, solving the problem going forward, but there is still the
issue of what to do about old versions of Nix up to now.

The solution here is to carefully catch the bad error from the daemon
that is likely to indicate this problem, and add some extra context to
it.

There is another "Ugly backwards compatibility hack" in
`remote-store.cc` that also works by transforming an error.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-07 10:39:37 -04:00
John Ericson
7ad66cb3ef Allow dynamic derivation deps in inputDrvs
We use the same nested map representation we used for goals, again in
order to save space. We might someday want to combine with `inputDrvs`,
by doing `V = bool` instead of `V = std::set<OutputName>`, but we are
not doing that yet for sake of a smaller diff.

The ATerm format for Derivations also needs to be extended, in addition
to the in-memory format. To accomodate this, we added a new basic
versioning scheme, so old versions of Nix will get nice errors. (And
going forward, if the ATerm format changes again the errors will be even
better.)

`parsedStrings`, an internal function used as part of parsing
derivations in A-Term format, used to consume the final `]` but expect
the initial `[` to already be consumed. This made for what looked like
unbalanced brackets at callsites, which was confusing. Now it consumes
both which is hopefully less confusing.

As part of testing, we also created a unit test for the A-Term format for
regular non-experimental derivations too.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

Apply suggestions from code review

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-07 10:39:37 -04:00
Valentin Gagarin
829d4d3e03 fix invalid anchor link 2023-09-07 15:13:22 +02:00
Valentin Gagarin
09eb7f1ef6 do not show configuration override flags for each command
this removes a lot of noise from the web search, which precludes finding
the actual documentation.

some configuration settings have enough documentation to warrant
individual pages, so the alternative of including full setting
documentation in each command page doesn't make much sense here.

this change technically means that the command line flags to override
settings are "invisible", and not exported as JSON. this may or may not
be desirable. a more explicit approach would be adding a `hidden` field
to the flag's JSON output, but would also require adjusting
post-processing of that JSON for manual rendering.
2023-09-07 02:02:27 +02:00
John Ericson
b7edc2099f Improve derivation parsing
- Don't assert: Derivation ATerms are not necessarily produced by Nix,
  and parsers should always throw graceful errors

- Improve error message from `static void except(..)`, shows both what
  we expected and what we actually got.

The intention is that we backport it, and then hopefully a few people
might get slightly better errors if they try out new experimental drv
files (for RFC 92) with an old version of Nix.
2023-09-06 11:44:06 -04:00
John Ericson
754528241a
Merge pull request #8927 from obsidiansystems/test-derivation-aterm
Test and begin documentation of the ATerm format for derivations
2023-09-06 09:24:52 -04:00
Christina Sørensen
2b3a17820f
Fix globals.hh typo 2023-09-06 04:19:40 +00:00
John Ericson
880d9cabed Test and begin documentation of the ATerm format for derivations
Wanted to do this before the last dynamic derivations PR when I
introduce a variation, to make sure I wasn't changing the old version by
mistake.
2023-09-05 11:16:39 -04:00
John Ericson
7ff43435f9 Unit test some worker protocol serializers
Continue with the characterization testing idioms begun in
c70484454f, but this time for unit tests.

Co-authored-by: Andreas Rammhold <andreas@rammhold.de>
2023-09-05 10:48:02 -04:00
Robert Hensing
477bc617bb traces: Add _NIX_DEVELOPER_SHOW_UNKNOWN_LOCATIONS 2023-09-03 14:08:03 +02:00
Robert Hensing
f1aeeea32b traces: DRY printPosMaybe 2023-09-03 14:00:55 +02:00
Robert Hensing
4c50f5d130 traces: Do not print unknown location
Solves 1/3 of the infinite recursion at unknown location meme.
See #8879 for ensuring we always have a trace (for stack overflows)
We might want to re-add this for finding missing location info
*while hacking on that problem only*.
2023-09-03 13:44:32 +02:00
Walter Franzini
3ae1489847 nix flakes metadata: Show lastModified timestamp for each input
Before this commit, the output of `nix flake metadata` in nix repo
looked like this:

    ...
    Last modified: 2023-07-09 16:00:16
    Inputs:
    ├───flake-compat: github:edolstra/flake-compat/35bb57c0c8d8b62bbfd284272c928ceb64ddbde9
    ├───lowdown-src: github:kristapsdz/lowdown/d2c2b44ff6c27b936ec27358a2653caaef8f73b8
    ├───nixpkgs: github:NixOS/nixpkgs/04a75b2eecc0acf6239acf9dd04485ff8d14f425
    └───nixpkgs-regression: github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2

This commit changes that to:

    ...
    Last modified: 2023-07-09 16:00:16
    Inputs:
    ├───flake-compat: github:edolstra/flake-compat/35bb57c0c8d8b62bbfd284272c928ceb64ddbde9 (2023-01-17 11:47:33)
    ├───lowdown-src: github:kristapsdz/lowdown/d2c2b44ff6c27b936ec27358a2653caaef8f73b8 (2021-10-06 10:00:07)
    ├───nixpkgs: github:NixOS/nixpkgs/04a75b2eecc0acf6239acf9dd04485ff8d14f425 (2022-12-08 01:04:00)
    └───nixpkgs-regression: github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2 (2022-01-24 19:20:45)
2023-09-03 12:35:29 +02:00
Eelco Dolstra
5c95b32c46 Fix warning 'catching polymorphic type by value' 2023-09-01 14:49:49 +02:00
Eelco Dolstra
919781cacc
Merge branch 'master' into valid_deriver_2 2023-09-01 13:35:05 +02:00
Eelco Dolstra
b88784278f
Merge pull request #8869 from hercules-ci/fix-issue-8838-pathExists-isDir
Fix #8838, pathExists: isDir when ends with `/ `
2023-09-01 13:15:54 +02:00
Eelco Dolstra
5b5f56a9d4
Merge pull request #8859 from edolstra/tarball-last-modified
Tarball trees: Propagate lastModified
2023-08-29 17:02:06 +02:00
Bryan Honof
736b9cede7
Port the flags of nix-daemon to nix daemon (#8788)
The new `nix daemon` command didn't accept the same flags that `nix-daemon` did.

* docs(daemon): clarify the daemon trust override flags
* fix: change declaration order
* docs: add examples of nix daemon usage
* Apply suggestions from code review

---------

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
Co-authored-by: tomberek <tomberek@users.noreply.github.com>
2023-08-28 13:43:34 +00:00
Robert Hensing
1e08e12d81 pathExists: isDir when endswith /
Fixes https://github.com/NixOS/nix/issues/8838
2023-08-25 17:18:37 +02:00
John Ericson
50f40ac4c0
Merge pull request #8829 from obsidiansystems/build-dynamic-derivations
Adapt scheduler to work with dynamic derivations
2023-08-25 11:13:15 -04:00
tomberek
b563ef38cc
Merge pull request #8819 from VertexA115/fix/deep-follow-paths
Fix follow path checking at depths greater than 2
2023-08-25 10:33:05 -04:00
John Ericson
cbd89568ca
Merge pull request #8814 from hercules-ci/exception-self-check
initLibUtil: Add exception handling self-check
2023-08-25 10:15:54 -04:00
John Ericson
5e3986f59c Adapt scheduler to work with dynamic derivations
To avoid dealing with an optional `drvPath` (because we might not know
it yet) everywhere, make an `CreateDerivationAndRealiseGoal`. This goal
just builds/substitutes the derivation file, and then kicks of a build
for that obtained derivation; in other words it does the chaining of
goals when the drv file is missing (as can already be the case) or
computed (new case).

This also means the `getDerivation` state can be removed from
`DerivationGoal`, which makes the `BasicDerivation` / in memory case and
`Derivation` / drv file file case closer together.

The map type is factored out for clarity, and because we will soon hvae
a second use for it (`Derivation` itself).

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-25 10:01:25 -04:00
John Ericson
692074f714 Use Worker::makeDerivationGoal less
We're about to split up `DerivationGoal` a bit. At that point
`makeDerivationGoal` will mean something more specific than it does
today. (Perhaps a future rename will make this clearer.)

On the other hand, the more public `Worker::makeGoal` function will
continue to work exactly as before. So by moving some call sites to use
that instead, we preemptively avoid issues in the next step.
2023-08-25 09:55:07 -04:00
John Ericson
1c4caef14b Throw MissingRealisation not plain Error in both resolveDerivedPath
Now we are consistent with the other `resolveDerivedPath`, and other
such functions.
2023-08-25 09:55:07 -04:00
John Ericson
2f5d3da806 Introduce OutputName and OutputNameView type aliases
Hopefully they make the code easier to understand!
2023-08-25 09:55:07 -04:00
Guillaume Girol
925a444b92 add nix-store --query --valid-derivers command
notably useful when nix-store --query --deriver returns a non-existing
path.

Co-authored-by: Felix Uhl <iFreilicht@users.noreply.github.com>
2023-08-24 11:37:24 +02:00
Eelco Dolstra
81045f243f Tarball trees: Propagate lastModified
This makes them behave consistently with GitHub/GitLab flakes.
2023-08-22 21:51:26 +02:00
Eelco Dolstra
3fcdea4ca0
Merge pull request #8836 from obsidiansystems/rm-addToSearchPath
Delete `EvalState::addToSearchPath`
2023-08-21 14:58:00 +02:00
Tom Bereknyei
c609be4072 doc: explain the . attrPath prefix notation 2023-08-19 17:19:52 -04:00
Tom Bereknyei
52248b1c27 feat: notation to refer to no attribute search prefix
An attrPath prefix of "." indicates no need to try default attrPath prefixes. For example 1nixpkgs#legacyPackages.x86_64-linux.ERROR` searches through

```
trying flake output attribute 'packages.x86_64-linux.legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute ''
trying flake output attribute 'legacyPackages.x86_64-linux.legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute 'legacyPackages.x86_64-linux'
trying flake output attribute 'legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute 'legacyPackages.x86_64-linux'
```

And there is no way to specify that one does not want the automatic
search behavior. Now one can specify
`nixpkgs#.legacyPackages.x86_64-linux.ERROR` to only refer to the rooted
attribute path without any default injection of attribute search path or
system.
2023-08-19 17:07:46 -04:00
Robert Hensing
10afcf06aa
Merge pull request #8812 from tweag/fix-clang-tidy
Fix some warnings/bugs found by clang-tidy
2023-08-19 16:00:12 +02:00
John Ericson
fe71faa920 Delete EvalState::addToSearchPath
This function is now trivial enough that it doesn't need to exist.

`EvalState` can still be initialized with a custom search path, but we
don't have a need to mutate the search path after it has been
constructed, and I don't see why we would need to in the future.

Fixes #8229
2023-08-18 14:04:33 -04:00
John Ericson
665ad4f7c5
Merge pull request #8839 from obsidiansystems/string-context-7479
Refactor Raw pattern, part of #7479
2023-08-18 13:47:01 -04:00
John Ericson
9121fed4b4 Fixing #7479
Types converted:

- `NixStringContextElem`
- `OutputsSpec`
- `ExtendedOutputsSpec`
- `DerivationOutput`
- `DerivationType`

Existing ones mostly conforming the pattern cleaned up:

- `ContentAddressMethod`
- `ContentAddressWithReferences`

The `DerivationGoal::derivationType` field had a bogus initialization,
now caught, so I made it `std::optional`. I think #8829 can make it
non-optional again because it will ensure we always have the derivation
when we construct a `DerivationGoal`.

See that issue (#7479) for details on the general goal.

`git grep 'Raw::Raw'` indicates the two types I didn't yet convert
`DerivedPath` and `BuiltPath` (and their `Single` variants) . This is
because @roberth and I (can't find issue right now...) plan on reworking
them somewhat, so I didn't want to churn them more just yet.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-08-18 11:44:00 -04:00
Cole Helbling
73696ec716 libutil: fix double-encoding of URLs
If you have a URL that needs to be percent-encoded, such as
`http://localhost:8181/test/+3d.tar.gz`, and try to lock that in a Nix
flake such as the following:

    {
      inputs.test = { url = "http://localhost:8181/test/+3d.tar.gz"; flake = false; };
      outputs = { test, ... }: {
        t = builtins.readFile test;
      };
    }

running `nix flake metadata` shows that the input URL has been
incorrectly double-encoded (despite the flake.lock being correctly
encoded only once):

    [...snip...]
    Inputs:
    └───test: http://localhost:8181/test/%252B3d.tar.gz?narHash=sha256-EFUdrtf6Rn0LWIJufrmg8q99aT3jGfLvd1//zaJEufY%3D

(Notice the `%252B`? That's just `%2B` but percent-encoded again)

With this patch, the double-encoding is gone; running `nix flake
metadata` will show the proper URL:

    [...snip...]
    Inputs:
    └───test: http://localhost:8181/test/%2B3d.tar.gz?narHash=sha256-EFUdrtf6Rn0LWIJufrmg8q99aT3jGfLvd1//zaJEufY%3D

---

As far as I can tell, this happens because Nix already percent-encodes
the URL and stores this as the value of `inputs.asdf.url`.

However, when Nix later tries to read this out of the eval state as a
string (via `getStrAttr`), it has to run it through `parseURL` again to
get the `ParsedURL` structure.

Now, this itself isn't a problem -- the true problem arises when using
`ParsedURL::to_string` later, which then _re-escapes the path_. It is
at this point that what would have been `%2B` (`+`) becomes `%252B`
(`%2B`).
2023-08-17 14:16:19 -07:00
Cole Helbling
1d7a57cfd9 libexpr/tests: test that parseFlakeRef doesn't percent-encode twice 2023-08-17 13:45:55 -07:00
Eelco Dolstra
db3bf180a5
Merge pull request #8833 from hercules-ci/jobcategory-doc
Document jobCategory()
2023-08-16 17:11:03 +02:00
Robert Hensing
d8079ee350 Document jobCategory() 2023-08-16 16:16:58 +02:00
Sergei Trofimovich
b74962c92b src/libexpr/search-path.cc: avoid out-of-bounds read on string_view
Without the change build with `-D_GLIBCXX_ASSERTIONS` exposes testsuite
assertion:

    $ gdb src/libexpr/tests/libnixexpr-tests
    Reading symbols from src/libexpr/tests/libnixexpr-tests...
    (gdb) break __glibcxx_assert_fail
    (gdb) run
    (gdb) bt
    in std::__glibcxx_assert_fail(char const*, int, char const*, char const*)@plt () from /mnt/archive/big/git/nix/src/libexpr/libnixexpr.so
    in std::basic_string_view<char, std::char_traits<char> >::operator[] (this=0x7fffffff56c0, __pos=4)
        at /nix/store/r74fw2j8rx5idb0w8s1s6ynwwgs0qmh9-gcc-14.0.0/include/c++/14.0.0/string_view:258
    in nix::SearchPath::Prefix::suffixIfPotentialMatch (this=0x7fffffff5780, path=...) at src/libexpr/search-path.cc:15
    in nix::SearchPathElem_suffixIfPotentialMatch_partialPrefix_Test::TestBody (this=0x555555a17540) at src/libexpr/tests/search-path.cc:62

As string sizes are usigned types `(a - b) > 0` effectively means
`a != b`. While the intention should be `a > b`.

The change fixes test suite pass.
2023-08-14 22:07:37 +01:00
Alex Zero
1ef8008ca7 Fix follow path checking at depths greater than 2
We need to recurse into the input tree to handle follows paths that
trarverse multiple inputs that may or may not be follow paths
themselves.
2023-08-14 18:55:46 +01:00
John Ericson
44c8d83831 Create outputOf primop.
In the Nix language, given a drv path, we should be able to construct
another string referencing to one of its output. We can do this today
with `(import drvPath).output`, but this only works for derivations we
already have.

With dynamic derivations, however, that doesn't work well because the
`drvPath` isn't yet built: importing it like would need to trigger IFD,
when the whole point of this feature is to do "dynamic build graph"
without IFD!

Instead, what we want to do is create a placeholder value with the right
string context to refer to the output of the as-yet unbuilt derivation.
A new primop in the language, analogous to `builtins.placeholder` can be
used to create one. This will achieve all the right properties. The
placeholder machinery also will match out the `outPath` attribute for CA
derivations works.

In 60b7121d2c we added that type of
placeholder, and the derived path and string holder changes necessary to
support it. Then in the previous commit we cleaned up the code
(inspiration finally hit me!) to deduplicate the code and expose exactly
what we need. Now, we can wire up the primop trivally!

Part of RFC 92: dynamic derivations (tracking issue #6316)

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-14 09:37:37 -04:00
John Ericson
e7c39ff00b Rework evaluator SingleDerivedPath infra
`EvalState::mkSingleDerivedPathString` previously contained its own
inverse (printing, rather than parsing) in order to validate what was
parsed. Now that is pulled out into its own separate function:
`EvalState::coerceToSingleDerivedPath`.

In additional that pulled out logic is deduplicated with
`EvalState::mkOutputString` via `EvalState::mkOutputStringRaw`, which is
itself deduplicated (and generalized) with
`DownstreamPlaceholder::mkOutputStringRaw`.

All these changes make the unit tests simpler.

(We would ideally write more unit tests for `mkSingleDerivedPathString`
`coerceToSingleDerivedPath` directly, but we cannot yet do that because
the IO in reading the store path won't work when the dummy store cannot
hold anything. Someday we'll have a proper in-memory store which will
work for this.)

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-14 08:44:50 -04:00
John Ericson
a04720e68c Rename optOutputPath to optStaticOutputPath
This choice of variable name makes it more clear what is going on.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-14 08:44:48 -04:00
Robert Hensing
584ff408a4
Merge pull request #8735 from obsidiansystems/defexpr
Factor out `nix-defexpr` path computation
2023-08-11 20:32:03 +02:00
Robert Hensing
c4dbb55ba9 initLibUtil: Add exception handling self-check 2023-08-11 17:25:42 +02:00
tomberek
010dc7958e
Merge pull request #8369 from obsidiansystems/inductive-derived-path
Make the Derived Path family of types inductive for dynamic derivations
2023-08-11 08:50:22 -05:00
Yorick van Pelt
e78e9a6bd1
SimpleLogger::log: fix unintended fallthrough 2023-08-11 12:05:45 +02:00
Yorick van Pelt
2e5096e4f0
FileTransfer::download: fix use-after-move
std::move(state->data) and data.empty() were called in a loop, and
could run with no other threads intervening. Accessing moved objects
is undefined behavior, and could cause a crash.
2023-08-11 12:00:31 +02:00
Yorick van Pelt
1ffb26311b
MultiCommand::toJSON: Fix use-after-move 2023-08-11 12:00:11 +02:00
Yorick van Pelt
b9b51f9579
Prevent overriding virtual methods that are called in a destructor
Virtual methods are no longer valid once the derived destructor has
run. This means the compiler is free to optimize them to be
non-virtual.

Found using clang-tidy
2023-08-11 11:58:33 +02:00
John Ericson
60b7121d2c Make the Derived Path family of types inductive for dynamic derivations
We want to be able to write down `foo.drv^bar.drv^baz`:
`foo.drv^bar.drv` is the dynamic derivation (since it is itself a
derivation output, `bar.drv` from `foo.drv`).

To that end, we create `Single{Derivation,BuiltPath}` types, that are
very similar except instead of having multiple outputs (in a set or
map), they have a single one. This is for everything to the left of the
rightmost `^`.

`NixStringContextElem` has an analogous change, and now can reuse
`SingleDerivedPath` at the top level. In fact, if we ever get rid of
`DrvDeep`, `NixStringContextElem` could be replaced with
`SingleDerivedPath` entirely!

Important note: some JSON formats have changed.

We already can *produce* dynamic derivations, but we can't refer to them
directly. Today, we can merely express building or example at the top
imperatively over time by building `foo.drv^bar.drv`, and then with a
second nix invocation doing `<result-from-first>^baz`, but this is not
declarative. The ethos of Nix of being able to write down the full plan
everything you want to do, and then execute than plan with a single
command, and for that we need the new inductive form of these types.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-08-10 00:08:32 -04:00
Peter Waller
4b1bd822ac Try to realise CA derivations during queryMissing
This enables nix to correctly report what will be fetched in the case
that everything is a cache hit.

Note however that if an intermediate build of something which is not
cached could still cause products to end up being substituted if the
intermediate build results in a CA path which is in the cache.

Fixes #8615.

Signed-off-by: Peter Waller <p@pwaller.net>
2023-08-09 20:57:04 +01:00
Théophane Hufschmitt
d00fe5f225
Merge pull request #8805 from tweag/fix-add-to-store-existing
[V2] Fix misread of source if path is already valid
2023-08-08 14:57:45 +02:00
Ben Radford
c0e6466a1e
Merge remote-tracking branch 'origin/master' into overlayfs-store 2023-08-08 13:39:18 +01:00
Eelco Dolstra
5624777988
Merge pull request #8786 from Ma27/fix-why-depends-precise
nix/why-depends: fix output of `--precise`
2023-08-07 19:32:49 +02:00
Théophane Hufschmitt
4999f42a70
Merge pull request #8322 from tweag/stabilize-discard-references
Stabilize `discard-references`
2023-08-07 17:35:02 +02:00
Eelco Dolstra
eb1302670e
Merge pull request #8769 from edolstra/generalize-tarball-urls
Don't require .tar/.zip extension for tarball flakerefs
2023-08-07 17:02:17 +02:00
Théophane Hufschmitt
ad410abbe0 Stabilize discard-references
It has been there for a few releases now (landed in 2.14.0), doesn't
seem to cause any major issue and is wanted in a few places
(https://github.com/NixOS/nix/pull/7087#issuecomment-1544471346).
2023-08-07 16:53:37 +02:00
Théophane Hufschmitt
5df0f1755f
Merge pull request #8692 from obsidiansystems/add-another-xp-check
Feature gate `DownstreamPlaceholder::unknownCaOutput`
2023-08-07 13:11:44 +02:00
Simon Rainerson
31a6e10fe5 Fix misread of source if path is already valid
When receiving a stream of NARs through the ssh-ng protocol, an already
existing path would cause the NAR archive to not be read in the stream,
resulting in trying to parse the NAR as a ValidPathInfo. This results in
the error message:
    error: not an absolute path: 'nix-archive-1'

Fixes #6253

Usually this problem is avoided by running QueryValidPaths before
AddMultipleToStore, but can arise when two parallel nix processes gets
the same response from QueryValidPaths. This makes the problem more
prominent when running builds in parallel.
2023-08-07 10:27:40 +02:00
John Ericson
9113b4252b
Merge pull request #8760 from iFreilicht/fix-json-load-assertion-errors
Fix derivation load assertion errors
2023-08-06 17:07:43 -07:00
Felix Uhl
3fefc2b284 Fix derivation load assertion errors
When loading a derivation from a JSON, malformed input would trigger
cryptic "assertion failed" errors. Simply replacing calls to `operator []`
with calls to `.at()` was not enough, as this would cause json.execptions
to be printed verbatim.

Display nice error messages instead and give some indication where the
error happened.

*Before:*

```
$ echo 4 | nix derivation add
error: [json.exception.type_error.305] cannot use operator[] with a string argument with number

$ nix derivation show nixpkgs#hello | nix derivation add
Assertion failed: (it != m_value.object->end()), function operator[], file /nix/store/8h9pxgq1776ns6qi5arx08ifgnhmgl22-nlohmann_json-3.11.2/include/nlohmann/json.hpp, line 2135.

$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: [json.exception.type_error.302] type must be string, but is object

$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error: [json.exception.type_error.302] type must be object, but is string

```

*After:*

```
$ echo 4 | nix derivation add
error: Expected JSON of derivation to be of type 'object', but it is of type 'number'

$ nix derivation show nixpkgs#hello | nix derivation add
error: Expected JSON object to contain key 'name' but it doesn't

$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: Expected JSON value to be of type 'string' but it is of type 'number'

$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error:
       … while reading key 'outputs'

       error: Expected JSON value to be of type 'object' but it is of type 'string'
```
2023-08-05 01:34:30 +02:00
Maximilian Bosch
7c09104a94
nix/why-depends: fix output of --precise
I haven't checked when this was exactly introduced, but on Nix 2.16 I
realized that the additional lines inserted when using `--precise` are
completely separated from the tree:

    nix why-depends /nix/store/ccgr4faaxys39s091qridxg1947lggh4-evcxr-0.14.2 /nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0 --precise --extra-experimental-features nix-command
    /nix/store/ccgr4faaxys39s091qridxg1947lggh4-evcxr-0.14.2
        → /nix/store/lcf37pgp3rgww67v9x2990hbfwx96c1w-gcc-wrapper-12.2.0
            → /nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0
    └───bin/evcxr: …':'}.PATH=${PATH/':''/nix/store/lcf37pgp3rgww67v9x2990hbfwx96c1w-gcc-wrapper-12.2.0/bin'':'/':'}…
        └───bin/cpp: …k disable=SC2193.[[ "/nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0/bin/cpp" = *++ ]] &&…

This is apparently because `std::cout` is buffered and flushed in the
end whereas the rest of the output isn't. The fix is rather simple, just
use `logger->cout` as it's already the case for the rest of the code.

This way we also don't need to insert additional newlines in the `hits`
map since that's something the logger takes care of.

Also added a small test to make sure that the layout of this is somehow
tested to reduce the risk of further regressions here.
2023-08-04 23:11:08 +02:00
John Ericson
d137002e94 Add API docs for all overridden local overlay methods
These docs explain the implementation relative to the local store
originals. The original declaration of virtual methods can still be
consulted for proper interface-level documentation.
2023-08-03 17:28:18 -04:00
John Ericson
7ad16c9d12 Add some docs for the local overlay store 2023-08-03 16:10:32 -04:00
John Ericson
4d99e407fd Remove FIXME on why something doesn't work
I now know it is due to
https://github.com/llvm/llvm-project/issues/64108. The workaround is
just fine and already in use in this codebase.
2023-08-03 12:12:54 -04:00
John Ericson
4f5b01f5cd Start to document the local-overlay store 2023-08-03 12:10:13 -04:00
John Ericson
4b9a621812 Guard the local overlay store behind an experimental feature 2023-08-02 20:30:42 -04:00
John Ericson
3fc838c8a8 Merge remote-tracking branch 'layered-store/experimental-stores' into overlayfs-store 2023-08-02 19:03:34 -04:00
John Ericson
3b592c880a Add infra for experimental store implemenations
This is analogous to that for experimental settings and flags that we
have also added as of late.
2023-08-02 15:46:38 -04:00
John Ericson
6b297e5895 Make verifyAllValidPaths more functional
return map rather than mutate one passed in by reference
2023-08-02 14:38:22 -04:00
John Ericson
73c9fc7ab1 Merge remote-tracking branch 'upstream/master' into overlayfs-store 2023-08-02 14:14:11 -04:00
John Ericson
3723363697
Merge pull request #8765 from NixLayeredStore/refactor-store-verify
More cleanups for `LocalStore::verifyPath`
2023-08-02 13:39:29 -04:00
John Ericson
9b908fa70a Factor out nix-defexpr path computation
Avoid duplicated code, and also avoid "on the fly" path construction
(which makes it harder to keep track of which paths we use).

The factored out code doesn't create the Nix state dir anymore, but this
is fine because other in nix-env and nix-channel does:

- nix-channel: Line 158 in this commit

- nix-env: Line 1407 in this commit
2023-08-02 12:54:48 -04:00
John Ericson
66550878df Add comment explaining the use of readDirectory(realStoreDir) 2023-08-02 12:46:07 -04:00
John Ericson
770d50e49c local-store verifying: Rename store to something more clear
It is not a `Store` but a `StorePathSet`.
2023-08-02 12:40:04 -04:00
Eelco Dolstra
d00469ebf9
Merge pull request #8762 from obsidiansystems/split-out-eval-settings
Move evaluator settings (type and global) to separate file/header
2023-08-02 16:54:02 +02:00
Eelco Dolstra
d9e7758f47 Don't require .tar/.zip extension for tarball flakerefs
Special-casing the file name is rather ugly, so we shouldn't do
that. So now any {file,http,https} URL is handled by
TarballInputScheme, except for non-flake inputs (i.e. inputs that have
the attribute `flake = false`).
2023-08-01 16:07:20 +02:00
Ben Radford
8225b7a011
Merge branch 'master' into overlayfs-store 2023-08-01 12:52:53 +01:00
Ben Radford
c712369ec5 Document remount-hook store parameter. 2023-08-01 12:48:02 +01:00
Ben Radford
c409a753db Fix new lines in comment. 2023-08-01 12:48:02 +01:00
Ben Radford
1255866e16 Update src/libstore/local-overlay-store.hh
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-08-01 12:48:02 +01:00
Ben Radford
6a8de4c9dc Avoid enumerating entire overlay store dir upfront.
As an optimisation for LocalStore, we read all the store directory entries into
a set. Checking for membership of this set is much faster than a stat syscall.
However for LocalOverlayStore, the lower store directory is expected to contain
a vast number of entries and reading them all can take a very long time.

So instead of enumerating them all upfront, we call pathExists as needed. This
means making stat syscalls for each store path, but the upper layer is expected
to be relatively small compared to the lower store so that should be okay.
2023-08-01 12:48:02 +01:00
Ben Radford
50ce8d15eb Preparatory refactor of LocalStore::verifyStore. 2023-08-01 12:48:02 +01:00
John Ericson
6525265f46 LocalStore::verifyPath: Try to clarify data flow with more scopes
It was initially unclear to me which of these are temporary state for
the verify paths computation, and which of these are the results of that
computation to be used in the rest of the function. Now, it is clear,
and enforced.
2023-07-31 12:44:27 -04:00
John Ericson
2a5f5fbb17 LocalStore::verifyPath: Use StorePathSet for store local var
We don't care about non-store-paths in there (things like `.links`, are,
in fact, allowed). So let's just skip them up front and be more strongly
typed.
2023-07-31 12:44:18 -04:00
John Ericson
dcdd5fed74
Merge pull request #8754 from NixLayeredStore/refactor-store-verify
Refactor `verifyPath` to take `StorePath` instead of `Path`
2023-07-31 12:01:36 -04:00
Ben Radford
c9a87ce7ca Refactor verifyPath to take StorePath instead of Path.
This way we avoid having to convert from Path to StorePath and vice versa in
the body of verifyPath.
2023-07-31 11:17:55 -04:00
John Ericson
1570e80219 Move evaluator settings (type and global) to separate file/header 2023-07-31 10:14:15 -04:00
Robert Hensing
33d58a90c2 toJSON: Add attribute path to trace 2023-07-31 13:02:54 +02:00
John Ericson
621bdbdb31 Merge branch 'overlayfs-store' into delete 2023-07-26 12:01:08 -04:00
Ben Radford
c2d54496a0 Forgot to check flag and early out. 2023-07-26 16:01:20 +01:00
Ben Radford
3a9fe1a085 Made remountRequired atomic to avoid concurrency issues. 2023-07-26 16:01:20 +01:00
Ben Radford
5744a500d6 Use debug instead of writing directly to stderr. 2023-07-26 16:01:20 +01:00
Ben Radford
ed14286924 Invoke remount-hook program when necessary. 2023-07-26 16:01:20 +01:00
Ben Radford
33ebae75ca Reuse deletion logic for optimiseStore and rename method. 2023-07-26 16:01:20 +01:00
Ben Radford
11c493f8fa Avoid creating whiteout for duplicate store paths. 2023-07-26 16:01:20 +01:00
Ben Radford
d9688ba708 Add new remount-hook store parameter. 2023-07-26 16:01:20 +01:00
John Ericson
07b34edc44 Fix deletion test
Lower layer references are ignored for deleting just in the upper layer.
2023-07-25 18:09:02 -04:00
Alex Ameen
2d1d81114d
Add parseFlakeRef and flakeRefToString builtins (#8670)
Over the last year or so I've run into several use cases where I need to
parse and/or serialize URLs for use by `builtins.fetchTree` or
`builtins.getFlake`, largely in order to produce _lockfile-like_ files
for lang2nix frameworks or tools which use `nix` internally to drive
builds.

I've gone through the painstaking process of emulating
`nix::FlakeRef::fromAttrs` and `nix::parseFlakeRef` several times with
mixed success; but these are difficult to create and even harder to
maintain if I hope to stay aligned with changes to the real
parser/serializer.

I understand why adding new `builtins` isn't something we want to do
flagrantly. I'm recommending this addition simply because I keep
encountering use cases where I need to parse/serialize these URIs in
`nix` expressions, and I want a reliable solution.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-07-25 17:43:33 +00:00
John Ericson
9ef0a9e8aa Fix hard linking issue causing overlay fs copy-ups 2023-07-25 10:28:11 -04:00
John Ericson
73a55e4005 Merge remote-tracking branch 'layered-store/overlayfs-store' into gc 2023-07-25 09:48:06 -04:00
Ben Radford
497464f494
Extend verify test to check that repair is supported. 2023-07-25 13:33:37 +01:00
John Ericson
0e595a52a3 Remove trailing whitespace 2023-07-24 15:39:59 -04:00
John Ericson
2cabf85b53 Merge remote-tracking branch 'upstream/master' into overlayfs-store 2023-07-24 15:39:36 -04:00
John Ericson
60d8dd7aea Clean up store hierarchy with IndirectRootStore
See the API doc comments for details.
2023-07-24 09:19:44 -04:00
John Ericson
13269ba93b Make RemoteStore::ConnectionHandle part of class and expose
Will need to do subclass-specific implementations in the next commit.
This isn't because there will be multiple variations of the daemon
protocol (whew!) but because different clients pick and choose different
parts to use.
2023-07-24 09:17:01 -04:00
John Ericson
0a30b07277 Move Store::Params typedef to StoreConfig::Params
This is because `StoreConfig` also uses it.
2023-07-24 09:03:44 -04:00
John Ericson
4685c9b55f
Merge pull request #8664 from ncfavier/merge-dynamic-attrs
parser: merge nested dynamic attributes
2023-07-23 11:03:38 -04:00
Naïm Favier
570a1a3ad7
parser: merge nested dynamic attributes
Fixes https://github.com/NixOS/nix/issues/7115
2023-07-21 17:14:03 +02:00
John Ericson
fe1fbdb5a1
Merge pull request #8724 from obsidiansystems/queryPartialDerivationOutputMap-evalStore
Give `queryPartialDerivationOutputMap` an `evalStore` parameter
2023-07-21 08:53:18 -04:00
Eelco Dolstra
7ac24d9525
Merge pull request #8650 from obsidiansystems/content-address-simpler
Simplify `ContentAddress`
2023-07-21 13:46:53 +02:00
John Ericson
6bc98c7fba Give queryPartialDerivationOutputMap an evalStore parameter
This makes it more useful. In general, the derivation will be in one
store, and the realisation info is in another.

This also helps us avoid duplication. See how `resolveDerivedPath` is
now simpler because it uses `queryPartialDerivationOutputMap`. In #8369
we get more flavors of derived path, and need more code to resolve them
all, and this problem only gets worse.

The fact that we need a new method to deal with the multiple dispatch is
unfortunate, but this generally relates to the fact that `Store` is a
sub-par interface, too bulky/unwieldy and conflating separate concerns.
Solving that is out of scope of this PR.

This is part of the RFC 92 work. See tracking issue #6316
2023-07-20 15:59:52 -04:00
John Ericson
f62543fe1c Remove unneeded copy
It appeared in 8eb73a8724 (by me!) without
justification.
2023-07-20 15:42:06 -04:00
John Ericson
7b30293d38 Tighten #includes: DerivedPath doesn't care about Realisation 2023-07-20 15:42:06 -04:00
Ben Radford
8ddbcb736a
Implement overlay store deduplication. 2023-07-20 10:42:39 +01:00
Ben Radford
614efc1240
Add test for store optimise path deduplication. 2023-07-20 10:42:39 +01:00
Ben Radford
d5cd74a401
Override verifyStore to always pass NoRepair for LocalOverlayStore. 2023-07-20 10:42:39 +01:00
Valentin Gagarin
0e4f6dfcf7 revert anchor prefix for builtin constants
the original change broke many pre-existing anchor links.

also change formatting of the constants listing slightly:
- the type should not be part of the anchor
- add highlight to the "impure only" note
2023-07-20 10:27:38 +02:00
Ben Radford
21b9e15d25
Merge branch 'master' into overlayfs-store 2023-07-19 15:00:47 +01:00
Robert Hensing
32494cbb29
Merge pull request #7973 from fricklerhandwerk/remove-channels
remove the Channels section
2023-07-19 14:02:26 +02:00
Valentin Gagarin
2fa90e5824 add more details on CA derivations 2023-07-19 13:59:18 +02:00
Valentin Gagarin
5f37ebcf83 document all special system features and their behavior 2023-07-19 13:34:03 +02:00
Robert Hensing
eca07b2a47
Merge pull request #8315 from fricklerhandwerk/doc-system
add information on the system type string
2023-07-19 12:07:23 +02:00
Valentin Gagarin
493ddf617f reformat system-features setting documentation 2023-07-19 11:07:47 +02:00
Valentin Gagarin
feb01b22ed add links to store API documentation 2023-07-19 11:07:47 +02:00
Valentin Gagarin
6c3cd429a6 fix broken links 2023-07-19 11:01:48 +02:00
Valentin Gagarin
0779005f49 expand on the extra-platforms option 2023-07-19 10:57:37 +02:00
Valentin Gagarin
fcadac0a02 mention extra-platforms 2023-07-19 10:38:42 +02:00
Valentin Gagarin
aba32def73 fix wording
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-07-19 10:38:30 +02:00
Valentin Gagarin
32de11923e add cross-links 2023-07-19 10:37:40 +02:00
Valentin Gagarin
3763c7bb5e shorten system setting description 2023-07-19 10:37:40 +02:00
Valentin Gagarin
c8a42039ea move docs of the current system to the system setting
add information what happens when Nix itself is cross-compiled
2023-07-19 10:37:40 +02:00
Valentin Gagarin
0751c1bfc6 one line per sentence for easier review 2023-07-19 10:37:40 +02:00
Robert Hensing
0e3a7e34a0
Merge pull request #8506 from corngood/ssh-master
Pass NIX_SSHOPTS when checking for an ssh master connection.
2023-07-18 15:47:57 +02:00
Théophane Hufschmitt
a8d5bb5e7e
Merge pull request #8342 from NixLayeredStore/best-effort-supplementary-groups
Best effort supplementary groups
2023-07-17 20:58:17 +02:00
John Ericson
0f7242ff87 Test nested sandboxing, and make nicer error
We were bedeviled by sandboxing issues when working on the layered
store. The problem ended up being that when we have nested nix builds,
and the inner store is inside the build dir (e.g. store is
`/build/nix-test/$name/store`, build dir is `/build`) bind mounts
clobber each other and store paths cannot be found.

After thoroughly cleaning up `local-derivation-goal.cc`, we might be
able to make that work. But that is a lot of work. For now, we just fail
earlier with a proper error message.

Finally, test this: nested sandboxing without the problematic store dir
should work, and with should fail with the expected error message.

Co-authored-by: Dylan Green <67574902+cidkidnix@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-07-14 09:40:24 -04:00
cidkidnix
04d5aa02e6 Merge branch 'best-effort-supplementary-groups' into overlayfs-store 2023-07-13 14:33:52 -05:00
John Ericson
caabc4f648 Feature gate DownstreamPlaceholder::unknownCaOutput
This is a part of CA derivations that we forgot to put behind the
experimental feature.

This was caught by @fricklerhandwerk in
https://github.com/NixOS/nix/pull/8369#discussion_r1258133719
2023-07-13 07:56:33 -04:00
John Ericson
2c3fb0eb33 Move BuiltPath to its own header/C++ file in libcmd
It is less important, and used less widely, than `DerivedPath`.
2023-07-12 23:01:10 -04:00
Ben Radford
a2acd23466
Update src/libstore/globals.hh
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-12 12:33:05 +01:00
Ben Radford
0309f6b5b8
Update src/libstore/globals.hh
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-12 12:32:57 +01:00
Ben Radford
b8e8dfc3e8
Say a bit about default value in setting description. 2023-07-11 11:25:49 +01:00
Ben Radford
a193ec4052
Default should depend on whether we are root. 2023-07-11 11:14:13 +01:00
Ben Radford
2b4c59dd99
Be clearer about the security implications. 2023-07-11 11:09:25 +01:00
Ben Radford
0caf28f238
Update description for require-drop-supplementary-groups. 2023-07-11 10:57:14 +01:00
Ben Radford
07dabcc90e
Always attempt setgroups but allow failure to be ignored. 2023-07-11 10:44:05 +01:00
Ben Radford
25b20b4ad2
Merge remote-tracking branch 'origin/master' into best-effort-supplementary-groups 2023-07-11 09:38:34 +01:00
John Ericson
4a880c3cc0
Merge pull request #8579 from obsidiansystems/findPath-cleanup-2
Further search path cleanups
2023-07-10 09:59:01 -04:00
Eelco Dolstra
fea7d3b1cd
Merge pull request #8681 from inclyc/libexpr/parser-move-noeffect
libexpr: remove std::move() for `basePath` in parser, it has no effect
2023-07-10 13:30:42 +02:00
Bader AlAttar
3fa0266e7a
Fix some grammar in installables doc (#8682) 2023-07-10 09:33:04 +00:00
Yingchi Long
3d74e7b811 libexpr: remove std::move() for basePath in parser, it has no effect 2023-07-10 12:02:29 +08:00
John Ericson
028b26a77f
Merge pull request #8370 from hercules-ci/fetchClosure-input-addressed
`fetchClosure`: input addressed and pure
2023-07-09 23:41:22 -04:00
John Ericson
be518e73ae Clean up SearchPath
- Better types

- Own header / C++ file pair

- Test factored out methods

- Pass parsed thing around more than strings

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-07-09 23:22:22 -04:00
John Ericson
87dcd09047 Clean up resolveSearchPathElem
We should use `std::optional<std::string>` not `std::pair<bool,
std::string>` for an optional string.
2023-07-09 23:13:30 -04:00
John Ericson
2add230939 Fix build 2023-07-09 21:53:06 -04:00
John Ericson
28398e6d02 Merge remote-tracking branch 'upstream/master' into overlayfs-store 2023-07-09 20:30:23 -04:00
John Ericson
903700c5e1 Simplify ContentAddress
Whereas `ContentAddressWithReferences` is a sum type complex because different
varieties support different notions of reference, and
`ContentAddressMethod` is a nested enum to support that,
`ContentAddress` can be a simple pair of a method and hash.

`ContentAddress` does not need to be a sum type on the outside because
the choice of method doesn't effect what type of hashes we can use.

Co-Authored-By: Cale Gibbard <cgibbard@gmail.com>
2023-07-07 07:30:01 -04:00
Robert Hensing
b4b02d084f fetchClosure: Interleave the examples in the docs 2023-07-07 11:40:40 +02:00
Robert Hensing
537e8beb77
fetchClosure: Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-07 11:00:40 +02:00
Eelco Dolstra
6db66ebfc5
Merge pull request #8631 from iFreilicht/profile-list
Profile list improvements
2023-07-05 16:06:17 +02:00
Théophane Hufschmitt
82d6699976
Document the path flakeref format (#8640)
* Document the path flakeref format

Fix https://github.com/NixOS/nix/issues/8482

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-04 08:39:08 +00:00
Eelco Dolstra
a353412c43 nix profile list: Add --json flag
This just dumps the profile manifest to stdout.
2023-07-03 12:23:57 +02:00
Eelco Dolstra
b8e8f27159 Rename 'resolvedRef' to 'lockedRef'
'resolvedRef' was incorrect, since a resolved ref is one after
registry resolution, which may still be unlocked (e.g. 'nixpkgs' ->
'github:NixOS/nixpkgs').
2023-07-03 12:23:57 +02:00
Yingchi Long
3c90340fe6 libexpr: use thread_local to make the parser thread-safe
If we call `adjustLoc`, the global variable `prev_yylloc` is shared
between threads and racy.

Currently, nix itself does not concurrently parsing files, but this is
helpful for libexpr users. (The parser is thread-safe except this.)
2023-07-03 16:05:43 +08:00
Eelco Dolstra
87b82db881 nix profile list: Improve readability of the output 2023-07-02 16:17:09 +02:00
Robert Hensing
7b39a388b3
Merge pull request #8566 from inclyc/nixd/value-print-depth
libexpr: extend `Value::print` to allow limited depth
2023-07-01 20:08:52 +02:00
Robert Hensing
40052c7613 fetchClosure: Docs and error message improvements
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-30 18:23:42 +02:00
Robert Hensing
50de11d662 doc: Improve fetchClosure documentation 2023-06-30 18:23:24 +02:00
Robert Hensing
dc79636007 fetchClosure: Refactor: replace enableRewriting
A single variable is nice and self-contained.
2023-06-30 18:22:47 +02:00
Robert Hensing
5bdca46117 fetchClosure: Split into three cases 2023-06-30 18:22:47 +02:00
Robert Hensing
55888633dd makeContentAddressed: Add single path helper 2023-06-30 18:22:47 +02:00
Robert Hensing
8dca95386c fetchClosure: Disallow toPath for inputAddressed = true 2023-06-30 18:22:47 +02:00
Robert Hensing
508aa58e67 fetchClosure: Always check that inputAddressed matches the result 2023-06-30 18:22:47 +02:00
Robert Hensing
ea30f152b7 fetchClosure: Allow input addressed paths in pure mode
When explicitly requested by the caller, as suggested in the meeting
(https://github.com/NixOS/nix/pull/8090#issuecomment-1531139324)

> @edolstra: { toPath } vs { fromPath } is too implicit

I've opted for the `inputAddressed = true` requirement, because it
we did not agree on renaming the path attributes.

> @roberth: more explicit
> @edolstra: except for the direction; not immediately clear in which direction the rewriting happens

This is in fact the most explicit syntax and a bit redundant, which is
good, because that redundancy lets us deliver an error message that
reminds expression authors that CA provides a better experience to
their users.
2023-06-30 18:22:43 +02:00
Robert Hensing
7e5b6d2c45 fetchClosure: Refactor: rename toCA -> enableRewriting 2023-06-30 18:19:56 +02:00
Robert Hensing
0f6d596df5 fetchClosure: Factor out attribute hint 2023-06-30 18:19:56 +02:00
Yueh-Shun Li
eebfe989a5 linkOrCopy: Fallback upon cross-device link error (EXDEV)
Fix building derivations in local chroot store on OpenAFS,
where hard linking accross directories causes cross-device link error
(EXDEV).
2023-06-30 21:12:26 +08:00
Eelco Dolstra
a0c617348b
Merge pull request #8589 from jfroche/sign-paths-as-allowed-user
Allow to sign path as unprivileged user
2023-06-30 13:13:42 +02:00
Robert Hensing
1632f08ea2
Merge pull request #8600 from inclyc/libexpr/fix-leaking-in-stripIndentation
libexpr: fix leaking `es2` in stripIndentation (parser.y)
2023-06-29 11:31:53 +02:00
Yingchi Long
3468cbaf47 libexpr: fix leaking es2 in stripIndentation (parser.y) 2023-06-28 22:38:44 +08:00
Jean-François Roche
80c9259756 Allow to sign path as unprivileged user
User can now sign path as unprivileged/allowed user

refs #1708
2023-06-27 18:31:31 +02:00
John Ericson
22b278e011 Automatically document builtin constants
This is done in roughly the same way builtin functions are documented.

Also auto-link experimental features for primops, subsuming PR #8371.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-27 09:37:54 -04:00
John Ericson
e8067daf09 Generialize showType 2023-06-27 09:11:42 -04:00
Maximilian Bosch
559fd7ffe7
nix flake check: improve error message if overlay is not a lambda (#8582)
* nix flake check: improve error message if overlay is not a lambda

Suppose you have an overlay like this

    {
      inputs = { /* ... */ };
      outputs = { flake-utils, ... }: flake-utils.lib.eachDefaultSystem
        (system: {
          overlays.default = final: prev: {

          };
        });
    }

then `nix flake check` (correctly) fails because `overlays` are supposed
to have the structure `overlays.<name> = final: prev: exp`. However, the
error-message is a little bit counter-intuitive:

    error: overlay does not take an argument named 'final'

While one might guess where the error actually comes from because the
trace above says `… while checking the overlay 'overlays.x86_64-linux'`
this is still pretty confusing because it complains about an argument
not being named `final` even though that's evidently the case.

With this change, the error-message actually makes it clear what's
wrong:

    [ma27@carsten:~/Projects/nix/tmp]$ nix flake check --extra-experimental-features 'nix-command flakes' path:$(pwd)
    error:
           … while checking flake output 'overlays'

             at /nix/store/clgblnxx003hyrq8qkz5ab6kgqkck6qc-source/flake.nix:4:5:

                3|   outputs = { ... }: {
                4|     overlays.x86_64-linux.snens = final: prev: {
                 |     ^
                5|       kek = throw "snens";

           … while checking the overlay 'overlays.x86_64-linux'

             at /nix/store/clgblnxx003hyrq8qkz5ab6kgqkck6qc-source/flake.nix:4:5:

                3|   outputs = { ... }: {
                4|     overlays.x86_64-linux.snens = final: prev: {
                 |     ^
                5|       kek = throw "snens";

           error: overlay is not a lambda, but a set instead
2023-06-27 14:58:29 +02:00
Eelco Dolstra
8e770dac9f
Merge pull request #8576 from obsidiansystems/findPath-cleanup
Use a struct not `std::pair` for `SearchPathElem`
2023-06-26 12:37:06 +02:00
Michal Sojka
f11445952f Document builtins.fetchTree
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

Supersedes #6740
2023-06-26 00:59:06 +02:00
Théophane Hufschmitt
60f06a1714
Merge pull request #5385 from Enzime/add/dirty-rev
Add `dirtyRev` and `dirtyShortRev` to `fetchGit`
2023-06-24 14:55:31 +02:00
Michael Hoang
a7b49086c7 Add dirtyRev and dirtyShortRev to fetchGit
Fixes #4682
2023-06-24 14:17:25 +10:00
John Ericson
484290a9e0 Use a struct not std::pair for SearchPathElem
I got very confused trying to keep all the `first` and `second` straight
reading the code, *especially* as there is also another `(boolean,
string)` pair type also being used.

Named fields is much better.

There are other cleanups that we can do (for example, the existing
TODO), but we can do them later. Doing them now would just make this
harder to review.
2023-06-23 12:01:10 -04:00
Robert Hensing
fd4f03b8fd
Merge pull request #8519 from fricklerhandwerk/reword-trusted-users
reword documentation on trusted users and substituters
2023-06-23 13:08:46 +02:00
Eelco Dolstra
8350f06d6c
Merge pull request #8574 from inclyc/nixd/remove-unused-token
libexpr: remove unused token `ATTRPATH` in token declaration
2023-06-23 12:02:40 +02:00
Yingchi Long
9d8c4ac446 libexpr: remove unused token ATTRPATH in token declaration 2023-06-23 13:35:41 +08:00
John Ericson
97df060588 Better document build failure exit codes
- Improved API docs from comment

- Exit codes are for `nix-build`, not just `nix-store --release`

- Make note in tests so the magic numbers are not surprising

Picking up where #8387 left off.
2023-06-22 14:29:45 -04:00
Yingchi Long
1400fde144 libexpr: extend Value::print to allow limited depth 2023-06-22 18:28:30 +08:00
Yingchi Long
c48277c1c1 libexpr: add tests for nix::Value::print 2023-06-22 18:28:30 +08:00
John Ericson
48fe0ed554
Merge pull request #8374 from obsidiansystems/improve-path-setting
Split `OptionalPathSetting` from `PathSetting`
2023-06-21 15:40:43 -04:00
John Ericson
9c0473120f Merge remote-tracking branch 'upstream/master' into overlayfs-store 2023-06-21 10:15:13 -04:00
Valentin Gagarin
e91d19db5f
be more serious about security risks with trusted users 2023-06-20 12:23:53 +02:00
Eelco Dolstra
b1352700c4
Merge pull request #8552 from edolstra/fix-eagain
GC server: Clear O_NONBLOCK on the right file descriptor
2023-06-20 12:12:53 +02:00
Ben Radford
6ae35534b7
Support opening local store with database on read-only filesystem (#8356)
Previously it was not possible to open a local store when its database is on a read-only filesystem. Obviously a store on a read-only filesystem cannot be modified, but it would still be useful to be able to query it.

This change adds a new read-only setting to LocalStore. When set to true, Nix will skip operations that fail when the database is on a read-only filesystem (acquiring big-lock, schema migration, etc), and the store database will be opened in immutable mode.

Co-authored-by: Ben Radford <benradf@users.noreply.github.com>
Co-authored-by: cidkidnix <cidkidnix@protonmail.com>
Co-authored-by: Dylan Green <67574902+cidkidnix@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-20 11:34:09 +02:00
Eelco Dolstra
a6a75ecad8 GC server: Clear O_NONBLOCK on the right file descriptor
The bug fix in 6d30f9e6fe erroneously
cleared O_NONBLOCK on the server rather than client FD (leaving both
in an incorrect state).

Fixes #8551.
2023-06-20 11:19:14 +02:00
Eelco Dolstra
c5fdbdae32 LocalStore::addTempRoot(): Handle ENOENT
If the garbage collector has acquired the global GC lock, but hasn't
created the GC socket yet, then a client attempting to connect would
get ENOENT. Note that this only happens when the GC runs for the first
time on a machine. Subsequently clients will get ECONNREFUSED which
was already handled.

Fixes #7370.
2023-06-20 11:14:55 +02:00
John Ericson
3859cf6b21 Remove unused #include from local-derivation-goal.cc
These were never needed for this file, and date back to before this was
split from `derivation-goal.cc`.
2023-06-19 12:18:04 -04:00
John Ericson
9f69b7dee9 Create worker_proto::{Read,Write}Conn
Pass this around instead of `Source &` and `Sink &` directly. This will
give us something to put the protocol version on once the time comes.

To do this ergonomically, we need to expose `RemoteStore::Connection`,
so do that too. Give it some more API docs while we are at it.
2023-06-19 12:08:23 -04:00
John Ericson
4e8b495ad7 Likewise namespace and enum struct-ify ServeCommand
The motivation is exactly the same as for the last commit. In addition,
this anticipates us formally defining separate serialisers for the serve
protocol.
2023-06-19 12:08:23 -04:00
John Ericson
95eae0c002 Put worker protocol items inside a WorkerProto struct
See API docs on that struct for why. The pasing as as template argument
doesn't yet happen in that commit, but will instead happen in later
commit.

Also make `WorkerOp` (now `Op`) and enum struct. This led us to catch
that two operations were not handled!

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-06-19 12:08:23 -04:00
John Ericson
469d06f9bc Split out worker protocol template definitions from declarations
This is generally a fine practice: Putting implementations in headers
makes them harder to read and slows compilation. Unfortunately it is
necessary for templates, but we can ameliorate that by putting them in a
separate header. Only files which need to instantiate those templates
will need to include the header with the implementation; the rest can
just include the declaration.

This is now documenting in the contributing guide.

Also, it just happens that these polymorphic serializers are the
protocol agnostic ones. (Worker and serve protocol have the same logic
for these container types.) This means by doing this general template
cleanup, we are also getting a head start on better indicating which
code is protocol-specific and which code is shared between protocols.
2023-06-19 11:45:59 -04:00
Ben Radford
71f3bad749
Merge branch 'read-only-local-store' into overlayfs-store 2023-06-19 16:12:04 +01:00
Ben Radford
4e72b8483e
Update src/libstore/sqlite.hh
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-06-19 16:01:43 +01:00
Ben Radford
0b3fa995e5
Merge branch 'master' into read-only-local-store 2023-06-19 14:09:20 +01:00
Ben Radford
ba492a98ba
Update src/libstore/local-store.hh
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-19 14:07:31 +01:00
Ben Radford
b09baa3bc3
Link to LocalStore section of nix3-help-stores section. 2023-06-19 13:57:10 +01:00
Ben Radford
ef40448b1c
Remove redundant description on experimental flag. 2023-06-19 13:52:32 +01:00
Ben Radford
feb8d552ae
Update src/libstore/local-store.hh
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-19 13:22:41 +01:00
Eelco Dolstra
f5e620bf2b
Merge pull request #8483 from edolstra/save-root
restoreMountNamespace(): Restore the original root directory
2023-06-19 12:54:05 +02:00
Eelco Dolstra
8f6255b86a
Merge pull request #8524 from amjoseph-nixpkgs/pr/doc/linkify-allowed-uris
src/libexpr/eval.hh: add link for allowed-uris option
2023-06-19 06:59:07 +02:00
John Ericson
c404623a1d
Clean up a few things related to profiles (#8526)
- Greatly expand API docs

- Clean up code in misc ways

  - Instead of a complicated single loop on generations, do different
    operations in successive subsequent steps.

  - Avoid `ref` in one place where `&` is fine

  - Just return path instead of mutating an argument in `makeName`

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-19 04:04:59 +00:00
Adam Joseph
6b06e97bde src/libexpr/eval.hh: add link for allowed-uris option
This commit adds a link to the documentation for `--option
allowed-uris` where that option is mentioned while describing
`restrict-eval`.
2023-06-18 23:36:32 -04:00
John Ericson
d2ce2e89b1 Split OptionalPathSetting from PathSetting
Rather than doing `allowEmpty` as boolean, have separate types and use
`std::optional`. This makes it harder to forget the possibility of an
empty path.

The `build-hook` setting was categorized as a `PathSetting`, but
actually it was split into arguments. No good! Now, it is
`Setting<Strings>` which actually reflects what it means and how it is
used.

Because of the subtyping, we now also have support for
`Setting<std::optional<String>>` in general. I imagine this can be used
to clean up many more settings also.
2023-06-18 23:31:18 -04:00
John Ericson
c8825e9d8c Create nlohmann serializers for std::optional and use
This is somewhat tricky.
2023-06-18 23:31:10 -04:00
Eelco Dolstra
e503eadafc
Merge pull request #8477 from edolstra/tarball-flake-redirects
Tarball flake improvements
2023-06-16 18:03:50 +02:00
Valentin Gagarin
f695a74751
Update src/libstore/globals.hh 2023-06-16 17:58:01 +02:00
Valentin Gagarin
baef05e6fe
fix typo
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-06-16 14:37:08 +02:00
Valentin Gagarin
126eea48e3
do not refer to trusted-users another time 2023-06-16 14:36:53 +02:00
Valentin Gagarin
1a8ca85d48
use "store URLs" consistently 2023-06-16 14:34:11 +02:00
Valentin Gagarin
4a33d5fe35
fix link text
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-06-16 14:33:31 +02:00
Eelco Dolstra
713836112c
Merge pull request #8517 from hercules-ci/fix-build-hook-error-for-lib-users
Fix build hook error for libstore library users
2023-06-16 13:20:50 +02:00
Théophane Hufschmitt
b2247ef4f6 Don't assume the type of string::size_type
The code accidentally conflated `std::string::size_type` and `long unsigned int`.
This was fine on 64bits machines where they are apparently the same in
practice, but not on 32bits. Fix that by using `std::string::size_type`
everywhere.
2023-06-15 21:24:14 +02:00
Valentin Gagarin
2ceacce484
Update src/libstore/globals.hh 2023-06-15 15:57:54 +02:00
Ben Radford
abb3bb7133
Merge branch 'master' into read-only-local-store 2023-06-15 13:42:14 +01:00
Ben Radford
04d8f202a7
Merge branch 'read-only-local-store' into overlayfs-store 2023-06-15 13:37:57 +01:00
Ben Radford
f5d83a8029
One line per sentence in markdown docs. 2023-06-15 13:36:28 +01:00
Ben Radford
f2fe9822c1
Comment explaining what schema version 0 means. 2023-06-15 13:34:42 +01:00
Ben Radford
4642b60afe
Update src/libstore/local-store.hh
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-15 13:33:26 +01:00
Ben Radford
a7b1b92d81
Update src/libstore/local-store.hh
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-15 13:32:56 +01:00
Ben Radford
984b01924a
Update src/libstore/local-store.cc
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-15 13:32:35 +01:00
Ben Radford
78e2f931d0
Update src/libstore/local-store.cc
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-15 13:32:16 +01:00
Robert Hensing
d2696cdd1e Fix build hook error for libstore library users
A library shouldn't require changes to the caller's argument handling,
especially if it doesn't have to, and indeed we don't have to.

This changes the lookup order to prioritize the hardcoded path to nix
if it exists. The static executable still finds itself through /proc
and the like.
2023-06-15 14:32:00 +02:00
Ben Radford
264b644c53
More detail on why read-only mode disables locking. 2023-06-15 13:22:17 +01:00
John Ericson
e1fa48f17c
Update src/nix/daemon.cc
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-15 07:41:37 -04:00
Adam Joseph
098fbf6273 src/libexpr/eval.hh: fix typo
The option name is `allowed-uris`, not `allowed-uri`.
2023-06-14 21:47:58 -07:00
Valentin Gagarin
b7d47e1d22
fix wording 2023-06-15 04:58:07 +02:00
Valentin Gagarin
e09b40e0d0 reword documentation on trusted users and substituters
this is to make it slightly easier to scan over
2023-06-15 02:19:13 +02:00
John Ericson
ca9f544160
Merge pull request #8516 from wentasah/remove-registerprimop
Remove RegisterPrimOp constructor without support for documentation
2023-06-15 01:13:46 +02:00
Valentin Gagarin
a1cf16563f
Fixup description of substituters (#8291)
Introduce what substituters "are" in the configuration option entry.
Remove arbitrary line breaks for easier editing in the future.
Link glossary some more.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-06-14 20:49:58 +00:00
Michal Sojka
a0c4d58549 Remove RegisterPrimOp constructor without support for documentation
The remaining constructor RegisterPrimOp::RegisterPrimOp(Info && info)
allows specifying the documentation in .args and .doc members of the
Info structure.

Commit 8ec1ba0210 removed all uses of the removed constructor in the
nix binary. Here, we remove the constructor completely as well as its
use in a plugin test. According to #8515, we didn't promis to maintain
compatibility with external plugins.

Fixes #8515
2023-06-14 22:37:52 +02:00
John Ericson
05eb06a1de
Merge pull request #8490 from flox/stdin_handling
fix: Do not apply default installables when using --stdin
2023-06-14 20:41:22 +02:00
John Ericson
e377746842
Merge pull request #8491 from wentasah/builtins-doc
Document several undocumented builtin functions
2023-06-14 20:32:37 +02:00
Daniel Asaturov
468add5aa0
Remove dead code (#8504)
`filesystem.cc` is the only place where `createSymlink()` is used with three arguments:
in the definition of `replaceSymlink()` with three parameters that _is not used at all_.

Closes #8495
2023-06-14 14:09:11 -04:00
John Ericson
ff905cb796
Merge pull request #4803 from ShamrockLee/nix-channel-list-generations
Add `nix-channel --list-generations`
2023-06-14 18:30:35 +02:00
John Ericson
61a3e1f2e2
Merge pull request #4282 from tweag/fix-ca-hash-rewriting
fix the hash rewriting for ca-derivations
2023-06-14 18:25:00 +02:00
John Ericson
32147c8c7b Merge remote-tracking branch 'upstream/master' into overlayfs-store 2023-06-14 10:29:55 -04:00
Michal Sojka
c6d7c4f9ec Document fromTOML, hasContext and getContext builtins
Until now, these functions were completely missing in the Nix manual.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-13 21:53:03 +02:00
Eelco Dolstra
1ad3328c5e Allow tarball URLs to redirect to a lockable immutable URL
Previously, for tarball flakes, we recorded the original URL of the
tarball flake, rather than the URL to which it ultimately
redirects. Thus, a flake URL like
http://example.org/patchelf-latest.tar that redirects to
http://example.org/patchelf-<revision>.tar was not really usable. We
couldn't record the redirected URL, because sites like GitHub redirect
to CDN URLs that we can't rely on to be stable.

So now we use the redirected URL only if the server returns the
`x-nix-is-immutable` or `x-amz-meta-nix-is-immutable` headers in its
response.
2023-06-13 14:17:45 +02:00
Eelco Dolstra
3402b650cd Add a generic check for rev attribute mismatches 2023-06-13 14:13:50 +02:00
David McFarland
d5e1eb20a2 Pass common ssh options in isMasterRunning 2023-06-13 00:55:34 -03:00
Tom Bereknyei
0e3849dc65 test: add test for non-defaulting for stding installable input 2023-06-12 08:40:17 -04:00
Eelco Dolstra
87c66f6b0f Don't include uds-remote-store.md from a header file
Closes #8484.
2023-06-12 11:10:55 +02:00
Michal Sojka
8ec1ba0210 Register all PrimOps via the Info structure
This will allow documenting them (in later commits).

Note that we keep the old constructor even if it is no longer used by
Nix code, because it is used in tests/plugins/plugintest.cc, which
suggests that it might be used by some external plugin.
2023-06-11 21:40:43 +02:00
Tom Bereknyei
08089fdd32 fix: Do not apply default installables when using --stdin 2023-06-11 13:33:38 -04:00
Eelco Dolstra
e54538c461 restoreMountNamespace(): Restore the original root directory
This is necessary when we're in a chroot environment, where the
process root is not the same as the root of the mount namespace
(e.g. in nixos-enter).

Fixes #7602.
2023-06-09 16:09:29 +02:00
Eelco Dolstra
381a32981b
Merge branch 'master' into angerman/mac-fix-recursive-nix 2023-06-09 13:06:47 +02:00
Andrea Bedini
3c78920f73
Parse TOML timestamps (#8120)
Currently `fromTOML` throws an exception when encountering a timestamp
since the Nix language lacks a way to represent them.

This patch changes this beaviour and makes `fromTOML` parse timestamps as
attrsets of the format

  { _type = "timestamp"; value = "1979-05-27T07:32:00Z"; }

This is guarded by an experimental feature flag to leave room for iterating on the representation.
2023-06-09 11:53:18 +02:00
Ben Radford
ee1241da86 Remove unnecessary overrides of add methods. 2023-06-06 12:21:17 +01:00
Ben Radford
8a9baa0a30 More sensible to have deleteGCPath in LocalStore. 2023-06-06 12:21:17 +01:00
Ben Radford
a48acfd684 Skip deletion of lower paths for overlay store GC. 2023-06-06 12:21:17 +01:00
Ben Radford
98edbb9686 Factor out GC path deletion so it can be overridden. 2023-06-06 12:21:17 +01:00
Eelco Dolstra
b56319a139
Merge pull request #8391 from aneeshusa/remove-wrong-default-value-in-docs-for-hashed-mirrors
Remove old default from docs for `hashed-mirrors`
2023-06-06 12:54:28 +02:00
Eelco Dolstra
864ab1a7ac
Merge pull request #8438 from zhaofengli/armv5tel-linux32
Also set the PER_LINUX32 personality flag on armv5tel-linux
2023-06-06 12:47:16 +02:00
Eelco Dolstra
51b3576855
Merge pull request #8456 from emilytrau/include-select
Add missing <sys/select.h> include
2023-06-06 12:45:51 +02:00
Ben Radford
b852bdb3f8
Merge branch 'read-only-local-store' into overlayfs-store 2023-06-06 11:09:26 +01:00
Ben Radford
c47f744e05
Also skip makeStoreWritable when read-only=true. 2023-06-06 11:07:55 +01:00
Ben Radford
e484278e9b
Merge branch 'master' into read-only-local-store 2023-06-06 11:07:36 +01:00
Eelco Dolstra
f5c6b29940 Fix SourcePath::resolveSymlinks()
This fixes handling of symlinks that start with '..', and symlink
targets that contain symlinks themselves.
2023-06-06 11:24:10 +02:00
Emily Trau
b37dd43db4 Add missing <sys/select.h> include
`select()` may not be ambiently available for use on every platform
2023-06-05 20:18:24 -07:00
figsoda
b64450ed4e Fix code block formatting in man page 2023-06-04 14:50:47 -04:00
Alexander Bantyev
a15b2c01c0
Document manual migration for use-xdg-base-directories (#8044)
* Document manual migration for use-xdg-base-directories

As there's currently no automatic migration for use-xdg-base-directories
option, add instructions for manual migration to the option's
description.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-02 15:38:17 +00:00
Zhaofeng Li
48632e6139 Also set the PER_LINUX32 personality flag on armv5tel-linux 2023-06-02 03:04:13 -06:00
Shamrock Lee
b292177eec Add nix-channel --list-generations
Add support to --list-generations
as another way to say
nix-env --profile /nix/var/nix/profiles/per-user/$USER/channels --list-generations
the way we did for nix-channel --rollback [generation id]
2023-06-02 10:12:35 +02:00
Eelco Dolstra
9a5b9c588f
Merge pull request #8421 from fricklerhandwerk/doc-replaceStrings
update documentation according to release notes
2023-05-31 12:10:32 +02:00
Valentin Gagarin
738c0d5064
Merge pull request #8318 from fricklerhandwerk/doc-currentTime
document `builtins.currentTime`
2023-05-31 03:15:54 +02:00
Valentin Gagarin
9afd525e18 update documentation according to release notes 2023-05-30 22:58:15 +02:00
Robert Hensing
bed2fe2312
Merge pull request #8398 from polykernel/perf/lazy-eval-replacements-replacestrings
primops: lazy evaluation of replaceStrings replacements
2023-05-30 22:49:42 +02:00
Eelco Dolstra
ecae62020b
Merge pull request #8406 from NixOS/fix-ca-attrs-false
Restore Nix 2.3 behaviour for {__impure,__contentAddressed} = false
2023-05-30 11:59:14 +02:00
Eelco Dolstra
61ddfa154b
Merge pull request #8399 from tweag/fix-chrooted-stores-error-path
Properly report build errors on chrooted stores
2023-05-27 17:55:57 +02:00
Eelco Dolstra
e4aebccf20 Restore Nix 2.3 behaviour for {__impure,__contentAddressed} = false
Fixes #8405.
2023-05-27 17:53:30 +02:00
Yorick van Pelt
2c462486fe
create pathAccessible, use it to infer default dirs 2023-05-26 15:36:47 +02:00
Yorick van Pelt
a6c78ba367
getDefaultNixPath: ignore EPERM 2023-05-26 15:36:46 +02:00
Yorick van Pelt
b7cde90c6b
Call getDefaultSSLCertFile() only when none is specified
This does pathExists on various paths, which crashes on EPERM in the
macOS sandbox.
2023-05-26 15:36:45 +02:00
Silvan Mosberger
be48907470
ci: Always run with sandbox, even on Darwin
And fix a test failure in the sandbox due to /home
existing on Darwin but not being accessible in the sandbox since it's a
symlink to /System/Volumes/Data/home, see
https://github.com/NixOS/nix/actions/runs/4205378453/jobs/7297384658#step:6:2127:

    C++ exception with description "error: getting status of /home/schnitzel/darmstadt/pommes: Operation not permitted" thrown in the test body.

On Linux this wasn't a problem because there /home doesn't exist in the sandbox
2023-05-26 15:36:44 +02:00
polykernel
a382919d7d
primops: lazy evaluation of replaceStrings replacements
The primop `builtins.replaceStrings` currently always strictly evaluates the
replacement strings, however time and space are wasted for their computation
if the corresponding pattern do not occur in the input string. This commit
makes the evaluation of the replacement strings lazy by deferring their
evaluation to when the corresponding pattern are matched and memoize the result
for efficient retrieval on subsequent matches.

The testcases for replaceStrings was updated to check for lazy evaluation
of the replacements. A note was also added in the release notes to
document the behavior change.
2023-05-25 18:35:23 -04:00
Théophane Hufschmitt
d16a1994fb Properly report build errors on chrooted stores
When encountering a build error, Nix moves the output paths out of the
chroot into their final location (for “easier debugging of build
failures”). However this was broken for chroot stores as it was moving
it to the _logical_ location, not the _physical_ one.

Fix it by moving to the physical (_real_) location.

Fix https://github.com/NixOS/nix/issues/8395
2023-05-25 16:38:29 +02:00
Ben Radford
8e4b15e6a0
Merge branch 'master' into read-only-local-store 2023-05-25 10:06:24 +01:00
Aneesh Agrawal
c694f1a2f3 Remove old default from docs for hashed-mirrors
The `hashed-mirrors` option did use to have this default value,
but it was removed and re-added with an empty default value.
As the autogenerated docs show the (actual) default values from code,
remove this incorrect reference from the docs.

I was updating my nix.conf settings after a few years and noticed this.
2023-05-24 11:05:40 -04:00
Théophane Hufschmitt
34e1b464f0 Normalize the hash-rewriting process when building derivations 2023-05-24 14:11:50 +02:00
Théophane Hufschmitt
a917fb0d53 Use a RewritingSink in derivation goal
Possibly this will make it stream
2023-05-24 14:11:50 +02:00
Théophane Hufschmitt
3ebe1341ab Make RewritingSink accept a map of rewrites
Giving it the same semantics as `rewriteStrings`.
Also add some tests for it
2023-05-24 14:11:50 +02:00
oxalica
303858afad Fix typo in error message of too long store path 2023-05-24 01:24:28 +08:00
Ben Radford
ff12cf3b94
Merge branch 'read-only-local-store' into overlayfs-store 2023-05-23 09:52:41 +01:00
Peter Becich
a420ccc6a8
nix flake check: skip derivations for foreign systems (#7759)
`nix flake show` now skips derivations for foreign systems: https://github.com/NixOS/nix/pull/6988

This commit borrows from that to implement the same behavior for `nix flake check`.

See "nix flake check breaks on IFD in multi-platform flake" https://github.com/NixOS/nix/issues/4265
2023-05-23 06:59:44 +02:00
Bernardo Meurer
bf693319f6
feat: add always-allow-substitutes
This adds a new configuration option to Nix, `always-allow-substitutes`,
whose effect is simple: it causes the `allowSubstitutes` attribute in
derivations to be ignored, and for substituters to always be used.

This is extremely valuable for users of Nix in CI, where usually
`nix-build-uncached` is used. There, derivations which disallow
substitutes cause headaches as the inputs for building already-cached
derivations need to be fetched to spuriously rebuild some simple text
file.

This option should be a good middle-ground, since it doesn't imply
rebuilding the world, such as the approach I took in
https://github.com/NixOS/nixpkgs/pull/221048
2023-05-22 19:42:29 +01:00
Théophane Hufschmitt
494a09c6df
Merge pull request #8377 from layus/fix-ssl-cert-mount
Make mounting ssl cert file optional
2023-05-22 14:06:57 +02:00
Ben Radford
32404292a3
Merge branch 'master' into read-only-local-store 2023-05-22 12:08:02 +01:00
Ben Radford
7251800086
Put read-only setting behind an experimental flag. 2023-05-22 11:38:37 +01:00
Théophane Hufschmitt
673fe85976
Merge pull request #8365 from obsidiansystems/proto-structs
Revert "Revert "Use template structs instead of phantoms""
2023-05-22 09:34:34 +02:00
Guillaume Maudoux
b14fea6fff Shortcircuit for empty caFile 2023-05-19 23:30:35 +02:00
Guillaume Maudoux
36b7e30c11 Make mounting ssl cert file optional 2023-05-19 22:47:40 +02:00
Eelco Dolstra
3305fd0cb1
Merge pull request #8354 from KasyanDiGris/git-fetcher-ask-credentials
Ask for git credentials in fetcher
2023-05-19 13:40:43 +02:00
Eelco Dolstra
34381d5747
Merge pull request #8215 from obsidiansystems/general-repair-path
Support `repairPath` on most stores.
2023-05-19 13:39:48 +02:00
Eelco Dolstra
e31d9b910d
Merge pull request #7312 from layus/fixed-output-system-cert
Allow system certs access to fixed-output derivations
2023-05-19 13:05:16 +02:00
Ben Radford
8ffeb1c4e5
Throw error instead of silently skipping CA migration. 2023-05-18 13:51:21 +01:00
Ben Radford
d55e38b98a
Check earlier whether schema migration is required. 2023-05-18 13:51:21 +01:00
Ben Radford
fe174d72a2
Fix spelling of 'immutable' in documentation. 2023-05-18 13:51:21 +01:00
Robert Hensing
b1c34c0ee8
Merge pull request #8366 from obsidiansystems/worker-proto-forward-decl-types
Don't use `store-api.hh` in `worker-protocol.hh`
2023-05-18 14:33:52 +02:00
Konstantin Vukolov
31ffd0c1fe Remove unused 2023-05-18 13:26:23 +03:00
Konstantin Vukolov
4c4ae887b8 Add option isInteractive 2023-05-18 13:18:34 +03:00
Dylan Green
60c0149721 Apply suggestions from code review
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-05-18 10:31:49 +01:00
cidkidnix
85a2453052 Add tests for read-only local store
Make sure we don't go down the path of making temproots when doing
operations on a read-only store
2023-05-18 10:31:49 +01:00
John Ericson
9923403d90 Don't use store-api.hh in worker-protocol.hh
Using abstract types like can help cut down on compilation time, both
from scratch, and especially incremental builds during development. The
idea is that `worker-protocol.hh` can declare all the (de)serializers, but
only again abstract types; when code needs to use some (de)serializers, it can
include headers just for the data types it needs to (de)serialize.

`store-api.hh` in particular is a bit of a sledgehammer, and the data
types we want to serialize have their own headers.
2023-05-18 00:20:24 -04:00
John Ericson
cb5052d98f Revert "Revert "Use template structs instead of phantoms""
This is the more typically way to do [Argument-dependent
lookup](https://en.cppreference.com/w/cpp/language/adl)-leveraging
generic serializers in C++. It makes the relationship between the `read`
and `write` methods more clear and rigorous, and also looks more
familiar to users coming from other languages that do not have C++'s
libertine ad-hoc overloading.

I am returning to this because during the review in
https://github.com/NixOS/nix/pull/6223, it came up as something that
would make the code easier to read --- easier today hopefully already,
but definitely easier if we were have multiple codified protocols with
code sharing between them as that PR seeks to accomplish.

If I recall correctly, the main criticism of this the first time around
(in 2020) was that having to specify the type when writing, e.g.
`WorkerProto<MyType>::write`, was too verbose and cumbersome. This is
now addressed with the `workerProtoWrite` wrapper function.

This method is also the way `nlohmann::json`, which we have used for a
number of years now, does its serializers, for what its worth.

This reverts commit 45a0ed82f0. That
commit in turn reverted 9ab07e99f5.
2023-05-17 22:44:47 -04:00
John Ericson
b9e5ce4a27 Upgrade downstreamPlaceholder to a type with methods
This gets us ready for dynamic derivation dependencies (part of RFC 92).
2023-05-17 17:41:16 -04:00
John Ericson
e7c1113a37 Add test for downstreamPlaceholder
This is good in general, but in particular ensures when we heavily
refactor it in the next commit there is less likelihood for an
unintentional change in behavior to sneak in.
2023-05-17 17:31:33 -04:00
John Ericson
32dc77ba5d
Merge pull request #8349 from tweag/fix-control-master
Fix ControlMaster behaviour
2023-05-17 12:17:09 -04:00
John Ericson
05cb93442c
Merge pull request #8360 from obsidiansystems/worker-protocol.cc
Move `worker_proto` defs out of `remote-store.cc` to own file
2023-05-17 11:04:29 -04:00
John Ericson
904878d6d2 Move worker_proto defs out of remote-store.cc to own file
These items are not templates, and they declared in
`worker-protocol.hh`; therefore they should live in a
`worker-protocol.cc`.

Anything else needlessly diverges from convention. After all, it is not
like this code is only used in `remote-store.cc`; it is also used in
`daemon.cc`. There is no good reason to place it with the client
implementation or the server implementation when it used equally by
both.
2023-05-17 10:36:03 -04:00
Valentin Gagarin
6a5a8f51bb add cross-references to pure evaluation mode
use consistent wording everywhere.
add some details on the configuration option documentation.
2023-05-17 15:01:54 +02:00
Ben Radford
5966b76c97
Document the new read-only local store setting. 2023-05-17 09:04:52 +01:00
Ben Radford
b1a7b26eef
Rename ReadOnly to Immutable and clarify its purpose. 2023-05-17 09:04:48 +01:00
Ben Radford
aa376f4ab1
Need to open database using immutable parameter.
This requires switching on SQLITE_OPEN_URI because there is no open flag to
make the database immutable. Without immutable, sqlite will still attempt to
create journal and wal files, even when the database is opened read-only.

https://www.sqlite.org/c3ref/open.html

The immutable parameter is a boolean query parameter that indicates that the
database file is stored on read-only media. When immutable is set, SQLite
assumes that the database file cannot be changed, even by a process with higher
privilege, and so the database is opened read-only and all locking and change
detection is disabled.
2023-05-17 08:55:04 +01:00
Ben Radford
78fdd6f24e
Open sqlite database according to new modes. 2023-05-17 08:55:04 +01:00
Ben Radford
afed9ccfad
Add enum for intended sqlite database open modes. 2023-05-17 08:55:03 +01:00
Ben Radford
7f443e0428
Do not check for write access to database when read-only. 2023-05-17 08:55:03 +01:00
Ben Radford
c22936ca6a
Do not attempt to migrate to CA schema when read-only. 2023-05-17 08:55:03 +01:00
Ben Radford
50bbdc65c8
Do not attempt to acquire big-lock when read-only. 2023-05-17 08:55:03 +01:00
Ben Radford
79583c2d38
Do not attempt to chmod per-user dir when read-only. 2023-05-17 08:55:03 +01:00
Ben Radford
de359da09a
Add read-only setting to LocalStoreConfig. 2023-05-17 08:55:03 +01:00
Alexander Schmolck
afd9bd787d Also use long options in src/nix/*.md 2023-05-17 08:10:30 +01:00
Konstantin Vukolov
25434df0d9 Ask for git credentials in fetcher 2023-05-17 02:00:32 +03:00
John Ericson
5fd161189d
Merge pull request #8346 from tweag/fix-nix-profile-install-conflict-segfault
Fix the segfault on `nix profile install` with conflict
2023-05-16 15:49:43 -04:00
Ben Radford
4d69bd034a
More detailed explanation of check-mount setting. 2023-05-16 17:27:49 +01:00
Alexander Bantyev
61cdb0b057
Fix ControlMaster behaviour 2023-05-16 18:50:09 +04:00
John Ericson
0a715ff9cf
Merge pull request #8154 from tweag/delete-old-on-all-profiles-dir
undefined
2023-05-16 09:51:41 -04:00
Théophane Hufschmitt
e997512523 Fix the printing of the installables on nix profile install conflict
- If the element comes from a flake, print the full flakeref (with the
  fragment part) and not just the reference to the flake itself
- If the element doesn't come from a flake, print its store path(s)

This is a bit too verbose, but has the advantages of being correct (and
not crashing), so it's strictly better than the previous situation

Fix https://github.com/NixOS/nix/issues/8284
2023-05-16 11:41:58 +02:00
Théophane Hufschmitt
3d144e778e Rename ProfileElement.describe() to .identifier
This method isn't used to describe what the element is, but to return a
unique identifier for it whithin the current profile
2023-05-16 11:41:58 +02:00
John Ericson
3496a5ee86 Merge branch 'best-effort-supplementary-groups' into overlayfs-store 2023-05-15 17:46:15 -04:00
John Ericson
746c6aae3f Merge remote-tracking branch 'upstream/master' into best-effort-supplementary-groups 2023-05-15 16:50:11 -04:00
John Ericson
72bb9604f3 Merge remote-tracking branch 'upstream/master' into overlayfs-store 2023-05-15 16:34:19 -04:00
John Ericson
2524a21186
Update src/libstore/build/local-derivation-goal.cc
Co-authored-by: Guillaume Girol <symphorien@users.noreply.github.com>
2023-05-15 12:38:39 -04:00
Ben Radford
0df37edb1c Make upper-layer a PathSetting instead of a Setting<string>. 2023-05-15 17:34:31 +01:00
Ben Radford
b0989cb10b Support percent encoded URIs for lower store. 2023-05-15 17:34:31 +01:00
Ben Radford
b5591ece4c Check that overlay store directory is mounted correctly.
Nix does not manage the overlayfs mount point itself, but the correct
functioning of the overlay store does depend on this mount point being set up
correctly. Rather than just assume this is the case, check that the lowerdir
and upperdir options are what we expect them to be. This check is on by
default, but can be disabled if needed.
2023-05-15 17:34:31 +01:00
John Ericson
f8a6a9e473
Merge pull request #8341 from obsidiansystems/dedup-gen-hh
Dedup some markdown -> C++ big literal stuff in build system
2023-05-15 11:44:35 -04:00
Robert Hensing
41591b33a9
Merge pull request #8340 from NixOS/delete-commited-build-artifacts
Delete commited build artifacts
2023-05-15 17:30:20 +02:00
John Ericson
98afd6ff76 Delete commited build artifacts
They were improperly added in 8a93b5a551.

They were not `.gitignore`d because they were stale in that commit --
build artifacts no longer used that name by then and so `.gitignore` was
updated accordingly.
2023-05-15 10:50:33 -04:00
John Ericson
bbd7d5de09 Fix some bounds in rapid check instances
`inRange` is exclusive not inclusive:
https://github.com/emil-e/rapidcheck/blob/master/doc/generators.md#usage

Furthermore, use `std::variant_size_v` so we use the right number
automatically.

Finally, make the `switch` assert the discriminant is in bounds as
expected.
2023-05-15 10:41:44 -04:00
John Ericson
f7c896cfd8 Dedup some markdown -> C++ big literal stuff in build system
This pattern rule was unwisely hidden in `src/libstore/local.mk`. Now it
is properly in `mk/` and we reuse it for the profile docs too.
2023-05-15 10:38:11 -04:00
John Ericson
d2162e7acd Make more string values work as installables
As discussed in #7417, it would be good to make more string values work
as installables. That is to say, if an installable refers to a value,
and the value is a string, it used to not work at all, since #7484, it
works somewhat, and this PR make it work some more.

The new cases that are added for `BuiltPath` contexts:

- Fixed input- or content-addressed derivation:

  ```
  nix-repl> hello.out.outPath
  "/nix/store/jppfl2bp1zhx8sgs2mgifmsx6dv16mv2-hello-2.12"

  nix-repl> :p builtins.getContext hello.out.outPath
  { "/nix/store/c7jrxqjhdda93lhbkanqfs07x2bzazbm-hello-2.12.drv" = { outputs = [ "out" ]; }; }

  The string matches the specified single output of that derivation, so
  it should also be valid.

- Floating content-addressed derivation:

  ```
  nix-repl> (hello.overrideAttrs (_: { __contentAddressed = true; })).out.outPath
  "/1a08j26xqc0zm8agps8anxpjji410yvsx4pcgyn4bfan1ddkx2g0"

  nix-repl> :p builtins.getContext (hello.overrideAttrs (_: { __contentAddressed = true; })).out.outPath
  { "/nix/store/qc645pyf9wl37c6qvqzaqkwsm1gp48al-hello-2.12.drv" = { outputs = [ "out" ]; }; }
  ```

  The string is not a path but a placeholder, however it also matches
  the context, and because it is a CA derivation we have no better
  option. This should also be valid.

We may also want to think about richer attrset based values (also
discussed in that issue and #6507), but this change "completes" our
string-based building blocks, from which the others can be desugared
into or at least described/document/taught in terms of.

Progress towards #7417

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-05-15 09:03:38 -04:00
John Ericson
5a23b80b0a Create EvalState::coerceToDerivedPath
This gives us some round trips to test.

`EvalState::coerceToDerivedPathUnchecked` is a factored out helper just
for unit testing.
2023-05-15 09:03:37 -04:00
John Ericson
8e1a990268 Expose mkOutputString as method of EvalState 2023-05-15 09:03:37 -04:00
John Ericson
0a9afce3b9 Split mkOutputString in two
This well help us with some unit testing
2023-05-15 09:03:37 -04:00
John Ericson
9550c3862f Remove unneeded argument for mkOutputString 2023-05-15 09:03:37 -04:00
John Ericson
914672dc4f
Merge pull request #8141 from tweag/user-files-doc
Document user files of nix
2023-05-15 07:11:47 -04:00
Théophane Hufschmitt
64b0cc5bc4 Make nix-collect-garbage -d look into more places
Make it look into the new-style profiles dir, the old-style one, and the
target of `~/.nix-profile` to be sure that we don't miss anything
2023-05-15 11:36:53 +02:00
Matej Urbas
c66a7af0c6 max-substitution-jobs release note entry 2023-05-14 09:51:11 +01:00
Eelco Dolstra
643b8d2126
Merge pull request #8299 from urbas/max-substitution-jobs
`max-substitution-jobs` setting
2023-05-12 12:24:27 +02:00
John Ericson
a93110ab19 Fix nix print-dev-env & nix develop with drv paths
Fixes #8309

This regression was because both `CmdDevelop` and `CmdPrintDevEnv` were
switched to be `InstallableValueCommand` subclasses, but actually
neither should have been.

The `nixpkgsFlakeRef` method should indeed not be on the base
installable class, because "flake refs" and "nixpkgs" are not
installable-wide notions, but that doesn't mean these commands should
only accept installable values.
2023-05-10 11:29:45 -04:00
John Ericson
53a1354acf
Merge pull request #3959 from obsidiansystems/ca-drv-exotic
Derivations can output "text-hashed" data
2023-05-10 10:41:59 -04:00
Théophane Hufschmitt
85ff212051
Merge pull request #7721 from yorickvP/post-build-hook
Also pass unwanted outputs to post-build-hook
2023-05-10 14:30:42 +02:00
John Ericson
4173743a3c Implement more queries 2023-05-09 17:20:58 -04:00
John Ericson
5059be53b1 Fix recursive ingestion from lower store 2023-05-09 16:42:28 -04:00
John Ericson
6a3a87a714 Improve error message for self reference with text hashing
The `ContentAddressWithReferences` method is made total, with error
handling now squarely the caller's job. This is better.
2023-05-09 14:44:08 -04:00
John Ericson
e514b3939a Add name to some error messages 2023-05-09 13:24:53 -04:00
John Ericson
753fc1661d Cleanups to content address types 2023-05-09 13:05:38 -04:00
John Ericson
d3c125e5a8
Apply suggestions from code review
Thanks!

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-05-09 12:45:51 -04:00
John Ericson
6513f4fe92 Fix bug, newInfo -> newInfo0
It appears we were checking a variable in the process of definining it.
2023-05-09 12:31:36 -04:00
John Ericson
35dcbe1c21 Fix spurious change
Didn't mean to use the private name that shouldn't be exposed.
2023-05-09 12:19:03 -04:00
Alex Ameen
82296f8113
prevent double quotation 2023-05-09 09:59:18 -05:00
Alex Ameen
b72bc4a972
libexpr: quote reserved keys when printing
This fixes a bug in commands like `nix eval' which would emit invalid attribute
sets if they contained reserved keywords such as "assert", "let", etc.

These keywords will not be quoted when printed, making them valid expressions.
All keywords recognized by the lexer are quoted except "or", which does not
require quotation.
2023-05-09 09:45:12 -05:00
John Ericson
e7c3399ed2 Specialize LocalOverlayStore::queryPathFromHashPart
With test
2023-05-09 10:40:10 -04:00
John Ericson
ddaf2750b5 Specialize more methods, fix tests 2023-05-09 10:22:38 -04:00
John Ericson
b3d320c594 Convert more methods
Fixed one test, broke another
2023-05-08 18:50:16 -04:00
John Ericson
59a8099038 Fix LocalOverlayStore::queryPathInfoUncached, FIXME in test 2023-05-08 17:37:40 -04:00
John Ericson
5406256d78 Specialize LocalOverlayStore::queryPathInfoUncached 2023-05-08 17:30:17 -04:00
John Ericson
31e98ed0a0 Specialize LocalOverlayStore::registerDrvOutput 2023-05-08 16:48:55 -04:00
John Ericson
9c9f5f0d12 Merge branch 'best-effort-supplementary-groups' into overlayfs-store 2023-05-08 14:47:46 -04:00
Matej Urbas
13185133bc introduces Goal::jobCategory 2023-05-08 19:45:46 +01:00
John Ericson
6d1aa523de Create escape hatch for supplementary group sandboxing woes
There is no obvious good solution for this that has occured to anyone.
2023-05-08 14:41:47 -04:00
Matej Urbas
1ea1e378de removes MaxSubstitutionJobsSetting 2023-05-08 19:21:57 +01:00
John Ericson
b5d9ef0a4c
Merge pull request #3921 from obsidiansystems/trustless-remote-builder-simple
Trustless remote building for input-addressed drvs
2023-05-08 10:43:37 -04:00
John Ericson
d5fe828d76 Merge remote-tracking branch 'upstream/master' into overlayfs-store 2023-05-08 10:20:18 -04:00
John Ericson
f0a176e2f1 Init local overlay store 2023-05-08 10:20:06 -04:00
John Ericson
df53a7d268 Split comment, match with each variable 2023-05-08 10:08:01 -04:00
John Ericson
cd0d8e0bd5
Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-05-08 09:57:05 -04:00
John Ericson
278c94d607 Rename a few things in new tests
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-05-08 08:01:58 -04:00
Yorick van Pelt
12685ef45f
CA: rewrite hashes for all outputs, not just the wanted ones 2023-05-08 12:58:59 +02:00
Yorick van Pelt
2ca2c80c4e
libstore: also pass unwanted outputs to the post-build-hook 2023-05-08 12:58:59 +02:00
Robert Hensing
cf8effdae2
Merge pull request #8263 from frederictobiasc/improve-doc-genericclosure
Documentation: Improve builtins.genericClosure
2023-05-08 12:23:19 +02:00
Matej Urbas
613bc699bb max-substitution-jobs setting 2023-05-07 20:22:18 +01:00
figsoda
0662fd8599 Fix hostRegex to accept hosts with a - 2023-05-03 18:59:44 -04:00
Eelco Dolstra
17e6b85d05 nix: Support the --repair flag 2023-04-28 17:03:04 +02:00
Valentin Gagarin
f8620758aa display documentation on manifest files separately
it's probably better not to show the manifest file documentation in the
command-specific pages, because these are implementation details that are not really practically useful.

this means no additional hassle for building the manual, but clutters
the table of contents a bit.
2023-04-28 12:10:36 +02:00
Valentin Gagarin
be7c236565 move compatibility info to nix profile docs 2023-04-26 15:39:35 +02:00
Alexander Bantyev
aee5e82b42 Include user-profiles.md into nix profile --help 2023-04-26 15:38:19 +02:00
Alexander Bantyev
8a93b5a551 Document user files of nix 2023-04-26 15:38:19 +02:00
Frédéric Christ
f0d2b7eef3 Doc: Improve builtins.genericClosure 2023-04-26 09:37:32 +02:00
Eelco Dolstra
87f676b3a0 Formatting 2023-04-25 16:52:02 +02:00
Eelco Dolstra
a74d397549 nix build --json: Only show non-zero startTime / stopTime 2023-04-25 16:43:47 +02:00
Eelco Dolstra
880e7b8ed6 TarArchive: Remove a duplicate constant and increase the buffer size 2023-04-25 16:43:10 +02:00
Moritz Angermann
0e18254aa8
Fix shutdown behavior and resource management for recursive-nix on macOS
Previously, we relied on the `shutdown()` function to terminate `accept()`
calls on a listening socket. However, this approach did not work on macOS as
the waiting `accept()` call is not considered a connected socket, resulting in
an `ENOTCONN` error. Instead, we now close the listening socket to terminate
the `accept()` call.

Additionally, we fixed a resource management issue where we set the
`daemonSocket` variable to -1, triggering resource cleanup and causing the
`stopDaemon` function to be called twice. This resulted in errors as the socket
was already closed by the time the second `stopDaemon` call was made. Instead of
setting `daemonSocket` to -1, we now release the socket using the `release()`
method on a unique pointer. This properly transfers ownership and allows for
correct resource cleanup.

These changes ensure proper behavior and resource management for the
recursive-nix feature on macOS.
2023-04-25 09:39:05 +08:00
Eelco Dolstra
5d3f6dbf59 Add some more SourcePath docs 2023-04-24 13:37:51 +02:00
Eelco Dolstra
ad57cff9bc
Document tMisc
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-24 13:34:46 +02:00
Eelco Dolstra
01232358ff Merge remote-tracking branch 'origin/master' into source-path 2023-04-24 13:20:36 +02:00
John Ericson
8eeaf591db
Add more docs to TextIngestionMethod
Thanks so much!

Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com>
2023-04-21 01:30:55 -04:00
John Ericson
85f0cdc370 Use std::set<StringContextElem> not PathSet for string contexts
Motivation

`PathSet` is not correct because string contexts have other forms
(`Built` and `DrvDeep`) that are not rendered as plain store paths.
Instead of wrongly using `PathSet`, or "stringly typed" using
`StringSet`, use `std::std<StringContextElem>`.

-----

In support of this change, `NixStringContext` is now defined as
`std::std<StringContextElem>` not `std:vector<StringContextElem>`. The
old definition was just used by a `getContext` method which was only
used by the eval cache. It can be deleted altogether since the types are
now unified and the preexisting `copyContext` function already suffices.

Summarizing the previous paragraph:

Old:

  - `value/context.hh`: `NixStringContext = std::vector<StringContextElem>`
  - `value.hh`: `NixStringContext Value::getContext(...)`
  - `value.hh`: `copyContext(...)`

New:

  - `value/context.hh`: `NixStringContext = std::set<StringContextElem>`
  - `value.hh`: `copyContext(...)`
----

The string representation of string context elements no longer contains
the store dir. The diff of `src/libexpr/tests/value/context.cc` should
make clear what the new representation is, so we recommend reviewing
that file first. This was done for two reasons:

Less API churn:

`Value::mkString` and friends did not take a `Store` before. But if
`NixStringContextElem::{parse, to_string}` *do* take a store (as they
did before), then we cannot have the `Value` functions use them (in
order to work with the fully-structured `NixStringContext`) without
adding that argument.

That would have been a lot of churn of threading the store, and this
diff is already large enough, so the easier and less invasive thing to
do was simply make the element `parse` and `to_string` functions not
take the `Store` reference, and the easiest way to do that was to simply
drop the store dir.

Space usage:

Dropping the `/nix/store/` (or similar) from the internal representation
will safe space in the heap of the Nix programming being interpreted. If
the heap contains many strings with non-trivial contexts, the saving
could add up to something significant.

----

The eval cache version is bumped.

The eval cache serialization uses `NixStringContextElem::{parse,
to_string}`, and since those functions are changed per the above, that
means the on-disk representation is also changed.

This is simply done by changing the name of the used for the eval cache
from `eval-cache-v4` to eval-cache-v5`.

----

To avoid some duplication `EvalCache::mkPathString` is added to abstract
over the simple case of turning a store path to a string with just that
string in the context.

Context

This PR picks up where #7543 left off. That one introduced the fully
structured `NixStringContextElem` data type, but kept `PathSet context`
as an awkward middle ground between internal `char[][]` interpreter heap
string contexts and `NixStringContext` fully parsed string contexts.

The infelicity of `PathSet context` was specifically called out during
Nix team group review, but it was agreeing that fixing it could be left
as future work. This is that future work.

A possible follow-up step would be to get rid of the `char[][]`
evaluator heap representation, too, but it is not yet clear how to do
that. To use `NixStringContextElem` there we would need to get the STL
containers to GC pointers in the GC build, and I am not sure how to do
that.

----

PR #7543 effectively is writing the inverse of a `mkPathString`,
`mkOutputString`, and one more such function for the `DrvDeep` case. I
would like that PR to have property tests ensuring it is actually the
inverse as expected.

This PR sets things up nicely so that reworking that PR to be in that
more elegant and better tested way is possible.

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-04-21 01:05:49 -04:00
John Ericson
7103c6da70 Remove references from fixed output derivation ab syntax
In other words, use a plain `ContentAddress` not
`ContentAddressWithReferences` for `DerivationOutput::CAFixed`.

Supporting fixed output derivations with (fixed) references would be a
cool feature, but it is out of scope at this moment.
2023-04-19 15:00:04 -04:00
John Ericson
aba8a8a83a Add a few more content addressing methods
Good to round out the library interface.
2023-04-19 14:13:30 -04:00
John Ericson
20decfd302 Gate dynamic-derivations with drv fromJSON too
Don't want `nix derivation add` to be a way to sneak by experimental
feature checks!
2023-04-19 12:44:38 -04:00
John Ericson
76baaeb341 Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2023-04-19 11:32:14 -04:00
Théophane Hufschmitt
d3e2394e91
Merge pull request #8233 from wentasah/narinfo-corrupt-reason
Make "NAR info file is corrupt" messages more informative
2023-04-19 06:57:08 +02:00
Robert Hensing
28d7ffd448
Merge pull request #8220 from accelbread/whitelist-commit-lockfile-summary
Add commit-lockfile-summary to flake nixConfig whitelist
2023-04-18 16:49:33 +02:00
Robert Hensing
5cd9890e8a src/nix/flake.md: Itemize safe nixConfigs 2023-04-18 16:06:58 +02:00
Michal Sojka
d30d2dc861 Make "NAR info file is corrupt" messages more informative
Recently, I encountered the "NAR info file 'xxxx' is corrupt" error
with my binary cache. The message is not helpful in determining, which
kind of corruption happened. The file, fetched with curl, looked
reasonably.

This commit adds more information to the error message, which should
allow debugging and hopefully fixing the problem.
2023-04-18 14:10:49 +02:00
John Ericson
668377f217 TextHashMethod -> TextIngestionMethod, gate with XP feature
I suppose we can use `dynamic-derivations` for the few things we neeed.
2023-04-17 19:02:45 -04:00
John Ericson
f56c4a5bdf Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2023-04-17 18:10:12 -04:00
John Ericson
aa74c7b0bc Gate experimental features in DerivationOutput::fromJSON
This is an entry point for outside data, so we need to check enabled
experimental features here.
2023-04-17 17:36:12 -04:00
John Ericson
ab5ca608bf Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-17 13:54:48 -04:00
John Ericson
23ee2d79a9 Use buildPathsWithResults in build-remote.cc trustless path
It handles failures more correctly; I am glad we have it now!
2023-04-17 13:49:14 -04:00
John Ericson
79ba0ba37a Improve the build remote comment. 2023-04-17 13:49:14 -04:00
Robert Hensing
64ee02890c
Merge pull request #8230 from obsidiansystems/daemon-trust-override
Experimentally allow forcing `nix-daemon` trust; use this to test
2023-04-17 19:43:41 +02:00
John Ericson
d41e1bed5e Experimentally allow forcing nix-daemon trust; use this to test
We finally test the status quo of remote build trust in a number of
ways. We create a new experimental feature on `nix-daemon` to do so.

PR #3921, which improves the situation with trustless remote building,
will build upon these changes. This code / tests was pull out of there
to make this, so everything is easier to review, and in particular we
test before and after so the new behavior in that PR is readily apparent
from the testsuite diff alone.
2023-04-17 13:06:21 -04:00
John Ericson
2c8475600d Fix some issues with experimental config settings
Issues:

1. Features gated on disabled experimental settings should warn and be
   ignored, not silently succeed.

2. Experimental settings in the same config "batch" (file or env var)
   as the enabling of the experimental feature should work.

3. For (2), the order should not matter.

These are analogous to the issues @roberth caught with my changes for
arg handling, but they are instead for config handling.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-17 12:41:04 -04:00
Robert Hensing
3f9589f17e
Merge pull request #6312 from obsidiansystems/keyed-build-result
Shuffle `BuildResult` data definition, make state machine clearer, introduce `SingleDrvOutputs`
2023-04-17 18:08:05 +02:00
John Ericson
2eb493ca51 Fix DerivationOutput::fromJSON 2023-04-17 10:28:54 -04:00
John Ericson
e12efa3654 Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2023-04-17 10:16:57 -04:00
Robert Hensing
e641de085b
Merge pull request #3746 from obsidiansystems/path-info
Introduce `StoreReferences` and `ContentAddressWithReferences`
2023-04-17 15:49:48 +02:00
John Ericson
e95db8f2b9 nix-testing -> daemon-trust-override
And only enable in the tests that need it. This makes it less of a
sledgehammer.
2023-04-17 09:35:43 -04:00
John Ericson
b1343e8ad1 Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-17 09:27:17 -04:00
John Ericson
537e8719f2
Explain various .self = false,
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-17 09:15:11 -04:00
John Ericson
72ffa7fedb
Merge pull request #7732 from hercules-ci/make-initLibStore-viable-alternative
Make `initLibStore` a viable alternative
2023-04-17 08:04:41 -04:00
Robert Hensing
cb2615cf47 Merge remote-tracking branch 'upstream/master' into source-path 2023-04-17 11:41:50 +02:00
Robert Hensing
9af9c260fc
Merge pull request #8193 from hercules-ci/dry-strings
Deduplicate string literal rendering, fix 4909
2023-04-17 11:19:40 +02:00
John Ericson
9800c1e807 Mark experimental configuration settings programmatically
Fix #8162

The test is changed to compare `nlohmann::json` values, not strings of dumped
JSON, which allows us to format things more nicely.
2023-04-16 10:58:04 -04:00
Robert Hensing
b6125772d7 libexpr: Move identifier-like printing to print.cc 2023-04-16 14:07:35 +02:00
Robert Hensing
28a5cdde02 libexpr/value/print.* -> libexpr/print.*
Generalizes the file to sensibly allow printing any part of the
language syntax.
2023-04-16 13:10:45 +02:00
Robert Hensing
1e2dd669bc printLiteral: Do not overload 2023-04-16 13:04:35 +02:00
Robert Hensing
9c74df5bb4
Format
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-04-15 20:56:51 +02:00
John Ericson
24866b71c4 Introduce SingleDrvOutputs
In many cases we are dealing with a collection of realisations, they are
all outputs of the same derivation. In that case, we don't need
"derivation hashes modulos" to be part of our map key, because the
output names alone will be unique. Those hashes are still part of the
realisation proper, so we aren't loosing any information, we're just
"normalizing our schema" by narrowing the "primary key".

Besides making our data model a bit "tighter" this allows us to avoid a
double `for` loop in `DerivationGoal::waiteeDone`. The inner `for` loop
was previously just to select the output we cared about without knowing
its hash. Now we can just select the output by name directly.

Note that neither protocol is changed as part of this: we are still
transferring `DrvOutputs` over the wire for `BuildResult`s. I would only
consider revising this once #6223 is merged, and we can mention protocol
versions inside factored-out serialization logic. Until then it is
better not change anything because it would come a the cost of code
reuse.
2023-04-15 12:51:19 -04:00
John Ericson
0f2b5146c7 Make restarting state machines explicit
If my memory is correct, @edolstra objected to modifying `wantedOutputs`
upon falling back to doing a build (as we did before), because we should
only modify it in response to new requests --- *actual* wants --- and
not because we are "incidentally" building all the outptus beyond what
may have been requested.

That's a fair point, and the alternative is to replace the boolean soup
with proper enums: Instead of modifying `wantedOuputs` som more, we'll
modify `needsRestart` to indicate we are passed the need.
2023-04-15 11:01:31 -04:00
John Ericson
37fca662b0 Make KeyedBuildResult, BuildResult like before, and fix bug another way
In https://github.com/NixOS/nix/pull/6311#discussion_r834863823, I
realized since derivation goals' wanted outputs can "grow" due to
overlapping dependencies (See `DerivationGoal::addWantedOutputs`, called
by `Worker::makeDerivationGoalCommon`), the previous bug fix had an
unfortunate side effect of causing more pointless rebuilds.

In paticular, we have this situation:

1. Goal made from `DerivedPath::Built { foo, {a} }`.

2. Goal gives on on substituting, starts building.

3. Goal made from `DerivedPath::Built { foo, {b} }`, in fact is just
   modified original goal.

4. Though the goal had gotten as far as building, so all outputs were
   going to be produced, `addWantedOutputs` no longer knows that and so
   the goal is flagged to be restarted.

This might sound far-fetched with input-addressed drvs, where we usually
basically have all our goals "planned out" before we start doing
anything, but with CA derivation goals and especially RFC 92, where *drv
resolution* means goals are created after some building is completed, it
is more likely to happen.

So the first thing to do was restore the clearing of `wantedOutputs` we
used to do, and then filter the outputs in `buildPathsWithResults` to
only get the ones we care about.

But fix also has its own side effect in that the `DerivedPath` in the
`BuildResult` in `DerivationGoal` cannot be trusted; it is merely the
*first* `DerivedPath` for which this goal was originally created.

To remedy this, I made `BuildResult` be like it was before, and instead
made `KeyedBuildResult` be a subclass wit the path. Only
`buildPathsWithResults` returns `KeyedBuildResult`s, everything else
just becomes like it was before, where the "key" is unambiguous from
context.

I think separating the "primary key" field(s) from the other fields is
good practical in general anyways. (I would like to do the same thing
for `ValidPathInfo`.) Among other things, it allows constructions like
`std::map<Key, ThingWithKey>` where doesn't contain duplicate keys and
just precludes the possibility of those duplicate keys being out of
sync.

We might leverage the above someday to overload `buildPathsWithResults`
to take a *set* of return a *map* per the above.

-----

Unfortunately, we need to avoid C++20 strictness on designated
initializers.

(BTW
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2287r1.html
this offers some new syntax for this use-case. Hopefully this will be
adopted and we can eventually use it.)

No having that yet, maybe it would be better to not make
`KeyedBuildResult` a subclass to just avoid this.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-15 11:01:31 -04:00
John Ericson
9df7f3f537 Introduce Worker::makeGoal
This takes a `DerivedPath` so the caller doesn't need to care about
which sort of goal does what.
2023-04-15 11:01:31 -04:00
Robert Hensing
ab228d73db
Merge pull request #8214 from raphaelr/installable-derived-path-warn
Display valid installable in `InstallableDerivedPath::parse` warning
2023-04-15 14:15:02 +02:00
John Ericson
ee420ac64e Legacy vs non-legacy to_string/parse for DerivedPath
As requested by @roberth, it is good to call out the specific instances
we care about, which is `!` for the RPC protocols, and `^` for humans.

This doesn't take advantage of parametricity as much, but since the
human and computer interfaces are good to decouple anyways (we don't
care if they drift further apart over time in the slightest) some
separation and slight duplication is fine.

Also, unit test both round trips.
2023-04-14 20:45:11 -04:00
Archit Gupta
bfc558c972 Whitelist commit-lockfile-summary in flake nixConfig 2023-04-14 11:33:38 -07:00
John Ericson
92346d1ed4
Merge pull request #8174 from NixOS/gloss-xp-features-note-installables
Start cross-referencing experimental features
2023-04-14 08:48:10 -04:00
John Ericson
a6f85e052c Support repairPath on most stores.
More progress on issue #5729

The method trivially generalizes to be store-implementation-agnostic, in
fact.

However, we force it to continue to be unimplemented with `RemoteStore`
and `LegacySSHStore` because the implementation we'd get via the
generalization is probably not the one users expect. This keeps our
hands untied to do it right going forward.

For more about the tension between the scheduler logic being
store-type-agnostic and remote stores doing their own scheduling, see
issues #5025 and #5056.
2023-04-14 08:29:06 -04:00
Robert Hensing
f5ab38a688
Merge pull request #8217 from obsidiansystems/push-get-fs-accessor-unsupported-down
Push `getFSAccessor` `unsupported(...)` down `Store` class hierarchy
2023-04-14 14:23:23 +02:00
John Ericson
d93e76fbb8 Start cross-referencing experimental features
- Create a glossary entry for experimental features.

- Have the man page experimental feature notice link `nix-commmand`.

  (Eventually this should be programmed, based on whether the command is
  experimental, and if so what experimental feature does it depend on.)

- Document which installables depend on which experimental features.

  I tried to use the same style (bold warning and block quote) that the
  top of the man page uses.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-04-14 07:45:08 -04:00
Eelco Dolstra
7eac8838df
Merge pull request #8199 from tweag/fix-sqlite-busy-reporting
Fix unnecessary reporting of SQLite busy errors
2023-04-14 10:50:50 +02:00
Eelco Dolstra
33fc09c2a5
Merge pull request #8176 from tweag/rename-confusing-write-method
Rename and protect `BufferedSink::write`
2023-04-14 10:44:36 +02:00
Raphael Robatsch
9e8f209036 Display valid installable in InstallableDerivedPath::parse warning
The warning message should produce an installable name that can be
passed to `nix build`, `nix path-info`, etc. again. Since the CLI
expects that the .drv path and the output names are separated by
a caret, the warning message must also separate the .drv path and output
names with a caret.

However, `DerivedPath::Built.to_string()` uses an exclamation point as
the separator instead. This commit adds a `separator` argument to the
to_string method.

This changes the warning message from:
If this command is now failing try again with '/nix/store/foo.drv!*'
to:
If this command is now failing try again with '/nix/store/foo.drv^*'
2023-04-14 10:32:03 +02:00
John Ericson
ee97f107e8 Push getFSAccessor unsupported(...) down Store class hierarchy
More progress on issue #5729.

Instead of having it by the default method in `Store` itself, have it be
the implementation in `DummyStore` and `LegacySSHStore`. Then just the
implementations which fail to provide the method pay the "penalty" of
dealing with the icky `unimplemented` function for non-compliance.

Combined with my other recent PRs, this finally makes `Store` have no
`unsupported` calls!
2023-04-13 13:39:44 -04:00
John Ericson
615c25b0dd Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-12 00:00:12 -04:00
Ben Radford
de3df3009b
Move warning timing logic into handleSQLiteBusy. 2023-04-11 16:14:16 +01:00
John Ericson
450e5ec618 Do not gate or hide experimental settings
This is somewhat hacky fix just for 2.15. I unintentionally hid them
from the manual, when no one wanted to hide them that (including
myself). I also required the experimental feature to be enabled in an
order-dependent way, which is not good.

The simplest fix for this immanent release is just to always show them,
and always allow them to be set.

Effectively undoes some changes from aa663b7e89
2023-04-11 10:56:48 -04:00
Eelco Dolstra
162c3f5f7c
Merge pull request #8170 from tweag/fix-aarch64-build
Explicitely define `LockFile::operator!=`
2023-04-11 12:35:23 +02:00
Ben Radford
7c56e84213
Warn after a second of being busy instead of immediately.
Getting the occasional SQLITE_BUSY is expected when the database is being
accessed concurrently. The retry will likely succeed so it is pointless to warn
immediately. Instead we track how long each retrySQLite block has been running,
and only begin warning after a second has elapsed (and then every 10 seconds
subsequently).
2023-04-11 11:15:34 +01:00
Eelco Dolstra
60a1bf08b6
Merge pull request #7798 from peeley/list-experimental-features
Documentation: list experimental features in manual
2023-04-11 11:55:35 +02:00
Eelco Dolstra
deb7f4b466
Nitpicks 2023-04-11 11:29:35 +02:00
Robert Hensing
4e0804c920 Deduplicate string literal rendering, fix 4909 2023-04-09 22:42:20 +02:00
John Ericson
51c8ffbc28 Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-08 18:49:26 -04:00
John Ericson
fd21f9d76e Merge remote-tracking branch 'upstream/master' into path-info 2023-04-07 20:39:04 -04:00
John Ericson
3ebd66c00e Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-07 19:54:34 -04:00
John Ericson
8f44edc83e Tidy up and comment daemon CLI
Some of the factoring out was taken from #7912 by @mupdt. Thanks!

No behavior should be changed in this commit.

Co-Authored-By: mupdt <25388474+mupdt@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-07 16:49:52 -04:00
Robert Hensing
4411c7d7e0
Merge pull request #8179 from tweag/disable-gc-on-coroutine
disable gc on coroutine
2023-04-07 18:21:11 +02:00
Robert Hensing
ddebeb934a libstore: Remove lockCPU dead code
Left over from 9747ea84b, https://github.com/NixOS/nix/pull/5821
2023-04-07 17:50:40 +02:00
Robert Hensing
1c0b680ef9 libstore: Remove lockCPU dead code
Left over from 9747ea84b, https://github.com/NixOS/nix/pull/5821
2023-04-07 17:50:40 +02:00
Robert Hensing
2445afd92c Require openssl >= 1.1.1
Versions older this are sufficiently old that we don't want to support
them, and they require extra support code.
2023-04-07 17:50:40 +02:00
Robert Hensing
2196fd1146 libutil: Provide alternatives to startSignalHandlerThread
How signals should be handled depends on what kind of process Nix
is integrated into. The signal handler thread used by the stand-alone
Nix commands / processes may not work well in the context of other
runtime systems, such as those of Python, Perl, or Haskell.
2023-04-07 17:50:40 +02:00
John Ericson
38ae7c2891 Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-07 11:13:35 -04:00
John Ericson
6e1e15ffec Fix it! 2023-04-07 11:13:23 -04:00
Robert Hensing
781d3dceb3 Move initLibUtil() from initNix to initLibStore
libutil is a dependency of libstore, so it should always be
initialized as such.
libutil is also a dependency of libmain. Being explicit about this
dependency might be good, but not worth the slight code complexity
until the library structure gets more advanced.

Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
2023-04-07 16:24:18 +02:00
Robert Hensing
1107ea363f libmain: Clarify the lack of initLibExpr()
Quote

    Why not initLibExpr()? initGC() is essentially that, but
    detectStackOverflow is not an instance of the init function concept, as
    it may have to be invoked more than once per process.

Furthermore, renaming initGC to initLibExpr is more trouble than it's
worth at this time.
2023-04-07 16:24:18 +02:00
Robert Hensing
52d6ce6515 Move macOS TMPDIR hack from initNix to initLibStore
This code is bad. We shouldn't unset variables in programs whose
children may need them. Fixing one issue at a time, so postponing.
See https://github.com/NixOS/nix/issues/7731

Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
2023-04-07 16:24:18 +02:00
Robert Hensing
e706ffa007 Move preloadNSS() from initNix to initLibStore
It is required for the sandbox, which is a libstore responsibility;
not just libmain.

Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
2023-04-07 16:24:18 +02:00
Robert Hensing
a58be39476 Move sodium_init() to initLibStore()
Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
2023-04-07 16:24:18 +02:00
Robert Hensing
9693076715 Move initLibStore() immediately after initLibUtil()
Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.

The goal of this reordering is to make initLibStore self-sufficient
in a following commit.
2023-04-07 16:24:18 +02:00
Robert Hensing
a692c43729 Move loadConfFile() to initLibStore
Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.

Using libstore without loading the config file is risky, as sqlite
may then be misconfigured. See https://github.com/cachix/cachix/issues/475
2023-04-07 16:24:18 +02:00
Robert Hensing
6e0b7109ab Move OpenSSL init to initLibUtil
Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
2023-04-07 16:24:18 +02:00
John Ericson
0746951be1
Finish converting existing comments for internal API docs (#8146)
* Finish converting existing comments for internal API docs

99% of this was just reformatting existing comments. Only two exceptions:

- Expanded upon `BuildResult::status` compat note

- Split up file-level `symbol-table.hh` doc comments to get
  per-definition docs

Also fixed a few whitespace goofs, turning leading tabs to spaces and
removing trailing spaces.

Picking up from #8133

* Fix two things from comments

* Use triple-backtick not indent for `dumpPath`

* Convert GNU-style `\`..'` quotes to markdown style in API docs

This will render correctly.
2023-04-07 13:55:28 +00:00
John Ericson
c036de086e Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-07 09:40:36 -04:00
Yorick van Pelt
62ddd8633c Add talkative msg for coro gc debug 2023-04-07 14:54:50 +02:00
Yorick van Pelt
58d24a4cb6 Always disable GC in a coroutine unless the patch is applied 2023-04-07 14:54:38 +02:00
Yorick van Pelt
00bc34430b DisableGC: replace by CoroutineContext, std::shared_ptr<void> 2023-04-07 14:53:40 +02:00
Yorick van Pelt
2c53ef1bfe Disable GC inside coroutines on mac OS 2023-04-07 14:52:59 +02:00
John Ericson
59e0728714 Create nix derivation add command
Also refine `nix derivation show`'s docs very slightly.
2023-04-07 08:34:58 -04:00
John Ericson
27597f8131 Rename files to reflect new nix derivation show name
This will match the files we added for `nix add derivation` in the rest
of this PR.
2023-04-07 08:34:58 -04:00
John Ericson
2b98af2e62 nix show-derivation -> nix derivation show 2023-04-07 08:34:58 -04:00
John Ericson
4e9f32f993 Liberate checkDerivationOutputs from LocalStore
Make it instead a method on `Derivation` that can work with any store.
We will need this for a CLI command to create a derivation.
2023-04-07 08:34:58 -04:00
John Ericson
b200784cec Include the name in the JSON for derivations
This is non-breaking change in the to-JSON direction. This *is* a
breaking change in the from-JSON direction, but we don't care, as that
is brand new in this PR.

`nix show-derivation --help` currently has the sole public documentation
of this format, it is updated accordingly.
2023-04-07 08:34:58 -04:00
John Ericson
fe9cbe838c Create Derivation::fromJSON
And test, of course
2023-04-07 08:34:58 -04:00
Eelco Dolstra
6960de73af
Typo 2023-04-07 13:08:32 +02:00
Théophane Hufschmitt
214f1d6791 Rename and protect BufferedSink::write
The `write` name is ambiguous and could lead to some funny bugs like
https://github.com/NixOS/nix/pull/8173#issuecomment-1500009480. So
rename it to the more explicit `writeUnbuffered`.
Besides, this method shouldn't be (and isn't) used outside of the class
implementation, so mark it `protected`.

This makes it more symetrical to `BufferedSource` which uses a
`protected readUnbuffered` method.
2023-04-07 09:21:50 +02:00
matthewcroughan
9207f94582 Add Store::isTrustedClient()
This function returns true or false depending on whether the Nix client
is trusted or not. Mostly relevant when speaking to a remote store with
a daemon.

We include this information in `nix ping store` and `nix doctor`

Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
2023-04-06 19:59:57 -04:00
John Ericson
bc192a95ef Describe active experimental features in the contributing guide
They are put in the manual separate pages under the new overarching
description of experimental features.

The settings page just lists the valid experimental feature names (so
people know what a valid setting entry looks like), with links to those
pages. It doesn't attempt to describe each experimental feature as that
is too much information for the configuration settings section.
2023-04-06 18:07:59 -04:00
John Ericson
6c4049b38a Link the new general documentation on xp features on the setting 2023-04-06 18:07:58 -04:00
Noah Snelson
e399cb49c2
Fix typo in no-url-literals experimental feature docs
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-04-06 15:02:19 -07:00
Noah Snelson
8a7790f46a Expand documentation for experimental-features
Adds examples and additional information to the `impure-derivations`,
`recursive-nix`, and `no-url-literals` experimental feature
documentation.
2023-04-06 14:52:34 -07:00
Henrik Karlsson
2265901e6e improved help command listing. 2023-04-06 16:37:41 +01:00
Eelco Dolstra
a9759407e5 Origin: Use SourcePath 2023-04-06 15:25:06 +02:00
Eelco Dolstra
94812cca98 Backport SourcePath from the lazy-trees branch
This introduces the SourcePath type from lazy-trees as an abstraction
for accessing files from inputs that may not be materialized in the
real filesystem (e.g. Git repositories). Currently, however, it's just
a wrapper around CanonPath, so it shouldn't change any behaviour. (On
lazy-trees, SourcePath is a <InputAccessor, CanonPath> tuple.)
2023-04-06 13:15:50 +02:00
Théophane Hufschmitt
faefaac875 Explicitely define LockFile::operator!=
It should be syntethised in terms of `operator==`, but the GCC version
used on aarch64-linux doesn't implement that (see
https://hydra.nixos.org/build/214848896=. So explicitely define it.

Fix https://github.com/NixOS/nix/issues/8159
2023-04-05 17:20:04 +02:00
John Ericson
53d0836347 Assemble experimental feature docs outside of Nix itself
Instead of constructing a markdown list in C++ (which involved all sorts
of nasty string literals), export some JSON and assemble it with the
manual build system.

Besides following the precedent set with other dumped data, this is a
better separate of content and presentation; if we decide for example we
want to display this information in a different way, or in a different
section of the manual, it will become much easier to do so.
2023-04-04 22:57:11 -04:00
John Ericson
3f98353f19 Merge remote-tracking branch 'upstream/master' into list-experimental-features 2023-04-04 21:34:20 -04:00
John Ericson
bdeeffff96
Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-04-04 19:16:10 -04:00
Théophane Hufschmitt
d93c1514eb
Merge pull request #7944 from yorickvP/fix-curRepl
NixRepl::mainLoop: restore old curRepl on function exit
2023-04-04 09:44:47 +02:00
Théophane Hufschmitt
523c3f7225
Merge pull request #8157 from hercules-ci/switch-bugs
Fix current and future `switch` bugs
2023-04-03 20:43:02 +02:00
Eelco Dolstra
81491e1379
Merge pull request #8042 from lheckemann/alt-lockfiles
Allow specifying alternative paths for reading/writing flake locks
2023-04-03 19:28:09 +02:00
Robert Hensing
bf2c5c3958 nix-store.cc: Refactor, remove qDefault 2023-04-03 18:45:20 +02:00
Robert Hensing
fba7be80eb Enable -Werror=switch-enum
switch statements must now match all enum values or disable the
warning.

Explicit is good. This has helped us find two bugs, after solving
another one by debugging.

From now on, adding to an enum will raise errors where they are
not explicitly handled, which is good for productivity, and helps
us decide the correct behavior in all usages.

Notably still excluded from this though are the cases where the
warning is disabled by local pragmas.

fromTOML.cc did not build despite a top-level pragma, so I've had
to resort to a makefile solution for that.
2023-04-03 18:45:20 +02:00
Robert Hensing
9470ee877d Allow open switch-enum in 5 places 2023-04-03 18:45:20 +02:00
Robert Hensing
3dac4c7874 Add explicit case statements where -Wswitch-enum would report them 2023-04-03 18:17:32 +02:00
Robert Hensing
ed7885017c Fix systemd logging for lvlNotice: eqv to lvlInfo, not lvlVomit 2023-04-03 18:17:32 +02:00
Robert Hensing
62cacc371f Fix BuildResult.toString() for NoSubstituters 2023-04-03 18:17:30 +02:00
John Ericson
4a0b893d5e Stuctured command stability
Prior to this, there was an ad-hoc whitelist in `main.cc`. Now, every
command states its stability.

In a future PR, we will adjust the manual to take advantage of this new
information in the JSON.
(It will be easier to do that once we have some experimental feature
docs to link too; see #5930 and #7798.)
2023-04-03 11:48:21 -04:00
Robert Hensing
f3a6de6ba9
Merge pull request #8148 from hercules-ci/fix-issue-8119-printValue-tBlackhole-abort
eval: Fix crash on missing printValue tBlackhole case
2023-04-03 16:05:55 +02:00
Eelco Dolstra
ac4318a1b9
Merge pull request #8152 from obsidiansystems/move-querySubstitutablePathInfos
Move `querySubstitutablePathInfos` from `LocalStore` to `Store`
2023-04-03 15:36:33 +02:00
Robert Hensing
1c55544a42 eval: Fix crash on missing printValue tBlackhole case
Fixes #8119
2023-04-03 15:32:42 +02:00
Théophane Hufschmitt
70bb7b7289
Merge pull request #7610 from obsidiansystems/gate-default-settings
Punt on improper global flags for now
2023-04-03 14:02:45 +02:00
John Ericson
9383520b75 Move querySubstitutablePathInfos from LocalStore to Store
The code is not local-store-specific, so we should share it with all
stores. More uniform behavior is better, and a less store-specific
functionality is more maintainable.

This fixes a FIXME added in f73d911628 by @edolstra himself.
2023-04-02 20:32:01 -04:00
John Ericson
32d72b1696 Add more API docs to experimental-features.hh 2023-04-02 18:57:46 -04:00
John Ericson
2585bcaa50 Rework a few things with the experimental features list
- Use struct not `std::pair`, designated initializers

- Use `constexpr` array that we can index by enum tag

- It no longer segfaults; not sure why.
2023-04-02 18:17:45 -04:00
John Ericson
b2c9315bf2 Merge remote-tracking branch 'upstream/master' into list-experimental-features 2023-04-02 16:21:38 -04:00
John Ericson
5d56e2daf7 Add comparison methods for content addresses 2023-04-01 16:52:23 -04:00
John Ericson
eeecfacb43 Merge branch 'path-info' into ca-drv-exotic 2023-04-01 16:40:32 -04:00
John Ericson
1f8e1edba9 Merge commit 'a6d00a7bfb18e7ec461ac1d54203cc628aca5c66' into ca-drv-exotic 2023-04-01 15:15:38 -04:00
John Ericson
f7f44f7c96 Merge commit 'aa99005004bccc9be506a2a2f162f78bad4bcb41' into ca-drv-exotic 2023-04-01 15:15:32 -04:00
John Ericson
2ef99cd104
Merge pull request #8122 from bjornfor/use-nix-store-l-unless-experimental-enabled
Don't recommend 'nix log' unless experimental feature is enabled
2023-04-01 14:50:16 -04:00
John Ericson
f4ab297b31 Ensure all headers have #pragma once and are in API docs
`///@file` makes them show up in the internal API dos. A tiny few were
missing `#pragma once`.
2023-03-31 23:19:44 -04:00
John Ericson
abd5e7dec0 Extend internal API docs, part 2
Picking up from #8111.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-03-31 23:01:40 -04:00
Patrick Jackson
2b905d1d35 Replace unnecessary Sync<uint64_t> with std::atomic<uint64_t> 2023-03-31 17:56:25 -07:00
Eelco Dolstra
936e7c66eb Indentation 2023-03-31 17:15:03 +02:00
Eelco Dolstra
7ebaf0252a Add CanonPath::makeRelative() 2023-03-31 17:15:03 +02:00
Théophane Hufschmitt
bb6e1244a3
Merge pull request #7406 from amesgen/gitlab-fetcher
Document `gitlab` input scheme
2023-03-31 13:21:55 +02:00
Théophane Hufschmitt
e32ca3cf16
Merge pull request #8018 from tweag/ssh-password-prompt
SSH: don't erase password prompt if it is displayed
2023-03-31 12:06:10 +02:00
Patrick Jackson
804180ad52 Only lock once 2023-03-30 18:06:27 -07:00
Patrick Jackson
309753ebb5 Fix data race in copyPaths 2023-03-30 16:59:34 -07:00
John Ericson
c51d554c93 Use "raw pattern" for content address types
We weren't because this ancient PR predated it!

This is actually a new version of the pattern which addresses some
issues identified in #7479.
2023-03-30 17:12:49 -04:00
John Ericson
a6d00a7bfb Fix warning 2023-03-30 16:29:13 -04:00
John Ericson
aa99005004 Merge remote-tracking branch 'upstream/master' into path-info
Also improve content-address.hh API docs.
2023-03-30 16:28:53 -04:00
John Ericson
06d87b95bc
Merge pull request #8111 from obsidiansystems/improve-internal-api-docs
Convert a bunch of comments in headers to Doxygen documentation
2023-03-30 09:40:22 -04:00
Valentin Gagarin
368c5b8b9a
Merge pull request #7518 from fricklerhandwerk/doc-commands
separate man pages for `nix-store` and `nix-env` subcommands
2023-03-30 15:33:08 +02:00
Eelco Dolstra
1cc5e1d5b6
Merge pull request #8123 from nomeata/import-flake-docs
Docs: Explain why `import nixpkgs` works in flakes
2023-03-30 11:20:55 +02:00
Eelco Dolstra
02050bd055
Typo 2023-03-30 10:49:21 +02:00
Joachim Breitner
1b6cfe6fa1 More precise wording 2023-03-30 10:40:04 +02:00
Alexander Bantyev
36b059748d Split nix-env and nix-store documentation per-subcommand
Documentation on "classic" commands with many sub-commands are
notoriously hard to discover due to lack of overview and anchor links.
Additionally the information on common options and environment variables
is not accessible offline in man pages, and therefore often overlooked
by readers.

With this change, each sub-command of nix-store and nix-env gets its
own page in the manual (listed in the table of contents), and each own
man page.

Also, man pages for each subcommand now (again) list common options
and environment variables. While this makes each page quite long and
some common parameters don't apply, this should still make it easier
to navigate as that additional information was not accessible on the
command line at all.

It is now possible to run 'nix-store --<subcommand> --help` to display
help pages for the given subcommand.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-03-30 09:46:28 +02:00
John Ericson
8cab89a94b Convert a bunch of comments in headers to Doxygen documentation
The internal API docs now contain more useful information.

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-03-29 23:22:07 -04:00
figsoda
081aadb76c Allow @ in git refs 2023-03-29 15:27:19 -04:00
Joachim Breitner
2e6b95dac7
Update src/libexpr/primops.cc
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-03-29 13:19:51 +02:00
Vaci Koblizek
a4dd87a2b3 avoid a string copy in printHash16 2023-03-29 10:54:27 +01:00
Vaci Koblizek
ccf7ce26fe return string_view from printHashType rather than string 2023-03-29 10:44:22 +01:00
Bjørn Forsman
74d94b2a13 Don't recommend 'nix log' unless experimental feature is enabled
This fixes the issue that `nix-build`, without experimental feature
'nix-command' enabled, recommends the experimental CLI `nix log` to view
build logs. Now it'll recommend the stable `nix-store -l` CLI instead.

Fixes https://github.com/NixOS/nix/issues/8118
2023-03-28 21:02:08 +02:00
Vaci Koblizek
957f832074 Avoid a string copy during Hash::to_string 2023-03-28 17:20:33 +01:00
Joachim Breitner
3ce6d72b2b
Update src/libexpr/primops.cc
Co-authored-by: asymmetric <lorenzo@mailbox.org>
2023-03-28 14:38:39 +02:00
Joachim Breitner
605c8f7789 Docs: Explain why import nixpkgs works in flakes 2023-03-28 09:35:49 +02:00
Noah Snelson
d1d1ae7a3b Documentation: list experimental features in manual
Lists all current experimental features in the `nix.conf` manual.
2023-03-27 17:12:49 -07:00
John Ericson
8df6f7bb0f Fix more profile dirs in docs
Picking up where #8078 left off.
2023-03-27 12:10:32 -04:00
Eelco Dolstra
5e3f855526
Merge pull request #7763 from obsidiansystems/installable-wide-info
Stratify `ExtraPathInfo` along `Installable` hierarchy
2023-03-27 17:04:08 +02:00
Eelco Dolstra
237587bc0a
Merge pull request #8084 from edolstra/store-docs
Auto-generate store documentation
2023-03-27 15:46:18 +02:00
John Ericson
1d539aa287
Merge pull request #8073 from tweag/fix-root-channels-location
Fix root channels location
2023-03-27 09:37:53 -04:00
Eelco Dolstra
5a0f5b5c34
Merge pull request #8062 from edolstra/ssl-cert-file
Add a setting for configuring the SSL certificates file
2023-03-27 15:37:15 +02:00
John Ericson
7c4dea3cf3 Punt on improper global flags for now
See the note in the test.

We don't want these flags showing up for commands where they are
irrelevant.

Eventually, this needs a proper fix, but it need not be a blocker for
stabilize: for a quick-n-dirty punt, just put these flags behind the
`nix-command` unstable feature.

This is fine because they are only relevant for commands which we don't
need to stabilize for a while.
2023-03-27 09:21:50 -04:00
John Ericson
570829d67e
Merge pull request #7609 from obsidiansystems/hide-experimental-settings
Hide experimental settings
2023-03-27 09:19:29 -04:00
Eelco Dolstra
a519436e9d ssl-cert-file: Use lists 2023-03-27 14:08:39 +02:00
Eelco Dolstra
2523448bad
Merge pull request #8104 from avdv/graphml-use-long-for-narsize
nix-store: Use `long` for `narSize` in graphml output
2023-03-27 13:10:28 +02:00
Théophane Hufschmitt
128994509f Make some comments in profiles.hh doxygen-enabled
These are proper documentation of the API, so they deserve to be here
2023-03-27 10:02:10 +02:00
oxalica
2941a599fa Catch eval errors in hasContent
`legacyPackages` of nixpkgs trigger eval errors in `hasContent`, causing
the whole `legacyPackages` being skipped. We should treat it as
has-content in that case.
2023-03-26 23:05:29 +08:00
Claudio Bley
e7b9dc7c89 nix-store: Use long for narSize in graphml output 2023-03-24 17:25:17 +01:00
John Ericson
256f3e3063 Stratify ExtraPathInfo along Installable hierarchy
Instead of having a bunch of optional fields, have a few subclasses
which can have mandatory fields.

Additionally, the new `getExtraPathInfo`, and `nixpkgsFlakeRef`, are
moved to `InstallableValue`.

I did these things because https://github.com/NixOS/rfcs/pull/134 ; with
these things moved to `InstallableValue`, the base `Installable` no
longer depends on libexpr! This is a major step towards that.

Also, add a bunch of doc comments for sake of the internal API docs.
2023-03-24 12:22:40 -04:00
Eelco Dolstra
e00abd3f56
Merge pull request #7757 from obsidiansystems/more-installable-value
Move value-only methods to `InstallableValue`
2023-03-24 14:07:18 +01:00
Théophane Hufschmitt
a4c2cd3c06
Merge pull request #8098 from hercules-ci/error-cc-Only-suggest-show-trace-when-truncated-trace-items-would-be-pri
error.cc: Only suggest `--show-trace` when relevant
2023-03-24 12:07:44 +01:00
Robert Hensing
233c4cf30f error.cc: Only suggest show-trace when truncated trace items would be printed
Otherwise, a trace consisting of

frame
frame
frame
non-frame

... would reach the non-frame and print the suggestion, even though
it would have ignored the non-frame anyway.

This resulted in a peculariar situation where --show-trace would have
no apparent effect, as the trace was actually already complete.
2023-03-23 17:29:06 +01:00
Eelco Dolstra
168b6021c5 Tweaks 2023-03-23 15:32:59 +01:00
Eelco Dolstra
b79df9dedc Register LocalStore to ensure it's included in the manual 2023-03-23 15:23:13 +01:00
Eelco Dolstra
161f4b0dea Document store URLs 2023-03-23 10:38:48 +01:00
Théophane Hufschmitt
fb67c1a1fb Factor out the generation of the profile/channel directory
Make sure that all the code paths use the same one, and that the
backwards-compatibility measures are probably in place when needed
2023-03-23 10:22:42 +01:00
Théophane Hufschmitt
82bd9535dd nix-channel: Restore the old root channels directory 2023-03-23 10:22:42 +01:00
Eelco Dolstra
abc449bc30
Update src/libstore/local-store.md
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-03-23 10:12:57 +01:00
Eelco Dolstra
80f0b8d307 Fix SSHStore 2023-03-23 09:35:35 +01:00
Eelco Dolstra
b134546f08 Fix clang build 2023-03-23 09:11:15 +01:00
John Ericson
e5146a6a50
Merge pull request #8078 from obsidiansystems/profile-md
Fix `nix profile --help`
2023-03-22 19:36:12 -04:00
Eelco Dolstra
5691bac202 Improve store setting descriptions / Markdown formatting 2023-03-22 14:23:36 +01:00
Alexander Bantyev
85a2d1d94f
Add a test for nix copy over ssh
Check that nix copy can copy stuff, refuses to copy unsigned paths by
default, and doesn't hide the ssh password prompt.
2023-03-22 09:45:08 +04:00
Alexander Bantyev
5291a82cd9
SSHMaster: pause logger to show password prompt
Pause logger before starting SSH connections, and resume it after the
connection is established, so that SSH password prompts are not erased
by the logger's updates.
2023-03-22 09:45:08 +04:00
Alexander Bantyev
85df7e7ea2
Logger, ProgressBar: add a way to pause/resume
Add new virtual methods pause and resume to the Logger class, and
implement them in ProgressBar to allow to pause the bar refreshing.
2023-03-22 09:45:08 +04:00
Eelco Dolstra
16cb9b9f04 Fix rendering of fetchGit documentation
stripIndentation() doesn't support tabs, so the entire markdown ended
up indented and thus rendered as a code block.
2023-03-21 14:48:15 +01:00
Eelco Dolstra
c967c29290 Add a "help" category
This makes the help commands show up prominently at the top of the
'nix' manpage.
2023-03-21 14:43:58 +01:00
Eelco Dolstra
233b063b08 Move store docs to 'nix help-stores'
Why not 'nix help stores'? Well, 'nix help <arg>' already means 'show
help on the "arg" subcommand'.
2023-03-21 14:37:09 +01:00
Eelco Dolstra
9eb53bbf17 Support per-store Markdown documentation 2023-03-21 14:03:40 +01:00
Eelco Dolstra
7704118d28 nix describe-stores: Remove
This command was intended for docs generation, but it was never used
for that and we don't need it.
2023-03-21 14:03:40 +01:00
Eelco Dolstra
8d6d59cb1b nix store --help: Include store type documentation 2023-03-21 14:03:40 +01:00
Eelco Dolstra
3ac3e31c4d Fix misrendering of 'nix store --help'
There are no categories underneath 'nix store', so having 'nix store
copy-log' in a category rendered as ':'.
2023-03-21 12:02:19 +01:00
John Ericson
f87f1bbfe6 Fix nix profile --help
The documentation hadn't be updated to reflect the new default profile
location.
2023-03-20 14:22:23 -04:00
Eelco Dolstra
515662ad70 Cleanup 2023-03-20 18:06:08 +01:00
Eelco Dolstra
16db8dc96f Open slave pseudoterminal before CLONE_NEWUSER
Otherwise, when running as root and user namespaces are enabled,
opening the slave fails with EPERM.

Fixes "opening pseudoterminal slave: Permission denied" followed by a
hang (https://hydra.nixos.org/build/213104244), and "error: getting
sandbox mount namespace: No such file or directory" (#8072), which
happens when the child fails very quickly and consequently reading
/proc/<child>/ns fails.
2023-03-20 17:58:36 +01:00
John Ericson
4607ac7aed Fix handling of experimental features mid-parse
If we conditionally "declare" the argument, as we did before, based upon
weather the feature is enabled, commands like

    nix --experimental-features=foo ... --thing-gated-on-foo

won't work, because the experimental feature isn't enabled until *after*
we start parsing.

Instead, allow arguments to also be associated with experimental
features (just as we did for builtins and settings), and then the
command line parser will filter out the experimental ones.

Since the effects of arguments (handler functions) are performed right
away, we get the required behavior: earlier arguments can enable later
arguments enabled!

There is just one catch: we want to keep non-positional
flags...non-positional. So if

    nix --experimental-features=foo ... --thing-gated-on-foo

works, then

    nix --thing-gated-on-foo --experimental-features=foo ...

should also work.

This is not my favorite long-term solution, but for now this is
implemented by delaying the requirement of needed experimental features
until *after* all the arguments have been parsed.
2023-03-20 11:35:34 -04:00
John Ericson
aa663b7e89 Mark experimental features on settings
We hide them in various ways if the experimental feature isn't enabled.

To do this, we had to move the experimental features list out of
libnixstore, because the setting machinary itself depends on it. To do
that, we made a new `ExperimentalFeatureSettings`.
2023-03-20 11:06:40 -04:00
John Ericson
296831f641 Move enabled experimental feature to libutil struct
This is needed in subsequent commits to allow the settings and CLI args
infrastructure itself to read this setting.
2023-03-20 11:05:22 -04:00
figsoda
54b8da0b10 docs: quote URL literals 2023-03-19 11:28:10 -04:00
Linus Heckemann
f1c9d83697 Only allow reference lock files when allow-dirty is set 2023-03-19 14:12:49 +01:00
Eelco Dolstra
e53e5c38d4 Add a setting for configuring the SSL certificates file
This provides a platform-independent way to configure the SSL
certificates file in the Nix daemon. Previously we provided
instructions for overriding the environment variable in launchd, but
that obviously doesn't work with systemd. Now we can just tell users
to add

  ssl-cert-file = /etc/ssl/my-certificate-bundle.crt

to their nix.conf.
2023-03-17 18:32:18 +01:00
John Ericson
790dd2555b
Merge pull request #7891 from mupdt/shared-fs-out-links
no-op refactor: extract outLink generation into a function
2023-03-17 10:57:07 -04:00
John Ericson
c998e0172f Move value-only methods to InstallableValue
These methods would previously fail on the other `Installable`s, so
moving them to this class is more correct as to where they actually
work.

Additionally, a `InstallableValueCommand` is created to make it easier
(or rather no worse than before) to write commands that just work on
`InstallableValue`s.

Besides being a cleanup to avoid failing default methods, this gets us
closer to https://github.com/NixOS/rfcs/pull/134.
2023-03-16 20:06:46 -04:00
Eelco Dolstra
7f46ebcf90
Merge pull request #8049 from edolstra/unexpected-eof
Fix "unexpected EOF" errors on macOS
2023-03-16 16:13:42 +01:00
Linus Heckemann
af4cbdafe7 LocalDerivationGoal: set NIX_ATTRS_*_FILE correctly for sandboxed builds 2023-03-16 15:26:52 +01:00
John Ericson
eb56cb7cc7
Merge pull request #7750 from obsidiansystems/no-args-prepare
Make command infra less stateful and more regular
2023-03-15 17:23:40 -04:00
John Ericson
bc23a44c54 Make command infra less stateful and more regular
Already, we had classes like `BuiltPathsCommand` and `StorePathsCommand`
which provided alternative `run` virtual functions providing the
implementation with more arguments. This was a very nice and easy way to
make writing command; just fill in the virtual functions and it is
fairly clear what to do.

However, exception to this pattern were `Installable{,s}Command`. These
two classes instead just had a field where the installables would be
stored, and various side-effecting `prepare` and `load` machinery too
fill them in. Command would wish out those fields.

This isn't so clear to use.

What this commit does is make those command classes like the others,
with richer `run` functions.

Not only does this restore the pattern making commands easier to write,
it has a number of other benefits:

- `prepare` and `load` are gone entirely! One command just hands just
  hands off to the next.

- `useDefaultInstallables` because `defaultInstallables`. This takes
  over `prepare` for the one case that needs it, and provides enough
  flexiblity to handle `nix repl`'s idiosyncratic migration.

- We can use `ref` instead of `std::shared_ptr`. The former must be
  initialized (so it is like Rust's `Box` rather than `Option<Box>`,
  This expresses the invariant that the installable are in fact
  initialized much better.

  This is possible because since we just have local variables not
  fields, we can stop worrying about the not-yet-initialized case.

- Fewer lines of code! (Finally I have a large refactor that makes the
  number go down not up...)

- `nix repl` is now implemented in a clearer way.

The last item deserves further mention. `nix repl` is not like the other
installable commands because instead working from once-loaded
installables, it needs to be able to load them again and again.

To properly support this, we make a new superclass
`RawInstallablesCommand`. This class has the argument parsing and
completion logic, but does *not* hand off parsed installables but
instead just the raw string arguments.

This is exactly what `nix repl` needs, and allows us to instead of
having the logic awkwardly split between `prepare`,
`useDefaultInstallables,` and `load`, have everything right next to each
other. I think this will enable future simplifications of that argument
defaulting logic, but I am saving those for a future PR --- best to keep
code motion and more complicated boolean expression rewriting separate
steps.

The "diagnostic ignored `-Woverloaded-virtual`" pragma helps because C++
doesn't like our many `run` methods. In our case, we don't mind the
shadowing it all --- it is *intentional* that the derived class only
provides a `run` method, and doesn't call any of the overridden `run`
methods.

Helps with https://github.com/NixOS/rfcs/pull/134
2023-03-15 16:29:07 -04:00
John Ericson
0a140a92bb
Merge pull request #7690 from ShamrockLee/nix-hash-sri
nix-hash: support base-64 and SRI format
2023-03-15 16:02:46 -04:00
Yueh-Shun Li
c27d358abb nix-hash: support base-64 and SRI format
Add the --base64 and --sri flags for the Base64 and SRI format output.

Add the --base16 flag to explicitly specify the hexadecimal format.

Add the --to-base64 and --to-sri flag to convert a hash to the above
mentioned format.
2023-03-16 03:08:42 +08:00
figsoda
31d588db8b respect allRefs=1 when using nix flake prefetch 2023-03-15 11:29:01 -04:00
Eelco Dolstra
5eb8bfd0f1 Remove a variable in LocalDerivationGoal 2023-03-15 10:59:10 +01:00
Eelco Dolstra
6029c763c2 Change builderOut from Pipe to AutoCloseFD 2023-03-15 10:59:10 +01:00
Eelco Dolstra
c536e00c9d Move pseudoterminal slave open to child
Hopefully this fixes "unexpected EOF" failures on macOS
(#3137, #3605, #7242, #7702).

The problem appears to be that under some circumstances, macOS
discards the output written to the slave side of the
pseudoterminal. Hence the parent never sees the "sandbox initialized"
message from the child, even though it succeeded. The conditions are:

* The child finishes very quickly. That's why this bug is likely to
  trigger in nix-env tests, since that uses a builtin builder. Adding
  a short sleep before the child exits makes the problem go away.

* The parent has closed its duplicate of the slave file
  descriptor. This shouldn't matter, since the child has a duplicate
  as well, but it does. E.g. moving the close to the bottom of
  startBuilder() makes the problem go away. However, that's not a
  solution because it would make Nix hang if the child dies before
  sending the "sandbox initialized" message.

* The system is under high load. E.g. "make installcheck -j16" makes
  the issue pretty reproducible, while it's very rare under "make
  installcheck -j1".

As a fix/workaround, we now open the pseudoterminal slave in the
child, rather than the parent. This removes the second condition
(i.e. the parent no longer needs to close the slave fd) and I haven't
been able to reproduce the "unexpected EOF" with this.
2023-03-15 10:58:37 +01:00
Eelco Dolstra
19326ac297 Simplify commonChildInit() 2023-03-14 18:58:12 +01:00
Linus Heckemann
3a1de4c3fe Apply review suggestions
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-03-14 12:02:03 +01:00
Linus Heckemann
e91596eb69 Allow specifying alternative paths for reading/writing flake locks
This allows having multiple separate lockfiles for a single
project, which can be useful for testing against different versions of
nixpkgs; it also allows tracking custom input overrides for remote
flakes without requiring local clones of these flakes.

For example, if I want to build Nix against my locally pinned nixpkgs,
and have a lock file tracking this override independently of future
updates to said nixpkgs:

nix flake lock --output-lock-file /tmp/nix-flake.lock --override-input nixpkgs flake:nixpkgs
nix build --reference-lock-file /tmp/nix-flake.lock

Co-Authored-By: Will Fancher <elvishjerricco@gmail.com>
2023-03-13 21:08:52 +01:00
Robert Hensing
a387f46967
Merge pull request #8033 from lbodor/stop-adding-dot-to-nix-dev-env-path
`print-dev-env`: stop inadvertently adding `.` to `PATH`
2023-03-13 19:52:41 +01:00
Eelco Dolstra
fcd0b0fbd5 Fix macOS warning 2023-03-13 05:31:03 -07:00
lbodor
526bdbda3c print-dev-env: stop inadvertently adding . to PATH 2023-03-12 22:40:47 +11:00
Théophane Hufschmitt
99af56cd0d
Merge pull request #8015 from tweag/progress-during-nix-copy
Display progress when running copyPaths (nix copy)
2023-03-10 16:23:22 +01:00
Eelco Dolstra
4f3a4b732c Make findRuntimeRoots() more resilient to disappearing processes
I saw this random failure in https://hydra.nixos.org/build/211811692:

  error: opening /proc/15307/fd: No such process

while running nix-collect-garbage in a readfile-context.sh. This is
because we're not handling ESRCH errors reading /proc/<pid>/fd. So
just move the read inside the try/catch where we do handle it.
2023-03-09 16:44:51 +01:00
Eelco Dolstra
8f96720b9f Fix another uninitialized variable
https://hydra.nixos.org/build/211811494
2023-03-09 15:30:53 +01:00
Alexander Bantyev
4bef2016a1
Display progress when running copyPaths (nix copy)
`nix copy` operations did not show progress. This is quite confusing.

Add a `progressSink` which displays the progress during `copyPaths`,
pretty much copied from `copyStorePath`.

Fixes https://github.com/NixOS/nix/issues/8000
2023-03-09 16:03:48 +04:00
Théophane Hufschmitt
39700c5cbe
Revert "Disable GC during coroutine execution + test" 2023-03-08 20:47:52 +01:00
Eelco Dolstra
1607ad5076 Fix uninitialized readFromStdIn variable
This was causing random failures in tests/ca/substitute.ca: 'nix copy
--file ./content-addressed.nix' wouldn't get the default installable
'.' applied in InstallablesCommand::load(), so it would do nothing.
2023-03-08 13:53:20 +01:00
Eelco Dolstra
df48040786
Merge pull request #7994 from edolstra/fix-ca-crash
Fix crash/hang with CA derivations
2023-03-08 12:15:55 +01:00
Eelco Dolstra
6b243ef2d5
Merge pull request #7993 from tweag/fix-profile-gc
Fix `nix-collect-garbage -d` with the new profile location
2023-03-08 11:10:45 +01:00
Eelco Dolstra
7bfed34367 Fix crash/hang with CA derivations
The curl download can outlive DrvOutputSubstitutionGoal (if some other
error occurs), so at shutdown setting the promise to an exception will
fail because 'this' is no longer valid in the callback. This can
manifest itself as a segfault, "corrupted double-linked list" or hang.
2023-03-08 11:09:15 +01:00
Théophane Hufschmitt
4a6244dcf7
Merge pull request #7725 from yorickvP/check-coro-gc
Disable GC during coroutine execution + test
2023-03-08 09:51:46 +01:00
Théophane Hufschmitt
2272bc6049 Fix nix-collect-garbage -d with the new profile location
Low-hanging fix for https://github.com/NixOS/nix/pull/5226#issuecomment-1454669399
2023-03-07 13:08:04 +01:00
Théophane Hufschmitt
ba0486f045
Merge pull request #7889 from sidkshatriya/sorted-fetch-paths
Print the store paths to be fetched sorted by StorePath name()
2023-03-07 11:58:10 +01:00
mupdt
762cf7f214 no-op refactor: extract outLink generation into a function 2023-03-06 10:20:21 -05:00
Eelco Dolstra
3f4dc61f7f
Merge pull request #7958 from amesgen/issue-7955
InstallableFlake: Apply nix config in `getCursors`
2023-03-06 11:08:51 +01:00
Eelco Dolstra
92771ca1be
Merge pull request #7975 from fricklerhandwerk/fix-fetchgit
remove indentation in `fetchGit` attribute listing
2023-03-06 11:00:40 +01:00
Valentin Gagarin
dfeb83cac1 output paths of store derivations are now addressed explicitly 2023-03-06 03:32:14 +01:00
Valentin Gagarin
f69d453c3e remove indentation in fetchGit attribute listing
also reword a confusing sentence and add links to Git terminology
2023-03-05 04:44:11 +01:00
Valentin Gagarin
2af9fd20c6 clarify definition of "installable"
the term was hard to discover, as its definition and explanation were in
a very long document lacking an overview section.
search did not help because it occurs so often.

- clarify wording in the definition
- add an overview of installable types
- add "installable" to glossary
- link to definition from occurrences of the term
- be more precise about where store derivation outputs are processed
- installable Nix expressions must evaluate to a derivation

Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com>
2023-03-05 01:46:17 +01:00
Valentin Gagarin
1e87d5f1ea make descriptions of each installable type an own subsection
this is easier to edit, provides anchors for free, and renders correctly
on the terminal without additional effort.
2023-03-05 01:46:17 +01:00
amesgen
77bb950566
InstallableFlake: Apply nix config in getCursors 2023-03-04 12:52:20 +01:00
Sidharth Kshatriya
427555861b Print the store paths to be fetched sorted by StorePath name() and not baseName
Presently when nix says something like:

```
these 486 paths will be fetched (511.54 MiB download, 6458.64 MiB unpacked):
 ...path1
 ...path2
 ...path3
    ...
    ...
 ...path486
```

It sorts path1, path2, path3, ..., path486 in lexicographic order of the
store path.

After this commit, nix will show path1, path2, path3, ..., path486 sorted by
StorePath name() (basically everything after the hash) rather than the store path.

This makes it easier to review what exactly is being downloaded at a glance,
especially when many paths need to be fetched.
2023-03-03 21:46:35 +05:30
Eelco Dolstra
0507462c06
Merge pull request #7918 from zimbatm/fix-empty-nix-store-env
treat empty NIX_STORE_DIR env vars as unset
2023-03-03 13:49:56 +01:00
Yorick van Pelt
2683734936
Add talkative msg for coro gc debug 2023-03-03 11:43:47 +01:00
zimbatm
dc8820c71f
fixup: use same style as getEnv 2023-03-03 11:34:36 +01:00
zimbatm
b96d9c1687
fixup: remove warning entirely
fixes 72e1e23051 (r1124211067)
2023-03-03 11:32:06 +01:00
Eelco Dolstra
f0908f592c
Merge pull request #7942 from edolstra/remove-format
Remove FormatOrString and remaining uses of format()
2023-03-03 10:22:11 +01:00
Valentin Gagarin
37c61907d7
Merge pull request #7605 from tweag/comments
add comments
2023-03-03 06:38:29 +01:00
Théophane Hufschmitt
1f394d2107
Merge branch 'master' into paths-from-stdin 2023-03-02 19:20:51 +01:00
Yorick van Pelt
5bb6e3bfaf
NixRepl::mainLoop: restore old curRepl on function exit
This fixes completion callbacks after entering and leaving a nested
debugger.
2023-03-02 17:24:58 +01:00
Eelco Dolstra
687d86f8ab
Merge pull request #7940 from tweag/fix-http-errors-encoding
Log the decompressed body in case of http errors
2023-03-02 16:25:09 +01:00
Jonas Chevalier
72e1e23051
Update src/libutil/util.cc
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-03-02 16:17:20 +01:00
Eelco Dolstra
29abc8e764 Remove FormatOrString and remaining uses of format() 2023-03-02 15:57:54 +01:00
Eelco Dolstra
b69a73a230 Get rid of some unchecked calls to std::cout 2023-03-02 15:02:24 +01:00
Théophane Hufschmitt
1a825b6919 Log the decompressed body in case of http errors
Don't show the users the raw (possibly compressed) error message as
everyone isn't able to decompress brotli in their brain.
2023-03-02 14:59:15 +01:00
Eelco Dolstra
09f5975c6a Logger::cout: Use fmt()
This ensures that in cout(s), 's' does not get interpreted as a format
string.
2023-03-02 14:52:37 +01:00
Eelco Dolstra
989b823ac5 nix store cat: Use writeFull()
Fixes #7939.
2023-03-02 14:51:14 +01:00
Eelco Dolstra
c13cbd20ab Logger::writeToStdout(): Use writeFull()
This ensures that write errors do not get ignored.
2023-03-02 14:46:28 +01:00
Eelco Dolstra
767974f411
Merge pull request #7924 from mkenigs/valid
Always set valid in path-info --json output
2023-03-02 09:58:20 +01:00
Félix Baylac Jacqué
25300c0ecd
Treat empty env var paths as unset
We make sure the env var paths are actually set (ie. not "") before
sending them to the canonicalization function. If we forget to do so,
the user will end up facing a puzzled failed assertion internal error.

We issue a non-failing warning as a stop-gap measure. We could want to
revisit this to issue a detailed failing error message in the future.
2023-03-01 20:50:07 +01:00
Théophane Hufschmitt
182129d985
Merge pull request #7932 from obsidiansystems/remove-sameMachine
Remove dead code `RemoteStore::sameMachine`
2023-03-01 20:05:39 +01:00
Valentin Gagarin
651dab55da
Merge pull request #7854 from aameen-tulip/patch-1 2023-03-01 18:24:53 +01:00
John Ericson
b7f01a82a9 Remove dead code RemoteStore::sameMachine
It has been dead code since 9747ea84b4.
2023-03-01 11:10:30 -05:00
Yorick van Pelt
176005749c
Always disable GC in a coroutine unless the patch is applied 2023-03-01 15:07:00 +01:00
Yorick van Pelt
4c73eab923
DisableGC: replace by CoroutineContext, std::shared_ptr<void> 2023-03-01 13:55:41 +01:00
Yorick van Pelt
53bb4a5327
tests/coro-gc: refactor and split into 3 tests 2023-03-01 13:55:41 +01:00
Yorick van Pelt
eaeb994d8b
Disable GC inside coroutines on mac OS 2023-03-01 13:55:41 +01:00
Yorick van Pelt
0fd8f542a8
tests/coro-gc: create test for boehm stack patch
Regression test for #7679
2023-03-01 13:55:37 +01:00
Valentin Gagarin
306e5c5ce5
Merge pull request #7788 from bobvanderlinden/pr-improve-nix-profile-install-error
Improve error on conflict for nix profile install
2023-03-01 11:48:43 +01:00
Bob van der Linden
12538605fd
nix-profile: add FIXME about using C++20 std::ranges 2023-03-01 07:40:53 +01:00
Matthew Kenigsberg
f86f2b973f Always set valid in path-info --json output
Currently the valid key is only present when the path is invalid, which
makes checking path validity more complex than it should be. With this
change, the valid key can always be used to check if a path is valid
2023-02-28 16:04:41 -07:00
John Ericson
ea0adfc582 Get rid of .drv special-casing for store path installables
The release notes document the change in behavior, I don't include it
here so there is no risk to it getting out of sync.

> Motivation

>> Plumbing CLI should be simple

Store derivation installations are intended as "plumbing": very simple
utilities for advanced users and scripts, and not what regular users
interact with. (Similarly, regular Git users will use branch and tag
names not explicit hashes for most things.)

The plumbing CLI should prize simplicity over convenience; that is its
raison d'etre. If the user provides a path, we should treat it the same
way not caring what sort of path it is.

>> Scripting

This is especially important for the scripting use-case. when arbitrary
paths are sent to e.g. `nix copy` and the script author wants consistent
behavior regardless of what those store paths are. Otherwise the script
author needs to be careful to filter out `.drv` ones, and then run `nix
copy` again with those paths and `--derivation`. That is not good!

>> Surprisingly low impact

Only two lines in the tests need changing, showing that the impact of
this is pretty light.

Many command, like `nix log` will continue to work with just the
derivation passed as before. This because we used to:

- Special case the drv path and replace it with it's outputs (what this
  gets rid of).

- Turn those output path *back* into the original drv path.

Now we just skip that entire round trip!

> Context

Issue #7261 lays out a broader vision for getting rid of `--derivation`,
and has this as one of its dependencies. But we can do this with or
without that.

`Installable::toDerivations` is changed to handle the case of a
`DerivedPath::Opaque` ending in `.drv`, which is new: it simply doesn't
need to do any extra work in that case. On this basis, commands like
`nix {show-derivation,log} /nix/store/...-foo.drv` still work as before,
as described above.

When testing older daemons, the post-build-hook will be run against the
old CLI, so we need the old version of the post-build-hook to support
that use-case.

Co-authored-by: Travis A. Everett <travis.a.everett@gmail.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-02-28 17:07:05 -05:00
Timothy DeHerrera
df643051e2
nix-store: read paths from standard input
Resolves #7437 for new `nix-store` by adding a `--stdin` flag.
2023-02-28 12:29:16 -07:00
Timothy DeHerrera
269caa5317
feat: read installable paths from stdin
Resolves #7437 for new `nix` commands only by adding a `--stdin` flag.

If paths are also passed on the cli they will be combined with the ones
from standard input.
2023-02-28 12:29:15 -07:00
John Ericson
5abd643c6d Merge branch 'path-info' into ca-drv-exotic 2023-02-28 12:46:00 -05:00
John Ericson
d381248ec0 No inheritance for TextInfo and FixedOutputInfo 2023-02-28 12:14:11 -05:00
John Ericson
85bb865d20 Revert "Remove some designated initializers"
This reverts commit ee9eb83a84.
2023-02-28 11:57:20 -05:00
John Ericson
123b11ff83 Clarify store path grammar and improve comment on makeType 2023-02-28 11:49:13 -05:00
John Ericson
d12f57c2c0 Merge remote-tracking branch 'upstream/master' into path-info 2023-02-28 11:34:34 -05:00
John Ericson
c36b584f8e Fix typo in the method name 2023-02-28 11:34:18 -05:00
Eelco Dolstra
a4a5d828e2
Merge pull request #7793 from layus/interrupt_downloads
Check interrupts even when download stalled
2023-02-28 13:29:29 +01:00
Eelco Dolstra
da1f49c4e3
Merge pull request #7904 from anatol/patch-1
Example uses gitlab.com thus clarify the comment
2023-02-28 13:29:10 +01:00
Bob van der Linden
872cdb4346
nix-profile-install: show helpful error upon package conflict
Whenever a file conflict happens during "nix profile install" an error
is shown that was previously thrown inside builtins.buildEnv.

We catch BuildProfileConflictError here so that we can provide the user
with more useful instructions on what to do next.

Most notably, we give the user concrete commands to use with all
parameters  already filled in. This avoids the need for the user to look
up these commands in manual pages.
2023-02-28 09:28:05 +01:00
Eelco Dolstra
3d15dbadc2
Merge pull request #7911 from edolstra/revert-7689
Revert #7689
2023-02-28 08:46:55 +01:00
Bob van der Linden
3113b13df9
buildenv: throw BuildEnvFileConflictError with more context
At the moment an Error is thrown that only holds an error message
regarding `nix-env` and `nix profile`. These tools make use of
builtins.buildEnv, but buildEnv is also used in other places. These
places are unrelated to Nix profiles, so the error shouldn't mention
these tools.

This generic error is now BuildEnvFileConflictError, which holds more
contextual information about the files that were conflicting while
building the environment.
2023-02-27 21:39:34 +01:00
Théophane Hufschmitt
eae89aca1b
Merge pull request #7776 from yorickvP/fix-path-escapes-7707
Properly escape local paths into URLs in fetchTree
2023-02-27 21:10:25 +01:00
Eelco Dolstra
f08ad5bdba
Merge pull request #7913 from fricklerhandwerk/master
add information on the `build-hook` setting
2023-02-27 17:33:25 +01:00
Valentin Gagarin
fd0e21475c add information on the build-hook setting
add a warning that you probably never want to change this.
2023-02-27 16:27:56 +01:00
Yorick van Pelt
2c0866fc3f
fetchTree: convert fs path to url via ParsedURL::to_string 2023-02-27 15:30:04 +01:00
Yorick van Pelt
0844856c84
url: make percentEncode stricter, expose and unit test it 2023-02-27 15:30:00 +01:00
Eelco Dolstra
dd93c12c6a Revert "getDefaultNixPath: actually respect {restrict,pure}-eval"
This reverts commit 1cba5984a6.
2023-02-27 15:11:36 +01:00
Eelco Dolstra
e928c72cf9 Revert "Document default nix-path value"
This reverts commit dba9173a1d.
2023-02-27 14:16:49 +01:00
Théophane Hufschmitt
995bfeef3b
Merge pull request #7796 from hercules-ci/fix-7263
Ensure that `self.outPath == ./.`
2023-02-27 10:26:02 +01:00
Anatol Pomozov
d731235f6b
Example uses gitlab.com thus clarify the comment 2023-02-24 07:17:47 -08:00
Théophane Hufschmitt
4a921ba43b
Merge pull request #7764 from yorickvP/build-remote-warning
build-remote: don't warn when all local build slots are taken
2023-02-22 17:43:20 +01:00
Robert Hensing
5d834c40d0 flakes: Differentiate self.outPath and self.sourceInfo.outPath
It would be incorrect to say that the `sourceInfo` has an `outPath`
that isn't the root. `sourceInfo` is about the root, whereas only
the flake may not be about the root. Thanks Eelco for pointing that
out.
2023-02-22 03:31:24 +01:00
Robert Hensing
904a107d16 flakes: Ensure that self.outPath == ./.
Users expect `self` to refer to the directory where the `flake.nix`
file resides.
2023-02-22 03:30:47 +01:00
Théophane Hufschmitt
8418d22ac1
Merge pull request #7874 from obsidiansystems/fix-no-gc-build
Fix the build without GC
2023-02-21 16:14:51 +01:00
Théophane Hufschmitt
c7885ab6f2
Merge pull request #7755 from obsidiansystems/mix-read-only-mode
Make `--read-only` a separate mixin
2023-02-21 16:13:48 +01:00
John Ericson
5b0175e81d Fix the build without GC
I had given it an improper trailing comma in
1bd03ad100.
2023-02-21 09:38:46 -05:00
Théophane Hufschmitt
532c70f531
Merge pull request #7856 from yorickvP/fix-nsswitch
Wait with making /etc unwritable until after build env setup
2023-02-21 09:39:10 +01:00
John Ericson
208c8d326d Derivation::toJSON: fix bug!
When I moved this code from the binary to libnixstore #7863, I forgot to
display the environment variables!
2023-02-20 17:38:57 -05:00
Robert Hensing
c7bd3a874f
Merge pull request #7863 from obsidiansystems/test-derivation-to-json
Move Derivation JSON printing logic to lib and test it
2023-02-20 23:21:50 +01:00
John Ericson
0258ac9c2a Make --read-only a separate mixin
It is independent of SourceExprCommand, which is about parsing
installables, except for the fact that parsing installables is one of
the many things influenced by read-only mode.
2023-02-20 10:39:18 -05:00
John Ericson
1bd03ad100 Split out CmdRepl and editorFor
The REPL itself and the `nix repl` CLI are conceptually different
things, and thus deserve to be in different files.
2023-02-20 09:45:29 -05:00
John Ericson
57a2e46ee0 Slight cleanup of InstallablesCommand::load 2023-02-20 09:09:11 -05:00
John Ericson
fa4733fce5 Split out InstallableFlake and InstallableAttrPath 2023-02-20 09:09:11 -05:00
Valentin Gagarin
dda83a59c1
Merge pull request #7158 from sternenseemann/foldl-strict-accumulation-value 2023-02-19 23:54:14 +01:00
John Ericson
7998686c00 Test toJSON of DerivationOutput and Derivation 2023-02-19 11:12:12 -05:00
John Ericson
cd583362ec Move Derivation toJSON logic to libnixstore 2023-02-19 10:06:40 -05:00
Yorick van Pelt
bbba49b3e4
Wait with making /etc unwritable until after build env setup
This fixes /etc/nsswitch.conf
2023-02-17 16:34:45 +01:00
aameen-tulip
b31d4b689c
Document hasAllInfo
If this documentation is inaccurate in any way please do not hesitate to suggest corrections.

My understanding of this function is strictly from reading the source code and some limited experience implementing fetchers.
2023-02-16 18:47:45 -06:00
Robert Hensing
a88ae62bc0
Merge pull request #7811 from Et7f3/fix_memory_leaks
Reduce memory leaks
2023-02-16 21:21:54 +01:00
Et7f3
cec23f5dda ExprOpHasAttr,ExprSelect,stripIndentation,binds,formals: delete losts objects
We are looking for *$ because it indicate that it was constructed with a new but
not release. De-referencing shallow copy so deleting as whole might create
dangling pointer that's why we move it so we delete a empty containers + the
nice perf boost.
2023-02-16 19:53:55 +01:00
Yorick van Pelt
49fd72a903
Make /etc writability conditional on uid-range feature 2023-02-14 13:55:41 +01:00
Yorick van Pelt
db41f74af3
Don't allow writing to /etc 2023-02-14 12:03:34 +01:00
tomberek
601faa00d7
Merge pull request #7744 from obsidiansystems/split-installable-store-path
Factor out `InstallableStorePath` to its own file, dedup
2023-02-13 08:57:19 -05:00
Eelco Dolstra
c205d10c66
Merge pull request #7616 from hercules-ci/fix-3898
Fix foreign key error inserting into NARs #3898
2023-02-13 13:02:19 +01:00
Eelco Dolstra
2037f8a3ee
Merge pull request #7804 from PJungkamp/fix-completions
Infer short completion descriptions for commandline flags
2023-02-13 11:26:38 +01:00
Et7f3
fa89d317b7 ExprString: Avoid copy of string 2023-02-12 05:49:45 +01:00
Et7f3
3d16f2a281 parser: use implicit rule 2023-02-12 05:49:45 +01:00
Philipp Jungkamp
30edd7af53 Completions::add use libutil trim() 2023-02-10 22:17:09 +01:00
Eelco Dolstra
b3d29e80e0
Merge pull request #7805 from edolstra/c++2a
Fix building with GCC 9
2023-02-10 20:41:29 +01:00
Eelco Dolstra
67451d8ed7
Merge pull request #7802 from edolstra/fix-7783
Fix PID namespace support check
2023-02-10 20:41:13 +01:00
Eelco Dolstra
5978ceb271 Fix building with GCC 9
Nixpkgs on aarch64-linux is currently stuck on GCC 9
(https://github.com/NixOS/nixpkgs/issues/208412) and using gcc11Stdenv
doesn't work either.

So use c++2a instead of c++20 for now. Unfortunately this means we
can't use some C++20 features for now (like std::span).
2023-02-10 18:38:57 +01:00
Théophane Hufschmitt
9ebbe35817
Merge pull request #5588 from tweag/balsoft/xdg
Follow XDG Base Directory standard
2023-02-10 18:05:50 +01:00
Philipp Jungkamp
a537095e1f Infer short completion descriptions for commandline flags
Descriptions for commandline flags may not include newlines and should
be rather short for display in a shell. Truncate the description string
of a flag on '\n' or '.' to and add an ellipsis if needed.
2023-02-10 18:03:19 +01:00
Alexander Bantyev
2384d36083
A setting to follow XDG Base Directory standard
XDG Base Directory is a standard for locations for storing various
files. Nix has a few files which seem to fit in the standard, but
currently use a custom location directly in the user's ~, polluting
it:

- ~/.nix-profile
- ~/.nix-defexpr
- ~/.nix-channels

This commit adds a config option (use-xdg-base-directories) to follow
the XDG spec and instead use the following locations:

- $XDG_STATE_HOME/nix/profile
- $XDG_STATE_HOME/nix/defexpr
- $XDG_STATE_HOME/nix/channels

If $XDG_STATE_HOME is not set, it is assumed to be ~/.local/state.

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
Co-authored-by: Tim Fenney <kodekata@gmail.com>
Co-authored-by: pasqui23 <pasqui23@users.noreply.github.com>
Co-authored-by: Artturin <Artturin@artturin.com>
Co-authored-by: John Ericson <Ericson2314@Yahoo.com>
2023-02-10 20:14:06 +04:00
Eelco Dolstra
c49b7472ea Fix macOS build 2023-02-10 17:08:33 +01:00
Eelco Dolstra
3e6e34cdf5 LocalDerivationGoal::startBuilder(): Use startProcess() to clone 2023-02-10 14:44:25 +01:00
Robert Hensing
37b1e93f4b daemon.cc: Rename UserSettings -> AuthorizationSettings
This is a bit more accurate.

It's a private name, but before you know it, someone might make it public!
2023-02-10 14:41:39 +01:00
Eelco Dolstra
f094ba7386 Simplify the PID namespace check: just try to mount /proc
Fixes #7783.
2023-02-10 14:38:14 +01:00
Guillaume Maudoux
e6ad8e8440 nit: cleaner diff 2023-02-10 00:57:56 +01:00
Guillaume Maudoux
aa18404ecb Flush data when download ends 2023-02-10 00:54:29 +01:00
Théophane Hufschmitt
5597d68e2d
Merge pull request #7754 from obsidiansystems/narrower-scope-derivation-flag
Scope down `--derivation` to just the commands that use it
2023-02-09 19:51:43 +01:00
Eelco Dolstra
0a7071ed33
Merge pull request #7774 from edolstra/submodule-fixes
Git submodule fixes
2023-02-09 17:19:48 +01:00
Eelco Dolstra
e46429f674
Merge pull request #7712 from Mic92/advertise-compressions
advertise transport encoding in http transfers to
2023-02-09 17:15:25 +01:00
Eelco Dolstra
862e56c23d
Improve comment
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-02-09 16:42:45 +01:00
Eelco Dolstra
15313bfdb7
Fix activity message
Co-authored-by: Josef Kemetmüller <josef.kemetmueller@gmail.com>
2023-02-09 16:42:14 +01:00
Guillaume Maudoux
78fea899e0 Check interupts even when download stalled 2023-02-09 13:56:50 +01:00
Joachim Breitner
e4a2a5c074
Documentation: builtins.fetchGit when used on a local path (#7706)
* Documentation:  builtins.fetchGit when used on a local path

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-02-08 11:53:28 +00:00
Robert Hensing
19b495a48a NarInfoDiskCache: Also test id consistency with updated fields
And clarify test
2023-02-07 23:34:36 +01:00
Robert Hensing
fb94d5cabd NarInfoDiskCache: Keep BinaryCache.id stable and improve test
Fixes #3898

The entire `BinaryCaches` row used to get replaced after it became
stale according to the `timestamp` column. In a concurrent scenario,
this leads to foreign key conflicts as different instances of the
in-process `state.caches` cache now differ, with the consequence that
the older process still tries to use the `id` number of the old record.

Furthermore, this phenomenon appears to have caused the cache for
actual narinfos to be erased about every week, while the default
ttl for narinfos was supposed to be 30 days.
2023-02-07 23:34:36 +01:00
Robert Hensing
2ceece3ef3 NarInfoDiskCache: Prepare reproducer for #3898 2023-02-07 23:34:36 +01:00
Robert Hensing
79f62d2dda NarInfoDiskCacheImpl: Make dbPath a parameter
This allows testing with a clean database.
2023-02-07 23:34:36 +01:00
Robert Hensing
29f0b196f4 NarInfoDiskCache: Rename cacheExists -> upToDateCacheExists
This is slightly more accurate considering that an outdated record
may exist in the persistent cache. Possibly-outdated records are
quite relevant as they may be foreign keys to more recent information
that we want to keep, but we will not return them here.
2023-02-07 23:34:36 +01:00
Robert Hensing
8a0ef5d58e sqlite.cc: Add SQL tracing
Set environment variable NIX_DEBUG_SQLITE_TRACES=1 to log all sql statements.
2023-02-07 23:34:36 +01:00
Eelco Dolstra
0a70b411e1 Print debug message if a namespace test fails 2023-02-07 23:01:39 +01:00
Eelco Dolstra
c5c0617d6f Mention --no-sandbox if sandboxing is unsupported 2023-02-07 22:59:46 +01:00
Eelco Dolstra
4e61877b5c More #ifdef 2023-02-07 22:51:53 +01:00
Eelco Dolstra
d834de2894 Fix macOS build 2023-02-07 22:51:53 +01:00
Eelco Dolstra
bc1d9fd8b5 Check whether we can use PID namespaces
In unprivileged podman containers, /proc is not fully visible (there
are other filesystems mounted on subdirectories of /proc). Therefore
we can't mount a new /proc in the sandbox that matches the PID
namespace of the sandbox. So this commit automatically disables
sandboxing if /proc is not fully visible.
2023-02-07 22:51:53 +01:00
Eelco Dolstra
fb2f7f5dcc Fix auto-uid-allocation in Docker containers
This didn't work because sandboxing doesn't work in Docker. However,
the sandboxing check is done lazily - after clone(CLONE_NEWNS) fails,
we retry with sandboxing disabled. But at that point, we've already
done UID allocation under the assumption that user namespaces are
enabled.

So let's get rid of the "goto fallback" logic and just detect early
whether user / mount namespaces are enabled.

This commit also gets rid of a compatibility hack for some ancient
Linux kernels (<2.13).
2023-02-07 22:51:53 +01:00
Eelco Dolstra
7a6daf61e8 Fix activity message 2023-02-07 22:22:50 +01:00
Eelco Dolstra
a8fe0dc16c Speed up fetching submodules
Previously we would completely refetch the submodules from the
network, even though the repo might already have them. Now we copy the
.git/modules directory from the repo as an optimisation. This speeds
up evaluating

  builtins.fetchTree { type = "git"; url = "/path/to/blender"; submodules = true; }

(where /path/to/blender already has the needed submodules) from 121s
to 57s.

This is still pretty inefficient and a hack, but a better solution is
best done on the lazy-trees branch.

This change also help in the case where the repo already has the
submodules but the origin is unfetchable for whatever reason
(e.g. there have been cases where Nix in a GitHub action doesn't have
the right authentication set up).
2023-02-07 16:01:36 +01:00
Eelco Dolstra
2edd5cf618 Fix the origin URL used for fetching submodules
We cannot use 'actualUrl', because for file:// repos that's not the
original URL that the repo was fetched from. This is a problem since
submodules may be relative to the original URL.

Fixes e.g.

  nix eval --impure --json --expr 'builtins.fetchTree { type = "git"; url = "/path/to/blender"; submodules = true; }'

where /path/to/blender is a clone of
https://github.com/blender/blender.git (which has several relative
submodules like '../blender-addons.git').
2023-02-07 16:01:36 +01:00
Eelco Dolstra
81e75e4bf6 Add some progress indication when fetching submodules 2023-02-07 16:01:36 +01:00
Yorick van Pelt
631ba6442a
build-remote: store maxBuildJobs before forcing it to 1 2023-02-07 12:08:00 +01:00
Yorick van Pelt
3050005211
build-remote: don't warn when all local build slots are taken
Previously, build-remote would show a warning if all build slots were
taken, even if they would open up later. This caused a lot of spam in
the logs. Disable this warning when maxJobs > 0.

See #6263
2023-02-06 17:53:03 +01:00
John Ericson
44bea52ae3 Scope down --derivation to just the commands that use it
Per the old FIXME, this flag was on too many commands, and mostly
ignored. Now it is just on the commands where it actually has an effect.

Per https://github.com/NixOS/nix/issues/7261, I would still like to get
rid of it entirely, but that is a separate project. This change should
be good with or without doing that.
2023-02-04 18:30:02 -05:00
John Ericson
45fa297e40 Factor out InstallableStorePath to its own file, dedup
`nix app` had something called `InstallableDerivedPath` which is
actually the same thing. We go with the later's name because it has
become more correct.

I originally did this change (more hurriedly) as part of #6225 --- a
mini store-only Nix and a full Nix need to share this code. In the first
RFC meeting for https://github.com/NixOS/rfcs/pull/134 we discussed how
some splitting of the massive `installables.cc` could begin prior, as
that is a good thing anyways. (@edolstra's words, not mine!) This would
be one such PR.
2023-02-03 11:26:39 -05:00
Jörg Thalheim
f20d3726dd advertise transport encoding in http transfers to
tl;dr: With this 1 line change I was able to get a speedup of 1.5x on 1Gbit/s
wan connections by enabling zstd compression in nginx.

Also nix already supported all common compression format for http
transfer, webservers usually only enable them if they are advertised
through the Accept-Encoding header.

This pull requests makes nix advertises content compression support for
zstd, br, gzip and deflate.

It's particular useful to add transparent compression for binary caches
that serve packages from the host nix store in particular nix-serve,
nix-serve-ng and harmonia.

I tried so far gzip, brotli and zstd, whereas only zstd was able to bring
me performance improvements for 1Gbit/s WAN connections.

The following nginx configuration was used in combination with the
[zstd module](https://github.com/tokers/zstd-nginx-module) and
[harmonia](https://github.com/nix-community/harmonia/)

```nix
{
  services.nginx.virtualHosts."cache.yourhost.com" = {
    locations."/".extraConfig = ''
      proxy_pass http://127.0.0.1:5000;
      proxy_set_header Host $host;
      proxy_redirect http:// https://;
      proxy_http_version 1.1;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;

      zstd on;
      zstd_types application/x-nix-archive;
    '';
  };
}
```

For testing I unpacked a linux kernel tarball to the nix store using
this command `nix-prefetch-url --unpack https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.1.8.tar.gz`.

Before:

```console
$ nix build && rm -rf /tmp/hello  && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwbb88sw36fvah-linux-6.1.8.tar.gz'
warning: Git tree '/scratch/joerg/nix' is dirty

real    0m18,375s
user    0m2,889s
sys     0m1,558s
```

After:

```console
$ nix build && rm -rf /tmp/hello  && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwb
b88sw36fvah-linux-6.1.8.tar.gz'

real    0m11,884s
user    0m4,130s
sys     0m1,439s
```

Signed-off-by: Jörg Thalheim <joerg@thalheim.io>

Update src/libstore/filetransfer.cc

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-02-03 12:33:38 +00:00
Eelco Dolstra
dbe0748f97
Merge pull request #7739 from obsidiansystems/user-settings
Move `trustedUsers` and `allowedUsers` to separate config struct
2023-02-03 11:55:37 +01:00
John Ericson
a47e055e09 Move trustedUsers and allowedUsers to separate config struct
These settings are not needed for libstore at all, they are just used by
the nix daemon *command* for authorization on unix domain sockets. My
moving them to a new configuration struct just in that file, we avoid
them leaking anywhere else.

Also, it is good to break up the mammoth `Settings` struct in general.
Issue #5638 tracks this.

The message is not changed because I do not want to regress in
convenience to the user. Just saying "this connection is not trusted"
doesn't tell them out to fix the issue. The ideal thing to do would be
to somehow parameterize `processCommand` on how the error should be
displayed, so different sorts of connections can display different
information to the user based on how authentication is performed for the
connection in question. This, however, is a good bit more work, so it is
left for the future.

This came up with me thinking about the tcp:// store (#5265). The larger
project is not TCP *per se*, but the idea that it should be possible for
something else to manage access control to services like the Nix Daemon,
and those services simply trust or trust the incoming connection as they
are told. This is a more capability-oriented way of thinking about trust
than "every server implements its own auth separately" as we are used to today.

Its very great that libstore itself already implements just this model,
and so via this refactor I basically want to "enshrine" that so it
continues to be the case.
2023-02-02 14:17:24 -05:00
John Ericson
479c011784 Get rid of the authHook parameter on processConnection
This is (morally) dead code.

As @edolstra pointed out in
https://github.com/NixOS/nix/pull/5226#discussion_r1073470813, this is
no longer needed.

I created this in 8d4162ff9e, so it is
fitting that I now destroy it :).
2023-02-02 12:02:03 -05:00
John Ericson
87ca46263d Merge branch 'master' into path-info 2023-02-02 09:55:07 -05:00
Eelco Dolstra
b574c70ccb
Merge pull request #7736 from shlevy/plugin-files-daemon
Don't send plugin-files to the daemon.
2023-02-02 12:35:51 +01:00
Shea Levy
895c525d04
daemon: Warn on old clients passing unexpected plugin-files.
The setting itself was already ignored due to exception trying to set pluginFiles.
2023-02-02 06:03:45 -05:00
Shea Levy
92edc38369
Don't send plugin-files to the daemon.
This is radically unsafe and the daemon has already loaded its plugins
anyway.

Fixes cachix/devenv#276
2023-02-01 20:05:56 -05:00
Jamie Quigley
32ca59649b
nix-shell: Colour the prompt red if the user is root
This matches the nixos prompt colours - green for standard user, red for
root
2023-02-01 20:50:44 +00:00
Eelco Dolstra
e8ca49f6ef Fix clang compilation 2023-02-01 20:34:44 +01:00
Eelco Dolstra
57a4258426 Remove an unused capture 2023-02-01 20:27:35 +01:00
John Ericson
ee9eb83a84 Remove some designated initializers
With the switch to C++20, the rules became more strict, and we can no
longer initialize base classes. Make them comments instead.

(BTW
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2287r1.html
this offers some new syntax for this use-case. Hopefully this will be
adopted and we can eventually use it.)
2023-02-01 11:25:56 -05:00
Eelco Dolstra
7a09bfbcb6
Merge pull request #7723 from yorickvP/nix-store-ping-json
nix store ping: add --json flag
2023-02-01 17:11:34 +01:00
John Ericson
59d3175649 Put back TODO
I don't think the `narHash` is in need of documentation more than the
other undocumented fields, but regardless this change has nothing to do
with that field and so we should leave the comment as is.
2023-02-01 10:09:25 -05:00
John Ericson
db759b1bc2 Undo style change
`&` without space before is far more common on this codebase than I
thought, so it is not worth changing just this one file. Maybe we will
adopt a formatter someday but until then this is fine.
2023-02-01 10:07:54 -05:00
John Ericson
0983a0bd30 Shrink diff in one place 2023-02-01 10:04:28 -05:00
John Ericson
43414738a0 Merge remote-tracking branch 'upstream/master' into path-info 2023-02-01 10:02:30 -05:00
Théophane Hufschmitt
518da6c6a3
Merge pull request #7716 from obsidiansystems/small-storePath-cleanups
Separate `path.hh` from `content-address.hh`
2023-02-01 16:00:28 +01:00
Eelco Dolstra
b55a946d8d
Merge pull request #7717 from obsidiansystems/delete-dead-code
Delete dead code
2023-02-01 15:57:04 +01:00
Théophane Hufschmitt
e32c5c2c77
Merge pull request #7667 from dramforever/flake-search-attr
Better error message for nix search when attr is not found
2023-02-01 15:56:22 +01:00
Eelco Dolstra
845b7f067d
Merge pull request #7726 from hercules-ci/flake-show-when-empty-hide
nix flake show: Ignore empty attrsets
2023-02-01 15:49:13 +01:00
Eelco Dolstra
14b0b9ea5a
Merge pull request #7203 from graham33/feature/cpp20
Proposal: Use C++20
2023-02-01 15:41:04 +01:00
Robert Hensing
60d48eda23 nix flake show: Ignore empty attrsets
For frameworks it's important that structures are as lazy as possible
to prevent infinite recursions, performance issues and errors that
aren't related to the thing to evaluate. As a consequence, they have
to emit more attributes than strictly (sic) necessary.
However, these attributes with empty values are not useful to the user
so we omit them.
2023-01-31 18:20:26 +01:00
Yorick van Pelt
4f4a6074e4
nix store ping: try to print json if connect() fails aswell 2023-01-31 15:10:39 +01:00
Yorick van Pelt
28648ed784
nix store ping: add --json flag 2023-01-31 13:24:23 +01:00
John Ericson
b6c98752f6 Merge remote-tracking branch 'upstream/master' into path-info 2023-01-30 18:04:54 -05:00
Robert Hensing
c9b9260f34
Merge pull request #7713 from obsidiansystems/more-rapid-check
Add more property tests
2023-01-30 18:54:53 +01:00
John Ericson
e21aa43212 Delete dead code
The references set seems to have been unused since `LegacySSHStore`
references were first created in
caa5793b4a.

The method decls never were upstream, and accidentally added by me in
062533f7cd (probably due to `git rerere`).
Sorry!

This reduces the diff from #3746.
2023-01-30 11:29:01 -05:00
dramforever
6b779e4b07 Fix extra "." in CmdSearch::getDefaultFlakeAttrPaths
No other getDefaultFlakeAttrPaths implementation has this trailing dot,
and the dot can show up in error messages like:

  error: flake '...' does not provide attribute 'packages.x86_64-linux.', ...
2023-01-31 00:04:05 +08:00
dramforever
377d5eb388 Installable::getCursors: Cleanup
- Clarify doc comments, Installables::getCursors returns non-empty
  vector
- Use vector::at in Installable::getCursor instead of checking for empty
  vector and throwing an exception with error message.
2023-01-31 00:04:05 +08:00
dramforever
b26562c629 InstallableFlake: Handle missing attr in getCursors
Handle the case where none of getActualAttrPaths() actually exists,
in which case instead of returning an empty vector.

This fixes the case where the user misspells the attribute name in nix
search. Instead of getting no search results, now it shows an error with
suggestions.

Also remove InstallableFlake::getCursor() override since it's now
equivalent to the base class version.
2023-01-30 23:59:09 +08:00
John Ericson
560142fec0 Make per-variant Arbitrary impls too
This is a nice idea that @roberth requested. If we could factor our a
generic `std::variant` impl as a follow-up it would be even better!
2023-01-30 10:56:00 -05:00
John Ericson
02e745ba5b Separate path.hh from content-address.hh
It is good to separate concerns; `StorePath` (in general) has nothing to
do with `ContentAddress` anyways.

This reduces the diff from #3746.
2023-01-30 10:14:03 -05:00
John Ericson
974a983351 Shrink diff in two places
Stuff crept in there.
2023-01-30 09:59:55 -05:00
John Ericson
adb3608034 Merge branch 'small-storePath-cleanups' into path-info 2023-01-30 09:46:43 -05:00
John Ericson
f3e272ba02 Avoid some StorePath <-> Path round trips
Avoid needless work and throwing away invariants.

These conversions date back to when `StorePath` was in Rust and there
were issues with it missing utility methods.
2023-01-30 09:37:57 -05:00
Eelco Dolstra
c79b1582a7
Merge pull request #5226 from NixOS/client-side-profiles
Move the default profiles to the user’s home
2023-01-30 12:21:47 +01:00
Théophane Hufschmitt
575d0aea5d
Merge pull request #6988 from max-privatevoid/pr-flake-show-foreign
nix flake show: don't evaluate derivations for foreign systems by default
2023-01-30 12:06:37 +01:00
Théophane Hufschmitt
de1b593233
Merge pull request #7087 from ncfavier/referenceablePaths
Self-contained outputs
2023-01-30 11:06:54 +01:00
Théophane Hufschmitt
7cd08ae379
Merge pull request #7645 from typetetris/fix-url-parsing-file-as-application-scheme
Fix url parsing for urls using `file+`
2023-01-30 10:42:03 +01:00
Théophane Hufschmitt
4aaf0ee52e
Merge branch 'master' into referenceablePaths 2023-01-30 10:31:00 +01:00
Théophane Hufschmitt
d70b890488
Merge pull request #7689 from ncfavier/nix-path-restrict-eval
getDefaultNixPath: actually respect `{restrict,pure}-eval`
2023-01-30 10:03:17 +01:00
John Ericson
ecd3e4ebd7 More property tests
Also put proper comparison methods on `DerivedPath` and
`NixStringContextElem`, which is needed for the tests but good in
general.
2023-01-29 17:09:59 -05:00
John Ericson
ec0c0efec6 Allow unit test infra to be reused across libs' tests
This allows using Arbitrary "instances" defined in libstore-tests in
libexpr-tests, something we will leverage in a moment.
2023-01-29 13:52:57 -05:00
Max
02e81cdf62 apply showAllSystems to legacyPackages as well 2023-01-27 23:59:48 +01:00
Naïm Favier
dba9173a1d
Document default nix-path value 2023-01-27 15:25:07 +01:00
Naïm Favier
1cba5984a6
getDefaultNixPath: actually respect {restrict,pure}-eval
Previously, getDefaultNixPath was called too early: at initialisation
time, before CLI and config have been processed, when `restrictEval` and
`pureEval` both have their default value `false`. Call it when
initialising the EvalState instead, and use `setDefault`.
2023-01-27 13:28:57 +01:00
Théophane Hufschmitt
ab424a39a9 Merge remote-tracking branch 'nixos/master' into pr-flake-show-foreign 2023-01-27 09:46:46 +01:00
Solène Rapenne
6b2729c81e improve documentation about substituters and trusted users
Co-authored-by: Théophane Hufschmitt <theophane.hufschmitt@tweag.io>
2023-01-26 09:56:44 +01:00
Solène Rapenne
64951d9125 Update src/libstore/daemon.cc
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-01-26 09:34:25 +01:00
Solène Rapenne
a96156c58f warnings: enhance the case of untrusted substituter for untrusted user 2023-01-26 09:34:25 +01:00
Guillaume Maudoux
734c5fdcd6 Fix 'destructor called on non-final ...' warning
clangStdenv compiles with a single warning:

```
warning: destructor called on non-final 'nix::PosAdapter' that has virtual functions but non-virtual destructor [-Wdelete-non-abstract-non-virtual-dtor]
```

This fixes the warning by making the destructor of PosAdapter virtual,
deffering to the correct destructor from the concrete child classes.
This has no impact in the end, as none of these classes have specific
destructors.

Technicaly, it may be faster not to have this indirection, but as per
the warning, there is only one place where we have to delete abstract
PosAdapter values.

Not worth bikesheding I guess.
2023-01-24 16:37:50 +01:00
John Ericson
e68e8e3cee Merge branch 'path-info' into ca-drv-exotic 2023-01-23 16:54:45 -05:00
John Ericson
4540e7b940 Don't add StorePathDescriptor for now
We don't need it yet, we can add it back later.
2023-01-23 12:58:27 -05:00
Eelco Dolstra
f503ba1b8b
Merge pull request #7595 from cole-h/show-setting-value
nix/show-config: allow getting the value of a specific setting
2023-01-23 17:56:39 +01:00
John Ericson
c67e0cc58c Merge remote-tracking branch 'upstream/master' into path-info 2023-01-23 11:47:20 -05:00
Robert Hensing
9b56683398
Merge pull request #7447 from aakropotkin/read-file-type
Read file type
2023-01-23 17:37:22 +01:00
Robert Hensing
0a9acefeb5
Merge pull request #7657 from obsidiansystems/fix-7655
Fix #7655
2023-01-23 15:42:59 +01:00
John Ericson
018e2571aa Test store paths, with property tests
The property test in fact found a bug: we were excluding numbers!
2023-01-23 07:05:50 -05:00
John Ericson
685395332d Better-scope Store forward declarations 2023-01-23 07:05:50 -05:00
John Ericson
7fe308c2f8 Add rapidcheck dependency for testing
Property tests are great!

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2023-01-23 07:05:50 -05:00
Alex Ameen
153ee460c5 primop: add readFileType, optimize readDir
Allows checking directory entry type of a single file/directory.

This was added to optimize the use of `builtins.readDir` on some
filesystems and operating systems which cannot detect this information
using POSIX's `readdir`.

Previously `builtins.readDir` would eagerly use system calls to lookup
these filetypes using other interfaces; this change makes these
operations lazy in the attribute values for each file with application
of `builtins.readFileType`.
2023-01-22 13:45:02 -06:00
John Ericson
0afdf4084c Fix #7655
We had some local variables left over from the older (more
complicated) implementation of this function. They should all be unused,
but one wasn't by mistake.

Delete them all, and replace the one that was still in use as intended.
2023-01-21 23:55:06 -05:00
Guillaume Maudoux
a0642305ab
Use complete '__toString' attribute name
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-01-20 13:06:00 +01:00
Guillaume Maudoux
ee4b849b17
Fix unreachable error message
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-01-20 13:01:03 +01:00
Eric Wolf
4d50995eff Fix url parsing for urls using file+
`file+https://example.org/test.mp4` should not be rejected with
`unexpected authority`.
2023-01-20 10:31:26 +01:00
Guillaume Maudoux
a9fa2c758b Always display addErrorContext messages in (expanded) traces 2023-01-19 14:14:19 +01:00
Guillaume Maudoux
6228b6b950 Discuss re-entrant errors and design 2023-01-19 14:12:26 +01:00
Guillaume Maudoux
ca7c5e08c1 Add tests for error traces, and fixes 2023-01-19 14:10:56 +01:00
Guillaume Maudoux
e4726a0c79 Revert "Revert "Merge pull request #6204 from layus/coerce-string""
This reverts commit 9b33ef3879.
2023-01-19 13:23:04 +01:00
Eelco Dolstra
0510aa40a3
Merge pull request #7631 from edolstra/output-names
OutputSpec: Allow all valid output names
2023-01-18 17:09:15 +01:00
Eelco Dolstra
75c89c3e5e Add test for OutputsSpec::Names
From @Ericson2314.
2023-01-18 16:34:37 +01:00
Eelco Dolstra
95cfd50d25 OutputSpec: Allow all valid output names
Fixes #7624.
2023-01-18 14:14:29 +01:00
Eelco Dolstra
01f268322a Restore support for channel: URLs in fetchTarball
Fixes #7625.
2023-01-18 12:57:03 +01:00
Robert Hensing
9b33ef3879 Revert "Merge pull request #6204 from layus/coerce-string"
This reverts commit a75b7ba30f, reversing
changes made to 9af16c5f74.
2023-01-18 01:34:07 +01:00
John Ericson
3965b0f75f Try again to fix aarch64-linux build failure
f419ab48e6 was on the right track, but
there are a few more missing `raw()` calls to fix.
2023-01-17 09:14:17 -05:00
Taeer Bar-Yam
b2752a4f74 add comments 2023-01-17 08:28:56 -05:00
Théophane Hufschmitt
6bdf4edb77 Keep the default profile the same
It's used as the “system” profile in a bunch of places, so better not
touch it. Besides, it doesn't hurt to keep it since it's owned by root
any way, so it doesn't have the `chown` problem that the user profiles
had and that led to wanting to move them on the client-side.
2023-01-17 14:17:28 +01:00
Théophane Hufschmitt
c80621dbac Don't try to migrate existing profiles
Doing so would be more dangerous than useful, better leave them as-is if
they already exist
2023-01-17 14:17:28 +01:00
Théophane Hufschmitt
0601050755 Migrate the old profiles to the new location
Make sure that we don’t just create the new profiles directory, but that
we also migrate every existing profile to it.
2023-01-17 14:17:28 +01:00
Théophane Hufschmitt
a5919f4754 Move the default profiles to the user’s home
Rather than using `/nix/var/nix/{profiles,gcroots}/per-user/`, put the user
profiles and gcroots under `$XDG_DATA_DIR/nix/{profiles,gcroots}`.

This means that the daemon no longer needs to manage these paths itself
(they are fully handled client-side). In particular, it doesn’t have to
`chown` them anymore (removing one need for root).

This does change the layout of the gc-roots created by nix-env, and is
likely to break some stuff, so I’m not sure how to properly handle that.
2023-01-17 14:17:28 +01:00
Eelco Dolstra
9a32f77a95
Merge pull request #7606 from hercules-ci/recognize-some-flake-attributes
flake check: Recognize well known community attributes
2023-01-17 13:27:30 +01:00
Robert Hensing
4e7592b593 flake check: Recognize well known community attributes
This avoids warning fatigue, making `nix flake check` more effective.
2023-01-16 20:16:45 +01:00
Eelco Dolstra
1df3d62c76
Merge pull request #7585 from NixOS/macos-disconnect
MonitorFdHup: Make it work on macOS again
2023-01-16 13:30:15 +01:00
John Ericson
f419ab48e6 Try to fix build failure
Failure: https://hydra.nixos.org/build/205357257/nixlog/1

The problem seems to be trying to `std::visit` a derived class of
`std::variant`. Per
https://stackoverflow.com/questions/63616709/incomplete-type-stdvariant-used-in-nested-name-specifier
certain C++ standard library implementations allow this, but others do
not.

The solution is simply to call the `raw` method, which upcasts the
reference back to the `std::variant`.
2023-01-15 15:16:14 -05:00
John Ericson
7c82213813 Merge branch 'path-info' into ca-drv-exotic 2023-01-14 17:09:58 -05:00
John Ericson
b3d91239ae Make ValidPathInfo have plain StorePathSet references like before
This change can wait for another PR.
2023-01-14 16:42:03 -05:00
John Ericson
056cc1c1b9 Merge remote-tracking branch 'upstream/master' into path-info 2023-01-14 14:27:28 -05:00
John Ericson
a416476217 Move ValidPathInfo defintions to path-info.cc
Originally there was no `path-info.*`, then there was `path-info.hh`,
then there was `path-info.cc`, but only for new things. Moving this
stuff over makes everything consistent.
2023-01-13 15:39:19 -05:00
John Ericson
2e7be46e73 Move new ValidPathInfo methods to path-info.cc
We'll move the old ones separately, so as not to clutter the diff.
2023-01-13 15:06:07 -05:00
Robert Hensing
fec527bba1
Merge pull request #7597 from tweag/move-implem-bit-to-implem-file
Move the `getBuildLog` implementation to its own implementation file
2023-01-13 20:16:33 +01:00
Robert Hensing
d21f54958e
Merge pull request #6815 from obsidiansystems/better-wanted-outputs
`OutputSpec` for `DerivationGoal` and `DerivedPath`, today's `OutputSpec` -> `ExtendedOutputSpec`
2023-01-13 16:03:12 +01:00
Théophane Hufschmitt
b8a0e9a9b8 Move the getBuildLog implementation to its own implementation file
Keep the header minimal and clean
2023-01-13 11:05:44 +01:00
Théophane Hufschmitt
bdeb6de889
Merge pull request #7430 from tweag/ca/fix-nix-log
Ca/fix nix log
2023-01-13 11:00:56 +01:00
John Ericson
d8512653d4 Write more (extended) output spec tests 2023-01-12 22:05:55 -05:00
John Ericson
d29eb08563 Assert on construction that OutputsSpec::Names is non-empty 2023-01-12 20:52:29 -05:00
John Ericson
e947aa5401 Unit test OuputsSpec::{union_, isSubsetOf} 2023-01-12 20:33:50 -05:00
John Ericson
31875bcfb7 Split OutputsSpec::merge into OuputsSpec::{union_, isSubsetOf}
Additionally get rid of the evil time we made an empty
`OutputSpec::Names()`.
2023-01-12 20:20:27 -05:00
Cole Helbling
1fc74afbba nix/show-config: allow getting the value of a specific setting
Instead of needing to run `nix show-config --json | jq -r
'."warn-dirty".value'` to view the value of `warn-dirty`, you can now
run `nix show-config warn-dirty`.
2023-01-12 13:56:35 -08:00
Théophane Hufschmitt
eaa20f2574
Merge pull request #7590 from fricklerhandwerk/remove-unnecessary-cast
remove unncessary cast
2023-01-12 14:00:43 +01:00
Valentin Gagarin
48b2a3a0d0 remove unncessary cast 2023-01-12 13:23:32 +01:00
John Ericson
0faf5326bd Improve tests for OutputsSpec 2023-01-11 19:09:21 -05:00
John Ericson
5ba6e5d0d9 Remove default constructor from OutputsSpec
This forces us to be explicit.

It also requires to rework how `from_json` works. A `JSON_IMPL` is added
to assist with this.
2023-01-11 19:08:19 -05:00
John Ericson
114a6e2b09 Make it hard to construct an empty OutputsSpec::Names
This should be a non-empty set, and so we don't want people doing this
by accident. We remove the zero-0 constructor with a little inheritance
trickery.
2023-01-11 19:08:19 -05:00
John Ericson
8a3b1b7ced Simplify and document store path installable parsing 2023-01-11 19:08:19 -05:00
John Ericson
ce2f91d356 Split OutputsSpec and ExtendedOutputsSpec, use the former more
`DerivedPath::Built` and `DerivationGoal` were previously using a
regular set with the convention that the empty set means all outputs.
But it is easy to forget about this rule when processing those sets.
Using `OutputSpec` forces us to get it right.
2023-01-11 18:57:18 -05:00
John Ericson
a7c0cff07f Rename OutputPath -> ExtendedOutputPath
Do this prior to making a new more limitted `OutputPath` we will use in
more places.
2023-01-11 18:55:29 -05:00
John Ericson
a8f45b5e5a Improve OutputsSpec slightly
A few little changes preparing for the rest.
2023-01-11 18:54:50 -05:00
Eelco Dolstra
9fc8d00d74 MonitorFdHup: Make it work on macOS again
It appears that on current macOS versions, our use of poll() to detect
client disconnects no longer works. As a workaround, poll() for
POLLRDNORM, since this *will* wake up when the client has
disconnected. The downside is that it also wakes up when input is
available. So just sleep for a bit in that case.  This means that on
macOS, a client disconnect may take up to a second to be detected,
but that's better than not being detected at all.

Fixes #7584.
2023-01-11 10:48:40 -08:00
Eelco Dolstra
7515617ad0 Backport getLine tests from lazy-trees 2023-01-11 13:49:39 +01:00
Théophane Hufschmitt
a3ba80357d
Merge pull request #7543 from obsidiansystems/typed-string-context
Parse string context elements properly
2023-01-11 07:09:37 +01:00
Robert Hensing
f58c301112
Merge pull request #7541 from hercules-ci/check-manual-links
Check links in the manual
2023-01-10 23:07:38 +01:00
Robert Hensing
fefa3a49ce doc/manual: Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-01-10 22:30:41 +01:00
Robert Hensing
e79f935718 doc/manual: Fix broken internal links
The targets I could find.
2023-01-10 22:30:41 +01:00
Robert Hensing
34a1e0d29b doc/manual: Introduce @docroot@ as a stable base for includable snippets
This way the links are clearly within the manual (ie not absolute paths),
while allowing snippets to reference the documentation root reliably,
regardless of at which base url they're included.
2023-01-10 22:30:41 +01:00
John Ericson
5576d5e987 Parse string context elements properly
Prior to this change, we had a bunch of ad-hoc string manipulation code
scattered around. This made it hard to figure out what data model for
string contexts is.

Now, we still store string contexts most of the time as encoded strings
--- I was wary of the performance implications of changing that --- but
whenever we parse them we do so only through the
`NixStringContextElem::parse` method, which handles all cases. This
creates a data type that is very similar to `DerivedPath` but:

 - Represents the funky `=<drvpath>` case as properly distinct from the
   others.

 - Only encodes a single output, no wildcards and no set, for the
   "built" case.

(I would like to deprecate `=<path>`, after which we are in spitting
distance of `DerivedPath` and could maybe get away with fewer types, but
that is another topic for another day.)
2023-01-10 13:10:49 -05:00
John Ericson
da64f026dd Make clear that StorePathWithOutputs is a deprecated type
- Add a comment

- Put `OutputsSpec` in a different header (First part of #6815)

- Make a few stray uses of it in new code use `DerivedPath` instead.
2023-01-10 11:27:19 -05:00
Eelco Dolstra
59cc920cc0 Add a FIXME 2023-01-10 15:20:30 +01:00
Eelco Dolstra
7f1af270dd Clean up toDerivedPaths() logic 2023-01-10 15:08:46 +01:00
Eelco Dolstra
1123c42f90
Apply suggestions from code review
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-01-10 14:57:35 +01:00
Eelco Dolstra
b4dc68f0be Show string in error message 2023-01-10 14:56:03 +01:00
Eelco Dolstra
b80e4b57da ExtraInfo -> ExtraPathInfo 2023-01-10 14:52:49 +01:00
Eelco Dolstra
8e923bf4c5 Merge remote-tracking branch 'origin/master' into fix-7417 2023-01-10 14:35:06 +01:00
Will Bush
05b13aff3d
Fix typo in example for builtin function map 2023-01-06 23:04:43 -06:00
John Ericson
81727f85cb Merge branch 'path-info' into ca-drv-exotic 2023-01-06 15:45:34 -05:00
John Ericson
46e942ff9e Do big rename to clean up code
- `PathReferences` -> `References`

- `PathReferences<StorePath>` -> `StoreReference`

- `references` -> `others`

- `hasSelfReference` -> `self`

And get rid of silly subclassing
2023-01-06 15:36:05 -05:00
John Ericson
7e1cfa97c6 Make derivation primop code for fixed output more concise 2023-01-06 12:52:16 -05:00
John Ericson
848b0832b5 Merge branch 'path-info' into ca-drv-exotic 2023-01-06 12:39:14 -05:00
John Ericson
9cfa78e58a Optimize ValidPathInfo construction a bit better 2023-01-06 12:26:15 -05:00
John Ericson
6a168254ce Use named field initialization for references 2023-01-06 12:24:20 -05:00
John Ericson
8623143921 Make formatting consistent 2023-01-06 11:18:14 -05:00
John Ericson
989b8065b4 Merge branch 'path-info' into ca-drv-exotic 2023-01-06 10:56:22 -05:00
John Ericson
e9fc1e4fdb Merge remote-tracking branch 'upstream/master' into path-info 2023-01-06 10:35:20 -05:00
Théophane Hufschmitt
8d88c3b347
Merge pull request #7307 from hercules-ci/derivation-goal-improve-comment
libstore/derivation-goal: Elaborate a TODO for performance concern
2023-01-06 13:07:57 +01:00
Eelco Dolstra
420ccecc1e
Merge pull request #7557 from NixOS/fix-7529
On macOS with auto-uid-allocation and sandboxing, use the correct gid
2023-01-06 12:35:55 +01:00
Naïm Favier
f1ee4ece80
Don't check NixOS modules
NixOS modules can be paths. Rather than dig further down into the layer
violation, don't check anything specific to NixOS modules.
2023-01-05 18:23:30 +01:00
Eelco Dolstra
3a98107170
Merge pull request #7542 from edolstra/gc-deadlock
Fix deadlock between auto-GC and addTempRoot()
2023-01-05 17:08:23 +01:00
Eelco Dolstra
0fe2b222d5
Merge pull request #7539 from tweag/fix-nix-why-depends--derivation
Fix `nix why-depends --derivation`
2023-01-05 15:32:04 +01:00
Eelco Dolstra
4e84b532ed On macOS with auto-uid-allocation and sandboxing, use the correct gid
macOS doesn't have user namespacing, so the gid of the builder needs
to be nixbld. The logic got "has sandboxing enabled" confused with
"has user namespaces".

Fixes #7529.
2023-01-05 04:58:55 -08:00
Eelco Dolstra
6991e558dd Move macOS sandbox files to sr/libstore/build 2023-01-04 04:50:45 -08:00
Eelco Dolstra
609a7dc059 Include macOS sandbox files in the Nix binary
This basically reverts 6e5165b773.
It fixes errors like

  sandbox-exec: <internal init prelude>:292:47: unable to open sandbox-minimal.sb: not found

when trying to run a development Nix installed in a user's home
directory.

Also, we're trying to minimize the number of installed files
to make it possible to deploy Nix as a single statically-linked
binary.
2023-01-04 04:36:07 -08:00
Naïm Favier
3c968191f1
move unsafeDiscardReferences out of outputChecks
It's not a check.
2023-01-03 18:53:01 +01:00
Eelco Dolstra
d4d1ca8b11 nix --version: Print the data directory 2023-01-03 08:30:49 -08:00
Naïm Favier
15f7fa59be
unsafeDiscardReferences
Adds a new boolean structured attribute
`outputChecks.<output>.unsafeDiscardReferences` which disables scanning
an output for runtime references.

    __structuredAttrs = true;
    outputChecks.out.unsafeDiscardReferences = true;

This is useful when creating filesystem images containing their own embedded Nix
store: they are self-contained blobs of data with no runtime dependencies.

Setting this attribute requires the experimental feature
`discard-references` to be enabled.
2023-01-03 17:19:16 +01:00
Eelco Dolstra
28d5b5cd45 Fix deadlock between auto-GC and addTempRoot()
Previously addTempRoot() acquired the LocalStore state lock and waited
for the garbage collector to reply. If the garbage collector is in the
same process (as it the case with auto-GC), this would deadlock as
soon as the garbage collector thread needs the LocalStore state lock.

So now addTempRoot() uses separate Syncs for the state that it
needs. As long at the auto-GC thread doesn't call addTempRoot() (which
it shouldn't), it shouldn't deadlock.

Fixes #3224.
2023-01-03 15:20:21 +01:00
Eelco Dolstra
224b56f10e Move creation of the temp roots file into its own function
This also moves the file handle into its own Sync object so we're not
holding the _state while acquiring the file lock. There was no real
deadlock risk here since locking a newly created file cannot block,
but it's still a bit nicer.
2023-01-03 14:51:23 +01:00
Eelco Dolstra
ae31b5f50f
Merge pull request #7497 from rski/master
src/libstore: Print the reason opening the DB failed
2023-01-03 12:44:14 +01:00
Eelco Dolstra
d33d15a48b Put the --show-trace hint in the logical place 2023-01-02 20:53:58 +01:00
Eelco Dolstra
c548e35498 Don't use state.positions[noPos]
This caused traces 'at «none»:0: (source not available)'.
2023-01-02 20:53:58 +01:00
Eelco Dolstra
6b69652385 Merge remote-tracking branch 'origin/master' into coerce-string 2023-01-02 20:53:39 +01:00
Théophane Hufschmitt
8cac451fce Fix why-depends for CA derivations (again)
This has the same goal as b13fd4c58e81b2b2b0d72caa5ce80de861622610,but
achieves it in a different way in order to not break
`nix why-depends --derivation`.
2023-01-02 17:42:22 +01:00
Théophane Hufschmitt
105d74eb81 Revert "Fix why-depends for CA derivations"
This reverts commit b13fd4c58e.
2023-01-02 15:44:04 +01:00
Théophane Hufschmitt
9af16c5f74
Merge pull request #5941 from hercules-ci/optimize-intersectAttrs
Optimize intersectAttrs performance
2023-01-02 15:22:38 +01:00
Valentin Gagarin
e0c4a95611 antiquotation -> string interpolation
as proposed by @mkaito[1] and @tazjin[2] and discussed with @edolstra
and Nix maintainers

[1]: https://github.com/NixOS/nix.dev/pull/267#issuecomment-1270076332
[2]: https://github.com/NixOS/nix.dev/pull/267#issuecomment-1270201979

Co-authored-by: John Ericson <git@JohnEricson.me>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-01-02 14:38:57 +01:00
Théophane Hufschmitt
fb8fc6fda6
Merge pull request #7478 from hercules-ci/make-sure-initNix-called
libstore: Make sure that initNix has been called
2023-01-02 14:12:49 +01:00
Valentin Gagarin
9cdf8ededb remove redundant re-definition of store derivations 2023-01-02 13:37:59 +01:00
Théophane Hufschmitt
cfd6c7fc9b
Merge pull request #7485 from fricklerhandwerk/doc-store-derivation
define "store derivation"
2023-01-02 13:26:41 +01:00
Théophane Hufschmitt
b3285c7722
Merge pull request #7351 from NaN-git/fix-mkString
cleanup eval.hh/eval.cc
2023-01-02 11:41:52 +01:00
Théophane Hufschmitt
e8a3e58171
Merge pull request #7521 from ncfavier/migration-deadlock
Release shared lock before acquiring exclusive lock
2023-01-02 11:08:43 +01:00
Steven Shaw
84b0893725
Fix error message 2023-01-01 12:37:43 +10:00
Eelco Dolstra
8c52f8ea9d
Merge pull request #7524 from ncfavier/sandbox-paths-closure
doc: sandbox-paths computes closures
2022-12-29 19:45:51 +01:00
Eelco Dolstra
8227fe819e
Merge pull request #7504 from edolstra/nix-develop-personality
nix develop: Set personality
2022-12-29 19:42:45 +01:00
Naïm Favier
d5d2f50ebb
doc: sandbox-paths computes closures 2022-12-28 17:09:20 +01:00
Naïm Favier
81c3f99b36
Release shared lock before acquiring exclusive lock
In principle, this should avoid deadlocks where two instances of Nix are
holding a shared lock on big-lock and are both waiting to get an
exclusive lock.

However, it seems like `flock(2)` is supposed to do this automatically,
so it's not clear whether this is actually where the problem comes from.
2022-12-27 15:58:14 +01:00
Robert Hensing
336908cf4c Optimize intersectAttrs performance
Always traverse the shortest set.
2022-12-24 14:51:05 +01:00
Robert Hensing
aba6eb348e libstore: Make sure that initNix has been called
Prevent bugs like https://github.com/cachix/cachix/pull/477
2022-12-24 14:39:30 +01:00
Philipp Otterbein
8af839f48c remove undefined function 2022-12-24 12:19:53 +01:00
Philipp Otterbein
a6e9d9cb2f remove function makeImmutableStringWithLen 2022-12-24 12:09:06 +01:00
Eelco Dolstra
c164d304f3 nix develop: Set personality
This makes 'nix develop' set the Linux personality in the same way
that the actual build does, allowing a command like 'nix develop
nix#devShells.i686-linux.default' on x86_64-linux to work correctly.
2022-12-23 16:33:55 +01:00
Eelco Dolstra
14f7dae3e4
Merge pull request #7503 from edolstra/fix-dirOf
Fix CanonPath::dirOf() returning a string_view of a temporary
2022-12-23 16:17:48 +01:00
Eelco Dolstra
64c60f7241 Fix CanonPath::dirOf() returning a string_view of a temporary
https://hydra.nixos.org/build/202837872
2022-12-23 15:32:54 +01:00
rski
d034ed1891 src/libstore: Print the reason opening the DB failed
Without this, the error is lost, and it makes for a hard to debug
situation. Also remove some of the busyness inside the sqlite_open_v2
args.

The errcode returned is not the extended one. The only way to make open
return an extended code, would be to add SQLITE_OPEN_EXRESCODE to the
flags. In the future it might be worth making this change,
which would also simplify the existing SQLiteError code.
2022-12-23 02:55:51 +02:00
Eelco Dolstra
c9eee5a84d
Merge pull request #7493 from mupdt/primops-storeDir-test-non-standard-path
primops `storeDir` test uses `settings.nixStore`
2022-12-22 16:18:48 +01:00
mupdt
a33e45b60b primops storeDir test uses settings.nixStore 2022-12-21 07:01:57 -05:00
Valentin Gagarin
7797661a70 link "store derivation" to glossary definition 2022-12-21 11:42:50 +01:00
Valentin Gagarin
3a66d82e1d update description of "store derivation" in installables section
a store derivation is not a store path itself, it has a store path.
2022-12-21 11:42:22 +01:00
mupdt
bc8ab21c5a [PDT] TDE-3114: prevent a race-condition when creating the S3 cache 2022-12-21 04:50:40 -05:00
Eelco Dolstra
5c97b5a398 InstallableFlake::toDerivedPaths(): Support paths and store paths
This makes 'nix build' work on paths (which will be copied to the
store) and store paths (returned as is). E.g. the following flake
output attributes can be built using 'nix build .#foo':

  foo = ./src;
  foo = self.outPath;
  foo = builtins.fetchTarball { ... };
  foo = (builtins.fetchTree { .. }).outPath;
  foo = builtins.fetchTree { .. } + "/README.md";
  foo = builtins.storePath /nix/store/...;

Note that this is potentially risky, e.g.

  foo = /.;

will cause Nix to try to copy the entire file system to the store.

What doesn't work yet:

  foo = self;
  foo = builtins.fetchTree { .. };

because we don't handle attrsets with an outPath attribute in it yet,
and

  foo = builtins.storePath /nix/store/.../README.md;

since result symlinks have to point to a store path currently (rather
than a file inside a store path).

Fixes #7417.
2022-12-20 15:11:44 +01:00
Eelco Dolstra
bda879170f EvalState::copyPathToStore(): Return a StorePath 2022-12-20 14:58:39 +01:00
Eelco Dolstra
845fc3f605 Merge toDerivations() into toDerivedPaths()
toDerivedPaths() now returns DerivedPathWithInfo, which is DerivedPath
with some attributes needed by 'nix profile' etc.

Preparation for #7417.
2022-12-20 14:24:14 +01:00
Eelco Dolstra
8332ac6a1d
Merge pull request #7451 from edolstra/abstract-pos
Introduce AbstractPos
2022-12-20 12:55:13 +01:00
Théophane Hufschmitt
000dd77d8d
Merge pull request #7473 from hercules-ci/sqlite-error
Improve sqlite error messages
2022-12-20 11:27:30 +01:00
Théophane Hufschmitt
c3d522dc51
Merge pull request #7480 from nrdxp/fix-develop-drv
fix(develop): make `nix develop` drv recreatable
2022-12-20 07:41:46 +01:00
Timothy DeHerrera
94cf0da7b2
fix(develop): make nix develop drv recreatable 2022-12-19 13:16:06 -07:00
Domen Kožar
f118e661e0
Merge pull request #7462 from rapenne-s/improve_messaging_reusing_trusted_substituter
Inform user instead of warning them when using a trusted substituter
2022-12-19 16:04:01 +00:00
Eelco Dolstra
d00bfe4833
Merge pull request #7450 from edolstra/canon-path
Add CanonPath wrapper to represent canonicalized paths
2022-12-19 16:21:11 +01:00
Robert Hensing
c965f35de7 Improve sqlite error messages
They did not include the detailed error message, losing essential
information for troubleshooting.

Example message:

    warning: creating statement 'insert or rplace into NARs(cache, hashPart, namePart, url, compression, fileHash, fileSize, narHash, narSize, refs, deriver, sigs, ca, timestamp, present) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1)': at offset 10: SQL logic error, near "rplace": syntax error (in '/tmp/nix-shell.grQ6f7/nix-test/tests/binary-cache/test-home/.cache/nix/binary-cache-v6.sqlite')

It's not the best example; more important information will be in
the message for e.g. a constraint violation.

I don't see why this specific error is printed as a warning, but
that's for another commit.
2022-12-17 14:51:37 +01:00
Solène Rapenne
d0660c6c0b
printMsg replacement by printInfo
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-12-16 09:34:22 +01:00
Taeer Bar-Yam
e5eb05c599 getBuildLog: factor out resolving derivations 2022-12-15 15:58:54 -05:00
Eelco Dolstra
0687e16c4a Fix a crash in DerivedPath::Built::toJSON() with impure derivations
The use of 'nullptr' here didn't result in a null JSON value, but in a
nullptr being cast to a string, which aborts.
2022-12-15 16:02:27 +01:00
Solène Rapenne
98e01da0b1 warnings: switch to info level when using a saved substituter 2022-12-14 14:11:21 +01:00
Naïm Favier
1f3c0a3c1d
Allow disabling build users by unsetting build-users-group
Unsetting `build-users-group` (without `auto-allocate-uids` enabled)
gives the following error:

```
src/libstore/lock.cc:25: static std::unique_ptr<nix::UserLock> nix::SimpleUserLock::acquire(): Assertion `settings.buildUsersGroup != ""' failed.
```

Fix the logic in `useBuildUsers` and document the default value
for `build-users-group`.
2022-12-14 00:40:30 +01:00
Eelco Dolstra
46b3c026fc
Merge pull request #7455 from rapenne-s/documentation_fix
Add anchors and links to definition
2022-12-13 17:26:52 +01:00
Eelco Dolstra
e2a4e7aecd
Merge pull request #4543 from obsidiansystems/indexed-store-path-outputs
Low level `<drvPath>^<outputName>` installable syntax to match existing `<highLevelInstallable>^<outputNames>` syntax
2022-12-13 17:22:29 +01:00
Eelco Dolstra
c9b0a85b08 Restore display of source lines for stdin/string inputs 2022-12-13 16:00:44 +01:00
Eelco Dolstra
aea97f07a3 Fix compilation 2022-12-13 15:23:12 +01:00
Eelco Dolstra
1315133b50
Improve cast safety
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-12-13 12:38:33 +01:00
Solène Rapenne
09860c16ce documentation: use sections instead of list items 2022-12-13 11:47:44 +01:00
Solène Rapenne
e43b0f5b12 documentation: link flake URL term to definition 2022-12-13 11:47:38 +01:00
Solène Rapenne
ae27181f16 documentation: fix link to definition 2022-12-13 11:47:22 +01:00
Théophane Hufschmitt
6e31d27cba
Merge pull request #6741 from Mindavi/nix-no-gc
support building with --enable-gc=no
2022-12-13 10:36:59 +01:00
Eelco Dolstra
b3fdab28a2 Introduce AbstractPos
This makes the position object used in exceptions abstract, with a
method getSource() to get the source code of the file in which the
error originated. This is needed for lazy trees because source files
don't necessarily exist in the filesystem, and we don't want to make
libutil depend on the InputAccessor type in libfetcher.
2022-12-13 00:50:43 +01:00
John Ericson
f61d575810 Merge branch 'indexed-store-path-outputs' of github.com:obsidiansystems/nix into indexed-store-path-outputs 2022-12-12 17:43:10 -05:00
John Ericson
5273cf4c97 Merge remote-tracking branch 'upstream/master' into indexed-store-path-outputs 2022-12-12 17:40:49 -05:00
John Ericson
32ae715db1
Fix typos in the docs
Thanks!

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2022-12-12 17:37:45 -05:00
John Ericson
dabb03b8d0 Merge remote-tracking branch 'upstream/master' into indexed-store-path-outputs 2022-12-12 17:36:02 -05:00
John Ericson
d8c1c24c78 Adjust docs 2022-12-12 17:32:24 -05:00
Rick van Schijndel
672ee88231 support building with --enable-gc=no
Some minor changes fixing the build without boehm.
Fixes NixOS#6250
2022-12-12 23:31:30 +01:00
John Ericson
dc075dcdd0
Apply suggestions from code review
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2022-12-12 16:26:10 -05:00
Eelco Dolstra
900b854084 Add CanonPath wrapper to represent canonicalized paths 2022-12-12 19:57:32 +01:00
Florian Friesdorf
8618c6cc75 Simplify loop, feedback from @tfc and @Ericson2314 2022-12-12 18:41:00 +00:00
John Ericson
7b122d43a4 Fix stack context notes to not rely on order
Make everything be in the form "while ..." (most things were already),
and in particular *don't* use other propositions that must go after or
before specific "while ..." clauses to make sense.
2022-12-12 18:41:00 +00:00
Florian Friesdorf
d269976be6 Show stack trace above error message
Save developers from scrolling by displaying the error message last,
below the stack trace.
2022-12-12 18:41:00 +00:00
Florian Friesdorf
173dcb0af9 Don't reverse stack trace when showing
When debugging nix expressions the outermost trace tends to be more useful
than the innermost. It is therefore printed last to save developers from
scrolling.
2022-12-12 18:41:00 +00:00
Eelco Dolstra
e408af82ab
Merge pull request #7436 from edolstra/enable-lang-tests
Enable some language tests that were accidentally disabled
2022-12-12 17:39:02 +01:00
Eelco Dolstra
17f81d3215 Fix unused variable warning 2022-12-12 16:41:46 +01:00
Benoit de Chezelles
a456630a5a Allow to disable global flake-registry with "" 2022-12-12 15:32:02 +01:00
Eelco Dolstra
7a85199f87 Add docs from the lazy-trees branch 2022-12-12 14:06:13 +01:00
Eelco Dolstra
fd0ed75118 Support flake references in the old CLI
Fixes #7026.
2022-12-12 14:05:52 +01:00
Eelco Dolstra
ae5f62a894 Move isUri() and resolveUri() out of filetransfer.cc
These are purely related to NIX_PATH / -I command line parsing, so put
them in libexpr.
2022-12-12 14:05:35 +01:00
Eelco Dolstra
7396844676
Merge pull request #7421 from edolstra/lazy-trees-trivial-changes
Trivial changes from the lazy-trees branch
2022-12-12 13:52:56 +01:00
John Ericson
1879c7c95e
Merge branch 'master' into indexed-store-path-outputs 2022-12-12 07:33:36 -05:00
Eelco Dolstra
e558e089ba -I description: Use -I examples 2022-12-12 12:51:23 +01:00
Eelco Dolstra
877ea1dab8
Use get_ptr()
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-12-12 12:46:13 +01:00
Eelco Dolstra
037d5c4299
Manual improvements
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2022-12-12 12:43:44 +01:00
Eelco Dolstra
786402365e Cleanup 2022-12-12 12:40:51 +01:00
Eelco Dolstra
f3d1e92856
Update URL
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2022-12-12 12:37:55 +01:00
Eelco Dolstra
8272cd9dec
Optimize string concatenation
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-12-12 12:36:19 +01:00
Andreas Rammhold
dbc8547664 Ignore the enforceDeterminism value
We used to set enforceDeterminism to true in the settings (by default)
and thus did send a non-zero value over the wire. The value should
probably be ignored as it should only matter if nrRounds is non-zero
as well.

Having the old code here where the value is expected to be zero only
works with the same version of Nix where we are sending zero. We
should always test this against older Nix versions being client or
server as otherwise upgrade in larger networks might be a pain.

Fixes 8e0946e8df
2022-12-10 17:55:07 +01:00
Théophane Hufschmitt
2affb19c92
Merge pull request #7409 from tweag/fix-6383
check the store for input before failing (hopefully fix #6383)
2022-12-09 06:33:30 +01:00
Taeer Bar-Yam
3b27181ee5 fix missing function after rebase 2022-12-08 16:59:21 -05:00
regnat
04b113f6cb Fix nix log with CA derivations
Fix #6209

When trying to run `nix log <installable>`, try first to resolve the derivation pointed to
by `<installable>` as it is the resolved one that holds the build log.

This has a couple of shortcomings:
1. It’s expensive as it requires re-reading the derivation
2. It’s brittle because if the derivation doesn’t exist anymore or can’t
   be resolved (which is the case if any one of its build inputs is missing),
   then we can’t access the log anymore

However, I don’t think we can do better (at least not right now).
The alternatives I see are:
1. Copy the build log for the un-resolved derivation. But that means a
   lot of duplication
2. Store the results of the resolving in the db. Which might be the best
   long-term solution, but leads to a whole new class of potential
   issues.
2022-12-08 16:03:20 -05:00
Eelco Dolstra
703d863a48 Trivial changes from the lazy-trees branch 2022-12-07 14:06:34 +01:00
amesgen
a5ab253e8f
Mention host parameter for GitHub 2022-12-07 12:38:18 +01:00
Linus Heckemann
8e0946e8df Remove repeat and enforce-determinism options
These only functioned if a very narrow combination of conditions held:

- The result path does not yet exist (--check did not result in
  repeated builds), AND
- The result path is not available from any configured substituters, AND
- No remote builders that can build the path are available.

If any of these do not hold, a derivation would be built 0 or 1 times
regardless of the repeat option. Thus, remove it to avoid confusion.
2022-12-07 11:36:48 +01:00
Taeer Bar-Yam
1c8de7d3d0 improve style 2022-12-06 11:25:38 -05:00
Eelco Dolstra
54906bc93c
Merge pull request #7382 from fricklerhandwerk/doc-automatic-uid
move documentation on `auto-allocate-uids` to options docs
2022-12-06 11:31:34 +01:00
amesgen
105677140d
Document gitlab input scheme 2022-12-06 10:36:39 +01:00
Eelco Dolstra
484578d3f9
Tweak option descriptions 2022-12-06 10:30:36 +01:00
Taeer Bar-Yam
8c7661da09 check the store for input before failing (hopefully fix #6383) 2022-12-05 23:22:38 -05:00
Théophane Hufschmitt
bfcf30f0ab
Merge pull request #7390 from Radvendii/fix-6383
check the store for input before failing (hopefully fix #6700)
2022-12-05 18:04:41 +01:00
Taeer Bar-Yam
e4f9f3bf24 check the store for input before failing (hopefully fix #6700) 2022-12-05 11:27:47 -05:00
Eelco Dolstra
e0ab2069c9
Consistent capitalisation 2022-12-05 16:55:55 +01:00
Jörg Thalheim
cccd57c022 getMaxCPU: fix cgroup path
Given this typo I am not sure if it has been tested.
2022-12-04 18:22:12 +01:00
Eelco Dolstra
c582150360
Merge pull request #7394 from edolstra/fix-7268
Lower verbosity of exceptions in getMaxCPU()
2022-12-02 17:02:32 +01:00
Eelco Dolstra
3a8341f57e
Merge pull request #7395 from obsidiansystems/evaluating-to-calling
Change "while evaluating <fun>" to "while *calling*" in trace
2022-12-02 15:53:57 +01:00
John Ericson
19c5394971 Change "while evaluating <fun>" to "while *calling*" in trace
The old way was not correct.

Here is an example:
```
 $ nix-instantiate --eval --expr 'let x = a: throw "asdf"; in x 1' --show-trace
error: asdf

       … while evaluating 'x'

       at «string»:1:9:

            1| let x = a: throw "asdf"; in x 1
             |         ^

       … from call site

       at «string»:1:29:

            1| let x = a: throw "asdf"; in x 1
             |                             ^
```
and yet also:
```
 $ nix-instantiate --eval --expr 'let x = a: throw "asdf"; in x' --show-trace
<LAMBDA>
```

Here is the thing: in both cases we are evaluating `x`!

Nix is a higher-order languages, and functions are a sort of value. When
we write `x = a: ...`, `a: ...` is the expression that `x` is being
defined to be, and that is already a value. Therefore, we should *never*
get an trace that says "while evaluating `x`", because evaluating `a:
...` is *trival* and nothing happens during it!

What is actually happening here is we are applying `x` and evaluating
its *body* with arguments substituted for parameters. I think the
simplest way to say is just "while *calling* `x`", and so that is what I
changed it to.
2022-12-02 09:14:12 -05:00
Eelco Dolstra
fa99ef6a87 getMaxCPU(): Lower verbosity level for ignored exceptions
Fixes #7268.
2022-12-02 15:03:40 +01:00
Valentin Gagarin
0ce5742bec
Merge pull request #7381 from aakropotkin/doc-ltoa-conflicts
doc: listToAttrs: document repeated keys
2022-12-02 14:19:34 +01:00
Eelco Dolstra
1e6a5d1ff6 Clean up cgroup handling in getMaxCPU()
Also, don't assume in LocalDerivationGoal that cgroups are mounted on
/sys/fs/cgroup.
2022-12-02 12:59:13 +01:00
Eelco Dolstra
1211e59a03 Move cgroup.{cc,hh} to libutil 2022-12-02 12:38:03 +01:00
Alex Ameen
ef524013aa doc: listToAttrs: add extra whitespace 2022-12-01 10:32:45 -06:00
Eelco Dolstra
f1e1ba9fe0 Really fix 'nix store make-content-addressed --json'
https://hydra.nixos.org/log/mcgypcf9vj4n8vdmw7lj3l05c899v73w-nix-2.12.0pre20221201_16b03f0-x86_64-unknown-linux-musl.drv
2022-12-01 16:29:09 +01:00
Alex Ameen
ec18b7d09b doc: listToAttrs: fix line wrapping 2022-11-30 23:21:09 -06:00
Alex Ameen
ad46726546 doc: listToAttrs: document repeated keys 2022-11-30 22:53:41 -06:00
Valentin Gagarin
0ea62670ed move documentation on auto-allocate-uids to options docs
this is where it belongs and can be found together with the other
options.
2022-12-01 04:40:02 +01:00
Eelco Dolstra
0b092bd87f nix store make-content-addressed: Fix JSON construction
Fixes

  error: [json.exception.type_error.301] cannot create object from initializer list

in tests/fetchClosure.sh.
2022-11-30 13:46:33 +01:00
Eelco Dolstra
fbc53e97ed
Merge pull request #3600 from NixOS/auto-uid-allocation
Automatic UID allocation
2022-11-29 14:01:42 +01:00
Eelco Dolstra
4f762e2b02 Restore ownership of / for non-uid-range builds 2022-11-29 13:10:53 +01:00
Eelco Dolstra
af8a32143b
Merge pull request #7358 from ncfavier/repl-exit-newline
repl: print a newline on ctrl-D
2022-11-29 11:31:43 +01:00
Eelco Dolstra
67bcb99700 Add a setting for enabling cgroups 2022-11-28 21:54:02 +01:00
Eelco Dolstra
ff12d1c1a1 Check that auto-allocated UIDs don't clash with existing accounts 2022-11-28 20:49:17 +01:00
Eelco Dolstra
dbf78a7ada
Merge pull request #7313 from yorickvP/nlohmann-everywhere
Replace src/libutil/json.cc with nlohmann
2022-11-28 15:03:48 +01:00
Naïm Favier
9b35cc716b
use logger->cout
in order to avoid potential problems with the progress bar

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2022-11-28 14:59:06 +01:00
Naïm Favier
04ec157517
repl: print a newline on ctrl-D 2022-11-28 10:38:23 +01:00
Eelco Dolstra
5b798f6cae Fix random client failures during GC server shutdown
We need to close the GC server socket before shutting down the active
GC client connections, otherwise a client may (re)connect and get
ECONNRESET. But also handle ECONNRESET for resilience.

Fixes random failures like

  GC socket disconnected
  connecting to '/tmp/nix-shell.y07M0H/nix-test/default/var/nix/gc-socket/socket'
  sending GC root '/tmp/nix-shell.y07M0H/nix-test/default/store/kb5yzija0f1x5xkqkgclrdzldxj6nnc6-non-blocking'
  reading GC root from client: error: unexpected EOF reading a line
  1 store paths deleted, 0.00 MiB freed
  error: reading from file: Connection reset by peer

in gc-non-blocking.sh.
2022-11-27 12:57:18 +01:00
Eelco Dolstra
0b4c4d7434 Don't use GC_STRNDUP
It calls strlen() on the input (rather than simply copying at most
`size` bytes), which can fail if the input is not zero-terminated and
is inefficient in any case.

Fixes #7347.
2022-11-25 22:30:56 +01:00
John Ericson
26534f141c
Merge branch 'master' into indexed-store-path-outputs 2022-11-25 08:14:32 -05:00
Théophane Hufschmitt
bc9692a6b7
Merge pull request #7337 from Radvendii/why-depends-ca
Fix why-depends for CA derivations
2022-11-23 20:16:14 +01:00
Taeer Bar-Yam
bd8571a5c3 add explanation and test 2022-11-23 12:06:47 -05:00
Taeer Bar-Yam
b13fd4c58e Fix why-depends for CA derivations
why-depends assumed that we knew the output path of the second argument.
For CA derivations, we might not know until it's built. One way to solve
this would be to build the second installable to get the output path.

In this case we don't need to, though. If the first installable (A)
depends on the second (B), then getting the store path of A will
necessitate having the store path B. The contrapositive is, if the store
path of B is not known (i.e. it's a CA derivation which hasn't been
built), then A does not depend on B.
2022-11-23 11:39:50 -05:00
Eelco Dolstra
6292d5616e Merge remote-tracking branch 'origin/master' into auto-uid-allocation 2022-11-23 11:16:09 +01:00
Eelco Dolstra
05d0892443
Merge pull request #7328 from edolstra/nix-build-stats
nix build --json: Include build statistics
2022-11-22 14:41:15 +01:00
Eelco Dolstra
3d23b9d032 SimpleUserLock::getSupplementaryGIDs(): Filter out main gid
This avoids having the user's gid in the supplementary group list as
well.
2022-11-22 10:26:17 +01:00
Eelco Dolstra
b37c2d84b6 Always call setgroups()
We shouldn't skip this if the supplementary group list is empty,
because then the sandbox won't drop the supplementary groups of the
parent (like "root").
2022-11-22 10:26:17 +01:00
Eelco Dolstra
02c02ee7c3
Merge pull request #6456 from amjoseph-nixpkgs/seccomp-mips
local-derivation-goal.cc: enable seccomp filters for mips{32,64}
2022-11-21 23:03:00 +01:00
Eelco Dolstra
c776dfbb35
Use hex for startId
Co-authored-by: Linus Heckemann <git@sphalerite.org>
2022-11-21 18:46:55 +01:00
Eelco Dolstra
9d17ce07e8 AutoUserLock: If sandboxing is disabled, use the build users group
We have to use a gid that has write access to the Nix store.
2022-11-21 12:55:49 +01:00
Eelco Dolstra
f0baa5c128 nix build --json: Include build statistics
Example:

  # nix build -L --extra-experimental-features cgroups --impure --expr 'with import <nixpkgs> {}; runCommand "foo" {} "dd if=/dev/urandom bs=1M count=1024 | md5sum; mkdir $out"' --json
  [
    {
      "cpuSystem": 1.911431,
      "cpuUser": 1.214249,
      "drvPath": "/nix/store/xzdqz67xba18hljhycp0hwfigzrs2z69-foo.drv",
      "outputs": {
        "out": "/nix/store/rh9mc9l2gkpq8kn2sgzndr6ll7ffjh6l-foo"
      },
      "startTime": 1669024076,
      "stopTime": 1669024079
    }
  ]
2022-11-21 12:06:01 +01:00
Eelco Dolstra
e7a5b76844 Rename derivedPathsWithHintsToJSON -> builtPathsToJSON 2022-11-21 11:56:20 +01:00
Eelco Dolstra
82d5cf2a76 Fix macOS build 2022-11-21 11:45:41 +01:00
Eelco Dolstra
653b32a78f Merge remote-tracking branch 'origin/master' into auto-uid-allocation 2022-11-21 11:33:23 +01:00
Eelco Dolstra
ec45f4b82e Fix indentation 2022-11-21 11:12:45 +01:00
Eelco Dolstra
300753d594 nix build --json: Include build statistics
Example:

  # nix build -L --extra-experimental-features cgroups --impure --expr 'with import <nixpkgs> {}; runCommand "foo" {} "dd if=/dev/urandom bs=1M count=1024 | md5sum; mkdir $out"' --json
  [
    {
      "cpuSystem": 1.911431,
      "cpuUser": 1.214249,
      "drvPath": "/nix/store/xzdqz67xba18hljhycp0hwfigzrs2z69-foo.drv",
      "outputs": {
        "out": "/nix/store/rh9mc9l2gkpq8kn2sgzndr6ll7ffjh6l-foo"
      },
      "startTime": 1669024076,
      "stopTime": 1669024079
    }
  ]
2022-11-21 10:49:01 +01:00
Eelco Dolstra
f538ee4342 Rename derivedPathsWithHintsToJSON -> builtPathsToJSON 2022-11-21 09:38:08 +01:00
Eelco Dolstra
e6b71f84a0 Use cgroup.kill to quickly kill cgroups 2022-11-18 16:59:36 +01:00
Eelco Dolstra
fa68eb367e Get CPU stats from the cgroup 2022-11-18 13:40:59 +01:00
Eelco Dolstra
20f66c6889 Indentation 2022-11-18 13:40:48 +01:00
Eelco Dolstra
128910ba23 Separate cgroup support from auto-uid-allocation
The new experimental feature 'cgroups' enables the use of cgroups for
all builds. This allows better containment and enables setting
resource limits and getting some build stats.
2022-11-18 10:39:28 +01:00
Eelco Dolstra
f1ab082ac4 createTempDir(): Use std::atomic 2022-11-18 09:37:11 +01:00
Eelco Dolstra
f423d4425f Fix segfault in unprivileged mode 2022-11-17 11:56:45 +01:00
Yorick van Pelt
09f00dd4d0
Replace src/libutil/json.cc with nlohmann json generation 2022-11-16 16:50:50 +01:00
Guillaume Maudoux
a7d2a3d087 Allow system certs access to fixed-output derivations 2022-11-16 15:23:59 +01:00
Théophane Hufschmitt
62960f3291
Merge pull request #7134 from yorickvP/disable-dbg-on-complete
Temporarily disable the debugger during completion evaluation
2022-11-16 11:28:40 +01:00
Théophane Hufschmitt
60dea270d0
Swallow the error in a more idiomatic way 2022-11-16 10:34:32 +01:00
Robert Hensing
bcd298d39b libstore/derivation-goal: Elaborate a TODO for performance concern 2022-11-15 17:57:40 +01:00
Théophane Hufschmitt
4bf70b74a7
Merge pull request #7294 from tobim/support-aws-sdk-1.10
libstore: link to aws-crt-cpp
2022-11-15 16:51:09 +01:00
Théophane Hufschmitt
3ade5f5d60
Merge pull request #7283 from hercules-ci/issue-6572
Fix #6572 `requires non-existent output`
2022-11-15 16:24:24 +01:00
Théophane Hufschmitt
daf1423a4a
Merge pull request #7260 from ncfavier/readFile-scan-references
Restrict `readFile` context to references that appear in the string
2022-11-15 16:22:28 +01:00
Robert Hensing
7e162c69fe derivation-goal: Fix requires non-existing output error
It occurred when a output of the dependency was already available,
so it didn't need rebuilding and didn't get added to the
inputDrvOutputs.
This process-related info wasn't suitable for the purpose of finding
the actual input paths for the builder. It is better to do this in
absolute terms by querying the store.
2022-11-14 17:52:55 +01:00
Théophane Hufschmitt
8b4352d79b Merge remote-tracking branch 'nixos/master' into readFile-scan-references 2022-11-14 15:00:05 +01:00
Et7f3
efadeee8fd
build: use pkg-config for lowdown 2022-11-12 23:04:58 +01:00
Tobias Mayer
07f2cb1e8f
libstore: link to aws-crt-cpp
This change is needed to support aws-sdk-cpp 1.10 and newer.

I opted not to make this dependent on the sdk version because
the crt dependency has been in the interface of the older
sdk as well, and it was only coincidence that libstore didn't
make use of any privately defined symbols directly.
2022-11-12 14:34:23 +01:00
Naïm Favier
e7ed9ae0c7
Restrict readFile context to references that appear in the string
When calling `builtins.readFile` on a store path, the references of that
path are currently added to the resulting string's context.

This change makes those references the *possible* context of the string,
but filters them to keep only the references whose hash actually appears
in the string, similarly to what is done for determining the runtime
references of a path.
2022-11-11 13:04:34 +01:00
Théophane Hufschmitt
9550b1d519
Merge pull request #7258 from ncfavier/fix-eval-error-fmt
Fix printing of eval errors with two format placeholders
2022-11-10 18:41:16 +01:00
Eelco Dolstra
6c6eff8ac4 Remove the SystemdCgroup feature 2022-11-10 17:24:12 +01:00
Sandro
481e4082bf
Update src/libutil/experimental-features.cc
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2022-11-09 12:03:53 +01:00
Eelco Dolstra
05d258667d Fix build on macOS 2022-11-08 08:00:29 -08:00
Eelco Dolstra
2fde7e0108 Split auto UID allocation from cgroups
Cgroups are now only used for derivations that require the uid-range
range feature. This allows auto UID allocation even on systems that
don't have cgroups (like macOS).

Also, make things work on modern systems that use cgroups v2 (where
there is a single hierarchy and no "systemd" controller).
2022-11-08 16:03:42 +01:00
Eelco Dolstra
b378876894
Merge pull request #7264 from patricksjackson/remote-brackets
build-remote: Add brackets to error message
2022-11-07 16:59:29 +01:00
Eelco Dolstra
995f5f2e41
Merge pull request #7267 from fricklerhandwerk/build-task
manual: build action -> build task
2022-11-07 16:58:47 +01:00
Valentin Gagarin
f7ab93b068 manual: build action -> build task
after discussing this with multiple people, I'm convinced that "build
task" is more precise: a derivation is not an action, but inert until it
is built. also it's easier to pronounce.

proposal: use "build task" for the generic concept "description of how
to derive new files from the contents of existing files". then it will
be easier to distinguish what we mean by "derivation" (a specific data
structure and Nix language value type) and "store derivation" (a
serialisation of a derivation into a file in the Nix store).
2022-11-06 13:28:18 +01:00
Patrick Jackson
907f52c337 build-remote: Add brackets to error message 2022-11-04 10:49:44 -07:00
Eelco Dolstra
40911d7dec Remove stray tab 2022-11-04 13:30:35 +01:00
Eelco Dolstra
6259fd7ea6 Fix indentation 2022-11-04 13:29:39 +01:00
Naïm Favier
dad859ba0f
Fix printing of eval errors with two format placeholders 2022-11-04 12:41:38 +01:00
Eelco Dolstra
b95faccf03 Merge remote-tracking branch 'origin/master' into auto-uid-allocation 2022-11-03 17:43:40 +01:00
Yorick van Pelt
16f1720fd2
SourceExprCommand: swallow EvalError, add tests for this
Completing things that would error would print an ugly error in
the middle of your command line. Avoid printing this error.
2022-11-03 10:11:28 +01:00
Yorick van Pelt
444af85552
Temporarily disable the debugger during completion evaluation
readline is not re-entrant, so entering the debugger from the
completioncallback results in an eventual segfault.

The workaround is to temporarily disable the debugger when searching
for possible completions.
2022-11-02 13:02:24 +01:00
Yorick van Pelt
34ea0e2e7b
tarfile: set directory mode to at least 0500, don't extract fflags
We don't need SGID, or any ACL's. We also want to keep every dir +rx.
2022-11-01 16:01:38 +01:00
Eelco Dolstra
0edba4cc1e
Merge pull request #7231 from jfroche/fix/savedArgv-access
Move savedArgv into libmain
2022-10-31 16:35:08 +01:00
Théophane Hufschmitt
90ed041677
Merge pull request #7039 from Mic92/libfetchers-variant
libfetchers: avoid api.github.com ratelimit if no github token is set
2022-10-31 11:48:51 +01:00
John Ericson
13f2a6f38d
Merge branch 'master' into indexed-store-path-outputs 2022-10-28 23:22:18 +01:00
Jean-François Roche
cd86eeb693
Move savedArgv into libmain
`savedArgv` is not accessible by plugins when defined in main binary.
Moving it into one of the nix lib fix the problem.
2022-10-28 12:19:37 +02:00
Eelco Dolstra
b7e8a3bf4c
Merge pull request #7217 from NixOS/restore-nix-build-remote
Fix `nix __build-remote`
2022-10-27 13:33:51 +02:00
Théophane Hufschmitt
f8d0193383 Pass the right argv when calling the build hook
Call it as `['nix', '__build-remote', ... ]` rather than the previous
`["__build-remote", "nix __build-remote", ... ]` which seemed to have
been most likely unintended
2022-10-27 11:53:04 +02:00
Eelco Dolstra
9323d139b0
Merge pull request #7174 from agbrooks/git-tag-bug
Prevent fetchGit from using incorrect cached rev for different refs
2022-10-26 16:48:48 +02:00
Théophane Hufschmitt
9bff7e8ee2 Fix nix __build-remote
Because of a wrong index, `nix __build-remote` wasn't working.

Fix the index to restore the command (and the build hook).
2022-10-26 11:53:46 +02:00
Yorick van Pelt
aff6d10934
nix run: fix "'defaultApp.x86_64-linux' should have type 'derivation'" 2022-10-26 10:05:27 +02:00
Théophane Hufschmitt
899878f77a
Merge pull request #7206 from akiekintveld/master
Defer to SSH config files for ForwardAgent option
2022-10-25 20:26:52 +02:00
Eelco Dolstra
fed1700754
Merge pull request #7207 from NixOS/flake-type
Mark flakes with ._type = "flake".
2022-10-25 17:31:23 +02:00
Eelco Dolstra
da2c61637b
Use _type
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2022-10-25 16:48:31 +02:00
Guillaume Maudoux
d6f5734c63 Complete genericClosure tests 2022-10-25 01:57:18 +02:00
Guillaume Maudoux
e93bf69b44 Rework error throwing, and test it 2022-10-25 01:46:10 +02:00
Michal Sojka
a9a868fe6a Improve --profile description
The description of the --profile option talks about the "update" operation.
This is probably meant for operations such as "nix profile install", but the
same option is reused in other subcommands, which do not update the profile,
such as "nix profile {list,history,diff-closures}".

We update the description to make sense in both contexts.
2022-10-24 08:49:46 +02:00
Shea Levy
334fa81d08
Mark flakes with .type = "flake".
Fixes #7186
2022-10-23 06:54:11 -04:00
Austin Kiekintveld
8e7804273c Defer to SSH config files for ForwardAgent option
Currently, Nix passes `-a` when it runs commands on a remote machine via
SSH, which disables agent forwarding. This causes issues when the
`ForwardAgent` option is set in SSH config files, as the command line
operation always overrides those.

In particular, this causes issues if the command being run is `sudo`
and the remote machine is configured with the equivalent of NixOS's
`security.pam.enableSSHAgentAuth` option. Not allowing SSH agent
forwarding can cause authentication to fail unexpectedly.

This can currently be worked around by setting `NIX_SSHOPTS="-A"`, but
we should defer to the options in the SSH config files to be least
surprising for users.
2022-10-22 19:51:22 -05:00
Guillaume Maudoux
8bd8583bc7 Try to please clang with convoluted templates 2022-10-23 00:11:44 +02:00
Guillaume Maudoux
8c3afd2d68 Introduce an Error builder to tackle complexity 2022-10-22 23:37:54 +02:00
Graham Bennett
4563e80363 Fix C++20 warnings 2022-10-22 15:16:46 +01:00
Graham Bennett
c5fd34a14e Build with C++20 2022-10-22 14:24:25 +01:00
Guillaume Maudoux
4a909c142c Rollback unneeded throwFrameErrorWithTrace function 2022-10-20 14:25:11 +02:00
Guillaume Maudoux
31ce52a045 Fix context message being printed twice with forceStringNoCtx 2022-10-20 14:18:35 +02:00
Guillaume Maudoux
512f6be9b5 Reword incomparable types message 2022-10-20 14:17:05 +02:00
Guillaume Maudoux
520404f450 Revert custom position of 'if' blocks 2022-10-20 13:55:15 +02:00
Eelco Dolstra
95331cb9c9
Merge pull request #7183 from n8henrie/patch-1
Fix typo -- dashes not underscores
2022-10-19 11:21:52 +02:00
Eelco Dolstra
e136d57f26
Implement BinaryCacheStore::queryPathFromHashPart() 2022-10-18 17:48:09 +02:00
Eelco Dolstra
61f89e954a
Add command 'nix store path-from-hash-part'
This exposes the Store::queryPathFromHashPart() interface in the CLI.
2022-10-18 16:51:12 +02:00
Nathan Henrie
af9c9504ca
Fix typo -- dashes not underscores 2022-10-17 14:15:32 -06:00
Guillaume Maudoux
b945b844a9 Initial frames support 2022-10-17 03:05:02 +02:00
Guillaume Maudoux
3f9f6ae127 Merge remote-tracking branch 'origin/master' into coerce-string 2022-10-16 20:39:19 +02:00
sternenseemann
d0f2da214b primops: make nature of foldl' strictness clearer
* Clarify the documentation of foldl': That the arguments are forced
  before application (?) of `op` is necessarily true. What is important
  to stress is that we force every application of `op`, even when the
  value turns out to be unused.

* Move the example before the comment about strictness to make it less
  confusing: It is a general example and doesn't really showcase anything
  about foldl' strictness.

* Add test cases which nail down aspects of foldl' strictness:
  * The initial accumulator value is not forced unconditionally.
  * Applications of op are forced.
  * The list elements are not forced unconditionally.
2022-10-16 14:29:12 +02:00
Andrew Brooks
a259084c50 Fix #7146
When fetching a non-local git repo by ref (and no rev), don't consider unrelated
cached revs for the same repository.
2022-10-14 18:04:47 -05:00
Eelco Dolstra
3093bd3a85
Merge pull request #7168 from NixOS/rosetta-test
Improve Rosetta detection
2022-10-14 17:35:57 +02:00
Robert Hensing
ab4eb39386 libmain: Make the entire stack overflow handler pluggable 2022-10-14 12:53:46 +02:00
Robert Hensing
0d75675787 libmain: Add extraStackOverflowHandler 2022-10-14 12:53:07 +02:00
Eelco Dolstra
285277a61a Remove useless debug statements
We haven't parsed the '-v' command line flags yet when this code executes,
so we can't actually get debug output here.
2022-10-14 00:35:33 -07:00
Eelco Dolstra
ddd5503950 Use /usr/bin/true 2022-10-14 00:34:31 -07:00
Eelco Dolstra
0359d6d123 Fix error display if execve() in the builder fails
After we've send "\2\n" to the parent, we can't send a serialized
exception anymore. It will show up garbled like

  $ nix-build --store /tmp/nix --expr 'derivation { name = "foo"; system = "x86_64-linux"; builder = "/foo/bar"; }'
  this derivation will be built:
    /nix/store/xmdip0z5x1zqpp6gnxld3vqng7zbpapp-foo.drv
  building '/nix/store/xmdip0z5x1zqpp6gnxld3vqng7zbpapp-foo.drv'...

  ErrorErrorEexecuting '/foo/bar': No such file or directory
  error: builder for '/nix/store/xmdip0z5x1zqpp6gnxld3vqng7zbpapp-foo.drv' failed with exit code 1
2022-10-13 21:35:16 +02:00
Eelco Dolstra
96eb5ef156 Improve Rosetta detection
Turns out that one of those *.plist files can exist even if Rosetta is
not installed. So let's just try to run an x86_64-darwin binary
directly.
2022-10-13 11:46:16 -07:00
Eelco Dolstra
59a304a9a8 Fix clang warnings 2022-10-13 11:25:49 -07:00
Eelco Dolstra
27ed3d0458
Merge pull request #7140 from crawford/chroot-store
Make warning about chroot store location more accurate
2022-10-13 16:35:31 +02:00
Steam Deck User
a86916eb72 Make warning about chroot store location more accurate
While trying to use an alternate directory for my Nix installation, I
noticed that nix's output didn't reflect the updated state
directory. This patch corrects that and now prints the warning before
attempting to create the directory (if the directory creation fails,
it wouldn't have been obvious why nix was attempting to create the
directory in the first place).

With this patch, I now get the following warning:

    warning: '/home/deck/.var/app/org.nixos.nix/var/nix' does not
    exist, so Nix will use '/home/deck/.local/share/nix/root' as a
    chroot store
2022-10-12 12:12:12 -07:00
Eelco Dolstra
eba610956b Move some options into a misc category
This unclutters the per-command options a bit by moving out some
global options.
2022-10-12 15:09:17 +02:00
Eelco Dolstra
ae2d330455 Merge branch 'print-common-flags' of github.com:Hoverbear/nix 2022-10-12 13:43:11 +02:00
Eelco Dolstra
f3193edd87
Merge pull request #7149 from amjoseph-nixpkgs/pr/intersectAttrs/values
parseDrvName: remove doc/impl discrepancy, add test covering the gap
2022-10-12 10:00:39 +02:00
Adam Joseph
7ef71cd21f src/libexpr/primops.cc: parseDrvName: make documentation follow implementation
The documentation for `parseDrvName` does not agree with the implementation when
the derivation name contains a dash which is followed by something that is
neither a letter nor a digit.  This commit corrects the documentation to agree
with the implementation.
2022-10-08 17:21:34 -07:00
Ana Hobden
069409d167 Print common flags in --help 2022-10-07 09:07:22 -07:00
Valentin Gagarin
927234cfb2
Merge pull request #6870 from amjoseph-nixpkgs/pr/doc/explain-local-remote-binary-substituter 2022-10-05 09:01:42 +02:00
Jörg Thalheim
d9abce4ad4 libfetchers: avoid api.github.com ratelimit if no github token is set
If we don't have any github token, we won't be able to fetch private
repos, but we are also more likely to run into API limits since
we don't have a token. To mitigate this only ever use the github api
if we actually have a token.
2022-10-01 10:38:09 +02:00
Eelco Dolstra
89ca75c9f9
Merge pull request #7080 from squalus/nar-close-file
archive: check close errors when extracting nars
2022-09-30 12:58:28 +02:00
Linus Heckemann
19f3ecd830 nix-build: remove unused --add-root arg
Fixes #1982
2022-09-27 00:07:42 +02:00
Eelco Dolstra
6806f20157
Merge pull request #7089 from NixOS/mention-accept-flake-config-in-warning
Mention `--accept-flake-config` in the related warning
2022-09-26 11:13:02 +02:00
Adam Joseph
dc205c75a2 src/libexpr/primops.cc: correct definition for intersectAttrs
The current definition of `intersectAttrs` is incorrect:

> Return a set consisting of the attributes in the set e2 that also exist in the
> set e1.

Recall that (Nix manual, section 5.1):

> An attribute set is a collection of name-value-pairs (called attributes)

According to the existing description of `intersectAttrs`, the following should
evaluate to the empty set, since no key-value *pair* (i.e. attribute) exists in
both sets:

```
builtins.intersectAttrs { x=3; } {x="foo";}
```

And yet:

```
nix-repl> builtins.intersectAttrs { x=3; } {x="foo";}
{ x = "foo"; }
```

Clearly the intent here was for the *names* of the resulting attribute set to be
the intersection of the *names* of the two arguments, and for the values of the
resulting attribute set to be the values from the second argument.

This commit corrects the definition, making it match the implementation and intent.
2022-09-24 23:51:09 -07:00
Théophane Hufschmitt
371013c08d
Merge pull request #7010 from edolstra/ignore-shell
nix develop: Ignore stdenv's $SHELL
2022-09-24 12:55:53 +02:00
Théophane Hufschmitt
db29ddd113
Merge pull request #7078 from obsidiansystems/trustworthy-signature
"valid signature" -> "signature by a trusted key"
2022-09-24 12:52:35 +02:00
Théophane Hufschmitt
74276cb354
Merge pull request #7079 from matthewbauer/allow-untrusted-settings
Allow pass max-silent-time and build-poll-interval to daemon untrusted
2022-09-24 12:47:49 +02:00
Théophane Hufschmitt
1a5d094be7
Mention --accept-flake-config in the related warning
Make sure that people who run Nix in non-interactive mode (and so don't have the possibility to interactively accept the individual flake configuration settings) are aware of this flag.

Fix #7086
2022-09-24 12:11:26 +02:00
Eelco Dolstra
c13007f012
Merge pull request #7059 from NixOS/remove-useless-ca-file-message
Remove a useless debug message in filetransfer.cc
2022-09-24 10:39:23 +02:00
John Ericson
60e23c8bae
Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Rune K. Svendsen <runesvend@gmail.com>
2022-09-23 13:57:57 -04:00
squalus
223f8dace0 archive: check close errors when extracting nars 2022-09-22 12:50:32 -07:00
Matthew Bauer
6e049ae607 Allow pass max-silent-time and build-poll-interval to daemon untrusted
These settings seem harmless, they control the same polling
functionality that timeout does, but with different behavior. Should
be safe for untrusted users to pass in.
2022-09-22 13:59:16 -05:00
John Ericson
a2a8cb10ac Dodge "trusted" vs "trustworthy" by being explicit
Hopefully this is best!
2022-09-22 14:37:52 -04:00
John Ericson
752f967c0f "valid signature" -> "trustworthy signature"
I just had a colleague get confused by the previous phrase for good
reason. "valid" sounds like an *objective* criterion, e.g. and *invalid
signature* would be one that would be trusted by no one, e.g. because it
misformatted or something.

What is actually going is that there might be a signature which is
perfectly valid to *someone else*, but not to the user, because they
don't trust the corresponding public key. This is a *subjective*
criterion, because it depends on the arbitrary and personal choice of
which public keys to trust.

I therefore think "trustworthy" is a better adjective to use. Whether
something is worthy of trust is clearly subjective, and then "trust"
within that word nicely evokes `trusted-public-keys` and friends.
2022-09-22 10:49:31 -04:00
Théophane Hufschmitt
f704c2720f
Merge pull request #7070 from alyssais/repl-history
nix repl: warn if creating dir for history fails
2022-09-21 13:50:49 +02:00
Alyssa Ross
d234d01f01
nix repl: warn if creating dir for history fails
The history is not critical to the functionality of nix repl, so it's
enough to warn here, rather than refuse to start if the directory Nix
thinks the history should live in can't be created.
2022-09-20 11:01:40 +00:00
squalus
1b595026e1 Improve durability of schema version file writes
- call close explicitly in writeFile to prevent the close exception
  from being ignored
- fsync after writing schema file to flush data to disk
- fsync schema file parent to flush metadata to disk

https://github.com/NixOS/nix/issues/7064
2022-09-19 20:13:30 -07:00
Théophane Hufschmitt
0f977bf91e
Remove a useless debug message in filetransfer.cc
Remove the `verify TLS: Nix CA file = 'blah'` message that Nix used to print when fetching anything as it's both useless (`libcurl` prints the same info in its logs) and misleading (gives the impression that a new TLS connection is being established which might not be the case because of multiplexing. See #7011 )
2022-09-19 08:42:43 +02:00
Adam Joseph
fb985f855c fetchurl.nix: change other use of __impure 2022-09-16 01:54:24 -07:00
Adam Joseph
673fd21b7c
Update src/libexpr/fetchurl.nix
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2022-09-16 08:51:14 +00:00
Adam Joseph
b3550d9179 libexpr/fetchurl.nix: allow __impure fetch
This commit adds an optional `__impure` parameter to fetchurl.nix, which allows
the caller to use `libfetcher`'s fetcher in an impure derivation.  This allows
nixpkgs' patch-normalizing fetcher (fetchpatch) to be rewritten to use nix's
internal fetchurl, thereby eliminating the awkward "you can't use fetchpatch
here" banners scattered all over the place.

See also: https://github.com/NixOS/nixpkgs/pull/188587
2022-09-16 00:48:01 -07:00
Eelco Dolstra
d365cced4f
Trim option descriptions
This removes unintended blank lines in Markdown when the description
is a multiline string literal.
2022-09-13 16:58:32 +02:00
Eelco Dolstra
c6ff33ff5c
RunPager: Stop the progress bar
In particular, the progress bar was interfering with 'less' rendering
in '--help' (e.g. run 'nix --help' and hit '/' to search).
2022-09-13 15:29:13 +02:00
Théophane Hufschmitt
0f64bf445a
Merge pull request #6994 from agbrooks/master
Prevent tempdir from being GC-ed before addToStoreFromDump completes
2022-09-13 09:23:16 +02:00
Andrew Brooks
565d888e0f Address PR feedback on #6694 2022-09-12 11:33:23 -05:00
Guillaume Maudoux
96f2dd99d3 fix remaining foo stuff 2022-09-11 23:09:36 +02:00
Guillaume Maudoux
e412bb6d30 fix remaining fixtures 2022-09-11 22:58:59 +02:00
Guillaume Maudoux
c7b901fd33 Cleanup error strings rebase 2022-09-11 01:34:19 +02:00
Valentin Gagarin
dceb4f9dbb
Merge pull request #6825 from rapenne-s/nix_shell_example
nix shell: document how to invoke multiple commands from the command …
2022-09-09 13:35:23 +02:00
Théophane Hufschmitt
c8e8eea95e
Merge pull request #7008 from jonringer/add-command-docs
Docs: Add nix develop --command entry
2022-09-08 13:34:40 +02:00
Max
6da4cc92d8 showForeign -> showAllSystems 2022-09-07 20:28:30 +02:00
Andrew Brooks
84fe75a12a Keep created temp dirs inside store, but protect from GC
Implements the approach suggested by feedback on PR #6994, where
tempdir paths are created in the store (now with an exclusive lock).

As part of this work, the currently-broken and unused
`createTempDirInStore` function is updated to create an exclusive lock
on the temp directory in the store.

The GC now makes a non-blocking attempt to lock any store directories
that "look like" the temp directories created by this function, and if
it can't acquire one, ignores the directory.
2022-09-06 17:48:00 -05:00
Guillaume Maudoux
eb460a9529 WIP: broken merge but need a git checkpoint 2022-09-07 00:34:03 +02:00
Eelco Dolstra
ece12a97d9 lockfile -> lock file for consistency 2022-09-06 19:20:31 +02:00
Eelco Dolstra
27be54ca53 nix develop: Ignore stdenv's $SHELL
Stdenv sets this to a bash that doesn't have readline/completion
support, so running 'nix (develop|shell)' inside a 'nix develop' gives
you a crippled shell. So let's just ignore the derivation's $SHELL.

This could break interactive use of build phases that use $SHELL, but
they appear to be fairly rare.
2022-09-06 18:27:39 +02:00
Eelco Dolstra
9600109879
Merge pull request #6995 from matthewbauer/disable-sa-restart-on-macos
Disable SA_RESTART for some signals on macOS
2022-09-06 17:36:31 +02:00
Jonathan Ringer
6ce2e96c88
Docs: Add nix develop --command entry
Add example of nix develop being used to execuate a series of script
commands. This is common when doing things like CI/CD, and should be
represented in the official documentation.

Also useful for people looking for the 'nix develop' equivalent of
'nix-shell --run'.

Related:
 - https://github.com/NixOS/nix/issues/6908
 - https://github.com/NixOS/nix/issues/6908#issuecomment-1229266853
2022-09-06 08:18:29 -07:00
Matthew Bauer
77d3d3d18d Remove some signals 2022-09-06 09:47:53 -05:00
Sandro
408dcfc0d3
Improve experimental-features error wording 2022-09-05 15:42:10 +02:00
Matthew Bauer
a47b5476e1 Add more signals 2022-09-03 16:06:33 -05:00
Matthew Bauer
102434e4cb Disable SA_RESTART for some signals on macOS
Disables the SA_RESTART behavior on macOS which causes:

> Restarting of pending calls is requested by setting the SA_RESTART bit
> in sa_flags. The affected system calls include read(2), write(2),
> sendto(2), recvfrom(2), sendmsg(2) and recvmsg(2) on a communications
> channel or a slow device (such as a terminal, but not a regular file)
> and during a wait(2) or ioctl(2).

From: https://man.openbsd.org/sigaction#SA_RESTART

This being set on macOS caused a bug where read() calls to the daemon
socket were blocking after a SIGINT was received. As a result,
checkInterrupt was never reached even though the signal was received
by the signal handler thread.

On Linux, SA_RESTART is disabled by default. This probably effects
other BSDs but I don’t have the ability to test it there right now.
2022-09-03 00:27:16 -05:00
Andrew Brooks
1f041ac54f Prevent tempdir from being GC-ed before addToStoreFromDump has renamed it
This fixes issue 6823 by placing the tempdir used in LocalStore::addToStoreFromDump
outside the Nix store, where automatic GC is no longer a concern.
2022-09-02 18:32:35 -05:00
Matthew Bauer
4894e567fb Don’t readDerivation if impure derivations feature is disabled
readDerivation is pretty slow, and while it may not be significant for
some use cases, on things like ghc-nix where we have thousands of
derivations is really slows things down.

So, this just doesn’t do the impure derivation check if the impure
derivation experimental feature is disabled. Perhaps we could cache
the result of isPure() and keep the check, but this is a quick fix to
for the slowdown introduced with impure derivations features in 2.8.0.
2022-09-02 11:46:34 -05:00
Adam Joseph
1ab913467e linkify mention of other options 2022-09-01 18:03:35 -07:00
Adam Joseph
59dc8346ca move substituter signature-checking conditions to configuration file documentation 2022-09-01 17:51:56 -07:00
Max
02bff90e7b nix flake show: don't evaluate derivations for foreign systems by default 2022-09-01 22:04:22 +02:00
Eelco Dolstra
a8b3d777fb Revert "Merge pull request #6621 from Kha/nested-follows"
This reverts commit c530cda345, reversing
changes made to 4adcdff5c1.
2022-09-01 15:26:19 +02:00
Théophane Hufschmitt
c530cda345
Merge pull request #6621 from Kha/nested-follows
Fix nested flake input overrides
2022-09-01 12:04:00 +02:00
Eelco Dolstra
1f2b12b30e
Merge pull request #6968 from fricklerhandwerk/man-no-weblinks
do not render relative links in help pages
2022-08-29 08:46:35 +02:00
Sebastian Ullrich
6f65c11780 Fix overlapping flake follows 2022-08-28 11:50:25 +02:00
Sebastian Ullrich
2b9d381301 Fix nested flake input overrides 2022-08-28 11:47:25 +02:00
Valentin Gagarin
e7dcacbd7c do not render relative links in help pages
this simplifies the setup a lot, and avoids weird looking `./file.md`
links showing up.

it also does not show regular URLs any more. currently the command
reference only has few of them, and not showing them in the offline
documentation is hopefully not a big deal.

instead of building more special-case solutions, clumsily preprocessing
the input, or issuing verbal rules on dealing with URLs, should better
be solved sustainably by not rendering relative links in `lowdown`:

https://github.com/kristapsdz/lowdown/issues/105
2022-08-26 17:10:41 +02:00
Eelco Dolstra
7918adbb62
Merge pull request #6954 from winterqt/darwin-sandbox-trustd
fix(libstore): allow access to trustd on macOS
2022-08-26 11:45:00 +02:00
Eelco Dolstra
bb411e4ae1 Fix progress bar flicker with -L
This was caused by -L calling setLogFormat() again, which caused the
creation of a new progress bar without destroying the old one. So we
had two progress bars clobbering each other.

We should change 'logger' to be a smart pointer, but I'll do that in a
future PR.

Fixes #6931.
2022-08-24 22:36:40 +02:00
Eelco Dolstra
a17ce0a8a9 Fix evaluation cache
98e361ad4c introduced a regression where
previously stored attributes were replaced by placeholders. As a
result, a command like 'nix build nixpkgs#hello' had to be executed at
least twice to get caching.

This code does not seem necessary for suggestions to work.
2022-08-24 21:19:43 +02:00
Winter
8e5659423e fix(libstore): allow access to trustd on macOS 2022-08-24 13:09:44 -04:00
Eelco Dolstra
04e74f7c8b
Merge pull request #6612 from NixOS/parallel-nix-copy
Make nix copy parallel again
2022-08-24 15:31:42 +02:00
Eelco Dolstra
56d97d4b4d Remove redundant Finally 2022-08-24 14:49:58 +02:00
Eelco Dolstra
8d906b1f3b Fix macOS build 2022-08-24 14:11:03 +02:00
Rickard Nilsson
c2d7456926 Fix a misplaced parenthese in serve protocol check
This issue made it impossible for clients using a serve protocol of
version <= 2.3 to use the `cmdBuildDerivation` command of servers using
a protocol of version >= 2.6. The faulty version check makes the server
send back build outputs that the client is not expecting.
2022-08-24 01:54:43 +02:00
Eelco Dolstra
4a0c4ca186
Merge pull request #6944 from edolstra/bash-vars
nix develop: Ignore some more bash special variables
2022-08-23 20:15:18 +02:00
Eelco Dolstra
0e54fab0dd
Merge pull request #6909 from ncfavier/json-paths
JSON: print paths as strings without copying them to the store
2022-08-23 15:19:33 +02:00
Eelco Dolstra
db026103b1 nix develop: Ignore some more bash special variables
Fixes #6940.
2022-08-23 14:57:08 +02:00
Eelco Dolstra
db71f5646f
Merge pull request #6887 from winterqt/specify-bashinteractive-outputs
nix-shell: specify which outputs from bashInteractive to build
2022-08-23 14:44:23 +02:00
Naïm Favier
ff0b5a778c
Revert to copyStore = true for nix-instantiate and nix-env 2022-08-23 14:40:27 +02:00
Eelco Dolstra
f0358ed465 Fix a hang in nix-copy-ssh.sh
This hang for some reason didn't trigger in the Nix build, but did
running 'make installcheck' interactively. What happened:

* Store::addMultipleToStore() calls a SinkToSource object to copy a
  path, which in turn calls LegacySSHStore::narFromPath(), which
  acquires a connection.

* The SinkToSource object is not destroyed after the last bytes has
  been read, so the coroutine's stack is still alive and its
  destructors are not run. So the connection is not released.

* Then when the next path is copied, because max-connections = 1,
  LegacySSHStore::narFromPath() hangs forever waiting for a connection
  to be released.

The fix is to make sure that the source object is destroyed when we're
done with it.
2022-08-23 14:19:53 +02:00
Eelco Dolstra
f865048332 Indentation 2022-08-22 15:32:53 +02:00
Eelco Dolstra
4c96761c2b Merge remote-tracking branch 'origin/master' into parallel-nix-copy 2022-08-22 15:29:10 +02:00
Eelco Dolstra
7c3ab57515
Merge pull request #6937 from edolstra/fix-repl
nix repl: Stop the progress bar
2022-08-22 15:10:01 +02:00
Naïm Favier
4c2ff4a0f4
JSON: add missing newlines
after `nix eval --json` and `nix-instantiate --eval --json`.
2022-08-22 15:07:52 +02:00
Naïm Favier
062e4fcdde
JSON: print paths as strings without copying them to the store
Makes `printValueAsJSON` not copy paths to the store for `nix eval
--json`, `nix-instantiate --eval --json` and `nix-env --json`.

Fixes https://github.com/NixOS/nix/issues/5612
2022-08-22 15:01:35 +02:00
Eelco Dolstra
8d84634e26
Merge pull request #6926 from rapenne-s/download_limit
add a nix.conf option to set a download speed limit
2022-08-22 14:57:19 +02:00
Eelco Dolstra
590660c1c6
Merge pull request #6932 from jack1142/patch-1
Fix default profile path for root in nix profile documentation
2022-08-22 14:55:04 +02:00
Eelco Dolstra
0d2163c6dc nix repl: Stop the progress bar
The repl was broken since c3769c6846. In
general, the progress bar is incompatible with the repl.
2022-08-22 14:27:36 +02:00
Eelco Dolstra
9ddef9f0be
Merge pull request #6923 from ncfavier/nix-env-incomplete-json
Avoid producing incomplete JSON
2022-08-22 14:15:31 +02:00
Eelco Dolstra
c21b1a7e67
Spelling 2022-08-22 14:14:14 +02:00
Solène Rapenne
caad87e6db
Better documentation wording
Co-authored-by: Anderson Torres <torres.anderson.85@protonmail.com>
2022-08-20 18:21:36 +02:00
Jakub Kuczys
7d800909e9
Fix default profile path for root in nix profile documentation 2022-08-20 03:48:42 +02:00
Solene Rapenne
0d2bf7acf9 add a nix.conf option to set a download speed limit 2022-08-19 12:40:22 +02:00
pennae
7d934f7880 don't read outputs into memory for output rewriting
RewritingSink can handle being fed input where a reference crosses a
chunk boundary. we don't need to load the whole source into memory, and
in fact *not* loading the whole source lets nix build FODs that do not
fit into memory (eg fetchurl'ing data files larger than system memory).
2022-08-19 11:26:26 +02:00
Naïm Favier
7535ee345d
nix-env: don't output incomplete JSON 2022-08-19 01:00:10 +02:00
Naïm Favier
8188b1d0ab
json: write null on abnormal placeholder destruction
Avoids leaving dangling attributes like

    {
        "foo":
    }

in case of exceptions.
2022-08-19 01:00:10 +02:00
Eelco Dolstra
c80a74b7d5 Don't pass --force to 'git add'
Fixes #5810.
2022-08-17 16:59:02 +02:00
Eelco Dolstra
53e7b7e8ac Remove warnLargeDump()
This message was unhelpful (#1184) and probably misleading since
memory is O(1) in most cases now.
2022-08-17 11:32:01 +02:00
Eelco Dolstra
c3769c6846 ProgressBar: Delay before showing a new activity
Some activities are numerous but usually very short (e.g. copying a
source file to the store) which would cause a lot of flickering. So
only show activities that have been running for at least 10 ms.
2022-08-17 11:31:44 +02:00
Eelco Dolstra
e62160579f nix flake metadata: Don't show "Inputs" if there are no inputs 2022-08-17 11:31:34 +02:00
Eelco Dolstra
3d4489b623 Show when we're evaluating a flake 2022-08-17 11:31:27 +02:00
David Arnold
b58785ab83
docfix: bundlers 2022-08-10 18:49:29 -05:00
Théophane Hufschmitt
8f3fdef1e0
Merge pull request #6850 from NinjaTrappeur/nin/build-check
Fix Nix build --check flag
2022-08-10 17:44:06 +02:00
Winter
5f37c5191a nix-shell: specify which outputs from bashInteractive to build 2022-08-09 22:54:22 -04:00
Théophane Hufschmitt
5192bb093a Merge remote-tracking branch 'upstream/master' into fix-mv-in-different-filesystems 2022-08-08 15:42:56 +02:00
Théophane Hufschmitt
f4a8426098 Remove the explicit c++fs linkage on darwin
Doesn't seem needed on a recent-enough clang anymore (and even seems to
break stuff)
2022-08-08 14:34:22 +02:00
Dave Nicponski
cb6794a0d9 Do not spam logs if the owned-homedir check results in a noop 2022-08-07 10:13:11 -04:00
Erik Arvstedt
4c8441be0a docs/flake-update: fix example 2022-08-04 09:45:30 +02:00
Erik Arvstedt
53833dfb40 libexpr/flake: remove FIXME
Line 593 checks that all overrides (i.e. all elements of
`lockFlags.inputOverrides`) are members of `overridesUsed`.
2022-08-04 09:45:29 +02:00
Erik Arvstedt
c9f446ede1 flakeref: fix comment 2022-08-04 09:45:28 +02:00
Eelco Dolstra
ccbd906c86 Fix NIX_COUNT_CALLS=1
Also, make the JSON writer support std::string_view.

Fixes #6857.
2022-08-03 17:46:51 +02:00
Théophane Hufschmitt
1ba5b3e001 Make moveFile more atomic
Rather than directly copying the source to its dest, copy it first to a
temporary location, and eventually move that temporary.
That way, the move is at least atomic from the point-of-view of the destination
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt
90f9680733 Only use renameFile where needed
In most places the fallback to copying isn’t needed and can actually be
bad, so we’d rather not transparently fallback
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt
d71d9e9fbf moveFile -> renameFile
`move` tends to have this `mv` connotation of “I will copy it for you if
needs be”
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt
a4f0fd633c Link against c++fs on darwin
Required by the old clang version
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt
c5db1821a9 Re-implement the recursive directory copy
The recursive copy from the stl doesn’t exactly do what we need because
1. It doesn’t delete things as we go
2. It doesn’t keep the mtime, which change the nars

So re-implement it ourselves. A bit dull, but that way we have what we want
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt
6f89fb6008 rename: Fallback to a copy if the filesystems mismatch
In `nix::rename`, if the call to `rename` fails with `EXDEV` (failure
because the source and the destination are in a different filesystems)
switch to copying and removing the source.

To avoid having to re-implement the copy manually, I switched the
function to use the c++17 `filesystem` library (which has a `copy`
function that should do what we want).

Fix #6262
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt
c2de0a232c Create a wrapper around stdlib’s rename
Directly takes some c++ strings, and gently throws an exception on error
(rather than having to inline this logic everywhere)
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt
8119390abc Move some fs-related functions to their own file
Unclutter `util.cc` a bit
2022-08-03 10:27:25 +02:00
Félix Baylac-Jacqué
1467a98d4c
derivation-goal.cc: remove bmCheck custom return branch on buildDone
Once a derivation goal has been completed, we check whether or not
this goal was meant to be repeated to check its output.

An early return branch was preventing the worker to reach that repeat
code branch, hence breaking the --check command (#2619).

It seems like this early return branch is an artifact of a passed
refactoring. As far as I can tell, buildDone's main branch also
cleanup the tmp directory before returning.
2022-08-01 11:39:19 +02:00
Théophane Hufschmitt
2805439335
Merge pull request #6814 from amjoseph-nixpkgs/pr/sandbox-error-messages
local-derivation-goal.cc: improve error messages when sandboxing fails
2022-07-22 13:27:52 +02:00
Solène Rapenne
92bae33ca5
nix shell: example shouldn't use an absolute path for the shell
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2022-07-21 14:25:07 +02:00
Solène Rapenne
64404220f5 nix shell: document how to invoke multiple commands from the command line 2022-07-20 14:54:46 +02:00
Théophane Hufschmitt
7ed91d6c6a
Merge branch 'master' into parallel-nix-copy 2022-07-20 10:05:34 +02:00
Alex Wied
722de8ddcc libstore/globals.cc: Move cgroup detection to libutil 2022-07-19 16:25:53 -04:00
Alex Wied
1af5d798a4 libstore/globals.cc: Automatically set cores based on cgroup CPU limit
By default, Nix sets the "cores" setting to the number of CPUs which are
physically present on the machine. If cgroups are used to limit the CPU
and memory consumption of a large Nix build, the OOM killer may be
invoked.

For example, consider a GitLab CI pipeline which builds a large software
package. The GitLab runner spawns a container whose CPU is limited to 4
cores and whose memory is limited to 16 GiB. If the underlying machine
has 64 cores, Nix will invoke the build with -j64. In many cases, that
level of parallelism will invoke the OOM killer and the build will
completely fail.

This change sets the default value of "cores" to be
ceil(cpu_quota / cpu_period), with a fallback to
std:🧵:hardware_concurrency() if cgroups v2 is not detected.
2022-07-19 16:03:58 -04:00
Théophane Hufschmitt
56f6f3725f Don't ultimately trust the signed paths
Like the old implem did (and like you'd want it to be anyways)
2022-07-19 19:46:00 +02:00
Adam Joseph
36e1383b6b local-derivation-goal.cc: save global errno to the stack before performing tests which might clobber it 2022-07-19 03:53:20 -07:00
Adam Joseph
a9e75eca00 error.hh: add additional constructor with explicit errno argument 2022-07-19 03:49:33 -07:00
Adam Joseph
99fcc91f67 as requested by @thufschmitt https://github.com/NixOS/nix/pull/6814#discussion_r924275777 2022-07-19 03:33:12 -07:00
Adam Joseph
5f51539f88 change warn() to notice() 2022-07-19 03:30:52 -07:00
Adam Joseph
c8c6203c2c local-derivation-goal.cc: detect unprivileged_userns_clone failure mode
The workaround for "Some distros patch Linux" mentioned in
local-derivation-goal.cc will not help in the `--option
sandbox-fallback false` case.  To provide the user more helpful
guidance on how to get the sandbox working, let's check to see if the
`/proc` node created by the aforementioned patch is present and
configured in a way that will cause us problems.  If so, give the user
a suggestion for how to troubleshoot the problem.
2022-07-17 01:27:22 -07:00
Adam Joseph
6fc56318bf local-derivation-goal.cc: add comment re: CLONE_NEWUSER
local-derivation-goal.cc contains a comment stating that "Some distros
patch Linux to not allow unprivileged user namespaces."  Let's give a
pointer to a common version of this patch for those who want more
details about this failure mode.
2022-07-17 01:23:32 -07:00
Adam Joseph
8d35f387dc local-derivation-goal.cc: warn if failing and /proc/self/ns/user missing
This commit causes nix to `warn()` if sandbox setup has failed and
`/proc/self/ns/user` does not exist.  This is usually a sign that the
kernel was compiled without `CONFIG_USER_NS=y`, which is required for
sandboxing.
2022-07-16 19:37:27 -07:00
Adam Joseph
90830b1074 local-derivation-goal.cc: warn if failing due to max_user_namespaces==0
This commit uses `warn()` to notify the user if sandbox setup fails
with errno==EPERM and /proc/sys/user/max_user_namespaces is missing or
zero, since that is at least part of the reason why sandbox setup
failed.

Note that `echo -n 0 > /proc/sys/user/max_user_namespaces` or
equivalent at boot time has been the recommended mitigation for
several Linux LPE vulnerabilities over the past few years.  Many users
have applied this mitigation and then forgotten that they have done
so.
2022-07-16 19:30:53 -07:00
Adam Joseph
8ea3a911aa local-derivation-goal.cc: improve error messages when sandboxing fails
The failure modes for nix's sandboxing setup are pretty complicated.
When nix is unable to set up the sandbox, let's provide more detail
about what went wrong.  Specifically:

* Make sure the error message includes the word "sandbox" so the user
  knows that the failure was related to sandboxing.

* If `--option sandbox-fallback false` was provided, and removing it
  would have allowed further attempts to make progress, let the user
  know.
2022-07-16 14:56:24 -07:00
Alex Wied
b88fb50e21 fix(libstore): allow Nix to access all Rosetta 2 paths on MacOS
Fixes: #5884
2022-07-15 12:10:56 -07:00
John Ericson
12461e246b Leverage existing docs for new store-path^outputs syntax 2022-07-15 14:04:22 +00:00
John Ericson
0e4ec98ae8
Fix typo in docs
Thanks!

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2022-07-15 09:49:23 -04:00
John Ericson
279ecf7cde Remove computed-derivations experimental feature
We don't need it yet.
2022-07-15 13:42:17 +00:00
Eelco Dolstra
59764eb842
Merge pull request #6810 from jfly/jfly/do-not-assume-savedvars-exist
nix develop: do not assume that saved vars are set
2022-07-15 13:59:25 +02:00
Eelco Dolstra
3bcd7a5474 Disable auto-chroot if $NIX_STATE_DIR is set
Issue #6732.
2022-07-15 12:32:29 +02:00
Jeremy Fleischman
04386f7d69
nix develop: do not assume that saved vars are set
This fixes https://github.com/NixOS/nix/issues/6809
2022-07-14 23:25:39 -07:00
John Ericson
8735f55dec Fix bug, test more, document more 2022-07-14 20:23:43 -04:00
John Ericson
f3262bc216 Combine InstallableStorePath with InstallableIndexedStorePath
No behavior should be changed, the `isDerivation` logic is moved from
the methods to the constructor.
2022-07-14 19:37:55 -04:00
John Ericson
6cafe308c9 Merge remote-tracking branch 'upstream/master' into indexed-store-path-outputs 2022-07-14 16:15:37 -04:00
Eelco Dolstra
ff49c75502 Disable auto-chroot if $NIX_STORE_DIR is set
Fixes #6732.
2022-07-14 17:47:09 +02:00
Eelco Dolstra
12df8885cc Simplify the check for overrides on non-existent inputs 2022-07-13 13:40:40 +02:00
Ikko Ashimine
694a9dc282
Fix typo in flake.cc
non-existant -> non-existent
2022-07-13 01:10:32 +09:00
Théophane Hufschmitt
2dbd5ed0b4
Merge pull request #6663 from Ma27/follows-invalid-input
flakes: throw an error if `follows`-declaration for an input is invalid
2022-07-12 16:44:22 +02:00
Eelco Dolstra
f6a434c8a4 Fix debug message 2022-07-12 11:53:34 +02:00
Maximilian Bosch
1f771065f1
Move follows-check into its own function 2022-07-12 11:25:33 +02:00
Maximilian Bosch
411111a3bc
Turn error for non-existant follows into a warning 2022-07-12 11:22:35 +02:00
Maximilian Bosch
c1c37f3200
flakes: throw an error if follows-declaration for an input is invalid
I recently got fairly confused why the following expression didn't have
any effect

    {
      description = "Foobar";
      inputs.sops-nix = {
        url = github:mic92/sops-nix;
        inputs.nixpkgs_22_05.follows = "nixpkgs";
      };
    }

until I found out that the input was called `nixpkgs-22_05` (please note
the dash vs. underscore).

IMHO it's not a good idea to not throw an error in that case and
probably leave end-users rather confused, so I implemented a small check
for that which basically checks whether `follows`-declaration from
overrides actually have corresponding inputs in the transitive flake.

In fact this was done by accident already in our own test-suite where
the removal of a `follows` was apparently forgotten[1].

Since the key of the `std::map` that holds the `overrides` is a vector
and we have to find the last element of each vector (i.e. the override)
this has to be done with a for loop in O(n) complexity with `n` being
the total amount of overrides (which shouldn't be that large though).

Please note that this doesn't work with nested expressions, i.e.

    inputs.fenix.inputs.nixpkgs.follows = "...";

which is a known problem[2].

For the expression demonstrated above, an error like this will be
thrown:

    error: sops-nix has a `follows'-declaration for a non-existant input nixpkgs_22_05!

[1] 2664a216e5
[2] https://github.com/NixOS/nix/issues/5790
2022-07-12 10:18:38 +02:00
Théophane Hufschmitt
f071eb3702
Merge pull request #6693 from ncfavier/complete-flake-inputs
Improve shell completion of flake inputs
2022-07-12 09:12:00 +02:00
Ben Burdette
cdcc349072
Merge branch 'master' into ignore-try 2022-07-11 11:29:22 -06:00
Ben Burdette
a3629ab0cc move ignore-try to EvalSettings 2022-07-11 10:47:09 -06:00
Ben Burdette
6ac8200ff5 use util.hh class instead of local 2022-07-11 10:21:40 -06:00
Ben Burdette
69ea265fd2 'tryEval' not 'try clause' 2022-07-11 10:21:12 -06:00
Naïm Favier
711b2e1f48
Fix flake input completion for InstallablesCommands
Defers completion of flake inputs until the whole command line is parsed
so that we know what flakes we need to complete the inputs of.

Previously, `nix build flake --update-input <Tab>` always behaved like
`nix build . --update-input <Tab>`.
2022-07-11 15:13:51 +02:00
Gytis Ivaskevicius
b2703c73a4 builtins.traceVerbose: Post rebase fixes 2022-07-05 19:56:39 +03:00
Gytis Ivaskevicius
ba1fe85b65 Add builtins.traceVerbose
Co-Authored-By: Silvan Mosberger <contact@infinisil.com>

Add builtins.traceVerbose tests
2022-07-05 19:44:26 +03:00
Théophane Hufschmitt
d63cd77549
Merge pull request #6664 from Ma27/innixshell-backwards-compat
nix-shell: restore backwards-compat with old nixpkgs
2022-07-05 15:57:20 +02:00
Théophane Hufschmitt
b7eb4ac169
Merge pull request #6699 from tennox/better-flake-new-error-message
flakes: apply templates partially on conflicts
2022-06-29 18:21:07 +02:00
Théophane Hufschmitt
4c8210095e
Merge pull request #6233 from flox/nix-repl-flakes
Nix repl flakes
2022-06-29 17:59:22 +02:00
Théophane Hufschmitt
3b18058969
Merge pull request #6674 from virusdave/dnicponski/scratch/dont_capture_stdout_in_nix_repl_master
Don't capture `stdout` when launching subshells in `nix repl`
2022-06-29 17:54:41 +02:00
Eelco Dolstra
4248174e71
Merge pull request #6702 from ncfavier/develop-prepend-vars
nix develop: save XDG_DATA_DIRS for loadable completion
2022-06-29 14:49:31 +02:00
Théophane Hufschmitt
f8fea9075c Merge remote-tracking branch 'origin/master' into better-flake-new-error-message 2022-06-29 13:44:14 +02:00
Eelco Dolstra
7633764342
Merge pull request #6739 from edolstra/ignore-chroot-error
Don't fail if we can't create ~/.local/share/nix/root
2022-06-29 13:06:35 +02:00
Eelco Dolstra
6cab528461 Don't fail if we can't create ~/.local/share/nix/root
https://hydra.nixos.org/build/182135943
2022-06-29 12:16:51 +02:00
Eelco Dolstra
455177cbe0 src/libexpr/tests/primops.cc: Quote Nix expressions
Otherwise they don't survive reformatting, see the failure in
https://github.com/NixOS/nix/pull/6721.
2022-06-29 11:37:25 +02:00
Eelco Dolstra
3c1814423b
Merge pull request #6737 from centromere/fix-typo
doc: Fix typo
2022-06-29 11:02:46 +02:00
Théophane Hufschmitt
56cf96a1b9
Merge pull request #6706 from lheckemann/cache-info-cache-invalidation
nar-info-disk-cache: refresh nix-cache-info weekly
2022-06-29 07:56:27 +02:00
Alex Wied
cd361b31fa doc: Fix typo 2022-06-28 22:43:37 -04:00
Manu [tennox]
ae4c9ef8e2 #6699 flake init: fix trying to add unchanged file
After skipping because of being of identical content it tried to git add it.
2022-06-26 21:35:21 +01:00
Manu [tennox]
4374e3ec67 #6699 flake init: Apply suggestions of @thufschmitt 2022-06-26 18:12:30 +01:00
Manuel
58cbbdc5e7
Update src/nix/flake.cc
https://github.com/NixOS/nix/pull/6699#discussion_r904097147

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-06-26 18:00:57 +01:00
Manuel
117baee1b7
Update src/nix/flake.cc
https://github.com/NixOS/nix/pull/6699#discussion_r904096906

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-06-26 18:00:34 +01:00
Guillaume Girol
e8109cf405 fetchGit: document shallow argument 2022-06-26 12:00:00 +00:00
Eelco Dolstra
49c4a37efc
Merge pull request #6717 from edolstra/move-flake-registry
Fetch flake-registry.json from channels.nixos.org
2022-06-25 21:17:57 +02:00
Eelco Dolstra
586fa707fc
Merge pull request #6714 from edolstra/auto-chroot-store
Automatically use a chroot store if /nix doesn't exist
2022-06-25 00:03:35 +02:00
Eelco Dolstra
30d4aa5dd6 Only do the auto chroot store on Linux 2022-06-24 23:35:21 +02:00
Théophane Hufschmitt
f06b6fa4fa
Merge pull request #6698 from tweag/forbid-tilde-paths-in-pure-mode
Forbid the tilde expansion in pure eval mode
2022-06-24 10:02:40 +02:00
Dave Nicponski
c6f7726f48 Don't capture stdout when launching subshells in nix repl 2022-06-23 22:45:13 -04:00
Eelco Dolstra
2ef6785d8e
Merge pull request #6716 from Mindavi/bugfix/cast
eval-cache: cast rowId to correct type
2022-06-24 00:28:38 +02:00
Eelco Dolstra
4b6cc3da62 Fetch flake-registry.json from channels.nixos.org
Using fastly is slightly faster, provides some resilience due to a
high stale TTL, and allows some usage metrics.
2022-06-23 23:56:26 +02:00
Rick van Schijndel
2beb929753 eval-cache: cast rowId to correct type
Prevents errors when running with UBSan:

/nix/store/j5vhrywqmz1ixwhsmmjjxa85fpwryzh0-gcc-11.3.0/include/c++/11.3.0/bits/stl_pair.h:353:4: runtime error: load of value 229, which is not a valid value for type 'AttrType'
2022-06-23 21:11:08 +02:00
Linus Heckemann
8cf26385cd [fixup] handle cache expiration in sqlite query 2022-06-23 14:54:25 -04:00
Cole Helbling
561a258f1d libstore/nar-info: drop unused system field
This was unused everywhere (and even the official NixOS binary cache
did not produce .narinfo files containing a "System:" field).
2022-06-23 14:25:10 -04:00
Eelco Dolstra
1cb376d60e
Fix typo
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2022-06-23 17:18:22 +02:00
Eelco Dolstra
2a9fddc0b1 Automatically use a chroot store if /nix doesn't exist
Specifically, if we're not root and the daemon socket does not exist,
then we use ~/.local/share/nix/root as a chroot store. This enables
non-root users to download nix-static and have it work out of the box,
e.g.

  ubuntu@ip-10-13-1-146:~$ ~/nix run nixpkgs#hello
  warning: '/nix' does not exists, so Nix will use '/home/ubuntu/.local/share/nix/root' as a chroot store
  Hello, world!
2022-06-23 16:29:50 +02:00
Eelco Dolstra
3c57db1a0f
Merge pull request #6710 from edolstra/embedded-sandbox-shell
Embed the sandbox shell into the statically linked 'nix' binary
2022-06-23 15:34:16 +02:00
Eelco Dolstra
0b2ea0023c Fix typo 2022-06-23 14:22:11 +02:00
Théophane Hufschmitt
027f6a735f
Merge pull request #6673 from asymmetric/warn
libstore: improve warning message on missing sig
2022-06-23 06:42:48 +02:00
Eelco Dolstra
925b975224 Embed the sandbox shell into the statically linked 'nix' binary
With this, Nix will write a copy of the sandbox shell to /bin/sh in
the sandbox rather than bind-mounting it from the host filesystem.
This makes /bin/sh work out of the box with nix-static, i.e. you no
longer get

  /nix/store/qa36xhc5gpf42l3z1a8m1lysi40l9p7s-bootstrap-stage4-stdenv-linux/setup: ./configure: /bin/sh: bad interpreter: No such file or directory
2022-06-23 04:08:28 +02:00
Eelco Dolstra
1e55ee2961 getSelfExe(): Support macOS 2022-06-23 01:32:46 +02:00
Eelco Dolstra
184f4e40de Remove NIX_LIBEXEC_DIR 2022-06-23 01:32:46 +02:00
Eelco Dolstra
d3176ce076 Fix build-remote in nix-static
'build-remote' is now executed via /proc/self/exe so it always works.
2022-06-23 01:32:46 +02:00
Naïm Favier
155c57c171
nix develop: save XDG_DATA_DIRS for loadable completion 2022-06-23 01:11:33 +02:00
Linus Heckemann
d533a88546 nar-info-disk-cache: refresh nix-cache-info weekly
This allows changes to nix-cache-info to be picked up by existing
clients. Previously, the only way for this to happen would be for
clients to delete binary-cache-v6.sqlite, which is quite awkward for
users.

On the other hand, updates to nix-cache-info should be pretty rare,
hence the choice of a fairly long TTL. Configurability is probably not
useful enough to warrant implementing it.
2022-06-22 10:49:18 -04:00
Eelco Dolstra
f6cf644e5f Style 2022-06-22 15:35:52 +02:00
Manu [tennox]
3a85fd077c #6542 Apply flake templates partially on conflicts
Will still exit with non-zero exit code and clearly prompt which files to merge:
 ```
nixx flake init -t github:numtide/devshell
wrote: /home/manu/dev/stuff/gopassbridge/.envrc
refusing to overwrite existing file '/home/manu/dev/stuff/gopassbridge/.gitignore'
-> merge manually with '/nix/store/ksmwhyghjwb4d9dw6hcpbvng1msdvjim-source/template/.gitignore'
wrote: /home/manu/dev/stuff/gopassbridge/devshell.toml
wrote: /home/manu/dev/stuff/gopassbridge/flake.nix
error: Encountered 1 conflicts - please merge manually
 ```
2022-06-21 17:37:02 +01:00
Eelco Dolstra
bbe7187db9
Merge pull request #6695 from afishhh/search-fix-exclude
Fix arity of `--exclude` flag in `nix search`
2022-06-21 16:12:04 +02:00
Théophane Hufschmitt
983efdbde4 Forbid the tilde expansion in pure eval mode
Fix #6684
2022-06-21 14:08:18 +02:00
Théophane Hufschmitt
16c6c6c963
Merge pull request #6676 from virusdave/dnicponski/scratch/swap_homedir_check_master
Verify this if `$HOME` exists, it is owned by current user in `getHome()`
2022-06-20 21:17:22 +02:00
Fishhh
4ade8a5f25
Fix arity of --exclude flag in nix search
Due to incorrectly using the Handler(vector<string>*) constructor the
`--exclude` flag would swallow all proceeding arguments instead of just
one.
2022-06-20 18:00:32 +02:00
Naïm Favier
d6d0e781bb
Complete flake inputs for all given flakes
Allow `nix build flake1 flake2 --update-input <Tab>` to complete the
inputs of both flakes.

Also do tilde expansion so that `nix build ~/flake --update-input <Tab>`
works.
2022-06-20 04:17:13 +02:00
Dave Nicponski
ca2be509b9 Verify $HOME is owned by current user in getHome(), if it exists.
Useful because a default `sudo` on darwin doesn't clear `$HOME`, so things like `sudo nix-channel --list`
will surprisingly return the USER'S channels, rather than `root`'s.

Other counterintuitive outcomes can be seen in this PR description:
  https://github.com/NixOS/nix/pull/6622
2022-06-17 17:42:08 -04:00
Théophane Hufschmitt
7e301fd74e
Merge pull request #6620 from maralorn/fix-print-missing
respect print-missing variable in new-style build command
2022-06-16 13:23:15 +02:00
Lorenzo Manacorda
475249db8a libstore: improve warning message on missing sig
Clarifies that the substitute will be ignored/skipped.
2022-06-15 17:34:28 +02:00
Tom Bereknyei
51268ceb79 Merge branch 'master' into nix-repl-flakes 2022-06-15 09:17:03 -04:00
tomberek
dae4a8a6c8
Apply suggestions from code review
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-06-15 09:02:36 -04:00
Maximilian Bosch
98946e2d9c
nix-shell: restore backwards-compat with old nixpkgs
Basically an attempt to resume fixing #5543 for a breakage introduced
earlier[1]. Basically, when evaluating an older `nixpkgs` with
`nix-shell` the following error occurs:

    λ ma27 [~] → nix-shell -I nixpkgs=channel:nixos-18.03 -p nix
    error: anonymous function at /nix/store/zakqwc529rb6xcj8pwixjsxscvlx9fbi-source/pkgs/top-level/default.nix:20:1 called with unexpected argument 'inNixShell'

           at /nix/store/zakqwc529rb6xcj8pwixjsxscvlx9fbi-source/pkgs/top-level/impure.nix:82:1:

               81|
               82| import ./. (builtins.removeAttrs args [ "system" "platform" ] // {
                 | ^
               83|   inherit config overlays crossSystem;

This is a problem because one of the main selling points of Nix is that
you can evaluate any old Nix expression and still get the same result
(which also means that it *still evaluates*). In fact we're deprecating,
but not removing a lot of stuff for that reason such as unquoted URLs[2]
or `builtins.toPath`. However this property was essentially thrown away
here.

The change is rather simple: check if `inNixShell` is specified in the
formals of an auto-called function. This means that

    { inNixShell ? false }:
    builtins.trace inNixShell
      (with import <nixpkgs> { }; makeShell { name = "foo"; })

will show `trace: true` while

    args@{ ... }:
    builtins.trace args.inNixShell
      (with import <nixpkgs> { }; makeShell { name = "foo"; })

will throw the following error:

    error: attribute 'inNixShell' missing

This is explicitly needed because the function in
`pkgs/top-level/impure.nix` of e.g. NixOS 18.03 has an ellipsis[3], but
passes the attribute-set on to another lambda with formals that doesn't
have an ellipsis anymore (hence the error from above). This was perhaps
a mistake, but we can't fix it anymore. This also means that there's
AFAICS no proper way to check if the attr-set that's passed to the Nix
code via `EvalState::autoCallFunction` is eventually passed to a lambda
with formals where `inNixShell` is missing.

However, this fix comes with a certain price. Essentially every
`shell.nix` that assumes `inNixShell` to be passed to the formals even
without explicitly specifying it would break with this[4]. However I think
that this is ugly, but preferable:

* Nix 2.3 was declared stable by NixOS up until recently (well, it still
  is as long as 21.11 is alive), so most people might not have even
  noticed that feature.

* We're talking about a way shorter time-span with this change being
  in the wild, so the fallout should be smaller IMHO.

[1] 9d612c393a
[2] https://github.com/NixOS/rfcs/pull/45#issuecomment-488232537
[3] https://github.com/NixOS/nixpkgs/blob/release-18.03/pkgs/top-level/impure.nix#L75
[4] See e.g. the second expression in this commit-message or the changes
    for `tests/ca/nix-shell.sh`.
2022-06-13 23:29:28 +02:00
Alexander Bantyev
d82a3dc70d
flake.cc: Make non-flake overrides sticky
Overrides for inputs with flake=false were non-sticky, since they
changed the `original` in `flake.lock`. This fixes it, by using the same
locked original for both flake and non-flake inputs.
2022-06-13 20:49:16 +04:00
Eelco Dolstra
9f58df4c91
Merge pull request #6619 from Jonpez2/patch-1
Add security.csm to ignored-acls
2022-06-13 16:23:13 +02:00
Eelco Dolstra
798efdf10b
Merge pull request #6027 from Ma27/pure-repl
repl: `--option pure-eval true` actually enables pure eval mode
2022-06-13 16:11:46 +02:00
Eelco Dolstra
2fef24f528
Merge pull request #6634 from lovesegfault/fix-getgrouplist
fix(libstore/lock): support users that belong to more than 10 groups
2022-06-13 15:45:09 +02:00
Eelco Dolstra
95eccfd5db
Merge pull request #6648 from andersk/missing-rethrow
Add missing rethrows in conditional exception handlers
2022-06-13 15:37:38 +02:00
Théophane Hufschmitt
03226aa053
Merge pull request #6651 from ncfavier/nix-build-stop-logger
nix-build: stop logger when appropriate
2022-06-13 10:43:14 +02:00
Gabriel Fontes
9f6b4639c2
fix sourcehut brach/tag resolving regression
nixos/nix#6290 introduced a regex pattern to account for tags when
resolving sourcehut refs. nixos/nix#4638 reafactored the code,
accidentally treating the pattern as a regular string, causing all
non-HEAD ref resolving to break.

This fixes the regression and adds more test cases to avoid future
breakage.
2022-06-11 16:52:20 -03:00
Naïm Favier
502d7d9092
nix-build: stop logger when appropriate
Reverts b944b588fa in `nix-build.cc`.
2022-06-11 15:17:35 +02:00
Ben Burdette
49ff4ef637 remove unused parameter 2022-06-10 12:22:36 -06:00
Anders Kaseorg
754cd53faf Add missing rethrows in conditional exception handlers
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2022-06-10 10:52:41 -07:00
Yuriy Taraday
65d09fce22 Mention that -f implies --impure for eval in docs
Right now this is not mentioned anywhere and it is unexpected.
2022-06-10 19:16:17 +04:00
Théophane Hufschmitt
45ebaab665
Merge pull request #6643 from ncfavier/fix-git-again
libfetchers/git: add missing `--git-dir` flags
2022-06-10 14:33:32 +02:00
Naïm Favier
da8f8668ca
libfetchers/git: add missing --git-dir flags 2022-06-10 12:57:13 +02:00
Sidharth Kshatriya
7868405d58 nix-env: A small std::move() optimization
Avoids doing a O(n) copy of Strings i.e. std::list<std::string>
2022-06-09 20:01:01 +05:30
Sidharth Kshatriya
3efea3d28a nix-store: small std::move() optimization 2022-06-09 16:25:26 +05:30
Bernardo Meurer
931930feb1 fix(libstore/lock): support users that belong to more than 10 groups
The manpage for `getgrouplist` says:

> If the number of groups of which user is a member is less than or
> equal to *ngroups, then the value *ngroups is returned.
>
> If the user is a member of more than *ngroups groups, then
> getgrouplist() returns -1.  In this case, the value returned in
> *ngroups can be used to resize the buffer passed to a further
> call getgrouplist().

In our original code, however, we allocated a list of size `10` and, if
`getgrouplist` returned `-1` threw an exception. In practice, this
caused the code to fail for any user belonging to more than 10 groups.

While unusual for single-user systems, large companies commonly have a
huge number of POSIX groups users belong to, causing this issue to crop
up and make multi-user Nix unusable in such settings.

The fix is relatively simple, when `getgrouplist` fails, it stores the
real number of GIDs in `ngroups`, so we must resize our list and retry.
Only then, if it errors once more, we can raise an exception.

This should be backported to, at least, 2.9.x.
2022-06-08 13:45:41 -04:00
Théophane Hufschmitt
480c2b6699 Rewrite the CA paths when moving them between store
Bring back the possibility to copy CA paths with no reference (like the
outputs of FO derivations or stuff imported at eval time) between stores
that have a different prefix.
2022-06-08 15:13:11 +02:00
Théophane Hufschmitt
cb0553ecd0 Restore the "low-latency" ssh copying 2022-06-08 14:03:46 +02:00
Lorenzo Manacorda
814ddfa5f5 Fix missing ` in key manual 2022-06-08 11:46:50 +02:00
Jonpez2
a7d25d339d
Add security.csm to the default ignore list 2022-06-08 09:32:14 +01:00
Eelco Dolstra
b2dea231cf
Merge pull request #6618 from afishhh/search-exclude
Add `-e`/`--exclude` flag to `nix search`
2022-06-07 19:01:36 +02:00
Fishhh
e009367c8d
Remove redundant std::moves in calls to hiliteMatches 2022-06-07 18:25:52 +02:00
Fishhh
b42358b9be
Add --exclude flag to nix search
If a package's attribute path, description or name contains matches for any of the
regexes specified via `-e` or `--exclude` that package is excluded from
the final output.
2022-06-07 18:25:48 +02:00
Eelco Dolstra
739c358c68
Merge pull request #6602 from cw789/patch-2
Explain exactly what nix-upgrade nix does
2022-06-07 13:08:56 +02:00
Malte Brandy
bf2f25e3d8 respect print-missing variable in new-style build command
Currently nix-build prints the "printMissing" information by default,
nix build doesn’t.
People generally don‘t notice this because the standard log-format of
nix build would not display the printMissing
output long enough to perceive the information.

This addresses https://github.com/NixOS/nix/issues/6561
2022-06-06 21:15:31 +02:00
Fishhh
d137ceccef
Fix incorrect comment in hiliteMatches 2022-06-05 20:30:18 +02:00
Théophane Hufschmitt
95f47c28fb Make nix copy parallel again
FILLME
2022-06-03 17:01:16 +02:00
Eelco Dolstra
2097c30b08
Merge pull request #6600 from asymmetric/hash-comment
schema.sql: add comment about hash being in base16
2022-06-03 08:51:07 +02:00
Tom Bereknyei
ffd41d1757 Merge branch 'master' into nix-repl-flakes 2022-06-02 16:58:35 -04:00
John Ericson
b585548dfe Merge remote-tracking branch 'upstream/master' into indexed-store-path-outputs 2022-06-02 19:53:11 +00:00
Eelco Dolstra
28e08822a3
Avoid unnecessary string copy 2022-06-02 21:26:28 +02:00
Eelco Dolstra
abb80cfa4c
Add operator for concatenating strings and string_views 2022-06-02 21:26:21 +02:00
Eelco Dolstra
81a486c607
Shut up clang warnings 2022-06-02 21:19:54 +02:00
Ben Burdette
8cf6ae8664 use Counter class to count tryEval levels 2022-06-02 12:29:38 -06:00
Ben Burdette
bc0d41e9ba print message with exceptions in a try clause 2022-06-02 12:17:28 -06:00
Ben Burdette
9151dbff88 ignore-try flag 2022-06-02 10:26:46 -06:00
Chris Wögi
51b3cc151e Explain exactly what nix-upgrade nix does 2022-06-02 15:05:27 +02:00
Théophane Hufschmitt
a5f7b934bd
Merge branch 'master' into lto 2022-06-01 17:55:03 +02:00
Lorenzo Manacorda
a9358a6097 schema.sql: add comment about hash being in base16 2022-06-01 14:59:57 +02:00
Maximilian Bosch
159b5815b5
repl: --option pure-eval true actually enables pure eval mode
To quote Eelco in #5867:

> Unfortunately we can't do
>
>     evalSettings.pureEval.setDefault(false);
>
> because then we have to do the same in main.cc (where
> pureEval is set to true), and that would allow pure-eval
> to be disabled globally from nix.conf.

Instead, a command should specify that it should be impure by
default. Then, `evalSettings.pureEval` will be set to `false;` unless
it's overridden by e.g. a CLI flag.

In that case it's IMHO OK to be (theoretically) able to override
`pure-eval` via `nix.conf` because it doesn't have an effect on commands
where `forceImpureByDefault` returns `false` (i.e. everything where pure
eval actually matters).

Closes #5867
2022-05-31 22:20:11 +02:00
Eelco Dolstra
078c80f750
Merge pull request #6582 from NixOS/debug-fetchgit-sigsev
Fix a segfault in the git fetcher
2022-05-31 17:17:37 +02:00
Eelco Dolstra
af23d38019
Merge pull request #6590 from edolstra/fix-noexcept
Fix noexcept violations
2022-05-30 17:45:44 +02:00
Eelco Dolstra
6378f0bb32
RemoteStore::queryRealisationUncached(): Fix potential noexcept violation 2022-05-30 13:27:13 +02:00
Eelco Dolstra
b8faa83742
HttpBinaryCacheStore::getFile(): Don't throw an exception
This violates the noexcept specification.

Fixes #6445.
2022-05-30 13:24:04 +02:00
Eelco Dolstra
8e8e9d8705
Respect the outputSpecified attribute
E.g. 'nix build nixpkgs#libxml2.dev' will build the 'dev' output.
2022-05-30 11:34:47 +02:00
Théophane Hufschmitt
027fd45230 Fix a segfault in the git fetcher
The git fetcher code used to dereference the (potentially empty) `ref`
input attribute. This was magically working, probably because the
compiler somehow outsmarted us, but is now blowing up with newer nixpkgs
versions.

Fix that by not trying to access this field while we don't know for sure
that it has been defined.

Fix #6554
2022-05-27 16:15:28 +02:00
Eelco Dolstra
ec07a70979
Style fix 2022-05-27 11:25:05 +02:00
Sergei Trofimovich
b36d5172cb src/libutil/json.cc: add missing <cstdint> include for gcc-13
Without the change llvm build fails on this week's gcc-13 snapshot as:

    src/libutil/json.cc: In function 'void nix::toJSON(std::ostream&, const char*, const char*)':
    src/libutil/json.cc:33:22: error: 'uint16_t' was not declared in this scope
       33 |             put(hex[(uint16_t(*i) >> 12) & 0xf]);
          |                      ^~~~~~~~
    src/libutil/json.cc:5:1: note: 'uint16_t' is defined in header '<cstdint>'; did you forget to '#include <cstdint>'?
        4 | #include <cstring>
      +++ |+#include <cstdint>
        5 |
2022-05-26 18:37:15 +01:00
Eelco Dolstra
1e67582d75
Merge pull request #6576 from hercules-ci/daemon-increase-socket-backlog
Fix `Connection refused` on daemon socket
2022-05-26 12:43:45 +02:00
Eelco Dolstra
9acc770ce4
Remove pre-C++11 hackiness 2022-05-26 12:40:01 +02:00
Robert Hensing
c156155239 createUnixDomainSocket: listen(unix, 5 -> 100)
This solves the error

    error: cannot connect to socket at '/nix/var/nix/daemon-socket/socket': Connection refused

on build farm systems that are loaded but operating normally.

I've seen this happen on an M1 mac running a loaded hercules-ci-agent.
Hercules CI uses multiple worker processes, which may connect to
the Nix daemon around the same time. It's not unthinkable that
the Nix daemon listening process isn't scheduled until after 6
workers try to connect, especially on a system under load with
many workers.

Is the increase safe?

The number is the number of connections that the kernel will buffer
while the listening process hasn't `accept`-ed them yet.
It did not - and will not - restrict the total number of daemon
forks that a client can create.

History

The number 5 has remained unchanged since the introduction in
nix-worker with 0130ef88ea in 2006.
2022-05-26 11:24:04 +02:00
Eelco Dolstra
762fa2b2ff
Merge pull request #6570 from edolstra/eof-in-string
Handle EOFs in string literals correctly
2022-05-26 08:13:39 +02:00
Ben Burdette
9068d32e12 remove parens from repl help 2022-05-25 12:55:58 -06:00
Ben Burdette
9a5ea6c359
Merge branch 'master' into debug-exploratory-PR 2022-05-25 10:41:10 -06:00
Ben Burdette
b4c24a29c6 back to ref<EvalState> in NixRepl 2022-05-25 10:21:20 -06:00
Eelco Dolstra
27ebb97d0a
Handle EOFs in string literals correctly
We can't return a STR token without setting a valid StringToken,
otherwise the parser will crash.

Fixes #6562.
2022-05-25 17:58:13 +02:00
Eelco Dolstra
d8398d33c9
Typo 2022-05-25 15:29:27 +02:00
Eelco Dolstra
89a8955e79
Merge remote-tracking branch 'origin/file-fetcher' 2022-05-25 15:15:16 +02:00
Eelco Dolstra
2f8a34cddc
Fix warning 2022-05-25 15:05:39 +02:00
pennae
b092afe77d
Merge branch 'master' into lto 2022-05-25 11:55:13 +00:00
Eelco Dolstra
91b7d5373a
Style tweaks 2022-05-25 12:32:22 +02:00
Maximilian Bosch
b916c08feb
libfetchers: drop getGitDir and hardcode .git
As discussed[1] this is most likely not desirable.

[1] https://github.com/NixOS/nix/pull/6440#issuecomment-1120876248
2022-05-24 14:20:48 +02:00
Ben Burdette
ba035f7dd0 comment 2022-05-23 10:13:47 -06:00
Théophane Hufschmitt
1e7c27cb91
Merge pull request #6560 from ncfavier/patch-1
typo: defaultApps → defaultApp
2022-05-23 17:08:06 +02:00
Théophane Hufschmitt
50ab1587bf
Merge pull request #6532 from flox/show_warning
fix: alignment during flake show of legacyPackages
2022-05-23 09:30:26 +02:00
Ben Burdette
13d02af079 remove redundant 'debugMode' flag 2022-05-22 21:45:24 -06:00
Ben Burdette
7ccb2700c0 comments 2022-05-22 19:15:58 -06:00
Ben Burdette
5bc97fdfa6 change state derefs 2022-05-22 19:12:03 -06:00
Ben Burdette
34ffaa9f57 changning repl to use EvalState& instead of ref 2022-05-22 18:57:45 -06:00
Naïm Camille Favier
81a9bf0ad2
typo: defaultApps → defaultApp 2022-05-21 14:41:24 +02:00
Ben Burdette
982c8a959b remove special tryEval behavior 2022-05-20 12:45:36 -06:00
Ben Burdette
884d591787 debugRepl ftn pointer 2022-05-20 10:33:50 -06:00
Tom Bereknyei
7a04fb1c56 repl: add repl-flake experimental feature for gating 2022-05-20 08:20:00 -04:00
tomberek
542e36c6e7
Apply suggestions from code review
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2022-05-20 01:48:24 -04:00
Tom Bereknyei
db613a85fb repl: allow --file to always utilize autoargs 2022-05-20 01:35:06 -04:00
tomberek
7d7e00272a
Apply suggestions from code review
Style fixes from @edolstra

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2022-05-20 01:28:20 -04:00
Ben Burdette
0600df86b8 'debugMode' 2022-05-19 17:01:23 -06:00
Ben Burdette
7ddef73d02 de-const evalState exceptions 2022-05-19 12:44:40 -06:00
Ben Burdette
f9cdb6af8d Merge branch 'debug-exploratory-PR' into debuggerHook-eval-arg 2022-05-19 11:07:18 -06:00
Ben Burdette
357fb84dba use an expr->StaticEnv table in evalState 2022-05-19 10:48:10 -06:00
Tony Olagbaiye
5b8c1deb18 fetchTree: Allow fetching plain files
Add a new `file` fetcher type, which will fetch a plain file over
http(s), or from the local file.

Because plain `http(s)://` or `file://` urls can already correspond to
`tarball` inputs (if the path ends-up with a know archive extension),
the URL parsing logic is a bit convuluted in that:

- {http,https,file}:// urls will be interpreted as either a tarball or a
  file input, depending on the extensions of the path part (so
  `https://foo.com/bar` will be a `file` input and
  `https://foo.com/bar.tar.gz` as a `tarball` input)
- `file+{something}://` urls will be interpreted as `file` urls (with
  the `file+` part removed)
- `tarball+{something}://` urls will be interpreted as `tarball` urls (with
  the `tarball+` part removed)

Fix #3785

Co-Authored-By: Tony Olagbaiye <me@fron.io>
2022-05-19 18:24:49 +02:00
Tom Bereknyei
f21dec5bef repl: hide flake behavior behind flag and provide warning 2022-05-19 01:02:24 -04:00
Tom Bereknyei
e1f308a1ec repl: provide backward compat with legacy usage 2022-05-18 22:28:15 -04:00
Tom Bereknyei
7534798eed refactor: factor out getValue 2022-05-18 21:36:50 -04:00
tomberek
9f8c1183fa Apply suggestions from code review
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-05-18 21:21:58 -04:00
Tom Bereknyei
1ca3f6035d repl: update docs with installables 2022-05-18 21:21:38 -04:00
Tom Bereknyei
5640b52834 repl: use installables 2022-05-18 21:20:59 -04:00
Tom Bereknyei
81567a0962 repl: allow loading installables from CLI
repl: search installable with findAlongAttrPath
repl: refactor handling of args
repl: temp
2022-05-18 21:20:59 -04:00
Timothy DeHerrera
06d57ce759 nix repl: load flakes from cli args
If experimental feature "flakes" is enabled, args passed to `nix repl`
will now be considered flake refs and imported using the existing
`:load-flake` machinery.

In addition, `:load-flake` now supports loading flake fragments.
2022-05-18 21:20:59 -04:00
Théophane Hufschmitt
452ffe5464
Hint at the source file on conflict in flake new
Add a pointer to the source file (from the template) when `nix flake new` (or `init`) encounters an already existing file

Fix #6542
2022-05-18 16:46:13 +02:00
Théophane Hufschmitt
78dc64ec1e
Merge pull request #6544 from ncfavier/getFlake-no-write-lock-file
Do not attempt to write a lock file in builtins.getFlake
2022-05-18 16:41:13 +02:00
Naïm Favier
169384abb2
Do not attempt to write a lock file in builtins.getFlake
Fixes https://github.com/NixOS/nix/issues/6541
2022-05-18 15:45:06 +02:00
zhujun
b8e44dc62b primop_match: fix example letter case in document 2022-05-18 14:05:26 +08:00
Ben Burdette
667074b586 first whack at passing evalState as an arg to debuggerHook. 2022-05-16 09:20:51 -06:00
Théophane Hufschmitt
43a2c13672 Make nix::eval_cache::int_t more idiomatic
Don’t explicitely give it a constructor, but use aggregate
initialization instead (also prevents having an implicit coertion, which
is probably good here)
2022-05-16 16:36:21 +02:00
Eli Kogan-Wang
e53349dd6e change priority conflict message 2022-05-16 16:16:06 +02:00
Eli Kogan-Wang
27d0f6747d resolve redundant priority passing, wrap NixInt in eval-cache variant 2022-05-16 15:17:35 +02:00
Eli Kogan-Wang
c81d24f1c7 Add int to eval-cache, bump eval cache schema version 2022-05-16 02:29:29 +02:00
elikoga
0cea59ab4b
Merge branch 'NixOS:master' into master 2022-05-16 01:01:37 +02:00
Ben Burdette
86ba0a702c fix thunk issue 2022-05-15 12:05:51 -06:00
Ben Burdette
6faa56ea1f remove extra argument 2022-05-15 12:05:34 -06:00
Eli Kogan-Wang
be2b19041e Integrate review changes 2022-05-13 22:02:28 +02:00
Tom Bereknyei
8150b93968 fix: alignment during flake show of legacyPackages
Fixes:
https://github.com/NixOS/nix/issues/6240
https://github.com/NixOS/nix/issues/6045
2022-05-13 11:12:11 -04:00
Eelco Dolstra
bf89cd95a4
Merge pull request #6525 from J-Swift/feature/bash-prefix
Add `bash-prompt-prefix` option
2022-05-13 14:06:07 +02:00
Ben Burdette
2acdb90438 remove debug code 2022-05-12 14:20:45 -06:00
Ben Burdette
2d0d1ec99d remove debug code 2022-05-12 14:15:35 -06:00
Ben Burdette
4f48095c66 Merge branch 'debugThrow' into debug-exploratory-PR 2022-05-12 14:11:35 -06:00
John Ericson
49ad315c03 Use ^ not ! in indexed store derivations installable syntax
Match the other syntax that was recently added
2022-05-12 20:10:02 +00:00
Ben Burdette
1ea13084c9 template-ize debugThrow 2022-05-12 13:59:58 -06:00
John Ericson
b18720ee17 Merge remote-tracking branch 'upstream/master' into indexed-store-path-outputs 2022-05-12 19:13:33 +00:00
Norbert Melzer
831e2743ea fix GitHub URL template 2022-05-12 00:56:39 +02:00
Théophane Hufschmitt
b944b588fa
Merge pull request #6523 from ncfavier/stop-logger-legacy
Stop the logger properly in legacy commands
2022-05-11 16:53:57 +02:00
Naïm Favier
1461e6cdda
Stop the logger properly in legacy commands
Ensures the logger is stopped on exit in legacy commands. Without this,
when using `nix-build --log-format bar` and stopping nix with CTRL+C,
the bar is not cleared from the screen.
2022-05-11 12:58:45 +02:00
Eli Kogan-Wang
aefc6c4f41 Add priority for nix profile install 2022-05-11 12:16:35 +02:00
Eelco Dolstra
54457382f9
Fix static build
https://hydra.nixos.org/build/176211267
2022-05-11 11:36:56 +02:00
Jimmy Reichley
584475acf9
Add documentation for bash-prompt-prefix 2022-05-10 16:55:25 -04:00
Jimmy Reichley
2998527b18
Allow setting bash-prompt-prefix nix develop configuration 2022-05-10 16:53:22 -04:00
Eelco Dolstra
eb957ad6d8
Merge pull request #6497 from danielfullmer/ghe-fetcher-url
Use correct URL for GitHub Enterprise
2022-05-10 19:23:22 +02:00
Eelco Dolstra
a9cbc2857f nix develop: Find bin/bash in the bashInteractive outputs 2022-05-10 16:43:41 +02:00
Ben Burdette
7cd7c7c91a
Merge branch 'master' into debug-exploratory-PR 2022-05-09 09:30:44 -06:00
Jan Tojnar
59d9551c25
libexpr: Fix manual link in error message
It was changed to the old manual in 8895fa70a4
2022-05-08 18:59:00 +02:00
Daniel Fullmer
7a3d5b2ff0 Use correct URL for GitHub Enterprise
For GitHub Enterprise, the API is accessed through a slightly different
URL. See [1], where it says:

> Use http(s)://[hostname]/api/v3 to access the API for GitHub
> Enterprise Server.

Also tested working on a GHE instance.

[1] https://docs.github.com/en/enterprise-server@3.3/rest/guides/getting-started-with-the-rest-api
2022-05-06 13:13:11 -07:00
Andreas Rammhold
059ae7f6c4
Add unit tests for libexpr (#5377)
* libexpr: fix builtins.split example

The example was previously indicating that multiple whitespaces would be
collapsed into a single captured whitespace. That isn't true and was
likely a mistake when being documented initially.

* Fix segfault on unitilized list when printing value

Since lists are just chunks of memory the individual elements in the
list might be unitilized when a programming error happens within Nix.

In this case the values are null-initialized (at least with Boehm GC)
and we can avoid a nullptr deref when printing them.

I ran into this issue while ensuring that new expression tests would
show the actual value on an assertion failure.

This is unlikely to cause any runtime performance regressions as
printing values is not really in the hot path (unless the repl is the
primary use case).

* Add operator<< for ValueTypes

* Add libexpr tests

This introduces tests for libexpr that evalulate various trivial Nix
language expressions and primop invocations that should be good smoke
tests wheter or not the implementation is behaving as expected.
2022-05-06 18:05:27 +02:00
Ben Burdette
fc66f48812 debugError() 2022-05-06 09:09:49 -06:00
Ben Burdette
2c9fafdc9e trying debugThrow 2022-05-06 08:47:21 -06:00
Eelco Dolstra
b470218d9a renderMarkdownToTerminal(): Avoid line overflow
Lowdown doesn't respect '.cols' exactly (maybe because of the
whitespace in front of each line), so adjust .cols a bit.
2022-05-06 13:14:49 +02:00
Eelco Dolstra
33affa0a02
Merge pull request #6483 from NixOS/fix-sourcehut-ref-parsing
Fix the parsing of the sourcehut refs file
2022-05-06 12:24:55 +02:00
Ben Burdette
99d69ac23f fix repl bug 2022-05-05 21:23:03 -06:00
Ben Burdette
dea998b2f2 traceable_allocator 2022-05-05 20:26:10 -06:00
Ben Burdette
f400c5466d rename valmap 2022-05-05 15:43:23 -06:00
Ben Burdette
09fcfee925 don't print the 'break' argument 2022-05-05 15:34:59 -06:00
Ben Burdette
ce304d0154 rename debug commands to be more gdb-like; hide them except in debug mode 2022-05-05 15:24:57 -06:00
Ben Burdette
0ac121a094
Merge branch 'NixOS:master' into debug-exploratory-PR 2022-05-05 13:54:20 -06:00
Eelco Dolstra
58645a78ab builtins.break: Return argument when debugging is not enabled 2022-05-05 17:17:03 +02:00
Eelco Dolstra
dd8b91eebc Style fixes
In particular, use std::make_shared and enumerate(). Also renamed some
fields to fit naming conventions.
2022-05-05 17:17:03 +02:00
Alexander Shpilkin
b3ed32d0fd Add forgotten null check 2022-05-04 22:13:49 +03:00
Théophane Hufschmitt
e68676e6c8 Fix the parsing of the sourcehut refs file
Since a26be9f3b8, the same parser is used
to parse the result of sourcehut’s `HEAD` endpoint (coming from [git
dumb protocol]) and the output of `git ls-remote`. However, they are very
slightly different (the former doesn’t specify the current reference
since it’s implied to be `HEAD`).

Unify both, and make the parser a bit more robust and understandable (by
making it more typed and adding tests for it)

[git dumb protocol]: https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols#_the_dumb_protocol
2022-05-04 14:38:59 +02:00
Eelco Dolstra
c98648bef0 Merge remote-tracking branch 'origin/master' into debug-exploratory-PR 2022-05-04 14:10:21 +02:00
Eelco Dolstra
107613ad2b Fix compiler warning 2022-05-04 11:31:39 +02:00
Eelco Dolstra
3e87c8e62b Move json stuff out of util.cc 2022-05-04 11:22:06 +02:00
Alain Zscheile
1385b20078
Get rid of most .at calls (#6393)
Use one of `get` or `getOr` instead which will either return a null-pointer (with a nicer error message) or a default value when the key is missing.
2022-05-04 07:44:32 +02:00
Eelco Dolstra
a3c6c5b1c7 nix profile: Support overriding outputs 2022-05-03 15:00:34 +02:00
Eelco Dolstra
4a79cba511 Allow selecting derivation outputs using 'installable!outputs'
E.g. 'nixpkgs#glibc^dev,static' or 'nixpkgs#glibc^*'.
2022-05-03 13:43:52 +02:00
Eelco Dolstra
404c222444
Merge pull request #6426 from edolstra/respect-outputs-to-install
nix: Respect meta.outputsToInstall, and use all outputs by default
2022-05-03 13:43:22 +02:00
Eelco Dolstra
61289ceee3 Style fixes 2022-05-02 13:37:53 +02:00
Eelco Dolstra
564faa6b4e
Merge pull request #6470 from Ma27/git-followup
libfetchers/git: fix every occasion of a permission error
2022-05-02 13:33:08 +02:00
Maximilian Bosch
1849e6a1f6
libfetchers/git: fix every occasion of a permission error
I'm afraid I missed a few problematic `git(1)`-calls while implementing
PR #6440, sorry for that! Upon investigating what went wrong, I realized
that I only tested against the "cached"-case by accident because my
git-checkout with my system's flake was apparently cached during my
debugging.

I managed to trigger the original issue again by running:

    $ git commit --allow-empty -m "tmp"
    $ sudo nixos-rebuild switch --flake .# -L --builders ''

Since `repoDir` points to the checkout that's potentially owned by
another user, I decided to add `--git-dir` to each call affecting
`repoDir`.

Since the `tmpDir` for the temporary submodule-checkout is created by
Nix itself, it doesn't seem to be an issue.

Sorry for that, it should be fine now.
2022-04-30 15:56:12 +02:00
Kjetil Orbekk
9bf296c970 Extract git reference parsing to a shared library
These utility functions can be shared between the git and github fetchers.
2022-04-29 18:46:21 -04:00
Kjetil Orbekk
c21afd684c Update nix flake documentation of ref handling
Update the documentation about how `ref` is resolved if it is not
specified.

Add a note about special handling of local workdirs with `git+file`.
2022-04-29 18:46:21 -04:00
Kjetil Orbekk
1203e48926 Store cached head in cached git repo
The previous head caching implementation stored two paths in the local
cache; one for the cached git repo and another textfile containing the
resolved HEAD ref. This commit instead stores the resolved HEAD by
setting the HEAD ref in the local cache appropriately.
2022-04-29 18:46:21 -04:00
Kjetil Orbekk
de54e1cd3f Refactor fetching of dirty workdir
Extract the handling of a local dirty workdir to a helper function.
2022-04-29 18:46:17 -04:00
Kjetil Orbekk
401e60f289 Resolve reference for remote repository
Resolves the HEAD reference from the remote repository instead
of assuming "master".
2022-04-29 18:42:28 -04:00
Ben Burdette
c81ffa692e remove 'libnix' 2022-04-29 11:35:50 -06:00
Ben Burdette
c941803861 spacing 2022-04-29 11:27:38 -06:00
Ben Burdette
172a83d22a line endings 2022-04-29 11:24:54 -06:00
Ben Burdette
ca6cba8b81 fix 'suggestions' error 2022-04-29 10:51:10 -06:00
Ben Burdette
2a5632c70d incorporate PosIdx changes, symbol changes. 2022-04-29 10:02:17 -06:00
Guillaume Maudoux
e93b59fbc5 Merge remote-tracking branch 'origin/master' into coerce-string 2022-04-29 00:12:25 +02:00
Ben Burdette
6e19947993 Merge branch 'master' into debug-merge-master 2022-04-28 12:32:57 -06:00
Eelco Dolstra
70a30dbc00 Fix libcxx build
Fixes #6458.
2022-04-28 14:37:05 +02:00
Eelco Dolstra
d77d813017 Shut up clang warning 2022-04-28 14:24:17 +02:00
Eelco Dolstra
4a9623b129 Fix passing $OUT_PATHS to the post-build hook
Fixes #6446.
2022-04-28 13:36:01 +02:00
Guillaume Maudoux
f6baa4d188 fixup! fix errors case and wording 2022-04-28 13:18:19 +02:00
Guillaume Maudoux
402ee8ab64 No point in passing string_views by reference 2022-04-28 13:02:39 +02:00
Guillaume Maudoux
5ef88457b8 Better document error location indent 2022-04-28 13:00:24 +02:00
Guillaume Maudoux
acf990c9ea fix errors case and wording 2022-04-28 12:54:14 +02:00
Adam Joseph
49119072e7 local-derivation-goal.cc: seccomp filters for MIPS secondary arch/abi
A mips64el Linux MIPS kernel can execute userspace code using any of
three ABIs:

  mips64el-linux-*abin64
  mips64el-linux-*abin32
    mipsel-linux-*

The first of these is the native 64-bit ABI, and the only ABI with
64-bit pointers; this is sometimes called "n64".  The last of these is
the old legacy 32-bit ABI, whose binaries can execute natively on
32-bit MIPS hardware; this is sometimes called "o32".

The second ABI, "n32" is essentially the 64-bit ABI with 32-bit
pointers and address space.  Hardware 64-bit integer/floating
arithmetic is still allowed, as well as the much larger mips64
register set and more-efficient calling convention.

Let's enable seccomp filters for all of these.  Likewise for big
endian (mips64-linux-*).
2022-04-27 21:39:26 -07:00
Eelco Dolstra
717298c749 Bump eval cache schema version 2022-04-26 17:17:51 +02:00
Eelco Dolstra
13d8400ac5 Remove obsolete FIXME 2022-04-26 17:17:27 +02:00
Eelco Dolstra
1ddabe1a01 nix: Respect meta.outputsToInstall, and use all outputs by default
'nix profile install' will now install all outputs listed in the
package's meta.outputsToInstall attribute, or all outputs if that
attribute doesn't exist. This makes it behave consistently with
nix-env. Fixes #6385.

Furthermore, for consistency, all other 'nix' commands do this as
well. E.g. 'nix build' will build and symlink the outputs in
meta.outputsToInstall, defaulting to all outputs. Previously, it only
built/symlinked the first output. Note that this means that selecting
a specific output using attrpath selection (e.g. 'nix build
nixpkgs#libxml2.dev') no longer works. A subsequent PR will add a way
to specify the desired outputs explicitly.
2022-04-26 17:17:27 +02:00
Eelco Dolstra
a81622c21d
Merge pull request #6447 from edolstra/eval-cache-symbols
EvalCache: Use Symbol in more places
2022-04-26 15:41:40 +02:00
Théophane Hufschmitt
fb5f13fb65
Merge pull request #6440 from Ma27/fix-nix-with-latest-git
libfetchers/git: hardcode `--git-dir`
2022-04-26 14:50:47 +02:00
Eelco Dolstra
b12c33510c EvalCache AttrKey: Use Symbol instead of std::string 2022-04-26 14:16:20 +02:00
Eelco Dolstra
474695975d EvalCache: Revert to using symbols in getAttr() 2022-04-26 14:01:42 +02:00
Eelco Dolstra
fab731a9d4 Don't pass Symbol by reference
Since Symbol is just an integer, passing it by const reference is
never advantageous.
2022-04-26 13:25:17 +02:00
pennae
d6d6bbd9ef
Merge branch 'master' into lto 2022-04-25 14:02:37 +00:00
pennae
a385e51a08 rename SymbolIdx -> Symbol, Symbol -> SymbolStr
after #6218 `Symbol` no longer confers a uniqueness invariant on the
string it wraps, it is now possible to create multiple symbols that
compare equal but whose string contents have different addresses. this
guarantee is now only provided by `SymbolIdx`, leaving `Symbol` only as
a string wrapper that knows about the intricacies of how symbols need to
be formatted for output.

this change renames `SymbolIdx` to `Symbol` to restore the previous
semantics of `Symbol` to that name. we also keep the wrapper type and
rename it to `SymbolStr` instead of returning plain strings from lookups
into the symbol table because symbols are formatted for output in many
places. theoretically we do not need `SymbolStr`, only a function that
formats a string for output as a symbol, but having to wrap every symbol
that appears in a message into eg `formatSymbol()` is error-prone and
inconvient.
2022-04-25 15:37:01 +02:00
Maximilian Bosch
d1f5356311
libfetchers/git: fix for nixos-rebuild
The `--git-dir=` must be `.` in some cases (for cached repos that are
"bare" repos in `~/.cache/nix/gitv3`). With this fix we can add
`--git-dir` to each `git`-invokation needed for `nixos-rebuild`.
2022-04-24 18:14:24 +02:00
Maximilian Bosch
0256e5578e
libfetchers/git: hardcode --git-dir
To demonstrate the problem:

* You need a `git` at 2.33.3 in your $PATH
* An expression like this in a git repository:

  ``` nix
  {
    outputs = { self, nixpkgs }: {
      packages.foo.x86_64-linux = with nixpkgs.legacyPackages.x86_64-linux;
        runCommand "snens" { } ''
          echo ${(builtins.fetchGit ./.).lastModifiedDate} > $out
        '';
    };
  }
  ```

Now, when instantiating the package via `builtins.getFlake`, it fails on
Nix 2.7 like this:

    $ nix-instantiate -E '(builtins.getFlake "'"$(pwd)"'").packages.foo.x86_64-linux'
    fatal: unsafe repository ('/nix/store/a7j3125km4h8l0p71q6ssfkxamfh5d61-source' is owned by someone else)
    To add an exception for this directory, call:

    	git config --global --add safe.directory /nix/store/a7j3125km4h8l0p71q6ssfkxamfh5d61-source
    error: program 'git' failed with exit code 128
    (use '--show-trace' to show detailed location information)

This breaks e.g. `nixops`-deployments using flakes with similar
expressions as shown above.

The cause for this is that `git(1)` tries to find the highest
`.git`-directory in the directory tree and if it finds a such a
directory, but with another owning user (root vs. the user who evaluates
the expression), it fails as above. This was changed recently to fix
CVE-2022-24765[1].

By explicitly specifying `--git-dir`, Git assumes to be in the top-level
directory and doesn't attempt to look for a `.git`-directory in the
parent directories and thus the code-path leading to said error is never
reached.

[1] https://lore.kernel.org/git/xmqqv8veb5i6.fsf@gitster.g/
2022-04-23 23:20:17 +02:00
Théophane Hufschmitt
197aa2fa96 Merge remote-tracking branch 'origin/master' into nixbuildaddprintstorepaths 2022-04-22 13:20:39 +02:00
Théophane Hufschmitt
be28603dca Merge remote-tracking branch 'origin/master' into nixbuildaddprintstorepaths 2022-04-22 11:11:01 +02:00
Théophane Hufschmitt
7b889f31ea Fix the darwin build
Looks like the auto-merge is indeed quite broken and merges even when the CI fails
2022-04-22 10:56:56 +02:00
Théophane Hufschmitt
c4ffc8e2f8
Merge pull request #6218 from pennae/pos-symbol-tables
reduce the size of Attr from 3 pointers to 2 on 64 bit machines
2022-04-22 10:28:06 +02:00
Théophane Hufschmitt
484badfa09 Add some tests for ChunkedVector 2022-04-22 10:03:44 +02:00