Commit graph

15363 commits

Author SHA1 Message Date
Valentin Gagarin
f00a5eb11b introduce lookup paths as a distinct language construct
so far they did not really have a name, and were at best referred to as
"angle bracket syntax".
2023-10-07 04:44:09 +02:00
Valentin Gagarin
6305801626 reword and reformat description of builtins.import 2023-10-07 03:57:08 +02:00
Valentin Gagarin
2fe1ccf797 describe the effect of restrict-eval in a more focused manner 2023-10-07 03:20:20 +02:00
Valentin Gagarin
a86a3e5e59 document that pure-eval also disables builtins.nixPath 2023-10-07 03:12:25 +02:00
Valentin Gagarin
a7ba8c3f4a complete example on attribute name interpolation 2023-10-07 02:49:55 +02:00
Valentin Gagarin
a67cee965a expand on interpolated expressions 2023-10-07 02:49:55 +02:00
Valentin Gagarin
c167ffe1cf
Merge pull request #9108 from fricklerhandwerk/fix-redirect
remove duplicate redirects entry
2023-10-07 00:06:33 +02:00
Valentin Gagarin
517c547dec remove duplicate redirects entry 2023-10-06 23:34:08 +02:00
John Ericson
644ebaab5f Define NixOS tests in tests/nixos/default.nix rather than flake.nix
I think the our `flake.nix` is currently too large and too scary looking.
I think this matters --- if Nix cannot dog-food itself in a way that is
elegant, why should other people have confidence that their own code can
be elegant and easy to maintain?

We could do this at many points in time, but I think around now, when we
are thinking about stabilizing parts of Flakes, is an especially good
time.

This is a first step to make the `flake.nix` smaller, and make
individual components responsible for their own packaging. I hope we can
do this many more follow-ups like it, until the top-level `flake.nix` is
very small and just coordinates between other things.
2023-10-06 10:58:17 -04:00
John Ericson
61720d0035
Merge pull request #9103 from obsidiansystems/functional-tests-dir
Put functional tests in `tests/functional`
2023-10-06 09:34:11 -04:00
John Ericson
68c81c7375 Put functional tests in tests/functional
I think it is bad for these reasons when `tests/` contains a mix of
functional and integration tests

 - Concepts is harder to understand, the documentation makes a good
   unit vs functional vs integration distinction, but when the
   integration tests are just two subdirs within `tests/` this is not
   clear.

 - Source filtering in the `flake.nix` is more complex. We need to
   filter out some of the dirs from `tests/`, rather than simply pick
   the dirs we want and take all of them. This is a good sign the
   structure of what we are trying to do is not matching the structure
   of the files.

With this change we have a clean:
```shell-session
$ git show 'HEAD:tests'
tree HEAD:tests

functional/
installer/
nixos/
```
2023-10-06 09:05:56 -04:00
Eelco Dolstra
3dd4475826
Merge pull request #8905 from hercules-ci/no-unknown-location
Don't print unknown locations unless requested for dev purposes
2023-10-06 14:41:01 +02:00
Eelco Dolstra
72c3990bb4
Merge pull request #9104 from fricklerhandwerk/nix-env-links
add links and anchors
2023-10-06 14:24:01 +02:00
Eelco Dolstra
8cfa582f43
Merge pull request #9096 from NixOS/dont-rebuild-twice-on-ci
Don't run the tests that require building if we're not building
2023-10-06 14:17:05 +02:00
Eelco Dolstra
62434951d9
Merge pull request #9095 from edef1c/reject-dot-paths
StorePath: reject names starting with '.'
2023-10-06 14:12:53 +02:00
Valentin Gagarin
369b076986 add links and anchors 2023-10-06 11:46:41 +02:00
Robert Hensing
f95364a803 eval: Run a full GC before printing stats
This makes the numbers more deterministic, especially when it comes
to the final heap size.
2023-10-06 00:21:05 +02:00
Robert Hensing
66a9fb0314
Merge pull request #9010 from fricklerhandwerk/first-steps
refer to nix.dev for installation instructions
2023-10-06 00:07:24 +02:00
Théophane Hufschmitt
92e8e1b1bb Poison the build on the test derivation
Make sure that we're not accidentally rebuilding Nix here as it's just
wasteful and awful for CI times.
2023-10-05 16:43:26 +02:00
Théophane Hufschmitt
eb68454be6 Don't run the tests that require building if we're not building
A couple of tests require building some libraries that depend on Nix,
and assume it to be built locally.
Don't run these if we only want to run the install tests.

This prevents the CI from rebuilding several times Nix (like in
https://github.com/NixOS/nix/actions/runs/6404422275/job/17384964033#step:6:6412), thus removing a fair amount of build time.
2023-10-05 16:43:26 +02:00
Valentin Gagarin
d12fb4b1f1
Merge pull request #9017 from fricklerhandwerk/contributing-docs
add contributing guide for documentation
2023-10-05 09:23:27 +02:00
Valentin Gagarin
80f734a803
more specific links to nix.dev 2023-10-05 09:20:42 +02:00
John Ericson
d344c112f7
Merge pull request #9098 from obsidiansystems/test-proto
Enable most of the third `BuildResult` worker protocol test
2023-10-04 22:44:22 -04:00
Valentin Gagarin
e0e47c0a68 accommodate inconsistent output from lowdown
the `term` output mode leaves inline HTML around verbatim, while `nroff`
mode (used for `man` pages) does not.

the correct solution would be to pre-render all output with a more
benign tool so we have less liabilities in our own code, but this has to
do for now.
2023-10-05 01:20:26 +02:00
edef
24bda0c7b3 StorePath: reject names starting with '.'
This has been the behaviour before Nix 2.4. It was dropped in a rewrite
in 759947bf72, allowing the creation of
store paths that aren't considered valid by older Nix versions or other
Nix tooling.

Nix 2.4 didn't ship in NixOS until 22.05, and stdenv.mkDerivation in
nixpkgs drops leading periods since April 2022, so it's unlikely anyone
is relying on the current lax behaviour.

Closes #9091.

Change-Id: I4a57bd9899e1b0dba56870ae5a1b680918a18ce9
2023-10-04 22:10:52 +00:00
Valentin Gagarin
8232711c9f fix wiring of baked-in Nix expressions 2023-10-05 00:05:08 +02:00
Valentin Gagarin
64b73476ce always show anchors on setting listings
refactor the templates for readability
2023-10-05 00:05:08 +02:00
John Ericson
e1af175707 Enable most of the third BuildResult worker protocol test
This was somewhat of a false alarm. The problem was not that the
protocol implementation actually failed to round trip, but that two of
the fields were ignored entirely --- not serialized and deserialized at
all.

For reference, those fields were added in
fa68eb367e.
2023-10-04 18:03:50 -04:00
John Ericson
2f1c16dfa2
Merge pull request #9094 from obsidiansystems/test-proto
Test the rest of the worker protocol serializers
2023-10-04 16:13:00 -04:00
John Ericson
632f24166d Test the rest of the worker protocol serializers
Part of the `BuildResult` test is commented out because we have caught a
roundtrip bug! A future PR will fix the bug and uncomment that test.
2023-10-04 15:31:52 -04:00
Robert Hensing
3c042f3b0b
Merge pull request #9032 from Ma27/structured-attrs-env-vars
structured attrs: improve support / usage of NIX_ATTRS_{SH,JSON}_FILE
2023-10-04 11:57:26 +02:00
John Ericson
b80207fad8
Merge pull request #9081 from obsidiansystems/hacky-fix-9052
Revert "Adapt scheduler to work with dynamic derivations"
2023-10-02 08:10:25 -04:00
John Ericson
8440afbed7 Revert "Adapt scheduler to work with dynamic derivations"
This reverts commit 5e3986f59c. This
un-implements RFC 92 but fixes the critical bug #9052 which many people
are hitting. This is a decent stop-gap until a minimal reproduction of
that bug is found and a proper fix can be made.

Mostly fixed #9052, but I would like to leave that issue open until we
have a regression test, so I can then properly fix the bug (unbreaking
RFC 92) later.
2023-10-01 23:43:12 -04:00
Valentin Gagarin
78e886bc5f refine the maintainer's process to unblock discussions more quickly
this addresses that we're too often running into open-ended discussions
about attempts to solve problems where neither the problem nor the
solution is well-understood enough to make decisions in a reasonable
amount of time.

this also prevents us from doing more work asynchronously.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-10-01 12:54:59 -04:00
Robert Hensing
7a0886e3cc tests/structured-attrs.sh: grep -q -> grepQuiet 2023-10-01 13:25:32 +01:00
Maximilian Bosch
42e3c6d658 doc: reference NIX_ATTRS_*_FILE vars at the env var reference for drvs 2023-10-01 13:25:29 +01:00
Maximilian Bosch
bfdd908f7d structured attrs: improve support / usage of NIX_ATTRS_{SH,JSON}_FILE
In #4770 I implemented proper `nix-shell(1)` support for derivations
using `__structuredAttrs = true;`. Back then we decided to introduce two
new environment variables, `NIX_ATTRS_SH_FILE` for `.attrs.sh` and
`NIX_ATTRS_JSON_FILE` for `.attrs.json`. This was to avoid having to
copy these files to `$NIX_BUILD_TOP` in a `nix-shell(1)` session which
effectively meant copying these files to the project dir without
cleaning up afterwords[1].

On last NixCon I resumed hacking on `__structuredAttrs = true;` by
default for `nixpkgs` with a few other folks and getting back to it,
I identified a few problems with the how it's used in `nixpkgs`:

* A lot of builders in `nixpkgs` don't care about the env vars and
  assume that `.attrs.sh` and `.attrs.json` are in `$NIX_BUILD_TOP`.
  The sole reason why this works is that `nix-shell(1)` sources
  the contents of `.attrs.sh` and then sources `$stdenv/setup` if it
  exists. This may not be pretty, but it mostly works. One notable
  difference when using nixpkgs' stdenv as of now is however that
  `$__structuredAttrs` is set to `1` on regular builds, but set to
  an empty string in a shell session.

  Also, `.attrs.json` cannot be used in shell sessions because
  it can only be accessed by `$NIX_ATTRS_JSON_FILE` and not by
  `$NIX_BUILD_TOP/.attrs.json`.

  I considered changing Nix to be compatible with what nixpkgs
  effectively does, but then we'd have to either move $NIX_BUILD_TOP for
  shell sessions to a temporary location (and thus breaking a lot of
  assumptions) or we'd reintroduce all the problems we solved back then
  by using these two env vars.

  This is partly because I didn't document these variables back
  then (mea culpa), so I decided to drop all mentions of
  `.attrs.{json,sh}` in the  manual and only refer to `$NIX_ATTRS_SH_FILE`
  and `$NIX_ATTRS_JSON_FILE`. The same applies to all our integration tests.
  Theoretically we could deprecated using `"$NIX_BUILD_TOP"/.attrs.sh` in
  the future now.

* `nix develop` and `nix print-dev-env` don't support this environment
  variable at all even though they're supposed to be part of the replacement
  for `nix-shell` - for the drv debugging part to be precise.

  This isn't a big deal for the vast majority of derivations, i.e.
  derivations relying on nixpkgs' `stdenv` wiring things together
  properly. This is because `nix develop` effectively "clones" the
  derivation and replaces the builder with a script that dumps all of
  the environment, shell variables, functions etc, so the state of
  structured attrs being "sourced" is transmitted into the dev shell and
  most of the time you don't need to worry about `.attrs.sh` not
  existing because the shell is correctly configured and the

      if [ -e .attrs.sh ]; then source .attrs.sh; fi

  is simply omitted.

  However, this will break when having a derivation that reads e.g. from
  `.attrs.json` like

      with import <nixpkgs> {};
      runCommand "foo" { __structuredAttrs = true; foo.bar = 23; } ''
        cat $NIX_ATTRS_JSON_FILE # doesn't work because it points to /build/.attrs.json
      ''

  To work around this I employed a similar approach as it exists for
  `nix-shell`: the `NIX_ATTRS_{JSON,SH}_FILE` vars are replaced with
  temporary locations.

  The contents of `.attrs.sh` and `.attrs.json` are now written into the
  JSON by `get-env.sh`, the builder that `nix develop` injects into the
  derivation it's debugging. So finally the exact file contents are
  present and exported by `nix develop`.

  I also made `.attrs.json` a JSON string in the JSON printed by
  `get-env.sh` on purpose because then it's not necessary to serialize
  the object structure again. `nix develop` only needs the JSON
  as string because it's only written into the temporary file.

  I'm not entirely sure if it makes sense to also use a temporary
  location for `nix print-dev-env` (rather than just skipping the
  rewrite in there), but this would probably break certain cases where
  it's relied upon `$NIX_ATTRS_SH_FILE` to exist (prime example are the
  `nix print-dev-env` test-cases I wrote in this patch using
  `tests/shell.nix`, these would fail because the env var exists, but it
  cannot read from it).

[1] https://github.com/NixOS/nix/pull/4770#issuecomment-836799719
2023-10-01 13:22:48 +01:00
John Ericson
ea2f74cbe1
Merge pull request #9022 from hercules-ci/fix-issue-8838-pathExists-isDir-slash-dot
pathExists: isDir when endswith /.
2023-09-30 00:49:17 -04:00
Robert Hensing
f8a3893e8d pathExists: isDir when endswith /. 2023-09-30 02:35:26 +01:00
Robert Hensing
9c84054f97
Merge pull request #9073 from fricklerhandwerk/pr-guidelines
contributor guide: emphasize solving a well-specified problem with each pull request
2023-09-29 18:11:54 +01:00
John Ericson
461902b860
Merge pull request #9069 from obsidiansystems/libfetchers-prep-0
`libfetchers` improvements without `libflake`
2023-09-29 10:39:45 -04:00
Valentin Gagarin
08145a5be5 contributor guide: emphasize solving a well-specified problem with each pull request
this moves the orientation step to the beginning, and adds notes how to
make sure that a problem is well-spefified and the according change more
likely to get accepted

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-29 16:19:14 +02:00
Théophane Hufschmitt
784c7df5bf
Merge pull request #9004 from fricklerhandwerk/release-support
add information on release cycle and backports
2023-09-29 15:04:00 +02:00
Théophane Hufschmitt
a66bd8f1f2
Merge pull request #9065 from fricklerhandwerk/testing-hints
add hint for troubleshooting tests
2023-09-29 13:52:54 +02:00
Valentin Gagarin
1dd03c62ad add hint for troubleshooting tests 2023-09-29 10:46:42 +02:00
John Ericson
89b3952063 Make the indirect fetcher input scheme part of the Flakes XP feature
I don't know much about it, but by the number of times "flake" appears
in the code it seems like is part of flakes, at least for now.
2023-09-28 21:35:36 -04:00
John Ericson
bfe1308d3f Add infra for InputSchemes to be experimental 2023-09-28 21:35:30 -04:00
John Ericson
c816c67eed Reword some comments/API docs to reflect libfetcher's multiple users
It's not just flakes, but also `builtins.fetchTree`. Also try to provide
some more info in general.
2023-09-28 21:10:51 -04:00
John Ericson
b912f3a937 Move flakeIdRegex{,S} from libutil to flakeref.{cc,hh
It isn't used, and doesn't belong in `libutil`.
2023-09-28 20:55:41 -04:00
John Ericson
cede94dbf7 builtins.fetchTree: Mark experimental the new way
This helps ensure uniform docs/error message.
2023-09-28 20:51:25 -04:00