Commit graph

2061 commits

Author SHA1 Message Date
Johannes Kirschbauer
42f26eb42e
doc: complexity for '?' operator (#9184)
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-10-20 02:45:47 +02:00
Eelco Dolstra
d2c0051784 Remove obsolete corepkgs references 2023-10-18 23:35:07 +02:00
Yueh-Shun Li
5088e6563a primops: add builtins.convertHash
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-10-19 00:58:56 +08:00
Eelco Dolstra
df73c6eb8c Introduce MemoryInputAccessor and use it for corepkgs
MemoryInputAccessor is an in-memory virtual filesystem that returns
files like <nix/fetchurl.nix>. This removes the need for special hacks
to handle those files.
2023-10-18 17:38:11 +02:00
Eelco Dolstra
201c115c3e
Merge pull request #9151 from edolstra/stabilize-fetchTree
Stabilize fetchTree
2023-10-18 10:54:08 +02:00
John Ericson
9d1f42db52
Merge pull request #9150 from vicky1999/fix/8914
`nix store ping` -> `nix store info`
2023-10-17 22:52:28 -04:00
vicky1999
891dfb4359 updated store ping to store info in files 2023-10-18 00:14:11 +05:30
Eelco Dolstra
4112dd1fc9 Mark fetchTree as stable 2023-10-13 16:45:08 +02:00
Robert Hensing
da2b59a088
Merge pull request #8047 from lovesegfault/always-allow-substitutes
feat: add always-allow-substitutes
2023-10-13 15:42:11 +02:00
Ninlives
94e91566ed
Allow CLI to pass environment variables to FOD builder (#8830)
Add a new experimental `impure-env` setting that is a key-value list of
environment variables to inject into FOD derivations that specify the
corresponding `impureEnvVars`.

This allows clients to make use of this feature (without having to change the
environment of the daemon itself) and might eventually deprecate the current
behaviour (pick whatever is in the environment of the daemon) as it's more
principled and might prevent information leakage.
2023-10-11 11:58:42 +00:00
Valentin Gagarin
1e1f08c045 annotate admonitions showing syntax
also fix typos
2023-10-10 00:45:19 +02:00
John Ericson
be81764320 Factor out bits of the worker protocol to use elsewhere
This introduces some shared infrastructure for our notion of protocols.
We can then define multiple protocols in terms of that notion.
We an also express how particular protocols depend on each other.

For example, we can define a common protocol and a worker protocol,
where the second depends on the first in terms of the data types it can
read and write.

The "serve" protocol can just use the common one for now, but will
eventually need its own machinary just like the worker protocol for
version-aware serialisers
2023-10-09 16:55:12 -04:00
John Ericson
2f0b508c29 Get rid of bootstrap.sh
For people working on Nix with `nix develop`, it's better to just use
`autoreconfPhase` and `configurePhase`, which is standard Nixpkgs / nix
shell make from Nixpkgs practice --- it is good to emphasize the degree
to which Nix is *just* a regular C++ project which can be worked on in
the regular way.

(For people running `nix-shell`, the story is similar, except
`configurePhase` would use non-writable store paths, which matters for
hte times we use output paths before `make install`, so I kept the
existing `./configure ...` instruction.)

For people building Nix without Nix (e.g. packaging it for another
distro) they also don't need `bootstrap.sh`, and can just run
`autoreconf -vfi` directly. (More likely, they have their own idioms to
do this just as we have `autoreconfPhase`.)
2023-10-09 12:55:58 -04:00
John Ericson
22513c91dc
Merge pull request #8942 from fricklerhandwerk/option-anchors
always show anchors on setting listings
2023-10-09 11:29:36 -04:00
John Ericson
838be5e4a0
Merge pull request #9114 from fricklerhandwerk/lookup-path
introduce lookup paths as a distinct language construct
2023-10-09 11:28:40 -04:00
John Ericson
ead65d3d8e
Merge pull request #9110 from fricklerhandwerk/interpolated-expression
expand on interpolated expressions
2023-10-09 10:56:19 -04:00
Eelco Dolstra
c8eb3b82e9
Merge pull request #9129 from fricklerhandwerk/ae-be
AE -> BE; fix redirects
2023-10-09 16:56:07 +02:00
John Ericson
217d863f7a Merge branch 'master' into lookup-path 2023-10-09 10:54:54 -04:00
John Ericson
67eddc05ef
Merge pull request #9116 from fricklerhandwerk/doc-import
reword and reformat description of `builtins.import`
2023-10-09 10:26:41 -04:00
John Ericson
81d3a8542a
doc: Slight reword of "interpolated expression" in paragraph
I was sleepy and confused that "interpolated expression" was a new type of thing at first. This nudges the reader to understand that its just a regular expression, and these conditions are imposed by the interpolation operation.
2023-10-09 10:21:18 -04:00
Valentin Gagarin
0246de1896 remove unnecessary indentation from markdown list
this makes it a bit easier to work with, as some tooling doesn't work
well with too much indentation.
2023-10-09 10:14:29 +02:00
Valentin Gagarin
896a905202 AE -> BE; fix redirects 2023-10-09 10:07:46 +02:00
Valentin Gagarin
f00a5eb11b introduce lookup paths as a distinct language construct
so far they did not really have a name, and were at best referred to as
"angle bracket syntax".
2023-10-07 04:44:09 +02:00
Valentin Gagarin
a7ba8c3f4a complete example on attribute name interpolation 2023-10-07 02:49:55 +02:00
Valentin Gagarin
a67cee965a expand on interpolated expressions 2023-10-07 02:49:55 +02:00
Valentin Gagarin
517c547dec remove duplicate redirects entry 2023-10-06 23:34:08 +02:00
John Ericson
68c81c7375 Put functional tests in tests/functional
I think it is bad for these reasons when `tests/` contains a mix of
functional and integration tests

 - Concepts is harder to understand, the documentation makes a good
   unit vs functional vs integration distinction, but when the
   integration tests are just two subdirs within `tests/` this is not
   clear.

 - Source filtering in the `flake.nix` is more complex. We need to
   filter out some of the dirs from `tests/`, rather than simply pick
   the dirs we want and take all of them. This is a good sign the
   structure of what we are trying to do is not matching the structure
   of the files.

With this change we have a clean:
```shell-session
$ git show 'HEAD:tests'
tree HEAD:tests

functional/
installer/
nixos/
```
2023-10-06 09:05:56 -04:00
Valentin Gagarin
369b076986 add links and anchors 2023-10-06 11:46:41 +02:00
Valentin Gagarin
d12fb4b1f1
Merge pull request #9017 from fricklerhandwerk/contributing-docs
add contributing guide for documentation
2023-10-05 09:23:27 +02:00
Valentin Gagarin
e0e47c0a68 accommodate inconsistent output from lowdown
the `term` output mode leaves inline HTML around verbatim, while `nroff`
mode (used for `man` pages) does not.

the correct solution would be to pre-render all output with a more
benign tool so we have less liabilities in our own code, but this has to
do for now.
2023-10-05 01:20:26 +02:00
Valentin Gagarin
64b73476ce always show anchors on setting listings
refactor the templates for readability
2023-10-05 00:05:08 +02:00
Maximilian Bosch
42e3c6d658 doc: reference NIX_ATTRS_*_FILE vars at the env var reference for drvs 2023-10-01 13:25:29 +01:00
Maximilian Bosch
bfdd908f7d structured attrs: improve support / usage of NIX_ATTRS_{SH,JSON}_FILE
In #4770 I implemented proper `nix-shell(1)` support for derivations
using `__structuredAttrs = true;`. Back then we decided to introduce two
new environment variables, `NIX_ATTRS_SH_FILE` for `.attrs.sh` and
`NIX_ATTRS_JSON_FILE` for `.attrs.json`. This was to avoid having to
copy these files to `$NIX_BUILD_TOP` in a `nix-shell(1)` session which
effectively meant copying these files to the project dir without
cleaning up afterwords[1].

On last NixCon I resumed hacking on `__structuredAttrs = true;` by
default for `nixpkgs` with a few other folks and getting back to it,
I identified a few problems with the how it's used in `nixpkgs`:

* A lot of builders in `nixpkgs` don't care about the env vars and
  assume that `.attrs.sh` and `.attrs.json` are in `$NIX_BUILD_TOP`.
  The sole reason why this works is that `nix-shell(1)` sources
  the contents of `.attrs.sh` and then sources `$stdenv/setup` if it
  exists. This may not be pretty, but it mostly works. One notable
  difference when using nixpkgs' stdenv as of now is however that
  `$__structuredAttrs` is set to `1` on regular builds, but set to
  an empty string in a shell session.

  Also, `.attrs.json` cannot be used in shell sessions because
  it can only be accessed by `$NIX_ATTRS_JSON_FILE` and not by
  `$NIX_BUILD_TOP/.attrs.json`.

  I considered changing Nix to be compatible with what nixpkgs
  effectively does, but then we'd have to either move $NIX_BUILD_TOP for
  shell sessions to a temporary location (and thus breaking a lot of
  assumptions) or we'd reintroduce all the problems we solved back then
  by using these two env vars.

  This is partly because I didn't document these variables back
  then (mea culpa), so I decided to drop all mentions of
  `.attrs.{json,sh}` in the  manual and only refer to `$NIX_ATTRS_SH_FILE`
  and `$NIX_ATTRS_JSON_FILE`. The same applies to all our integration tests.
  Theoretically we could deprecated using `"$NIX_BUILD_TOP"/.attrs.sh` in
  the future now.

* `nix develop` and `nix print-dev-env` don't support this environment
  variable at all even though they're supposed to be part of the replacement
  for `nix-shell` - for the drv debugging part to be precise.

  This isn't a big deal for the vast majority of derivations, i.e.
  derivations relying on nixpkgs' `stdenv` wiring things together
  properly. This is because `nix develop` effectively "clones" the
  derivation and replaces the builder with a script that dumps all of
  the environment, shell variables, functions etc, so the state of
  structured attrs being "sourced" is transmitted into the dev shell and
  most of the time you don't need to worry about `.attrs.sh` not
  existing because the shell is correctly configured and the

      if [ -e .attrs.sh ]; then source .attrs.sh; fi

  is simply omitted.

  However, this will break when having a derivation that reads e.g. from
  `.attrs.json` like

      with import <nixpkgs> {};
      runCommand "foo" { __structuredAttrs = true; foo.bar = 23; } ''
        cat $NIX_ATTRS_JSON_FILE # doesn't work because it points to /build/.attrs.json
      ''

  To work around this I employed a similar approach as it exists for
  `nix-shell`: the `NIX_ATTRS_{JSON,SH}_FILE` vars are replaced with
  temporary locations.

  The contents of `.attrs.sh` and `.attrs.json` are now written into the
  JSON by `get-env.sh`, the builder that `nix develop` injects into the
  derivation it's debugging. So finally the exact file contents are
  present and exported by `nix develop`.

  I also made `.attrs.json` a JSON string in the JSON printed by
  `get-env.sh` on purpose because then it's not necessary to serialize
  the object structure again. `nix develop` only needs the JSON
  as string because it's only written into the temporary file.

  I'm not entirely sure if it makes sense to also use a temporary
  location for `nix print-dev-env` (rather than just skipping the
  rewrite in there), but this would probably break certain cases where
  it's relied upon `$NIX_ATTRS_SH_FILE` to exist (prime example are the
  `nix print-dev-env` test-cases I wrote in this patch using
  `tests/shell.nix`, these would fail because the env var exists, but it
  cannot read from it).

[1] https://github.com/NixOS/nix/pull/4770#issuecomment-836799719
2023-10-01 13:22:48 +01:00
Théophane Hufschmitt
784c7df5bf
Merge pull request #9004 from fricklerhandwerk/release-support
add information on release cycle and backports
2023-09-29 15:04:00 +02:00
Théophane Hufschmitt
a66bd8f1f2
Merge pull request #9065 from fricklerhandwerk/testing-hints
add hint for troubleshooting tests
2023-09-29 13:52:54 +02:00
Valentin Gagarin
1dd03c62ad add hint for troubleshooting tests 2023-09-29 10:46:42 +02:00
tomberek
976f596579
Merge branch 'master' into tomberek.absolute.attrpath.notation 2023-09-28 10:01:57 -04:00
Ilan Joselevich
13ed5d7106 flakes: adopt repl-flake behavior as default 2023-09-27 20:47:10 -04:00
Robert Hensing
ea2fa8b6f3
Merge pull request #8241 from Luabee/patch-1
Mention `$DRV_PATH` in post-build-hook docs
2023-09-28 00:11:00 +01:00
Robert Hensing
dafa38213b
Update doc/manual/src/advanced-topics/post-build-hook.md
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-09-27 23:10:39 +01:00
Théophane Hufschmitt
c6faef61a6
Merge pull request #8923 from obsidiansystems/test-proto
Unit test some worker protocol serializers
2023-09-26 17:12:24 +02:00
Eelco Dolstra
706b91ef62
Merge pull request #9044 from fricklerhandwerk/doc-realise
realisation: reformat for readability
2023-09-26 15:28:20 +02:00
Valentin Gagarin
a757749fcf reword for readability 2023-09-26 06:28:17 +02:00
Valentin Gagarin
9428d7dcd1
Merge pull request #9045 from fricklerhandwerk/fix-redirects
fix broken redirects script
2023-09-26 04:27:33 +02:00
Valentin Gagarin
503b02d3f8
Merge pull request #9046 from fricklerhandwerk/fixup-derivation
fix broken reference link
2023-09-26 04:27:23 +02:00
Valentin Gagarin
60a155d01c
Merge pull request #8706 from fricklerhandwerk/doc-system-features
document system features
2023-09-26 04:21:31 +02:00
Valentin Gagarin
b17f200b11
Document "Import From Derivation" (#7332)
* document "Import From Derivation"

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-09-26 01:49:03 +00:00
Valentin Gagarin
c6f8247032 fix broken reference link 2023-09-26 03:10:12 +02:00
Théophane Hufschmitt
9a78d87bc0
Merge pull request #6614 from RasmusRendal/spaces
Implement support for percent encoded filepaths for flakerefs
2023-09-26 02:27:09 +02:00
Valentin Gagarin
45de35bcf1 fix broken redirects script 2023-09-26 01:24:07 +02:00
Valentin Gagarin
887cbcd395 add contributing guide for documentation 2023-09-26 01:06:47 +02:00
Valentin Gagarin
5b0336b3b1 reword example for clarity 2023-09-26 00:58:50 +02:00
Valentin Gagarin
7de66f19f8 example: headers -> dev
make the example more realistic, since `headers` is not an output name
used in Nixpkgs

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-26 00:46:27 +02:00
Valentin Gagarin
75a231147f be more precise about system semantics 2023-09-26 00:45:00 +02:00
Valentin Gagarin
5e4734a08b reword to avoid saying "build"
derivations are about data transformation, so the term "build" does not
add any information. there was also some feedback that "build task" is
not more helpful than "derivation" if you have no prior experience with
Nix or build systems, while existing associations may be misleading.
2023-09-26 00:45:00 +02:00
Valentin Gagarin
17884f54d1 clarification on extra attributes
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-26 00:45:00 +02:00
Valentin Gagarin
026c24e378 add example for store path using the given name 2023-09-26 00:45:00 +02:00
Valentin Gagarin
d621dd17f2 more precise wording
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-26 00:45:00 +02:00
Valentin Gagarin
e2f118efed restructure and reword input attributes section on derivation 2023-09-26 00:43:58 +02:00
Valentin Gagarin
541890463d make separate section for builder execution 2023-09-26 00:43:58 +02:00
Valentin Gagarin
cba53b3a18 reformat for readability 2023-09-26 00:42:23 +02:00
John Ericson
728767db03
Merge pull request #9028 from Ericson2314/nix3-config-options
Misc options rendering adjustments
2023-09-25 09:04:56 -04:00
John Ericson
4606a07bb6
generate-manpage.nix: Add comment explaining one bit 2023-09-25 08:20:39 -04:00
Maximilian Bosch
89e5e68799
doc/hacking: fix make target to build the docs (#9033)
Was confused why `make html` didn't work while working on #9032, but
then I realized that after this section was written, the target was
renamed to `manual-html` in 6910f5dcb6.
2023-09-24 20:28:34 +02:00
John Ericson
9f93972c4d manual / manpages: Make option category names a proper subheader
Before they were an "ad-hoc" header with bold and a colon; now they are
a proper subheader.

For the man pages, this doesn't make much of a difference, but it will
help more on for the HTML manual, where things can be restyled. Again,
good separation of content vs presentation.
2023-09-23 00:37:44 -04:00
John Ericson
1d9fd3a6f8 manual / manpages: Adjust option filter filtering, move from C++ to Nix
Behavior change:

Before we only showed uption if the command-specific options were
non-empty. But that is somewhat odd since we also show common options.
Now, we do everything based on the union of both sorts of options (with
hidden-categories filtered, as before).

Implementation change:

The JSON dumping once again includes all options; the filtering of
hidden categories is done in the Nix instead. This is better separation
of "content" vs "presentation", and prepare the way for the HTML manual
vs manpages / `--help` doing different things.
2023-09-23 00:34:51 -04:00
John Ericson
9c640c1229 doc: showOptions: Simplify code with builtins.groupBy
This makes grouping options by category much nicer. No behavior should
be changed.
2023-09-23 00:28:16 -04:00
John Ericson
694810ba34 doc: showOptions: Move union to caller
`showOptions` itself doesn't care, so it shouldn't take two separate
arguments.
2023-09-22 23:54:04 -04:00
Théophane Hufschmitt
f89b84919c
Merge pull request #8931 from fricklerhandwerk/nix3-config-options
do not show configuration override flags for each command
2023-09-22 14:13:51 +02:00
Rasmus Rendal
3411507696 Document the percent-encoding mechanism 2023-09-22 10:07:14 +02:00
Valentin Gagarin
1b560ea502
more detail on backports
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-09-21 23:23:14 +02:00
John Ericson
6b6cfec8a5
Merge pull request #8990 from fricklerhandwerk/example-nix-env-install-outputs
add clarifying example to `nix-env` output selection
2023-09-21 12:27:35 -04:00
John Ericson
0273ad73fa
Merge pull request #9012 from fricklerhandwerk/fix-multiline-doc
fix rendering error for consecutive spaces
2023-09-21 11:57:39 -04:00
John Ericson
7bafcdba79
Merge pull request #9013 from fricklerhandwerk/test-coverage-docs
Move test coverage section to testing page
2023-09-21 11:50:04 -04:00
Valentin Gagarin
cf6ba7256f delete unused files 2023-09-21 02:57:00 +02:00
Valentin Gagarin
02649d247b move test coverage section to testing page 2023-09-21 02:52:28 +02:00
Valentin Gagarin
954890a42f add information on release cycle and backports 2023-09-21 02:52:05 +02:00
Valentin Gagarin
747b2baf21 fix rendering error for consecutive spaces 2023-09-21 02:51:54 +02:00
Eelco Dolstra
10ad052f7d Release notes 2023-09-20 11:42:49 +02:00
Valentin Gagarin
408055a9dd add clarifying example to nix-env output selection
there is a very confusing warning in the Nixpkgs manual that
mischaracterises `nix-env` behavior, and this example shows what's
really happening.

note that it doesn't use `pkgs.runCommand` or other `pkgs.stdenv`
facilities, as deep down those set `meta.outputsToInstall` to very
particular defaults that do not generally apply to Nix.
2023-09-18 20:11:42 +02:00
Emil Nikolov
2cdc9c32e7
docs: fixed the default priority of nix-env --install (#8945) 2023-09-09 06:54:39 +00:00
John Ericson
7ad66cb3ef Allow dynamic derivation deps in inputDrvs
We use the same nested map representation we used for goals, again in
order to save space. We might someday want to combine with `inputDrvs`,
by doing `V = bool` instead of `V = std::set<OutputName>`, but we are
not doing that yet for sake of a smaller diff.

The ATerm format for Derivations also needs to be extended, in addition
to the in-memory format. To accomodate this, we added a new basic
versioning scheme, so old versions of Nix will get nice errors. (And
going forward, if the ATerm format changes again the errors will be even
better.)

`parsedStrings`, an internal function used as part of parsing
derivations in A-Term format, used to consume the final `]` but expect
the initial `[` to already be consumed. This made for what looked like
unbalanced brackets at callsites, which was confusing. Now it consumes
both which is hopefully less confusing.

As part of testing, we also created a unit test for the A-Term format for
regular non-experimental derivations too.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

Apply suggestions from code review

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-07 10:39:37 -04:00
Valentin Gagarin
aa46f536e8 add note on overriding settings for stable commands 2023-09-07 02:02:27 +02:00
Valentin Gagarin
09eb7f1ef6 do not show configuration override flags for each command
this removes a lot of noise from the web search, which precludes finding
the actual documentation.

some configuration settings have enough documentation to warrant
individual pages, so the alternative of including full setting
documentation in each command page doesn't make much sense here.

this change technically means that the command line flags to override
settings are "invisible", and not exported as JSON. this may or may not
be desirable. a more explicit approach would be adding a `hidden` field
to the flag's JSON output, but would also require adjusting
post-processing of that JSON for manual rendering.
2023-09-07 02:02:27 +02:00
Valentin Gagarin
b0fe7f560d add missing link 2023-09-07 01:38:53 +02:00
Valentin Gagarin
589fd897fb add todo on store docs 2023-09-07 01:38:53 +02:00
John Ericson
3a62651bd6
Merge pull request #8933 from fricklerhandwerk/option-anchors
Add anchors to option listings
2023-09-06 09:39:33 -04:00
John Ericson
216c66ddcf
Merge pull request #8932 from fricklerhandwerk/formatting
dedent common options listing; one sentence per line
2023-09-06 09:30:25 -04:00
John Ericson
754528241a
Merge pull request #8927 from obsidiansystems/test-derivation-aterm
Test and begin documentation of the ATerm format for derivations
2023-09-06 09:24:52 -04:00
John Ericson
d568877eab
Retitle section as Robert suggests
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-06 08:43:16 -04:00
Valentin Gagarin
391f18063c add anchors to option listings 2023-09-06 09:52:56 +02:00
Valentin Gagarin
02c2679f0e dedent common options listing; one sentence per line
this is a pure reformatting, contents were not changed

one sentence per line makes reviewing diffs and making suggestions much
more convenient. the indentation was an artifat of the DocBook
migration.
2023-09-06 09:49:00 +02:00
Valentin Gagarin
5c23d3a90c disambiguate output from output path 2023-09-06 09:09:02 +02:00
John Ericson
880d9cabed Test and begin documentation of the ATerm format for derivations
Wanted to do this before the last dynamic derivations PR when I
introduce a variation, to make sure I wasn't changing the old version by
mistake.
2023-09-05 11:16:39 -04:00
John Ericson
7ff43435f9 Unit test some worker protocol serializers
Continue with the characterization testing idioms begun in
c70484454f, but this time for unit tests.

Co-authored-by: Andreas Rammhold <andreas@rammhold.de>
2023-09-05 10:48:02 -04:00
Valentin Gagarin
f34484d18a
Merge pull request #7592 from fricklerhandwerk/nix-store-realise
reword description of how realisation works
2023-09-05 11:55:01 +02:00
Eelco Dolstra
919781cacc
Merge branch 'master' into valid_deriver_2 2023-09-01 13:35:05 +02:00
Valentin Gagarin
1ac181759d revert some random change 2023-08-31 21:27:13 +02:00
Valentin Gagarin
d38a539437 make description open-ended, add TODO
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-31 21:27:11 +02:00
Valentin Gagarin
894cbe43bc don't invent terms yet 2023-08-31 21:26:46 +02:00