Document string context (#8595)

* Document string context

Now what we have enough primops, we can document how string contexts
work.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Felix Uhl <iFreilicht@users.noreply.github.com>

doc/manual/src/SUMMARY.md.in

@@ -27,6 +27,7 @@
   - [Language Constructs](language/constructs.md)
     - [String interpolation](language/string-interpolation.md)
     - [Lookup path](language/constructs/lookup-path.md)
+    - [String context](language/string-context.md)
   - [Operators](language/operators.md)
   - [Derivations](language/derivations.md)
     - [Advanced Attributes](language/advanced-attributes.md)

doc/manual/src/glossary.md

@@ -218,6 +218,17 @@
 - [output closure]{#gloss-output-closure}\
   The [closure] of an [output path]. It only contains what is [reachable] from the output.
 
+- [deriving path]{#gloss-deriving-path}
+
+  Deriving paths are a way to refer to [store objects][store object] that ar not yet [realised][realise].
+  This is necessary because, in general and particularly for [content-addressed derivations][content-addressed derivation], the [output path] of an [output] is not known in advance.
+  There are two forms:
+
+  - *constant*: just a [store path]
+    It can be made [valid][validity] by copying it into the store: from the evaluator, command line interface or another store.
+
+  - *output*: a pair of a [store path] to a [derivation] and an [output] name.
+
 - [deriver]{#gloss-deriver}
 
   The [store derivation] that produced an [output path].

doc/manual/src/language/string-context.md

@@ -0,0 +1,134 @@
+# String context
+
+> **Note**
+>
+> This is an advanced topic.
+> The Nix language is designed to be used without the programmer consciously dealing with string contexts or even knowing what they are.
+
+A string in the Nix language is not just a sequence of characters like strings in other languages.
+It is actually a pair of a sequence of characters and a *string context*.
+The string context is an (unordered) set of *string context elements*.
+
+The purpose of string contexts is to collect non-string values attached to strings via
+[string concatenation](./operators.md#string-concatenation),
+[string interpolation](./string-interpolation.md),
+and similar operations.
+The idea is that a user can combine together values to create a build instructions for derivations without manually keeping track of where they come from.
+Then the Nix language implicitly does that bookkeeping to efficiently obtain the closure of derivation inputs.
+
+> **Note**
+>
+> String contexts are *not* explicitly manipulated in idiomatic Nix language code.
+
+String context elements come in different forms:
+
+- [deriving path]{#string-context-element-derived-path}
+
+  A string context element of this type is a [deriving path](@docroot@/glossary.md#gloss-deriving-path).
+  They can be either of type [constant](#string-context-constant) or [output](#string-context-output), which correspond to the types of deriving paths.
+
+  - [Constant string context elements]{#string-context-constant}
+
+    > **Example**
+    >
+    > [`builtins.storePath`] creates a string with a single constant string context element:
+    >
+    > ```nix
+    > builtins.getContext (builtins.storePath "/nix/store/wkhdf9jinag5750mqlax6z2zbwhqb76n-hello-2.10")
+    > ```
+    > evaluates to
+    > ```nix
+    > {
+    >   "/nix/store/wkhdf9jinag5750mqlax6z2zbwhqb76n-hello-2.10" = {
+    >     path = true;
+    >   };
+    > }
+    > ```
+
+    [deriving path]: @docroot@/glossary.md#gloss-deriving-path
+    [store path]: @docroot@/glossary.md#gloss-store-path
+    [`builtins.storePath`]: ./builtins.md#builtins-storePath
+
+  - [Output string context elements]{#string-context-output}
+
+    > **Example**
+    >
+    > The behavior of string contexts are best demonstrated with a built-in function that is still experimental: [`builtins.outputOf`].
+    > This example will *not* work with stable Nix!
+    >
+    > ```nix
+    > builtins.getContext
+    >   (builtins.outputOf
+    >     (builtins.storePath "/nix/store/fvchh9cvcr7kdla6n860hshchsba305w-hello-2.12.drv")
+    >     "out")
+    > ```
+    > evaluates to
+    > ```nix
+    > {
+    >   "/nix/store/fvchh9cvcr7kdla6n860hshchsba305w-hello-2.12.drv" = {
+    >     outputs = [ "out" ];
+    >   };
+    > }
+    > ```
+
+    [`builtins.outputOf`]: ./builtins.md#builtins-outputOf
+
+- [*derivation deep*]{#string-context-element-derivation-deep}
+
+  *derivation deep* is an advanced feature intended to be used with the
+  [`exportReferencesGraph` derivation attribute](./advanced-attributes.html#adv-attr-exportReferencesGraph).
+  A *derivation deep* string context element is a derivation path, and refers to both its outputs and the entire build closure of that derivation:
+  all its outputs, all the other derivations the given derivation depends on, and all the outputs of those.
+
+  > **Example**
+  >
+  > The best way to illustrate *derivation deep* string contexts is with [`builtins.addDrvOutputDependencies`].
+  > Take a regular constant string context element pointing to a derivation, and transform it into a "Derivation deep" string context element.
+  >
+  > ```nix
+  > builtins.getContext
+  >   (builtins.addDrvOutputDependencies
+  >     (builtins.storePath "/nix/store/fvchh9cvcr7kdla6n860hshchsba305w-hello-2.12.drv"))
+  > ```
+  > evaluates to
+  > ```nix
+  > {
+  >   "/nix/store/fvchh9cvcr7kdla6n860hshchsba305w-hello-2.12.drv" = {
+  >     allOutputs = true;
+  >   };
+  > }
+  > ```
+
+  [`builtins.addDrvOutputDependencies`]: ./builtins.md#builtins-addDrvOutputDependencies
+  [`builtins.unsafeDiscardOutputDependency`]: ./builtins.md#builtins-unsafeDiscardOutputDependency
+
+## Inspecting string contexts
+
+Most basically, [`builtins.hasContext`] will tell whether a string has a non-empty context.
+
+When more granular information is needed, [`builtins.getContext`] can be used.
+It creates an [attribute set] representing the string context, which can be inspected as usual.
+
+[`builtins.hasContext`]: ./builtins.md#builtins-hasContext
+[`builtins.getContext`]: ./builtins.md#builtins-getContext
+[attribute set]: ./values.md#attribute-set
+
+## Clearing string contexts
+
+[`buitins.unsafeDiscardStringContext`](./builtins.md#builtins-unsafeDiscardStringContext) will make a copy of a string, but with an empty string context.
+The returned string can be used in more ways, e.g. by operators that require the string context to be empty.
+The requirement to explicitly discard the string context in such use cases helps ensure that string context elements are not lost by mistake.
+The "unsafe" marker is only there to remind that Nix normally guarantees that dependencies are tracked, whereas the returned string has lost them.
+
+## Constructing string contexts
+
+[`builtins.appendContext`] will create a copy of a string, but with additional string context elements.
+The context is specified explicitly by an [attribute set] in the format that [`builtins.hasContext`] produces.
+A string with arbitrary contexts can be made like this:
+
+1. Create a string with the desired string context elements.
+   (The contents of the string do not matter.)
+2. Dump its context with [`builtins.getContext`].
+3. Combine it with a base string and repeated [`builtins.appendContext`] calls.
+
+[`builtins.appendContext`]: ./builtins.md#builtins-appendContext

src/libexpr/primops/context.cc

@@ -14,8 +14,11 @@ static void prim_unsafeDiscardStringContext(EvalState & state, const PosIdx pos,
 
 static RegisterPrimOp primop_unsafeDiscardStringContext({
     .name = "__unsafeDiscardStringContext",
-    .arity = 1,
+    .args = {"s"},
-    .fun = prim_unsafeDiscardStringContext
+    .doc = R"(
+        Discard the [string context](@docroot@/language/string-context.md) from a value that can be coerced to a string.
+    )",
+    .fun = prim_unsafeDiscardStringContext,
 });
 
 
@@ -75,7 +78,11 @@ static RegisterPrimOp primop_unsafeDiscardOutputDependency({
     .name = "__unsafeDiscardOutputDependency",
     .args = {"s"},
     .doc = R"(
-      Create a copy of the given string where every "derivation deep" string context element is turned into a constant string context element.
+      Create a copy of the given string where every
+      [derivation deep](@docroot@/language/string-context.md#string-context-element-derivation-deep)
+      string context element is turned into a
+      [constant](@docroot@/language/string-context.md#string-context-element-constant)
+      string context element.
 
       This is the opposite of [`builtins.addDrvOutputDependencies`](#builtins-addDrvOutputDependencies).
 
@@ -137,7 +144,11 @@ static RegisterPrimOp primop_addDrvOutputDependencies({
     .name = "__addDrvOutputDependencies",
     .args = {"s"},
     .doc = R"(
-      Create a copy of the given string where a single constant string context element is turned into a "derivation deep" string context element.
+      Create a copy of the given string where a single
+      [constant](@docroot@/language/string-context.md#string-context-element-constant)
+      string context element is turned into a
+      [derivation deep](@docroot@/language/string-context.md#string-context-element-derivation-deep)
+      string context element.
 
       The store path that is the constant string context element should point to a valid derivation, and end in `.drv`.