Commit graph

864 commits

Author SHA1 Message Date
Eelco Dolstra
ab35cbd675 StaticEnv: Use std::vector instead of std::map 2021-11-04 15:03:34 +01:00
regnat
5b2aa61f1b Don’t require ca-derivations when __contentAddressed = false
If we explicitely opt-out of it, there’s no need to require the
experimental feature
2021-11-03 06:51:32 +01:00
regnat
af99941279 Make experimental-features a proper type
Rather than having them plain strings scattered through the whole
codebase, create an enum containing all the known experimental features.

This means that
- Nix can now `warn` when an unkwown experimental feature is passed
  (making it much nicer to spot typos and spot deprecated features)
- It’s now easy to remove a feature altogether (once the feature isn’t
  experimental anymore or is dropped) by just removing the field for the
  enum and letting the compiler point us to all the now invalid usages
  of it.
2021-10-26 07:02:31 +02:00
Shay Bergmann
ba81e871b2
toJSON: report error position for fancier output
Given flake:

```nix
{ description = "nix json error provenance";
  inputs = {};
  outputs = { self }: {
    jsonFunction = _: "function";
    json = builtins.toJSON (_: "function");
  };
}

```
- Before:

```console
❯ nix eval --json .#jsonFunction
error: cannot convert a function to JSON
```

- After:

```console
❯ nix eval --json .#jsonFunction
error: cannot convert a function to JSON

       at /nix/store/b7imf1c2j4jnkg3ys7fsfbj02s5j0i4f-source/testflake/flake.nix:4:5:

            3|   outputs = { self }: {
            4|     jsonFunction = _: "function";
             |     ^
            5|     json = builtins.toJSON (_: "function");
```
2021-10-25 21:17:52 +00:00
Ben Burdette
e54f17eb46 remove more debug code 2021-10-22 14:27:04 -06:00
Ben Burdette
427fb8d158 comment out debugs 2021-10-11 16:48:10 -06:00
regnat
7466048d39 (partially) Revert "Don't copy in rethrow"
This reverts some parts of commit
8430a8f086 which was trying to rethrow
some exceptions while we weren’t in the context of a `catch` block,
causing some weird “terminate called without an active exception”
errors.

Fix #5368
2021-10-11 10:51:22 +02:00
John Ericson
195daa8299 Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2021-10-08 23:59:15 +00:00
Eelco Dolstra
d39692e6b3 Make builtins.{path,filterSource} work with chroot stores 2021-10-07 14:22:39 +02:00
Eelco Dolstra
c4dcf3cf25 Add a trace to all errors in addPath() 2021-10-07 13:47:15 +02:00
Eelco Dolstra
4806f2f6b0 Allow builtins.{path,filterSource} on paths with a context
We now build the context (so this has the side-effect of making
builtins.{path,filterSource} work on derivations outputs, if IFD is
enabled) and then check that the path has no references (which is what
we really care about).
2021-10-07 13:43:17 +02:00
Eelco Dolstra
7c50568788 Remove unnecessary call to queryMissing()
Worker::run() already does this.
2021-10-07 13:15:01 +02:00
Eelco Dolstra
cfaad7168e Refactoring: Add allowPath() method 2021-10-07 12:11:00 +02:00
Eelco Dolstra
c9ee634f75
Merge pull request #5341 from andir/libexpr-formals
libexpr: remove matchAttrs boolean from ExprLambda
2021-10-07 11:58:56 +02:00
Eelco Dolstra
53e4794289
Merge pull request #5286 from ilkecan/add-a-warning-to-filterSource
Warn about the usage of filterSource with Nix store paths
2021-10-06 21:02:39 +02:00
ilkecan
a4a6ef4fb2 Add a warning to filterSource
Warn about the usage of `filterSource` with Nix store paths
2021-10-06 19:25:33 +03:00
Eelco Dolstra
0dc8172458 Remove no-op call to realiseContext() 2021-10-06 18:08:37 +02:00
Eelco Dolstra
c497fce011 Merge branch 'flakes_filterSource' of https://github.com/tomberek/nix 2021-10-06 18:08:18 +02:00
Andreas Rammhold
cae41eebff libexpr: remove matchAttrs boolean from ExprLambda
The boolean is only used to determine if the formals are set to a
non-null pointer in all our cases. We can get rid of that allocation and
instead just compare the pointer value with NULL. Saving up to
sizeof(bool) + platform specific alignment per ExprLambda instace.
Probably not a lot of memory but perhaps a few kilobyte with nixpkgs?

This also gets rid of a potential issue with dereferencing formals based on
the value of the boolean that didn't have to be aligned with the formals
pointer but was in all our cases.
2021-10-06 17:24:06 +02:00
John Ericson
edf67e1508 Merge branch 'path-info' into ca-drv-exotic 2021-10-01 17:25:22 +00:00
John Ericson
13b6b64589 Merge remote-tracking branch 'upstream/master' into path-info 2021-10-01 17:12:54 +00:00
John Ericson
9af9ab4212 Merge branch 'path-info' into ca-drv-exotic 2021-09-30 22:42:15 +00:00
John Ericson
f4f3203aa7 Merge remote-tracking branch 'upstream/master' into path-info 2021-09-30 22:41:53 +00:00
John Ericson
242f9bf3dc std::visit by reference
I had started the trend of doing `std::visit` by value (because a type
error once mislead me into thinking that was the only form that
existed). While the optomizer in principle should be able to deal with
extra coppying or extra indirection once the lambdas inlined, sticking
with by reference is the conventional default. I hope this might even
improve performance.
2021-09-30 21:35:09 +00:00
Eelco Dolstra
fd01c48d34
Merge pull request #5301 from Ma27/builtins-missing-feature-error
libexpr: throw a more helpful eval-error if a builtin is not available due to a missing feature-flag
2021-09-29 12:53:29 +02:00
Maximilian Bosch
2b02ce0e48
libexpr: throw a more helpful eval-error if a builtin is not available due to a missing feature-flag
I found it somewhat confusing to have an error like

    error: attribute 'getFlake' missing

if the required experimental-feature (`flakes`) is not enabled. Instead,
I'd expect Nix to throw an error just like it's the case when using e.g. `nix
flake` without `flakes` being enabled.

With this change, the error looks like this:

    $ nix-instantiate -E 'builtins.getFlake "nixpkgs"'
    error: Cannot call 'builtins.getFlake' because experimental Nix feature 'flakes' is disabled. You can enable it via '--extra-experimental-features flakes'.

           at «string»:1:1:

                1| builtins.getFlake "nixpkgs"
                 | ^

I didn't use `settings.requireExperimentalFeature` here on purpose
because this doesn't contain a position. Also, it doesn't seem as if we
need to catch the error and check for the missing feature here since
this already happens at evaluation time.
2021-09-29 11:57:15 +02:00
Eelco Dolstra
8430a8f086 Don't copy in rethrow 2021-09-27 14:38:10 +02:00
Ben Burdette
c7e3d830c1 more debug stuff 2021-09-22 16:22:53 -06:00
Eelco Dolstra
ff28fffce2 Don't cache realiseContext() errors
Errors that depend on the configuration (such as whether
allow-import-from-derivation is set) should not be cached.
2021-09-22 14:00:56 +02:00
Ben Burdette
cd8c232b55 add cout debugging 2021-09-15 16:16:53 -06:00
Eelco Dolstra
991cc53386 Revert "Disallow reading flake.lock"
This reverts commit e5596113f7.
2021-09-15 18:30:37 +02:00
Eelco Dolstra
e5596113f7 Disallow reading flake.lock
With --no-write-lock-file, it's possible that flake.lock is out of
sync with the actual inputs used by the evaluation. So doing fromJSON
(readFile ./flake.lock) will give wrong results.

Fixes #4639.
2021-09-14 21:09:11 +02:00
Ben Burdette
21071bfdeb shared_ptr for StaticEnv 2021-09-14 10:49:22 -06:00
kvtb
c6fa7775de hashFile, hashString: realize context before calculation, and discard afterwards 2021-09-13 22:34:58 +02:00
Eelco Dolstra
323cafcb4e
Merge pull request #5191 from hercules-ci/evalstate-lifetime-hygiene
EvalState lifetime hygiene
2021-08-30 12:23:09 +02:00
Robert Hensing
8656b130ea Fix use after free with vImportedDrvToDerivation 2021-08-29 20:42:57 +02:00
Robert Hensing
f10465774f Force all Pos* to be non-null
This fixes a class of crashes and introduces ptr<T> to make the
code robust against this failure mode going forward.

Thanks regnat for the idea of a ref<T> without overhead!

Closes #4895
Closes #4893
Closes #5127
Closes #5113
2021-08-29 18:11:58 +02:00
Tom Bereknyei
d90582be33 Allow use of path and filterSource in flakes
As filterSource and path perform work, add paths to allowedPaths.
2021-08-22 18:45:42 -04:00
Eelco Dolstra
8943e3176d
Merge pull request #5111 from Pamplemousse/clean
Minor maintenance cleaning
2021-08-09 20:05:03 +02:00
Pamplemousse
2de7a1fe67 libexpr: Squash similar conditions
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-08-09 10:10:11 -07:00
Niels Egberts
ae0ed53b09 toString also coerces a set with an outPath attribute to a string
nix-repl> builtins.toString { outPath = "somestring"; }
"somestring"
2021-07-09 21:50:10 +01:00
Maximilian Bosch
bec33858bb
primops: math.h -> cmath
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-05-10 16:41:10 +02:00
Maximilian Bosch
7f7f99f350
Implement builtins.floor and builtins.ceil using the C library functions internally
Closes #4782

Note: even though the type is internally called `NixFloat`, it's
actually a `double`.
2021-05-10 12:19:32 +02:00
Domen Kožar
e3e78ee2a2
Merge pull request #4751 from Ma27/storepath-pos
primops/storePath: add trace to pure mode error
2021-04-27 19:05:23 +02:00
Maximilian Bosch
e6a360a4d9
primops/storePath: add trace to pure mode error
As described in #4745 it's otherwise fairly hard to understand where
this is coming from. Say you have an expression which uses e.g.
`types.package`:

``` nix
{ outputs = { self, nixpkgs }: {
    packages.x86_64-linux.hello = let
      foo = nixpkgs.lib.evalModules {
        modules = [
          {
            options.foo.bar = with nixpkgs.lib; mkOption { type = types.package; };
          }
          {
            foo.bar = ./.;
          }
        ];
      };
    in builtins.trace foo.config.foo.bar.outPath nixpkgs.legacyPackages.x86_64-linux.hello;
    defaultPackage.x86_64-linux = self.packages.x86_64-linux.hello;
  };
}
```

Then you'll get an error trace like this:

```
error: 'builtins.storePath' is not allowed in pure evaluation mode

       at /nix/store/p4h2x6r80njkb0j2rc1xjhhl99yri3zb-source/lib/attrsets.nix:328:15:

          327|     let
          328|       path' = builtins.storePath path;
             |               ^
          329|       res =

       … while evaluating the attribute 'config.foo.bar.outPath'

       at /nix/store/p4h2x6r80njkb0j2rc1xjhhl99yri3zb-source/lib/attrsets.nix:332:11:

          331|           name = sanitizeDerivationName (builtins.substring 33 (-1) (baseNameOf path'));
          332|           outPath = path';
             |           ^
          333|           outputs = [ "out" ];

       … while evaluating the attribute 'packages.x86_64-linux.hello'

       at /nix/store/6c1rfsqzrhjw1235palzjmf5vihcpci7-source/flake.nix:3:5:

            2| { outputs = { self, nixpkgs }: {
            3|     packages.x86_64-linux.hello = let
             |     ^
            4|       foo = nixpkgs.lib.evalModules {
```

Fixes #4745
2021-04-27 17:24:38 +02:00
regnat
31313d1401 Replace the trailing markdown spaces by a backslash
They are equivalent according to
<https://spec.commonmark.org/0.29/#hard-line-breaks>,
and the trailing spaces tend to be a pain (because the make git
complain, editors tend to want to remove them − the `.editorconfig`
actually specifies that − etc..).
2021-04-23 14:37:21 +02:00
Maximilian Bosch
864ef0e93d
libexpr/primops: review 2021-04-13 23:12:39 +02:00
Maximilian Bosch
ad7b47dc85
primops/libexpr: use new attr-call extractor everywhere; use function's pos if attr-set pos == noPos 2021-04-13 23:12:38 +02:00
Maximilian Bosch
f60473a077
libexpr/primops: Move attr name extraction into its own function
This now takes care of providing positioning for both the faulting value
and the faulting function call in case of an error.
2021-04-13 23:12:38 +02:00
Maximilian Bosch
7c76964daa
libexpr: misc improvements for proper error position
When working on some more complex Nix code, there are sometimes rather
unhelpful or misleading error messages, especially if coerce-errors are
thrown.

This patch is a first steps towards improving that. I'm happy to file
more changes after that, but I'd like to gather some feedback first.

To summarize, this patch does the following things:

* Attrsets (a.k.a. `Bindings` in `libexpr`) now have a `Pos`. This is
  helpful e.g. to identify which attribute-set in `listToAttrs` is
  invalid.

* The `Value`-struct has a new method named `determinePos` which tries
  to guess the position of a value and falls back to a default if that's
  not possible.

  This can be used to provide better messages if a coercion fails.

* The new `determinePos`-API is used by `builtins.concatMap` now. With
  that change, Nix shows the exact position in the error where a wrong
  value was returned by the lambda.

  To make sure it's still obvious that `concatMap` is the problem,
  another stack-frame was added.

* The changes described above can be added to every other `primop`, but
  first I'd like to get some feedback about the overall approach.
2021-04-13 23:12:38 +02:00
Maximilian Bosch
3550a32b25
primops/derivation: use position of currently evaluated attribute
* The position of the `name`-attribute appears in the trace.
* If e.g. `meta` has no `outPath`-attribute, a `cannot coerce set to
  string` error will be thrown where `pos` points to `name =` which is
  highly misleading.
2021-04-13 23:12:38 +02:00
John Ericson
d0ed11ca72 Merge commit '1b6cf0d5f56e166a1cbbf38142375b7a92fc88f2' into ca-drv-exotic 2021-04-05 19:06:43 -04:00
John Ericson
386765e3ff Merge commit 'd5cef6c33a051dfc672cb1e5f4739948b167315b' into ca-drv-exotic 2021-04-05 19:06:37 -04:00
John Ericson
1b6cf0d5f5 Merge remote-tracking branch 'upstream/master' into path-info 2021-04-05 18:47:33 -04:00
John Ericson
d5cef6c33a Merge commit '9dfb97c987d8b9d6a3d15f016e40f22f91deb764' into path-info 2021-04-05 18:40:30 -04:00
John Ericson
9b805d36ac Rename Buildable 2021-04-05 09:52:25 -04:00
John Ericson
255d145ba7 Use BuildableReq for buildPaths and ensurePath
This avoids an ambiguity where the `StorePathWithOutputs { drvPath, {}
}` could mean "build `brvPath`" or "substitute `drvPath`" depending on
context.

It also brings the internals closer in line to the new CLI, by
generalizing the `Buildable` type is used there and makes that
distinction already.

In doing so, relegate `StorePathWithOutputs` to being a type just for
backwards compatibility (CLI and RPC).
2021-04-05 08:33:00 -04:00
John Ericson
90d76fa399 Merge remote-tracking branch 'obsidian/path-info' into ca-drv-exotic 2021-02-25 21:58:41 +00:00
John Ericson
ca0994819d Merge remote-tracking branch 'upstream/master' into path-info 2021-02-25 21:51:05 +00:00
sternenseemann
76d8bdfe35 Include note about type of catched errors in tryEval documentation
Reference #356.
2021-02-03 17:14:40 +01:00
Eelco Dolstra
12de0466fe Add trace to build errors during import-from-derivation
Example:

error: builder for '/nix/store/9ysqfidhipyzfiy54mh77iqn29j6cpsb-failing.drv' failed with exit code 1;
       last 1 log lines:
       > FAIL
       For full logs, run 'nix log /nix/store/9ysqfidhipyzfiy54mh77iqn29j6cpsb-failing.drv'.

       … while importing '/nix/store/pfp4a4bjh642ylxyipncqs03z6kkgfvy-failing'

       at /nix/store/25wgzr2qrqqiqfbdb1chpiry221cjglc-source/flake.nix:58:15:

           57|
           58|         ifd = import self.hydraJobs.broken;
             |               ^
           59|
2021-01-27 14:46:10 +01:00
Eelco Dolstra
8d4268d190 Improve error formatting
Changes:

* The divider lines are gone. These were in practice a bit confusing,
  in particular with --show-trace or --keep-going, since then there
  were multiple lines, suggesting a start/end which wasn't the case.

* Instead, multi-line error messages are now indented to align with
  the prefix (e.g. "error: ").

* The 'description' field is gone since we weren't really using it.

* 'hint' is renamed to 'msg' since it really wasn't a hint.

* The error is now printed *before* the location info.

* The 'name' field is no longer printed since most of the time it
  wasn't very useful since it was just the name of the exception (like
  EvalError). Ideally in the future this would be a unique, easily
  googleable error ID (like rustc).

* "trace:" is now just "…". This assumes error contexts start with
  something like "while doing X".

Example before:

  error: --- AssertionError ---------------------------------------------------------------------------------------- nix
  at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

       6|
       7|   x = assert false; 1;
        |       ^
       8|

  assertion 'false' failed
  ----------------------------------------------------- show-trace -----------------------------------------------------
  trace: while evaluating the attribute 'x' of the derivation 'hello-2.10'
  at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix

     191|         // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) {
     192|           name = "${attrs.pname}-${attrs.version}";
        |           ^
     193|         } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {

Example after:

  error: assertion 'false' failed

         at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

              6|
              7|   x = assert false; 1;
               |       ^
              8|

         … while evaluating the attribute 'x' of the derivation 'hello-2.10'

         at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix

            191|         // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) {
            192|           name = "${attrs.pname}-${attrs.version}";
               |           ^
            193|         } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {
2021-01-21 11:02:09 +01:00
Eelco Dolstra
75efa42134 Move <nix/fetchurl.nix> into the nix binary
This makes the statically linked nix binary just work, without needing
any additional files.
2020-12-22 14:43:20 +01:00
Eelco Dolstra
ec4a5c5b0b
Merge pull request #4355 from Infinisil/private-value-type
Refactoring for private Value type
2020-12-21 12:38:47 +01:00
Silvan Mosberger
b70d22baca
Replace Value type setters with mk* functions
Move clearValue inside Value

mkInt instead of setInt

mkBool instead of setBool

mkString instead of setString

mkPath instead of setPath

mkNull instead of setNull

mkAttrs instead of setAttrs

mkList instead of setList*

mkThunk instead of setThunk

mkApp instead of setApp

mkLambda instead of setLambda

mkBlackhole instead of setBlackhole

mkPrimOp instead of setPrimOp

mkPrimOpApp instead of setPrimOpApp

mkExternal instead of setExternal

mkFloat instead of setFloat

Add note that the static mk* function should be removed eventually
2020-12-18 21:48:22 +01:00
Silvan Mosberger
12e65078ef
Rename Value::normalType() -> Value::type() 2020-12-17 14:45:45 +01:00
Eelco Dolstra
3765174691
Merge pull request #4348 from NixOS/ca/use-hashmodulo
Use the hash modulo in the derivation outputs
2020-12-16 12:48:44 +01:00
Maximilian Bosch
f890830b33
primops/fromJSON: add error position in case of parse error
This makes it easier to track down where invalid JSON was passed to
`builtins.fromJSON`.
2020-12-13 13:55:32 +01:00
Silvan Mosberger
bf98903967
Add ValueType checking functions for types that have the same NormalType 2020-12-12 03:31:50 +01:00
Silvan Mosberger
22ead43a0b
Use Value::normalType on all forced values instead of Value::type 2020-12-12 03:31:48 +01:00
regnat
bab1cda0e6 Use the hash modulo in the derivation outputs
Rather than storing the derivation outputs as `drvPath!outputName` internally,
store them as `drvHashModulo!outputName` (or `outputHash!outputName` for
fixed-output derivations).

This makes the storage slightly more opaque, but enables an earlier
cutoff in cases where a fixed-output dependency changes (but keeps the
same output hash) − same as what we already do for input-addressed
derivations.
2020-12-11 21:17:23 +01:00
John Ericson
2113ae2d85 Make drv hash modulo memo table thread-safe
Let's get one step closer to the daemon not needing to fork.
2020-11-19 16:50:06 +00:00
regnat
c092fa4702 Allow non-CA derivations to depend on CA derivations 2020-10-27 07:29:23 +01:00
John Ericson
a4e5de1b9d Derivations can output "text-hashed" data
In particular, this means that derivations can output derivations. But
that ramification isn't (yet!) useful as we would want, since there is
no way to have a dependent derivation that is itself a dependent
derivation.
2020-10-13 02:15:48 +00:00
John Ericson
f8d562c0a7 Use PathReferences more widely 2020-10-07 15:00:10 +00:00
Eelco Dolstra
cbb9862cd9
Merge pull request #3626 from W95Psp/master
Make `functionArgs` primitive accept primops (fix #3624)
2020-09-25 15:14:18 +02:00
Eelco Dolstra
e8e1d420f3 Don't include <regex> in header files
This reduces compilation time by ~15 seconds (CPU time).

Issue #4045.
2020-09-21 18:22:45 +02:00
Eelco Dolstra
10d1865f5f Remove corepkgs/derivation.nix 2020-09-17 09:41:02 +02:00
Eelco Dolstra
0066ef6c59 Fix doc generation 2020-09-16 16:56:28 +02:00
Eelco Dolstra
2eacc1bc00 builtins.toFile: Fix indentation 2020-09-16 14:18:46 +02:00
John Ericson
3a5cdd737c Rename Derivation::pathOpt to Derivation::path
We no longer need the `*Opt` to disambiguate.
2020-09-15 15:21:39 +00:00
John Ericson
5aed6f9b25 Document mkOutputString 2020-09-04 15:58:42 +00:00
John Ericson
ef278d00f9 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-01 18:01:48 +00:00
Eelco Dolstra
84f5cabbea Merge remote-tracking branch 'origin/master' into markdown 2020-08-31 14:24:26 +02:00
John Ericson
8017fe7487 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-28 19:59:14 +00:00
Eelco Dolstra
eb75282b8d
Merge pull request #3434 from Ericson2314/derivation-header-include-order
Revise division of labor in deserialization of derivations
2020-08-27 16:39:28 +02:00
Eelco Dolstra
4bf5faf416 Merge remote-tracking branch 'origin/master' into markdown 2020-08-25 19:47:34 +02:00
Eelco Dolstra
7a02865b94
Move import docs 2020-08-25 14:06:01 +02:00
Eelco Dolstra
2a2121d264
Use RegisterPrimOp for some undocumented primops 2020-08-25 11:25:01 +02:00
Eelco Dolstra
b8416779e3
Document some primops 2020-08-25 11:16:45 +02:00
Eelco Dolstra
a990f063ff
Move primop docs inline
This makes them available to 'nix repl'.
2020-08-24 14:31:10 +02:00
Eelco Dolstra
33b1679d75
Allow primops to have Markdown documentation 2020-08-24 13:16:02 +02:00
John Ericson
35e6288be1 writeDerivation just needs a plain store reference 2020-08-23 15:01:11 +00:00
John Ericson
3a7b330b64 "Downstream placeholders" should not be store paths
Insead they should be opaque `/<hash>` like the placeholders we already
have.
2020-08-21 19:35:35 +00:00
John Ericson
27a3f82c0b Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-20 18:28:17 +00:00
John Ericson
45a2f1baab Rename drv output querying functions, like master
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
 - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap
2020-08-20 18:14:12 +00:00
John Ericson
950ddfdb82 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-08-18 14:36:44 +00:00
Eelco Dolstra
1c8b550e34
Merge pull request #3917 from obsidiansystems/output-env-var-unconditional
Simplify code as output env vars are unconditional
2020-08-18 16:21:17 +02:00
John Ericson
3c8b5b6219 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-14 17:00:13 +00:00
Eelco Dolstra
13e49be660
Merge pull request #3875 from obsidiansystems/new-interface-for-path-pathOpt
Offer a safer interface for path and pathOpt
2020-08-14 17:19:19 +02:00
John Ericson
d3fa8c04c6 Simplify code as output env vars are unconditional
Since the jsonObject unique ptr is reset to flush the string to make
`__json`, all these `!jsonObject` conditions will always be true.
2020-08-11 01:13:26 +00:00
John Ericson
1b5c24662b Merge branch 'small-drv-serialize-cleanup' of github.com:obsidiansystems/nix into single-ca-drv-build 2020-08-10 01:57:54 +00:00
John Ericson
bcd0629c2e Remove name parameter from writeDerivation
The name is now stored with the derivation itself.
2020-08-10 01:35:59 +00:00
John Ericson
e913a2989f Squashed get CA derivations building 2020-08-07 19:51:55 +00:00
Carlo Nucera
1d2e80ddd6 Merge branch 'master' of github.com:NixOS/nix into new-interface-for-path-pathOpt 2020-08-05 15:45:33 -04:00
John Ericson
b3e73547a0
Update src/libexpr/primops.cc
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-08-05 11:05:46 -04:00
John Ericson
92ad550e96 Merge remote-tracking branch 'obsidian/misc-ca' into derivation-primop-floating-output 2020-08-05 14:51:41 +00:00
John Ericson
b9ebe373bb Sed some names to perhaps avoid conflicts 2020-08-05 14:49:25 +00:00
John Ericson
e7b0847f2d Make names more consistent 2020-08-05 14:44:39 +00:00
John Ericson
03f4fafc27 Merge remote-tracking branch 'upstream/master' into misc-ca 2020-08-05 14:36:25 +00:00
John Ericson
9357512d73 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-08-01 19:38:35 +00:00
Matthew Bauer
cdc2386644 Make expectedHash optional in prim_path
This fixes an error found in builtins.path that looks like:

store path mismatch in (possibly filtered) path added from '/private/tmp/nix-shell.CyXViH/nix-test/filter-source/filterin'

when no hash is specified
2020-07-30 16:40:40 -05:00
Eelco Dolstra
3f6e88a552 unsigned long long -> uint64_t 2020-07-30 13:34:04 +02:00
Carlo Nucera
c318d398f3 Merge branch 'misc-ca' of github.com:obsidiansystems/nix into new-interface-for-path-pathOpt 2020-07-28 14:22:24 -04:00
Carlo Nucera
7ef1e3cd14 Use the new interface 2020-07-28 13:59:24 -04:00
John Ericson
951415b568 Require ca-derivations everywhere we create a CA derivation
"create" as in read one in from a serialized form, or build one from
scratch in memory.
2020-07-27 17:56:36 +00:00
John Ericson
e32a9e124b Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output 2020-07-27 17:50:06 +00:00
John Ericson
7cf978440c Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca 2020-07-27 17:47:40 +00:00
John Ericson
43f2bd8dc5 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-27 16:13:57 +00:00
John Ericson
9423f64ee2 Parse CA derivations using new output variants
We no longer need `ParsedDerivation` because everything libstore needs
to know about is in the `BasicDerivation` proper.
2020-07-22 23:59:25 +00:00
Carlo Nucera
5cb840541b Merge branch 'multi-output-hashDerivationModulo' of github.com:Ericson2314/nix into misc-ca 2020-07-17 10:28:33 -04:00
Carlo Nucera
048e916f64 Merge branch 'master' of github.com:NixOS/nix into optional-derivation-output-storepath 2020-07-16 13:32:28 -04:00
John Ericson
5ea817dace Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-16 14:58:53 +00:00
Carlo Nucera
455bdee205 Merge branch 'master' of github.com:NixOS/nix into derivation-header-include-order 2020-07-15 17:58:30 -04:00
Carlo Nucera
d090562348 Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type 2020-07-15 17:21:01 -04:00
Eelco Dolstra
832e111494 Merge remote-tracking branch 'origin/master' into flakes 2020-07-14 13:56:18 +02:00
Eelco Dolstra
c0dd05131e toStorePath(): Return a StorePath and the suffix 2020-07-13 16:25:48 +02:00
John Ericson
4415765385 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-13 03:01:44 +00:00
John Ericson
503b425690 DerivationOutputExtensional -> DerivationOutputInputAddressed
Thanks @regnat for the great name.
2020-07-12 15:56:20 +00:00
John Ericson
18152406ce String .drv suffix to create derivation name 2020-07-12 15:40:14 +00:00
John Ericson
13ec627e0a Set derivation name in dervationStrict 2020-07-12 03:03:12 +00:00
John Ericson
1c9bec226f Don't improperly assume path is store path 2020-07-12 02:38:03 +00:00
Matthew Bauer
a7884970c5 Fix DerivationOutputExtensional name 2020-07-09 11:37:18 -04:00
Matthew Bauer
8e0d0689be Only store hash of fixed derivation output
we don’t need a full storepath for a fixedoutput derivation. So just
putting the ingestion method + the hash is sufficient.
2020-07-08 19:11:39 -04:00
Matthew Bauer
af95a7c16b Add name to BasicDerivation
We always have a name for BasicDerivation, since we have a derivation
store path that has a name.
2020-07-08 15:38:01 -04:00
Eelco Dolstra
54712aaf8a Merge remote-tracking branch 'origin/master' into flakes 2020-07-06 16:40:10 +02:00
John Ericson
a38ab99d57 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-07-05 21:49:01 +00:00
Ben Burdette
3629b0585a don't include errpos for addErrorContext 2020-07-01 11:49:01 -06:00
Eelco Dolstra
50f13b06fb EvalCache: Store string contexts 2020-06-29 19:08:37 +02:00
Ben Burdette
023912def3 convenience form of addTrace 2020-06-24 13:46:25 -06:00
Ben Burdette
1d43a6e123 use plain errPos instead of nixCode; fix tests 2020-06-23 15:30:13 -06:00
John Ericson
98e5d1af03 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-06-23 17:03:37 +00:00
Ben Burdette
abe0552504 Merge remote-tracking branch 'upstream/master' into add-trace 2020-06-23 09:40:28 -06:00
John Ericson
8313f0e939 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-06-21 20:39:10 +00:00
John Ericson
fdeabf7160 Merge remote-tracking branch 'upstream/master' into multi-output-hashDerivationModulo 2020-06-21 16:43:17 +00:00
Carlo Nucera
e7a14118df WIP bug fixing 2020-06-19 16:50:28 -04:00
Ben Burdette
54e8f550c9 addErrorTrace 2020-06-19 13:44:08 -06:00
John Ericson
68294746ae Merge remote-tracking branch 'upstream/master' into no-hash-type-unknown 2020-06-19 17:53:34 +00:00
John Ericson
2f0e395c99 Merge remote-tracking branch 'me/no-stringly-typed-derivation-output' into validPathInfo-ca-proper-datatype 2020-06-19 15:26:59 +00:00
John Ericson
01dc8b0bab Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output 2020-06-19 14:59:05 +00:00
John Ericson
145d88cb2a Use designated initializers for DerivationOutputHash 2020-06-19 14:58:30 +00:00
John Ericson
237d88c97e FileSystemHash -> DerivationOutputHash 2020-06-19 14:47:10 +00:00
John Ericson
669c3992e8 Merge branch 'no-hash-type-unknown' into validPathInfo-temp 2020-06-18 22:33:07 +00:00
John Ericson
15abb2aa2b Revert the enum struct change
Not a regular git revert as there have been many merges and things.
2020-06-18 22:11:26 +00:00
John Ericson
bbbf3602a3 Merge branch 'enum-class' into no-hash-type-unknown 2020-06-18 22:11:19 +00:00
John Ericson
40526fbea5 Merge remote-tracking branch 'upstream/master' into enum-class 2020-06-18 21:38:15 +00:00
Ben Burdette
4d1a4f0217 addTrace 2020-06-18 15:25:26 -06:00
Eelco Dolstra
5771c8bbf2 Don't provide 'getFlake' if the 'flakes' feature is not enabled
(cherry picked from commit 0a1d3c1dd3)
2020-06-18 14:03:00 +02:00
Eelco Dolstra
2a61bbf77f Some backports from the flakes branch 2020-06-18 14:03:00 +02:00
John Ericson
517f5980e2 Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output 2020-06-17 04:58:43 +00:00
John Ericson
18493fd9c4 Move some Store functions from derivations.cc to store-api.cc
This further continues with the dependency inverstion. Also I just went
ahead and exposed `parseDerivation`: it seems like the more proper
building block, and not a bad thing to expose if we are trying to be
less wedded to drv files on disk anywas.
2020-06-17 04:30:16 +00:00
John Ericson
02928f76fd Merge remote-tracking branch 'upstream/master' into multi-output-hashDerivationModulo 2020-06-17 03:51:02 +00:00
Eelco Dolstra
29542865ce Remove StorePath::clone() and related functions 2020-06-16 22:20:18 +02:00
Eelco Dolstra
1fb762d11f Get rid of explicit ErrorInfo constructors 2020-06-15 14:06:58 +02:00
Eelco Dolstra
7a77762961 Merge branch 'errors-phase-2' of https://github.com/bburdette/nix 2020-06-15 11:46:31 +02:00
Eelco Dolstra
00fa7e2205
Merge pull request #3674 from matthewbauer/allow-empty-hash2
Allow empty hash in derivations
2020-06-12 18:18:12 +02:00
Matthew Bauer
b260c9ee03 Add newHashAllowEmpty helper function
This replaces the copy&paste with a helper function in hash.hh.
2020-06-12 10:11:16 -05:00
Eelco Dolstra
9f736dd89d Add Store::readDerivation() convenience function 2020-06-12 13:04:52 +02:00
Eelco Dolstra
045b07200c Remove Store::queryDerivationOutputNames()
This function was used in only one place, where it could easily be
replaced by readDerivation() since it's not
performance-critical. (This function appears to have been modelled
after queryDerivationOutputs(), which exists only to make the garbage
collector faster.)
2020-06-12 12:46:33 +02:00
Matthew Bauer
19aa892f20 Support empty hash in fetchers
fetchTarball, fetchTree, and fetchGit all have *optional* hash attrs.
This means that we need to be careful with what we allow to avoid
accidentally making these defaults. When ‘hash = ""’ we assume the
empty hash is wanted.
2020-06-09 11:10:54 -05:00
Matthew Bauer
762273f1fd Allow empty hash in derivations
follow up of https://github.com/NixOS/nix/pull/3544

This allows hash="" so that it can be used for debugging purposes. For
instance, this gives you an error message like:

  warning: found empty hash, assuming you wanted 'sha256:0000000000000000000000000000000000000000000000000000'
  hash mismatch in fixed-output derivation '/nix/store/asx6qw1r1xk6iak6y6jph4n58h4hdmbm-nix':
    wanted: sha256:0000000000000000000000000000000000000000000000000000
    got:    sha256:0fpfhipl9v1mfzw2ffmxiyyzqwlkvww22bh9wcy4qrfslb4jm429
2020-06-09 01:23:37 -05:00
John Ericson
450dcf2c1b Remove HashType::Unknown
Instead, `Hash` uses `std::optional<HashType>`. In the future, we may
also make `Hash` itself require a known hash type, encoraging people to
use `std::optional<Hash>` instead.
2020-06-02 15:52:13 +00:00
Ben Burdette
734283d636 Merge remote-tracking branch 'upstream/master' into errors-phase-2 2020-05-29 09:51:37 -06:00
Carlo Nucera
6dd471ebf6 Fixing the result of merge 2020-05-28 12:14:36 -04:00
Carlo Nucera
4f597fb901 Merge branch 'master' of github.com:NixOS/nix into enum-class 2020-05-28 10:58:22 -04:00
John Ericson
5b4cd84bc2 Merge remote-tracking branch 'me/more-rust-ffi' into no-stringly-typed-derivation-output 2020-05-28 10:35:53 -04:00
Matthew Bauer
c66441a646 Rename some variables named “recursive” to “method”
This is much less confusing since recursive is no longer a boolean.
2020-05-27 13:21:26 -05:00
Carlo Nucera
d49e65ba9d Merge remote-tracking branch 'john-ericson/enum-FileIngestionMethod' into no-stringly-typed-derivation-output 2020-05-26 12:30:48 -04:00
Carlo Nucera
6d73c10041 Merge remote-tracking branch 'origin/master' into enum-FileIngestionMethod 2020-05-26 11:14:08 -04:00
Lucas Franceschino
b2748c6e99 Make functionArgs primitive accept primops 2020-05-25 19:07:38 +02:00
Ben Burdette
4daccb279c formatting 2020-05-14 10:28:17 -06:00
Ben Burdette
ef9dd9f9bc formatting and a few minor changes 2020-05-13 15:56:39 -06:00
Ben Burdette
b93c1bf3d6 fixes to merged code 2020-05-11 15:52:15 -06:00
Ben Burdette
59b1f5c701 Merge branch 'master' into errors-phase-2 2020-05-11 14:35:30 -06:00
Ben Burdette
55eb717148 add pos to errorinfo, remove from hints 2020-05-08 18:18:28 -06:00
Ben Burdette
e4fb9a3849 remove 'format' from Error constructor calls 2020-04-21 17:07:07 -06:00
Domen Kožar
25ed842725
Merge pull request #3502 from NixOS/more-pos
pass Pos to forceValue to improve infinite recursion error
2020-04-18 14:05:21 +02:00
Eelco Dolstra
fcd048a526 Use RootValue 2020-04-16 18:02:59 +02:00
Domen Kožar
b865b5b40c
pass Pos to forceValue to improve infinite recursion error 2020-04-16 12:32:07 +02:00
Domen Kožar
fc144242d5
Merge pull request #3447 from DavHau/improve-tofile-error-msg
improve toFile error message when containing potential drv path
2020-04-11 13:12:11 +02:00
DavHau
fc14213d2d improve toFile error message when containing potential drv path 2020-04-11 10:54:26 +00:00
Eelco Dolstra
9ed097db7b
Merge pull request #3468 from Infinisil/functionArgsPositions
Make function arguments retain position info
2020-04-08 15:29:39 +02:00
Eelco Dolstra
462421d345 Backport libfetchers from the flakes branch
This provides a pluggable mechanism for defining new fetchers. It adds
a builtin function 'fetchTree' that generalizes existing fetchers like
'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a
set of attributes, e.g.

  fetchTree {
    type = "git";
    url = "https://example.org/repo.git";
    ref = "some-branch";
    rev = "abcdef...";
  }

The existing fetchers are just wrappers around this. Note that the
input attributes to fetchTree are the same as flake input
specifications and flake lock file entries.

All fetchers share a common cache stored in
~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching
mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}).

This also adds support for Git worktrees (c169ea5904).
2020-04-07 09:03:14 +02:00
Silvan Mosberger
c34e96f7e0
Make function arguments retain position info
This allows querying the location of function arguments. E.g.

  builtins.unsafeGetAttrPos "x" (builtins.functionArgs ({ x }: null))

  => { column = 57; file = "/home/infinisil/src/nix/inst/test.nix"; line = 1; }
2020-04-02 05:52:52 +02:00
John Ericson
7e9a2718f0 s/outputHashRecursive/ingestionMethod/c 2020-03-30 22:36:15 +00:00
John Ericson
51afea3af2 Never cast FileIngestionMethod to or from boolean 2020-03-30 22:31:51 +00:00
John Ericson
bbbb7c1bc7 Use auto with some FileIngestionMethod local variables 2020-03-30 18:15:55 -04:00
John Ericson
832bd534dc Store parsed hashes in DerivationOutput
It's best to detect invalid data as soon as possible, with data types
that make storing it impossible.
2020-03-30 11:33:35 -04:00
John Ericson
225e62a56a Replace some bool recursive with a new FileIngestionMethod enum 2020-03-29 15:16:20 -04:00
John Ericson
87b32bab05 Use enum struct and drop prefixes
This does a few enums; the rest will be gotten in subsequent commits.
2020-03-29 11:23:15 -04:00
John Ericson
f1cf3ab870 hashDerivationModulo: Generalize for multiple fixed ouputs per drv
See documentattion in header and comments in implementation for details.

This is actually done in preparation for floating ca derivations, not
multi-output fixed ca derivations, but the distinction doesn't yet
mattter.

Thanks @cole-h for finding and fixing a bunch of typos.
2020-03-19 10:30:49 -04:00
Eelco Dolstra
9950cdec35 Move some corepkgs into the nix binary 2020-03-11 16:57:48 +01:00
Eelco Dolstra
401b5bc541
builtins.cache: Cache regular expressions
The evaluator was spending about 1% of its time compiling a small
number of regexes over and over again.
2020-03-04 11:44:33 +01:00
Eelco Dolstra
aef635da78 Fix derivation computation with __structuredAttrs and multiple outputs
Fixes

  error: derivation '/nix/store/klivma7r7h5lndb99f7xxmlh5whyayvg-zlib-1.2.11.drv' has incorrect output '/nix/store/fv98nnx5ykgbq8sqabilkgkbc4169q05-zlib-1.2.11-dev', should be '/nix/store/adm7pilzlj3z5k249s8b4wv3scprhzi1-zlib-1.2.11-dev'
2020-01-21 21:14:13 +01:00
Eelco Dolstra
bbe97dff8b Make the Store API more type-safe
Most functions now take a StorePath argument rather than a Path (which
is just an alias for std::string). The StorePath constructor ensures
that the path is syntactically correct (i.e. it looks like
<store-dir>/<base32-hash>-<name>). Similarly, functions like
buildPaths() now take a StorePathWithOutputs, rather than abusing Path
by adding a '!<outputs>' suffix.

Note that the StorePath type is implemented in Rust. This involves
some hackery to allow Rust values to be used directly in C++, via a
helper type whose destructor calls the Rust type's drop()
function. The main issue is the dynamic nature of C++ move semantics:
after we have moved a Rust value, we should not call the drop function
on the original value. So when we move a value, we set the original
value to bitwise zero, and the destructor only calls drop() if the
value is not bitwise zero. This should be sufficient for most types.

Also lots of minor cleanups to the C++ API to make it more modern
(e.g. using std::optional and std::string_view in some places).
2019-12-10 22:06:05 +01:00
Eelco Dolstra
2d6f1ddbb5
Remove builtins.valueSize
Fixes #3246.
2019-11-28 13:52:42 +01:00
Eelco Dolstra
ba87b08f85
getEnv(): Return std::optional
This allows distinguishing between an empty value and no value.
2019-11-22 16:18:13 +01:00
Eelco Dolstra
99b73fb507
OCD performance fix: {find,count}+insert => insert 2019-10-09 16:06:29 +02:00
Eelco Dolstra
168a887916
Fix fetchTarball with chroot stores
Fixes #2405.
2019-10-01 07:51:06 +00:00
Bas van Dijk
89865144c3 Allow builtins.pathExists to check the existence of /nix/store paths
This makes it consitent with builtins.readDir.
2019-07-30 11:27:35 +02:00
Eelco Dolstra
f8b30338ac
Refactor downloadCached() interface
(cherry picked from commit df3f5a78d5)
2019-06-24 22:12:26 +02:00
Eelco Dolstra
dc29e9fb47
downloadCached: Return ETag
(cherry picked from commit 529add316c)
2019-06-24 21:58:33 +02:00
Daniel Schaefer
3f192ac80c Add builtins.hashFile
For text files it is possible to do it like so:
`builtins.hashString "sha256" (builtins.readFile /tmp/a)`
but that doesn't work for binary files.

With builtins.hashFile any kind of file can be conveniently hashed.
2019-05-03 17:23:36 +02:00
zimbatm
514b3c7f83
Add isPath primop
this is added for completeness' sake since all the other possible
`builtins.typeOf` results have a corresponding `builtins.is<Type>`
2019-03-24 11:36:49 +01:00
Eelco Dolstra
ef52ccf035
experimental/optional -> optional 2019-03-14 14:10:52 +01:00
Shea Levy
1d757292d0
Add builtins.getContext.
This can be very helpful when debugging, as well as enabling complex
black magic like surgically removing a single dependency from a
string's context.
2019-01-14 11:27:10 -05:00
Shea Levy
087be7281a
Treat plain derivation paths in context as normal paths.
Previously, plain derivation paths in the string context (e.g. those
that arose from builtins.storePath on a drv file, not those that arose
from accessing .drvPath of a derivation) were treated somewhat like
derivaiton paths derived from .drvPath, except their dependencies
weren't recursively added to the input set. With this change, such
plain derivation paths are simply treated as paths and added to the
source inputs set accordingly, simplifying context handling code and
removing the inconsistency. If drvPath-like behavior is desired, the
.drv file can be imported and then .drvPath can be accessed.

This is a backwards-incompatibility, but storePath is never used on
drv files within nixpkgs and almost never used elsewhere.
2019-01-13 11:29:55 -05:00
Eelco Dolstra
6024dc1d97
Support SRI hashes
SRI hashes (https://www.w3.org/TR/SRI/) combine the hash algorithm and
a base-64 hash. This allows more concise and standard hash
specifications. For example, instead of

  import <nix/fetchurl.nl> {
    url = https://nixos.org/releases/nix/nix-2.1.3/nix-2.1.3.tar.xz;
    sha256 = "5d22dad058d5c800d65a115f919da22938c50dd6ba98c5e3a183172d149840a4";
  };

you can write

  import <nix/fetchurl.nl> {
    url = https://nixos.org/releases/nix/nix-2.1.3/nix-2.1.3.tar.xz;
    hash = "sha256-XSLa0FjVyADWWhFfkZ2iKTjFDda6mMXjoYMXLRSYQKQ=";
  };

In fixed-output derivations, the outputHashAlgo is no longer mandatory
if outputHash specifies the hash (either as an SRI or in the old
"<type>:<hash>" format).

'nix hash-{file,path}' now print hashes in SRI format by default. I
also reverted them to use SHA-256 by default because that's what we're
using most of the time in Nixpkgs.

Suggested by @zimbatm.
2018-12-13 14:30:52 +01:00
Eelco Dolstra
8327a7a8fa
Merge branch 'dirOf-relative' of https://github.com/lheckemann/nix 2018-09-13 14:33:12 +02:00
Eelco Dolstra
9b1bdf2db8
FIx floating point evaluation
Fixes #2361.
2018-08-19 11:59:49 +02:00
Linus Heckemann
d7402c9cd5 dirOf: allow use on non-absolute paths 2018-08-13 11:27:35 +02:00
Eelco Dolstra
45bcf5416a
Merge branch 'prim_mapAttr-fix' of https://github.com/volth/nix 2018-07-31 20:05:07 +02:00
volth
e2b114cfe1
prim_foldlStrict: call forceValue() before value is copied
forceValue() were called after a value is copied effectively forcing only one of the copies keeping another copy not evaluated.
This resulted in its evaluation of the same lazy value more than once (the number of hits is not big though)
2018-07-21 06:44:42 +00:00
volth
c03d73c1cf
prim_map, prim_genList: no need to force function argument which will be stored in mkApp 2018-07-07 23:30:17 +00:00
volth
627e28ba33 prim_mapAttrs: `f' must be evaluated lazily to avoid infinite recursion 2018-07-06 21:52:54 +00:00
volth
1515c65616 prim_concatMap: no need to force value 2018-07-05 15:33:33 +00:00
volth
e6bf1a79d7 prim_mapAttrs: must be lazy to avoid infinite recursion 2018-07-05 15:33:12 +00:00
volth
841747b0e6
prim_concatMap: allocate intermediate list on stack 2018-07-05 12:37:37 +00:00
volth
ee218f99ca
primops.cc: fix comment 2018-07-05 11:58:15 +00:00
volth
403a76a18f lib.concatMap and lib.mapAttrs to be builtins 2018-07-05 02:54:09 +00:00
Eelco Dolstra
1672bcd230
Move evaluator-specific settings out of libstore 2018-05-30 13:29:50 +02:00
volth
6cc28c0589
add `mod' and bitwise builtins: camel-case function names 2018-05-16 10:52:19 +00:00
volth
f3c090f91c
add mod' and bitwise builtins: remove mod' and shifts 2018-05-16 06:57:11 +00:00
volth
8a6a14e1f5 add `mod' and bitwise builtins 2018-05-12 16:50:39 +00:00
Shea Levy
b37f5ae31d
isFunction: True on primops.
Fixes #2073
2018-04-17 14:33:12 -04:00
Shea Levy
fd98fca7bb
Make prim_exec and prim_importNative available to plugins 2018-04-09 10:26:50 -04:00
Shea Levy
cfdbfa6b2c
Catch more possible instances of passing NULL to memcpy.
Actually fixes #1976.
2018-03-14 23:53:43 -04:00
Shea Levy
14ca85688c
Actually fix nixDataDir in non-canonical path 2018-02-28 06:19:40 -05:00
Tuomas Tynkkynen
77e9e1ed91 libexpr: Fix prim_replaceStrings() to work on an empty source string
Otherwise, running e.g.

nix-instantiate --eval -E --strict 'builtins.replaceStrings [""] ["X"] "abc"'

would just hang in an infinite loop.

Found by afl-fuzz.

First attempt of this was reverted in e2d71bd186 because it caused
another infinite loop, which is fixed now and a test added.
2018-02-26 19:49:13 +02:00
Eelco Dolstra
e2d71bd186
Revert "libexpr: Fix prim_replaceStrings() to work on an empty source string"
This reverts commit 4ea9707591.

It causes an infinite loop in Nixpkgs evaluation,
e.g. "nix-instantiate -A hello" hung.

PR #1886.
2018-02-21 15:35:28 +01:00
Tuomas Tynkkynen
4ea9707591 libexpr: Fix prim_replaceStrings() to work on an empty source string
Otherwise, running e.g.

nix-instantiate --eval -E --strict 'builtins.replaceStrings [""] ["X"] "abc"'

would just hang in an infinite loop.

Found by afl-fuzz.
2018-02-19 23:20:26 +02:00
Tuomas Tynkkynen
37264ed0ad libexpr: Avoid an unnecessary string copy in prim_derivationStrict 2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen
66eeff3345 libexpr: Remove unnecessary drvName assignment in prim_derivationStrict
drvName is already assigned to the same value right at the start of the
function.
2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen
7e0360504d libexpr: Optimize prim_derivationStrict by using more symbol comparisons 2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen
b8bed7da14 libexpr: Optimize prim_attrNames a bit
Instead of having lexicographicOrder() create a temporary sorted array
of Attr*:s and copying attr names from that, copy the attr names
first and then sort that.
2018-02-17 16:54:21 +02:00
Shea Levy
b095c06139
Add splitVersion primop.
Fixes #1868.
2018-02-14 09:55:43 -05:00
Eelco Dolstra
abe6be578b
Merge pull request #1816 from shlevy/add-path
Add path primop.
2018-02-07 13:32:35 +01:00
Shea Levy
69d82e5c58
Add path primop.
builtins.path allows specifying the name of a path (which makes paths
with store-illegal names now addable), allows adding paths with flat
instead of recursive hashes, allows specifying a filter (so is a
generalization of filterSource), and allows specifying an expected
hash (enabling safe path adding in pure mode).
2018-02-06 16:48:08 -05:00
Eelco Dolstra
43f8ef73c6
realiseContext(): Add derivation outputs to the allowed paths
This makes import-from-derivation work in restricted mode again.
2018-02-06 15:38:45 +01:00
Eelco Dolstra
0c95776c3e
Don't define builtins.{currentSystem,currentTime} in pure mode
This makes it easier to provide a default, e.g.

  system = builtins.currentSystem or "x86_64-linux";
2018-01-18 16:38:48 +01:00
Eelco Dolstra
d8b4cfad82
Typo 2018-01-17 11:53:08 +01:00
Eelco Dolstra
d4dcffd643
Add pure evaluation mode
In this mode, the following restrictions apply:

* The builtins currentTime, currentSystem and storePath throw an
  error.

* $NIX_PATH and -I are ignored.

* fetchGit and fetchMercurial require a revision hash.

* fetchurl and fetchTarball require a sha256 attribute.

* No file system access is allowed outside of the paths returned by
  fetch{Git,Mercurial,url,Tarball}. Thus 'nix build -f ./foo.nix' is
  not allowed.

Thus, the evaluation result is completely reproducible from the
command line arguments. E.g.

  nix build --pure-eval '(
    let
      nix = fetchGit { url = https://github.com/NixOS/nixpkgs.git; rev = "9c927de4b179a6dd210dd88d34bda8af4b575680"; };
      nixpkgs = fetchGit { url = https://github.com/NixOS/nixpkgs.git; ref = "release-17.09"; rev = "66b4de79e3841530e6d9c6baf98702aa1f7124e4"; };
    in (import (nix + "/release.nix") { inherit nix nixpkgs; }).build.x86_64-linux
  )'

The goal is to enable completely reproducible and traceable
evaluation. For example, a NixOS configuration could be fully
described by a single Git commit hash. 'nixos-rebuild' would do
something like

  nix build --pure-eval '(
    (import (fetchGit { url = file:///my-nixos-config; rev = "..."; })).system
  ')

where the Git repository /my-nixos-config would use further fetchGit
calls or Git externals to fetch Nixpkgs and whatever other
dependencies it has. Either way, the commit hash would uniquely
identify the NixOS configuration and allow it to reproduced.
2018-01-16 19:23:18 +01:00
Eelco Dolstra
74f75c8558
import, builtins.readFile: Handle diverted stores
Fixes #1791
2018-01-12 17:31:08 +01:00
Shea Levy
689b2783fc
Add hasContext primop 2018-01-02 12:25:14 -05:00
Eelco Dolstra
cd532a9251
Fix filterSource 2017-10-31 16:18:32 +01:00
Eelco Dolstra
812e027e1d
Add option allowed-uris
This allows network access in restricted eval mode.
2017-10-30 12:41:49 +01:00
Eelco Dolstra
f9686885be
enable-http2 -> http2 2017-10-30 11:00:59 +01:00
Eelco Dolstra
23ce4b3393
fetchTarball: Use "source" as the default name
This ensures that it produces the same output as fetchgit:

  $ nix eval --raw '(builtins.fetchgit https://github.com/NixOS/patchelf.git)'
  /nix/store/ghigrkw02l440g8vfxa9wj4c3zpfmw99-source

  $ nix eval --raw '(fetchTarball https://github.com/NixOS/patchelf/archive/master.tar.gz)'
  /nix/store/ghigrkw02l440g8vfxa9wj4c3zpfmw99-source
2017-10-30 10:22:58 +01:00
Eelco Dolstra
66ddbef754
fetchurl/fetchTarball: Respect name changes
The computation of urlHash didn't take the name into account, so
subsequent fetchurl calls with the same URL but a different name would
resolve to the same cached store path.
2017-10-30 10:22:58 +01:00
Eelco Dolstra
8191992c83
Bump language version 2017-10-25 15:18:37 +02:00
Eelco Dolstra
2d5b1b24bf
Pass lists/attrsets to bash as (associative) arrays 2017-10-25 13:01:50 +02:00
Nicolas B. Pierron
b8867a0239 Add builtins.string function.
The function 'builtins.split' takes a POSIX extended regular expression
and an arbitrary string. It returns a list of non-matching substring
interleaved by lists of matched groups of the regular expression.

```nix
with builtins;
assert split "(a)b" "abc"      == [ "" [ "a" ] "c" ];
assert split "([ac])" "abc"    == [ "" [ "a" ] "b" [ "c" ] "" ];
assert split "(a)|(c)" "abc"   == [ "" [ "a" null ] "b" [ null "c" ] "" ];
assert split "([[:upper:]]+)" "  FOO   "
                               == [ "  " [ "FOO" ] "   " ];
```
2017-08-15 20:04:11 +00:00
Jörg Thalheim
2fd8f8bb99 Replace Unicode quotes in user-facing strings by ASCII
Relevant RFC: NixOS/rfcs#4

$ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 12:32:45 +01:00
Eelco Dolstra
1762b9616c Merge pull request #1428 from rimmington/clearer-regex-space-error
Clearer error message when regex exceeds space limit
2017-07-10 11:45:05 +02:00
Rhys
17bb00d378 Clearer error message when regex exceeds space limit 2017-07-10 09:35:53 +10:00
Eelco Dolstra
c0015e87af
Support base-64 hashes
Also simplify the Hash API.

Fixes #1437.
2017-07-04 15:07:41 +02:00
Eelco Dolstra
e46090edb1
builtins.match: Improve error message for bad regular expression
Issue #1331.
2017-05-17 11:58:01 +02:00
Eelco Dolstra
b01d62285c
Improve progress indicator 2017-05-16 16:09:57 +02:00
Eelco Dolstra
c05d9ae7a5
Disallow outputHash being null or an empty string
Fixes #1384.
2017-05-15 18:47:13 +02:00
Shea Levy
3ecb09a40a builtins.exec: Make the argument just a list 2017-03-31 11:58:41 -04:00
Shea Levy
0bb8db257d Add exec primop behind allow-unsafe-native-code-during-evaluation.
Execute a given program with the (optional) given arguments as the
user running the evaluation, parsing stdout as an expression to be
evaluated.

There are many use cases for nix that would benefit from being able to
run arbitrary code during evaluation, including but not limited to:

* Automatic git fetching to get a sha256 from a git revision
* git rev-parse HEAD
* Automatic extraction of information from build specifications from
  other tools, particularly language-specific package managers like
  cabal or npm
* Secrets decryption (e.g. with nixops)
* Private repository fetching

Ideally, we would add this functionality in a more principled way to
nix, but in the mean time 'builtins.exec' can be used to get these
tasks done.

The primop is only available when the
'allow-unsafe-native-code-during-evaluation' nix option is true. That
flag also enables the 'importNative' primop, which is strictly more
powerful but less convenient (since it requires compiling a plugin
against the running version of nix).
2017-03-30 08:04:21 -04:00
Shea Levy
93f863be96 Add option to disable import-from-derivation completely, even if the drv is already realized 2017-03-08 08:46:12 -05:00
Eelco Dolstra
6de33a9c67
Add support for passing structured data to builders
Previously, all derivation attributes had to be coerced into strings
so that they could be passed via the environment. This is lossy
(e.g. lists get flattened, necessitating configureFlags
vs. configureFlagsArray, of which the latter cannot be specified as an
attribute), doesn't support attribute sets at all, and has size
limitations (necessitating hacks like passAsFile).

This patch adds a new mode for passing attributes to builders, namely
encoded as a JSON file ".attrs.json" in the current directory of the
builder. This mode is activated via the special attribute

  __structuredAttrs = true;

(The idea is that one day we can set this in stdenv.mkDerivation.)

For example,

  stdenv.mkDerivation {
    __structuredAttrs = true;
    name = "foo";
    buildInputs = [ pkgs.hello pkgs.cowsay ];
    doCheck = true;
    hardening.format = false;
  }

results in a ".attrs.json" file containing (sans the indentation):

  {
    "buildInputs": [],
    "builder": "/nix/store/ygl61ycpr2vjqrx775l1r2mw1g2rb754-bash-4.3-p48/bin/bash",
    "configureFlags": [
      "--with-foo",
      "--with-bar=1 2"
    ],
    "doCheck": true,
    "hardening": {
      "format": false
    },
    "name": "foo",
    "nativeBuildInputs": [
      "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10",
      "/nix/store/4jnvjin0r6wp6cv1hdm5jbkx3vinlcvk-cowsay-3.03"
    ],
    "propagatedBuildInputs": [],
    "propagatedNativeBuildInputs": [],
    "stdenv": "/nix/store/f3hw3p8armnzy6xhd4h8s7anfjrs15n2-stdenv",
    "system": "x86_64-linux"
  }

"passAsFile" is ignored in this mode because it's not needed - large
strings are included directly in the JSON representation.

It is up to the builder to do something with the JSON
representation. For example, in bash-based builders, lists/attrsets of
string values could be mapped to bash (associative) arrays.
2017-01-26 20:40:33 +01:00
Eelco Dolstra
54801ed6ad
Bindings: Add a method for iterating in lexicographically sorted order 2017-01-26 20:40:33 +01:00
Eelco Dolstra
4e6a2fbc56
Revert "Propagate path context via builtins.readFile"
This reverts commit f7f0116dd7.

Issue #1174.
2017-01-24 15:31:54 +01:00
Eelco Dolstra
fa738e50bc
Revert "builtins.readFile: Put the references of the file, not those needed to realize the file, into the context"
Reverting commit 451c223dee for now
because it breaks http://hydra.nixos.org/build/46805136, not clear
why.
2017-01-24 15:29:46 +01:00
Shea Levy
451c223dee builtins.readFile: Put the references of the file, not those needed to realize the file, into the context 2017-01-10 12:22:22 -05:00
Eelco Dolstra
215b70f51e
Revert "Get rid of unicode quotes (#1140)"
This reverts commit f78126bfd6. There
really is no need for such a massive change...
2016-11-26 00:38:01 +01:00
Guillaume Maudoux
f78126bfd6 Get rid of unicode quotes (#1140) 2016-11-25 15:48:27 +01:00
Tuomas Tynkkynen
f0fc3dd88b Fix SIGFPE from integer overflow during division
On some architectures (like x86_64 or i686, but not ARM for example)
overflow during integer division causes a crash due to SIGFPE.
Reproduces on a 64-bit system with:

    nix-instantiate --eval -E '(-9223372036854775807 - 1) / -1'

The only way this can happen is when the smallest possible integer is
divided by -1, so just special-case that.
2016-10-26 18:41:59 +03:00
Alexander Ried
c935e8eeaf fixup! replace own regex class with std::regex 2016-10-19 16:03:53 +02:00
Alexander Ried
b05b98df75 replace own regex class with std::regex 2016-10-18 20:22:25 +02:00
Eelco Dolstra
c55bf085eb printMsg(lvlError, ...) -> printError(...) etc. 2016-09-21 16:54:53 +02:00
Eelco Dolstra
90ad02bf62 Enable HTTP/2 support
The binary cache store can now use HTTP/2 to do lookups. This is much
more efficient than HTTP/1.1 due to multiplexing: we can issue many
requests in parallel over a single TCP connection. Thus it's no longer
necessary to use a bunch of concurrent TCP connections (25 by
default).

For example, downloading 802 .narinfo files from
https://cache.nixos.org/, using a single TCP connection, takes 11.8s
with HTTP/1.1, but only 0.61s with HTTP/2.

This did require a fairly substantial rewrite of the Downloader class
to use the curl multi interface, because otherwise curl wouldn't be
able to do multiplexing for us. As a bonus, we get connection reuse
even with HTTP/1.1. All downloads are now handled by a single worker
thread. Clients call Downloader::enqueueDownload() to tell the worker
thread to start the download, getting a std::future to the result.
2016-09-14 16:36:02 +02:00
Eelco Dolstra
c42d1acfeb forceBool(): Show position info 2016-08-29 19:37:19 +02:00
Eelco Dolstra
26d92017d3 Add builtin function "partition"
The implementation of "partition" in Nixpkgs is O(n^2) (because of the
use of ++), and for some reason was causing stack overflows in
multi-threaded evaluation (not sure why).

This reduces "nix-env -qa --drv-path" runtime by 0.197s and memory
usage by 298 MiB (in non-Boehm mode).
2016-08-29 19:36:54 +02:00
Eelco Dolstra
22d6e31fc6 Add a mechanism for derivation attributes to reference the derivation's outputs
For example, you can now say:

  configureFlags = "--prefix=${placeholder "out"} --includedir=${placeholder "dev"}";

The strings returned by the ‘placeholder’ builtin are replaced at
build time by the actual store paths corresponding to the specified
outputs.

Previously, you had to work around the inability to self-reference by doing stuff like:

  preConfigure = ''
    configureFlags+=" --prefix $out --includedir=$dev"
  '';

or rely on ad-hoc variable interpolation semantics in Autoconf or Make
(e.g. --prefix=\$(out)), which doesn't always work.
2016-08-17 17:19:32 +02:00
Eelco Dolstra
ac841a4679 Merge pull request #1031 from abbradar/replacestrings-context
Allow contexted strings in replaceStrings
2016-08-16 12:47:20 +02:00
Nikolay Amiantov
f1b8dd43be Allow contexted strings in replaceStrings 2016-08-16 13:20:56 +03:00
Shea Levy
d52d391164 builtins.fetch{url,tarball}: Allow name attribute 2016-08-15 07:37:11 -04:00
Eelco Dolstra
ee22a91ab8 makeFixedOutputPath(): Drop superfluous HashType argument 2016-07-26 21:25:52 +02:00
Eelco Dolstra
06bbfb6004 builtins.{fetchurl,fetchTarball}: Support a sha256 attribute
Also, allow builtins.{fetchurl,fetchTarball} in restricted mode if a
hash is specified.
2016-07-26 21:16:52 +02:00
Eelco Dolstra
7850d3d279 Make the store directory a member variable of Store 2016-06-01 16:24:17 +02:00
Eelco Dolstra
6c75cf69c3 Cleanup: Remove singleton() 2016-05-04 16:16:53 +02:00
Eelco Dolstra
41633f9f73 Improved logging abstraction
This also gets rid of --log-type, since the nested log type isn't
useful in a multi-threaded situation, and nobody cares about the
"pretty" log type.
2016-04-25 19:18:45 +02:00
Eelco Dolstra
363f37d084 Make the search path lazier with non-fatal errors
Thus, -I / $NIX_PATH entries are now downloaded only when they are
needed for evaluation. An error to download an entry is a non-fatal
warning (just like non-existant paths).

This does change the semantics of builtins.nixPath, which now returns
the original, rather than resulting path. E.g., before we had

  [ { path = "/nix/store/hgm3yxf1lrrwa3z14zpqaj5p9vs0qklk-nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ]

but now

  [ { path = "https://nixos.org/channels/nixos-16.03/nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ]

Fixes #792.
2016-04-14 15:32:24 +02:00
Eelco Dolstra
12b257f045 Make primop registration pluggable
This way we don't have to put all primops in one giant file.
2016-04-14 12:50:01 +02:00
Nikolay Amiantov
f7f0116dd7 Propagate path context via builtins.readFile 2016-03-04 14:04:51 +03:00
Eelco Dolstra
e9c50064b5 Add an HTTP binary cache store
Allowing stuff like

  NIX_REMOTE=https://cache.nixos.org nix-store -qR /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1

or

  NIX_REMOTE=https://cache.nixos.org nix-store --export /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 | nix-store --import
2016-02-29 18:15:20 +01:00
Eelco Dolstra
b3e8d72770 Merge pull request #762 from ctheune/ctheune-floats
Implement floats
2016-02-12 12:49:59 +01:00