Commit graph

1148 commits

Author SHA1 Message Date
John Ericson
c251b011cd Merge remote-tracking branch 'upstream/master' into enum-FileIngestionMethod 2020-03-30 18:16:44 -04:00
John Ericson
bbbb7c1bc7 Use auto with some FileIngestionMethod local variables 2020-03-30 18:15:55 -04:00
John Ericson
832bd534dc Store parsed hashes in DerivationOutput
It's best to detect invalid data as soon as possible, with data types
that make storing it impossible.
2020-03-30 11:33:35 -04:00
Eelco Dolstra
e322a16523 Remove global -I flags
(cherry picked from commit 2c692a3b14)
2020-03-30 15:30:19 +02:00
Julian Stecklina
40c023ecfe fetchGit: don't use std::filesystem to filter git repos
Using std::filesystem means also having to link with -lstdc++fs on
some platforms and it's hard to discover for what platforms this is
needed. As all the functionality is already implemented as utilities,
use those instead.
2020-03-30 00:32:42 +02:00
Bjørn Forsman
f686efeed4 fetchGit: fix submodule corner case by fetching all refs from cacheDir
Due to fetchGit not checking if rev is an ancestor of ref (there is even
a FIXME comment about it in the code), the cache repo might not have the
ref even though it has the rev. This doesn't matter when submodule =
false, but the submodule = true code blows up because it tries to fetch
the (missing) ref from the cache repo.

Fix this in the simplest way possible: fetch all refs from the local
cache repo when submodules = true.

TODO: Add tests.
2020-03-29 22:29:58 +02:00
Bjørn Forsman
cc522d0d23 fetchGit: fix submodules = true for dirty trees 2020-03-29 22:29:58 +02:00
Bjørn Forsman
b306b7039e fetchGit: checkout rev instead of latest ref
Major bugfix for the submodules = true code path.

TODO: Add tests.
2020-03-29 22:29:58 +02:00
Bjørn Forsman
369fffd6f1 fetchGit: add submodules attribute to the .link file
The .link file is used as a lock, so I think we should put the
"submodule" attribute in there since turning on submodules creates a new
.link file path.
2020-03-29 22:29:58 +02:00
Bjørn Forsman
6864ad7cf5 fetchGit: fix submodule output attribute
Before this change it would be false for all evaluations but the first.
Now it follows the input argument (as it should).
2020-03-29 22:29:58 +02:00
Bjørn Forsman
002a3a95dc fetchGit: fix "fatal: couldn't find remote ref refs/heads/master" issue with submodules 2020-03-29 22:29:58 +02:00
Julian Stecklina
cc4fe977e5 Link to stdc++fs
Some platforms seem to still require linking with stdc++fs to enable
STL std::filesystem support.
2020-03-29 22:29:58 +02:00
Julian Stecklina
435366ed3c Rename fetchGit fetchSubmodules to just submodules 2020-03-29 22:29:58 +02:00
Julian Stecklina
c8d33de777 Add git submodule fixes from @bjornfor
This fixes fetching repositories with no submodules and also cleans up
.git files in checkouts.
2020-03-29 22:29:57 +02:00
Julian Stecklina
f58604ac32 Add fetchSubmodules to builtins.fetchGit
There are some downsides to this features:

 - Submodules are not cached (unlike the root repo),
 - Full checkouts are created in a temporary directory.
2020-03-29 22:29:57 +02:00
John Ericson
225e62a56a Replace some bool recursive with a new FileIngestionMethod enum 2020-03-29 15:16:20 -04:00
John Ericson
87b32bab05 Use enum struct and drop prefixes
This does a few enums; the rest will be gotten in subsequent commits.
2020-03-29 11:23:15 -04:00
John Ericson
bcde5456cc Flip dependency so store-api.hh includes derivations.hh
I think it makes more sense to define the data model (derivations),
before the operations (store api).
2020-03-24 20:39:45 +00:00
Eelco Dolstra
0a10854f85 Misc changes from the flakes branch 2020-03-24 14:34:47 +01:00
Eelco Dolstra
76e7d958ed Fix coverage build
https://hydra.nixos.org/build/110757285
(cherry picked from commit b430a81a1f)
2020-03-24 14:06:47 +01:00
Eelco Dolstra
231a8aa2c2 nix edit: Support non-derivation attributes
E.g.

  $ nix edit .#nixosConfigurations.bla

now works.

(cherry picked from commit d2032edb2f)
2020-03-24 14:06:47 +01:00
Eelco Dolstra
c1ca4f0acc findAlongAttrPath(): Return position
(cherry picked from commit 0b013a54dc)
2020-03-24 14:06:47 +01:00
Eelco Dolstra
1eb952d27a findAlongAttrPath(): Throw AttrPathNotFound
(cherry picked from commit 6b0ca8e803)
2020-03-24 14:06:47 +01:00
Eelco Dolstra
c34a20e1f6 EvalState::allocAttr(): Add convenience method
(cherry picked from commit c02da99757)
2020-03-24 13:26:37 +01:00
John Ericson
68fe0d9809 Add missing #include <regex> 2020-03-13 21:24:35 -04:00
Eelco Dolstra
9950cdec35 Move some corepkgs into the nix binary 2020-03-11 16:57:48 +01:00
Eelco Dolstra
e02481ded2 parseExprFromString(): Use std::string_view 2020-03-11 16:56:29 +01:00
Eelco Dolstra
401b5bc541
builtins.cache: Cache regular expressions
The evaluator was spending about 1% of its time compiling a small
number of regexes over and over again.
2020-03-04 11:44:33 +01:00
Eelco Dolstra
9af10b753c Bindings::get(): std::optional<Attr *> -> Attr *
Returning a nullable type in an optional is silly.
2020-02-13 17:15:05 +01:00
Eelco Dolstra
aef635da78 Fix derivation computation with __structuredAttrs and multiple outputs
Fixes

  error: derivation '/nix/store/klivma7r7h5lndb99f7xxmlh5whyayvg-zlib-1.2.11.drv' has incorrect output '/nix/store/fv98nnx5ykgbq8sqabilkgkbc4169q05-zlib-1.2.11-dev', should be '/nix/store/adm7pilzlj3z5k249s8b4wv3scprhzi1-zlib-1.2.11-dev'
2020-01-21 21:14:13 +01:00
Eelco Dolstra
c3181e21e7 Tweak error message 2020-01-13 21:52:03 +01:00
Eelco Dolstra
bfaa4db7bd Merge branch 'assert-show-expression' of https://github.com/LnL7/nix 2020-01-13 21:49:55 +01:00
Eelco Dolstra
835e541144 Fix build
https://hydra.nixos.org/eval/1564374
2020-01-13 21:34:54 +01:00
Daiderd Jordan
307bcb9a8e
libexpr: show expression in assertion errors
Includes the expression of the condition in the assertion message if
the assertion failed, making assertions much easier to debug. eg.

    error: assertion (withPython -> (python2Packages != null)) failed at pkgs/tools/security/nmap/default.nix:11:1
2020-01-11 15:45:41 +01:00
Yorick van Pelt
a350d0beb0
json-to-value: use unique_ptr instead of raw pointers 2020-01-09 22:46:41 +01:00
Yorick van Pelt
f1fac0b5c3
builtins.fromJSON: use nlohmann/json parser instead of custom parser 2020-01-09 17:38:27 +01:00
Nikola Knezevic
52a8f9295b Add support for \u escape in fromJSON
As fromTOML supports \u and \U escapes, bring fromJSON on par. As JSON defaults
to UTF-8 encoding (every JSON parser must support UTF-8), this change parses the
`\u hex hex hex hex` sequence (\u followed by 4 hexadecimal digits) into an
UTF-8 representation.

Add a test to verify correct parsing, using all escape sequences from json.org.
2020-01-07 00:09:58 +01:00
Nikola Knezevic
cb2d348d48 Remove redundant check in parseJSONString 2020-01-07 00:09:58 +01:00
Eelco Dolstra
cb90e382b5 Hide FunctionCallTrace constructor/destructor
This prevents them from being inlined. On gcc 9, this reduces the
stack size needed for

  nix-instantiate '<nixpkgs>' -A texlive.combined.scheme-full --dry-run

from 12.9 MiB to 4.8 MiB.
2020-01-05 16:21:34 +01:00
Eelco Dolstra
54bf5ba422 nix-store -r: Handle symlinks to store paths
Fixes #3270.
2019-12-16 19:11:47 +01:00
Eelco Dolstra
acb71aa5c6 Tweak error message 2019-12-15 10:44:53 +01:00
Dima
d89d9958a7 bugfix: Adding depth limit to resolveExprPath
There is no termination condition for evaluation of cyclical
expression paths which can lead to infinite loops. This addresses
one spot in the parser in a similar fashion as utils.cc/canonPath
does.

This issue can be reproduced by something like:

```
ln -s a b
ln -s b a

nix-instantiate -E 'import ./a'
```
2019-12-13 14:51:30 +01:00
Eelco Dolstra
bbe97dff8b Make the Store API more type-safe
Most functions now take a StorePath argument rather than a Path (which
is just an alias for std::string). The StorePath constructor ensures
that the path is syntactically correct (i.e. it looks like
<store-dir>/<base32-hash>-<name>). Similarly, functions like
buildPaths() now take a StorePathWithOutputs, rather than abusing Path
by adding a '!<outputs>' suffix.

Note that the StorePath type is implemented in Rust. This involves
some hackery to allow Rust values to be used directly in C++, via a
helper type whose destructor calls the Rust type's drop()
function. The main issue is the dynamic nature of C++ move semantics:
after we have moved a Rust value, we should not call the drop function
on the original value. So when we move a value, we set the original
value to bitwise zero, and the destructor only calls drop() if the
value is not bitwise zero. This should be sufficient for most types.

Also lots of minor cleanups to the C++ API to make it more modern
(e.g. using std::optional and std::string_view in some places).
2019-12-10 22:06:05 +01:00
Eelco Dolstra
98ef11677c EvalState::callFunction(): Make FunctionCallTrace use less stack space
The FunctionCallTrace object consumes a few hundred bytes of stack
space, even when tracing is disabled. This was causing stack overflows:

  $ nix-instantiate '<nixpkgs> -A texlive.combined.scheme-full --dry-run
  error: stack overflow (possible infinite recursion)

This is with the default stack size of 8 MiB.

Putting the object on the heap reduces stack usage to < 5 MiB.
2019-12-10 13:32:30 +01:00
Eelco Dolstra
0678e4d56a Move #include
(cherry picked from commit 8beedd4486)
2019-12-05 20:30:29 +01:00
Eelco Dolstra
79142cbbe1 Bindings: Add convenience method for requiring an attribute
(cherry picked from commit fb692e5f7b)
2019-12-05 20:29:15 +01:00
Eelco Dolstra
0d118ef0c9 Bindings::get(): Add convenience method
This allows writing attribute lookups as

    if (auto name = value.attrs->get(state.sName))
      ...

(cherry picked from commit f216c76c56)
2019-12-05 20:29:00 +01:00
Eelco Dolstra
f102d793f1
Merge pull request #2748 from edolstra/rust
Make nix/unpack-channel.nix a builtin builder
2019-11-29 19:33:31 +01:00
Eelco Dolstra
2d6f1ddbb5
Remove builtins.valueSize
Fixes #3246.
2019-11-28 13:52:42 +01:00
Eelco Dolstra
8918bae098 Drop remaining uses of external "tar"
Also, fetchGit now runs in O(1) memory since we pipe the output of
'git archive' directly into unpackTarball() (rather than first reading
it all into memory).
2019-11-26 22:07:28 +01:00
Eelco Dolstra
872740cf60
Merge pull request #3238 from puckipedia/attrset-overrides-dynamic
Ensure enough space in attrset bindings
2019-11-26 20:14:55 +01:00
Eelco Dolstra
1ec6e6e11e
Add feature to disable URL literals
E.g.

  $ nix-build '<nixpkgs>' -A hello --experimental-features no-url-literals
  error: URL literals are disabled, at /nix/store/vsjamkzh15r3c779q2711az826hqgvzr-nixpkgs-20.03pre194957.bef773ed53f/nixpkgs/pkgs/top-level/all-packages.nix:1236:11

Helps with implementing https://github.com/NixOS/rfcs/pull/45.
2019-11-26 19:48:34 +01:00
Puck Meerburg
cd55f91ad2 Ensure enough space in attrset bindings when using both __overrides and dynamic attributes 2019-11-25 12:37:14 +00:00
Eelco Dolstra
d12d69ea1a
Turn NIX_PATH into a config setting
This allows it to be set in nix.conf.
2019-11-22 23:07:35 +01:00
Eelco Dolstra
ec9dd9a5ae
Provide a default value for NIX_PATH 2019-11-22 22:08:51 +01:00
Eelco Dolstra
ba87b08f85
getEnv(): Return std::optional
This allows distinguishing between an empty value and no value.
2019-11-22 16:18:13 +01:00
Eelco Dolstra
5ee23c35b9
Merge pull request #3219 from Ericson2314/semicolons
Fix extra semicolons warnings
2019-11-11 12:13:51 +01:00
John Ericson
4c34054673 Remove unneeded semicolons 2019-11-10 11:24:47 -05:00
John Ericson
96e6e680c1 Fix extra ; warnings involving MakeError 2019-11-10 11:24:47 -05:00
Peter Kolloch
2ba9f22715
De-duplicate struct PrimOp forward declaration 2019-11-10 10:02:22 +01:00
Eelco Dolstra
852554bb16
Merge branch 'nix-repl-e' of https://github.com/zimbatm/nix 2019-11-05 11:20:53 +01:00
Benjamin Hipple
80d5ec6ff4 Minor updates to inline comments
Add missing docstring on InstallableCommand. Also, some of these were wrapped
when they're right next to a line longer than the unwrapped line, so we can just
unwrap them to save vertical space.
2019-10-31 05:56:37 -04:00
zimbatm
9a25059656
findDerivationFilename: add FIXME 2019-10-28 21:40:02 +01:00
zimbatm
ec448f8bb6
libexpr: findDerivationFilename return Pos instead of tuple 2019-10-28 21:29:54 +01:00
Eelco Dolstra
e012384fe9
Merge branch 'tojson-tostring-fix' of https://github.com/mayflower/nix 2019-10-27 12:18:35 +01:00
Robin Gloster
e583df5280
builtins.toJSON: fix __toString usage 2019-10-27 10:15:51 +01:00
zimbatm
59c7249769
libexpr: add findDerivationFilename
extract the derivation to filename:lineno heuristic
2019-10-23 17:21:16 +02:00
Eelco Dolstra
926d3e5bb0
Fix Bison 2.4 warning 2019-10-09 22:57:37 +02:00
Eelco Dolstra
99b73fb507
OCD performance fix: {find,count}+insert => insert 2019-10-09 16:06:29 +02:00
Eelco Dolstra
168a887916
Fix fetchTarball with chroot stores
Fixes #2405.
2019-10-01 07:51:06 +00:00
Eelco Dolstra
bd79c1f6f6 Don't catch exceptions by value
(cherry picked from commit 893be6f5e3)
2019-09-22 21:56:56 +02:00
zimbatm
619cc4af85
function-trace: always show the trace
If the user invokes nix with --trace-function-calls it means that they
want to see the trace.
2019-09-18 23:23:21 +02:00
Graham Christensen
ee9c988a1b
Track function start and ends for flame graphs
With this patch, and this file I called `log.py`:

    #!/usr/bin/env nix-shell
    #!nix-shell -i python3 -p python3 --pure

    import sys
    from pprint import pprint

    stack = []
    timestack = []

    for line in open(sys.argv[1]):
        components = line.strip().split(" ", 2)
        if components[0] != "function-trace":
            continue

        direction = components[1]
        components = components[2].rsplit(" ", 2)

        loc = components[0]
        _at = components[1]
        time = int(components[2])

        if direction == "entered":
            stack.append(loc)
            timestack.append(time)
        elif direction == "exited":
            dur = time - timestack.pop()
            vst = ";".join(stack)
            print(f"{vst} {dur}")
            stack.pop()

and:

    nix-instantiate --trace-function-calls -vvvv ../nixpkgs/pkgs/top-level/release.nix -A unstable > log.matthewbauer 2>&1
    ./log.py ./log.matthewbauer > log.matthewbauer.folded
    flamegraph.pl --title matthewbauer-post-pr log.matthewbauer.folded > log.matthewbauer.folded.svg

I can make flame graphs like: http://gsc.io/log.matthewbauer.folded.svg

---

Includes test cases around function call failures and tryEval. Uses
RAII so the finish is always called at the end of the function.
2019-08-14 16:09:35 -04:00
Bas van Dijk
89865144c3 Allow builtins.pathExists to check the existence of /nix/store paths
This makes it consitent with builtins.readDir.
2019-07-30 11:27:35 +02:00
Eelco Dolstra
33db1d35ae
Merge pull request #2582 from LnL7/fetchgit-refs
fetchGit: allow fetching explicit refs
2019-07-02 15:44:31 +02:00
Eelco Dolstra
64ec087f58
Fix 32-bit overflow with --no-net
--no-net causes tarballTtl to be set to the largest 32-bit integer,
which causes comparison like 'time + tarballTtl < other_time' to
fail on 32-bit systems. So cast them to 64-bit first.

https://hydra.nixos.org/build/95076624
(cherry picked from commit 29ccb2e969)
2019-06-24 22:16:43 +02:00
Eelco Dolstra
f8b30338ac
Refactor downloadCached() interface
(cherry picked from commit df3f5a78d5)
2019-06-24 22:12:26 +02:00
Eelco Dolstra
b43e1e186e
CachedDownloadResult: Include store path
Also, make fetchGit and fetchMercurial update allowedPaths properly.

(Maybe the evaluator, rather than the caller of the evaluator, should
apply toRealPath(), but that's a bigger change.)

(cherry picked from commit 5c34d66538)
2019-06-24 21:59:27 +02:00
Eelco Dolstra
dc29e9fb47
downloadCached: Return ETag
(cherry picked from commit 529add316c)
2019-06-24 21:58:33 +02:00
Eelco Dolstra
94f11d0a61
Fix abort in fromTOML
Fixes #2969.
2019-06-24 17:09:09 +02:00
Eelco Dolstra
22f2744afd
Iterate over references 2019-05-28 23:05:08 +02:00
Eelco Dolstra
9eaebbf575
Merge branch 'attrPaths' of https://github.com/NinjaTrappeur/nix 2019-05-28 22:59:05 +02:00
Daniel Schaefer
3f192ac80c Add builtins.hashFile
For text files it is possible to do it like so:
`builtins.hashString "sha256" (builtins.readFile /tmp/a)`
but that doesn't work for binary files.

With builtins.hashFile any kind of file can be conveniently hashed.
2019-05-03 17:23:36 +02:00
Eelco Dolstra
bb6e6923f2 Add environment variable NIX_SHOW_SYMBOLS for dumping the symbol table 2019-04-11 23:04:13 +02:00
Eelco Dolstra
f32fbf952d Fix Bison 3.3 warning 2019-03-27 21:09:31 +01:00
zimbatm
514b3c7f83
Add isPath primop
this is added for completeness' sake since all the other possible
`builtins.typeOf` results have a corresponding `builtins.is<Type>`
2019-03-24 11:36:49 +01:00
Linus Heckemann
2aa89daab3 eval: improve type description for primops and applied primops
This can make type errors a little easier to understand.
2019-03-21 15:31:46 +01:00
Eelco Dolstra
1f64f4c7c8
pkg-config files: Use c++17 2019-03-14 14:11:12 +01:00
Eelco Dolstra
ef52ccf035
experimental/optional -> optional 2019-03-14 14:10:52 +01:00
Shea Levy
b30be6b450
Add builtins.appendContext.
A partner of builtins.getContext, useful for the same reasons.
2019-01-31 08:52:23 -05:00
Shea Levy
1d757292d0
Add builtins.getContext.
This can be very helpful when debugging, as well as enabling complex
black magic like surgically removing a single dependency from a
string's context.
2019-01-14 11:27:10 -05:00
Shea Levy
087be7281a
Treat plain derivation paths in context as normal paths.
Previously, plain derivation paths in the string context (e.g. those
that arose from builtins.storePath on a drv file, not those that arose
from accessing .drvPath of a derivation) were treated somewhat like
derivaiton paths derived from .drvPath, except their dependencies
weren't recursively added to the input set. With this change, such
plain derivation paths are simply treated as paths and added to the
source inputs set accordingly, simplifying context handling code and
removing the inconsistency. If drvPath-like behavior is desired, the
.drv file can be imported and then .drvPath can be accessed.

This is a backwards-incompatibility, but storePath is never used on
drv files within nixpkgs and almost never used elsewhere.
2019-01-13 11:29:55 -05:00
Eelco Dolstra
c7bf1cdb4e
Merge pull request #2608 from dtzWill/fix/issue-2546
EvalState::resetFileCache: clear parse cache as well as eval cache
2019-01-10 20:56:31 +01:00
John Ericson
fef9f5653b Remove mentions of libformat, it no longer exists 2019-01-05 14:31:29 -05:00
Will Dietz
21ea00d3ec EvalState::resetFileCache: clear parse cache as well as eval cache
Fixes #2546.

(at least the basic reproduction I've been testing)
2018-12-31 10:18:28 -06:00
Daiderd Jordan
7e35e914c1
fetchGit: allow fetching explicit refs
Trying to fetch refs that are not in refs/heads currently fails because
it looks for refs/heads/refs/foo instead of refs/foo.

eg.

	builtins.fetchGit {
	  url = https://github.com/NixOS/nixpkgs.git;
	  ref = "refs/pull/1024/head;
	}
2018-12-14 20:12:22 +01:00
Eelco Dolstra
6024dc1d97
Support SRI hashes
SRI hashes (https://www.w3.org/TR/SRI/) combine the hash algorithm and
a base-64 hash. This allows more concise and standard hash
specifications. For example, instead of

  import <nix/fetchurl.nl> {
    url = https://nixos.org/releases/nix/nix-2.1.3/nix-2.1.3.tar.xz;
    sha256 = "5d22dad058d5c800d65a115f919da22938c50dd6ba98c5e3a183172d149840a4";
  };

you can write

  import <nix/fetchurl.nl> {
    url = https://nixos.org/releases/nix/nix-2.1.3/nix-2.1.3.tar.xz;
    hash = "sha256-XSLa0FjVyADWWhFfkZ2iKTjFDda6mMXjoYMXLRSYQKQ=";
  };

In fixed-output derivations, the outputHashAlgo is no longer mandatory
if outputHash specifies the hash (either as an SRI or in the old
"<type>:<hash>" format).

'nix hash-{file,path}' now print hashes in SRI format by default. I
also reverted them to use SHA-256 by default because that's what we're
using most of the time in Nixpkgs.

Suggested by @zimbatm.
2018-12-13 14:30:52 +01:00
CHEIKH Chawki
fa5143c722 Solve hg "abandoned transaction" issue 2018-12-06 13:57:59 +01:00
Eelco Dolstra
4aee93d5ce
fetchGit: Drop unnecessary localRef 2018-11-20 20:59:44 +01:00
Eelco Dolstra
3f4de91d80
Merge branch 'better-git-cache' of https://github.com/graham-at-target/nix 2018-11-20 20:41:19 +01:00
Jan Path
d1b049c4ea Fix typo in comments 2018-10-31 20:50:18 +01:00
Guillaume Maudoux
6a5bf9b143 simplify handling of extra '}' 2018-10-27 00:14:51 +02:00
Eelco Dolstra
8327a7a8fa
Merge branch 'dirOf-relative' of https://github.com/lheckemann/nix 2018-09-13 14:33:12 +02:00
Eelco Dolstra
91405986f4
Convert NIX_COUNT_CALLS to JSON too 2018-09-05 21:57:54 +02:00
Eelco Dolstra
0a2545f95c
Log stats to stderr
We shouldn't pollute stdout.
2018-09-05 21:35:58 +02:00
Michael Bishop
4b034f390c remove the old text format output 2018-09-02 18:25:23 -03:00
Michael Bishop
2fd1008c70 add JSON to NIX_SHOW_STATS 2018-09-01 20:05:06 -03:00
Michael Bishop
c29e5fbb13 improve the stats when profiling 2018-09-01 17:11:56 -03:00
Eelco Dolstra
475a0a54a9
fetchGit/fetchMercurial: Don't absolutize paths
This is already done by coerceToString(), provided that the argument
is a path (e.g. 'fetchGit ./bla'). It fixes the handling of URLs like
git@github.com:owner/repo.git. It breaks 'fetchGit "./bla"', but that
was never intended to work anyway and is inconsistent with other
builtin functions (e.g. 'readFile "./bla"' fails).
2018-09-01 00:19:49 +02:00
aszlig
0ad643ed5c
libexpr: Use int64_t for NixInt
Using a 64bit integer on 32bit systems will come with a bit of a
performance overhead, but given that Nix doesn't use a lot of integers
compared to other types, I think the overhead is negligible also
considering that 32bit systems are in decline.

The biggest advantage however is that when we use a consistent integer
size across all platforms it's less likely that we miss things that we
break due to that. One example would be:

https://github.com/NixOS/nixpkgs/pull/44233

On Hydra it will evaluate, because the evaluator runs on a 64bit
machine, but when evaluating the same on a 32bit machine it will fail,
so using 64bit integers should make that consistent.

While the change of the type in value.hh is rather easy to do, we have a
few more options available for doing the conversion in the lexer:

  * Via an #ifdef on the architecture and using strtol() or strtoll()
    accordingly depending on which architecture we are. For the #ifdef
    we would need another AX_COMPILE_CHECK_SIZEOF in configure.ac.
  * Using istringstream, which would involve copying the value.
  * As we're already using boost, lexical_cast might be a good idea.

Spoiler: I went for the latter, first of all because lexical_cast does
have an overload for const char* and second of all, because it doesn't
involve copying around the input string. Also, because istringstream
seems to come with a bigger overhead than boost::lexical_cast:

https://www.boost.org/doc/libs/release/doc/html/boost_lexical_cast/performance.html

The first method (still using strtol/strtoll) also wasn't something I
pursued further, because it is also locale-aware which I doubt is what
we want, given that the regex for int is [0-9]+.

Signed-off-by: aszlig <aszlig@nix.build>
Fixes: #2339
2018-08-29 01:05:52 +02:00
Eelco Dolstra
c651b7bdc9
Revert "Fix parser/lexer generation with parallel make"
This reverts commit d277442df5.

Make sucks.
2018-08-23 00:23:35 +02:00
Eelco Dolstra
9b1bdf2db8
FIx floating point evaluation
Fixes #2361.
2018-08-19 11:59:49 +02:00
Graham Christensen
02098d2073 fetchGit: use a better caching scheme
The current usage technically works by putting multiple different
repos in to the same git directory. However, it is very slow as
Git tries very hard to find common commits between the two
repositories. If the two repositories are large (like Nixpkgs and
another long-running project,) it is maddeningly slow.

This change busts the cache for existing deployments, but users
will be promptly repaid in per-repository performance.
2018-08-17 11:27:34 -04:00
Eelco Dolstra
d277442df5
Fix parser/lexer generation with parallel make
Fun fact: rules with multiple targets don't work properly with 'make
-j'. For example, a rule like

  a b: c
    touch a b

is equivalent to

  a: c
    touch a b

  b: c
    touch a b

so with 'make -j', the 'touch' command will be run twice. See
e.g. https://stackoverflow.com/questions/2973445/gnu-makefile-rule-generating-a-few-targets-from-a-single-source-file.
2018-08-17 12:59:23 +02:00
Linus Heckemann
d7402c9cd5 dirOf: allow use on non-absolute paths 2018-08-13 11:27:35 +02:00
Eelco Dolstra
bc65e02d96
Merge pull request #2326 from aszlig/fix-symlink-leak
Fix symlink leak in restricted eval mode
2018-08-03 17:01:34 +02:00
aszlig
43e28a1b75
Fix symlink leak in restricted eval mode
In EvalState::checkSourcePath, the path is checked against the list of
allowed paths first and later it's checked again *after* resolving
symlinks.

The resolving of the symlinks is done via canonPath, which also strips
out "../" and "./". However after the canonicalisation the error message
pointing out that the path is not allowed prints the symlink target in
the error message.

Even if we'd suppress the message, symlink targets could still be leaked
if the symlink target doesn't exist (in this case the error is thrown in
canonPath).

So instead, we now do canonPath() without symlink resolving first before
even checking against the list of allowed paths and then later do the
symlink resolving and checking the allowed paths again.

The first call to canonPath() should get rid of all the "../" and "./",
so in theory the only way to leak a symlink if the attacker is able to
put a symlink in one of the paths allowed by restricted evaluation mode.

For the latter I don't think this is part of the threat model, because
if the attacker can write to that path, the attack vector is even
larger.

Signed-off-by: aszlig <aszlig@nix.build>
2018-08-03 06:46:43 +02:00
Eelco Dolstra
45bcf5416a
Merge branch 'prim_mapAttr-fix' of https://github.com/volth/nix 2018-07-31 20:05:07 +02:00
Eelco Dolstra
a7fb7d3cde
Merge pull request #2303 from volth/patch-4
parser.y: fix assoc of -> and < > <= >=
2018-07-23 11:38:15 +02:00
volth
deaa6e9a34
parser.y: right-associativity of -> 2018-07-23 07:28:48 +00:00
volth
85fe4a819c
parser.y: fix assoc of -> and < > <= >=
The parser allowed senseless `a > b > c` but disallowed `a -> b -> c` which seems valid
It might be a typo
2018-07-21 15:24:51 +00:00
volth
e2b114cfe1
prim_foldlStrict: call forceValue() before value is copied
forceValue() were called after a value is copied effectively forcing only one of the copies keeping another copy not evaluated.
This resulted in its evaluation of the same lazy value more than once (the number of hits is not big though)
2018-07-21 06:44:42 +00:00
Eelco Dolstra
c2de2ff385
Fix build on 32-bit systems and macOS
Apparently, on macOS, 'long' != 'int64_t'.

https://hydra.nixos.org/build/77100756
2018-07-11 21:12:18 +02:00
Eelco Dolstra
efd04888ca
Shup up a warning 2018-07-11 21:05:09 +02:00
Eelco Dolstra
875cd9da2b
Remove unused function printStats2()
Closes #2282.
2018-07-11 20:29:18 +02:00
volth
c03d73c1cf
prim_map, prim_genList: no need to force function argument which will be stored in mkApp 2018-07-07 23:30:17 +00:00
volth
627e28ba33 prim_mapAttrs: `f' must be evaluated lazily to avoid infinite recursion 2018-07-06 21:52:54 +00:00
volth
1515c65616 prim_concatMap: no need to force value 2018-07-05 15:33:33 +00:00
volth
e6bf1a79d7 prim_mapAttrs: must be lazy to avoid infinite recursion 2018-07-05 15:33:12 +00:00
volth
841747b0e6
prim_concatMap: allocate intermediate list on stack 2018-07-05 12:37:37 +00:00
volth
ee218f99ca
primops.cc: fix comment 2018-07-05 11:58:15 +00:00
volth
403a76a18f lib.concatMap and lib.mapAttrs to be builtins 2018-07-05 02:54:09 +00:00
Eelco Dolstra
ea3c9dab5f
Include cpptoml for build simplicity 2018-07-03 18:39:36 +02:00
Eelco Dolstra
3b1f54cf06
Add a fromTOML primop
This is primarily useful for processing Cargo.lock files.
2018-07-03 18:39:36 +02:00
Eelco Dolstra
a92ed973e5
Store floating point numbers in double precision
Even on 32-bit systems, Value has enough space to hold a double.
2018-07-03 18:39:32 +02:00
Eelco Dolstra
c905d8b0a8
GC_malloc -> GC_MALLOC
This makes it possible to build with -DGC_DEBUG.
2018-06-12 17:49:55 +02:00
Eelco Dolstra
455d1f01d0
Don't scan for roots in dynamic libraries
This reduces the risk of object liveness misdetection. For example,
Glibc has an internal variable "mp_" that often points to a Boehm
object, keeping it alive unnecessarily. Since we don't store any
actual roots in global variables, we can just disable data segment
scanning.

With this, the max RSS doing 100 evaluations of
nixos.tests.firefox.x86_64-linux.drvPath went from 718 MiB to 455 MiB.
2018-06-12 17:49:55 +02:00
Eelco Dolstra
30964103dc
Add temporary stats 2018-06-12 17:49:55 +02:00
Eelco Dolstra
24c6806994
Cache parse trees
This prevents EvalState::resetFileCache() from parsing everything all
over again.
2018-06-12 17:49:55 +02:00
Eelco Dolstra
6ad0a2f749
Remove duplicate definition of allocBytes() 2018-06-12 17:49:51 +02:00
Eelco Dolstra
5d4a9d5677
Fix static assertion failure on 32-bit systems 2018-05-30 17:47:30 +02:00
Eelco Dolstra
1672bcd230
Move evaluator-specific settings out of libstore 2018-05-30 13:29:50 +02:00
Eelco Dolstra
bd56b5fe3f
Merge pull request #2187 from bgamari/stoi-exceptions
json-to-value: Use strtol instead of strtoi
2018-05-28 14:54:49 +02:00
Ben Gamari
7f560b81ea json-to-value: Use strtol instead of strtoi
NixInt is long, so strtoi is too restrictive.
2018-05-26 18:43:46 -04:00
Will Dietz
93ae90de0f eval.cc: add message to static_assert, message can be omitted w/c++17 2018-05-26 14:04:41 -05:00
Eelco Dolstra
743359bc8a
Merge pull request #2157 from volth/bitwise
add builtins: __bitAnd,  __bitOr,  __bitXor
2018-05-24 15:00:39 +02:00
Eelco Dolstra
4bb8741b98
Make Env self-describing
If the Env denotes a 'with', then values[0] may be an Expr* cast to a
Value*. For code that generically traverses Values/Envs, it's useful
to know this.
2018-05-22 16:02:32 +02:00
Eelco Dolstra
9fd7cf98db
Memoise checkSourcePath()
This prevents hydra-eval-jobs from statting the same files over and
over again.
2018-05-22 13:02:14 +02:00
volth
6cc28c0589
add `mod' and bitwise builtins: camel-case function names 2018-05-16 10:52:19 +00:00
volth
f3c090f91c
add mod' and bitwise builtins: remove mod' and shifts 2018-05-16 06:57:11 +00:00