Commit graph

1289 commits

Author SHA1 Message Date
Ben Burdette
9d1cb0c5e6 with normaltxt, elide yellow color code instead of canceling it; use normaltxt on plain_string hintfmt 2020-06-22 11:32:20 -06:00
Ben Burdette
28b079067f
Update src/libutil/fmt.hh
Co-authored-by: John Ericson <git@JohnEricson.me>
2020-06-22 10:00:37 -06:00
Ben Burdette
be4f444175 tidying up 2020-06-19 16:58:12 -06:00
Ben Burdette
0309488a66 fmt -> hintfmt test 2020-06-19 16:46:49 -06:00
Ben Burdette
397dbe114e remove formathelper 2020-06-19 15:57:19 -06:00
John Ericson
507aa48739 WIP: Make Hash always store a valid hash type 2020-06-19 21:48:57 +00:00
Ben Burdette
b193aca4ae escape percents 2020-06-19 15:29:19 -06:00
Ben Burdette
db475f9e7e too few, too many args 2020-06-19 15:28:13 -06:00
Ben Burdette
cdddf24f25 add hintfmt test 2020-06-19 14:54:41 -06:00
Ben Burdette
54e8f550c9 addErrorTrace 2020-06-19 13:44:08 -06:00
John Ericson
e288c0987a Merge remote-tracking branch 'upstream/master' into validPathInfo-ca-proper-datatype 2020-06-19 18:44:24 +00:00
John Ericson
c98081d270 Merge remote-tracking branch 'upstream/master' into no-hash-type-unknown 2020-06-19 17:50:05 +00:00
John Ericson
c1892a5316 tabs -> spaces 2020-06-19 17:49:57 +00:00
John Ericson
2f0e395c99 Merge remote-tracking branch 'me/no-stringly-typed-derivation-output' into validPathInfo-ca-proper-datatype 2020-06-19 15:26:59 +00:00
John Ericson
b90cac3bad Remove uneeded = default for Hash 2020-06-19 15:00:38 +00:00
John Ericson
01dc8b0bab Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output 2020-06-19 14:59:05 +00:00
Eelco Dolstra
2886c92aef
Merge pull request #3669 from gilligan/add-compression-tests
Add compression unit tests
2020-06-19 13:59:04 +02:00
John Ericson
3fc58a9638 Remove some Base:: that crept in 2020-06-19 00:24:47 +00:00
John Ericson
669c3992e8 Merge branch 'no-hash-type-unknown' into validPathInfo-temp 2020-06-18 22:33:07 +00:00
John Ericson
15abb2aa2b Revert the enum struct change
Not a regular git revert as there have been many merges and things.
2020-06-18 22:11:26 +00:00
John Ericson
bbbf3602a3 Merge branch 'enum-class' into no-hash-type-unknown 2020-06-18 22:11:19 +00:00
John Ericson
40526fbea5 Merge remote-tracking branch 'upstream/master' into enum-class 2020-06-18 21:38:15 +00:00
Ben Burdette
4d1a4f0217 addTrace 2020-06-18 15:25:26 -06:00
Ben Burdette
e6f93b94fc Merge branch 'master' into caveman-LOCs 2020-06-18 13:07:53 -06:00
Eelco Dolstra
4d5169bdd5
Merge pull request #3707 from p01arst0rm/outdated-function-fix
replaced uncaught_exception with uncaught_exceptions
2020-06-17 18:26:01 +02:00
John Ericson
6403508f5a Use ansicolor.hh in nix repl rather than duplicates 2020-06-17 15:13:00 +00:00
Eelco Dolstra
1524752c17 Merge remote-tracking branch 'origin/master' into flakes 2020-06-17 10:26:52 +02:00
John Ericson
517f5980e2 Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output 2020-06-17 04:58:43 +00:00
p01arst0rm
e9970a34e8 appended ' __attribute__((weak)); ' to 'extern char * * environ ' 2020-06-17 03:25:34 +01:00
p01arst0rm
c9d06558b6 replaced uncaught_exception with uncaught_exceptions 2020-06-17 03:15:47 +01:00
Eelco Dolstra
df4da4f5da
Merge pull request #3702 from NixOS/store-path-cxx
Rewrite StorePath class in C++
2020-06-16 21:37:26 +02:00
John Ericson
fbf90bd693 Add another missing #include 2020-06-16 14:19:49 +00:00
Eelco Dolstra
759947bf72 StorePath: Rewrite in C++
On nix-env -qa -f '<nixpkgs>', this reduces maximum RSS by 20970 KiB
and runtime by 0.8%. This is mostly because we're not parsing the hash
part as a hash anymore (just validating that it consists of base-32
characters).

Also, replace storePathToHash() by StorePath::hashPart().
2020-06-16 14:28:41 +02:00
Tobias Pflug
cd8214c398 Fix logging unit tests 2020-06-16 10:23:15 +02:00
John Ericson
7e7e3b71f3 Add mising #include for strerror 2020-06-15 23:35:07 +00:00
Eelco Dolstra
8b099812ea Respect terminal width printing error messages 2020-06-15 19:35:31 +02:00
Eelco Dolstra
f20bb983ca Cleanup 2020-06-15 18:16:03 +02:00
Eelco Dolstra
5ed5d7acbd Improve "waiting for locks" messages
These are now shown in the progress bar.

Closes #3577.
2020-06-15 16:03:29 +02:00
Eelco Dolstra
e14e62fddd Remove trailing whitespace 2020-06-15 14:12:39 +02:00
Eelco Dolstra
1fb762d11f Get rid of explicit ErrorInfo constructors 2020-06-15 14:06:58 +02:00
Eelco Dolstra
fd64e4fb96 Disambiguate BaseError(Args) constructor
This means that 'throw Error({ ... ErrorInfo ... })' now works.
2020-06-15 13:50:33 +02:00
Eelco Dolstra
7a77762961 Merge branch 'errors-phase-2' of https://github.com/bburdette/nix 2020-06-15 11:46:31 +02:00
John Ericson
f6f01416b7 Use std::string_view in a few more places 2020-06-12 21:32:30 +00:00
Eelco Dolstra
2853ba4ab2 Fix build 2020-06-12 19:00:48 +02:00
Eelco Dolstra
00fa7e2205
Merge pull request #3674 from matthewbauer/allow-empty-hash2
Allow empty hash in derivations
2020-06-12 18:18:12 +02:00
Matthew Bauer
ea0d29d99a Provide base argument to to_string 2020-06-12 10:18:27 -05:00
Matthew Bauer
b260c9ee03 Add newHashAllowEmpty helper function
This replaces the copy&paste with a helper function in hash.hh.
2020-06-12 10:11:16 -05:00
Eelco Dolstra
4a4c063222
Merge pull request #3670 from gilligan/add-pool-tests
Add tests for pool.hh
2020-06-12 11:19:05 +02:00
Ben Burdette
ef1b3f21b6 Merge remote-tracking branch 'upstream/master' into errors-phase-2 2020-06-11 14:06:35 -06:00
Eelco Dolstra
ac4d43a31b
Merge pull request #3073 from tweag/machine-logs
Add an option to print the logs in a machine-readable format
2020-06-11 15:45:18 +02:00
Eelco Dolstra
dd9bb11d0d Move names.{cc,hh} to libstore 2020-06-11 15:42:18 +02:00
Tobias Pflug
4750d98bbd Add tests for pool.hh 2020-06-10 22:29:50 +02:00
Eelco Dolstra
b9ae1bdd7a
Merge pull request #3655 from zimbatm/hash-encoding-prepare
libutils/hash: remove default encoding
2020-06-10 11:48:38 +02:00
Ben Burdette
2f19650768 add file origin to Pos in stests 2020-06-08 11:21:17 -06:00
Ben Burdette
b1c53b034c Merge branch 'errors-phase-2' into caveman-LOCs 2020-06-08 11:10:13 -06:00
regnat
801112de1a Move progress-bar.cc to libmain
Needed so that we can include it as a logger in loggers.cc without
adding a dependency on nix

This also requires moving names.hh to libutil to prevent a circular
dependency between libmain and libexpr
2020-06-08 17:16:52 +02:00
Eelco Dolstra
6470450ab4 Add completion for --update-input 2020-06-08 16:20:00 +02:00
Tobias Pflug
cd6dbf951a Add compression unit tests 2020-06-08 11:34:37 +02:00
regnat
4983401440 Unify the printing of the logs between bar-with-logs and raw
Make the printing of the build logs systematically go through the
logger, and replicate the behavior of `no-build-output` by having two
different loggers (one that prints the build logs and one that doesn't)
2020-06-08 09:31:15 +02:00
regnat
170e86dff5 Make the logger customisable
Add a new `--log-format` cli argument to change the format of the logs.
The possible values are
- raw (the default one for old-style commands)
- bar (the default one for new-style commands)
- bar-with-logs (equivalent to `--print-build-logs`)
- internal-json (the internal machine-readable json format)
2020-06-08 09:31:15 +02:00
Ben Burdette
94c347577e set verbosity levels 2020-06-07 07:24:49 -06:00
Tobias Pflug
952e72c804 Add tests for logging.hh 2020-06-06 10:22:32 +02:00
John Ericson
e5cc1ebc5d Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output 2020-06-04 21:04:35 +00:00
John Ericson
a7b82fd006 Remove file which shouldn't be committed 2020-06-04 21:04:20 +00:00
Ben Burdette
94427ffee3 add some comments 2020-06-04 11:53:19 -06:00
Eelco Dolstra
f85606c431 Merge remote-tracking branch 'origin/master' into flakes 2020-06-04 13:16:28 +02:00
Eelco Dolstra
0f44b60e6d Make 'nix dev-shell' a deprecated alias for 'nix develop' 2020-06-04 11:14:19 +02:00
Ben Burdette
4335ba999b Merge remote-tracking branch 'upstream/master' into errors-phase-2 2020-06-03 17:00:00 -06:00
Ben Burdette
f97576c5d9 newline-as-prefix; no final newline in output. 2020-06-03 14:47:00 -06:00
Eelco Dolstra
c20591ddc3 Merge remote-tracking branch 'origin/master' into flakes 2020-06-03 16:15:22 +02:00
zimbatm
6ee03b8444
libutils/hash: remove default encoding
This will make it easier to reason about the hash encoding and switch to
SRI everywhere where possible.
2020-06-03 13:49:51 +02:00
John Ericson
01572c2198
Missing #include <cassert> in lru-cache.hh (#3654)
This was a latent bug that just appeared because of the tests that were
added. Remember to wait for CI! :)
2020-06-03 10:15:22 +00:00
John Ericson
1fcd3afc38 Fix hashes 2020-06-02 20:35:17 +00:00
John Ericson
c664e68b87 Fix to-base --type handler to correctly set std::optional flag
Now that we have a separate flag function, also describe why it is
optional.
2020-06-02 18:25:32 +00:00
John Ericson
d73dbc8e4c Remove hashingWithUnknownAlgoExits
A valid hash type must be provided now. The hash itself can still be
invalid, but that doesn't cause an `abort()`.
2020-06-02 16:28:54 +00:00
John Ericson
64cffb804a Merge remote-tracking branch 'upstream/master' into no-hash-type-unknown 2020-06-02 16:07:25 +00:00
John Ericson
450dcf2c1b Remove HashType::Unknown
Instead, `Hash` uses `std::optional<HashType>`. In the future, we may
also make `Hash` itself require a known hash type, encoraging people to
use `std::optional<Hash>` instead.
2020-06-02 15:52:13 +00:00
Ben Burdette
156d4f8bc8 remove extra space in SysErrors 2020-06-02 08:45:37 -06:00
Ben Burdette
d82d230b40 elide the 'ErrorInfo' in logError and logWarning calls 2020-06-02 08:22:24 -06:00
Eelco Dolstra
c16fdda3a6 Merge branch 'lru-tests' of https://github.com/gilligan/nix 2020-06-02 12:07:48 +02:00
Eelco Dolstra
e9fee8e6a7
src/libutil/tests/lru-cache.cc: Check erase()
Co-authored-by: James Lee <jbit@jbit.net>
2020-06-02 12:06:59 +02:00
Tobias Pflug
eca1ff7a9f Add tests for lru-cache.hh 2020-05-31 01:05:05 +02:00
Nikola Knezevic
77007d4eab Improve ref validity checking in fetchGit
The previous regex was too strict and did not match what git was allowing. It
could lead to `fetchGit` not accepting valid branch names, even though they
exist in a repository (for example, branch names containing `/`, which are
pretty standard, like `release/1.0` branches).

The new regex defines what a branch name should **NOT** contain. It takes the
definitions from `refs.c` in https://github.com/git/git and `git help
check-ref-format` pages.

This change also introduces a test for ref name validity checking, which
compares the result from Nix with the result of `git check-ref-format --branch`.
2020-05-30 12:29:35 +02:00
Ben Burdette
734283d636 Merge remote-tracking branch 'upstream/master' into errors-phase-2 2020-05-29 09:51:37 -06:00
Carlo Nucera
6dd471ebf6 Fixing the result of merge 2020-05-28 12:14:36 -04:00
Carlo Nucera
4f597fb901 Merge branch 'master' of github.com:NixOS/nix into enum-class 2020-05-28 10:58:22 -04:00
John Ericson
5b4cd84bc2 Merge remote-tracking branch 'me/more-rust-ffi' into no-stringly-typed-derivation-output 2020-05-28 10:35:53 -04:00
John Ericson
ef71caba29 Merge remote-tracking branch 'upstream/master' into more-rust-ffi 2020-05-28 10:31:46 -04:00
Eelco Dolstra
17ca997fc6 Merge remote-tracking branch 'origin/master' into flakes 2020-05-28 12:55:24 +02:00
Andreas Rammhold
fc137d2f00
config.hh: Add documentation
Provides some general overview on the mechanics of Config/Setting and
comments for the public methods of Config.
2020-05-27 17:47:18 +02:00
Andreas Rammhold
9df3d8ccd7
tests/config.cc: add tests for Config::applyConfig 2020-05-27 17:47:18 +02:00
Andreas Rammhold
e1b8c64c04
config.cc: extract parts of applyConfigFile into applyConfig
This moves the actual parsing of configuration contents into applyConfig
which applyConfigFile is then going to call. By changing this we can now
test the configuration file parsing without actually create a file on
disk.
2020-05-27 17:47:18 +02:00
Andreas Rammhold
93129cf1dd
Add unit tests for config.cc 2020-05-27 17:47:17 +02:00
Eelco Dolstra
66d3ac94c9 Merge pull request #3621 from gilligan/add-json-tests
Add unit tests for "json.hh"
2020-05-27 11:08:12 +02:00
Eelco Dolstra
dae6a267a8
Merge pull request #3625 from gilligan/xml-writer-tests
Add unit tests for xml-writer
2020-05-27 11:07:53 +02:00
Carlo Nucera
89a5ac9d3b Merge remote-tracking branch 'john-ericson/more-rust-ffi' into no-stringly-typed-derivation-output 2020-05-26 12:31:26 -04:00
Carlo Nucera
d49e65ba9d Merge remote-tracking branch 'john-ericson/enum-FileIngestionMethod' into no-stringly-typed-derivation-output 2020-05-26 12:30:48 -04:00
Carlo Nucera
0f3f901071 Merge remote-tracking branch 'origin/master' into more-rust-ffi 2020-05-26 11:46:42 -04:00
Carlo Nucera
c2f33edd1f
Update src/libutil/rust-ffi.hh
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2020-05-26 11:43:18 -04:00
Tobias Pflug
4b388e8431 Add unit tests for xml-writer 2020-05-25 18:34:55 +02:00
Tobias Pflug
c284700867 Add unit tests for "json.hh" 2020-05-25 11:57:45 +02:00
Tobias Pflug
ecc5c90dfc Add unit tests for hashing functions 2020-05-25 11:50:41 +02:00
Ben Burdette
0e49de6a2b position for stdin, string; (string) for trace; fix tests 2020-05-21 14:28:45 -06:00
Ben Burdette
6a420d672c print LOC for stdin, string args 2020-05-20 22:18:26 -06:00
Ben Burdette
85ce455b85 get code lines from the nix file 2020-05-20 17:25:02 -06:00
Tobias Pflug
a73a820a5d Add unit testes for url.cc
This adds tests for

- parseURL
- percentDecode
- decodeQuery
2020-05-20 16:37:35 +02:00
Ben Burdette
19694aa213 fix compile errors 2020-05-14 12:28:18 -06:00
Ben Burdette
ef9dd9f9bc formatting and a few minor changes 2020-05-13 15:56:39 -06:00
Ben Burdette
bfca5fc395 change status messages to info level 2020-05-13 09:52:36 -06:00
Ben Burdette
ecbb8e9c0a no blank line if no LOC 2020-05-12 14:41:30 -06:00
Ben Burdette
19cffc29c9 remove unused extra json fields 2020-05-12 12:09:12 -06:00
Eelco Dolstra
437614b479 Fix macOS build
macOS doesn't have GLOB_ONLYDIR.
2020-05-12 11:09:09 +02:00
Ben Burdette
b93c1bf3d6 fixes to merged code 2020-05-11 15:52:15 -06:00
Ben Burdette
59b1f5c701 Merge branch 'master' into errors-phase-2 2020-05-11 14:35:30 -06:00
Eelco Dolstra
27d34ef770 When completing flakerefs, only return directories 2020-05-11 22:04:13 +02:00
Eelco Dolstra
e917332d63 Shut up warnings while running completers 2020-05-11 21:38:17 +02:00
Ben Burdette
958e81987b switch from printError warnings to logWarnings 2020-05-11 13:02:16 -06:00
Eelco Dolstra
4c3c638a05 Cleanup 2020-05-11 15:57:45 +02:00
Eelco Dolstra
0884f180f5 Simplify 2020-05-10 21:50:32 +02:00
Eelco Dolstra
e0c19ee620 Add completion for paths 2020-05-10 21:35:07 +02:00
Eelco Dolstra
91ddee6bf0 nix: Implement basic bash completion 2020-05-10 20:32:21 +02:00
Ben Burdette
55eb717148 add pos to errorinfo, remove from hints 2020-05-08 18:18:28 -06:00
Tobias Pflug
181a47d884 Enable toLower umlauts test
Update comment and enable the test
2020-05-08 15:13:55 +02:00
Tobias Pflug
2191141274 Enable baseNameOf test
Add note about removal of trailing slashes in the doc comment of
baseNameOf and enabled the test.
2020-05-08 15:07:40 +02:00
Tobias Pflug
e3df9c2a6e Enable dirOf test
Adjusted the doc comment for `dirOf` to reflect the implementation
behavior.
2020-05-08 15:03:44 +02:00
Eelco Dolstra
5b8883faac
configure: Look for gtest 2020-05-08 12:09:37 +02:00
Eelco Dolstra
ca657525b8
Don't install unit tests 2020-05-08 12:03:27 +02:00
Eelco Dolstra
7898cdb75a
make check: Run unit tests 2020-05-08 11:49:40 +02:00
Eelco Dolstra
72b9d971bc
Fix warning 2020-05-08 11:35:57 +02:00
Eelco Dolstra
7cc7cef950
Move unit tests to sr/libutil/tests, use mk make rules 2020-05-08 11:34:09 +02:00
Ben Burdette
1b801cec40 pretending to be const 2020-05-07 16:43:36 -06:00
Tobias Pflug
1f3602a2c9 Remove replaceInSet
The function isn't being used anywhere so it seems safe to remove
2020-05-07 18:15:13 +02:00
Ben Burdette
e76ad2e48a implement SysError errno handling 2020-05-06 14:07:20 -06:00
Eelco Dolstra
2f8ee4578f
Merge remote-tracking branch 'origin/master' into flakes 2020-05-06 12:01:40 +02:00
Eelco Dolstra
74a1bfdcab
Merge pull request #3546 from guibou/nix_readfile_on_0_sized_files
builtins.readFile: do not truncate content
2020-05-06 11:33:55 +02:00
Eelco Dolstra
fd4911269f
Revert "Merge pull request #3558 from LnL7/ssh-ng-stderr"
This reverts commit 3ebfbecdd1, reversing
changes made to c089c52d5f.

https://github.com/NixOS/nix/pull/3558
2020-05-06 10:54:18 +02:00
Eelco Dolstra
6f3244ce45 Merge remote-tracking branch 'origin/master' into flakes 2020-05-05 18:59:33 +02:00
Eelco Dolstra
f132d82a79 nix --help: Group commands 2020-05-05 15:18:23 +02:00
Ben Burdette
7ffb5efdbc appending to hints; remove _printError 2020-05-04 16:19:57 -06:00
Ben Burdette
f30de61578 add normaltxt, yellowify->yellowtxt 2020-05-04 16:19:20 -06:00
Ben Burdette
afaa541013 affinity operator<< 2020-05-04 14:44:00 -06:00
Eelco Dolstra
a721a0b114 Flag: Use designated initializers 2020-05-04 22:40:19 +02:00
Ben Burdette
ab6f0b9641 convert some printError calls to logError 2020-05-03 08:01:25 -06:00
Daiderd Jordan
4769eea5e2
logging: handle build log lines in simple logger
The raw stderr output isn't logged anymore so the build logs need to be
printed by the default logger in order for the old commands like
nix-build to still show build output.
2020-05-02 23:40:53 +02:00
Ben Burdette
39ff80d031 errorinfo constructor test 2020-04-29 18:57:05 -06:00
Guillaume Bouchard
2e5be2a749 StringSink pre allocate
When used with `readFile`, we have a pretty good heuristic of the file
size, so `reserve` this in the `string`. This will save some allocation
/ copy when the string is growing.
2020-04-29 18:44:01 +02:00
Guillaume Bouchard
7afcb5af98 Remove the drain argument from readFile
Now it is always `drain` (see previous commit).
2020-04-29 18:43:45 +02:00
Ben Burdette
e2f61263eb uncrustify formatting 2020-04-29 10:14:32 -06:00
Guillaume Bouchard
5a34a473dd builtins.readFile: do not truncate content
This closes #3026 by allowing `builtins.readFile` to read a file with a
wrongly reported file size, for example, files in `/proc` may report a
file size of 0. Reading file in `/proc` is not a good enough motivation,
however I do think it just makes nix more robust by allowing more file
to be read.  Especially, I do considerer the previous behavior to be
dangerous because nix was previously reading truncated files. Examples
of file system which incorrectly report file size may be network file
system or dynamic file system (for performance reason, a dynamic file
system such as FUSE may generate the content of the file on demand).

```
nix-repl> builtins.readFile "/proc/version"
""
```

With this commit:

```
nix-repl> builtins.readFile "/proc/version"
"Linux version 5.6.7 (nixbld@localhost) (gcc version 9.3.0 (GCC)) #1-NixOS SMP Thu Apr 23 08:38:27 UTC 2020\n"
```

Here is a summary of the behavior changes:

- If the reported size is smaller, previous implementation
was silently returning a truncated file content. The new implementation
is returning the correct file content.

- If a file had a bigger reported file size, previous implementation was
failing with an exception, but the new implementation is returning the
correct file content. This change of behavior is coherent with this pull
request.

Open questions

- The behavior is unchanged for correctly reported file size, however
performances may vary because it uses the more complex sink interface.
Considering that sink is used a lot, I don't think this impacts the
performance a lot.
- `builtins.readFile` on an infinite file, such as `/dev/random` may
fill the memory.
- it does not support adding file to store, such as `${/proc/version}`.
2020-04-29 14:50:52 +02:00
Eelco Dolstra
5ada0831cf Merge remote-tracking branch 'origin/master' into flakes 2020-04-29 13:32:27 +02:00
Ben Burdette
22e6490311 Error classname as name 2020-04-28 21:06:08 -06:00
Ben Burdette
e51a757720 astyle format 2020-04-27 15:15:08 -06:00
Alyssa Ross
c05e20daa1
Fix long paths permanently breaking GC
Suppose I have a path /nix/store/[hash]-[name]/a/a/a/a/a/[...]/a,
long enough that everything after "/nix/store/" is longer than 4096
(MAX_PATH) bytes.

Nix will happily allow such a path to be inserted into the store,
because it doesn't look at all the nested structure.  It just cares
about the /nix/store/[hash]-[name] part.  But, when the path is deleted,
we encounter a problem.  Nix will move the path to /nix/store/trash, but
then when it's trying to recursively delete the trash directory, it will
at some point try to unlink
/nix/store/trash/[hash]-[name]/a/a/a/a/a/[...]/a.  This will fail,
because the path is too long.  After this has failed, any store deletion
operation will never work again, because Nix needs to delete the trash
directory before recreating it to move new things to it.  (I assume this
is because otherwise a path being deleted could already exist in the
trash, and then moving it would fail.)

This means that if I can trick somebody into just fetching a tarball
containing a path of the right length, they won't be able to delete
store paths or garbage collect ever again, until the offending path is
manually removed from /nix/store/trash.  (And even fixing this manually
is quite difficult if you don't understand the issue, because the
absolute path that Nix says it failed to remove is also too long for
rm(1).)

This patch fixes the issue by making Nix's recursive delete operation
use unlinkat(2).  This function takes a relative path and a directory
file descriptor.  We ensure that the relative path is always just the
name of the directory entry, and therefore its length will never exceed
255 bytes.  This means that it will never even come close to AX_PATH,
and Nix will therefore be able to handle removing arbitrarily deep
directory hierachies.

Since the directory file descriptor is used for recursion after being
used in readDirectory, I made a variant of readDirectory that takes an
already open directory stream, to avoid the directory being opened
multiple times.  As we have seen from this issue, the less we have to
interact with paths, the better, and so it's good to reuse file
descriptors where possible.

I left _deletePath as succeeding even if the parent directory doesn't
exist, even though that feels wrong to me, because without that early
return, the linux-sandbox test failed.

Reported-by: Alyssa Ross <hi@alyssa.is>
Thanks-to: Puck Meerburg <puck@puckipedia.com>
Tested-by: Puck Meerburg <puck@puckipedia.com>
Reviewed-by: Puck Meerburg <puck@puckipedia.com>
2020-04-27 20:50:17 +00:00
Ben Burdette
1ff42722ce error.hh 2020-04-26 14:47:41 -06:00
Ben Burdette
d4fd7b543e print dashes instead of empty name string 2020-04-25 12:05:26 -06:00
Ben Burdette
cdac083dc5 don't print blank lines for blank description 2020-04-24 21:40:13 -06:00
Ben Burdette
d8d4844b88 all things error to error.hh 2020-04-24 14:57:51 -06:00
Ben Burdette
d9632765a8 add has_value check; remove obslete friend class 2020-04-24 12:44:23 -06:00
Ben Burdette
833501f6f1 'what' string 2020-04-23 15:55:34 -06:00
Eelco Dolstra
ef4d3fc111 Merge remote-tracking branch 'origin/master' into flakes 2020-04-23 15:16:18 +02:00
Eelco Dolstra
c9d0cf7e02
Don't include error.hh in util.hh to prevent header bloat 2020-04-22 15:29:27 +02:00
Eelco Dolstra
16e3bf4537
Merge branch 'error-format' of https://github.com/bburdette/nix 2020-04-22 15:29:10 +02:00
Ben Burdette
e4fb9a3849 remove 'format' from Error constructor calls 2020-04-21 17:07:07 -06:00
Ben Burdette
d3052197fe add ErrorInfo to BaseError 2020-04-21 13:25:41 -06:00
Ben Burdette
4697552948 demoing other error levels than warn/error; rename line and file fields in errPos 2020-04-17 15:50:46 -06:00
Ben Burdette
3d5b1032a1 logError, logWarning; Logger functions; switch to Verbosity enum 2020-04-17 15:07:44 -06:00
Ben Burdette
12814806ef iomanip no longer needed 2020-04-16 10:48:15 -06:00
Eelco Dolstra
0858738355 Merge remote-tracking branch 'origin/master' into flakes 2020-04-16 18:27:37 +02:00
Eelco Dolstra
67a5941472 Logger: Add method for writing to stdout
Usually this just writes to stdout, but for ProgressBar, we need to
clear the current line, write the line to stdout, and then redraw the
progress bar.

(cherry picked from commit 696c026006)
2020-04-16 18:03:38 +02:00
Ben Burdette
96262e744e switch to structs, which don't need public: 2020-04-16 09:55:38 -06:00
Eelco Dolstra
696c026006 Logger: Add method for writing to stdout
Usually this just writes to stdout, but for ProgressBar, we need to
clear the current line, write the line to stdout, and then redraw the
progress bar.
2020-04-16 13:47:59 +02:00
Ben Burdette
057e5b6b2e move implementation to cc 2020-04-15 10:09:43 -06:00
Ben Burdette
adf03b0b8e Merge branch 'initializer-style' into error-format 2020-04-15 10:06:20 -06:00
zimbatm
895516cadf
add NIX_USER_CONF_FILES
Motivation: maintain project-level configuration files.

Document the whole situation a bit better so that it corresponds to the
implementation, and add NIX_USER_CONF_FILES that allows overriding
which user files Nix will load during startup.
2020-04-14 18:45:06 +02:00
Eelco Dolstra
d103c79144
Merge remote-tracking branch 'origin/master' into flakes 2020-04-09 15:54:21 +02:00
Nikola Knezevic
c330109bfa DataTransfer -> FileTransfer 2020-04-08 22:26:57 +02:00
Nikola Knezevic
a0c5931208 actDownload -> actDataTransfer 2020-04-08 22:26:57 +02:00
Ben Burdette
805ffe1bc9 indention 2020-04-08 11:33:46 -06:00
Ben Burdette
8c2bf15c4f format -> fmt 2020-04-08 11:17:02 -06:00
Ben Burdette
555baa8fb0 comments 2020-04-08 09:56:10 -06:00
Ben Burdette
54f91923c8 return of NixCode 2020-04-08 09:48:21 -06:00
Ben Burdette
47ed067d45 initializer style 2020-04-08 09:07:58 -06:00
Ben Burdette
00c507cc52 columnRange -> column 2020-04-07 14:36:32 -06:00
Ben Burdette
20c0984a46 remove columnrange; switch to fmt in error.cc 2020-04-07 10:14:15 -06:00
Eelco Dolstra
462421d345 Backport libfetchers from the flakes branch
This provides a pluggable mechanism for defining new fetchers. It adds
a builtin function 'fetchTree' that generalizes existing fetchers like
'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a
set of attributes, e.g.

  fetchTree {
    type = "git";
    url = "https://example.org/repo.git";
    ref = "some-branch";
    rev = "abcdef...";
  }

The existing fetchers are just wrappers around this. Note that the
input attributes to fetchTree are the same as flake input
specifications and flake lock file entries.

All fetchers share a common cache stored in
~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching
mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}).

This also adds support for Git worktrees (c169ea5904).
2020-04-07 09:03:14 +02:00
Ben Burdette
55c96b64e4 comment cleanup 2020-04-06 20:14:48 -06:00
Ben Burdette
ec449c8450 constructor style basically working 2020-04-06 19:43:22 -06:00
Ben Burdette
216263c36f Merge branch 'master' into error-format 2020-04-06 10:00:00 -06:00
Ben Burdette
9bb528d392 handle Pos instead of individual file/line/columnrange args 2020-04-03 13:15:59 -06:00
Ben Burdette
7b7801d3f0 variadic args for hint format 2020-04-03 08:48:20 -06:00
Ben Burdette
c6b3fcddb0 formatted with astyle 2020-04-02 16:02:40 -06:00
Ben Burdette
1c329ca433 indenting 2020-04-02 14:25:43 -06:00
Ben Burdette
e697884f65 using std:: everywhere; fix a formatting error; add exception flags 2020-04-01 21:30:19 -06:00
Ben Burdette
dd7b8183a5 indenting 2020-04-01 16:20:20 -06:00
Ben Burdette
8713aeac5e remove using std::*, switch to include guard 2020-04-01 15:51:14 -06:00
Ben Burdette
5b3aefff85 add some explanatory comments 2020-03-31 12:42:41 -06:00
Ben Burdette
9e7b89bf10 rename errors/warnings 2020-03-31 11:56:37 -06:00
Ben Burdette
09652f597c enum style 2020-03-31 09:36:20 -06:00
Eelco Dolstra
e1a94ad852 Backport 'nix dev-shell' from the flakes branch
This also adds a '--profile' option to 'nix build' (replacing 'nix-env
--set').
2020-03-30 19:16:45 +02:00
Ben Burdette
35c7bab09a build with make 2020-03-30 09:14:29 -06:00
Eelco Dolstra
e0a0ae0467 Move fetchers from libstore to libfetchers 2020-03-30 14:04:53 +02:00
John Ericson
87b32bab05 Use enum struct and drop prefixes
This does a few enums; the rest will be gotten in subsequent commits.
2020-03-29 11:23:15 -04:00
Ben Burdette
759f39800b remove util.hh from deps 2020-03-27 10:55:09 -06:00
Ben Burdette
00eb3fcb7a more cleanup 2020-03-27 10:13:46 -06:00
Ben Burdette
a3ef00be6c camelcase; optional hint 2020-03-27 10:03:02 -06:00
John Ericson
e433d4af4c Extend Rust FFI
Do idiomatic C++ copy and move constructors for a few things, so
wrapping structs' defaults can work.
2020-03-25 16:12:14 -04:00
Ben Burdette
d44c9c5581 some colors 2020-03-25 11:20:44 -06:00
Ben Burdette
3582dc3c88 programName as static member var 2020-03-25 10:52:03 -06:00
Ben Burdette
fc310eda3a switch to one level of builder function, not subobject functions 2020-03-24 14:24:57 -06:00
Ben Burdette
0166e7ab6d MkNixCode, MkErrLine approach 2020-03-24 11:21:35 -06:00
Ben Burdette
4171ab4bbd renaming 2020-03-24 09:18:23 -06:00
Eelco Dolstra
0a10854f85 Misc changes from the flakes branch 2020-03-24 14:34:47 +01:00
Eelco Dolstra
7a8de57d3e Pretty-print 'nix why-depends' / 'nix-store -q --tree' output
Extracted from 678301072f.
2020-03-24 14:26:23 +01:00
Eelco Dolstra
4260a22a55 absPath(): Use std::optional
(cherry picked from commit 1bf9eb21b7)
2020-03-24 14:25:28 +01:00
Eelco Dolstra
edc34cc1a2 Add function for quoting strings
(cherry picked from commit 7dcf5b011a)
2020-03-24 13:44:04 +01:00
Ben Burdette
aadd59d005 error test 2020-03-23 15:29:49 -06:00
Ben Burdette
f694f43d7d straightforward port of rust mockup code 2020-03-22 12:25:47 -06:00
Eelco Dolstra
6208d24c38 Merge remote-tracking branch 'origin/master' into flakes 2020-02-14 22:42:35 +01:00
Eelco Dolstra
d8972317fc Prevent uninitialized StorePath creation 2020-02-13 16:12:16 +01:00
Eelco Dolstra
8451298b35
Factor out TreeInfo 2020-02-01 22:06:55 +01:00
Eelco Dolstra
c5319e5d0b
Show "warning:" in yellow instead of red 2020-02-01 12:37:22 +01:00
Eelco Dolstra
8414685c0f Change lock file format to use an attribute representation of flake refs rather than URLs 2020-01-31 19:16:40 +01:00
Eelco Dolstra
e91f32f2b5 Use light box drawing symbols 2020-01-31 14:09:27 +01:00
Eelco Dolstra
678301072f nix flake list-inputs: Pretty-print the tree 2020-01-31 14:09:27 +01:00
Eelco Dolstra
9f4d8c6170 Pluggable fetchers
Flakes are now fetched using an extensible mechanism. Also lots of
other flake cleanups.
2020-01-21 22:56:04 +01:00
Eelco Dolstra
1bf9eb21b7 absPath(): Use std::optional 2020-01-21 22:39:43 +01:00
Eelco Dolstra
c7866733d7 Merge remote-tracking branch 'origin/master' into flakes 2019-12-20 12:45:58 +01:00
Eelco Dolstra
c84c843e33
tarfile.cc: Restore timestamps
This is needed to get the lastModified attribute of GitHub flakes.
2019-12-19 15:09:54 +01:00
Eelco Dolstra
2550c11373
tarfile.cc: Don't change the cwd
Nix is multithreaded so it's not safe to change the cwd.
2019-12-19 15:08:16 +01:00
Eelco Dolstra
be32da0ed0
tarfile.cc: Style fixes 2019-12-19 15:01:58 +01:00
Eelco Dolstra
ee235e764c
Merge branch 'libarchive' of https://github.com/yorickvP/nix 2019-12-19 14:47:18 +01:00
Eelco Dolstra
87873d0d65 Merge remote-tracking branch 'origin/master' into flakes 2019-12-18 14:25:25 +01:00
Eelco Dolstra
f8abbdd456 Add priority setting to stores
This allows overriding the priority of substituters, e.g.

  $ nix-store --store ~/my-nix/ -r /nix/store/df3m4da96d84ljzxx4mygfshm1p0r2n3-geeqie-1.4 \
    --substituters 'http://cache.nixos.org?priority=100 daemon?priority=10'

Fixes #3264.
2019-12-17 17:17:53 +01:00
Eelco Dolstra
ad6b738ed8 Merge remote-tracking branch 'origin/master' into flakes 2019-12-16 20:17:21 +01:00
Eelco Dolstra
54bf5ba422 nix-store -r: Handle symlinks to store paths
Fixes #3270.
2019-12-16 19:11:47 +01:00
Eelco Dolstra
ac9cc2ec08 Move some code 2019-12-13 19:10:39 +01:00
Eelco Dolstra
b4edc3ca61 Don't leak exceptions 2019-12-13 19:05:26 +01:00
Eelco Dolstra
e6bd88878e Improve gzip error message 2019-12-13 19:05:26 +01:00
Eelco Dolstra
ca87707c90 Get rid of CBox 2019-12-13 19:05:26 +01:00
Eelco Dolstra
3e787423c2 Remove FIXME 2019-12-13 12:55:52 +01:00
Tom Bereknyei
c6295a3afd Initial gzip support
Closes #3256
2019-12-13 03:34:15 -05:00
Eelco Dolstra
ecb3a1afa2 Merge remote-tracking branch 'origin/master' into flakes 2019-12-11 14:53:30 +01:00
Eelco Dolstra
bbe97dff8b Make the Store API more type-safe
Most functions now take a StorePath argument rather than a Path (which
is just an alias for std::string). The StorePath constructor ensures
that the path is syntactically correct (i.e. it looks like
<store-dir>/<base32-hash>-<name>). Similarly, functions like
buildPaths() now take a StorePathWithOutputs, rather than abusing Path
by adding a '!<outputs>' suffix.

Note that the StorePath type is implemented in Rust. This involves
some hackery to allow Rust values to be used directly in C++, via a
helper type whose destructor calls the Rust type's drop()
function. The main issue is the dynamic nature of C++ move semantics:
after we have moved a Rust value, we should not call the drop function
on the original value. So when we move a value, we set the original
value to bitwise zero, and the destructor only calls drop() if the
value is not bitwise zero. This should be sufficient for most types.

Also lots of minor cleanups to the C++ API to make it more modern
(e.g. using std::optional and std::string_view in some places).
2019-12-10 22:06:05 +01:00
Yorick van Pelt
eba82b7c88
further clean up libarchive code 2019-12-09 17:21:46 +07:00
Puck Meerburg
28ee687adf Clean up libarchive support 2019-12-07 18:12:21 +00:00
Yorick van Pelt
fe7ec70e6b
remove rust unpack_tarfile ffi 2019-12-07 23:28:31 +07:00
Yorick van Pelt
1355554d12
code 'cleanup' 2019-12-07 23:23:11 +07:00
Yorick van Pelt
f54c168031
add wrapper function around libarchive to c++ errors 2019-12-07 23:10:27 +07:00
Yorick van Pelt
232b390766
fixup! libarchive proof of concept 2019-12-07 23:00:37 +07:00
Yorick van Pelt
9ff5f6492f
libarchive proof of concept 2019-12-07 22:35:14 +07:00
Eelco Dolstra
ab88f4bbd4 Merge remote-tracking branch 'origin/master' into flakes 2019-12-05 20:53:32 +01:00
Eelco Dolstra
5e449b43ed Initialize Command::_name
(cherry picked from commit d0a769cb06)
2019-12-05 20:21:22 +01:00
Eelco Dolstra
ac67685606 Make subcommand construction in MultiCommand lazy
(cherry picked from commit a0de58f471)
2019-12-05 20:19:26 +01:00
Eelco Dolstra
f964f428fe Move Command and MultiCommand to libutil
(cherry picked from commit f70434b1fb)
2019-12-05 20:13:47 +01:00
Eelco Dolstra
092af3c826 Eliminate more pass-by-value in variadic calls 2019-12-05 19:58:52 +01:00
Eelco Dolstra
603b2f583c Revert "Make fmt() non-recursive"
This reverts commit 2b761d5f50.

Also *really* make fmt() take arguments by reference.
2019-12-05 19:58:49 +01:00
Eelco Dolstra
334b8f8af1 fmt(): Pass arguments by reference rather than by value 2019-12-05 17:40:46 +01:00
Eelco Dolstra
c3c23a52ee Merge remote-tracking branch 'origin/master' into flakes 2019-12-04 00:31:09 +01:00
Eelco Dolstra
949dc84894 Fix segfault on i686-linux
https://hydra.nixos.org/build/107467517

Seems that on i686-linux, gcc and rustc disagree on how to return
1-word structs: gcc has the caller pass a pointer to the result, while
rustc has the callee return the result in a register. Work around this
by using a bare pointer.
2019-11-27 14:17:15 +01:00
Eelco Dolstra
8918bae098 Drop remaining uses of external "tar"
Also, fetchGit now runs in O(1) memory since we pipe the output of
'git archive' directly into unpackTarball() (rather than first reading
it all into memory).
2019-11-26 22:07:28 +01:00
Eelco Dolstra
d33dd6e6c0 Move code around 2019-11-26 22:07:28 +01:00
Eelco Dolstra
b7fba16613 Move code around 2019-11-26 22:07:28 +01:00
Eelco Dolstra
ba87b08f85
getEnv(): Return std::optional
This allows distinguishing between an empty value and no value.
2019-11-22 16:18:13 +01:00
Eelco Dolstra
5ee23c35b9
Merge pull request #3219 from Ericson2314/semicolons
Fix extra semicolons warnings
2019-11-11 12:13:51 +01:00
John Ericson
4c34054673 Remove unneeded semicolons 2019-11-10 11:24:47 -05:00
John Ericson
96e6e680c1 Fix extra ; warnings involving MakeError 2019-11-10 11:24:47 -05:00
Eelco Dolstra
2c1e05ae93
Merge remote-tracking branch 'origin/master' into flakes 2019-11-08 15:22:54 +01:00
Eelco Dolstra
0d6774468c
Move editorFor srom libutil to nix
libutil should not depend on libexpr.
2019-11-08 15:13:32 +01:00
Eelco Dolstra
f730841db4
Merge remote-tracking branch 'origin/master' into flakes 2019-11-07 11:44:02 +01:00
Eelco Dolstra
39a2e166dd
Cleanup 2019-11-06 16:53:02 +01:00
Eelco Dolstra
88c452d160
Merge remote-tracking branch 'origin/master' into flakes 2019-11-06 10:56:33 +01:00
Eelco Dolstra
852554bb16
Merge branch 'nix-repl-e' of https://github.com/zimbatm/nix 2019-11-05 11:20:53 +01:00
Eelco Dolstra
1b600ecd14
Don't use SOCK_CLOEXEC on macOS
https://hydra.nixos.org/build/105428308
2019-11-05 10:25:09 +01:00
Eelco Dolstra
b81d9d26f5
Merge remote-tracking branch 'origin/master' into flakes 2019-11-04 22:29:31 +01:00
Harald van Dijk
c935ad3f02
Fix progress bar when nix-prefetch-url is piped.
The intent of the code was that if the window size cannot be determined,
it would be treated as having the maximum possible size. Because of a
missing assignment, it was actually treated as having a width of 0.

The reason the width could not be determined was because it was obtained
from stdout, not stderr, even though the printing was done to stderr.

This commit addresses both issues.
2019-11-03 21:46:59 +00:00
Eelco Dolstra
63b99af85a
Move Unix domain socket creation to libutil
Also drop multithread-unfriendly hacks like doing a temporary
chmod/umask.
2019-10-29 13:30:51 +01:00
zimbatm
3774fe55fd
editorFor: take a pos object instead 2019-10-28 21:36:34 +01:00
Eelco Dolstra
f1c0b2c0e1
Add O(1)-memory copyPath() function 2019-10-27 18:18:58 +01:00
zimbatm
207a537343
libutil: add editorFor heuristic 2019-10-23 16:48:28 +02:00
Eelco Dolstra
9a18f544ac Merge remote-tracking branch 'origin/master' into flakes 2019-10-21 18:48:21 +02:00
Eelco Dolstra
0abb3ad537 Allow content-addressable paths to have references
This adds a command 'nix make-content-addressable' that rewrites the
specified store paths into content-addressable paths. The advantage of
such paths is that 1) they can be imported without signatures; 2) they
can enable deduplication in cases where derivation changes do not
cause output changes (apart from store path hashes).

For example,

  $ nix make-content-addressable -r nixpkgs.cowsay
  rewrote '/nix/store/g1g31ah55xdia1jdqabv1imf6mcw0nb1-glibc-2.25-49' to '/nix/store/48jfj7bg78a8n4f2nhg269rgw1936vj4-glibc-2.25-49'
  ...
  rewrote '/nix/store/qbi6rzpk0bxjw8lw6azn2mc7ynnn455q-cowsay-3.03+dfsg1-16' to '/nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16'

We can then copy the resulting closure to another store without
signatures:

  $ nix copy --trusted-public-keys '' ---to ~/my-nix /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16

In order to support self-references in content-addressable paths,
these paths are hashed "modulo" self-references, meaning that
self-references are zeroed out during hashing. Somewhat annoyingly,
this means that the NAR hash stored in the Nix database is no longer
necessarily equal to the output of "nix hash-path"; for
content-addressable paths, you need to pass the --modulo flag:

  $ nix path-info --json /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16  | jq -r .[].narHash
  sha256:0ri611gdilz2c9rsibqhsipbfs9vwcqvs811a52i2bnkhv7w9mgw

  $ nix hash-path --type sha256 --base32 /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16
  1ggznh07khq0hz6id09pqws3a8q9pn03ya3c03nwck1kwq8rclzs

  $ nix hash-path --type sha256 --base32 /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16 --modulo iq6g2x4q62xp7y7493bibx0qn5w7xz67
  0ri611gdilz2c9rsibqhsipbfs9vwcqvs811a52i2bnkhv7w9mgw
2019-10-21 17:47:24 +02:00
Eelco Dolstra
90df25ef7e
Fix build 2019-10-10 15:07:50 +02:00
Eelco Dolstra
e99bb91217
Merge remote-tracking branch 'origin/master' into flakes 2019-10-10 12:54:37 +02:00
Eelco Dolstra
9348f9291e
nix-env: Create ~/.nix-profile automatically 2019-10-09 23:35:01 +02:00
Eelco Dolstra
55bba8e4f5
Make std::uncaught_exception warning less noisy 2019-10-09 23:04:11 +02:00
Eelco Dolstra
a323b7826c
Merge remote-tracking branch 'origin/master' into flakes 2019-10-04 17:26:32 +02:00
Eelco Dolstra
93b1ce1ac5
Revert "std::uncaught_exception() -> std::uncaught_exceptions()"
This reverts commit 6b83174fff because
it doesn't work on macOS yet.

https://hydra.nixos.org/build/102617587
2019-10-04 16:34:59 +02:00
Eelco Dolstra
092ee24627
Merge remote-tracking branch 'origin/master' into flakes 2019-09-18 12:02:50 +02:00
Eelco Dolstra
6b83174fff std::uncaught_exception() -> std::uncaught_exceptions()
The former is deprecated in C++17. Fixes a clang warning.
2019-09-13 20:05:44 +02:00
Eelco Dolstra
4caeefaf00
Revert "Remove obsolete OpenSSL locking code"
This reverts commit aeb695c007.
2019-09-04 14:06:52 +02:00
Eelco Dolstra
e302ba0e65
Merge remote-tracking branch 'origin/master' into flakes 2019-09-04 13:30:11 +02:00
Eelco Dolstra
f186000367
Add some noexcepts
This is to assert that callback functions should never throw (since
the context in which they're called may not be able to handle the
exception).
2019-09-03 13:45:35 +02:00
Eelco Dolstra
7348653ff4
Ensure that Callback is called only once
Also, make Callback movable but uncopyable.
2019-09-03 13:45:35 +02:00
Eelco Dolstra
aeb695c007
Remove obsolete OpenSSL locking code
OpenSSL 1.1.1 no longer needs this (2e52e7df51).

This shuts up a clang warning about opensslLockCallback being unused.
2019-09-02 17:50:44 +02:00
Eelco Dolstra
c693f80b81
Shut up some clang warnings 2019-09-02 17:43:27 +02:00
Eelco Dolstra
ebc4dae517 Merge remote-tracking branch 'origin/master' into flakes 2019-08-29 16:11:38 +02:00
Eelco Dolstra
a2c4fcd5e9 Don't rely on st_blocks
It doesn't seem very reliable on ZFS.
2019-08-29 14:49:58 +02:00
zimbatm
91b00b145f
libutil: add SizedSource
Introduce the SizeSource which allows to bound how much data is being
read from a source. It also contains a drainAll() function to discard
the rest of the source, useful to keep the nix protocol in sync.
2019-08-16 15:05:40 +02:00
Eelco Dolstra
1d750e0587
Merge remote-tracking branch 'origin/master' into flakes 2019-08-08 15:49:13 +02:00
regnat
7c5596734f
Add a post-build-hook
Passing `--post-build-hook /foo/bar` to a nix-* command will cause
`/foo/bar` to be executed after each build with the following
environment variables set:

    DRV_PATH=/nix/store/drv-that-has-been-built.drv
    OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev

This can be useful in particular to upload all the builded artifacts to
the cache (including the ones that don't appear in the runtime closure
of the final derivation or are built because of IFD).

This new feature prints the stderr/stdout output to the `nix-build`
and `nix build` client, and the output is printed in a Nix 2
compatible format:

    [nix]$ ./inst/bin/nix-build ./test.nix
    these derivations will be built:
      /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv
    building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'...
    hello!
    bye!
    running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'...
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + printf 'very important stuff'
    /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation

    [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix
    my-example-derivation> hello!
    my-example-derivation> bye!
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + printf 'very important stuff'
    [1 built, 0.0 MiB DL]

Co-authored-by: Graham Christensen <graham@grahamc.com>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-08-02 10:48:15 -04:00
Eelco Dolstra
4205234f26
Merge remote-tracking branch 'origin/master' into flakes 2019-07-11 13:13:19 +02:00
Eelco Dolstra
03f09e1d18
Revert "Fix 'error 9 while decompressing xz file'"
This reverts commit 78fa47a7f0.
2019-07-10 19:46:15 +02:00
Eelco Dolstra
cc218b15ba Merge remote-tracking branch 'origin/master' into flakes 2019-07-06 21:06:22 +02:00
Niklas Hambüchen
a96006d97f Get BOOST_LDFLAGS from autoconf, fix Ubuntu 16.04 build.
Our use of boost::coroutine2 depends on -lboost_context,
which in turn depends on `-lboost_thread`, which in turn depends
on `-lboost_system`.

I suspect that this builds on nix only because of low-level hacks
like NIX_LDFLAGS.

This commit passes the proper linker flags, thus fixing bootstrap
builds on non-nix distributions like Ubuntu 16.04.

With these changes, I can build Nix on Ubuntu 16.04 using:

    ./bootstrap.sh
    ./configure --prefix=$HOME/editline-prefix \
      --disable-doc-gen \
      CXX=g++-7 \
      --with-boost=$HOME/boost-prefix \
      EDITLINE_CFLAGS=-I$HOME/editline-prefix/include \
      EDITLINE_LIBS=-leditline \
      LDFLAGS=-L$HOME/editline-prefix/lib
    make

where

* g++-7 comes from gcc-7 from
  https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test,
* editline 1.14 from https://github.com/troglobit/editline/releases/tag/1.14.0
	was installed into `$HOME/editline-prefix`
  (because Ubuntu 16.04's `editline` is too old to have the function nix uses),
* boost 1.66 from
	https://www.boost.org/doc/libs/1_66_0/more/getting_started/unix-variants.html
	was installed into $HOME/boost-prefix (because Ubuntu 16.04 only has 1.58)
2019-07-03 04:32:25 +02:00
Eelco Dolstra
5600b070a7
Add "warning" verbosity level
This ensures that "nix" shows warnings. Previously these were hidden
because they were at "info" level.

(cherry picked from commit 615a9d031d)
2019-06-25 12:44:17 +02:00
Eelco Dolstra
78fa47a7f0
Fix 'error 9 while decompressing xz file'
Once we've started writing data to a Sink, we can't restart a download
request, because then we end up writing duplicate data to the
Sink. Therefore we shouldn't handle retries in Downloader but at a
higher level (in particular, in copyStorePath()).

Fixes #2952.

(cherry picked from commit a67cf5a358)
2019-06-24 21:59:51 +02:00
Eelco Dolstra
a67cf5a358
Fix 'error 9 while decompressing xz file'
Once we've started writing data to a Sink, we can't restart a download
request, because then we end up writing duplicate data to the
Sink. Therefore we shouldn't handle retries in Downloader but at a
higher level (in particular, in copyStorePath()).

Fixes #2952.
2019-06-24 21:50:08 +02:00
Eelco Dolstra
d0a769cb06
Initialize Command::_name 2019-06-19 23:37:40 +02:00
Eelco Dolstra
a0de58f471
Make subcommand construction in MultiCommand lazy 2019-06-18 16:25:31 +02:00
Eelco Dolstra
615a9d031d
Add "warning" verbosity level
This ensures that "nix" shows warnings. Previously these were hidden
because they were at "info" level.
2019-06-17 09:12:03 +02:00
Eelco Dolstra
ae7b56cd9a
Get last commit time of github flakes 2019-05-29 10:10:40 +02:00
Eelco Dolstra
0f5032c5a4 Merge remote-tracking branch 'origin/master' into flakes 2019-05-15 20:51:29 +02:00
Eelco Dolstra
8f6c72faee
Merge pull request #2810 from NixOS/print-build-logs
nix: Add --print-build-logs flag
2019-05-15 20:38:38 +02:00
Eelco Dolstra
66b8a62101 nix: Add --print-build-logs flag
This causes 'nix' to print build log output to stderr rather than
showing the last log line in the progress bar. Log lines are prefixed
by the name of the derivation (minus the version string), e.g.

  binutils> make[1]: Leaving directory '/build/binutils-2.31.1'
  binutils-wrapper> unpacking sources
  binutils-wrapper> patching sources
  ...
  binutils-wrapper> Using dynamic linker: '/nix/store/kr51dlsj9v5cr4n8700jliyz8v5b2q7q-bootstrap-stage0-glibc/lib/ld-linux-x86-64.so.2'
  bootstrap-stage2-gcc-wrapper> unpacking sources
  ...
  linux-headers> unpacking sources
  linux-headers> unpacking source archive /nix/store/8javli69jhj3bkql2c35gsj5vl91p382-linux-4.19.16.tar.xz
2019-05-15 17:33:56 +02:00
Graham Christensen
f1b8e9efe7
runProgram: Uncomment chdir support 2019-05-12 17:03:01 -04:00
Graham Christensen
b4a05edbfe
runProgram: support gid, uid, chdir 2019-05-12 13:17:27 -04:00
Graham Christensen
6df61db060
diff hook: execute as the build user, and pass the temp dir 2019-05-12 13:17:26 -04:00
Eelco Dolstra
2bc55aba1e
Merge remote-tracking branch 'origin/master' into flakes 2019-05-08 14:30:27 +02:00
Eelco Dolstra
f9a2ea4486
Fix "Bad system call" running i686-linux binaries on x86_64-linux
To determine which seccomp filters to install, we were incorrectly
using settings.thisSystem, which doesn't denote the actual system when
--system is used.

Fixes #2791.
2019-05-03 10:48:33 +02:00
Eelco Dolstra
8ec77614f6 Move createTempFile to libutil 2019-05-02 21:28:41 +02:00
Eelco Dolstra
7dcf5b011a Add function for quoting strings 2019-05-02 21:09:52 +02:00
Eelco Dolstra
21d5abfc14 Merge remote-tracking branch 'tweag/flake-clone' into flakes 2019-04-24 12:47:35 +02:00
Eelco Dolstra
6960ee929d Clean up exportGit argument handling 2019-04-19 11:34:23 +02:00
Nick Van den Broeck
b42ba08fc8 Add command flake clone 2019-04-17 13:31:09 +02:00
Eelco Dolstra
edb3836696 Merge remote-tracking branch 'tweag/auto-update-flake' into flakes 2019-03-26 11:50:22 +01:00
Eelco Dolstra
b5565a7081 Merge remote-tracking branch 'origin/master' into flakes 2019-03-26 11:44:14 +01:00
Nick Van den Broeck
d4ee8afd59 Implemented --flake flag for nix build
Also fixed Eelco's PR comments
2019-03-22 11:16:20 +01:00
Eelco Dolstra
ef52ccf035
experimental/optional -> optional 2019-03-14 14:10:52 +01:00
Will Dietz
0963479741
archive.cc: ignore more posix_fallocate "not supported" error codes
Fixes w/musl.
2019-03-01 10:31:17 +08:00
Eelco Dolstra
c8a0b9d5cb
experimental/optional -> optional 2019-02-12 13:43:32 +01:00
Eelco Dolstra
f70434b1fb
Move Command and MultiCommand to libutil 2019-02-11 11:55:25 +01:00
Eelco Dolstra
01d07b1e92
Revert "Restore parent mount namespace before executing a child process"
This reverts commit a0ef21262f. This
doesn't work in 'nix run' and nix-shell because setns() fails in
multithreaded programs, and Boehm GC mark threads are uncancellable.

Fixes #2646.
2019-02-05 10:49:19 +01:00
John Ericson
fef9f5653b Remove mentions of libformat, it no longer exists 2019-01-05 14:31:29 -05:00
John Ericson
e10d6ed2a7 brotli is only used as a library now 2019-01-05 14:25:54 -05:00
Eelco Dolstra
6024dc1d97
Support SRI hashes
SRI hashes (https://www.w3.org/TR/SRI/) combine the hash algorithm and
a base-64 hash. This allows more concise and standard hash
specifications. For example, instead of

  import <nix/fetchurl.nl> {
    url = https://nixos.org/releases/nix/nix-2.1.3/nix-2.1.3.tar.xz;
    sha256 = "5d22dad058d5c800d65a115f919da22938c50dd6ba98c5e3a183172d149840a4";
  };

you can write

  import <nix/fetchurl.nl> {
    url = https://nixos.org/releases/nix/nix-2.1.3/nix-2.1.3.tar.xz;
    hash = "sha256-XSLa0FjVyADWWhFfkZ2iKTjFDda6mMXjoYMXLRSYQKQ=";
  };

In fixed-output derivations, the outputHashAlgo is no longer mandatory
if outputHash specifies the hash (either as an SRI or in the old
"<type>:<hash>" format).

'nix hash-{file,path}' now print hashes in SRI format by default. I
also reverted them to use SHA-256 by default because that's what we're
using most of the time in Nixpkgs.

Suggested by @zimbatm.
2018-12-13 14:30:52 +01:00
volth
21d494da83
probably typo
...at least MSVC unable to compile this
2018-12-13 02:45:50 +00:00
Eelco Dolstra
a32ff2573b
Fix 'Read-only file system' when building a derivation 2018-11-15 13:20:39 +01:00
Eelco Dolstra
a0ef21262f
Restore parent mount namespace before executing a child process
This ensures that they can't write to /nix/store. Fixes #2535.
2018-11-13 16:15:30 +01:00
Linus Heckemann
f3b8173a93 config: use all of XDG_CONFIG_DIRS
Previously, config would only be read from XDG_CONFIG_HOME. This change
allows reading config from additional directories, which enables e.g.
per-project binary caches or chroot stores with the help of direnv.
2018-10-31 09:24:43 +01:00
Eelco Dolstra
79e358ce6d RemoteStore: Close connection if an exception occurs
Fixes #2075.
2018-10-16 23:36:15 +02:00
Eelco Dolstra
ed78582847
sinkToSource(): Start the coroutine lazily
In particular this causes copyStorePath() from HttpBinaryCacheStore to
only start a download if needed. E.g. if the destination LocalStore
goes to sleep waiting for the path lock and another process creates
the path, then LocalStore::addToStore() will never read from the
source so we don't have to do the download.
2018-09-26 21:19:34 +02:00
Eelco Dolstra
44e86304b6
Make NAR header check more robust
Changes

  std::bad_alloc

into

  bad archive: input doesn't look like a Nix archive
2018-09-26 12:03:58 +02:00
Eelco Dolstra
901dfc7978
Fix build failure if parallel xz is not available 2018-09-17 16:36:11 +02:00
Eelco Dolstra
8327a7a8fa
Merge branch 'dirOf-relative' of https://github.com/lheckemann/nix 2018-09-13 14:33:12 +02:00
Matthew Bauer
74f6d8767d Get effective user in Nix commands
‘geteuid’ gives us the user that the command is being run as,
including in setuid modes. By using geteuid to determind id, we can
avoid the ‘sudo -i’ hack when upgrading Nix. So now, upgrading Nix on
macOS is as simple as:

$ sudo nix-channel --update
$ sudo nix-env -u

$ sudo launchctl stop org.nixos.nix-daemon
$ sudo launchctl start org.nixos.nix-daemon
or
$ sudo systemctl restart nix-daemon
2018-09-04 19:32:39 -05:00
Eelco Dolstra
ebe3d2d370 Improve 'coroutine has finished' error message 2018-08-21 15:22:04 +02:00
Eelco Dolstra
6317c65937 Fix warnings in compression.cc 2018-08-21 15:20:23 +02:00
Linus Heckemann
d7402c9cd5 dirOf: allow use on non-absolute paths 2018-08-13 11:27:35 +02:00
Eelco Dolstra
d3761f5f8b
Fix Brotli decompression in 'nix log'
This didn't work anymore since decompression was only done in the
non-coroutine case.

Decompressors are now sinks, just like compressors.

Also fixed a bug in bzip2 API handling (we have to handle BZ_RUN_OK
rather than BZ_OK), which we didn't notice because there was a missing
'throw':

  if (ret != BZ_OK)
      CompressionError("error while compressing bzip2 file");
2018-08-06 15:40:29 +02:00
Eelco Dolstra
fa4def3d46
Require libbrotli 2018-08-06 14:06:54 +02:00
Eelco Dolstra
d9e8ab0ff2
Add a check for broken Boost versions
In some Boost versions, coroutines don't propagate exceptions
properly, causing Nix to fail with the exception 'coroutine has
finished'.
2018-07-31 16:35:05 +02:00
Eelco Dolstra
a92ed973e5
Store floating point numbers in double precision
Even on 32-bit systems, Value has enough space to hold a double.
2018-07-03 18:39:32 +02:00
Will Dietz
f601bc0492 sync.hh: return cv_status instead of dropping it, in case useful. 2018-06-18 17:31:00 -05:00
Eelco Dolstra
7d21863bb3
Make <nix/fetchurl.nix> run in constant memory
E.g.

  nix-build --store ~/my-nix/ -E 'import <nix/fetchurl.nix> { url = https://cache.nixos.org/nar/0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv.nar.xz; sha256 = "0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv"; }'

now runs in 17 MiB (was 70 MiB), while

  nix-build --store ~/my-nix/ -E 'import <nix/fetchurl.nix> { url = https://cache.nixos.org/nar/0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv.nar.xz; sha256 = "0d2fxljdih3nc5dqx41hjzic3141ajil94m8kdbpryq569dpsbvb"; unpack = true; }'

runs in 17 MiB (was 346 MiB).
2018-05-30 13:42:51 +02:00
Eelco Dolstra
08ec757726
Make LocalBinaryCacheStore::narFromPath() run in constant memory
This reduces memory consumption of

  nix copy --from file://... --to ~/my-nix /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79

from 514 MiB to 18 MiB for an uncompressed binary cache, and from 192
MiB to 53 MiB for a bzipped binary cache. It may also be faster
because fetching can happen concurrently with decompression/writing.

Continuation of 48662d151b.

Issue https://github.com/NixOS/nix/issues/1681.
2018-05-30 13:34:57 +02:00
Eelco Dolstra
81ea8bd5ce
Simplify the callback mechanism 2018-05-30 13:34:37 +02:00
Eelco Dolstra
737ed88f35
Modularize config settings
Allow global config settings to be defined in multiple Config
classes. For example, this means that libutil can have settings and
evaluator settings can be moved out of libstore. The Config classes
are registered in a new GlobalConfig class to which config files
etc. are applied.

Relevant to https://github.com/NixOS/nix/issues/2009 in that it
removes the need for ad hoc handling of useCaseHack, which was the
underlying cause of that issue.
2018-05-30 13:28:01 +02:00
Will Dietz
b08923b4a0 serialise: fix buffer size used, hide method for internal use only
Fixes #2169.
2018-05-21 17:30:38 -05:00
Eelco Dolstra
53ec5ac69f
Fix some random -Wconversion warnings 2018-05-02 13:56:34 +02:00
Eelco Dolstra
4a2c948943
Fix bzip2 compression of files > 4 GiB
Bzip2's 'avail_in' parameter is declared as an unsigned int, so
assigning a size_t length to it led to silent truncation.

Fixes #2111.
2018-05-02 12:54:30 +02:00
Eelco Dolstra
28eb9621cc
Fix build 2018-03-22 13:46:19 +01:00
Eelco Dolstra
dce8af59a6
Tweak 2018-03-22 13:32:18 +01:00
Eelco Dolstra
f87e286e82
Merge branch 'fix/avoid-large-stack-buffers' of https://github.com/dtzWill/nix 2018-03-22 13:19:25 +01:00
Eelco Dolstra
47f7e5585b Make 'nix copy --from ssh://...' run in constant memory
For instance, this reduced the memory consumption of

  $ nix copy --from ssh://localhost --to ~/my-nix /nix/store/1n7x0yv8vq6zi90hfmian84vdhd04bgp-blender-2.79a

from 632 MiB to 16 MiB.
2018-03-21 23:10:40 +01:00
Eelco Dolstra
1aca195e52
Hack to get SSH error messages from build-remote
E.g.

  cannot build on 'ssh://mac1': cannot connect to 'mac1': bash: nix-store: command not found
  cannot build on 'ssh://mac2': cannot connect to 'mac2': Host key verification failed.
  cannot build on 'ssh://mac3': cannot connect to 'mac3': Received disconnect from 213... port 6001:2: Too many authentication failures
  Authentication failed.
2018-03-20 15:25:26 +01:00
Will Dietz
f5a2136a40 serialise.cc: remove pessimising move
from clang6:

src/libutil/serialise.cc:189:23: warning: moving a temporary object prevents copy elision [-Wpessimizing-move]
2018-03-19 19:18:47 -05:00
Will Dietz
9b33201e72 util: rename stdout/stdin members to avoid conflicts w/standard macro
(cherry picked from commit c389a7fb617ed7bcd617efa68c6a48c00405310d)
2018-03-19 11:18:56 -05:00
Eelco Dolstra
48662d151b
Reduce substitution memory consumption
copyStorePath() now pipes the output of srcStore->narFromPath()
directly into dstStore->addToStore(). The sink used by the former is
converted into a source usable by the latter using
boost::coroutine2. This is based on [1].

This reduces the maximum resident size of

  $ nix build --store ~/my-nix/ /nix/store/b0zlxla7dmy1iwc3g459rjznx59797xy-binutils-2.28.1 --substituters file:///tmp/binary-cache-xz/ --no-require-sigs

from 418592 KiB to 53416 KiB. (The previous commit also reduced the
runtime from ~4.2s to ~3.4s, not sure why.) A further improvement will
be to download files into a Sink.

[1] https://github.com/NixOS/nix/compare/master...Mathnerd314:dump-fix-coroutine#diff-dcbcac55a634031f9cc73707da6e4b18

Issue #1969.
2018-03-16 20:35:59 +01:00
Eelco Dolstra
3e6b194d78
decompress(): Use a Source and Sink
This allows decompression to happen in O(1) memory.
2018-03-16 20:35:59 +01:00
Eelco Dolstra
4868721506
Filter ANSI colors when not writing to a terminal
Fixes https://github.com/NixOS/nixpkgs/issues/37114.
2018-03-15 16:08:07 +01:00
Eelco Dolstra
7b8914825a
Merge pull request #1965 from masaeedu/master
Wrap thread local in function for Cygwin
2018-03-15 14:56:21 +01:00
Shea Levy
cfdbfa6b2c
Catch more possible instances of passing NULL to memcpy.
Actually fixes #1976.
2018-03-14 23:53:43 -04:00
Asad Saeeduddin
be54f4a0b6 Wrap thread local in function for Cygwin
Fixes #1826. See #1352 for a previous instance of a similar change.
2018-03-12 00:56:41 -04:00
Eelco Dolstra
24b739817f
Fix double free in Store::queryPathInfo()
It was holding on to a Value* (i.e. a std::shared_ptr<ValidPathInfo>*)
outside of the pathInfoCache lock, so the std::shared_ptr could be
destroyed between the release of the lock and the decrement of the
std::shared_ptr refcount. This can happen if more than
'path-info-cache-size' paths are added in the meantime, *or* if
clearPathInfoCache() is called. The hydra-queue-runner queue monitor
thread periodically calls the later, so is likely to trigger a crash.

Fixes https://github.com/NixOS/hydra/issues/542.
2018-03-09 14:19:51 +01:00
Will Dietz
e917c05628 logging: Don't throw from Activity destructor
Move definition of destructor to logging.cc for
access to util.hh's ignoreException.
2018-03-06 10:42:58 -06:00
Will Dietz
c89a3d5368 don't allocate large buffers on the stack 2018-03-02 10:52:04 -06:00
Shea Levy
aa8bbbf69d
Merge branch 'write-failure-fixes' of git://github.com/lheckemann/nix 2018-03-02 10:59:59 -05:00
Eelco Dolstra
24ec750003
nix run: Fix segfault on macOS
Note that clearenv() is not available on macOS.

Fixes #1907.
2018-02-26 18:29:40 +01:00
Tuomas Tynkkynen
546f98dace libutil: Fix invalid assert on decoding base64 hashes
The assertion is broken because there is no one-to-one mapping from
length of a base64 string to the length of the output.

E.g.

"1q69lz7Empb06nzfkj651413n9icx0njmyr3xzq1j9q=" results in a 32-byte output.
"1q69lz7Empb06nzfkj651413n9icx0njmyr3xzq1j9qy" results in a 33-byte output.

To reproduce, evaluate:

builtins.derivationStrict {
    name = "0";
    builder = "0";
    system = "0";
    outputHashAlgo = "sha256";
    outputHash = "1q69lz7Empb06nzfkj651413n9icx0njmyr3xzq1j9qy";
}

Found by afl-fuzz.
2018-02-19 23:20:26 +02:00
Tuomas Tynkkynen
1d0e42879f libutil: Fix infinite loop in filterANSIEscapes on '\r'
E.g. nix-instantiate --eval -E 'abort "\r"' hangs.

Found by afl-fuzz.
2018-02-19 23:20:26 +02:00
Eelco Dolstra
ed73d40c3b
Config::handleUnknownSettings(): Remove unused 'fatal' argument 2018-02-19 14:00:34 +01:00
Eelco Dolstra
75a1d96cfd
Merge branch 'register-settings' of https://github.com/shlevy/nix 2018-02-19 13:58:34 +01:00
Eelco Dolstra
7fe5910bf8
Merge pull request #1857 from dtzWill/fix/check-for-lzma-mt
configure.ac: check if lzma has MT support, fix deb build/etc.
2018-02-19 13:06:14 +01:00
Shea Levy
ac973a6d3c
monitor-fds: Fix on macOS.
Fixes #1871.
2018-02-14 18:26:37 -05:00
Shea Levy
de4934ab3b
Allow plugins to define new settings. 2018-02-13 14:43:32 -05:00
Linus Heckemann
78ac3eb4eb Fix #1762
nix-store --export, nix-store --dump, and nix dump-path would previously
fail silently if writing the data out failed, because
 a) FdSink::write ignored exceptions, and
 b) the commands relied on FdSink's destructor, which ignores
    exceptions, to flush the data out.

This could cause rather opaque issues with installing nixos, because
nix-store --export would happily proceed even if it couldn't write its
data out (e.g. if nix-store --import on the other side of the pipe
failed).

This commit adds tests that expose these issues in the nix-store
commands, and fixes them for all three.
2018-02-13 14:26:19 +00:00
Shea Levy
6eb1040e90
Allow includes from nix.conf 2018-02-13 08:16:32 -05:00
Will Dietz
c6209030c4 compression: make parallel sink separate class 2018-02-11 13:50:28 -06:00
Will Dietz
a0bdc96726 compression: print warning if parallel requested but not supported 2018-02-11 13:03:47 -06:00
Will Dietz
5a082ad15a configure.ac: check if lzma has MT support, fix deb build/etc. 2018-02-09 21:02:25 -06:00
AmineChikhaoui
55ecdfe2a8
make multi threaded compression configurable and use single threaded
by default.
2018-02-07 17:54:08 +01:00
AmineChikhaoui
163e39547a
Merge branch 'master' of https://github.com/NixOS/nix into parallel-xz 2018-02-07 17:53:50 +01:00
Eelco Dolstra
84989d3af2
Improve filtering of ANSI escape sequences in build logs
All ANSI sequences except color setting are now filtered out. In
particular, terminal resets (such as from NixOS VM tests) are filtered
out.

Also, fix the completely broken tab character handling.
2018-02-07 15:23:57 +01:00
AmineChikhaoui
9d1e22f743
set block size to 0 to let the lzma lib choose the right one, add
some comments about possible improvements wrt memory usage/threading.
2018-02-07 11:18:55 +01:00
AmineChikhaoui
bc7e3a4dd6
support multi threaded xz encoder, this might be particularly useful in
the case of hydra where the overhead of single threaded encoding is more
noticeable e.g most of the time spent in "Sending inputs"/"Receiving outputs"
is due to compression while the actual upload to the binary cache seems
to be negligible.
2018-02-06 22:42:02 +01:00
Eelco Dolstra
d4dcffd643
Add pure evaluation mode
In this mode, the following restrictions apply:

* The builtins currentTime, currentSystem and storePath throw an
  error.

* $NIX_PATH and -I are ignored.

* fetchGit and fetchMercurial require a revision hash.

* fetchurl and fetchTarball require a sha256 attribute.

* No file system access is allowed outside of the paths returned by
  fetch{Git,Mercurial,url,Tarball}. Thus 'nix build -f ./foo.nix' is
  not allowed.

Thus, the evaluation result is completely reproducible from the
command line arguments. E.g.

  nix build --pure-eval '(
    let
      nix = fetchGit { url = https://github.com/NixOS/nixpkgs.git; rev = "9c927de4b179a6dd210dd88d34bda8af4b575680"; };
      nixpkgs = fetchGit { url = https://github.com/NixOS/nixpkgs.git; ref = "release-17.09"; rev = "66b4de79e3841530e6d9c6baf98702aa1f7124e4"; };
    in (import (nix + "/release.nix") { inherit nix nixpkgs; }).build.x86_64-linux
  )'

The goal is to enable completely reproducible and traceable
evaluation. For example, a NixOS configuration could be fully
described by a single Git commit hash. 'nixos-rebuild' would do
something like

  nix build --pure-eval '(
    (import (fetchGit { url = file:///my-nixos-config; rev = "..."; })).system
  ')

where the Git repository /my-nixos-config would use further fetchGit
calls or Git externals to fetch Nixpkgs and whatever other
dependencies it has. Either way, the commit hash would uniquely
identify the NixOS configuration and allow it to reproduced.
2018-01-16 19:23:18 +01:00
Will Dietz
9dd2b8ac7b use libbrotli directly when available
* Look for both 'brotli' and 'bro' as external command,
  since upstream has renamed it in newer versions.
  If neither are found, current runtime behavior
  is preserved: try to find 'bro' on PATH.
* Limit amount handed to BrotliEncoderCompressStream
  to ensure interrupts are processed in a timely manner.
  Testing shows negligible performance impact.
  (Other compression sinks don't seem to require this)
2017-12-30 20:26:33 -06:00
Will Dietz
e4d43f1f94 logging.cc: add missing 'override' 2017-12-11 12:08:40 -06:00
Eelco Dolstra
6cdaa858d0
Propagate flags like --sandbox to the daemon properly 2017-11-21 18:50:56 +01:00
Eelco Dolstra
d0b88db441
Cleanup 2017-11-20 17:32:58 +01:00