Commit graph

4011 commits

Author SHA1 Message Date
Gregory Hale
faa5607f54 Merge remote-tracking branch 'origin/master' into github-api-token 2020-09-25 12:10:58 -04:00
Eelco Dolstra
7d81582488
Merge pull request #3953 from obsidiansystems/basic-derivation-goal-outputs
Deduplicate basic derivation goals too
2020-09-25 17:21:19 +02:00
Kevin Quick
5a35cc29bf
Re-add support for github-access-token, but mark as deprecated. 2020-09-25 08:09:56 -07:00
Kevin Quick
ef2a14be19
Fix reference to older name for access-tokens config value. 2020-09-25 08:08:27 -07:00
Kevin Quick
8fba2a8b54
Update to use access-tokens configuration for github/gitlab access.
This change provides support for using access tokens with other
instances of GitHub and GitLab beyond just github.com and
gitlab.com (especially company-specific or foundation-specific
instances).

This change also provides the ability to specify the type of access
token being used, where different types may have different handling,
based on the forge type.
2020-09-24 22:49:44 -07:00
Alexander Bantyev
ed218e1d6c
Fix max-jobs option
After 0ed946aa61, max-jobs setting (-j/--max-jobs)
stopped working.

The reason was that nrLocalBuilds (which compared to maxBuildJobs to figure
out whether the limit is reached or not) is not incremented yet when tryBuild
is started; So, the solution is to move the check to tryLocalBuild.

Closes https://github.com/nixos/nix/issues/3763
2020-09-25 00:07:42 +03:00
John Ericson
45ca7c3e4b Merge remote-tracking branch 'upstream/master' into path-info 2020-09-23 22:35:41 +00:00
Eelco Dolstra
4ce8a3ed45 Hopefully fix EPERM on macOS 2020-09-23 21:29:10 +02:00
Eelco Dolstra
9a24ece122 Fix exception 2020-09-23 20:21:08 +02:00
Eelco Dolstra
236d9ee7f7 lstat() cleanup 2020-09-23 19:17:28 +02:00
Eelco Dolstra
688bd4fb50 After rewriting a path, make it read-only 2020-09-23 19:10:16 +02:00
Eelco Dolstra
cec9473871 DerivationGoal::registerOutputs(): Don't canonicalize twice
Fixes #4021.
2020-09-23 18:21:22 +02:00
Eelco Dolstra
d4f8163d10 canonicalisePathMetaData_(): Change assertion to error message 2020-09-23 18:21:22 +02:00
Eelco Dolstra
e8f0b1e996 DerivationGoal::registerOutputs(): Fix bad format string 2020-09-23 18:21:22 +02:00
John Ericson
3f226f71c1 Return more info from BinaryCacheStore::addToStoreCommon
We don't need it yet, but we could/should in the future, and it's a
cost-free change since we already have the reference. I like it.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2020-09-23 14:40:41 +00:00
John Ericson
412b3a54fb
Clarify FIXME in BinaryCacheStore::addToStoreCommon
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2020-09-23 10:36:55 -04:00
John Ericson
9fbc31a65b Get rid of Hash::dummy from BinaryCacheStore 2020-09-23 04:56:04 +00:00
John Ericson
993229cdaf Deduplicate basic derivation goals too
See comments for security concerns.

Also optimize goal creation by not traversing map twice.
2020-09-22 17:13:59 +00:00
John Ericson
e9fc2031f0 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-22 14:18:31 +00:00
regnat
67cc356bb3 Merge remote-tracking branch 'origin/master' into non-ca-depending-on-ca 2020-09-22 16:11:25 +02:00
Eelco Dolstra
980edd1f3a RemoteStore::addCAToStore(): Don't hold connection while calling queryPathInfo()
This leads to a deadlock if we're at the connection limit.
2020-09-22 15:28:20 +02:00
Eelco Dolstra
5b107f2c5f
Merge pull request #4038 from maljub01/master
Add a nix.conf option for allowing a symlinked store
2020-09-22 13:19:22 +02:00
Eelco Dolstra
35a0ac1838 Style fixes 2020-09-22 11:40:19 +02:00
Eelco Dolstra
92ac8df0ec Merge branch 'add-ca-to-store' of https://github.com/hercules-ci/nix 2020-09-22 11:31:33 +02:00
John Ericson
3786a801c3 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-22 04:15:55 +00:00
John Ericson
b92d3b2edd Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-22 00:45:55 +00:00
Cole Helbling
ba37299a03
Serialize SandboxMode enum to string for JSON
Rather than showing an integer as the default, instead show the boolean
referenced in the description.

The nix.conf.5 manpage used to show "default: 0", which is unnecessarily
opaque and confusing (doesn't 0 mean false, even though the default is
true?); now it properly shows that the default is true.
2020-09-21 10:36:45 -07:00
Eelco Dolstra
0716adaa8b abstractsettingtojson.hh -> abstract-setting-to-json.hh 2020-09-21 18:49:43 +02:00
Eelco Dolstra
340ca382c4 Don't include nlohmann/json.hpp in globals.hh
This reduces compilation time by 207s.

Issue #4045.
2020-09-21 18:47:18 +02:00
Eelco Dolstra
d51ba43047 Move Callback into its own header
This gets rid of the inclusion of <future> in util.hh, cutting
compilation time by ~20s (CPU time).

Issue #4045.
2020-09-21 18:42:21 +02:00
Marwan Aljubeh
f80ffeb8c9 Update the variable name accordingly 2020-09-21 17:29:08 +01:00
Eelco Dolstra
e8e1d420f3 Don't include <regex> in header files
This reduces compilation time by ~15 seconds (CPU time).

Issue #4045.
2020-09-21 18:22:45 +02:00
Marwan Aljubeh
4e1a04733d Use a better name for the config option 2020-09-21 16:32:22 +01:00
Robert Hensing
ca30abb3fb Document addCAToStore/addToStoreFromDump source drainage
Also checked that all usages satisfy the requirement and
removed dead code.
2020-09-21 07:55:47 +02:00
Robert Hensing
fa08db5c4c wopAddToStore: return ValidPathInfo
A ValidPathInfo is created anyway. By returning it we can save a
roundtrip and we have a nicer interface.
2020-09-21 07:55:47 +02:00
Robert Hensing
7c68264085 wopAddToStore: add RepairFlag 2020-09-21 07:55:47 +02:00
Robert Hensing
fbf509c113 parseContentAddressMethodPrefix: use string_view
Co-authored-by: John Ericson <git@JohnEricson.me>
2020-09-21 07:55:47 +02:00
Robert Hensing
8279178b07 Move FramedSink next to FramedSource 2020-09-21 07:55:47 +02:00
Robert Hensing
ecc8088cb7 wopAddToStore: Throw to clarify unused refs
Co-authored-by: John Ericson <git@JohnEricson.me>
2020-09-21 07:55:47 +02:00
Robert Hensing
c602ebfb34 Refactor wopAddToStore to make wopAddTextToStore obsolete 2020-09-21 07:55:45 +02:00
Robert Hensing
e34fe47d0c Overhaul wopAddToStore 2020-09-21 07:54:05 +02:00
Marwan Aljubeh
e40772cd35 Lint issue: replacing tabs with spaces 2020-09-18 17:18:45 +01:00
Marwan Aljubeh
c00e078343 Add a nix.conf option for allowing a symlinked store 2020-09-18 17:10:39 +01:00
Robert Hensing
14b30b3f3d Move FramedSource and FramedSink, extract withFramedSink 2020-09-17 20:21:04 +02:00
Robert Hensing
dfa547c6a8 Add ContentAddressMethod and parse/render it 2020-09-17 20:21:04 +02:00
Robert Hensing
9ee3122ec7 Remove redundant import 2020-09-17 20:21:04 +02:00
John Ericson
b7df353f27 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-17 16:33:10 +00:00
Eelco Dolstra
649d3aaf24
Merge pull request #3829 from obsidiansystems/remove-storetype-delegate-regStore
Remove storetype delegate reg store -- contains #3736
2020-09-17 13:55:01 +02:00
Eelco Dolstra
b94a35ef40
Merge pull request #4027 from tweag/fix-gc-of-ca-derivations
Fix garbage collection of CA derivations
2020-09-17 13:46:26 +02:00
regnat
520895b1da Fix garbage collection of CA derivations
Fix #4026
2020-09-17 13:36:58 +02:00
John Ericson
f60b380a7f Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore 2020-09-16 22:35:24 +00:00
John Ericson
2741fffa35 Ensure resolved CA derivations are written
so we can link outputs to deriver and thus properly cache.
2020-09-16 17:57:32 +00:00
John Ericson
7fdbb377ba Start to fix floating CA + remote building 2020-09-16 17:50:48 +00:00
John Ericson
c5ccebae00 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-16 17:50:40 +00:00
Greg Hale
a303c0b6dc Fetch commits from github/gitlab using Auth header
`nix flake info` calls the github 'commits' API, which requires
authorization when the repository is private. Currently this request
fails with a 404.

This commit adds an authorization header when calling the 'commits' API.
It also changes the way that the 'tarball' API authenticates, moving the
user's token from a query parameter into the Authorization header.

The query parameter method is recently deprecated and will be disallowed
in November 2020. Using them today triggers a warning email.
2020-09-16 13:46:48 -04:00
Théophane Hufschmitt
77a0e2c5be
Remove useless exception copy
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-09-16 14:00:21 +02:00
regnat
d72927aa7a Fix the s3 store
Add some necessary casts in the initialisation of the store's config
2020-09-16 13:53:09 +02:00
regnat
e0817cbcdc Don't include nlohmann/json.hpp in config.hh
Instead make a separate header with the template implementation of
`BaseSetting<T>::toJSONObj` that can be included where needed
2020-09-16 13:53:09 +02:00
regnat
fc2d31c423 Add (StoreConfig*) casts to work around a GCC bug
Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80431 that was
already there in the code but was accidentally removed in the last
commits
2020-09-16 13:53:09 +02:00
regnat
888f7afe9f Fix build issues with gcc 2020-09-16 13:53:09 +02:00
regnat
b73adacc1e Add a name to the stores
So that it can be printed by `nix describe-stores`
2020-09-16 13:53:09 +02:00
regnat
f24f0888f9 Document the new store hierarchy 2020-09-16 13:53:09 +02:00
regnat
29a632386e fixup! Make the store plugins more introspectable 2020-09-16 13:53:09 +02:00
regnat
1129913c4e fixup! Correctly call all the parent contructors of the stores 2020-09-16 13:53:09 +02:00
regnat
7f103dcddd Properly filter the stores according to their declared uriSchemes
When opening a store, only try the stores whose `uriSchemes()` include
the current one
2020-09-16 13:53:09 +02:00
regnat
5895184df4 Correctly call all the parent contructors of the stores
Using virtual inheritance means that only the default constructors of
the parent classes will be called, which isn't what we want
2020-09-16 13:53:09 +02:00
regnat
22afa8fb4d Separate store configs from the implems
Rework the `Store` hierarchy so that there's now one hierarchy for the
store configs and one for the implementations (where each implementation
extends the corresponding config). So a class hierarchy like

```
StoreConfig-------->Store
    |                 |
    v                 v
SubStoreConfig----->SubStore
    |                 |
    v                 v
SubSubStoreConfig-->SubSubStore
```

(with virtual inheritance to prevent DDD).

The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-16 13:53:08 +02:00
regnat
aa4eac3788 fixup! Separate the instantiation and initialisation of the stores 2020-09-16 13:53:08 +02:00
regnat
3c525d1590 Complete the toJSON instance for Setting<T>
Don't let it just contain the value, but also the other fields of the
setting (description, aliases, etc..)
2020-09-16 13:53:08 +02:00
regnat
3b57181f8e Separate the instantiation and initialisation of the stores
Add a new `init()` method to the `Store` class that is supposed to
handle all the effectful initialisation needed to set-up the store.
The constructor should remain side-effect free and just initialize the
c++ data structure.

The goal behind that is that we can create “dummy” instances of each
store to query static properties about it (the parameters it accepts for
example)
2020-09-16 13:53:08 +02:00
regnat
fa32560169 Fix the registration of stores 2020-09-16 13:53:08 +02:00
regnat
7d5bdf8b56 Make the store plugins more introspectable
Directly register the store classes rather than a function to build an
instance of them.
This gives the possibility to introspect static members of the class or
choose different ways of instantiating them.
2020-09-16 13:53:08 +02:00
John Ericson
3a5cdd737c Rename Derivation::pathOpt to Derivation::path
We no longer need the `*Opt` to disambiguate.
2020-09-15 15:21:39 +00:00
John Ericson
6387550d58 Get rid of confusing std::optional<bool> for validity 2020-09-15 15:19:45 +00:00
John Ericson
c4bf219b55 Don't link deriver until after any delayed exception is thrown
Otherwise, we will associate fixed-output derivations with outputs that
they did indeed produce, but which had the wrong hash. That's no good.
2020-09-15 14:28:06 +00:00
John Ericson
3ba552b245 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-15 14:17:06 +00:00
John Ericson
c08c9f08c7 Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore 2020-09-15 14:08:35 +00:00
regnat
057c6203b5 gracefully handle old daemon versions
Add a fallback path in `queryPartialDerivationOutputMap` for daemons
that don't support it.

Also upstreams a couple methods from `SSHStore` to `RemoteStore` as this
is needed to handle the fallback path.
2020-09-15 09:58:01 +02:00
Eelco Dolstra
a59e77d9e5 nix-daemon: Lower verbosity of restricted setting warning
Fixes #3992.
2020-09-14 13:48:51 +02:00
aszlig
525b38eee8
Fix unspecified behaviour in readStorePathCAMap
When deploying a Hydra instance with current Nix master, most builds
would not run because of errors like this:

  queue monitor: error: --- Error --- hydra-queue-runner
  error: --- UsageError --- nix-daemon
  not a content address because it is not in the form '<prefix>:<rest>': /nix/store/...-somedrv

The last error message is from parseContentAddress, which expects a
colon-separated string, however what we got here is a store path.

Looking at the worker protocol, the following message sent to the Nix
daemon caused the error above:

  0x1E -> wopQuerySubstitutablePathInfos
  0x01 -> Number of paths
  0x16 -> Length of string
  "/nix/store/...-somedrv"
  0x00 -> Length of string
  ""

Looking at writeStorePathCAMap, the store path is indeed the first field
that's transmitted. However, readStorePathCAMap expects it to be the
*second* field *on my machine*, since expression evaluation order is a
classic form of unspecified behaviour[1] in C++.

This has been introduced in https://github.com/NixOS/nix/pull/3689,
specifically in commit 66a62b3189.

[1]: https://en.wikipedia.org/wiki/Unspecified_behavior#Order_of_evaluation_of_subexpressions

Signed-off-by: aszlig <aszlig@nix.build>
2020-09-13 02:40:51 +02:00
John Ericson
98dfd7531d Fix querying outputs for CA derivations some more
If we resolve using the known path of a derivation whose output we
didn't have, we previously blew up. Now we just fail gracefully,
returning the map of all outputs unknown.
2020-09-04 18:33:58 +00:00
John Ericson
075d399e3f Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-09-04 16:04:35 +00:00
John Ericson
e9fad3006b Fix some of the issues raised by @edolstra
- More and better comments

 - The easier renames
2020-09-04 15:18:25 +00:00
John Ericson
e86dd59dcc
Apply suggestions from code review
Thanks!

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-09-04 10:48:50 -04:00
John Ericson
c9f1ed912c Don't chmod symlink before moving outputs around
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-09-04 14:41:53 +00:00
John Ericson
aad4abcc9c Fix floating CA tests
We will sometimes try to query the outputs of derivations we can't
resolve. That's fine; it just means we don't know what those outputs are
yet.
2020-09-04 01:17:38 +00:00
John Ericson
975a47f7fe Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-09-03 22:09:04 +00:00
John Ericson
e7d93e7ece Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-03 15:43:17 +00:00
Eelco Dolstra
8a945d6ddb
Remove gc-check-reachability 2020-09-03 11:30:15 +02:00
Eelco Dolstra
82b77a7726
addPermRoot(): Remove indirect flag 2020-09-03 11:29:34 +02:00
Eelco Dolstra
b07167be5a
createGeneration(): Always create an indirect root
This means profiles outside of /nix/var/nix/profiles don't get
garbage-collected. It also means we don't need to scan
/nix/var/nix/profiles for GC roots anymore, except for compatibility
with previously existing generations.
2020-09-03 11:13:53 +02:00
Eelco Dolstra
b74f5cdd23
createGeneration(): Take a StorePath 2020-09-03 11:06:56 +02:00
Domen Kožar
dd4b56c87f
Allow HTTP binary cache to request absolute uris 2020-09-01 21:35:48 +02:00
John Ericson
ef278d00f9 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-01 18:01:48 +00:00
Eelco Dolstra
84f5cabbea Merge remote-tracking branch 'origin/master' into markdown 2020-08-31 14:24:26 +02:00
John Ericson
02e0001fc0 Merge branch 'ca-floating-upstream' of github.com:obsidiansystems/nix into ca-floating-upstream 2020-08-28 21:23:09 +00:00
John Ericson
421ed527c7
Update src/libstore/build.cc
Thanks for catching, @regnat.
2020-08-28 17:22:57 -04:00
John Ericson
48e8828050 Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-08-28 20:38:25 +00:00
John Ericson
8017fe7487 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-28 19:59:14 +00:00
Eelco Dolstra
eb75282b8d
Merge pull request #3434 from Ericson2314/derivation-header-include-order
Revise division of labor in deserialization of derivations
2020-08-27 16:39:28 +02:00
Eelco Dolstra
a0f19d9f3a RemoteStore::addToStore(): Fix race between stderrThread and NAR writer
As pointed out by @B4dM4n, the call to to.flush() on stderrThread is
unsafe because the NAR writer thread is also writing to 'to'.

Fixes #3943.
2020-08-27 14:50:51 +02:00
Eelco Dolstra
4bf5faf416 Merge remote-tracking branch 'origin/master' into markdown 2020-08-25 19:47:34 +02:00
John Ericson
8eb73a8724 CA derivations that depend on other CA derivations
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-08-24 19:01:15 +00:00
John Ericson
e0b0e18905 Add constructor for BasicDerivation -> Derivation 2020-08-24 19:01:07 +00:00
John Ericson
59979e7053 Fix bad debug format string 2020-08-24 19:01:07 +00:00
Eelco Dolstra
6a67e57019
Add DummyStore (dummy://)
DummyStore does not allow building or adding paths. This is useful for
evaluation tests when you don't want to initialize a "proper" store.
2020-08-24 18:54:16 +02:00
Eelco Dolstra
2530cc6827
Merge pull request #3950 from obsidiansystems/tabs-spaces
tabs -> spaces
2020-08-24 10:20:28 +02:00
John Ericson
b0b59fd05a Merge remote-tracking branch 'obsidian/write-derivation-borrow' into HEAD 2020-08-23 15:11:10 +00:00
John Ericson
35e6288be1 writeDerivation just needs a plain store reference 2020-08-23 15:01:11 +00:00
John Ericson
3a7b330b64 "Downstream placeholders" should not be store paths
Insead they should be opaque `/<hash>` like the placeholders we already
have.
2020-08-21 19:35:35 +00:00
John Ericson
422affe102 tabs -> spaces
Sorry I let the tab sneak in there in the first place.
2020-08-21 19:19:24 +00:00
John Ericson
c265e0ea82 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-08-20 20:24:22 +00:00
John Ericson
27a3f82c0b Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-20 18:28:17 +00:00
John Ericson
45a2f1baab Rename drv output querying functions, like master
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
 - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap
2020-08-20 18:14:12 +00:00
Eelco Dolstra
4d77513d97
Merge pull request #3859 from obsidiansystems/drv-outputs-map-allow-missing
`queryDerivationOutputMap` no longer assumes all outputs have a mapping
2020-08-20 16:49:23 +02:00
John Ericson
9a9d834dc7 Rename drv output querying functions
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
 - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap`
2020-08-20 14:12:51 +00:00
Eelco Dolstra
acb99f03f9
Config: Use nlohmann/json 2020-08-20 11:02:16 +02:00
John Ericson
a83694c7a1 Use RemoteStore to open connection for proxying daemon
Removes duplicate websocket opening code, and also means we should be
able to to ssh-ssh-... daemon relays, not just uds-uds-... ones.
2020-08-19 19:34:47 +00:00
Eelco Dolstra
c8fa39324a
Generate the nix.conf docs from the source code
This means we don't have two (divergent) sets of option descriptions
anymore.
2020-08-19 18:28:04 +02:00
John Ericson
d5af5763cf Merge branch 'master' of github.com:NixOS/nix into remove-storetype-delegate-regStore 2020-08-19 15:54:50 +00:00
John Ericson
be0d429b95 Merge branch 'master' of github.com:NixOS/nix into templated-daemon-protocol 2020-08-19 03:17:41 +00:00
John Ericson
950ddfdb82 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-08-18 14:36:44 +00:00
Eelco Dolstra
dfeb76dbf9
Merge pull request #3930 from obsidiansystems/legacy-ssh-build-paths
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths`
2020-08-18 16:07:40 +02:00
Eelco Dolstra
7cdc739ece
Merge remote-tracking branch 'origin/master' into markdown 2020-08-17 13:43:39 +02:00
Eelco Dolstra
847a5392f4 Merge branch 'ca-no-need-trust' of https://github.com/obsidiansystems/nix 2020-08-17 10:57:02 +02:00
Eelco Dolstra
3c619f6290 Merge branch 'test-RemoteStore-buildDerivation' of https://github.com/obsidiansystems/nix 2020-08-17 10:53:18 +02:00
John Ericson
6f7ac5e865 Remove extra closing paren 2020-08-14 21:59:31 +00:00
John Ericson
3c8b5b6219 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-14 17:00:13 +00:00
John Ericson
4b571ea321
Update src/libstore/daemon.cc
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-08-14 11:52:37 -04:00
Eelco Dolstra
13e49be660
Merge pull request #3875 from obsidiansystems/new-interface-for-path-pathOpt
Offer a safer interface for path and pathOpt
2020-08-14 17:19:19 +02:00
Eelco Dolstra
7714d9a943
Merge pull request #3924 from obsidiansystems/features-per-store
Make `system-features` a store setting
2020-08-14 17:13:07 +02:00
Eelco Dolstra
9b9d5297ba
Merge pull request #3909 from matthewbauer/readd-hashed-mirrors
Add hashed-mirrors back
2020-08-14 17:05:47 +02:00
Eelco Dolstra
d81f13f7cb
Merge pull request #3899 from obsidiansystems/make-narHash-not-optional
Make narHash in ValidPathInfo not optional
2020-08-14 17:00:18 +02:00
John Ericson
e1308b1211 Define LegacySSHStore::buildPaths using cmdBuildPaths
Evidentally this was never implemented because Nix switched to using
`buildDerivation` exclusively before `build-remote.pl` was rewritten.

The `nix-copy-ssh` test (already) tests this.
2020-08-13 21:27:55 +00:00
John Ericson
5ccd94501d Allow trustless building of CA derivations
Include a long comment explaining the policy. Perhaps this can be moved
to the manual at some point in the future.

Also bump the daemon protocol minor version, so clients can tell whether
`wopBuildDerivation` supports trustless CA derivation building. I hope
to take advantage of this in a follow-up PR to support trustless remote
building with the minimal sending of derivation closures.
2020-08-13 18:15:57 +00:00
John Ericson
85aacbee64 Use TeeSink and TeeSouce in a few more places 2020-08-13 14:51:17 +00:00
John Ericson
d2f2be0f70 Test RemoteStore::buildDerivation
Fix `wopNarFromPath` which needed a `toRealPath`.
2020-08-13 04:07:14 +00:00
John Ericson
5d67f18c86 Merge branch 'daemon-auth-cleanup' of github.com:obsidiansystems/nix into HEAD 2020-08-12 18:22:31 +00:00
John Ericson
4720853129 Make system-features a store setting
This seems more correct. It also means one can specify the features a
store should support with --store and remote-store=..., which is useful.
I use this to clean up the build remotes test.
2020-08-12 18:13:00 +00:00
John Ericson
8d4162ff9e Separate auth and logic for the daemon
Before, processConnection wanted to know a user name and user id, and
`nix-daemon --stdio`, when it isn't proxying to an underlying daemon,
would just assume "root" and 0. But `nix-daemon --stdio` (no proxying)
shouldn't make guesses about who holds the other end of its standard
streams.

Now processConnection takes an "auth hook", so `nix-daemon` can provide
the appropriate policy and daemon.cc doesn't need to know or care what
it is.
2020-08-12 15:22:33 +00:00
John Ericson
5f80aea795 Break out lambda so output can be matched just once
This is much better.
2020-08-12 02:23:31 +00:00
John Ericson
18834f7764 Recheck path validity after acquiring lock
It might have changed, and in any event this is how the cod used to work
so let's just keep it.
2020-08-11 23:44:02 +00:00
John Ericson
2de201254e Don't assume a total output map in two places in build.cc
Thanks @regnat for catching one of them. The other follows for many of
the same reasons. I'm find fixing others on a need-to-fix basis,
provided their are no regressions.
2020-08-11 23:07:50 +00:00
John Ericson
4c6aac8fdf Clarify comment on sandbox and temp fresh paths 2020-08-11 22:46:05 +00:00
John Ericson
6d57139050 Clarify outputReferences variable with self-describing type
Thanks for the idea, @Regnat!
2020-08-11 22:34:09 +00:00
John Ericson
8a068bd025 Remove redundant equality check 2020-08-11 21:25:40 +00:00
John Ericson
07e3466eb4 Float comment to out describe gaveUpOnSubstitution in general 2020-08-11 21:16:14 +00:00
John Ericson
d0f6e338dd
Apply suggestions from code review
Thanks!!

Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-08-11 16:49:10 -04:00
John Ericson
2a0902634e Fix error in merge breaking floating CA drvs
Forgot to add this hunk!
2020-08-11 00:13:19 +00:00
John Ericson
1b5c24662b Merge branch 'small-drv-serialize-cleanup' of github.com:obsidiansystems/nix into single-ca-drv-build 2020-08-10 01:57:54 +00:00
John Ericson
581183d4d5 Deduplicate parsing and reading derivations 2020-08-10 01:40:50 +00:00
John Ericson
bcd0629c2e Remove name parameter from writeDerivation
The name is now stored with the derivation itself.
2020-08-10 01:35:59 +00:00
John Ericson
e913a2989f Squashed get CA derivations building 2020-08-07 19:51:55 +00:00
Eelco Dolstra
edfd676e05 Fix .ls file names in binary caches
These are not supposed to include the 'name' part of the store
path. This was broken by 759947bf72.
2020-08-07 21:18:29 +02:00
John Ericson
8f92bb5ad9 Merge branch 'drv-outputs-map-allow-missing' of github.com:obsidiansystems/nix into templated-daemon-protocol 2020-08-07 18:51:01 +00:00
John Ericson
f7ba16f9cb Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing 2020-08-07 17:46:39 +00:00
John Ericson
47644e49ca Specialize std::optional<StorePath> so this is backwards compatible
While I am cautious to break parametricity, I think it's OK in this
cases---we're not about to try to do some crazy polymorphic protocol
anytime soon.
2020-08-07 17:05:14 +00:00
Matthew Bauer
96c158d6e1 Fix build 2020-08-06 21:04:31 -05:00
Matthew Bauer
8abc577cc2 Merge remote-tracking branch 'origin/master' into readd-hashed-mirrors 2020-08-06 18:58:59 -05:00
Carlo Nucera
46f9dd56da Fix bug due to non-deterministic arg eval order 2020-08-06 19:30:05 -04:00
Matthew Bauer
641c950701 Add hashed-mirrors back
Some users have their own hashed-mirrors setup, that is used to mirror
things in addition to what’s available on tarballs.nixos.org. Although
this should be feasable to do with a Binary Cache, it’s not always
easy, since you have to remember what "name" each of the tarballs has.
Continuing to support hashed-mirrors is cheap, so it’s best to leave
support in Nix. Note that NIX_HASHED_MIRRORS is also supported in
Nixpkgs through fetchurl.nix.

Note that this excludes tarballs.nixos.org from the default, as in
\#3689. All of these are available on cache.nixos.org.
2020-08-06 18:19:09 -05:00
Carlo Nucera
9ab07e99f5 Use template structs instead of phantoms 2020-08-06 18:04:13 -04:00
Carlo Nucera
3d8240c32e Remove leftover commented code 2020-08-06 16:04:18 -04:00
Carlo Nucera
f795f0fabc Merge branch 'drv-outputs-map-allow-missing-namespace' of github.com:obsidiansystems/nix into templated-daemon-protocol 2020-08-06 15:53:09 -04:00
John Ericson
e89b5bd0bf Minimize the usage of Hash::dummy 2020-08-06 18:31:48 +00:00
John Ericson
5e59b25a23 Merge remote-tracking branch 'upstream/master' into make-narHash-not-optional 2020-08-06 15:34:15 +00:00
Carlo Nucera
8b175f58d1 Simplify the namespace 2020-08-05 17:57:07 -04:00
Carlo Nucera
0739d428e0 Solve template deduction problem
We had to predeclare our template functions
2020-08-05 17:49:45 -04:00
John Ericson
6c66331d5c WIP: Put the worker protocol read and write in a namespace to disambig 2020-08-05 20:37:48 +00:00
Carlo Nucera
1d2e80ddd6 Merge branch 'master' of github.com:NixOS/nix into new-interface-for-path-pathOpt 2020-08-05 15:45:33 -04:00
John Ericson
ed96e603e1 Proxy -> Phantom to match Rust
Sorry, Haskell.
2020-08-05 19:44:08 +00:00
Carlo Nucera
8241e660ba Remove Hash::operator bool ()
Since the hash is not optional anymore
2020-08-05 15:30:38 -04:00
Carlo Nucera
be6e1c6457 Merge branch 'master' of github.com:NixOS/nix into make-narHash-not-optional 2020-08-05 15:14:47 -04:00
Carlo Nucera
1ad6394b33 Add Hash::dummy to signal default value
We did this in the same spirit of the dummy value that's present in
libstore/path.hh
2020-08-05 15:11:49 -04:00
Carlo Nucera
1d71028f4d Remove optionality in ValidPathInfo::narInfo 2020-08-05 14:42:48 -04:00
John Ericson
7302761f64 Merge remote-tracking branch 'obsidian/drv-outputs-map-allow-missing' into templated-daemon-protocol 2020-08-05 17:53:24 +00:00
Ben Burdette
f1a47a96b6 error messages for issue 2238 2020-08-05 10:58:00 -06:00
John Ericson
cf939055c8 Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing 2020-08-05 16:43:30 +00:00
John Ericson
0559ff3d8b Merge branch 'master' of github.com:NixOS/nix into derivation-primop-floating-output 2020-08-05 14:56:40 +00:00
John Ericson
92ad550e96 Merge remote-tracking branch 'obsidian/misc-ca' into derivation-primop-floating-output 2020-08-05 14:51:41 +00:00
Eelco Dolstra
790b694be7 Style fix 2020-08-05 16:51:06 +02:00
Eelco Dolstra
e48f944e9d Merge branch 'misc-ca' of https://github.com/obsidiansystems/nix 2020-08-05 16:50:05 +02:00
John Ericson
b9ebe373bb Sed some names to perhaps avoid conflicts 2020-08-05 14:49:25 +00:00
Eelco Dolstra
25f7912156 Style fix 2020-08-05 16:47:48 +02:00
John Ericson
e561a13a58 Reanme DerivationType::Regular defintion too
This is the one non-prefixed occurence
2020-08-05 14:45:56 +00:00
John Ericson
e7b0847f2d Make names more consistent 2020-08-05 14:44:39 +00:00
John Ericson
839f0fe095 Merge remote-tracking branch 'upstream/master' into misc-ca 2020-08-05 14:40:01 +00:00
John Ericson
03f4fafc27 Merge remote-tracking branch 'upstream/master' into misc-ca 2020-08-05 14:36:25 +00:00
Eelco Dolstra
b91dc7ebad
Merge pull request #3730 from obsidiansystems/better-ca-parse-errors
Improve hash parsing and errors
2020-08-05 16:33:07 +02:00
Eelco Dolstra
75f220a595
Merge pull request #3864 from obsidiansystems/more-topo-sort
Abstract out topo sorting logic
2020-08-05 16:07:29 +02:00
John Ericson
e61061c88e Remove stray tabs 2020-08-04 23:17:11 +00:00
John Ericson
36dd7920a2 Merge remote-tracking branch 'upstream/master' into path-info 2020-08-04 23:16:18 +00:00
John Ericson
16c98bf57c Get rid of some unneeded temporaries 2020-08-04 22:36:31 +00:00
John Ericson
1dfcbebc95 Organize and format code a bit 2020-08-04 22:28:10 +00:00
John Ericson
45b6fdb22b Remove unused functions 2020-08-04 22:10:13 +00:00
John Ericson
1bab8a321f Remove unneeded definition
Template instantiations will cover this case fine.
2020-08-04 21:56:42 +00:00
John Ericson
2f2ae993dc WIP systematize more of the worker protocol
This refactor should *not* change the wire protocol.
2020-08-04 19:02:05 +00:00
John Ericson
fbeb8695fb Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing 2020-08-04 18:35:33 +00:00
John Ericson
d3452a5ed6 Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors 2020-08-04 16:13:20 +00:00
Eelco Dolstra
4e7f1c7f11 S3BinaryCacheStore: Fix size determination 2020-08-04 16:01:13 +02:00
Eelco Dolstra
dfe66420e7 Revert "Remove putBytes"
This reverts commit b8eea7e81a.
2020-08-04 15:56:10 +02:00
Eelco Dolstra
327b1bf378 BinaryCacheStore: Explicitly flush file sink
The file sink is also flushed in its destructor, but we ignore any
exceptions in the destructor.

Issue #3886.
2020-08-04 14:54:07 +02:00
Eelco Dolstra
ed52cf632b
Merge pull request #3892 from obsidiansystems/path-info-header
Move ValidPathInfo to its own header
2020-08-04 10:44:47 +02:00
John Ericson
062533f7cd Merge remote-tracking branch 'upstream/master' into path-info-header 2020-08-03 21:02:28 +00:00
Eelco Dolstra
24e07c428f Delete compressed NARs
Fixes #3891.
2020-08-03 18:34:52 +02:00
John Ericson
54281f3ac1 addToStore in terms of addToStoreFromDump is not local-store-specific 2020-08-03 04:13:45 +00:00
John Ericson
9357512d73 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-08-01 19:38:35 +00:00
John Ericson
0e9e599b00 Merge remote-tracking branch 'upstream/master' into path-info 2020-08-01 00:06:24 +00:00
Carlo Nucera
b6d97fdbf4 Merge branch 'master' of github.com:NixOS/nix into drv-outputs-map-allow-missing 2020-07-31 13:12:51 -04:00
Eelco Dolstra
bf290c2306
Merge remote-tracking branch 'origin/master' into markdown 2020-07-31 16:07:04 +02:00
Eelco Dolstra
1d0a7b54fa
Enable syntax highlighting 2020-07-31 15:43:25 +02:00
Carlo Nucera
eee6ef86cd Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors 2020-07-30 18:27:25 -04:00
Matthew Bauer
05ac4db39a Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-07-30 12:38:24 -05:00
Eelco Dolstra
3f6e88a552 unsigned long long -> uint64_t 2020-07-30 13:34:04 +02:00
Eelco Dolstra
ebee2b7852 receiveContents(): unsigned int -> size_t 2020-07-30 13:00:30 +02:00
Eelco Dolstra
f63839bfa4 Cleanup 2020-07-29 18:04:51 +02:00
Eelco Dolstra
4c0077a07d Fix RemoteStore::addToStore() latency
Since 6185d25e52, this was very
latency-bound since it required a round-trip for every 32 KiB. So for
example copying a 514 MiB closure over a virtual ethernet device with
a articial delay of just 1 ms took 343s. Now it takes 2.7s.

Fixes #3372.
2020-07-29 00:48:39 +02:00
Eelco Dolstra
c159f48a39 Cleanup 2020-07-29 00:24:55 +02:00
John Ericson
df707d05d1 Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca 2020-07-28 21:12:36 +00:00
Carlo Nucera
2980b244b7 Use assert(false) instead of abort() 2020-07-28 15:39:45 -04:00
Carlo Nucera
cfe96bd7ce Merge branch 'master' of github.com:NixOS/nix into ca-drv 2020-07-28 15:36:45 -04:00
Carlo Nucera
c318d398f3 Merge branch 'misc-ca' of github.com:obsidiansystems/nix into new-interface-for-path-pathOpt 2020-07-28 14:22:24 -04:00
Carlo Nucera
7ef1e3cd14 Use the new interface 2020-07-28 13:59:24 -04:00
Carlo Nucera
d564ac1c50 Offer a safer interface for pathOpt
The new interface we offer provides a way of getting all the
DerivationOutputs with the storePaths directly, based on the observation
that it's the most common usecase.
2020-07-28 13:55:15 -04:00
Eelco Dolstra
b8eea7e81a Remove putBytes
istream->tellg() returns -1 so we can't get the number of bytes
written.

Fixes 'uploaded 's3://nix-cache/nar/00819r9lp5kajr6baxfw5dhhc0cx8ndxaz43qmd2f0gn1hk1ynlp.nar.xz' (-1 bytes) in 11620 ms' messages.
2020-07-28 13:27:56 +02:00
John Ericson
8065c6d160 Abstract out topo sorting logic 2020-07-27 20:45:34 +00:00
John Ericson
951415b568 Require ca-derivations everywhere we create a CA derivation
"create" as in read one in from a serialized form, or build one from
scratch in memory.
2020-07-27 17:56:36 +00:00
John Ericson
e32a9e124b Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output 2020-07-27 17:50:06 +00:00
John Ericson
7cf978440c Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca 2020-07-27 17:47:40 +00:00
John Ericson
e6115ca004 Merge remote-tracking branch 'upstream/master' into ca-derivation-data-types 2020-07-27 17:41:43 +00:00
John Ericson
1d7d94ceea Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-07-27 16:17:50 +00:00
John Ericson
43f2bd8dc5 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-27 16:13:57 +00:00
John Ericson
387699fe7f Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca 2020-07-27 15:00:28 +00:00
John Ericson
78466bcb2f Merge branch 'optional-derivation-output-storepath' into ca-derivation-data-types 2020-07-27 14:40:08 +00:00
John Ericson
d5bb67cfa4 Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath 2020-07-27 14:29:32 +00:00
John Ericson
699fc89b39 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-27 14:22:45 +00:00
John Ericson
2c7557481b queryDerivationOutputMap no longer assumes all outputs have a mapping
This assumption is broken by CA derivations. Making a PR now to do the
breaking daemon change as soon as possible (if it is already too late,
we can bump protocol intead).
2020-07-24 21:14:06 +00:00
John Ericson
58e55c0923 Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output 2020-07-23 19:17:52 +00:00
John Ericson
a2af25c55a Merge remote-tracking branch 'upstream/master' into misc-ca 2020-07-23 19:14:17 +00:00
John Ericson
c36916eca0 Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output 2020-07-23 15:06:56 +00:00
John Ericson
6fcd9966bf Merge remote-tracking branch 'upstream/master' into misc-ca 2020-07-23 15:02:29 +00:00
John Ericson
66a2067288 Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors 2020-07-23 14:56:35 +00:00
John Ericson
2274f63453 Get rid of basicDerivation::findOutput
It's a tiny function which is:

 - hardly worth abstrating over, and also only used once.

 - doesn't work once we get CA drvs

I rewrote the one callsite to be forwards compatable with CA
derivations, and also potentially more performant: instead of reading in
the derivation it can ust consult the SQLite DB in the common case.
2020-07-23 14:39:02 +00:00
John Ericson
9423f64ee2 Parse CA derivations using new output variants
We no longer need `ParsedDerivation` because everything libstore needs
to know about is in the `BasicDerivation` proper.
2020-07-22 23:59:25 +00:00
John Ericson
c56356bacc Separate concerns in scanForReferences with TeeSink
This also will make it easier to use a `HashModuloSink` instead for CA
derivations.
2020-07-22 22:13:44 +00:00
Eelco Dolstra
ff314f186e
Merge pull request #3736 from obsidiansystems/allow-relative-paths-in-store-option
Allow relative paths in --store option
2020-07-21 19:43:07 +02:00
Eelco Dolstra
0951330680
Merge pull request #3714 from obsidiansystems/add-body-to-network-errors
Add response body to network errors
2020-07-21 18:09:42 +02:00
John Ericson
02639716ea Merge branch 'allow-relative-paths-in-store-option' into remove-storetype-delegate-regStore 2020-07-21 15:55:27 +00:00
John Ericson
6cce32c8e8 Change logic for deciding what is a relative path for the local store
The was Eelco's prefered logic, and it looks good to me!
2020-07-21 15:39:47 +00:00
Carlo Nucera
922a845ffc Update chunkSize to the suggested value
This was a suggested course of action in a review in one of our earlier
commits, https://github.com/NixOS/nix/pull/3801#discussion_r457557079
2020-07-21 10:24:19 -04:00
John Ericson
54e507a7aa Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors 2020-07-21 14:17:59 +00:00
John Ericson
5ce95b9529
Update src/libstore/build.cc 2020-07-21 09:47:40 -04:00
John Ericson
cdb3f39b64 Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors 2020-07-21 13:43:20 +00:00
John Ericson
5055c595bd Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into ca-derivation-data-types 2020-07-21 01:20:53 +00:00
John Ericson
5a37413796 Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into from-dump-stream 2020-07-21 01:20:14 +00:00
John Ericson
bf0dca7790 Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into misc-ca 2020-07-21 01:18:33 +00:00
Carlo Nucera
9aae179f34 Correct bug, thoroughly document addToStoreSlow 2020-07-20 20:18:12 -04:00
Carlo Nucera
0ca9744694 Use heuristics to decide when to show the response
Due to https://github.com/NixOS/nix/issues/3841 we don't know how print
different messages for different verbosity levels.
2020-07-20 15:57:58 -04:00
Carlo Nucera
6357b1b0fb Add another Unimplemented case 2020-07-20 14:17:25 -04:00
Carlo Nucera
007bf6d17a Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca 2020-07-20 14:14:39 -04:00
Carlo Nucera
362ae93851 Add UnimplementedError to ease grepping for these 2020-07-20 14:13:37 -04:00
John Ericson
bf61871271 parser.hh -> split.hh 2020-07-20 17:42:34 +00:00
John Ericson
eab945f591 Merge remote-tracking branch 'upstream/master' into from-dump-stream 2020-07-20 17:29:23 +00:00
Eelco Dolstra
1c5f8bbfb5
Merge pull request #3822 from obsidiansystems/dump-thrice-fixme
Optimize `addToStoreSlow` and remove `TeeParseSink`
2020-07-20 18:55:05 +02:00
John Ericson
ac2fc7ba1f
Apply suggestions from code review
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-07-20 11:29:46 -04:00
Carlo Nucera
0aa79dcc6f Remove StoreType abstraction and delegate regStore
to each Store implementation. The generic regStore implementation will
only be for the ambiguous shorthands, like "" and "auto".

This also could get us close to simplifying the daemon command.
2020-07-17 17:24:02 -04:00
John Ericson
6756cecfcf Add DerivationType::CAFloating 2020-07-17 19:55:41 +00:00
Carlo Nucera
fefd6c9e5f Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option 2020-07-17 14:34:20 -04:00
Carlo Nucera
bbc633c98c Revert "Don't anticipate CA but not fixed outputs for now"
This reverts commit 3a9e4c3262.
2020-07-17 13:10:32 -04:00
Carlo Nucera
205dcd140d Revert "Don't anticipate multiple CA outputs for now"
This reverts commit 74b251b2f3.
2020-07-17 12:43:46 -04:00
Carlo Nucera
1feb8981df Revert "Don't anticipate hash algo without hash in derivation for now"
This reverts commit 3804e3df9b.
2020-07-17 12:05:54 -04:00
Carlo Nucera
487c5751c6 Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca 2020-07-17 11:32:59 -04:00
Carlo Nucera
5cb840541b Merge branch 'multi-output-hashDerivationModulo' of github.com:Ericson2314/nix into misc-ca 2020-07-17 10:28:33 -04:00
Carlo Nucera
745a03cef5 Merge branch 'optional-derivation-output-storepath' of github.com:obsidiansystems/nix into ca-derivation-data-types 2020-07-16 13:36:01 -04:00
Carlo Nucera
048e916f64 Merge branch 'master' of github.com:NixOS/nix into optional-derivation-output-storepath 2020-07-16 13:32:28 -04:00
John Ericson
cc0d77f8c9 Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-07-16 17:28:52 +00:00
Carlo Nucera
e3a3406db8 Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors 2020-07-16 12:04:10 -04:00
John Ericson
5ea817dace Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-16 14:58:53 +00:00
John Ericson
3dcca18c30 Fix bug in TeeSource
We use this to simplify `LocalStore::addToStoreFromDump`.

Also, hope I fixed build error with old clang (used in Darwin CI).
2020-07-16 13:39:27 +00:00
Carlo Nucera
e3b394b6e8 Small namespace fix 2020-07-16 09:36:02 -04:00
Eelco Dolstra
5517eee17e Generations API cleanup 2020-07-16 15:18:21 +02:00
Eelco Dolstra
3f264916db Merge remote-tracking branch 'origin/flakes' 2020-07-16 14:07:32 +02:00
John Ericson
68dfb8c6ae Optimize addToStoreSlow and remove TeeParseSink 2020-07-16 05:09:41 +00:00
John Ericson
5602637d9e Revert "LocalStore::addToStoreFromDump copy in chunks"
This reverts commit 592851fb67. We don't
need this extra feature anymore
2020-07-15 23:37:49 +00:00
John Ericson
bc109648c4 Get rid of LocalStore::addToStoreCommon
I got it to just become `LocalStore::addToStoreFromDump`, cleanly taking
a store and then doing nothing too fancy with it.

`LocalStore::addToStore(...Path...)` is now just a simple wrapper with a
bare-bones sinkToSource of the right dump command.
2020-07-15 23:14:30 +00:00
Carlo Nucera
455bdee205 Merge branch 'master' of github.com:NixOS/nix into derivation-header-include-order 2020-07-15 17:58:30 -04:00
Carlo Nucera
d090562348 Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type 2020-07-15 17:21:01 -04:00
John Ericson
aff69e229c Merge remote-tracking branch 'obsidian/fix-add-to-star-narhash' into from-dump-stream 2020-07-15 21:05:29 +00:00
Carlo Nucera
2d2a10e79a Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors 2020-07-15 13:06:08 -04:00
Eelco Dolstra
cff2157185 Revert "LocalStore::addToStore(srcPath): Handle the flat case"
This reverts commit a2c27022e9. See
addToStoreSlow(), we don't need to handle this case efficiently
anymore. In fact, we can almost remove the method/hashAlgo arguments
since the non-recursive and/or non-SHA256 are almost not used anymore.
2020-07-15 12:49:03 +02:00
John Ericson
650c2c6558 Rename variable nar -> dump according to TODO 2020-07-14 21:28:50 +00:00
John Ericson
8173e7bfef Fix localhost::addToStore(...Path...)
We were calculating the nar hash wrong when the file ingestion method
was flat. I don't think there's anything we can do in that case but dump
the file again, so that's what I do.

As an optomization, we again could reuse the original dump for just the
recursive and non-sha256 case, but I rather do that after this fix, and
after my other PRs which deduplicate this code.
2020-07-14 21:12:11 +00:00
John Ericson
592851fb67 LocalStore::addToStoreFromDump copy in chunks
Rather than copying byte-by-byte, we let the coroutine know how much
data we would like it to send back to us.
2020-07-14 13:56:36 +00:00
John Ericson
9de96ef7d4 Dedup LocalStore::addToStore*
The downsides is that the coroutine has byte-by-byte loop transfer. Will
fix that next.
2020-07-14 13:56:36 +00:00
John Ericson
c86fc3a965 Crudely make addToStoreFromDump take Source not string
I just as little beyond the type as possible, so the implementation
changes this enables can be reviewed separately.
2020-07-14 13:56:36 +00:00
John Ericson
9ec10046e0 Narrow scope of temporary value 2020-07-14 13:54:29 +00:00
Eelco Dolstra
832e111494 Merge remote-tracking branch 'origin/master' into flakes 2020-07-14 13:56:18 +02:00
Eelco Dolstra
926c3a6664 Doh 2020-07-14 11:55:54 +02:00
Eelco Dolstra
43b8e96d30 Fix 'nix verify --all' on a binary cache (cached case) 2020-07-13 20:17:00 +02:00
Eelco Dolstra
7c2fef0a81 Make 'nix copy' to s3:// binary caches run in constant memory 2020-07-13 20:07:19 +02:00
Eelco Dolstra
493961b689 Remove istringstream_nocopy 2020-07-13 18:31:19 +02:00
Eelco Dolstra
545bb2ed03 Remove 'accessor' from addToStore()
This is only used by hydra-queue-runner and it's better to implement
it there.
2020-07-13 18:31:19 +02:00
Eelco Dolstra
0a9da00a10 NarAccessor: Run in constant memory 2020-07-13 17:30:42 +02:00
Eelco Dolstra
fc84c358d9 Make 'nix copy' to file:// binary caches run in constant memory 2020-07-13 16:28:45 +02:00
Eelco Dolstra
400f1a9b59 Store::pathInfoToJSON(): Use consistent format for downloadHash 2020-07-13 16:26:09 +02:00
Eelco Dolstra
c0dd05131e toStorePath(): Return a StorePath and the suffix 2020-07-13 16:25:48 +02:00
Eelco Dolstra
1d01ae816b Fix 'nix verify --all' on a binary cache and add a test 2020-07-13 14:35:01 +02:00
John Ericson
c466cb2091 Merge branch 'hash-always-has-type' into better-ca-parse-errors 2020-07-13 03:02:09 +00:00
John Ericson
4415765385 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-13 03:01:44 +00:00
John Ericson
230c9b4329 Change types to prepare the way for CA derivations
We've added the variant to `DerivationOutput` to support them, but made
`DerivationOutput::path` partial to avoid actually implementing them.

With this chage, we can all collaborate on "just" removing
`DerivationOutput::path` calls to implement CA derivations.
2020-07-12 23:53:33 +00:00
John Ericson
fedfc913ad Use more std::visit to prepare for new variant
N.B. not using `std::visit` for fetchurl because there is no attempt to
handle all the cases (e.g. no `else`) and lambda complicates early
return.
2020-07-12 23:51:40 +00:00
John Ericson
ffc18583b1 Move C++17 "pattern matching" boilerplat to utils.hh 2020-07-12 22:15:14 +00:00
John Ericson
503b425690 DerivationOutputExtensional -> DerivationOutputInputAddressed
Thanks @regnat for the great name.
2020-07-12 15:56:20 +00:00
John Ericson
a8d4707107 Undo erroneous indentation change 2020-07-12 15:54:12 +00:00
John Ericson
18152406ce String .drv suffix to create derivation name 2020-07-12 15:40:14 +00:00
John Ericson
5d0b75e5b6 std::string_view for new derivation name parameters 2020-07-12 15:02:36 +00:00
John Ericson
abea26a968 BasicDerivation::findOutput cannot return reference anymore 2020-07-12 01:57:06 +00:00
John Ericson
442d43178b Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath 2020-07-11 02:27:45 +00:00
Matthew Bauer
acb74d4d94 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-07-10 18:13:45 -04:00
Eelco Dolstra
8efa23bb99 Avoid a redundant hash 2020-07-10 15:56:24 +02:00
Eelco Dolstra
5dff49f661 Factor out commonality between nix-prefetch-url and nix-store --add-fixed 2020-07-10 13:21:37 +02:00
John Ericson
323d51a7a0 Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors 2020-07-09 19:07:06 +00:00
Matthew Bauer
a7884970c5 Fix DerivationOutputExtensional name 2020-07-09 11:37:18 -04:00
Eelco Dolstra
a2c27022e9 LocalStore::addToStore(srcPath): Handle the flat case
This helps nix-prefetch-url when using a local store.
2020-07-09 15:54:32 +02:00
Matthew Bauer
06a4e15478 Fix build.cc on linux 2020-07-08 19:27:51 -04:00
Matthew Bauer
8e0d0689be Only store hash of fixed derivation output
we don’t need a full storepath for a fixedoutput derivation. So just
putting the ingestion method + the hash is sufficient.
2020-07-08 19:11:39 -04:00
Eelco Dolstra
b981e5aacf Cleanup 2020-07-08 22:07:21 +02:00
Eelco Dolstra
34f25124ba Make LocalStore::addToStore(srcPath) run in constant memory
This reduces memory consumption of

  nix-instantiate \
    -E 'with import <nixpkgs> {}; runCommand "foo" { src = ./blender; } "echo foo"' \
    --option nar-buffer-size 10000

(where ./blender is a 1.1 GiB tree) from 1716 to 36 MiB, while still
ensuring that we don't do any write I/O for small source paths (up to
'nar-buffer-size' bytes). The downside is that large paths are now
always written to a temporary location in the store, even if they
produce an already valid store path. Thus, adding large paths might be
slower and run out of disk space. ¯\_(ツ)_/¯ Of course, you can always
restore the old behaviour by setting 'nar-buffer-size' to a very high
value.
2020-07-08 22:07:21 +02:00
Matthew Bauer
af95a7c16b Add name to BasicDerivation
We always have a name for BasicDerivation, since we have a derivation
store path that has a name.
2020-07-08 15:38:01 -04:00
Eelco Dolstra
9223603908 Merge remote-tracking branch 'origin/master' into flakes 2020-07-08 15:55:19 +02:00
Eelco Dolstra
16ec7785ca Fix 'got unknown message type 1 from Nix daemon'
Example:

  $ nix-build -E 'with import <nixpkgs> {}; runCommand "foo" { x = runCommand "bar" {} "exit 1"; } "echo foo; exit 1"'
  warning: unknown setting 'auto-allocate-uids'
  these 2 derivations will be built:
    /nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv
    /nix/store/k4fsvrjl7cp2xpz7927iv7g0dqj1zyhs-foo.drv
  warning: unknown setting 'auto-allocate-uids'
  building '/nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv'...
  error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------- nix-daemon
  builder for '/nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv' failed with exit code 1
  error: --- Error ------------------------------------------------------------------------------------------------------------------------------------------------------------------ nix-build
  got unknown message type 1 from Nix daemon
2020-07-08 15:53:14 +02:00
Eelco Dolstra
1ab9da9154 Merge remote-tracking branch 'origin/master' into flakes 2020-07-07 14:38:57 +02:00
Eelco Dolstra
7c9ece5dca exportReferencesGraph: Fix support for non-top-level store paths
Fixes #3471.
2020-07-07 14:25:43 +02:00
Carlo Nucera
0f05a36e20 Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors 2020-07-06 16:57:09 -04:00
Eelco Dolstra
54712aaf8a Merge remote-tracking branch 'origin/master' into flakes 2020-07-06 16:40:10 +02:00
Eelco Dolstra
7349f257da Only mount /sys in uid-range builds
Maybe this should be a separate system feature... /sys exposes a lot
of impure info about the host system.
2020-07-06 13:50:33 +02:00
Eelco Dolstra
8c4cce553c Fix macOS build 2020-07-06 13:50:33 +02:00
Eelco Dolstra
ba50c3efa3 Add "uid-range" and "systemd-cgroup" system features
"uid-range" provides 65536 UIDs to a build and runs the build as root
in its user namespace. "systemd-cgroup" allows the build to mount the
systemd cgroup controller (needed for running systemd-nspawn and NixOS
containers).

Also, add a configuration option "auto-allocate-uids" which is needed
to enable these features, and some experimental feature gates.

So to enable support for containers you need the following in
nix.conf:

  experimental-features = auto-allocate-uids systemd-cgroup
  auto-allocate-uids = true
  system-features = uid-range systemd-cgroup
2020-07-06 13:50:33 +02:00
Eelco Dolstra
570c443f56 Simplify cgroup creation 2020-07-06 13:50:33 +02:00
Eelco Dolstra
7bdcf43b40 Destroy the cgroup prior to building 2020-07-06 13:50:33 +02:00
Eelco Dolstra
ca2f64bcda Reduce # of UIDs per build to 65536
2^18 was overkill. The idea was to enable multiple containers to run
inside a build. However, those containers can use the same UID range -
we don't really care about perfect isolation between containers inside
a build.
2020-07-06 13:50:33 +02:00
Eelco Dolstra
f5fa3de759 Run builds in their own cgroup
Also, run builds in a cgroup namespace (ensuring /proc/self/cgroup
doesn't leak information about the outside world) and mount /sys. This
enables running systemd-nspawn and thus NixOS containers in a Nix
build.
2020-07-06 13:50:33 +02:00
Eelco Dolstra
c3e0a68c7e canonicalisePathMetaData(): Support a UID range 2020-07-06 13:50:33 +02:00
Eelco Dolstra
836573a9a2 Dynamically allocate UIDs
Rather than rely on a nixbld group, we now allocate UIDs/GIDs
dynamically starting at a configurable ID (872415232 by default).

Also, we allocate 2^18 UIDs and GIDs per build, and run the build as
root in its UID namespace. (This should not be the default since it
breaks some builds. We probably should enable this conditional on a
requiredSystemFeature.) The goal is to be able to run (NixOS)
containers in a build. However, this will also require some cgroup
initialisation.

The 2^18 UIDs/GIDs is intended to provide enough ID space to run
multiple containers per build, e.g. for distributed NixOS tests.
2020-07-06 13:50:33 +02:00
John Ericson
a38ab99d57 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-07-05 21:49:01 +00:00
John Ericson
465daa9396 Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors 2020-07-03 17:08:39 +00:00
John Ericson
3134db1a83 Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-07-03 14:12:38 +00:00
John Ericson
dbffd309fe Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type 2020-07-03 14:11:38 +00:00
Ben Burdette
b29a4ea1dc Merge branch 'master' into add-trace 2020-07-03 07:57:36 -06:00
regnat
223fbe644a Shorten the path to the test root
Fix a socket length failure on the OSX builders
2020-07-03 09:20:01 +02:00
John Ericson
13796be78d Have splitPrefix and splitPrefixTo parser helpers 2020-07-02 23:18:22 +00:00
John Ericson
2f93d9f2ba Merge branch 'hash-always-has-type' into HEAD 2020-07-02 21:47:51 +00:00
John Ericson
1be279af26 Fix Narinfo corruption detection bug
The aim of this check was just to ensure each key occurs once.
2020-07-02 21:46:10 +00:00
Carlo Nucera
1fc835aa22 Tighten parsing for drv files and pathinfo 2020-07-02 11:57:21 -04:00
Matthew Bauer
fc2ab42e86 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-07-02 11:14:04 -04:00
Matthew Bauer
d2e8b9ff0e Store subPath in SubstitutionGoal 2020-07-02 11:12:05 -04:00
Ben Burdette
bf2788e4c1 move showTrace to new loggerSettings 2020-07-02 09:04:31 -06:00
Matthew Bauer
1f9cb06db2 Try next when no ca exists and have different store dirs 2020-07-02 10:59:24 -04:00
Carlo Nucera
263ccdd489 Rename two hash constructors to proper functions 2020-07-01 18:34:18 -04:00
Eelco Dolstra
38ccf2e241 Cleanup 2020-07-01 15:31:34 +02:00
Eelco Dolstra
86a4aba6c4 Merge branch 'remote-query-outputs' of https://github.com/tweag/nix 2020-07-01 15:10:29 +02:00
Ben Burdette
9159dfe3d8 comments and cleanup 2020-06-30 16:31:55 -06:00
Matthew Bauer
09833126f3 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-30 16:35:08 -04:00
Ben Burdette
ddb81ca126 Merge branch 'master' into add-trace 2020-06-30 12:21:45 -06:00
Carlo Nucera
77b51f4598 Factor the prefix splitting in content-address 2020-06-30 11:57:46 -04:00
John Ericson
6683406843 Fix nar info parsing 2020-06-29 19:59:26 +00:00
John Ericson
70ed47c1cb Fix some things in remote store 2020-06-29 19:21:46 +00:00
John Ericson
a83566e5bc Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-06-29 18:40:34 +00:00
John Ericson
987a4a0be9 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-06-29 18:03:21 +00:00
John Ericson
a9c0ea30bf Backport fix from #3754 branch 2020-06-29 17:59:58 +00:00
Ben Burdette
8f81fae116 showTrace flag in loggers 2020-06-29 10:20:51 -06:00
Eelco Dolstra
58bc3b6578
Merge pull request #3729 from obsidiansystems/simpler-hased-mirror
hashed-mirrors: Use parsed derivation output rather than reconstructing it
2020-06-29 14:04:12 +02:00
Eelco Dolstra
64232f3ea6
Merge pull request #3749 from rodarima/master
Fall back to copyPath if link fails with EPERM
2020-06-29 13:31:24 +02:00
Carlo Nucera
278a679782 Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option 2020-06-26 10:40:13 -04:00
Eelco Dolstra
adf2fbbdc2 Merge remote-tracking branch 'origin/master' into flakes 2020-06-26 08:46:46 +02:00
John Ericson
71e4c9c505 WIP: store separate hasValidPath bool 2020-06-25 17:53:31 +00:00
John Ericson
1722ae6ece Pull out PathReferences super class 2020-06-25 17:49:22 +00:00
John Ericson
d92d4f85a5 Move ValidPathInfo to its own header 2020-06-25 17:48:49 +00:00
Eelco Dolstra
de2641ae99 Fix empty std::optional dereference in writeDerivation()
https://hydra.nixos.org/build/123017579
2020-06-25 15:50:30 +02:00
Rodrigo
3a642187c3 Fall back to copyPath if link fails with EPERM
BeeGFS doesn't allow hard-links and returns EPERM, so we fall back
to copyPath. See https://github.com/NixOS/nix/issues/3748
2020-06-25 12:03:26 +02:00
regnat
d38f860c3e Add a way to get all the outputs of a derivation with their label
Generalize `queryDerivationOutputNames` and `queryDerivationOutputs` by
adding a `queryDerivationOutputMap` that returns the map
`outputName=>outputPath`

(not that this is not equivalent to merging the results of
`queryDerivationOutputs` and `queryDerivationOutputNames` as sets don't
preserve the order, so we would end up with an incorrect mapping).

squash! Add a way to get all the outputs of a derivation with their label

Rename StorePathMap to OutputPathMap
2020-06-24 20:38:40 +02:00
John Ericson
d9fb9e9f26 Merge remote-tracking branch 'upstream/master' into ca-drv 2020-06-24 15:41:29 +00:00
John Ericson
98e5d1af03 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-06-23 17:03:37 +00:00
Carlo Nucera
e197bc6229 Enable the --store option to take relative paths
In nix commands which accept --store options, we can now specify a
relative path, which will be canonicalized.
2020-06-23 11:13:18 -04:00
John Ericson
55d4bd6e0e Improve content address parsing
- Ensure hash is in form <algo>-<prefix> and not SRI.

 - Better errors if something goes wrong

 - string_view for no coppying
2020-06-22 18:08:27 +00:00
Matthew Bauer
66a62b3189 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-22 13:08:11 -04:00
John Ericson
f4a5913125 hashed-mirrors: Use parsed derivation output rather than reconstructing it
Now the derivation outputs are parsed up front, we can avoid a reparse
by doing it. Also, this just feels a bit better as the `output*` env
vars are more of a `libnixexpr` interface than `libnixstore` interface:
ultimately, it's the derivation outputs that decide whether the
derivation is fixed-output.

Yes, hashed mirrors might go away with #3689, but this bit of code would
be moved rather than deleted, so it's worth doing a cleanup anyways I
think.
2020-06-22 15:17:20 +00:00
John Ericson
93bbe6e8ab Merge remote-tracking branch 'upstream/master' into ca-drv 2020-06-21 21:11:58 +00:00
John Ericson
3804e3df9b Don't anticipate hash algo without hash in derivation for now
When we merge with master, the new lack of string types make this case
impossible (after parsing). Later, when we actually implemenent
CA-derivations, we'll change the types to allow that.
2020-06-21 21:05:37 +00:00
John Ericson
8313f0e939 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-06-21 20:39:10 +00:00
John Ericson
bf9f040112 Tweak declaration
I think this is clearer
2020-06-21 16:51:39 +00:00
John Ericson
fdeabf7160 Merge remote-tracking branch 'upstream/master' into multi-output-hashDerivationModulo 2020-06-21 16:43:17 +00:00
Matthew Bauer
f2a6cee334 Update worker protocol to support sending storepath maps
We need to also send the ca to daemon in addition to the path.
2020-06-19 18:06:19 -04:00
John Ericson
507aa48739 WIP: Make Hash always store a valid hash type 2020-06-19 21:48:57 +00:00
Matthew Bauer
e8e1f5282f Replace error message in getDerivationCA 2020-06-19 15:19:09 -04:00
Matthew Bauer
0c9c1b8826 Return map of StorePaths in copyPaths
This allows the caller to know what values were actually added to the
store.
2020-06-19 14:48:57 -04:00
Matthew Bauer
7e11cf3399 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-19 14:40:49 -04:00
Matthew Bauer
2796b01e86 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-19 14:22:06 -04:00
John Ericson
68294746ae Merge remote-tracking branch 'upstream/master' into no-hash-type-unknown 2020-06-19 17:53:34 +00:00
John Ericson
911fc88bcb More designated initializers 2020-06-19 17:42:56 +00:00
John Ericson
2f0e395c99 Merge remote-tracking branch 'me/no-stringly-typed-derivation-output' into validPathInfo-ca-proper-datatype 2020-06-19 15:26:59 +00:00
John Ericson
fb39a5e00c Remove unneeded constructor for DerivationOutputHash 2020-06-19 15:11:11 +00:00
John Ericson
01dc8b0bab Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output 2020-06-19 14:59:05 +00:00
John Ericson
145d88cb2a Use designated initializers for DerivationOutputHash 2020-06-19 14:58:30 +00:00
John Ericson
237d88c97e FileSystemHash -> DerivationOutputHash 2020-06-19 14:47:10 +00:00
John Ericson
3fc58a9638 Remove some Base:: that crept in 2020-06-19 00:24:47 +00:00
John Ericson
3f8dcfe3fd Merge branch 'validPathInfo-temp' into validPathInfo-ca-proper-datatype 2020-06-18 23:01:58 +00:00
John Ericson
669c3992e8 Merge branch 'no-hash-type-unknown' into validPathInfo-temp 2020-06-18 22:33:07 +00:00
John Ericson
15abb2aa2b Revert the enum struct change
Not a regular git revert as there have been many merges and things.
2020-06-18 22:11:26 +00:00
John Ericson
bbbf3602a3 Merge branch 'enum-class' into no-hash-type-unknown 2020-06-18 22:11:19 +00:00
John Ericson
40526fbea5 Merge remote-tracking branch 'upstream/master' into enum-class 2020-06-18 21:38:15 +00:00
John Ericson
1b23fe4afb Fix bugs
- Bad dynamic cast target ...classic

 - std::shared_ptr need explicit deref
2020-06-18 19:03:10 +00:00
John Ericson
fb432b0bd2 Merge remote-tracking branch 'obsidian/fix-url-format' into add-body-to-network-errors 2020-06-18 18:08:02 +00:00
John Ericson
639e20dc3e Prevent '%' in URL from causing crashes
We have a larger problem that passsing computed strings to the first
variable argument of many exception constructors is unsafe because that
first variable argument is interpreted not as a plain string, but format
string, and if it contains '%' boost::format will abort, since there are
no arguments to the format string.

In this particular instance '%' was used as part of an escape code in a
URL, which, when the download failed, caused Nix to abort displaying the
`FileTransferError`.
2020-06-18 17:54:16 +00:00
John Ericson
74b219ef6e Adjust FileTransferError message to use opt response 2020-06-18 14:48:45 +00:00
regnat
4fef2ba7e4 Rename content-addressed-paths into ca-derivations
See <https://github.com/NixOS/nix/pull/3710#issuecomment-645480333>
2020-06-18 09:25:55 +02:00
Carlo Nucera
004570a377 Add HTTP responses to FileTransferErrors 2020-06-17 22:34:31 +00:00
Carlo Nucera
a835c740ca Replace TransferItem::status with a local variable
Everywhere seems to use `getHTTPStatus` now.
2020-06-17 22:16:24 +00:00
Carlo Nucera
fb05a6adcf Eliminate old TeeSink abstraction
This was introduced in fa125b9b28, and
then "reverted" in 1cf4801108, except that
revert left the struct around doing nothing useful.

We're removing it all the way now because we want to make a new
`TeeSink` complementing the already-exiting `TeeSource`, that is
actually a completely different concept as far as the class hierarchy is
concerned.
2020-06-17 22:07:22 +00:00
Eelco Dolstra
2b8f33bf5f
Merge pull request #3713 from matthewbauer/cleanup-warnings
Cleanup class StorePath warning
2020-06-17 21:19:30 +02:00
Matthew Bauer
5e631e3304 Add StorePathCAMap for querySubstitutablePathInfos
I’m not 100% sure this is wanted since it kind of makes everything
have to know about ca even if they don’t really want to. But it also
make things easier in dealing with looking up ca.
2020-06-17 15:03:05 -04:00
Matthew Bauer
be50de1142 Make sure references are empty for store path replacing
also copy info2 instead of casting
2020-06-17 14:15:08 -04:00
Matthew Bauer
8974755d19 Add assert for replaced storePath 2020-06-17 14:04:46 -04:00
Eelco Dolstra
3078404e35
Merge pull request #3712 from obsidiansystems/make-http-successful-states-coherent
Make successful states coherent
2020-06-17 19:37:46 +02:00
Matthew Bauer
7bd88cc1dc Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-17 13:32:29 -04:00
Matthew Bauer
22d7d36703 Remove unused narInfoFile in binary-cache-store 2020-06-17 13:27:19 -04:00
Matthew Bauer
f767bedfac Replace struct StorePath with class StorePath
also a similar case with struct Goal
2020-06-17 13:26:37 -04:00
Carlo Nucera
4930cb48a2 Include review comments 2020-06-17 12:58:59 -04:00
Matthew Bauer
6438ba1e99 Update strings from review comment 2020-06-17 12:31:01 -04:00
Eelco Dolstra
2f51cd8dc9
Merge pull request #3710 from tweag/reserve_ca_derivations
Reserve the `__contentAddressed` derivation parameter
2020-06-17 18:28:26 +02:00
Eelco Dolstra
4d5169bdd5
Merge pull request #3707 from p01arst0rm/outdated-function-fix
replaced uncaught_exception with uncaught_exceptions
2020-06-17 18:26:01 +02:00
regnat
480b54e1c6 fixup! Reserve the __contentAddressed derivation parameter 2020-06-17 17:37:04 +02:00
Carlo Nucera
079c6e87de Make successful states coherent
The successful states used in these two places in the code were slightly
different. Should they be the same list?
2020-06-17 11:16:16 -04:00
regnat
56d75bf4fc Reserve the __contentAddressed derivation parameter
Not implementing anything here, just throwing an error if a derivation
sets `__contentAddressed = true` without
`--experimental-features content-addressed-paths`
(and also with it as there's nothing implemented yet)
2020-06-17 15:41:17 +02:00
Eelco Dolstra
fdff09e57c Fix coverage build 2020-06-17 15:18:10 +02:00
Eelco Dolstra
ccbea8255c
Merge pull request #3657 from obsidiansystems/sligthly-improve-store-path-documentation
Clarify the description of StorePath inputs
2020-06-17 14:54:37 +02:00
Eelco Dolstra
1524752c17 Merge remote-tracking branch 'origin/master' into flakes 2020-06-17 10:26:52 +02:00
John Ericson
517f5980e2 Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output 2020-06-17 04:58:43 +00:00
John Ericson
18493fd9c4 Move some Store functions from derivations.cc to store-api.cc
This further continues with the dependency inverstion. Also I just went
ahead and exposed `parseDerivation`: it seems like the more proper
building block, and not a bad thing to expose if we are trying to be
less wedded to drv files on disk anywas.
2020-06-17 04:30:16 +00:00
John Ericson
7130f0a3a6 Don't need abstract struct Derivation in local-store 2020-06-17 04:05:38 +00:00
John Ericson
21ef342172 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-06-17 03:52:01 +00:00
John Ericson
02928f76fd Merge remote-tracking branch 'upstream/master' into multi-output-hashDerivationModulo 2020-06-17 03:51:02 +00:00
John Ericson
49308ef039 Merge remote-tracking branch 'upstream/master' into ca-drv 2020-06-17 03:39:39 +00:00
p01arst0rm
c9d06558b6 replaced uncaught_exception with uncaught_exceptions 2020-06-17 03:15:47 +01:00
Eelco Dolstra
29542865ce Remove StorePath::clone() and related functions 2020-06-16 22:20:18 +02:00
Eelco Dolstra
759947bf72 StorePath: Rewrite in C++
On nix-env -qa -f '<nixpkgs>', this reduces maximum RSS by 20970 KiB
and runtime by 0.8%. This is mostly because we're not parsing the hash
part as a hash anymore (just validating that it consists of base-32
characters).

Also, replace storePathToHash() by StorePath::hashPart().
2020-06-16 14:28:41 +02:00
Eelco Dolstra
72e17290d4 Fix FTP support
Fixes #3618.
2020-06-16 11:53:04 +02:00
Eelco Dolstra
a588b6b19d Print only one error message if a build fails
E.g. instead of

  error: --- BuildError ----------------------------------------------- nix
  builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1
  error: --- Error ---------------------------------------------------- nix
  build of '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed

we now get

  error: --- Error ---------------------------------------------------- nix
  builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1
2020-06-15 19:35:31 +02:00
Eelco Dolstra
31707735b6 Remove unnecessary amDone() overrides 2020-06-15 16:47:21 +02:00
Eelco Dolstra
5ed5d7acbd Improve "waiting for locks" messages
These are now shown in the progress bar.

Closes #3577.
2020-06-15 16:03:29 +02:00
Eelco Dolstra
e14e62fddd Remove trailing whitespace 2020-06-15 14:12:39 +02:00
Eelco Dolstra
1fb762d11f Get rid of explicit ErrorInfo constructors 2020-06-15 14:06:58 +02:00
Eelco Dolstra
7a77762961 Merge branch 'errors-phase-2' of https://github.com/bburdette/nix 2020-06-15 11:46:31 +02:00
Matthew Bauer
c214cda940 Correctly substitute from different storeDir
Originally, the test was only checking for different “real” storeDir.
That’s an easy case to handle, but the much harder one is if different
virtual store dirs are used. To do this, we need the SubstitutionGoal
to know about the ca, so it can recalculate the path to copy it over.
An important note here is that the store path passed to copyStorePath
needs to be one for srcStore - so that queryPathInfo works properly.

This also adds an error message when the store path from queryPathInfo
is different from the one we requested.
2020-06-13 00:07:42 -05:00
Matthew Bauer
b2cb288cdd Add makeFixedOutputPathFromCA function
This puts what we are already doing into a shared method. It just
needs a path name and a ca and produces a store path.
2020-06-12 16:36:35 -05:00
John Ericson
f6f01416b7 Use std::string_view in a few more places 2020-06-12 21:32:30 +00:00
Matthew Bauer
88120442d2 Debug when storePath changes
these rewrites should be transparent, but they are important to know
about when debugging
2020-06-12 15:32:52 -05:00
Matthew Bauer
5b386b05f5 Recompute storePath based on isContentAddressed 2020-06-12 14:57:45 -05:00
Matthew Bauer
85d01e1f65 Don’t use makeStorePath 2020-06-12 14:53:30 -05:00
Matthew Bauer
9077b9e6b2 Separate dstStore path from srcStore path 2020-06-12 14:27:28 -05:00
Matthew Bauer
2f2ac850b5 Compute new store path correctly 2020-06-12 14:03:17 -05:00
Matthew Bauer
11c97070f3 Fix storeDir != storeDir condition
this needs to only continue if the path replacement fails.
2020-06-12 10:14:03 -05:00
Matthew Bauer
3e3eaa90dd Remove hashed-mirrors 2020-06-12 09:51:44 -05:00
Matthew Bauer
79c169d1c6 Allow substituting from different storeDir
Substituters can substitute from one store dir to another with a
little bit of help. The store api just needs to have a CA so it can
recompute the store path based on the new store dir. We can only do
this for fixed output derivations with no references, though.
2020-06-12 09:49:09 -05:00
Eelco Dolstra
9f736dd89d Add Store::readDerivation() convenience function 2020-06-12 13:04:52 +02:00
Eelco Dolstra
045b07200c Remove Store::queryDerivationOutputNames()
This function was used in only one place, where it could easily be
replaced by readDerivation() since it's not
performance-critical. (This function appears to have been modelled
after queryDerivationOutputs(), which exists only to make the garbage
collector faster.)
2020-06-12 12:46:33 +02:00
Ben Burdette
ef1b3f21b6 Merge remote-tracking branch 'upstream/master' into errors-phase-2 2020-06-11 14:06:35 -06:00
Eelco Dolstra
ac4d43a31b
Merge pull request #3073 from tweag/machine-logs
Add an option to print the logs in a machine-readable format
2020-06-11 15:45:18 +02:00
Eelco Dolstra
dd9bb11d0d Move names.{cc,hh} to libstore 2020-06-11 15:42:18 +02:00
Eelco Dolstra
b9ae1bdd7a
Merge pull request #3655 from zimbatm/hash-encoding-prepare
libutils/hash: remove default encoding
2020-06-10 11:48:38 +02:00
Matthew Bauer
b2c8061b44
Disable extra-platforms = i686-linux on wsl1 (#3676)
WSL1 doesn’t support i686-linux emulation, see https://github.com/microsoft/wsl/issues/2468
2020-06-09 21:53:53 +00:00
Eelco Dolstra
29e0748847 Show HTTP status message
For example:

  warning: unable to download 'https://api.github.com/repos/edolstra/dwarffs/commits/master': HTTP error 403 ('rate limit exceeded'); using cached version
2020-06-09 14:20:22 +02:00
Eelco Dolstra
447ea52b07 FileTransfer: Don't store status since curl already does that 2020-06-09 14:05:15 +02:00
regnat
4983401440 Unify the printing of the logs between bar-with-logs and raw
Make the printing of the build logs systematically go through the
logger, and replicate the behavior of `no-build-output` by having two
different loggers (one that prints the build logs and one that doesn't)
2020-06-08 09:31:15 +02:00
Carlo Nucera
d614166cb6 Fix condition error and make test suite pass 2020-06-04 17:21:21 -04:00
John Ericson
e5cc1ebc5d Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output 2020-06-04 21:04:35 +00:00
John Ericson
94ddea9e2f Use readString rather than >> temporary
Fixed the rest of these before, but this one slipped through.
2020-06-04 20:55:08 +00:00
John Ericson
2041499b5e Flip boolean
Thanks Matt!
2020-06-04 20:42:25 +00:00
John Ericson
ed86acf02a Use some std::optional::has_value for clarity 2020-06-04 20:42:02 +00:00
Matthew Bauer
2299ef705c Add error message when FileIngestionMethod is out of bounds
bool coerces anything >0 to true, but in the future we may have other
file ingestion methods. This shows a better error message when the
“recursive” byte isn’t 1.
2020-06-04 11:32:39 -05:00
John Ericson
53bc8ff152 No C++ designated initializers yet with Clang 7 2020-06-03 20:45:14 -04:00
Ben Burdette
4335ba999b Merge remote-tracking branch 'upstream/master' into errors-phase-2 2020-06-03 17:00:00 -06:00
Carlo Nucera
132d6f2c24 Clarify the description of StorePath construction 2020-06-03 16:08:32 -04:00
John Ericson
74b251b2f3 Don't anticipate multiple CA outputs for now 2020-06-03 18:53:04 +00:00
John Ericson
3a9e4c3262 Don't anticipate CA but not fixed outputs for now 2020-06-03 18:50:45 +00:00
John Ericson
2500403059 Use enum and predicates rather than bitfile for derivation type 2020-06-03 17:38:54 +00:00
John Ericson
6b7f4ec4ab Merge remote-tracking branch 'upstream/master' into ca-drv 2020-06-03 16:36:06 +00:00
Eelco Dolstra
c20591ddc3 Merge remote-tracking branch 'origin/master' into flakes 2020-06-03 16:15:22 +02:00
zimbatm
6ee03b8444
libutils/hash: remove default encoding
This will make it easier to reason about the hash encoding and switch to
SRI everywhere where possible.
2020-06-03 13:49:51 +02:00
John Ericson
3c78ac348c Merge remote-tracking branch 'obsidian/no-hash-type-unknown' into validPathInfo-ca-proper-datatype 2020-06-03 04:44:24 +00:00
John Ericson
406dbb7fce outputHashAlgo can be blank so parse accordingly
It is blank for SRI hashes.
2020-06-02 21:09:15 +00:00
Carlo Nucera
75d2581390 Typo 2020-06-02 16:21:18 -04:00
Carlo Nucera
78f137e931 Validate text version instead, throw Errors 2020-06-02 16:20:22 -04:00
Carlo Nucera
a5cdf1867e Add assertions for SHA256 in fixed case 2020-06-02 16:13:08 -04:00
Carlo Nucera
fd2eb41e64 Move file-hash to content-address 2020-06-02 15:44:58 -04:00
Carlo Nucera
343c20a404 WIP Completed implementation 2020-06-02 15:23:21 -04:00
Carlo Nucera
390bf64858 WIP 2020-06-02 14:15:58 -04:00
John Ericson
a33270ce1d Clean up ValidPathInfo::isContentAddressed with std::visit 2020-06-02 17:04:21 +00:00
John Ericson
25e61812f3
Apply suggestions from code review
Co-authored-by: Matthew Bauer <mjbauer95@gmail.com>
2020-06-02 12:47:18 -04:00
John Ericson
64cffb804a Merge remote-tracking branch 'upstream/master' into no-hash-type-unknown 2020-06-02 16:07:25 +00:00
John Ericson
450dcf2c1b Remove HashType::Unknown
Instead, `Hash` uses `std::optional<HashType>`. In the future, we may
also make `Hash` itself require a known hash type, encoraging people to
use `std::optional<Hash>` instead.
2020-06-02 15:52:13 +00:00
Carlo Nucera
a5d820a0a3 Change parseCa(Opt) to parseContentAddress(Opt) 2020-06-02 11:00:10 -04:00
John Ericson
1b6461f671 Merge remote-tracking branch 'upstream/master' into validPathInfo-ca-proper-datatype 2020-06-02 14:31:18 +00:00
Ben Burdette
d82d230b40 elide the 'ErrorInfo' in logError and logWarning calls 2020-06-02 08:22:24 -06:00
John Ericson
efcd30da89 WIP 2020-06-02 00:37:43 +00:00
John Ericson
754c910953 WIP more progress 2020-06-01 19:26:40 -04:00
Carlo Nucera
da39092a39 WIP 2020-06-01 18:53:31 -04:00
Carlo Nucera
0e9438b6d3 Create new file-hash files 2020-06-01 17:32:40 -04:00
Carlo Nucera
0cb67ecbd3 Merge branch 'derivation-header-include-order' of github.com:Ericson2314/nix into validPathInfo-ca-proper-datatype 2020-06-01 17:13:11 -04:00
Carlo Nucera
f4b89e11a4 Merge branch 'no-stringly-typed-derivation-output' of github.com:Ericson2314/nix into validPathInfo-ca-proper-datatype 2020-06-01 17:12:50 -04:00
John Ericson
fac0c2d54a Remove addToStore variant as requested by FIXME
The idea is it's always more flexible to consumer a `Source` than a
plain string, and it might even reduce memory consumption.

I also looked at `addToStoreFromDump` with its `// FIXME: remove?`, but
the worked needed for that is far more up for interpretation, so I
punted for now.
2020-05-29 17:02:32 -04:00
Ben Burdette
734283d636 Merge remote-tracking branch 'upstream/master' into errors-phase-2 2020-05-29 09:51:37 -06:00
Carlo Nucera
6dd471ebf6 Fixing the result of merge 2020-05-28 12:14:36 -04:00
Carlo Nucera
4f597fb901 Merge branch 'master' of github.com:NixOS/nix into enum-class 2020-05-28 10:58:22 -04:00
John Ericson
5b4cd84bc2 Merge remote-tracking branch 'me/more-rust-ffi' into no-stringly-typed-derivation-output 2020-05-28 10:35:53 -04:00
John Ericson
ef71caba29 Merge remote-tracking branch 'upstream/master' into more-rust-ffi 2020-05-28 10:31:46 -04:00
Eelco Dolstra
17ca997fc6 Merge remote-tracking branch 'origin/master' into flakes 2020-05-28 12:55:24 +02:00
Matthew Bauer
c66441a646 Rename some variables named “recursive” to “method”
This is much less confusing since recursive is no longer a boolean.
2020-05-27 13:21:26 -05:00
Carlo Nucera
d49e65ba9d Merge remote-tracking branch 'john-ericson/enum-FileIngestionMethod' into no-stringly-typed-derivation-output 2020-05-26 12:30:48 -04:00
Carlo Nucera
0f3f901071 Merge remote-tracking branch 'origin/master' into more-rust-ffi 2020-05-26 11:46:42 -04:00
Carlo Nucera
6d73c10041 Merge remote-tracking branch 'origin/master' into enum-FileIngestionMethod 2020-05-26 11:14:08 -04:00
Eelco Dolstra
5ef64f05e6 Cleanup 2020-05-18 15:50:29 +02:00
Eelco Dolstra
0ed946aa61 Merge branch 'wait-for-builders' of https://github.com/serokell/nix 2020-05-18 13:48:45 +02:00
Ben Burdette
92123c6c79 Merge remote-tracking branch 'upstream/master' into errors-phase-2 2020-05-15 07:00:36 -06:00
Alexander Bantyev
183dd28266
Don't lock a user while doing remote builds 2020-05-14 17:00:54 +03:00
Ben Burdette
ef9dd9f9bc formatting and a few minor changes 2020-05-13 15:56:39 -06:00
Ben Burdette
c79d4addab consistent capitalization 2020-05-13 10:02:18 -06:00
Ben Burdette
bfca5fc395 change status messages to info level 2020-05-13 09:52:36 -06:00
Ben Burdette
19cffc29c9 remove unused extra json fields 2020-05-12 12:09:12 -06:00
Ben Burdette
ec870b9c85 new pos format for more errors 2020-05-12 10:52:26 -06:00
Eelco Dolstra
ebc024df22
Show hint how to enable experimental features 2020-05-12 15:47:09 +02:00
Ben Burdette
b93c1bf3d6 fixes to merged code 2020-05-11 15:52:15 -06:00
Ben Burdette
59b1f5c701 Merge branch 'master' into errors-phase-2 2020-05-11 14:35:30 -06:00
Ben Burdette
958e81987b switch from printError warnings to logWarnings 2020-05-11 13:02:16 -06:00
Domen Kožar
5bdb67c843
Merge pull request #3568 from kolloch/outputHashModeError
libstore/build.cc: more explicit error about form of output
2020-05-11 18:14:32 +02:00
Domen Kožar
1d8144e36b
Update src/libstore/build.cc 2020-05-11 18:14:23 +02:00
Benjamin Hipple
146f9c114f doc: consistently refer to 'fixed-output' with a dash
General cleanup that makes it easier to search for the term.
2020-05-09 10:58:43 -04:00
Alexander Bantyev
772e5db828
Mention build users in the 'waiting for' message 2020-05-08 12:29:00 +03:00
Alexander Bantyev
14073fb76b
Don't block while waiting for build users 2020-05-08 12:22:39 +03:00
Ben Burdette
e3901638b5 todo removal 2020-05-06 15:01:13 -06:00
Eelco Dolstra
2f8ee4578f
Merge remote-tracking branch 'origin/master' into flakes 2020-05-06 12:01:40 +02:00
Eelco Dolstra
74a1bfdcab
Merge pull request #3546 from guibou/nix_readfile_on_0_sized_files
builtins.readFile: do not truncate content
2020-05-06 11:33:55 +02:00
Peter Kolloch
9be46859a9 libstore/build.cc: more explicit about form of output
Be more explicit about why we expect a regular file as output
when outputHashMode=flat for a fixed output derivation.
2020-05-06 11:21:12 +02:00
Eelco Dolstra
02c5914ea4
Merge pull request #3562 from pikajude/master
Use fragment size for autoGC capacity calculation
2020-05-06 11:19:30 +02:00
Eelco Dolstra
fd4911269f
Revert "Merge pull request #3558 from LnL7/ssh-ng-stderr"
This reverts commit 3ebfbecdd1, reversing
changes made to c089c52d5f.

https://github.com/NixOS/nix/pull/3558
2020-05-06 10:54:18 +02:00
Eelco Dolstra
6f3244ce45 Merge remote-tracking branch 'origin/master' into flakes 2020-05-05 18:59:33 +02:00
Alexander Bantyev
04967dee9d
Wait for build users when none are available 2020-05-05 13:04:36 +03:00
Ben Burdette
7ffb5efdbc appending to hints; remove _printError 2020-05-04 16:19:57 -06:00
Jude Taylor
e2fc575c61 nix auto-gc: use fragment size 2020-05-04 14:42:06 -07:00
Eelco Dolstra
a721a0b114 Flag: Use designated initializers 2020-05-04 22:40:19 +02:00
Ben Burdette
ab6f0b9641 convert some printError calls to logError 2020-05-03 08:01:25 -06:00
Daiderd Jordan
f16e24f95e
remote-store: don't log raw stderr by default
For remote stores the log messages are already forwarded as structured
STDERR_RESULT messages so the old format is duplicate information.  But
still included with -vvv since it could be useful for debugging
problems.

    $ nix build -L /nix/store/nl71b2niws857ffiaggyrkjwgx9jjzc0-foo.drv --store ssh-ng://localhost
    Hello World!
    foo> Hello World!
    [1/0/1 built] building foo

Fixes #3556
2020-05-02 23:40:50 +02:00
Ben Burdette
4b99c09f5c convert some errors 2020-05-01 14:32:06 -06:00
Eelco Dolstra
941f95284a Merge remote-tracking branch 'origin/master' into flakes 2020-05-01 11:59:56 +02:00
Ben Burdette
39ff80d031 errorinfo constructor test 2020-04-29 18:57:05 -06:00
Guillaume Bouchard
7afcb5af98 Remove the drain argument from readFile
Now it is always `drain` (see previous commit).
2020-04-29 18:43:45 +02:00
Antoine Eiche
ca93b26db6 Only call grantpt on MacOS systems
The commit 3cc1125595 adds a `grantpt`
call on the builder pseudo terminal fd. This call is actually only
required for MacOS, but it however requires a RW access to /dev/pts
which is only RO bindmounted in the Bazel Linux sandbox. So, Nix can
not be actually run in the Bazel Linux sandbox for unneeded reasons.
2020-04-29 15:43:20 +02:00
Ben Burdette
22e6490311 Error classname as name 2020-04-28 21:06:08 -06:00
Eelco Dolstra
ef4d3fc111 Merge remote-tracking branch 'origin/master' into flakes 2020-04-23 15:16:18 +02:00
Dustin DeWeese
c0d940978a Replace select() with poll() to allow waiting on more than FD_SETSIZE fds 2020-04-21 16:21:28 -07:00
Ben Burdette
e4fb9a3849 remove 'format' from Error constructor calls 2020-04-21 17:07:07 -06:00
Ben Burdette
15e9564fd1 logEI for tunnelLogger and progressbar 2020-04-19 17:16:51 -06:00
Ben Burdette
3d5b1032a1 logError, logWarning; Logger functions; switch to Verbosity enum 2020-04-17 15:07:44 -06:00
Eelco Dolstra
aaa109565e Use a more space/time-efficient representation for the eval cache 2020-04-17 23:04:21 +02:00
Eelco Dolstra
7a9687ba30 SQLiteStmt: Use std::string_view 2020-04-17 01:00:56 +02:00
Eelco Dolstra
0858738355 Merge remote-tracking branch 'origin/master' into flakes 2020-04-16 18:27:37 +02:00
Eelco Dolstra
a118293bd0
Merge pull request #3458 from zimbatm/nix-user-conf-dir
NIX_USER_CONF_FILES
2020-04-15 13:00:28 +02:00
zimbatm
895516cadf
add NIX_USER_CONF_FILES
Motivation: maintain project-level configuration files.

Document the whole situation a bit better so that it corresponds to the
implementation, and add NIX_USER_CONF_FILES that allows overriding
which user files Nix will load during startup.
2020-04-14 18:45:06 +02:00
Eelco Dolstra
c0c2cb871d Merge remote-tracking branch 'origin/master' into flakes 2020-04-14 13:02:55 +02:00
Bruce Toll
8132d0a12e Fix nix-build --check -K in sandbox w/o root
Temporarily add user-write permission to build directory so that it
can be moved out of the sandbox to the store with a .check suffix.

This is necessary because the build directory has already had its
permissions set read-only, but write permission is required
to update the directory's parent link to move it out of the sandbox.

Updated the related --check "derivation may not be deterministic"
messages to consistently use the real store paths.

Added test for non-root sandbox nix-build --check -K to demonstrate
issue and help prevent regressions.
2020-04-10 16:23:10 -04:00
Domen Kožar
db25a6d7bb
Merge pull request #2689 from tollb/fix/delete_tmp_dir_when_build_check_ok
Delete temporary directory on successful build
2020-04-10 09:47:57 +02:00
Bruce Toll
16a4864759 Delete temporary directory on successful build
With --check and the --keep-failed (-K) flag, the temporary directory
was being retained regardless of whether the build was successful and
reproducible.  This removes the temporary directory, as expected, on
a reproducible check build.

Added tests to verify that temporary build directories are not
retained unnecessarily, particularly when using --check with
--keep-failed.
2020-04-09 16:37:41 -04:00
Philipp Middendorf
04bedda0b6 gc.cc: Ignore hidden files in temproots 2020-04-09 17:05:29 +02:00
Eelco Dolstra
d103c79144
Merge remote-tracking branch 'origin/master' into flakes 2020-04-09 15:54:21 +02:00
Nikola Knezevic
f5095594e7 datatransfer.{cc,hh} -> filetransfer.{cc,hh} 2020-04-08 22:26:57 +02:00
Nikola Knezevic
c330109bfa DataTransfer -> FileTransfer 2020-04-08 22:26:57 +02:00
Nikola Knezevic
7848372b0f Add upload method 2020-04-08 22:26:57 +02:00
Nikola Knezevic
a0c5931208 actDownload -> actDataTransfer 2020-04-08 22:26:57 +02:00
Nikola Knezevic
c4c1ae0a00 DownloadError -> DataTransferError 2020-04-08 22:26:57 +02:00
Nikola Knezevic
213d124277 DownloadItem -> TransferItem 2020-04-08 22:26:57 +02:00
Nikola Knezevic
cd391206e6 {get,make,new}Downloader -> DataTransfer 2020-04-08 22:26:57 +02:00
Nikola Knezevic
2df2741ec6 enqueueDownload -> enqueueDataTransfer 2020-04-08 22:26:57 +02:00
Nikola Knezevic
142ed7fe45 DownloadResult -> DataTransferResult 2020-04-08 22:26:57 +02:00
Nikola Knezevic
e5cc53beec DownloadSettings -> DataTransferSettings 2020-04-08 22:26:57 +02:00
Nikola Knezevic
741e9012d3 Rename src/lib/download.* to src/lib/datatransfer.* 2020-04-08 22:26:57 +02:00
Nikola Knezevic
65ef57e0cb DownloadRequest -> DataTransferRequest 2020-04-08 22:26:57 +02:00
Domen Kožar
bf81b31559
build.cc: improve message if home directory exists 2020-04-08 18:27:10 +02:00
Eelco Dolstra
1ab8d6ac18
Downloader: Only write data to the sink on a 200 response
Hopefully fixes #3278.
2020-04-08 15:27:09 +02:00
Eelco Dolstra
54955867a6 Merge remote-tracking branch 'origin/master' into flakes 2020-04-07 14:29:45 +02:00
Eelco Dolstra
cd39709003 Cleanup 2020-04-07 09:27:17 +02:00
Eelco Dolstra
462421d345 Backport libfetchers from the flakes branch
This provides a pluggable mechanism for defining new fetchers. It adds
a builtin function 'fetchTree' that generalizes existing fetchers like
'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a
set of attributes, e.g.

  fetchTree {
    type = "git";
    url = "https://example.org/repo.git";
    ref = "some-branch";
    rev = "abcdef...";
  }

The existing fetchers are just wrappers around this. Note that the
input attributes to fetchTree are the same as flake input
specifications and flake lock file entries.

All fetchers share a common cache stored in
~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching
mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}).

This also adds support for Git worktrees (c169ea5904).
2020-04-07 09:03:14 +02:00
Cole Helbling
c976cb0b8a
Don't retry on "unsupported protocol" error
When encountering an unsupported protocol, there's no need to retry.
Chances are, it won't suddenly be supported between retry attempts;
error instead. Otherwise, you see something like the following:

    $ nix-env -i -f git://git@github.com/foo/bar
    warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 335 ms
    warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 604 ms
    warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 1340 ms
    warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 2685 ms

With this change, you now see:

    $ nix-env -i -f git://git@github.com/foo/bar
    error: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1)
2020-04-05 09:00:34 -07:00
Eelco Dolstra
4fdec5f61d Merge remote-tracking branch 'origin/master' into flakes 2020-04-01 00:03:50 +02:00
mlatus
12556e5709 fix placeholder not substituted in passAsFile 2020-03-31 19:40:16 +08:00
John Ericson
51afea3af2 Never cast FileIngestionMethod to or from boolean 2020-03-30 22:31:51 +00:00
John Ericson
c251b011cd Merge remote-tracking branch 'upstream/master' into enum-FileIngestionMethod 2020-03-30 18:16:44 -04:00
John Ericson
832bd534dc Store parsed hashes in DerivationOutput
It's best to detect invalid data as soon as possible, with data types
that make storing it impossible.
2020-03-30 11:33:35 -04:00
John Ericson
f5494d9442 Merge remote-tracking branch 'me/enum-FileIngestionMethod' into HEAD 2020-03-30 11:08:13 -04:00
Eelco Dolstra
d15d91cad1 Makefile cleanup 2020-03-30 16:50:13 +02:00
Eelco Dolstra
e322a16523 Remove global -I flags
(cherry picked from commit 2c692a3b14)
2020-03-30 15:30:19 +02:00
Eelco Dolstra
2c692a3b14 Remove global -I flags 2020-03-30 14:39:33 +02:00
Eelco Dolstra
e0a0ae0467 Move fetchers from libstore to libfetchers 2020-03-30 14:04:53 +02:00
Eelco Dolstra
2287cc6486 Fix segfault 2020-03-30 13:31:55 +02:00
John Ericson
225e62a56a Replace some bool recursive with a new FileIngestionMethod enum 2020-03-29 15:16:20 -04:00
John Ericson
87b32bab05 Use enum struct and drop prefixes
This does a few enums; the rest will be gotten in subsequent commits.
2020-03-29 11:23:15 -04:00
Eelco Dolstra
2287e2f279 nix flake info: Show flake subdirectory 2020-03-28 18:05:50 +01:00
John Ericson
e433d4af4c Extend Rust FFI
Do idiomatic C++ copy and move constructors for a few things, so
wrapping structs' defaults can work.
2020-03-25 16:12:14 -04:00
John Ericson
bcde5456cc Flip dependency so store-api.hh includes derivations.hh
I think it makes more sense to define the data model (derivations),
before the operations (store api).
2020-03-24 20:39:45 +00:00
Eelco Dolstra
0a10854f85 Misc changes from the flakes branch 2020-03-24 14:34:47 +01:00
Eelco Dolstra
777e21e596 nix path-info --json: Print hash in SRI format
(cherry picked from commit 442e665d6d)
2020-03-24 14:26:23 +01:00
Eelco Dolstra
f9611c7ae4 buildenv: Eliminate global variables, other cleanup
(cherry picked from commit b82f75464d)
2020-03-24 14:06:47 +01:00
Eelco Dolstra
edc34cc1a2 Add function for quoting strings
(cherry picked from commit 7dcf5b011a)
2020-03-24 13:44:04 +01:00
Eelco Dolstra
5a7e7fc35f Use std::string_view
(cherry picked from commit 6529490cc1)
2020-03-24 13:26:37 +01:00
Eelco Dolstra
7abe3bde8a Merge remote-tracking branch 'origin/master' into flakes 2020-03-23 13:27:54 +01:00
Eelco Dolstra
5885e20404
Merge pull request #3429 from LnL7/darwin-sandbox
darwin sandbox
2020-03-23 09:13:36 +01:00
Daiderd Jordan
2e9bc1245c
sandbox: fix /bin/sh on catalina
Sadly 10.15 changed /bin/sh to a shim which executes bash, this means it
can't be used anymore without also opening up the sandbox to allow bash.

    Failed to exec /bin/bash as variant for /bin/sh (1: Operation not permitted).
2020-03-20 22:12:30 +01:00
Daiderd Jordan
f6c122aaeb
sandbox: allow pty devices
Nix now runs builds with a pseudo-terminal to enable colored build
output.
2020-03-20 21:58:45 +01:00
Daiderd Jordan
7f2df903d9
libstore: relax default sandbox-paths on darwin 2020-03-20 21:31:20 +01:00
Daiderd Jordan
afb78ebd34
libstore: disable resolve-system-dependencies hook
This is used to determine the dependency tree of impure libraries so nix
knows what paths to open in the sandbox.  With the less restrictive
defaults it isn't needed anymore.
2020-03-20 21:21:56 +01:00
John Ericson
e317324236
Apply suggestions from code review 2020-03-19 23:38:51 -04:00
John Ericson
d5b3328dd1
Apply suggestions from code review
Co-Authored-By: Cole Helbling <cole.e.helbling@outlook.com>
2020-03-19 23:37:52 -04:00
John Ericson
f1cf3ab870 hashDerivationModulo: Generalize for multiple fixed ouputs per drv
See documentattion in header and comments in implementation for details.

This is actually done in preparation for floating ca derivations, not
multi-output fixed ca derivations, but the distinction doesn't yet
mattter.

Thanks @cole-h for finding and fixing a bunch of typos.
2020-03-19 10:30:49 -04:00
Eelco Dolstra
8aa354fdfd
Register flake-registry.json as a GC root again 2020-03-19 11:45:34 +01:00
Eelco Dolstra
1c127e6a82
downloadFile(): Use expired file if the download fails 2020-03-19 11:42:50 +01:00
John Ericson
049179ba07
Fix typos
Thanks @asymmetric

I failed to do them all in one batch

Co-Authored-By: asymmetric <lorenzo@mailbox.org>
2020-03-18 19:07:05 -04:00
Eelco Dolstra
f6ddf48882
Get rid of downloadCached()
Everything uses the generic caching system now.
2020-03-18 17:24:19 +01:00
Eelco Dolstra
c5ec95e2c7
tarball.cc: Use ETags 2020-03-18 15:14:23 +01:00
Eelco Dolstra
1e7ce1d6da
tarball / github fetchers: Use generic caching system 2020-03-18 14:08:25 +01:00
Eelco Dolstra
38e360154d
Git: Use unified caching system 2020-03-17 22:35:29 +01:00
Eelco Dolstra
d1165d8791
Require shallow clones to be requested explicitly
If you do a fetchTree on a Git repository, whether the result contains
a revCount attribute should not depend on whether that repository
happens to be a shallow clone or not. That would complicate caching a
lot and would be semantically messy. So applying fetchTree/fetchGit to
a shallow repository is now an error unless you pass the attribute
'shallow = true'. If 'shallow = true', we don't return revCount, even
if the repository is not actually shallow.

Note that Nix itself is not doing shallow clones at the moment. But it
could do so as an optimisation if the user specifies 'shallow = true'.

Issue #2988.
2020-03-17 22:35:29 +01:00
Eelco Dolstra
2a4e4f6a6e
Unified fetcher caching system 2020-03-17 22:35:29 +01:00
John Ericson
e5178fd22d
Fix typos
Thanks @asymmetric!

Co-Authored-By: asymmetric <lorenzo@mailbox.org>
2020-03-16 16:40:13 -04:00
Eelco Dolstra
fbcb897e21
Add a test for shallow Git clones
Also, don't return a revCount anymore for shallow or dirty Git trees,
since it's incorrect.

Closes #2988.
2020-03-16 13:20:32 +01:00
John Ericson
2be64efb02 Generalize isFixedOutput in preparation for CA drvs
Today's fixed output derivations and regular derivations differ in a few
ways which are largely orthogonal. This replaces `isFixedOutput` with a
`type` that returns an enum of possible combinations.
2020-03-15 11:05:37 -04:00
Eelco Dolstra
0c2088d438 Merge remote-tracking branch 'origin/master' into flakes 2020-03-13 17:03:30 +01:00
Eelco Dolstra
b816515f61
Fix ca-references feature check
Fixes #3406.
2020-03-13 13:15:51 +01:00
Robert Hensing
3f55f8a8fb pathInfoCache: Respect disk cache TTLs #3398 2020-03-12 10:30:28 +01:00
Eelco Dolstra
e188fe7c6d Move call-flake.nix into libexpr 2020-03-11 17:04:51 +01:00
Eelco Dolstra
73b6d87e17
Merge remote-tracking branch 'origin/master' into flakes 2020-03-04 13:58:42 +01:00
Eelco Dolstra
d37dc71e3c
nix-build: Fix !<output> handling
This was broken by 22a754c091.

https://hydra.nixos.org/eval/1573669
2020-03-04 13:56:17 +01:00
Eelco Dolstra
75db069f92
Optimise Derivation::unparse()
In

  nix-instantiate --dry-run '<nixpkgs/nixos/release-combined.nix>' -A nixos.tests.simple.x86_64-linux

this reduces time spent in unparse() from 9.15% to 4.31%. The main
culprit was appending characters one at a time to the destination
string. Even though the string has enough capacity, push_back() still
needs to check this on every call.
2020-03-04 11:44:45 +01:00
Eelco Dolstra
22a754c091
Fix GC failures on bad store path names
It failed on names like '/nix/store/9ip48nkc9rfy0a4yaw98lp6gipqlib1a-'.
2020-02-28 18:07:10 +01:00
Eelco Dolstra
4a4521f462 Fix nlohmann::json exception 2020-02-20 23:47:02 +01:00
Eelco Dolstra
73c9840569 Restore subdir -> dir
Got this mixed up somewhere.
2020-02-20 23:44:06 +01:00
Eelco Dolstra
d068f9ffff Restore subdir support in registries
Hacky...
2020-02-20 22:14:44 +01:00
Eelco Dolstra
1d99c4ab25
Merge pull request #3229 from Ma27/flakes-fetchgit-worktree-support
builtins.fetchGit: Fix build when fetching a git worktree
2020-02-19 14:11:18 +01:00
Maximilian Bosch
c169ea5904
builtins.fetchGit: Fix build when fetching a git worktree
Worktrees[1] are a feature of git which allow you to check out a ref in
a different directory.

While playing around with flakes I realized that git repositories in a
worktree checkout break when trying to build a flake:

```
$ git worktree add ../nixpkgs-flakes nixpkgs-flakes
$ cd ../nixpkgs-flakes
$ nix build .#hello
error: opening directory '/home/ma27/Projects/nixpkgs-flakes/.git/refs/heads': Not a directory
```

This issue has been fixed by determining with `git rev-parse --git-common-dir`
where the actual `.git` directory is.

Please note that this issue only exists on the `flakes` branch, fetching
worktree checkouts with Nix master seems to work fine.

[1] https://git-scm.com/docs/git-worktree
2020-02-19 14:00:36 +01:00
Eelco Dolstra
f3505a7899 Merge remote-tracking branch 'origin/master' into flakes 2020-02-19 12:57:45 +01:00
Eelco Dolstra
c4d3674de6
Merge pull request #3353 from tbsmoest/priv_tobias_pr_set_deathsig-1.4
Fix PR_SET_PDEATHSIG results in Broken pipe (#2395)
2020-02-19 12:29:12 +01:00
Eelco Dolstra
553e584f92
LocalStore::checkDerivationOutputs(): Improve error message 2020-02-18 17:51:48 +01:00
Eelco Dolstra
b5e3c04c03
Fix URL parser
Fixes #3361.
2020-02-18 12:51:26 +01:00
Eelco Dolstra
6529490cc1 nix eval-hydra-jobs: Support job names as aggregate constituents
Fixes https://github.com/NixOS/hydra/issues/715.
2020-02-17 15:53:59 +01:00
Eelco Dolstra
8a78bcf6a2 LocalStore::checkDerivationOutputs(): Improve error message 2020-02-17 15:46:07 +01:00
Eelco Dolstra
6208d24c38 Merge remote-tracking branch 'origin/master' into flakes 2020-02-14 22:42:35 +01:00
Tobias Möst
3e347220c8 Fix PR_SET_PDEATHSIG results in Broken pipe (#2395)
The ssh client is lazily started by the first worker thread, that
requires a ssh connection. To avoid the ssh client to be killed, when
the worker process is stopped, do not set PR_SET_PDEATHSIG.
2020-02-14 07:51:44 +01:00
Eelco Dolstra
d8972317fc Prevent uninitialized StorePath creation 2020-02-13 16:12:16 +01:00
Eelco Dolstra
d4df99a334 Parse narHash attribute for all input types 2020-02-11 23:53:24 +01:00
Eelco Dolstra
442e665d6d nix path-info --json: Print hash in SRI format 2020-02-11 23:50:16 +01:00
Eelco Dolstra
84a3a5c3cd Merge remote-tracking branch 'origin/master' into flakes 2020-02-07 13:57:31 +01:00
Eelco Dolstra
d82b78bf51
Fix segfault in gcc on i686-linux
src/libstore/ssh-store.cc: In constructor 'nix::SSHStore::SSHStore(const string&, const Params&)':
  src/libstore/ssh-store.cc:31:21: internal compiler error: Segmentation fault
               compress)
                       ^
  Please submit a full bug report,
  with preprocessed source if appropriate.

https://hydra.nixos.org/build/111545609
2020-02-07 13:01:48 +01:00
Eelco Dolstra
379852a152 Registry: Use attr notation instead of URLs 2020-02-06 14:27:31 +01:00
Eelco Dolstra
022287060b Merge remote-tracking branch 'origin/master' into flakes 2020-02-05 15:41:51 +01:00
Eelco Dolstra
9d7fb62db6 Add option --commit-lock-file 2020-02-05 14:48:49 +01:00
Eelco Dolstra
e2213d77a2 Set 'ref' properly for local trees 2020-02-04 21:55:57 +01:00
Eelco Dolstra
db88cb401b
Merge pull request #3344 from LnL7/ssh-ng-remote-params
ssh-store: add remote-store and remote-program query params
2020-02-04 10:10:08 +01:00
Daiderd Jordan
8745c63d3c
ssh-store: add remote-store and remote-program query params
Brings the functionality of ssh-ng:// in sync with the legacy ssh://
implementation.  Specifying the remote store uri enables various useful
things. eg.

    $ nix copy --to ssh-ng://cache?remote-store=file://mnt/cache --all
2020-02-03 23:22:28 +01:00
Eelco Dolstra
0a4e911cf4 Install headers in the correct location 2020-02-03 18:04:09 +01:00
Eelco Dolstra
a2628b43bb Fix URL parser
Fixes #3062.
2020-02-03 15:27:26 +01:00
Eelco Dolstra
d070e1c532 Fix parsing of '#nixosConfigurations."hostname".config...'
This is not strictly speaking valid but who cares.
2020-02-03 14:29:34 +01:00
Eelco Dolstra
d5334c466b
Automatically do git/hg add on flake.lock 2020-02-02 16:33:22 +01:00
Eelco Dolstra
a9ebc3ea5d
Remove the git+ and hg+ prefixes from structured input refs 2020-02-02 13:06:00 +01:00
Eelco Dolstra
fad9faf354
Add TreeInfo::computeStorePath() 2020-02-02 12:29:53 +01:00
Eelco Dolstra
7bcc9f2aaf
Add convenience overload for toRealPath() 2020-02-02 12:28:56 +01:00
Eelco Dolstra
958ec5de56
Cleanup 2020-02-02 11:31:58 +01:00
Eelco Dolstra
887730aab3
Remove superfluous TreeInfo::rev field 2020-02-01 23:54:20 +01:00
Eelco Dolstra
b9d64f9318
Record TreeInfo in the lock file
Necessary for #3253.
2020-02-01 23:33:44 +01:00
Eelco Dolstra
8451298b35
Factor out TreeInfo 2020-02-01 22:06:55 +01:00
Eelco Dolstra
185c3c8240 Cleanup 2020-01-31 19:35:28 +01:00
Eelco Dolstra
8414685c0f Change lock file format to use an attribute representation of flake refs rather than URLs 2020-01-31 19:16:40 +01:00
Eelco Dolstra
88b44b1e94 Fix flake update check 2020-01-29 23:14:23 +01:00
Eelco Dolstra
7be1a07a45
Merge pull request #3335 from domenkozar/retry-429
retry on HTTP status code 429
2020-01-29 16:22:46 +01:00
Domen Kožar
48ddb8e481
retry on HTTP status code 429 2020-01-29 11:47:39 +01:00
Eelco Dolstra
ab41e9d543 Merge remote-tracking branch 'origin/master' into flakes 2020-01-28 13:19:30 +01:00
Eelco Dolstra
1af7b94c1d Add support for tarball flake inputs
For example,

  $ nix flake info https://github.com/edolstra/dwarffs/archive/master.tar.gz

Fixes #2929.
2020-01-28 13:11:02 +01:00
Eelco Dolstra
5046233b5a Add Mercurial tests 2020-01-27 13:45:49 +01:00
Eelco Dolstra
cc22cf662b Respect lock files of inputs + fine-grained lock file control
When computing a lock file, we now respect the lock files of flake
inputs. This is important for usability / reproducibility. For
example, the 'nixops' flake depends on the 'nixops-aws' and
'nixops-hetzner' repositories. So when the 'nixops' flake is used in
another flake, we want the versions of 'nixops-aws' and
'nixops-hetzner' locked by the the 'nixops' flake because those
presumably have been tested.

This can lead to a proliferation of versions of flakes like 'nixpkgs'
(since every flake's lock file could depend on a different version of
'nixpkgs'). This is not a major issue when using Nixpkgs overlays or
NixOS modules, since then the top-level flake composes those
overlays/modules into *its* version of Nixpkgs and all other versions
are ignored. Lock file computation has been made a bit more lazy so it
won't try to fetch all those versions of 'nixpkgs'.

However, in case it's necessary to minimize flake versions, there now
are two input attributes that allow this. First, you can copy an input
from another flake, as follows:

  inputs.nixpkgs.follows = "dwarffs/nixpkgs";

This states that the calling flake's 'nixpkgs' input shall be the same
as the 'nixpkgs' input of the 'dwarffs' input.

Second, you can override inputs of inputs:

  inputs.nixpkgs.url = github:edolstra/nixpkgs/<hash>;
  inputs.nixops.inputs.nixpkgs.url = github:edolstra/nixpkgs/<hash>;

or equivalently, using 'follows':

  inputs.nixpkgs.url = github:edolstra/nixpkgs/<hash>;
  inputs.nixops.inputs.nixpkgs.follows = "nixpkgs";

This states that the 'nixpkgs' input of the 'nixops' input shall be
the same as the calling flake's 'nixpkgs' input.

Finally, at '-v' Nix now prints the changes to the lock file, e.g.

  $ nix flake update ~/Misc/eelco-configurations/hagbard
  inputs of flake 'git+file:///home/eelco/Misc/eelco-configurations?subdir=hagbard' changed:
    updated 'nixpkgs': 'github:edolstra/nixpkgs/7845bf5f4b3013df1cf036e9c9c3a55a30331db9' -> 'github:edolstra/nixpkgs/03f3def66a104a221aac8b751eeb7075374848fd'
    removed 'nixops'
    removed 'nixops/nixops-aws'
    removed 'nixops/nixops-hetzner'
    removed 'nixops/nixpkgs'
2020-01-24 22:05:11 +01:00
Robin Gloster
f8dbde0813
structured-attrs: chown .attrs.* files to builder
Otherwise `chmod .`'ing the build directory doesn't work anymore, which
is done in nixpkgs if sourceRoot is set to '.'.
2020-01-23 17:38:07 +01:00
Eelco Dolstra
9a5ca802c7 clang fixes
https://hydra.nixos.org/build/110757171
2020-01-22 21:26:19 +01:00
Eelco Dolstra
d506bd587a Fix clang warning 2020-01-22 21:20:01 +01:00
Eelco Dolstra
b5c9dbc84f Fix --override-flake and add a test 2020-01-22 20:00:58 +01:00
Eelco Dolstra
b33b94748c Convert fetchMercurial to a input type
This enables Mercurial flakes. It also fixes a bug in pure mode where
you could use a branch/tag name rather than a revision.
2020-01-21 23:49:32 +01:00
Eelco Dolstra
9f4d8c6170 Pluggable fetchers
Flakes are now fetched using an extensible mechanism. Also lots of
other flake cleanups.
2020-01-21 22:56:04 +01:00
Eelco Dolstra
6fadb3fc03 Merge remote-tracking branch 'origin/master' into flakes 2020-01-21 21:18:52 +01:00
Eelco Dolstra
aef635da78 Fix derivation computation with __structuredAttrs and multiple outputs
Fixes

  error: derivation '/nix/store/klivma7r7h5lndb99f7xxmlh5whyayvg-zlib-1.2.11.drv' has incorrect output '/nix/store/fv98nnx5ykgbq8sqabilkgkbc4169q05-zlib-1.2.11-dev', should be '/nix/store/adm7pilzlj3z5k249s8b4wv3scprhzi1-zlib-1.2.11-dev'
2020-01-21 21:14:13 +01:00
Daiderd Jordan
8b3217f832
build: remove warning when in sandboxing test mode
Introduced in 66fccd5832, but somehow
breaks the secure-drv-outputs test.
2020-01-13 22:09:18 +01:00
John Ericson
d64ab5131c unbreak build without pch 2020-01-13 21:45:33 +01:00
Eelco Dolstra
bc22a7ee6a Fix use of uninitialized store path
Fixes 'building of '/nix/store/00000000000000000000000000000000-': ...'.
2020-01-06 22:20:10 +01:00
Eelco Dolstra
e2988f48a1
Merge pull request #3303 from LnL7/darwin-sandbox
build: fix sandboxing on darwin
2020-01-06 20:56:35 +01:00
Eelco Dolstra
1dc29df1d3 Merge remote-tracking branch 'origin/master' into flakes 2020-01-06 12:43:07 +01:00
Daiderd Jordan
66fccd5832
build: fix sandboxing on darwin
Starting ba87b08f85 getEnv now returns an
std::optional which means these getEnv() != "" conditions no longer happen
if the variables are not defined.
2020-01-05 20:23:52 +01:00
Daiderd Jordan
7d448bc966
build: fix path repairing when hash rewriting is required
Handle store path repairing on darwin when sandboxing is enabled. Unlike
on linux sandboxing on darwin still requires hash rewriting.
2020-01-04 20:25:25 +01:00
Daiderd Jordan
b33fefcb92
build: recover store path when replacing fails
This shouldn't happen in normal circumstances, but just in case
attempt to move the temporary path back if possible.
2020-01-04 20:24:27 +01:00
edef
c65a6fa86a passAsFile: leave out the hash prefix
Having a colon in the path may cause issues, and having the hash
function indicated isn't actually necessary. We now verify the path 
format in the tests to prevent regressions.
2020-01-02 23:56:06 +00:00
Puck Meerburg
515c0a263e passAsFile: hash the attribute name instead of numbering sequentially
This makes the paths consistent without relying on ordering.

Co-authored-by: edef <edef@edef.eu>
2020-01-02 22:56:03 +00:00
Graham Christensen
c502831a1d
exportReferencesGraph: support working
Before, we would get:

    [deploy@bastion:~]$ nix-store -r /nix/store/grfnl76cahwls0igd2by2pqv0dimi8h2-nixos-system-eris-19.09.20191213.03f3def.drv
    these derivations will be built:
      /nix/store/3ka4ihvwh6wsyhpd2qa9f59506mnxvx1-initrd-linux-4.19.88.drv
      /nix/store/ssxwmll7v21did1c8j027q0m8w6pg41i-unit-prometheus-alertmanager-irc-notifier.service.drv
      /nix/store/mvyvkj46ay7pp7b1znqbkck2mq98k0qd-unit-script-network-local-commands-start.drv
      /nix/store/vsl1y9mz38qfk6pyirjwnfzfggz5akg6-unit-network-local-commands.service.drv
      /nix/store/wi5ighfwwb83fdmav6z6n2fw6npm9ffl-unit-prometheus-hydra-exporter.service.drv
      /nix/store/x0qkv535n75pbl3xn6nn1w7qkrg9wwyg-unit-prometheus-packet-sd.service.drv
      /nix/store/lv491znsjxdf51xnfxh9ld7r1zg14d52-unit-script-packet-sd-env-key-pre-start.drv
      /nix/store/nw4nzlca49agsajvpibx7zg5b873gk9f-unit-script-packet-sd-env-key-start.drv
      /nix/store/x674wwabdwjrkhnykair4c8mpxa9532w-unit-packet-sd-env-key.service.drv
      /nix/store/ywivz64ilb1ywlv652pkixw3vxzfvgv8-unit-wireguard-wg0.service.drv
      /nix/store/v3b648293g3zl8pnn0m1345nvmyd8dwb-unit-script-acme-selfsigned-status.nixos.org-start.drv
      /nix/store/zci5d3zvr6fgdicz6k7jjka6lmx0v3g4-unit-acme-selfsigned-status.nixos.org.service.drv
      /nix/store/f6pwvnm63d0kw5df0v7sipd1rkhqxk5g-system-units.drv
      /nix/store/iax8071knxk9c7krpm9jqg0lcrawf4lc-etc.drv
      /nix/store/grfnl76cahwls0igd2by2pqv0dimi8h2-nixos-system-eris-19.09.20191213.03f3def.drv
    error: invalid file name 'closure-init-0' in 'exportReferencesGraph'

This was tough to debug, I didn't figure out which one was broken until I did:

    nix-store -r /nix/store/grfnl76cahwls0igd2by2pqv0dimi8h2-nixos-system-eris-19.09.20191213.03f3def.drv 2>&1 | grep  nix/store | xargs -n1 nix-store -r

and then looking at the remaining build graph:

    $ nix-store -r /nix/store/grfnl76cahwls0igd2by2pqv0dimi8h2-nixos-system-eris-19.09.20191213.03f3def.drv
    these derivations will be built:
      /nix/store/3ka4ihvwh6wsyhpd2qa9f59506mnxvx1-initrd-linux-4.19.88.drv
      /nix/store/grfnl76cahwls0igd2by2pqv0dimi8h2-nixos-system-eris-19.09.20191213.03f3def.drv
    error: invalid file name 'closure-init-0' in 'exportReferencesGraph'

and knowing the initrd build is before the system, then:

    $ nix show-derivation /nix/store/3ka4ihvwh6wsyhpd2qa9f59506mnxvx1-initrd-linux-4.19.88.drv
    {
      "/nix/store/3ka4ihvwh6wsyhpd2qa9f59506mnxvx1-initrd-linux-4.19.88.drv": {
        [...]
        "exportReferencesGraph": "closure-init-0 /nix/store/...-stage-1-init.sh closure-mdadm.conf-1 /nix/store/...-mdadm.conf closure-ubuntu.conf-2 ...",
        [...]
      }
    }

I then searched the repo for "in 'exportReferencesGraph'", found this
recently updated regex, and realized it was missing a "-".
2020-01-01 20:50:40 -05:00
Brian McKenna
d25923263e Disable use-sqlite-wal under WSL
Before:

    $ nix-channel --update
    unpacking channels...
    warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy (SQLITE_PROTOCOL)
    warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy (SQLITE_PROTOCOL)
    warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy (SQLITE_PROTOCOL)
    warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy (SQLITE_PROTOCOL)
    warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy (SQLITE_PROTOCOL)

After:

    $ inst/bin/nix-channel --update
    unpacking channels...
    created 1 symlinks in user environment

I've seen complaints that "sandbox" caused problems under WSL but I'm
having no problems. I think recent changes could have fixed the issue.
2019-12-21 08:14:19 +11:00
Eelco Dolstra
c7866733d7 Merge remote-tracking branch 'origin/master' into flakes 2019-12-20 12:45:58 +01:00
Eelco Dolstra
ee235e764c
Merge branch 'libarchive' of https://github.com/yorickvP/nix 2019-12-19 14:47:18 +01:00
Eelco Dolstra
4da1cd59ba Fix getting the timestamp of GitHub flakes 2019-12-18 17:19:30 +01:00
Eelco Dolstra
87873d0d65 Merge remote-tracking branch 'origin/master' into flakes 2019-12-18 14:25:25 +01:00
Eelco Dolstra
f8abbdd456 Add priority setting to stores
This allows overriding the priority of substituters, e.g.

  $ nix-store --store ~/my-nix/ -r /nix/store/df3m4da96d84ljzxx4mygfshm1p0r2n3-geeqie-1.4 \
    --substituters 'http://cache.nixos.org?priority=100 daemon?priority=10'

Fixes #3264.
2019-12-17 17:17:53 +01:00
Eelco Dolstra
ad6b738ed8 Merge remote-tracking branch 'origin/master' into flakes 2019-12-16 20:17:21 +01:00
Eelco Dolstra
54bf5ba422 nix-store -r: Handle symlinks to store paths
Fixes #3270.
2019-12-16 19:11:47 +01:00
Eelco Dolstra
e8aa2290ed Only install *.sb files on macOS 2019-12-13 14:42:55 +01:00
Eelco Dolstra
d1b238ec3c Simplify 2019-12-13 12:53:20 +01:00
Profpatsch
38b29fb72c libstore/ssh: Improve error message on failing execvp
If the `throw` is reached, this means that execvp into `ssh` wasn’t
successful. We can hint at a usual problem, which is a missing `ssh`
executable.

Test with:

```
env PATH= ./result/bin/nix-copy-closure --builders '' unusedhost
```

and the bash version with

```
env PATH= ./result/bin/nix-copy-closure --builders '' localhost
```
2019-12-12 15:32:17 +01:00
Eelco Dolstra
a045f93396 Temporarily revert to using 'tar' until we have gzip support 2019-12-12 15:07:31 +01:00
Eelco Dolstra
ecb3a1afa2 Merge remote-tracking branch 'origin/master' into flakes 2019-12-11 14:53:30 +01:00
Eelco Dolstra
bbe97dff8b Make the Store API more type-safe
Most functions now take a StorePath argument rather than a Path (which
is just an alias for std::string). The StorePath constructor ensures
that the path is syntactically correct (i.e. it looks like
<store-dir>/<base32-hash>-<name>). Similarly, functions like
buildPaths() now take a StorePathWithOutputs, rather than abusing Path
by adding a '!<outputs>' suffix.

Note that the StorePath type is implemented in Rust. This involves
some hackery to allow Rust values to be used directly in C++, via a
helper type whose destructor calls the Rust type's drop()
function. The main issue is the dynamic nature of C++ move semantics:
after we have moved a Rust value, we should not call the drop function
on the original value. So when we move a value, we set the original
value to bitwise zero, and the destructor only calls drop() if the
value is not bitwise zero. This should be sufficient for most types.

Also lots of minor cleanups to the C++ API to make it more modern
(e.g. using std::optional and std::string_view in some places).
2019-12-10 22:06:05 +01:00
Eelco Dolstra
61cc9f34d2 Remove UserLock self-lock check
This is no longer needed since we're not using POSIX locks anymore.
2019-12-09 23:57:33 +01:00
Yorick van Pelt
9ff5f6492f
libarchive proof of concept 2019-12-07 22:35:14 +07:00
Eelco Dolstra
048ef27326 Typo 2019-12-05 20:34:34 +01:00
Eelco Dolstra
47a937d512 Show hash mismatch warnings in SRI format
(cherry picked from commit 63c5c91cc0)
2019-12-05 20:32:25 +01:00
Eelco Dolstra
c3c23a52ee Merge remote-tracking branch 'origin/master' into flakes 2019-12-04 00:31:09 +01:00
Eelco Dolstra
ac2bc721d8 Merge remote-tracking branch 'origin/recursive-nix' 2019-12-02 12:34:46 +01:00
Brian Wignall
8737980e75 Fix typos 2019-11-30 19:04:14 -05:00
Eelco Dolstra
8918bae098 Drop remaining uses of external "tar"
Also, fetchGit now runs in O(1) memory since we pipe the output of
'git archive' directly into unpackTarball() (rather than first reading
it all into memory).
2019-11-26 22:07:28 +01:00
Eelco Dolstra
d33dd6e6c0 Move code around 2019-11-26 22:07:28 +01:00
Eelco Dolstra
b7fba16613 Move code around 2019-11-26 22:07:28 +01:00
Eelco Dolstra
f738cd4d97 More Rust FFI adventures
We can now convert Rust Errors to C++ exceptions. At the Rust->C++ FFI
boundary, Result<T, Error> will cause Error to be converted to and
thrown as a C++ exception.
2019-11-26 22:07:28 +01:00
Eelco Dolstra
8110b4ebb2 Rust cleanup 2019-11-26 22:07:28 +01:00
Eelco Dolstra
045708db43 Make <nix/unpack-channel.nix> a builtin builder
This was the last function using a shell script, so this allows us to
get rid of tar, coreutils, bash etc.
2019-11-26 22:07:28 +01:00
Eelco Dolstra
313106d549
Fix clang warnings 2019-11-26 21:07:44 +01:00
Eelco Dolstra
425991883a
Merge pull request #3141 from xbreak/nocafile
Downloader: Log configured CA file
2019-11-26 20:52:25 +01:00
Eelco Dolstra
7c8d7c17f8
Merge pull request #3144 from matthewbauer/fix-sandbox-fallback
Fix sandbox fallback settings
2019-11-26 20:51:52 +01:00
Eelco Dolstra
ec5e7b44ff
Simplify 2019-11-26 20:26:22 +01:00
Eelco Dolstra
96e1c39bb7
Merge branch 'repair-bad-links' of https://github.com/chkno/nix 2019-11-26 20:21:48 +01:00
Eelco Dolstra
c13193017f
Disallow empty store path names
Fixes #3239.
2019-11-26 20:12:15 +01:00
Eelco Dolstra
89db9353d7
Doh 2019-11-26 20:08:25 +01:00
Eelco Dolstra
1ec6e6e11e
Add feature to disable URL literals
E.g.

  $ nix-build '<nixpkgs>' -A hello --experimental-features no-url-literals
  error: URL literals are disabled, at /nix/store/vsjamkzh15r3c779q2711az826hqgvzr-nixpkgs-20.03pre194957.bef773ed53f/nixpkgs/pkgs/top-level/all-packages.nix:1236:11

Helps with implementing https://github.com/NixOS/rfcs/pull/45.
2019-11-26 19:48:34 +01:00
Eelco Dolstra
ba87b08f85
getEnv(): Return std::optional
This allows distinguishing between an empty value and no value.
2019-11-22 16:18:13 +01:00
Chuck
3e2c77d001 Check for and repair bad .links entries
A corrupt entry in .links prevents adding a fixed version of that file
to the store in any path.  The user experience is that corruption
present in the store 'spreads' to new paths added to the store:

(With store optimisation enabled)

1. A file in the store gets corrupted somehow (eg: filesystem bug).
2. The user tries to add a thing to the store which contains a good copy
   of the corrupted file.
3. The file being added to the store is hashed, found to match the bad
   .links entry, and is replaced by a link to the bad .links entry.
   (The .links entry's hash is not verified during add -- this would
   impose a substantial performance burden.)
4. The user observes that the thing in the store that is supposed to be
   a copy of what they were trying to add is not a correct copy -- some
   files have different contents!  Running "nix-store --verify
   --check-contents --repair" does not fix the problem.

This change makes "nix-store --verify --check-contents --repair" fix
this problem.  Bad .links entries are simply removed, allowing future
attempts to insert a good copy of the file to succeed.
2019-11-15 11:55:36 -08:00
Eelco Dolstra
5ee23c35b9
Merge pull request #3219 from Ericson2314/semicolons
Fix extra semicolons warnings
2019-11-11 12:13:51 +01:00
John Ericson
8669db1dcc Clean up semicolon and comma
Thanks @bhipple for catching!
2019-11-10 16:21:59 -05:00
John Ericson
96e6e680c1 Fix extra ; warnings involving MakeError 2019-11-10 11:24:47 -05:00
Eelco Dolstra
f730841db4
Merge remote-tracking branch 'origin/master' into flakes 2019-11-07 11:44:02 +01:00
Eelco Dolstra
04bf9acd22
Remove #include 2019-11-07 10:12:35 +01:00
Eelco Dolstra
39a2e166dd
Cleanup 2019-11-06 16:53:02 +01:00
Eelco Dolstra
69326f3637
Recursive Nix: Handle concurrent client connections 2019-11-06 00:55:03 +01:00
Eelco Dolstra
c119ab9db0
Enable recursive Nix using a feature
Derivations that want to use recursion should now set

  requiredSystemFeatures = [ "recursive-nix" ];

to make the daemon socket appear.

Also, Nix should be configured with "experimental-features =
recursive-nix".
2019-11-06 00:55:03 +01:00
Eelco Dolstra
c921074c19
RestrictedStore: Implement addToStore() 2019-11-06 00:55:03 +01:00
Eelco Dolstra
c4d7c76b64
Recursive Nix support
This allows Nix builders to call Nix to build derivations, with some
limitations.

Example:

  let nixpkgs = fetchTarball channel:nixos-18.03; in

  with import <nixpkgs> {};

  runCommand "foo"
    {
      buildInputs = [ nix jq ];
      NIX_PATH = "nixpkgs=${nixpkgs}";
    }
    ''
      hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })')

      $hello/bin/hello

      mkdir -p $out/bin
      ln -s $hello/bin/hello $out/bin/hello

      nix path-info -r --json $hello | jq .
    ''

This derivation makes a recursive Nix call to build GNU Hello and
symlinks it from its $out, i.e.

  # ll ./result/bin/
  lrwxrwxrwx 1 root root 63 Jan  1  1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello

  # nix-store -qR ./result
  /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131
  /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5
  /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo

This is implemented as follows:

* Before running the outer builder, Nix creates a Unix domain socket
  '.nix-socket' in the builder's temporary directory and sets
  $NIX_REMOTE to point to it. It starts a thread to process
  connections to this socket. (Thus you don't need to have nix-daemon
  running.)

* The daemon thread uses a wrapper store (RestrictedStore) to keep
  track of paths added through recursive Nix calls, to implement some
  restrictions (see below), and to do some censorship (e.g. for
  purity, queryPathInfo() won't return impure information such as
  signatures and timestamps).

* After the build finishes, the output paths are scanned for
  references to the paths added through recursive Nix calls (in
  addition to the inputs closure). Thus, in the example above, $out
  has a reference to $hello.

The main restriction on recursive Nix calls is that they cannot do
arbitrary substitutions. For example, doing

  nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10

is forbidden unless /nix/store/kmwd... is in the inputs closure or
previously built by a recursive Nix call. This is to prevent
irreproducible derivations that have hidden dependencies on
substituters or the current store contents. Building a derivation is
fine, however, and Nix will use substitutes if available. In other
words, the builder has to present proof that it knows how to build a
desired store path from scratch by constructing a derivation graph for
that path.

Probably we should also disallow instantiating/building fixed-output
derivations (specifically, those that access the network, but
currently we have no way to mark fixed-output derivations that don't
access the network). Otherwise sandboxed derivations can bypass
sandbox restrictions and access the network.

When sandboxing is enabled, we make paths appear in the sandbox of the
builder by entering the mount namespace of the builder and
bind-mounting each path. This is tricky because we do a pivot_root()
in the builder to change the root directory of its mount namespace,
and thus the host /nix/store is not visible in the mount namespace of
the builder. To get around this, just before doing pivot_root(), we
branch a second mount namespace that shares its /nix/store mountpoint
with the parent.

Recursive Nix currently doesn't work on macOS in sandboxed mode
(because we can't change the sandbox policy of a running build) and in
non-root mode (because setns() barfs).
2019-11-06 00:52:38 +01:00
Eelco Dolstra
b81d9d26f5
Merge remote-tracking branch 'origin/master' into flakes 2019-11-04 22:29:31 +01:00
Eelco Dolstra
6bff1aa46d
Merge pull request #3182 from bhipple/fixup/comments
Minor updates to inline comments
2019-10-31 14:14:35 +01:00
Benjamin Hipple
80d5ec6ff4 Minor updates to inline comments
Add missing docstring on InstallableCommand. Also, some of these were wrapped
when they're right next to a line longer than the unwrapped line, so we can just
unwrap them to save vertical space.
2019-10-31 05:56:37 -04:00
Will Dietz
0e9b72e097 minor: fix mismatch of struct/class forward decl of 'Source'
Fixes the following warning and the indicate potential issue:

src/libstore/worker-protocol.hh:66:1: warning: class 'Source' was previously declared as a struct; this is valid, but may result in linker errors
under the Microsoft C++ ABI [-Wmismatched-tags]

(cherry picked from commit 6e1bb04870b1b723282d32182af286646f13bf3c)
2019-10-30 14:39:01 -05:00
Eelco Dolstra
e5319a87ce
queryPathInfoUncached(): Return const ValidPathInfo 2019-10-29 13:53:04 +01:00
Eelco Dolstra
992a2ad475
Move addToStoreFromDump to Store 2019-10-29 13:38:24 +01:00
Eelco Dolstra
05819d013f
Don't create a Store in processConnection() 2019-10-29 13:36:19 +01:00
Eelco Dolstra
2d37e88319
Move most of the daemon implementation to libstore 2019-10-29 13:25:33 +01:00
Eelco Dolstra
95c727caef
Remove the check against concurrent builds in the same process 2019-10-29 12:43:20 +01:00
Eelco Dolstra
f7ce80f90a
Factor out linkOrCopy() 2019-10-27 18:19:13 +01:00
Eelco Dolstra
e30a0155d4 Add "nix profile remove" command 2019-10-22 13:06:32 +02:00
Eelco Dolstra
45b740c18b Use upstream json_fwd.hpp to speed up compilation 2019-10-21 22:11:21 +02:00
Eelco Dolstra
9a18f544ac Merge remote-tracking branch 'origin/master' into flakes 2019-10-21 18:48:21 +02:00
Eelco Dolstra
629b9b0049 Mark content-addressable paths with references as experimental 2019-10-21 18:05:31 +02:00
Eelco Dolstra
0abb3ad537 Allow content-addressable paths to have references
This adds a command 'nix make-content-addressable' that rewrites the
specified store paths into content-addressable paths. The advantage of
such paths is that 1) they can be imported without signatures; 2) they
can enable deduplication in cases where derivation changes do not
cause output changes (apart from store path hashes).

For example,

  $ nix make-content-addressable -r nixpkgs.cowsay
  rewrote '/nix/store/g1g31ah55xdia1jdqabv1imf6mcw0nb1-glibc-2.25-49' to '/nix/store/48jfj7bg78a8n4f2nhg269rgw1936vj4-glibc-2.25-49'
  ...
  rewrote '/nix/store/qbi6rzpk0bxjw8lw6azn2mc7ynnn455q-cowsay-3.03+dfsg1-16' to '/nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16'

We can then copy the resulting closure to another store without
signatures:

  $ nix copy --trusted-public-keys '' ---to ~/my-nix /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16

In order to support self-references in content-addressable paths,
these paths are hashed "modulo" self-references, meaning that
self-references are zeroed out during hashing. Somewhat annoyingly,
this means that the NAR hash stored in the Nix database is no longer
necessarily equal to the output of "nix hash-path"; for
content-addressable paths, you need to pass the --modulo flag:

  $ nix path-info --json /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16  | jq -r .[].narHash
  sha256:0ri611gdilz2c9rsibqhsipbfs9vwcqvs811a52i2bnkhv7w9mgw

  $ nix hash-path --type sha256 --base32 /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16
  1ggznh07khq0hz6id09pqws3a8q9pn03ya3c03nwck1kwq8rclzs

  $ nix hash-path --type sha256 --base32 /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16 --modulo iq6g2x4q62xp7y7493bibx0qn5w7xz67
  0ri611gdilz2c9rsibqhsipbfs9vwcqvs811a52i2bnkhv7w9mgw
2019-10-21 17:47:24 +02:00
Eelco Dolstra
b82f75464d buildenv: Eliminate global variables, other cleanup 2019-10-21 17:40:40 +02:00
Eelco Dolstra
a7aabd7cc7 Add getDefaultProfile() function 2019-10-21 16:07:19 +02:00
Eelco Dolstra
4a1cd10495 Merge remote-tracking branch 'origin/master' into flakes 2019-10-21 13:52:55 +02:00
Eelco Dolstra
aabf5c86c9
Add experimental-features setting
Experimental features are now opt-in. There is currently one
experimental feature: "nix-command" (which enables the "nix"
command. This will allow us to merge experimental features more
quickly, without committing to supporting them indefinitely.

Typical usage:

$ nix build --experimental-features 'nix-command flakes' nixpkgs#hello

(cherry picked from commit 8e478c2341,
without the "flakes" feature)
2019-10-21 13:34:44 +02:00
xbreak
7c568d4c6e Downloader: Warn if no trusted CA file has been configured 2019-10-18 19:08:33 +00:00
Eelco Dolstra
8e478c2341
Add experimental-features setting
Experimental features are now opt-in. There are currently two
experimental features: "nix-command" (which enables the "nix"
command), and "flakes" (which enables support for flakes). This will
allow us to merge experimental features more quickly, without
committing to supporting them indefinitely.

Typical usage:

$ nix build --experimental-features 'nix-command flakes' nixpkgs#hello
2019-10-16 17:49:01 +02:00
Matthew Bauer
96c84937c4 Move tmpDirInSandbox to initTmpDir 2019-10-13 16:41:49 -04:00
Matthew Bauer
499b038875 Fix sandbox fallback settings
The tmpDirInSandbox is different when in sandboxed vs. non-sandboxed.
Since we don’t know ahead of time here whether sandboxing is enabled,
we need to reset all of the env vars we’ve set previously. This fixes
the issue encountered in https://github.com/NixOS/nixpkgs/issues/70856.
2019-10-12 19:22:13 -04:00
Eelco Dolstra
906d56a96b
ssh-ng: Don't set CPU affinity on the remote
Fixes #3138.
2019-10-11 18:49:46 +02:00
Eelco Dolstra
e99bb91217
Merge remote-tracking branch 'origin/master' into flakes 2019-10-10 12:54:37 +02:00
Eelco Dolstra
bda64a2b0f
Doh
https://hydra.nixos.org/build/102803044
2019-10-10 00:12:30 +02:00
Eelco Dolstra
20eec802ff
Force per-user group to a known value 2019-10-09 23:35:02 +02:00
Eelco Dolstra
d7bae5680f
Go back to 755 permission on per-user directories
700 is pointless since the store is world-readable anyway. And
per-user/root/channels must be world-readable.
2019-10-09 23:35:02 +02:00
Eelco Dolstra
9348f9291e
nix-env: Create ~/.nix-profile automatically 2019-10-09 23:35:01 +02:00
Eelco Dolstra
5a303093dc
Remove world-writability from per-user directories
'nix-daemon' now creates subdirectories for users when they first
connect.

Fixes #509 (CVE-2019-17365).
Should also fix #3127.
2019-10-09 23:34:48 +02:00
Eelco Dolstra
99b73fb507
OCD performance fix: {find,count}+insert => insert 2019-10-09 16:06:29 +02:00
Matthew Bauer
199e888785 Handle empty sandbox_shell
Previously, SANDBOX_SHELL was set to empty when unavailable. This
caused issues when actually generating the sandbox. Instead, just set
SANDBOX_SHELL when --with-sandbox-shell= is non-empty. Alternative
implementation to https://github.com/NixOS/nix/pull/3038.
2019-10-08 23:12:54 -04:00
Eelco Dolstra
a323b7826c
Merge remote-tracking branch 'origin/master' into flakes 2019-10-04 17:26:32 +02:00
Eelco Dolstra
93b1ce1ac5
Revert "std::uncaught_exception() -> std::uncaught_exceptions()"
This reverts commit 6b83174fff because
it doesn't work on macOS yet.

https://hydra.nixos.org/build/102617587
2019-10-04 16:34:59 +02:00
Domen Kožar
2d2769f68c
Merge pull request #2338 from bobvanderlinden/pr-cannot-delete-alive-why
mention `nix-store --query --roots` when a path cannot be deleted
2019-09-30 14:06:52 +02:00
Eelco Dolstra
c32bba7489
Shut up some warnings 2019-09-24 17:28:18 +02:00
Eelco Dolstra
5038e1bec4
Merge pull request #3103 from bhipple/fix/spelling
Fix spelling in comment
2019-09-23 10:46:40 +02:00
Eelco Dolstra
02b4632e77
Merge pull request #3104 from zimbatm/no-show-trace-forwarding
libstore: don't forward --show-trace
2019-09-23 10:44:42 +02:00
Eelco Dolstra
3a022d4599 Shut up some warnings
(cherry picked from commit 99e8e58f2d)
2019-09-22 21:57:05 +02:00
Eelco Dolstra
bd79c1f6f6 Don't catch exceptions by value
(cherry picked from commit 893be6f5e3)
2019-09-22 21:56:56 +02:00
Eelco Dolstra
893be6f5e3 Don't catch exceptions by value 2019-09-22 21:29:33 +02:00
zimbatm
e63c9e73e3
libstore: don't forward --show-trace 2019-09-22 10:57:20 +00:00
Benjamin Hipple
c6a542f22a Fix spelling in comment 2019-09-21 18:53:15 -04:00
Eelco Dolstra
99e8e58f2d Shut up some warnings 2019-09-20 13:48:53 +02:00
Eelco Dolstra
092ee24627
Merge remote-tracking branch 'origin/master' into flakes 2019-09-18 12:02:50 +02:00
Eelco Dolstra
6b83174fff std::uncaught_exception() -> std::uncaught_exceptions()
The former is deprecated in C++17. Fixes a clang warning.
2019-09-13 20:05:44 +02:00
Eelco Dolstra
a25c022af3 Merge remote-tracking branch 'origin/master' into flakes 2019-09-13 19:52:03 +02:00
Eelco Dolstra
5dafde28db
BinaryCacheStore: Add index-debug-info option
This integrates the functionality of the index-debuginfo program in
nixos-channel-scripts to maintain an index of DWARF debuginfo files in
a format usable by dwarffs. Thus the debug info index is updated by
Hydra rather than by the channel mirroring script.

Example usage:

  $ nix copy --to 'file:///tmp/binary-cache?index-debug-info=true' /nix/store/vr9mhcch3fljzzkjld3kvkggvpq38cva-nix-2.2.2-debug

  $ cat /tmp/binary-cache/debuginfo/036b210b03bad75ab2d8fc80b7a146f98e7f1ecf.debug
  {"archive":"../nar/0313h2kdhk4v73xna9ysiksp2v8xrsk5xsw79mmwr3rg7byb4ka8.nar.xz","member":"lib/debug/.build-id/03/6b210b03bad75ab2d8fc80b7a146f98e7f1ecf.debug"}

Fixes #3083.
2019-09-04 19:28:26 +02:00
Eelco Dolstra
e302ba0e65
Merge remote-tracking branch 'origin/master' into flakes 2019-09-04 13:30:11 +02:00
Eelco Dolstra
e07ec8d27e
Support allowSubstitutes attribute in structured attribute derivations
Hopefully fixes #3081 (didn't test).
2019-09-03 16:03:49 +02:00
Eelco Dolstra
f186000367
Add some noexcepts
This is to assert that callback functions should never throw (since
the context in which they're called may not be able to handle the
exception).
2019-09-03 13:45:35 +02:00
Eelco Dolstra
7348653ff4
Ensure that Callback is called only once
Also, make Callback movable but uncopyable.
2019-09-03 13:45:35 +02:00
Eelco Dolstra
8c4ea7a451
Downloader: Remove a possible double call to Callback 2019-09-03 13:45:32 +02:00
Eelco Dolstra
ebc4dae517 Merge remote-tracking branch 'origin/master' into flakes 2019-08-29 16:11:38 +02:00
Eelco Dolstra
84de821004
Merge pull request #3069 from matthewbauer/max-name
Set maximum name length in Nix
2019-08-29 15:22:36 +02:00
Eelco Dolstra
a2c4fcd5e9 Don't rely on st_blocks
It doesn't seem very reliable on ZFS.
2019-08-29 14:49:58 +02:00
Eelco Dolstra
f27e53f77e Cleanup 2019-08-29 12:09:58 +02:00
Matthew Bauer
693e68e09c Set maximum name length in Nix
Previously we allowed any length of name for Nix derivations. This is
bad because different file systems have different max lengths. To make
things predictable, I have picked a max. This was done by trying to
build this derivation:

  derivation {
    name = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
    builder = "/no-such-path";
    system = "x86_64-linux";
  }

Take off one a and it will not lead to file name too long. That ends
up being 212 a’s. An even smaller max could be picked if we want to
support more file systems.

Working backwards, this is why:

aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-${name}.drv.chroot

> 255 - 32 - 1 - 4 - 7 = 211
2019-08-28 12:32:54 -04:00
Eelco Dolstra
7298a38a07
Don't send certain setting overrides to the daemon
These are already handled separately. This fixes warnings like

  warning: ignoring the user-specified setting 'max-jobs', because it is a restricted setting and you are not a trusted user

when using the -j flag.
2019-08-28 16:29:44 +02:00
Eelco Dolstra
e5b397b2c7 Merge branch 'test-sandboxing' of https://github.com/matthewbauer/nix 2019-08-27 20:58:47 +02:00
Matthew Bauer
5c06a8d328 Reset tmpDirInSandbox for unsandboxed 2019-08-23 20:24:39 -04:00
Eelco Dolstra
35ebae198f
Merge pull request #3031 from grahamc/low-speed-limit
conf: stalled-download-timeout: make tunable
2019-08-08 22:06:26 +02:00
Graham Christensen
a02457db71
conf: stalled-download-timeout: make tunable
Make curl's low speed limit configurable via stalled-download-timeout.
Before, this limit was five minutes without receiving a single byte.
This is much too long as if the remote end may not have even
acknowledged the HTTP request.
2019-08-08 10:22:13 -04:00
Eelco Dolstra
1d750e0587
Merge remote-tracking branch 'origin/master' into flakes 2019-08-08 15:49:13 +02:00
Eelco Dolstra
f9021c4c6c
Merge pull request #3030 from dtzWill/fix/missing-include-ocloexec
pathlocks: add include to fcntl.h for O_CLOEXEC
2019-08-07 22:03:09 +02:00
Eelco Dolstra
56df30cd3f
Merge pull request #2995 from tweag/post-build-hook
Add a post build hook
2019-08-07 15:02:29 +02:00
Will Dietz
c3fefd1a6e
pathlocks: add include to fcntl.h for O_CLOEXEC 2019-08-07 07:41:22 -05:00
Eelco Dolstra
399b6f3c46
nix-store --verify: Don't repair while holding the GC lock 2019-08-02 18:48:26 +02:00
Eelco Dolstra
a2597d5f27
Simplify
With BSD locks we don't have to guard against reading our own
temproots.
2019-08-02 18:39:16 +02:00
Eelco Dolstra
e349f2c0a3
Use BSD instead of POSIX file locks
POSIX file locks are essentially incompatible with multithreading. BSD
locks have much saner semantics. We need this now that there can be
multiple concurrent LocalStore::buildPaths() invocations.
2019-08-02 18:39:16 +02:00
Eelco Dolstra
ec415d7166
Add a test for auto-GC
This currently fails because we're using POSIX file locks. So when the
garbage collector opens and closes its own temproots file, it causes
the lock to be released and then deleted by another GC instance.
2019-08-02 18:39:16 +02:00
regnat
7c5596734f
Add a post-build-hook
Passing `--post-build-hook /foo/bar` to a nix-* command will cause
`/foo/bar` to be executed after each build with the following
environment variables set:

    DRV_PATH=/nix/store/drv-that-has-been-built.drv
    OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev

This can be useful in particular to upload all the builded artifacts to
the cache (including the ones that don't appear in the runtime closure
of the final derivation or are built because of IFD).

This new feature prints the stderr/stdout output to the `nix-build`
and `nix build` client, and the output is printed in a Nix 2
compatible format:

    [nix]$ ./inst/bin/nix-build ./test.nix
    these derivations will be built:
      /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv
    building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'...
    hello!
    bye!
    running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'...
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + printf 'very important stuff'
    /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation

    [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix
    my-example-derivation> hello!
    my-example-derivation> bye!
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + printf 'very important stuff'
    [1 built, 0.0 MiB DL]

Co-authored-by: Graham Christensen <graham@grahamc.com>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-08-02 10:48:15 -04:00
Matthew Bauer
9a0855bbb6 Don’t rely on EPERM
startProcess does not appear to send the exit code to the helper
correctly. Not sure why this is, but it is probably safe to just
fallback on all sandbox errors.
2019-07-30 17:53:37 -04:00
Bas van Dijk
ee1e3132ca Disable findRuntimeRoots on darwin when running tests because lsof is slow
See: https://github.com/NixOS/nix/issues/3011
2019-07-30 11:29:03 +02:00
Matthew Bauer
11d8534629 Use sandbox fallback when cloning fails in builder
When sandbox-fallback = true (the default), the Nix builder will fall
back to disabled sandbox mode when the kernel doesn’t allow users to
set it up. This prevents hard errors from occuring in tricky places,
especially the initial installer. To restore the previous behavior,
users can set:

  sandbox-fallback = false

in their /etc/nix/nix.conf configuration.
2019-07-25 14:42:30 -04:00
Matthew Bauer
d171090530 Disable CLONE_NEWUSER when it’s unavailable
Some kernels disable "unpriveleged user namespaces". This is
unfortunate, but we can still use mount namespaces. Anyway, since each
builder has its own nixbld user, we already have most of the benefits
of user namespaces.
2019-07-25 14:42:25 -04:00
Eelco Dolstra
2f853b20df
Merge pull request #2975 from matthewbauer/fix-nsswitch-issue
Don’t use entire /etc/nsswitch.conf file
2019-07-13 17:08:02 +02:00
Eelco Dolstra
4205234f26
Merge remote-tracking branch 'origin/master' into flakes 2019-07-11 13:13:19 +02:00
Eelco Dolstra
53247d6b11
Resume NAR downloads
This is a much simpler fix to the 'error 9 while decompressing xz
file' problem than 78fa47a7f0. We just
do a ranged HTTP request starting after the data that we previously
wrote into the sink.

Fixes #2952, #379.
2019-07-10 23:12:17 +02:00
Eelco Dolstra
00f6fafad6
HttpBinaryCacheStore: Use default number of retries for NARs 2019-07-10 23:05:04 +02:00
Eelco Dolstra
f76b2a7fdd
Downloader: Use warn() 2019-07-10 22:27:50 +02:00
Eelco Dolstra
03f09e1d18
Revert "Fix 'error 9 while decompressing xz file'"
This reverts commit 78fa47a7f0.
2019-07-10 19:46:15 +02:00
Eelco Dolstra
cc218b15ba Merge remote-tracking branch 'origin/master' into flakes 2019-07-06 21:06:22 +02:00
Eelco Dolstra
7e1c85c5fb
Merge pull request #2779 from LnL7/build-exit-codes
build: add exit code for hash and check mismatches
2019-07-02 17:37:49 +02:00
Eelco Dolstra
db700f730e
Merge pull request #2974 from grahamc/invalid-name
checkStoreName: give more precise/verbose error information
2019-07-02 16:12:01 +02:00
Graham Christensen
17d3ec3405
checkStoreName: give more precise/verbose error information
$ sudo ./inst/bin/nix-instantiate -E '"${./.git}"'
error: The path name '.git' is invalid: it is illegal to start the
name with a period. Path names are alphanumeric and can include the
symbols +-._?= and must not begin with a period. Note: If '.git' is a
source file and you cannot rename it on disk,
builtins.path { name = ... } can be used to give it an alternative
name.
2019-07-02 08:41:53 -04:00
Daiderd Jordan
a52c331edb
build: replace 100 offset for build exit codes 2019-07-02 00:12:38 +02:00
Daiderd Jordan
cbf84bcce7
build: use binary mask for build status flags
If multiple builds with fail with different errors it will be reflected
in the status code.

eg.

	103 => timeout + hash mismatch
	105 => timeout + check mismatch
	106 => hash mismatch + check mismatch
	107 => timeout + hash mismatch + check mismatch
2019-07-02 00:12:38 +02:00
Daiderd Jordan
97baf32fbc
build: add exit code for hash and check mismatches
Makes it easier to identify the failure reason in other tooling, eg.
differentiate between a non-deterministic --check vs a failed build.

	$ nix-build '<nix/fetchurl.nix>' --argstr url http://example.org --argstr sha256 0000000000000000000000000000000000000000000000000000
	hash mismatch in fixed-output derivation '/nix/store/nzi9ck45rwlxzcwr25is7qlf3hs5xl83-example.org':
	  wanted: sha256:0000000000000000000000000000000000000000000000000000
	  got:    sha256:08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm
	$ echo $?
	102

	$ nix-build -E 'with import <nixpkgs> {}; runCommand "foo" {} "date +%s > $out"' --check
	warning: rewriting hashes in '/nix/store/g3k47g0399fvjmbm0p0mnad74k4w8vkz-foo'; cross fingers
	error: derivation '/nix/store/mggc8dz13ackb49qca6m23zq4fpq132q-foo.drv' may not be deterministic: output '/nix/store/g3k47g0399fvjmbm0p0mnad74k4w8vkz-foo' differs
	$ echo $?
	104
2019-07-02 00:12:34 +02:00
Aniket Deshpande
ec58ba38c5
Fix http2 = false having no effect. Fixes #2971.
Setting `http2 = false` in nix config (e.g. /etc/nix/nix.conf)
had no effect, and `nix-env -vvvvv -i hello` still downloaded .nar
packages using HTTP/2.

In `src/libstore/download.cc`, the `CURL_HTTP_VERSION_2TLS` option was
being explicitly set when `downloadSettings.enableHttp2` was `true`,
but, `CURL_HTTP_VERSION_1_1` option was not being explicitly set when
`downloadSettings.enableHttp2` was `false`.

This may be because `https://curl.haxx.se/libcurl/c/libcurl-env.html` states:
"You have to set this option if you want to use libcurl's HTTP/2 support."
but, also, in the changelog, states:
"DEFAULT
Since curl 7.62.0: CURL_HTTP_VERSION_2TLS
Before that: CURL_HTTP_VERSION_1_1"

So, the default setting for `libcurl` is HTTP/2 for version >= 7.62.0.

In this commit, option `CURLOPT_HTTP_VERSION` is explicitly set to
`CURL_HTTP_VERSION_1_1` when `downloadSettings.enableHttp2` nix config
setting is `false`.

This can be tested by running `nix-env -vvvvv -i hello | grep HTTP`
2019-06-28 20:44:46 +05:30
Matthew Bauer
ec0087df0a Don’t use entire /etc/nsswitch.conf file
The default nsswitch.conf(5) file in most distros can handle many
different things including host name, user names, groups, etc. In Nix,
we want to limit the amount of impurities that come from these things.
As a result, we should only allow nss to be used for gethostbyname(3)
and getservent(3).

/cc @Ericson2314
2019-06-27 14:31:29 -04:00
Eelco Dolstra
2cc248c4fd
Merge remote-tracking branch 'origin/master' into flakes 2019-06-25 14:27:57 +02:00
Eelco Dolstra
88571219d9
nix-channel: Don't fetch binary-cache-url
This has been ignored since the Perl->C++ rewrite.
2019-06-25 13:27:16 +02:00
Eelco Dolstra
64ec087f58
Fix 32-bit overflow with --no-net
--no-net causes tarballTtl to be set to the largest 32-bit integer,
which causes comparison like 'time + tarballTtl < other_time' to
fail on 32-bit systems. So cast them to 64-bit first.

https://hydra.nixos.org/build/95076624
(cherry picked from commit 29ccb2e969)
2019-06-24 22:16:43 +02:00
Eelco Dolstra
f8b30338ac
Refactor downloadCached() interface
(cherry picked from commit df3f5a78d5)
2019-06-24 22:12:26 +02:00
Eelco Dolstra
7b9c68766d
Add '--no-net' convenience flag
This flag

* Disables substituters.

* Sets the tarball-ttl to infinity (ensuring e.g. that the flake
  registry and any downloaded flakes are considered current).

* Disables retrying downloads and sets the connection timeout to the
  minimum. (So it doesn't completely disable downloads at the moment.)

(cherry picked from commit 8ea842260b)
2019-06-24 22:07:29 +02:00
Eelco Dolstra
78fa47a7f0
Fix 'error 9 while decompressing xz file'
Once we've started writing data to a Sink, we can't restart a download
request, because then we end up writing duplicate data to the
Sink. Therefore we shouldn't handle retries in Downloader but at a
higher level (in particular, in copyStorePath()).

Fixes #2952.

(cherry picked from commit a67cf5a358)
2019-06-24 21:59:51 +02:00
Eelco Dolstra
2fef4dd296
Downloader: Propagate exceptions from decompressionSink->finish()
(cherry picked from commit 15fa70cd1b)
2019-06-24 21:59:31 +02:00
Eelco Dolstra
b43e1e186e
CachedDownloadResult: Include store path
Also, make fetchGit and fetchMercurial update allowedPaths properly.

(Maybe the evaluator, rather than the caller of the evaluator, should
apply toRealPath(), but that's a bigger change.)

(cherry picked from commit 5c34d66538)
2019-06-24 21:59:27 +02:00
Eelco Dolstra
dc29e9fb47
downloadCached: Return ETag
(cherry picked from commit 529add316c)
2019-06-24 21:58:33 +02:00
Eelco Dolstra
a67cf5a358
Fix 'error 9 while decompressing xz file'
Once we've started writing data to a Sink, we can't restart a download
request, because then we end up writing duplicate data to the
Sink. Therefore we shouldn't handle retries in Downloader but at a
higher level (in particular, in copyStorePath()).

Fixes #2952.
2019-06-24 21:50:08 +02:00
Eelco Dolstra
15fa70cd1b
Downloader: Propagate exceptions from decompressionSink->finish() 2019-06-24 21:06:37 +02:00
Eelco Dolstra
d132d057a8
Handle store symlinks in flake directories
E.g. 'nix path-info ./result' inside a flake directory now works
again.
2019-06-21 15:29:05 +02:00
Eelco Dolstra
29ccb2e969
Fix 32-bit overflow with --no-net
--no-net causes tarballTtl to be set to the largest 32-bit integer,
which causes comparison like 'time + tarballTtl < other_time' to
fail on 32-bit systems. So cast them to 64-bit first.

https://hydra.nixos.org/build/95076624
2019-06-21 12:53:52 +02:00
Eelco Dolstra
3871131308
Merge pull request #2949 from NixOS/no-net
Add --no-net flag
2019-06-18 15:59:42 +02:00
Eelco Dolstra
d6c4fe55db
Merge remote-tracking branch 'origin/master' into flakes 2019-06-17 14:45:12 +02:00
Eelco Dolstra
8ea842260b
Add '--no-net' convenience flag
This flag

* Disables substituters.

* Sets the tarball-ttl to infinity (ensuring e.g. that the flake
  registry and any downloaded flakes are considered current).

* Disables retrying downloads and sets the connection timeout to the
  minimum. (So it doesn't completely disable downloads at the moment.)
2019-06-17 08:43:45 +02:00
Eelco Dolstra
3cc1125595
Another attempt at getting pseudoterminals to work on macOS 2019-06-17 08:08:04 +02:00
Eelco Dolstra
2743bf0bb1 Hopefully fix macOS tests 2019-06-16 20:02:40 +02:00
Eelco Dolstra
82ca6ef390 Set $TERM 2019-06-16 17:36:30 +02:00
Eelco Dolstra
e84c265645 Run builds in a pseudo-terminal
This allows many programs (e.g. gcc, clang, cmake) to print colorized
log output (assuming $TERM is set to a value like "xterm").

There are other ways to get colors, in particular setting
CLICOLOR_FORCE, but they're less widely supported and can break
programs that parse tool output.
2019-06-16 17:36:30 +02:00
Eelco Dolstra
6644b6099b
Add flake evaluation cache
This exploits the hermetic nature of flake evaluation to speed up
repeated evaluations of a flake output attribute.

For example (doing 'nix build' on an already present package):

  $ time nix build nixpkgs:firefox

  real    0m1.497s
  user    0m1.160s
  sys     0m0.139s

  $ time nix build nixpkgs:firefox

  real    0m0.052s
  user    0m0.038s
  sys     0m0.007s

The cache is ~/.cache/nix/eval-cache-v1.sqlite, which has entries like

  INSERT INTO Attributes VALUES(
    X'92a907d4efe933af2a46959b082cdff176aa5bfeb47a98fabd234809a67ab195',
    'packages.firefox',
    1,
    '/nix/store/pbalzf8x19hckr8cwdv62rd6g0lqgc38-firefox-67.0.drv /nix/store/g6q0gx0v6xvdnizp8lrcw7c4gdkzana0-firefox-67.0 out');

where the hash 92a9... is a fingerprint over the flake store path and
the contents of the lockfile. Because flakes are evaluated in pure
mode, this uniquely identifies the evaluation result.
2019-06-07 22:25:48 +02:00
Eelco Dolstra
a2f86ac647
Merge remote-tracking branch 'origin/master' into flakes 2019-06-03 09:22:19 +02:00
Eelco Dolstra
aec545c20b Fix segfault in builtin fetchurl with hashed mirrors + SRI hashes 2019-06-01 15:27:43 +02:00
Eelco Dolstra
63c5c91cc0
Show hash mismatch warnings in SRI format 2019-05-31 18:48:28 +02:00
Eelco Dolstra
ae7b56cd9a
Get last commit time of github flakes 2019-05-29 10:10:40 +02:00
Eelco Dolstra
f0d6d67af9 Prevent the global registry from being GC'ed
Issue #2868.
2019-05-22 23:43:58 +02:00
Eelco Dolstra
df3f5a78d5 Refactor downloadCached() interface 2019-05-22 23:36:29 +02:00
Eelco Dolstra
b531695331
Merge pull request #96 from tweag/support-chroot-store
Make flakes work with 'nix build --store ...'
2019-05-16 10:51:40 +02:00
Eelco Dolstra
0f5032c5a4 Merge remote-tracking branch 'origin/master' into flakes 2019-05-15 20:51:29 +02:00
Eelco Dolstra
5c34d66538 Make flakes work with 'nix build --store ...'
It was getting confused between logical and real store paths.

Also, make fetchGit and fetchMercurial update allowedPaths properly.

(Maybe the evaluator, rather than the caller of the evaluator, should
apply toRealPath(), but that's a bigger change.)
2019-05-15 15:38:24 +02:00
Eelco Dolstra
5f6840fbb4
Merge pull request #2802 from LnL7/fix-needs-hashrewrite
build: move needsHashRewrite initialization to startBuilder
2019-05-13 08:53:48 +02:00
Daiderd Jordan
ce02fc74b2
build: make needsHashRewrite a method 2019-05-12 22:56:27 +02:00
Graham Christensen
73b797c207
handleDiffHook: stop passing allowVfork 2019-05-12 13:44:22 -04:00
Graham Christensen
b4a05edbfe
runProgram: support gid, uid, chdir 2019-05-12 13:17:27 -04:00
Graham Christensen
dde8eeb39a
chdir, setgroups 2019-05-12 13:17:26 -04:00
Graham Christensen
6df61db060
diff hook: execute as the build user, and pass the temp dir 2019-05-12 13:17:26 -04:00
Graham Christensen
c78686e411
build: run diff-hook under --check and document diff-hook 2019-05-12 13:17:24 -04:00
Daiderd Jordan
ff6867ab94
build: move needsHashRewrite initialization to startBuilder
The value of useChroot is not set yet in the constructor, resulting in
hash rewriting being enabled in certain cases where it should not be.

Fixes #2801
2019-05-12 15:53:40 +02:00
Eelco Dolstra
2bc55aba1e
Merge remote-tracking branch 'origin/master' into flakes 2019-05-08 14:30:27 +02:00
Eelco Dolstra
3e940bbf2d
Merge pull request #2790 from samueldr/fix/minor-comment-NIX_ROOT_FINDER
findRootsNoTemp: fixes comment about findRuntimeRoots
2019-05-08 10:13:09 +02:00
Eelco Dolstra
f9a2ea4486
Fix "Bad system call" running i686-linux binaries on x86_64-linux
To determine which seccomp filters to install, we were incorrectly
using settings.thisSystem, which doesn't denote the actual system when
--system is used.

Fixes #2791.
2019-05-03 10:48:33 +02:00
Eelco Dolstra
7dcf5b011a Add function for quoting strings 2019-05-02 21:09:52 +02:00
Samuel Dionne-Riel
cbc7d9a412 findRootsNoTemp: fixes comment about findRuntimeRoots
The NIX_ROOT_FINDER environment variable was removed in
3c46fe62b8 when porting from perl to C.
2019-04-30 22:43:24 -04:00
Eelco Dolstra
0cbda84f5b exportGit: Don't clone local repositories
This ensures that commands like 'nix flake info /my/nixpkgs' don't
copy a gigabyte of crap to ~/.cache/nix.

Fixes #60.
2019-04-19 14:10:57 +02:00
Daiderd Jordan
b614e0e53d
build: only skip hash rewriting for sandboxing on linux
The sandbox on darwin, and possibly other future platforms, doesn't have
enough isolation to redirect outputs without hash rewriting.
2019-04-17 23:41:23 +02:00
Nick Van den Broeck
b3d33b02e3 Added support for private github repositories 2019-04-16 08:14:46 +02:00
Eelco Dolstra
9b7eac332b Merge remote-tracking branch 'origin/master' into flakes 2019-04-08 14:23:48 +02:00
Eelco Dolstra
41ba5135e0 primeCache(): Barf if builds are needed but not allowed
Fixes #2753.
2019-04-01 21:09:49 +02:00
Eelco Dolstra
2bc6304793 getMachines(): Cache result 2019-04-01 21:09:35 +02:00
Eelco Dolstra
b5565a7081 Merge remote-tracking branch 'origin/master' into flakes 2019-03-26 11:44:14 +01:00
Will Dietz
42e2d5e7b7
store-api.hh: add missing include for unordered_map 2019-03-25 09:18:30 +01:00
Chaker Benhamed
81a23fa7e2 Add a 5 seconds of timeouts to connect to S3.
The default is 1000ms, but we can hit it a lot of we don't have direct
link to AWS (e.g. using VPN).
2019-03-15 13:23:58 +01:00
Eelco Dolstra
1f64f4c7c8
pkg-config files: Use c++17 2019-03-14 14:11:12 +01:00
Eelco Dolstra
ef52ccf035
experimental/optional -> optional 2019-03-14 14:10:52 +01:00
Eelco Dolstra
53522cb6ac
findRoots(): Add 'censor' parameter
This is less brittle than filtering paths after the fact in
nix-daemon.
2019-03-14 13:53:34 +01:00
Eelco Dolstra
a3f37d87ea
findRuntimeRoots: Simplify/fix handling of /proc files
Scanning of /proc/<pid>/{exe,cwd} was broken because '{memory:' was
prepended twice. Also, get rid of the whole '{memory:...}' thing
because it's unnecessary, we can just list the file in /proc directly.
2019-03-14 13:30:25 +01:00
Guillaume Maudoux
38ee16ae9c Unify internal findRootsXxx() api 2019-03-10 00:56:09 +01:00
Guillaume Maudoux
ebc86550f9 Make roots a map of store paths to pinning links
This new structure makes more sense as there may be many sources rooting
the same store path. Many profiles can reference the same path but this
is even more true with /proc/<pid>/maps where distinct pids can and
often do map the same store path.
This implementation is also more efficient as the `Roots` map contains
only one entry per rooted store path.
2019-03-10 00:56:09 +01:00
Guillaume Maudoux
a17f86ce3a Obfuscate memory roots for non-root users 2019-03-10 00:56:09 +01:00
Guillaume Maudoux
43331d6344 detail memory roots 2019-03-10 00:56:09 +01:00
Dzmitry Zaitsau
06d6335987 fix indentation 2019-02-26 11:07:37 +01:00
Dzmitry Zaitsau
ac200c3678 Apply param values on S3Helper initialization 2019-02-25 18:06:19 +01:00
Dzmitry Zaitsau
56c18c67d9 Extend S3 URL parsing with parameters extraction 2019-02-25 18:00:55 +01:00
Dzmitry Zaitsau
07f992a74b Extract and expose splitUriAndParams function
which splits a URL into localtor and parameter parts
2019-02-25 17:59:26 +01:00
Eelco Dolstra
529add316c
downloadCached: Return ETag
This allows fetchFlake() to return the Git revision of a GitHub
archive.
2019-02-25 23:20:50 +08:00
Eelco Dolstra
0cd7f2cd8d
pkg-config files: Use c++17 2019-02-12 13:44:20 +01:00
Eelco Dolstra
c8a0b9d5cb
experimental/optional -> optional 2019-02-12 13:43:32 +01:00
Eelco Dolstra
01d07b1e92
Revert "Restore parent mount namespace before executing a child process"
This reverts commit a0ef21262f. This
doesn't work in 'nix run' and nix-shell because setns() fails in
multithreaded programs, and Boehm GC mark threads are uncancellable.

Fixes #2646.
2019-02-05 10:49:19 +01:00
Eelco Dolstra
7cc1a2593e
unsupported(): Show the name of the unsupported operation 2019-01-18 13:34:53 +01:00
John Ericson
fef9f5653b Remove mentions of libformat, it no longer exists 2019-01-05 14:31:29 -05:00
zimbatm
5e6fa9092f
libstore: improve hash mismatch error messages
Use the same output ordering and format everywhere.

This is such a common issue that we trade the single-line error message for
more readability.

Old message:
```
fixed-output derivation produced path '/nix/store/d4nw9x2sy9q3r32f3g5l5h1k833c01vq-example.com' with sha256 hash '08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm' instead of the expected hash '1xzwnipjd54wl8g93vpw6hxnpmdabq0wqywriiwmh7x8k0lvpq5m'
```

New message:
```
hash mismatch in fixed-output derivation '/nix/store/d4nw9x2sy9q3r32f3g5l5h1k833c01vq-example.com':
  wanted: sha256:1xzwnipjd54wl8g93vpw6hxnpmdabq0wqywriiwmh7x8k0lvpq5m
  got:    sha256:08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm

```
2018-12-12 17:42:00 +01:00
Eelco Dolstra
f7425d55df
Fix assertion failure in NarInfoDiskCache
https://hydra.nixos.org/build/85827920
2018-12-12 17:13:22 +01:00
Daiderd Jordan
6f89053108
nar-info-disk-cache: include ca in the cache entries
Without this information the content addressable state and hashes are
lost after the first request, this causes signatures to be required for
everything even tho the path could be verified without signing.
2018-12-12 01:08:47 +01:00
Daiderd Jordan
898823b67d
s3: make scheme configurable
This enables using for http for S3 request for debugging or
implementations that don't have https configured.  This is not a problem
for binary caches since they should not contain sensitive information.
Both package signatures and AWS auth already protect against tampering.
2018-12-07 23:55:38 +01:00
Eelco Dolstra
ebeea068d5
Merge pull request #2216 from dtzWill/fix/curl-760-timeout
download: if there are active requests, never sleep for 10s
2018-11-20 19:07:56 +01:00
Eelco Dolstra
a32ff2573b
Fix 'Read-only file system' when building a derivation 2018-11-15 13:20:39 +01:00
Eelco Dolstra
a0ef21262f
Restore parent mount namespace before executing a child process
This ensures that they can't write to /nix/store. Fixes #2535.
2018-11-13 16:15:30 +01:00
Eelco Dolstra
812e39313c
Enable sandboxing by default
Closes #179.
2018-11-07 17:08:28 +01:00
Eelco Dolstra
6323b0729a
Disable the S3 content-encoding patch
Since we're not using multi-part uploads at the moment, we can drop
this patch.
2018-11-01 15:18:38 +01:00
Eelco Dolstra
fb35aaa422
Merge pull request #2501 from mayflower/xdg_config_dirs
config: use all of XDG_CONFIG_DIRS
2018-10-31 12:37:23 +01:00
Linus Heckemann
f3b8173a93 config: use all of XDG_CONFIG_DIRS
Previously, config would only be read from XDG_CONFIG_HOME. This change
allows reading config from additional directories, which enables e.g.
per-project binary caches or chroot stores with the help of direnv.
2018-10-31 09:24:43 +01:00
Eelco Dolstra
38d76d6d78
Remove redundant message 2018-10-30 19:28:42 +01:00
Eelco Dolstra
9f99d62480
S3BinaryCacheStore: Allow disabling multipart uploads
The use of TransferManager has several issues, including that it
doesn't allow setting a Content-Encoding without a patch, and it
doesn't handle exceptions in worker threads (causing termination on
memory allocation failure).

Fixes #2493.
2018-10-30 14:25:00 +01:00
Eelco Dolstra
0163e8928c
Fix broken uploadProgressCallback closure
Since the callback is global we can't refer to 'path' in it. This
could cause a segfault or printing of arbitrary data.
2018-10-30 11:45:31 +01:00
Falco Peijnenburg
49e272f647 copyStorePath: Fix hash errors when copying from older store
This commit partially reverts 48662d151b. When
copying from an older store (in my case a store running Nix 1.11.7), nix would
throw errors about there being no hash. This is fixed by recalculating the hash.
2018-10-29 20:24:37 +01:00
Eelco Dolstra
18b4c53f71
Restore old (dis)allowedRequisites behaviour for self-references
stdenv relies on this. So ignore self-references (but only in legacy non-structured attributes mode).
2018-10-27 15:41:53 +02:00
Eelco Dolstra
63575ffa38
Merge branch 'nix-doctor' of https://github.com/LnL7/nix 2018-10-27 12:54:22 +02:00
Will Dietz
3283c0dc45 remote-store.hh: ConnectionHandle is struct, minor fix warning 2018-10-26 13:15:48 -05:00
Eelco Dolstra
c47e14ee45
Merge pull request #2483 from mayflower/tail-config
Promote verbose-build and log-lines
2018-10-23 16:59:52 +02:00
Linus Heckemann
a25abe823f Promote log-lines to a fully-qualified option
This allows commands like

nix build --log-lines 30 nixpkgs.hello

in order to obtain more information in case of a failure.
2018-10-23 16:54:24 +02:00
Eelco Dolstra
3cd15c5b1f
Per-output reference and closure size checks
In structured-attributes derivations, you can now specify per-output
checks such as:

  outputChecks."out" = {
    # The closure of 'out' must not be larger than 256 MiB.
    maxClosureSize = 256 * 1024 * 1024;

    # It must not refer to C compiler or to the 'dev' output.
    disallowedRequisites = [ stdenv.cc "dev" ];
  };

  outputChecks."dev" = {
    # The 'dev' output must not be larger than 128 KiB.
    maxSize = 128 * 1024;
  };

Also fixed a bug in allowedRequisites that caused it to ignore
self-references.
2018-10-23 01:29:16 +02:00
Eelco Dolstra
bd78544f66 Fix assertion failure in Store::queryPathInfo()
$ nix-store -qR /nix/store/fnord
  nix-store: src/libstore/store-api.cc:80: std::__cxx11::string nix::storePathToHash(const Path&): Assertion `base.size() >= storePathHashLen' failed.
  Aborted
2018-10-16 23:39:36 +02:00
Eelco Dolstra
79e358ce6d RemoteStore: Close connection if an exception occurs
Fixes #2075.
2018-10-16 23:36:15 +02:00
Eelco Dolstra
ba51100d64 Get rid of UDSRemoteStore::Connection
Since its superclass RemoteStore::Connection contains 'to' and 'from'
fields that refer to the file descriptor maintained in the subclass,
it was possible for the flush() call in Connection::~Connection() to
write to a closed file descriptor (or worse, a file descriptor now
referencing another file). So make sure that the file descriptor
survives 'to' and 'from'.
2018-10-16 22:58:01 +02:00
Eelco Dolstra
1e7b8deea7
Check requiredSystemFeatures for local builds
For example, this prevents a "kvm" build on machines that don't have
KVM.

Fixes #2012.
2018-09-28 16:10:27 +02:00
Eelco Dolstra
7ae7a38c9a
Move structured attrs handling into a separate class
This is primarily because Derivation::{can,will}BuildLocally() depends
on attributes like preferLocalBuild and requiredSystemFeatures, but it
can't handle them properly because it doesn't have access to the
structured attributes.
2018-09-28 14:32:46 +02:00
Eelco Dolstra
99d4bb2d4c
Remove obsolete check on system features 2018-09-28 13:28:15 +02:00
Eelco Dolstra
c9ba33870e
Support special attributes in structured attributes derivations
E.g. __noChroot and allowedReferences now work correctly. We also now
check that the attribute type is correct. For instance, instead of

  allowedReferences = "out";

you have to write

  allowedReferences = [ "out" ];

Fixes #2453.
2018-09-28 12:43:01 +02:00
Eelco Dolstra
63786cbd3b
S3BinaryCacheStore: Don't create buckets
This meant that making a typo in an s3:// URI would cause a bucket to
be created. Also it didn't handle eventual consistency very well. Now
it's up to the user to create the bucket.
2018-09-27 21:34:41 +02:00
Eelco Dolstra
9750430003
Ensure download thread liveness
* Don't wait forever for the client to remove data from the
  buffer. This does mean that the buffer can grow without bounds
  (e.g. when downloading is faster than writing to disk), but meh.

* Don't hold the state lock while calling the sink. The sink could
  take any amount of time to process the data (in particular when it's
  actually a coroutine), so we don't want to block the download
  thread.
2018-09-26 21:47:34 +02:00
Eelco Dolstra
98b2cc2e6e
Untabify 2018-09-26 21:39:06 +02:00
Eelco Dolstra
bba3f0a308
If fallback is enabled, temporarily disable substituters after a failure
Otherwise, we just keep asking the substituter for other .narinfo
files, which can take a very long time due to retries/timeouts.
2018-09-07 17:08:43 +02:00
Eelco Dolstra
33c3f91885
Handle queryPathInfo() failure from substituters when fallback is enabled
Fixes #1990.
2018-09-07 16:35:48 +02:00
Eelco Dolstra
2d91012754
fetchurl: Respect unpack
Fixes #2393.
2018-09-05 21:22:37 +02:00
Will Dietz
28418af920 download: fix size reported to progress bar 2018-09-03 14:12:54 -05:00
Daiderd Jordan
070823baa4
Store: expose the protocol version used by a store 2018-09-02 12:54:29 +02:00
Eelco Dolstra
c0c31b58a4
Add localhost alias for ::1 to the sandbox 2018-08-31 11:12:51 +02:00
Michael Bishop
0767e402f1 fix error: unknown serve command 9 2018-08-30 20:10:08 -03:00
Eelco Dolstra
ebe3d2d370 Improve 'coroutine has finished' error message 2018-08-21 15:22:04 +02:00
Eelco Dolstra
17a92dfb7d Fix another 'coroutine has finished' during decompression
https://hydra.nixos.org/build/79867739
2018-08-21 15:19:20 +02:00
Symphorien Gibol
5b19a6663b ignore when listxattr fails with ENODATA
This happens on CIFS and means the remote filesystem has no extended
attributes.
2018-08-13 21:11:36 +02:00
Eelco Dolstra
ada4e90267
S3BinaryCacheStore: Share TransferManager
TransferManager allocates a lot of memory (50 MiB by default), and it
might leak but I'm not sure about that. In any case it was causing
OOMs in hydra-queue-runner. So allocate only one TransferManager per
S3BinaryCacheStore.

Hopefully fixes https://github.com/NixOS/hydra/issues/586.
2018-08-09 20:48:45 +02:00
Eelco Dolstra
a0b971dd9c
S3BinaryCacheStore: Don't use the transfer status callback
This callback is executed on a different thread, so exceptions thrown
from the callback are not caught:

  Aug 08 16:25:48 chef hydra-queue-runner[11967]: terminate called after throwing an instance of 'nix::Error'
  Aug 08 16:25:48 chef hydra-queue-runner[11967]:   what():  AWS error: failed to upload 's3://nix-cache/19dbddlfb0vp68g68y19p9fswrgl0bg7.ls'

Therefore, just check the transfer status after it completes. Also
include the S3 error message in the exception.
2018-08-08 21:39:11 +02:00
Bob van der Linden
58a85fa462 mention nix-store --query --roots when a path cannot be deleted 2018-08-08 21:21:21 +02:00
Eelco Dolstra
ed6c646f44
Doh 2018-08-06 17:27:08 +02:00
Eelco Dolstra
7de3e00ad9
Fix setting Content-Encoding in S3 uploads
Fixes https://github.com/NixOS/nix/issues/2333 and https://github.com/NixOS/nixpkgs/issues/44337.
2018-08-06 17:20:23 +02:00
Eelco Dolstra
d3761f5f8b
Fix Brotli decompression in 'nix log'
This didn't work anymore since decompression was only done in the
non-coroutine case.

Decompressors are now sinks, just like compressors.

Also fixed a bug in bzip2 API handling (we have to handle BZ_RUN_OK
rather than BZ_OK), which we didn't notice because there was a missing
'throw':

  if (ret != BZ_OK)
      CompressionError("error while compressing bzip2 file");
2018-08-06 15:40:29 +02:00
Eelco Dolstra
4361a4331f
Fix reporting of HTTP body size when a result callback is used 2018-08-06 11:31:14 +02:00
Eelco Dolstra
2825e05d21
Make adding paths via nix-store --serve run in constant memory
It adds a new operation, cmdAddToStoreNar, that does the same thing as
the corresponding nix-daemon operation, i.e. call addToStore(). This
replaces cmdImportPaths, which has the major issue that it sends the
NAR first and the store path second, thus requiring us to store the
incoming NAR either in memory or on disk until we decide what to do
with it.

For example, this reduces the memory usage of

  $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79

from 267 MiB to 12 MiB.

Probably fixes #1988.
2018-08-03 21:21:16 +02:00
Eelco Dolstra
848a9375c3
Support escaping in store URIs 2018-08-03 21:20:38 +02:00
Eelco Dolstra
4e7d5f660c
SSHMaster: Bypass SSH when connecting to localhost
This is primarily useful for testing since it removes the need to have
SSH working.
2018-08-03 21:20:38 +02:00
Eelco Dolstra
e268bbc054
LegacySSHStore: Add remote-store option
This is primarily useful for testing, e.g.

  $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' ...
2018-08-03 21:20:38 +02:00
Graham Christensen
49a53c1d3f s3 binary cache: support specifying an endpoint
Works for uploading and not downloading.
2018-08-02 08:19:50 -04:00
Eelco Dolstra
4b4adbc93d
Fix race downloading into a sink
This fixes 'error 10 while decompressing xz file'.

https://hydra.nixos.org/build/78308551
2018-07-31 17:26:04 +02:00
Linus Heckemann
9ac1a79882 copyPathsToStore: honour keep-going 2018-07-24 17:05:06 +02:00
Eelco Dolstra
b920b90857
Detect redirect loops
Fixes #2225.
2018-06-18 10:36:19 +02:00
Eelco Dolstra
2e244fb68f
Merge pull request #2234 from symphorien/esrch2
libstore/gc.cc: ignore ESRCH when reading symlinks in /proc
2018-06-18 09:01:55 +02:00
Symphorien Gibol
ef8122064b libstore/gc.cc: ignore ESRCH when reading symlinks in /proc
readlink is also affected by the problem fixed for regular files in
c567afe355df
2018-06-16 18:04:09 +02:00
Matthew Justin Bauer
6d09e4400c
Fix optimise-store.cc to skip more paths on macOS
I hate to make this such a large check but the lack of documentation means we really have no idea what's allowed. All of them reported so far have been within ".app/Contents" directories. That appears to be a safe starting point. However, I would not be surprised to also find more paths that are disallowed for instance in .framework or .bundle directories.

Fixes #2031
Fixes #2229
2018-06-14 12:34:55 -04:00
Eelco Dolstra
7902fccf6f
hashDerivationModulo(): Handle chroot stores
This makes 'nix copy --to /mnt /nix/store/bla.drv' work.
2018-06-13 18:00:54 +02:00
Eelco Dolstra
3c5d9f478d
std::random_shuffle -> std::shuffle
The former is removed in C++17.
2018-06-13 16:56:19 +02:00
Yorick van Pelt
72a78beb34
Fix #2162: use getaddrinfo instead of curl to preload NSS 2018-06-12 13:05:14 +02:00
Symphorien Gibol
8c567afe35 libstore/gc.cc: ignore ESRCH when reading /proc
If a process disappears between the time /proc/[pid]/maps is opened and
the time it is read, the read() syscall will return ESRCH. This should be ignored.
2018-06-11 16:29:47 +02:00
Will Dietz
6bf7dc1baf download: if there are active requests, never sleep for 10s
In some versions/configurations libcurl doesn't handle timeouts
(especially DNS timeouts) in a way that wakes curl_multi_wait.

This doesn't appear to be a problem if using c-ares, FWIW.
2018-06-08 21:12:43 -05:00
AmineChikhaoui
170165ee5a
use debug instead of printTalkative to avoid spamming hydra queue
runner logs.
2018-06-08 11:43:32 +01:00
Eelco Dolstra
4ac4f675df
Don't require --fallback to recover from disappeared binary cache NARs 2018-06-05 16:04:41 +02:00
Eelco Dolstra
691b7582c7
Handle HTTP 410 Gone errors 2018-06-05 16:03:32 +02:00
Eelco Dolstra
ca06a9cea7
Improve binary cache fallback test 2018-06-05 15:26:12 +02:00
Eelco Dolstra
a2ec7a3bfd
Further improve upload messages 2018-06-05 14:37:26 +02:00
Will Dietz
54b1c59643 download.cc: fix readCallback return type (int -> size_t)
I'm not sure if curl ever asks for enough data at once
for truncation to occur but better safe than sorry.
2018-06-01 10:59:26 -05:00
Eelco Dolstra
73357500ac
Improve binary cache upload messages
Don't say "download" when we mean "upload".
2018-06-01 14:14:22 +02:00
Peter Simons
93aa3bea2e
Merge pull request #767 from mogorman/garbage_collect_keep_last_few
Implement --delete-generations + flag for keeping last N number of gens
2018-05-31 10:00:21 +02:00
Eelco Dolstra
3c2de9830d
RemoteStore::addToStore(): Ensure capacity is decreased on exceptions 2018-05-30 13:48:31 +02:00
Eelco Dolstra
a4c1618876
Fix deadlock in RemoteStore::queryPathInfoUncached() 2018-05-30 13:46:14 +02:00
Eelco Dolstra
7d21863bb3
Make <nix/fetchurl.nix> run in constant memory
E.g.

  nix-build --store ~/my-nix/ -E 'import <nix/fetchurl.nix> { url = https://cache.nixos.org/nar/0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv.nar.xz; sha256 = "0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv"; }'

now runs in 17 MiB (was 70 MiB), while

  nix-build --store ~/my-nix/ -E 'import <nix/fetchurl.nix> { url = https://cache.nixos.org/nar/0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv.nar.xz; sha256 = "0d2fxljdih3nc5dqx41hjzic3141ajil94m8kdbpryq569dpsbvb"; unpack = true; }'

runs in 17 MiB (was 346 MiB).
2018-05-30 13:42:51 +02:00
Eelco Dolstra
5a654fd7dd
getDownloader(): Simplify 2018-05-30 13:42:30 +02:00
Eelco Dolstra
e87e4a60d6
Make HttpBinaryCacheStore::narFromPath() run in constant memory
This reduces memory consumption of

  nix copy --from https://cache.nixos.org --to ~/my-nix /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79

from 176 MiB to 82 MiB. (The remaining memory is probably due to xz
decompression overhead.)

Issue https://github.com/NixOS/nix/issues/1681.
Issue https://github.com/NixOS/nix/issues/1969.
2018-05-30 13:42:29 +02:00
Eelco Dolstra
08ec757726
Make LocalBinaryCacheStore::narFromPath() run in constant memory
This reduces memory consumption of

  nix copy --from file://... --to ~/my-nix /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79

from 514 MiB to 18 MiB for an uncompressed binary cache, and from 192
MiB to 53 MiB for a bzipped binary cache. It may also be faster
because fetching can happen concurrently with decompression/writing.

Continuation of 48662d151b.

Issue https://github.com/NixOS/nix/issues/1681.
2018-05-30 13:34:57 +02:00
Eelco Dolstra
81ea8bd5ce
Simplify the callback mechanism 2018-05-30 13:34:37 +02:00
Eelco Dolstra
1672bcd230
Move evaluator-specific settings out of libstore 2018-05-30 13:29:50 +02:00
Eelco Dolstra
c1d445ecec
Move some Download-specific settings to download.cc 2018-05-30 13:29:50 +02:00
Eelco Dolstra
737ed88f35
Modularize config settings
Allow global config settings to be defined in multiple Config
classes. For example, this means that libutil can have settings and
evaluator settings can be moved out of libstore. The Config classes
are registered in a new GlobalConfig class to which config files
etc. are applied.

Relevant to https://github.com/NixOS/nix/issues/2009 in that it
removes the need for ad hoc handling of useCaseHack, which was the
underlying cause of that issue.
2018-05-30 13:28:01 +02:00
Eelco Dolstra
e606cd412f
Fix assertion failure in storePathToHash()
Fixes https://github.com/NixOS/nix/issues/2015.
2018-05-30 13:27:19 +02:00
Eelco Dolstra
6185d25e52
Make 'nix copy --to daemon' run in constant memory (daemon side)
Continuation of 97002b684c. This makes
the daemon use constant memory. For example, it reduces the daemon's
maximum RSS on

  $ nix copy --from ~/my-nix --to daemon /nix/store/1n7x0yv8vq6zi90hfmian84vdhd04bgp-blender-2.79a

from 264 MiB to 7 MiB.

We now use a TunnelSource to prevent the connection from ending up in
an undefined state if an exception is thrown while the NAR is being
sent.

Issue https://github.com/NixOS/nix/issues/1681.
2018-05-30 13:26:48 +02:00
Eelco Dolstra
23d6bb583a
LocalStore::addToStore: Throw an exception if no hash is given 2018-05-30 13:25:40 +02:00
Eelco Dolstra
3cab639e21
Use extended SQLite error codes 2018-05-30 13:25:03 +02:00
Will Dietz
ca9c6cb95d libstore/download: remove unused function "getTime()" 2018-05-25 19:49:45 -05:00
Eelco Dolstra
966407bcf1
Merge pull request #2145 from serokell/uncached-channel
make sure not to use cached channels for nix-channel --update
2018-05-15 14:57:18 +02:00
Eelco Dolstra
2b3c1b3a88
Remove mention of default since we already know the default 2018-05-10 12:06:28 +02:00
Eelco Dolstra
38def17627
Merge branch 's3-multipart-uploads' of https://github.com/AmineChikhaoui/nix 2018-05-10 12:06:02 +02:00
Yorick van Pelt
b9289e4855
make sure not to use cached channels for nix-channel --update
fixes #1964
2018-05-09 16:18:20 +02:00
AmineChikhaoui
854c0860f4
share the executor between multiple copy threads. 2018-05-07 15:07:00 +01:00
AmineChikhaoui
591e75cd01
add a FIXME note to find a way to abort the multipart uploads in
case the nix command is interrupted.
2018-05-07 14:27:53 +01:00
AmineChikhaoui
129394fb95
Support multi-part uploads for large NARs that exceed the size of
5Gb.
2018-05-07 14:23:51 +01:00
Eelco Dolstra
c717d8e3bd
Don't set a CA assertion for paths with references
Really fixes #2133.
2018-05-03 14:49:56 +02:00
Eelco Dolstra
80a7b16593
LocalStore::addValidPath(): Throw error when the CA assertion is wrong
Closes #2133.
2018-05-03 11:02:40 +02:00
Eelco Dolstra
53ec5ac69f
Fix some random -Wconversion warnings 2018-05-02 13:56:34 +02:00
Will Dietz
8e7d77d494 globals.hh: don't use '==' to compare string literals
Saw this in logs, also reported here:

8e6108ff71 (r28707288)
2018-04-23 20:57:05 -05:00
Shea Levy
8e6108ff71
Merge branch 'aarch64-armv7' of git://github.com/lheckemann/nix
Support extra compatible architectures (#1916)
2018-04-23 08:48:22 -04:00
Shea Levy
58f9ae0dcd
Merge branch 'hash-mismatch' of git://github.com/LnL7/nix 2018-04-23 08:40:34 -04:00
Peter Simons
040acdcee2
Merge pull request #1707 from dtzWill/fix/issue-1692
ignore "interrupted" exception in progress callback
2018-04-21 16:51:19 +02:00
Daiderd Jordan
13d4d1c0a2
download: make hash mismatch error consistent with fetchurl 2018-04-20 21:07:32 +02:00
Will Dietz
6d9129014d build.cc: fix bind-mount of /dev/{pts,ptmx} fallback
Don't bind-mount these to themselves,
mount them into the chroot directory.

Fixes pty issues when using sandbox on CentOS 7.4.
(build of perlPackages.IOTty fails before this change)
2018-04-20 12:24:23 -05:00
Eelco Dolstra
c74f838620
exportReferences: Check whether the path is in the input closure
Fixes #2082.
2018-04-17 12:03:27 +02:00
Eelco Dolstra
d34fa2bcc3
Fix #1921 2018-04-13 15:42:35 +02:00
Shea Levy
dc0a542c9f
initPlugins: Fix dlopen error message. 2018-04-11 21:02:50 -04:00
Eelco Dolstra
d6aaa35478
Fix perl-bindings
https://hydra.nixos.org/build/72636781
2018-04-10 13:11:43 +02:00
Shea Levy
a38fe5c1a8
Export required C++ version in pkgconfig. 2018-04-09 11:32:43 -04:00
AmineChikhaoui
62d75ad3e1
rename the options to mention it's a narinfo TTL as disk cache is used all over the place for other operations 2018-04-06 11:05:15 +01:00
AmineChikhaoui
86930ed414
add documentation for the local disk cache TTL config 2018-04-06 10:51:18 +01:00
AmineChikhaoui
2855c3d965
Make the TTL for disk cache configurable, we can now completely disable
disk cache lookup for example by doing:
  nix copy --from <binary-cahe> <store-path> --option \
  positive-disk-cache-ttl 0

Issues: #1885 #2035
2018-04-06 10:51:18 +01:00
Eelco Dolstra
1d9742b95f
<nix/buildenv.nix>: Ignore dangling symlinks
Fixes

  error: getting status of '/nix/store/j8p0vv89k1pf0cn7kmfsdcs7bshwga1i-firefox-52.7.2esr/share/icons/hicolor/48x48/apps/firefox.png': No such file or directory

https://github.com/NixOS/nix/issues/1934

Also improve error message on directory/non-directory collisions.
2018-04-05 20:04:42 +02:00
Eelco Dolstra
5c904e10eb
<nix/buildenv.nix>: Skip non-directory top-level paths
Fixes https://github.com/NixOS/nix/issues/1934 (at least the "error:
opening directory '/nix/store/...-linux-config-4.4.14': Not a
directory" issue).
2018-04-05 19:34:56 +02:00
Eelco Dolstra
2b61c74922
Process --option use-case-hack properly
Fixes https://github.com/NixOS/nix/issues/2009.
2018-03-29 12:28:29 +02:00
Eelco Dolstra
9d1220a01d
ValidPathInfo::isContentAddressed(): Ensure there are no references 2018-03-29 12:27:42 +02:00
Will Dietz
74da813912 download: improve error for hash mismatch ("store mismatch")
Fixes #1905
2018-03-22 15:22:06 -05:00
Will Dietz
59b32403f2 download.cc: fix path for cached files 2018-03-22 15:05:46 -05:00
Eelco Dolstra
f87e286e82
Merge branch 'fix/avoid-large-stack-buffers' of https://github.com/dtzWill/nix 2018-03-22 13:19:25 +01:00
Eelco Dolstra
ec91840e70 Also make the backwards compatible case of RemoteStore::addToStore run in constant memory
Just because we can.
2018-03-21 23:46:03 +01:00
Eelco Dolstra
97002b684c Make 'nix copy --to daemon' run in constant memory 2018-03-21 23:42:52 +01:00
Eelco Dolstra
11898d6a37 Make 'nix copy --to ssh://...' run in constant memory 2018-03-21 23:13:06 +01:00
Eelco Dolstra
47f7e5585b Make 'nix copy --from ssh://...' run in constant memory
For instance, this reduced the memory consumption of

  $ nix copy --from ssh://localhost --to ~/my-nix /nix/store/1n7x0yv8vq6zi90hfmian84vdhd04bgp-blender-2.79a

from 632 MiB to 16 MiB.
2018-03-21 23:10:40 +01:00
Eelco Dolstra
8615cfb130
LegacySSHStore: Allow overriding the path to nix-store 2018-03-21 15:10:46 +01:00
Eelco Dolstra
668ac3ea2c
Make <nix/buildenv.nix> a builtin builder
This avoids sandbox annoyances.
2018-03-20 17:28:09 +01:00
Eelco Dolstra
9d40787938
Move builtinFetchurl to its own file 2018-03-20 17:01:36 +01:00
Eelco Dolstra
5df427f35b
Slight simplification 2018-03-20 16:32:59 +01:00
Eelco Dolstra
21ebf6a160
SSHMaster: Optionally pass -v to ssh 2018-03-20 15:30:55 +01:00
Eelco Dolstra
1aca195e52
Hack to get SSH error messages from build-remote
E.g.

  cannot build on 'ssh://mac1': cannot connect to 'mac1': bash: nix-store: command not found
  cannot build on 'ssh://mac2': cannot connect to 'mac2': Host key verification failed.
  cannot build on 'ssh://mac3': cannot connect to 'mac3': Received disconnect from 213... port 6001:2: Too many authentication failures
  Authentication failed.
2018-03-20 15:25:26 +01:00
Linus Heckemann
637701b604 rename build-extra-platforms -> extra-platforms
also document it
2018-03-16 22:50:27 +00:00
Eelco Dolstra
48662d151b
Reduce substitution memory consumption
copyStorePath() now pipes the output of srcStore->narFromPath()
directly into dstStore->addToStore(). The sink used by the former is
converted into a source usable by the latter using
boost::coroutine2. This is based on [1].

This reduces the maximum resident size of

  $ nix build --store ~/my-nix/ /nix/store/b0zlxla7dmy1iwc3g459rjznx59797xy-binutils-2.28.1 --substituters file:///tmp/binary-cache-xz/ --no-require-sigs

from 418592 KiB to 53416 KiB. (The previous commit also reduced the
runtime from ~4.2s to ~3.4s, not sure why.) A further improvement will
be to download files into a Sink.

[1] https://github.com/NixOS/nix/compare/master...Mathnerd314:dump-fix-coroutine#diff-dcbcac55a634031f9cc73707da6e4b18

Issue #1969.
2018-03-16 20:35:59 +01:00
Eelco Dolstra
3e6b194d78
decompress(): Use a Source and Sink
This allows decompression to happen in O(1) memory.
2018-03-16 20:35:59 +01:00
Eelco Dolstra
eb75bc5afb
Don't retry on CURLE_SSL_CACERT_BADFILE
The certificates won't get any better if we retry.
2018-03-16 12:03:18 +01:00
Eelco Dolstra
7b8914825a
Merge pull request #1965 from masaeedu/master
Wrap thread local in function for Cygwin
2018-03-15 14:56:21 +01:00
Shea Levy
cfdbfa6b2c
Catch more possible instances of passing NULL to memcpy.
Actually fixes #1976.
2018-03-14 23:53:43 -04:00
Eelco Dolstra
ca14b14200
Use boost::format from the boost package
Note that this only requires headers from boost so it doesn't add a
runtime dependency.

Also, use Nixpkgs 18.03.
2018-03-14 19:24:04 +01:00
Eelco Dolstra
6e60141a80
Typo 2018-03-13 17:48:09 +01:00
Asad Saeeduddin
be54f4a0b6 Wrap thread local in function for Cygwin
Fixes #1826. See #1352 for a previous instance of a similar change.
2018-03-12 00:56:41 -04:00
Eelco Dolstra
16551f54c9
Fix #1957 2018-03-07 11:40:50 +01:00
Will Dietz
56253bb08f rand() -> random(), since we use srandom().
rand() requires we call srand() instead,
but might as well use random().
2018-03-06 17:44:25 -06:00
Shea Levy
088ef81759
ssh-ng: Don't forward options to the daemon.
This can be iterated on and currently leaves out settings we know we
want to forward, but it fixes #1713 and fixes #1935 and isn't
fundamentally broken like the status quo. Future changes are suggested
in a comment.
2018-03-05 07:42:15 -05:00
Will Dietz
6b9a03f5d8 hoist vector out of loop just in case 2018-03-02 10:52:05 -06:00
Will Dietz
c89a3d5368 don't allocate large buffers on the stack 2018-03-02 10:52:04 -06:00
Matthew O'Gorman
467fdd8ca4 only delete things older than current gen and update logic in doc as
well
2018-03-01 22:59:00 -05:00
Matthew O'Gorman
3c16044cb0 remove unused variable and make sure to check that the current
generation is not the one we are deleting
2018-03-01 22:22:02 -05:00
Matthew O'Gorman
0312d30315 this updates issues that were addressed by people in pr 2018-03-01 21:47:57 -05:00
Matthew O'Gorman
429154b74c Implement --delete-generations + flag for keeping last N number of generations 2018-03-01 21:47:57 -05:00
Linus Heckemann
919c3c20b3 seccomp: add 32-bit ARM on aarch64-linux
This allows building armv[67]l-linux derivations on compatible aarch64
machines. Failure to add the architecture may result from missing
hardware support, in which case we can't run 32-bit binaries and don't
need to restrict them with seccomp anyway,
2018-02-27 00:32:15 +00:00
Linus Heckemann
eb03a296c1 Add build-extra-platforms setting
This allows specifying additional systems that a machine is able to
build for. This may apply on some armv7-capable aarch64 processors, or
on systems using qemu-user with binfmt-misc to support transparent
execution of foreign-arch programs.

This removes the previous hard-coded assumptions about which systems are
ABI-compatible with which other systems, and instead relies on the user
to specify any additional platforms that they have ensured compatibility
for and wish to build for locally.

NixOS should probably add i686-linux on x86_64-linux systems for this
setting by default.
2018-02-27 00:27:58 +00:00
Eelco Dolstra
64e486ab63
Fix downloadCached() with a chroot store
E.g.

  nix run --store ~/my-nix -f channel:nixos-17.03 hello -c hello

This problem was mentioned in #1897.
2018-02-23 13:00:42 +01:00
Eelco Dolstra
e8d53bfdc9
Revert "Enable sandbox builds on Linux by default"
This reverts commit ddc58e7896.

https://hydra.nixos.org/eval/1435322
2018-02-22 14:20:07 +01:00
Eelco Dolstra
ddc58e7896
Enable sandbox builds on Linux by default
The overhead of sandbox builds is a problem on NixOS (since building a
NixOS configuration involves a lot of small derivations) but not for
typical non-NixOS use cases. So outside of NixOS we can enable it.

Issue #179.
2018-02-22 12:27:25 +01:00
Eelco Dolstra
d4e93532e2
Fix incorrect (and unnecessary) format string
https://hydra.nixos.org/eval/1434547#tabs-now-fail
2018-02-19 20:46:39 +01:00
Shea Levy
e59a8a63e1
Don't silently succeed seccomp setup when !HAVE_SECCOMP.
Running Nix with build users without seccomp on Linux is dangerous,
and administrators should very explicitly opt-in to it.
2018-02-19 09:56:24 -05:00
Eelco Dolstra
75a1d96cfd
Merge branch 'register-settings' of https://github.com/shlevy/nix 2018-02-19 13:58:34 +01:00
Shea Levy
690ac7c90b
configure: Add a flag to disable seccomp.
This is needed for new arches where libseccomp support doesn't exist
yet.

Fixes #1878.
2018-02-18 02:35:01 -05:00
Matthew Bauer
8f186722a9 Set backup MANPATH in case man path isn’t set correctly.
Previously, this would fail at startup for non-NixOS installs:

nix-env --help

The fix for this is to just use "nixManDir" as the value for MANPATH
when spawning "man".

To test this, I’m using the following:

$ nix-build release.nix -A build
$ MANPATH= ./result/bin/nix-env --help

Fixes #1627
2018-02-15 10:20:02 -06:00
Shea Levy
de4934ab3b
Allow plugins to define new settings. 2018-02-13 14:43:32 -05:00
Shea Levy
b8739f2fb3
Enable specifying directories in plugin-files. 2018-02-13 11:25:01 -05:00
Eelco Dolstra
1c10a74c73
Merge branch 'plugins' of https://github.com/shlevy/nix 2018-02-13 12:15:27 +01:00
Eelco Dolstra
9bcb4d2dd9 Fix hang in build-remote 2018-02-12 22:48:55 +01:00
Eelco Dolstra
4f09ce7940
Fix 'deadlock: trying to re-acquire self-held lock'
This was caused by derivations with 'allowSubstitutes = false'. Such
derivations will be built locally. However, if there is another
SubstitionGoal that has the output of the first derivation in its
closure, then the path will be simultaneously built and substituted.

There was a check to catch this situation (via pathIsLockedByMe()),
but it no longer worked reliably because substitutions are now done in
another thread. (Thus the comment 'It can't happen between here and
the lockPaths() call below because we're not allowing multi-threading'
was no longer valid.)

The fix is to handle the path already being locked in both
SubstitutionGoal and DerivationGoal.
2018-02-12 17:06:06 +01:00
Eelco Dolstra
960e9c560e
nix: Ensure that the user sees errors from substituters 2018-02-09 15:06:47 +01:00
Eelco Dolstra
aa02cdc33c
getDefaultSubstituters(): Skip broken substituters
Fixes #1340.
2018-02-09 15:06:47 +01:00
Shea Levy
88cd2d41ac
Add plugins to make Nix more extensible.
All plugins in plugin-files will be dlopened, allowing them to
statically construct instances of the various Register* types Nix
supports.
2018-02-08 12:44:37 -05:00
AmineChikhaoui
47ad88099b
move the parallel-compression setting to binary-cache-store, the setting
can be done now from the url e.g s3://nix-cache?parallel-compression=1
instead of nix.conf.
2018-02-07 21:06:11 +01:00
AmineChikhaoui
55ecdfe2a8
make multi threaded compression configurable and use single threaded
by default.
2018-02-07 17:54:08 +01:00
Eelco Dolstra
84989d3af2
Improve filtering of ANSI escape sequences in build logs
All ANSI sequences except color setting are now filtered out. In
particular, terminal resets (such as from NixOS VM tests) are filtered
out.

Also, fix the completely broken tab character handling.
2018-02-07 15:23:57 +01:00
Eelco Dolstra
abe6be578b
Merge pull request #1816 from shlevy/add-path
Add path primop.
2018-02-07 13:32:35 +01:00
Shea Levy
69d82e5c58
Add path primop.
builtins.path allows specifying the name of a path (which makes paths
with store-illegal names now addable), allows adding paths with flat
instead of recursive hashes, allows specifying a filter (so is a
generalization of filterSource), and allows specifying an expected
hash (enabling safe path adding in pure mode).
2018-02-06 16:48:08 -05:00
Eelco Dolstra
bb1d046f5c
Allow substituters to be marked as trusted
This is needed by nixos-install, which uses the Nix store on the
installation CD as a substituter. We don't want to disable signature
checking entirely because substitutes from cache.nixos.org should
still be checked. So now we can pas "local?trusted=1" to mark only the
Nix store in /nix as not requiring signatures.

Fixes #1819.
2018-02-05 18:08:30 +01:00
Eelco Dolstra
2175eee9fe
Fix segfault using non-binary cache stores as substituters 2018-02-05 17:46:43 +01:00
Eelco Dolstra
84722d67d2
Remove nix-build --hash
Instead, if a fixed-output derivation produces has an incorrect output
hash, we now unconditionally move the outputs to the path
corresponding with the actual hash and register it as valid. Thus,
after correcting the hash in the Nix expression (e.g. in a fetchurl
call), the fixed-output derivation doesn't have to be built again.

It would still be good to have a command for reporting the actual hash
of a fixed-output derivation (instead of throwing an error), but
"nix-build --hash" didn't do that.
2018-02-03 10:08:05 +01:00
Shea Levy
de96daf54f
Merge branch 'master' of git://github.com/catern/nix 2018-02-01 13:21:45 -05:00
Eelco Dolstra
855699855f
Remove obsolete references to manifests
Closes #323.
2018-02-01 10:39:16 +01:00
Spencer Baugh
e5432574e2 document ability to set NIX_REMOTE=unix://path/to/socket 2018-01-31 22:47:16 +00:00
Eelco Dolstra
c129fc6ee8
Merge pull request #1801 from catern/master
remote_store: register for NIX_REMOTE=unix://path
2018-01-31 23:12:39 +01:00
Eelco Dolstra
88f4f0231b Fix building without aws-sdk-cpp 2018-01-31 22:34:51 +01:00
Eelco Dolstra
c287d73121
Rename 1.12 -> 2.0
Following discussion with Shea and Graham. It's a big enough change
from the last release. Also, from a semver perspective, 2.0 makes more
sense because we did remove some interfaces (like nix-pull/nix-push).
2018-01-31 18:58:45 +01:00
Eelco Dolstra
30370f168f
Cleanup 2018-01-31 15:14:03 +01:00
Eelco Dolstra
6270b2e50f
Merge branch 'http-binary-cache-put-upsert' of https://github.com/adelbertc/nix 2018-01-31 15:10:12 +01:00
Eelco Dolstra
478e3e4649
Indent properly 2018-01-31 13:46:38 +01:00
Spencer Baugh
746f8aed86 remote_store: register for NIX_REMOTE=unix://path
This allows overriding the socket path so the daemon may be listening at
an arbitrary Unix domain socket location.

Fixes #1800
2018-01-26 22:05:16 +00:00
Shea Levy
1d5d277ac7
HttpBinaryCacheStore: Support upsertFile with PUT.
Some servers, such as Artifactory, allow uploading with PUT and BASIC
auth. This allows nix copy to work to upload binaries to those
servers.

Worked on together with @adelbertc
2018-01-26 11:12:30 -08:00
Eelco Dolstra
e09161d05c
Remove signed-binary-caches as the default for require-sigs
This was for backward compatibility. However, with security-related
configuration settings, it's best not to have any confusion.

Issue #495.
2018-01-26 17:12:15 +01:00
Dan Peebles
d43a8b25f0 Fix obscure corner case in name resolution for builtin:fetchurl in sandboxed environments 2018-01-23 14:45:50 -05:00
Eelco Dolstra
3c4c30eadd
Rewrite builtin derivation environment
Also add a test.

Fixes #1803.
Closes #1805.
2018-01-19 15:00:38 +01:00
Eelco Dolstra
2896bb6826
Don't retry CURLE_URL_MALFORMAT 2018-01-19 14:05:08 +01:00
Eelco Dolstra
75b9670df6
Make show-trace a config setting 2018-01-16 19:24:23 +01:00
Eelco Dolstra
d4dcffd643
Add pure evaluation mode
In this mode, the following restrictions apply:

* The builtins currentTime, currentSystem and storePath throw an
  error.

* $NIX_PATH and -I are ignored.

* fetchGit and fetchMercurial require a revision hash.

* fetchurl and fetchTarball require a sha256 attribute.

* No file system access is allowed outside of the paths returned by
  fetch{Git,Mercurial,url,Tarball}. Thus 'nix build -f ./foo.nix' is
  not allowed.

Thus, the evaluation result is completely reproducible from the
command line arguments. E.g.

  nix build --pure-eval '(
    let
      nix = fetchGit { url = https://github.com/NixOS/nixpkgs.git; rev = "9c927de4b179a6dd210dd88d34bda8af4b575680"; };
      nixpkgs = fetchGit { url = https://github.com/NixOS/nixpkgs.git; ref = "release-17.09"; rev = "66b4de79e3841530e6d9c6baf98702aa1f7124e4"; };
    in (import (nix + "/release.nix") { inherit nix nixpkgs; }).build.x86_64-linux
  )'

The goal is to enable completely reproducible and traceable
evaluation. For example, a NixOS configuration could be fully
described by a single Git commit hash. 'nixos-rebuild' would do
something like

  nix build --pure-eval '(
    (import (fetchGit { url = file:///my-nixos-config; rev = "..."; })).system
  ')

where the Git repository /my-nixos-config would use further fetchGit
calls or Git externals to fetch Nixpkgs and whatever other
dependencies it has. Either way, the commit hash would uniquely
identify the NixOS configuration and allow it to reproduced.
2018-01-16 19:23:18 +01:00
Eelco Dolstra
ba75c69e00
Barf when using a diverted store on macOS
Fixes #1792.
2018-01-15 12:14:43 +01:00
Eelco Dolstra
74f75c8558
import, builtins.readFile: Handle diverted stores
Fixes #1791
2018-01-12 17:31:08 +01:00
Renzo Carbonara
b0328c244d nix.conf: builders-use-substitutes
Fixes #937
2018-01-09 22:40:07 +01:00
Eelco Dolstra
44272d8719
Rename "use-substitutes" to "substitute"
Commit c2154d4c84 renamed
"build-use-substitutes" to "use-substitutes", but that broke
"nix-copy-closure --use-substitutes".
2018-01-04 16:58:39 +01:00
Eelco Dolstra
aa43cbb764
Check aws-sdk-cpp version 2017-12-22 12:05:13 +01:00
Eelco Dolstra
8efe937a35 Fix build on gcc 7
Fixes #1738.
2017-12-16 00:49:31 +01:00
Eelco Dolstra
5d5b931fb1 Fix build against current aws-sdk-cpp 2017-12-14 16:45:40 +01:00
Eelco Dolstra
1ca8e95178
Merge pull request #1722 from bhipple/fix-for-older-libcurl
Fix for builds with system libcurl < 7.30
2017-12-12 11:48:08 +01:00
Will Dietz
00e0c416ff Mark comparison call operator as const 2017-12-11 12:08:40 -06:00
Benjamin Hipple
d76c4fe770 Fix for builds with system libcurl < 7.30
CentOS 7.4 and RHEL 7.4 ship with libcurl-devel-7.29.0-42.el7.x86_64; this flag
was added in 7.30.0
https://curl.haxx.se/libcurl/c/CURLMOPT_MAX_TOTAL_CONNECTIONS.html
2017-12-09 20:02:21 -05:00
Eelco Dolstra
2df9cbeb47 Provide random access to cached NARs
E.g.

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nars \
    /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79/share/icons/hicolor/scalable/apps/blender.svg > /dev/null
  real    0m4.139s

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nars \
    /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79/share/icons/hicolor/scalable/apps/blender.svg > /dev/null
  real    0m0.024s

(Before, the second call took ~0.220s.)

This will use a NAR listing in
/tmp/nars/b0w2hafndl09h64fhb86kw6bmhbmnpm1.ls containing all metadata,
including the offsets of regular files inside the NAR. Thus, we don't
need to read the entire NAR. (We do read the entire listing, but
that's generally pretty small. We could use a SQLite DB by borrowing
some more code from nixos-channel-scripts/file-cache.hh.)

This is primarily useful when Hydra is serving files from an S3 binary
cache, in particular when you have giant NARs. E.g. we had some 12 GiB
NARs, so accessing individuals files was pretty slow.
2017-12-07 01:07:07 +01:00
Eelco Dolstra
338f29dbd4 nix ls-{nar,store}: Return offset of files in the NAR if known
E.g.

  $ nix ls-store --json --recursive --store https://cache.nixos.org /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79 \
    | jq .entries.bin.entries.blender.narOffset
  400
2017-12-07 01:07:07 +01:00
Daiderd Jordan
2cb8aaa610
ssh-store: fix length when writing nar
This fixes nix copy and other things that use copyStorePath.
2017-12-06 11:41:08 +01:00
Will Dietz
79f5c296c0 ignore "interrupted" exception in progress callback
Context/discusson:

https://github.com/NixOS/nix/issues/1692#issuecomment-348282301
2017-11-30 12:45:54 -06:00
Eelco Dolstra
b52846ab5b
Show log tail when a remote build fails 2017-11-28 15:36:28 +01:00
Eelco Dolstra
549c3706a5
nix path-info: Show URL of NARs in binary caches 2017-11-24 18:08:50 +01:00
Eelco Dolstra
6cdaa858d0
Propagate flags like --sandbox to the daemon properly 2017-11-21 18:50:56 +01:00
Eelco Dolstra
91a1987607
signed-binary-caches -> require-sigs
Unlike signed-binary-caches (which could only be '*' or ''),
require-sigs is a proper Boolean option. The default is true.
2017-11-20 17:44:07 +01:00
Eelco Dolstra
7a2b64e55c
binary-cache-public-keys -> trusted-public-keys
The name had become a misnomer since it's not only for substitution
from binary caches, but when adding/copying any
(non-content-addressed) path to a store.
2017-11-20 17:32:34 +01:00
Eelco Dolstra
7474ac871b
nix copy: Abbreviate "daemon" 2017-11-20 15:17:11 +01:00
Eelco Dolstra
8956ae1987
Add a "profile" option to S3BinaryCacheStore
This allows specifying the AWS configuration profile to use. E.g.

  nix copy --from s3://my-cache?profile=aws-dev-account /nix/store/cf3isrlqavvd5w7rpky1fa8j9lcnlggm-...
2017-11-15 14:18:20 +01:00
Eelco Dolstra
ec5b04862b
nix sign-paths: Support binary caches 2017-11-14 18:44:05 +01:00
Eelco Dolstra
5773d667ee
nix ls-{nar,store}: Don't abort on missing files 2017-11-14 14:49:06 +01:00
Eelco Dolstra
4db0a9555e
nix ls-{nar,store} --json: Respect -R 2017-11-14 14:31:38 +01:00
Eelco Dolstra
bac8055652
nix ls-{store,nar}: Add --json flag 2017-11-14 14:23:53 +01:00
Eelco Dolstra
9d87d03331
Remove extraneous comment 2017-11-08 16:17:32 +01:00
Eelco Dolstra
513b143cd8
Merge pull request #1650 from copumpkin/darwin-sandbox-unix-socket
Always allow builds to use unix domain sockets in Darwin sandbox
2017-11-08 16:16:42 +01:00
Eelco Dolstra
dc30856141
Merge pull request #1632 from AmineChikhaoui/sigint-copy
run query paths in parallel during nix copy and handle SIGINT
2017-11-06 13:36:28 +01:00
Eelco Dolstra
9ccea31dc2
Merge pull request #1651 from LnL7/darwin-sandbox-getpwuid
Allow getpwuid in the darwin sandbox
2017-11-03 10:55:31 +01:00
Daiderd Jordan
453f675810
Allow getpwuid in the darwin sandbox. 2017-11-03 10:50:49 +01:00
Daniel Peebles
3105679226 Don't freak out if we get a 403 from S3
As far as we're concerned, not being able to access a file just means
the file is missing. Plus, AWS explicitly goes out of its way to
return a 403 if the file is missing and the requester doesn't have
permission to list the bucket.

Also getting rid of an old hack that Eelco said was only relevant
to an older AWS SDK.
2017-11-03 01:31:42 +01:00
Eelco Dolstra
1969f357b7
Add fetchMercurial primop
E.g.

  $ nix eval '(fetchMercurial https://www.mercurial-scm.org/repo/hello)'
  { branch = "default"; outPath = "/nix/store/alvb9y1kfz42bjishqmyy3pphnrh1pfa-source"; rev = "82e55d328c8ca4ee16520036c0aaace03a5beb65"; revCount = 1; shortRev = "82e55d328c8c"; }

  $ nix eval '(fetchMercurial { url = https://www.mercurial-scm.org/repo/hello; rev = "0a04b987be5ae354b710cefeba0e2d9de7ad41a9"; })'
  { branch = "default"; outPath = "/nix/store/alvb9y1kfz42bjishqmyy3pphnrh1pfa-source"; rev = "0a04b987be5ae354b710cefeba0e2d9de7ad41a9"; revCount = 0; shortRev = "0a04b987be5a"; }

  $ nix eval '(fetchMercurial /tmp/unclean-hg-tree)'
  { branch = "default"; outPath = "/nix/store/cm750cdw1x8wfpm3jq7mz09r30l9r024-source"; rev = "0000000000000000000000000000000000000000"; revCount = 0; shortRev = "000000000000"; }
2017-11-01 17:45:32 +01:00
Dan Peebles
bc6b3f7e8f Always allow builds to use unix domain sockets in Darwin sandbox 2017-10-31 15:33:57 +01:00
Eelco Dolstra
197922ea4e
Merge pull request #1646 from copumpkin/optional-sandbox-local-network
Allow optional localhost network access to sandboxed derivations
2017-10-30 18:54:40 +01:00
Dan Peebles
4a4a009f78 Allow optional localhost network access to sandboxed derivations
This will allow bind and connect to 127.0.0.1, which can reduce purity/
security (if you're running a vulnerable service on localhost) but is
also needed for a ton of test suites, so I'm leaving it turned off by
default but allowing certain derivations to turn it on as needed.

It also allows DNS resolution of arbitrary hostnames but I haven't found
a way to avoid that. In principle I'd just want to allow resolving
localhost but that doesn't seem to be possible.

I don't think this belongs under `build-use-sandbox = relaxed` because we
want it on Hydra and I don't think it's the end of the world.
2017-10-30 17:59:12 +01:00
Eelco Dolstra
812e027e1d
Add option allowed-uris
This allows network access in restricted eval mode.
2017-10-30 12:41:49 +01:00
Eelco Dolstra
f9686885be
enable-http2 -> http2 2017-10-30 11:00:59 +01:00
Eelco Dolstra
66ddbef754
fetchurl/fetchTarball: Respect name changes
The computation of urlHash didn't take the name into account, so
subsequent fetchurl calls with the same URL but a different name would
resolve to the same cached store path.
2017-10-30 10:22:58 +01:00
AmineChikhaoui
0f9a7225ab respect SIGINT in nix copy during the paths queries #1629 2017-10-25 16:51:45 +01:00
AmineChikhaoui
54a2cd9ce4 Merge branch 'master' of github.com:NixOS/nix into sigint-copy 2017-10-25 16:50:52 +01:00
Eelco Dolstra
9971d875a4
Fix building on clang
https://hydra.nixos.org/build/62945761
2017-10-25 17:20:47 +02:00
AmineChikhaoui
9f01a3f0a8 attempt to fix #1630: make the queries of store paths run in parallel using a thread pool 2017-10-25 16:13:49 +01:00
Eelco Dolstra
82327e3cc4
exportReferencesGraph: Allow exporting a list of store paths 2017-10-25 15:18:49 +02:00
Eelco Dolstra
3395e3bbc4
Fix exportReferencesGraph in the structured attrs case 2017-10-25 14:08:29 +02:00
Eelco Dolstra
2d5b1b24bf
Pass lists/attrsets to bash as (associative) arrays 2017-10-25 13:01:50 +02:00
Eelco Dolstra
d16fd24973
Allow shorter syntax for chroot stores
You can now say '--store /tmp/nix' instead of '--store local?root=/tmp/nix'.
2017-10-24 15:32:38 +02:00
Eelco Dolstra
3460e4cf00
More progress indicator improvements
In particular, don't show superfluous "fetching path" and "building
path(s)" messages, and show the current round (with --repeat).
2017-10-24 15:32:38 +02:00
Eelco Dolstra
96051dd057
More progress indicator improvements
Fixes #1599.
2017-10-24 14:47:23 +02:00
Eelco Dolstra
be220702a7
Progress indicator: Show on what machine we're building
E.g.

  $ nix build nixpkgs.hello --builders 'root@wendy'
  [1/0/1 built] building hello-2.10 on ssh://root@wendy: checking for minix/config.h... no
2017-10-24 14:24:57 +02:00
Eelco Dolstra
fe9d2f974d
Handle log messages from build-remote
This makes the progress indicator show statuses like "connecting to
'root@machine'".
2017-10-24 14:05:22 +02:00
Eelco Dolstra
0d59f1ca49
nix: Respect -I, --arg, --argstr
Also, random cleanup to argument handling.
2017-10-24 12:58:34 +02:00
Eelco Dolstra
25f32625e2
Remove the remote-builds option
This is superfluous since you can now just set "builders" to empty,
e.g. "--builders ''".
2017-10-24 11:00:16 +02:00
Eelco Dolstra
af241ae7d3
Remove the builder-files option
You can now include files via the "builders" option, using the syntax
"@<filename>". Having only one option makes it easier to override
builders completely.

For backward compatibility, the default is "@/etc/nix/machines", or
"@<filename>" for each file name in NIX_REMOTE_SYSTEMS.
2017-10-24 10:54:43 +02:00
Eelco Dolstra
37fbfffd8e
Pass all settings to build-remote
This ensures that command line flags such as --builders get passed
correctly.
2017-10-23 20:50:28 +02:00
Eelco Dolstra
f32cdc4fab
Turn $NIX_REMOTE into a configuration option 2017-10-23 20:50:28 +02:00
Eelco Dolstra
f31c66d392 Merge pull request #1616 from copumpkin/separate-darwin-sandbox
Shift Darwin sandbox to separate installed files
2017-10-20 12:00:27 +02:00
Eelco Dolstra
ef1d0142a0
Revert "Let's not populate the NAR cache from hydra-queue-runner for now"
This reverts commit 908590dc6c. Since
hydra-server can have a different store URI from hydra-queue-runner
now, we don't really need this.
2017-10-19 12:11:07 +02:00
Eelco Dolstra
908590dc6c
Let's not populate the NAR cache from hydra-queue-runner for now 2017-10-18 15:45:39 +02:00
Eelco Dolstra
75cd75b1ae
Errors writing to the NAR cache should not be fatal 2017-10-18 15:41:14 +02:00
Eelco Dolstra
d8306148e0
Suppress "copying 0 paths" message 2017-10-18 15:02:58 +02:00
Dan Peebles
6e5165b773 Shift Darwin sandbox to separate installed files
This makes it slightly more manageable to see at a glance what in a
build's sandbox profile is unique to the build and what is standard. Also
a first step to factoring more of our Darwin logic into scheme functions
that will allow us a bit more flexibility. And of course less of that
nasty codegen in C++! 😀
2017-10-17 19:15:51 -04:00
Eelco Dolstra
b24b8ef77c BinaryCacheStore::addToStore(): Add NARs to the local cache 2017-10-17 21:39:48 +02:00
Eelco Dolstra
ca580bec35 BinaryCacheStore: Support local caching of NARs
This speeds up commands like "nix cat-store". For example:

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nar-cache /nix/store/i60yncmq6w9dyv37zd2k454g0fkl3arl-systemd-234/etc/udev/udev.conf
  real    0m4.336s

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nar-cache /nix/store/i60yncmq6w9dyv37zd2k454g0fkl3arl-systemd-234/etc/udev/udev.conf
  real    0m0.045s

The primary motivation is to allow hydra-server to serve files from S3
binary caches. Previously Hydra had a hack to do "nix-store -r
<path>", but that fetches the entire closure so is prohibitively
expensive.

There is no garbage collection of the NAR cache yet. Also, the entire
NAR is read when accessing a single member file. We could generate the
NAR listing to provide random access.

Note: the NAR cache is indexed by the store path hash, not the content
hash, so NAR caches should not be shared between binary caches, unless
you're sure that all your builds are binary-reproducible.
2017-10-17 21:15:33 +02:00
Eelco Dolstra
11ba4302e3 Remove BinaryCacheStoreAccessor
Probably as a result of a bad merge in
4b8f1b0ec0, we had both a
BinaryCacheStoreAccessor and a
RemoteFSAccessor. BinaryCacheStore::getFSAccessor() returned the
latter, but BinaryCacheStore::addToStore() checked for the
former. This probably caused hydra-queue-runner to download paths that
it just uploaded.
2017-10-17 20:51:42 +02:00
Eelco Dolstra
1dd29d7aeb
Add option to disable the seccomp filter
I needed this to test ACL/xattr removal in
canonicalisePathMetaData(). Might also be useful if you need to build
old Nixpkgs that doesn't have the required patches to remove
setuid/setgid creation.
2017-10-12 18:21:55 +02:00
Dan Peebles
476493dbf5 Reverse retry logic to retry in all but a few cases
It was getting too much like whac-a-mole listing all the retriable error
conditions, so we now retry by default and list the cases where retrying
is almost certainly hopeless.
2017-10-02 23:22:02 -04:00
Eelco Dolstra
aca4f7dff0
Don't remove Content-Encoding in fetchurl / nix-prefetch-url
Fixes #1568.
2017-09-18 11:07:28 +02:00
Eelco Dolstra
89dc62c174
RemoteStore: Add option to drop old connections from the pool
This is a hack to make hydra-queue-runner free its temproots
periodically, thereby ensuring that garbage collection of the
corresponding paths is not blocked until the queue runner is
restarted.

It would be better if temproots could be released earlier than at
process exit. I started working on a RAII object returned by functions
like addToStore() that releases temproots. However, this would be a
pretty massive change so I gave up on it for now.
2017-09-14 18:10:38 +02:00
Eelco Dolstra
308ecf6361
Show the pid of temporary roots 2017-09-14 15:02:52 +02:00
Eelco Dolstra
da1e4fdfb5
nix-store -q --roots / --gc --print-roots: Print temporary / in-memory roots
For example,

  $ nix-store -q --roots /nix/store/7phd2sav7068nivgvmj2vpm3v47fd27l-patchelf-0.8pre845_0315148
  {temp:1}

denotes that the path is only being kept alive by a temporary root
(i.e. /nix/var/nix/temproots/). Similarly,

  $ nix-store --gc --print-roots
  ...
  {memory:9} -> /nix/store/094gpjn9f15ip17wzxhma4r51nvsj17p-curl-7.53.1

shows that curl is being used by some process.
2017-09-14 14:38:36 +02:00
Eelco Dolstra
3ed8290e53
LegacySSHStore: Include signatures etc. 2017-09-08 16:55:27 +02:00
Eelco Dolstra
6a888ec29a
copyStorePath(): Fill in missing narHash regardless of checkSigs
I don't remember what the reasoning was here, but security is provided
by the signatures, not by whether the hash is provided by the other
store.
2017-09-08 14:48:08 +02:00
Eelco Dolstra
88e6bb76de Communicate the structured log FD to builders
Since we may use a dedicated file descriptor in the future, this
allows us to change it. So builders can do

  if [[ -n $NIX_LOG_FD ]]; then
    echo "@nix { message... }" >&$NIX_LOG_FD
  fi
2017-09-08 11:27:10 +02:00
Eelco Dolstra
1277aab219
Fix abort when the GC thread gets an exception 2017-09-06 11:37:58 +02:00
Eelco Dolstra
0b606aad46
Add automatic garbage collection
Nix can now automatically run the garbage collector during builds or
while adding paths to the store. The option "min-free = <bytes>"
specifies that Nix should run the garbage collector whenever free
space in the Nix store drops below <bytes>. It will then delete
garbage until "max-free" bytes are available.

Garbage collection during builds is asynchronous; running builds are
not paused and new builds are not blocked. However, there also is a
synchronous GC run prior to the first build/substitution.

Currently, no old GC roots are deleted (as in "nix-collect-garbage
-d").
2017-09-05 20:43:42 +02:00
Eelco Dolstra
b932ea58ec
GC: Don't delete own temproots file
Since file locks are per-process rather than per-file-descriptor, the
garbage collector would always acquire a lock on its own temproots
file and conclude that it's stale.
2017-09-05 20:39:57 +02:00
Eelco Dolstra
8215b75d36
Abort curl downloads if there is no progress for 5 minutes
Maybe this will fix the curl hangs on macOS. (We could also use
CURLOPT_TIMEOUT but that seems more of a sledgehammer.)
2017-09-01 16:51:26 +02:00
Eelco Dolstra
bbdf08bc0f
Call queryMissing() prior to building
Without this, substitute info is fetched sequentially, which is
superslow. In the old UI (e.g. nix-build), we call printMissing(),
which calls queryMissing(), thereby preheating the binary cache
cache. But the new UI doesn't do that.
2017-08-31 16:05:07 +02:00
Eelco Dolstra
fd73c1e20a
Add an activity for binary cache queries 2017-08-31 15:25:58 +02:00
Eelco Dolstra
c2154d4c84
Rename a few configuration options
In particular, drop the "build-" and "gc-" prefixes which are
pointless. So now you can say

  nix build --no-sandbox

instead of

  nix build --no-build-use-sandbox
2017-08-31 14:28:25 +02:00
Eelco Dolstra
2cc345b95f
Give activities a verbosity level again
And print them (separately from the progress bar) given sufficient -v
flags.
2017-08-28 19:13:24 +02:00
Eelco Dolstra
cfc8132391
Don't send progress messages to older clients 2017-08-28 18:54:23 +02:00
Eelco Dolstra
fe34b91289
Tunnel progress messages from the daemon to the client
This makes the progress bar work for non-root users.
2017-08-28 18:49:42 +02:00
Eelco Dolstra
94a0548dc4
Simplify 2017-08-25 21:26:37 +02:00
Eelco Dolstra
0ac35b67b8
Allow derivations to update the build phase
So the progress bar can show

  [1/0/1 built, 0.0 MiB DL] building hello-2.10 (configuring): checking whether pread is declared without a macro... yes
2017-08-25 18:04:05 +02:00
Eelco Dolstra
c137c0a5eb
Allow activities to be nested
In particular, this allows more relevant activities ("substituting X")
to supersede inferior ones ("downloading X").
2017-08-25 17:49:40 +02:00
Eelco Dolstra
0e9ddcc306
Restore activity metadata
This allows the progress bar to display "building perl-5.22.3" instead
of "building /nix/store/<hash>-perl-5.22.3.drv".
2017-08-25 15:58:35 +02:00
Eelco Dolstra
1f56235438
Clean up JSON construction 2017-08-21 12:18:46 +02:00
Eelco Dolstra
4af2611bd1
Allow builders to create activities
Actually, currently they can only create download activities. Thus,
downloads by builtins.fetchurl show up in the progress bar.
2017-08-21 12:18:46 +02:00
Eelco Dolstra
b4ed97e3a3
nix optimise-store: Show how much space has been freed 2017-08-16 20:56:03 +02:00
Eelco Dolstra
23b8b7e096
nix optimise-store: Add
This replaces "nix-store --optimise". Main difference is that it has a
progress indicator.
2017-08-16 20:56:03 +02:00
Eelco Dolstra
40bffe0a43
Progress indicator: Cleanup 2017-08-16 20:56:03 +02:00
Eelco Dolstra
dff12b38f9
Progress indicator: More improvements 2017-08-16 20:56:03 +02:00
Eelco Dolstra
bf1f123b09
Progress indicator: Show number of active items 2017-08-16 20:56:03 +02:00
Eelco Dolstra
0e0dcf2c7e
Progress indicator: Unify "copying" and "substituting"
They're the same thing after all.

Example:

  $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped
  [0/1 built, 49/98 copied, 16.3/92.8 MiB DL, 55.8/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/0pl9li1jigcj2dany47hpmn0r3r48wc4nz48v5mqhh426lgz3bz6.nar.xz'
2017-08-16 20:56:03 +02:00
Eelco Dolstra
c36467ad2e
Improve substitution progress indicator
E.g.

  $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped
  [0/1 built, 1/97/98 fetched, 65.8/92.8 MiB DL, 203.2/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/1czm9fk0svacy4h6a3fzkpafi4f7a9gml36kk8cq1igaghbspg3k.nar.xz'
2017-08-16 20:56:02 +02:00
Eelco Dolstra
b29b6feaba
nix copy: Improve progress indicator
It now shows the amount of data copied:

  [8/1038 copied, 160.4/1590.9 MiB copied] copying path '...'
2017-08-16 20:56:02 +02:00
Eelco Dolstra
c5e4404580
nix copy: Revive progress bar 2017-08-16 20:56:02 +02:00
Eelco Dolstra
e4bd42f98f
Disallow SSH host names starting with a dash 2017-08-16 20:55:58 +02:00
Eelco Dolstra
b6ee5e5bf0
Style fix 2017-08-10 13:51:07 +02:00
Eelco Dolstra
af765a8eab
Use /proc/self/fd to efficiently close all FDs on Linux
Issue #1506.
2017-08-09 16:22:05 +02:00
Eelco Dolstra
a2778988f2
Merge branch 'macOS' of https://github.com/davidak/nix 2017-07-31 10:31:51 +02:00
Jörg Thalheim
2fd8f8bb99 Replace Unicode quotes in user-facing strings by ASCII
Relevant RFC: NixOS/rfcs#4

$ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 12:32:45 +01:00
davidak
fcb8d6a7a0 replace "OS X" with "macOS" 2017-07-30 12:28:50 +02:00
Eelco Dolstra
fc3568e263
FD_SETSIZE check: BuildError -> Error
BuildError denotes a permanent build failure, which is not the case
here.
2017-07-20 13:33:13 +02:00
Dmitry Kalinkin
d5e1bffd2a
Do not try to fill fd_set with fd>=FD_SETSIZE
This is UB and causes buffer overflow and crash on linux.
2017-07-18 17:51:50 -04:00
Eelco Dolstra
bf6792c0df
Always use base-16 for hashed mirror lookups
In particular, don't use base-64, which we don't support. (We do have
base-32 redirects for hysterical reasons.)

Also, add a test for the hashed mirror feature.
2017-07-17 13:13:18 +02:00
Eelco Dolstra
49304bae81
Make the hashes mirrors used by builtins.fetchurl configurable
In particular, this allows it to be disabled in our tests.
2017-07-17 13:07:08 +02:00
Eelco Dolstra
4ec6eb1fdf
Fix accidental printError 2017-07-17 11:38:15 +02:00
Eelco Dolstra
766ad5db3b
nix path-info: Show download sizes for binary cache stores
E.g.

  $ nix path-info --json --store https://cache.nixos.org nixpkgs.thunderbird -S
  ...
      "downloadHash": "sha256:1jlixpzi225wwa0f4xdrwrqgi47ip1qpj9p06fyxxg07sfmyi4q0",
      "downloadSize": 43047620,
      "closureDownloadSize": 84745960
    }
  ]
2017-07-14 18:29:10 +02:00
Eelco Dolstra
3908d3929c
nix path-info: Don't barf on invalid paths
Now you get

  [
    {
      "path": "/nix/store/fzvliz4j5xzvnd0w5zgw2l0ksqh578yk-bla",
      "valid": false
    }
  ]
2017-07-14 18:29:10 +02:00
Eelco Dolstra
8e8caf7f3e fetchTarball: Prevent concurrent downloads of the same file
Fixes #849.
2017-07-11 23:21:24 +02:00
Eelco Dolstra
d3713716b6 Merge pull request #1445 from matthewbauer/macos-skip-hardlink
Don’t hardlink disallowed paths in OS X.
2017-07-07 11:05:21 +02:00
Matthew Bauer
72e80c59b5 Don’t hardlink disallowed paths in OS X.
Fixes #1443
2017-07-06 19:30:19 -07:00
Eelco Dolstra
a3dc1e65ab
Add X32 to the seccomp filter
Fixes #1432.
2017-07-04 19:00:51 +02:00
Eelco Dolstra
42c5774e78
Sort substituters by priority
Fixes #1438.
2017-07-04 16:34:53 +02:00
Eelco Dolstra
b7203e853e
getDefaultSubstituters(): Simplify initialisation
As shlevy pointed out, static variables in C++11 have thread-safe
initialisation.
2017-07-04 16:26:48 +02:00
Eelco Dolstra
6cf23c3e8f
Add allow-new-privileges option
This allows builds to call setuid binaries. This was previously
possible until we started using seccomp. Turns out that seccomp by
default disallows processes from acquiring new privileges. Generally,
any use of setuid binaries (except those created by the builder
itself) is by definition impure, but some people were relying on this
ability for certain tests.

Example:

  $ nix build '(with import <nixpkgs> {}; runCommand "foo" {} "/run/wrappers/bin/ping -c 1 8.8.8.8; exit 1")' --no-allow-new-privileges
  builder for ‘/nix/store/j0nd8kv85hd6r4kxgnwzvr0k65ykf6fv-foo.drv’ failed with exit code 1; last 2 log lines:
    cannot raise the capability into the Ambient set
    : Operation not permitted

  $ nix build '(with import <nixpkgs> {}; runCommand "foo" {} "/run/wrappers/bin/ping -c 1 8.8.8.8; exit 1")' --allow-new-privileges
  builder for ‘/nix/store/j0nd8kv85hd6r4kxgnwzvr0k65ykf6fv-foo.drv’ failed with exit code 1; last 6 log lines:
    PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
    64 bytes from 8.8.8.8: icmp_seq=1 ttl=46 time=15.2 ms

Fixes #1429.
2017-07-04 15:48:25 +02:00
Eelco Dolstra
c0015e87af
Support base-64 hashes
Also simplify the Hash API.

Fixes #1437.
2017-07-04 15:07:41 +02:00
Eelco Dolstra
fcca702a96
Replace a few bool flags with enums
Functions like copyClosure() had 3 bool arguments, which creates a
severe risk of mixing up arguments.

Also, implement copyClosure() using copyPaths().
2017-07-03 11:38:08 +02:00
David McFarland
596b0e0a04 Call SetDllDirectory("") after sqlite3 init on cygwin
Cygwin sqlite3 is patched to call SetDllDirectory("/usr/bin") on init, which
affects the current process and is inherited by child processes.  It causes
DLLs to be loaded from /usr/bin/ before $PATH, which breaks all sorts of
things.  A typical failures would be header/lib version mismatches (e.g.
openssl when running checkPhase on openssh).  We'll just set it back to the
default value.

Note that this is a problem with the cygwin version of sqlite3 (currently
3.18.0).  nixpkgs doesn't have the problematic patch.
2017-06-20 10:59:13 -03:00
Eelco Dolstra
a1355917ec
Disable use of virtual hosting in aws-sdk-cpp
Recently aws-sdk-cpp quietly switched to using S3 virtual host URIs
(https://github.com/aws/aws-sdk-cpp/commit/69d9c53882), i.e. it sends
requests to http://<bucket>.<region>.s3.amazonaws.com rather than
http://<region>.s3.amazonaws.com/<bucket>. However this interacts
badly with curl connection reuse. For example, if we do the following:

1) Check whether a bucket exists using GetBucketLocation.
2) If it doesn't, create it using CreateBucket.
3) Do operations on the bucket.

then 3) will fail for a minute or so with a NoSuchBucket exception,
presumably because the server being hit is a fallback for cases when
buckets don't exist.

Disabling the use of virtual hosts ensures that 3) succeeds
immediately. (I don't know what S3's consistency guarantees are for
bucket creation, but in practice buckets appear to be available
immediately.)
2017-06-19 18:51:33 +02:00
Eelco Dolstra
82a0d614cf
Support creating S3 caches in other regions than us-east-1 2017-06-19 18:51:33 +02:00
Eelco Dolstra
b33621d425
Handle S3Errors::RESOURCE_NOT_FOUND from aws-sdk-cpp
This is returned by recent versions. Also handle NO_SUCH_KEY even
though the library doesn't actually return that at the moment.
2017-06-19 18:51:32 +02:00
Eelco Dolstra
1c969611ba
Suppress "will retry in N ms" for non-retriable errors
Newer versions of aws-sdk-cpp call CalculateDelayBeforeNextRetry()
even for non-retriable errors (like NoSuchKey) whih causes log spam in
hydra-queue-runner.
2017-06-19 18:51:32 +02:00
Eelco Dolstra
00aa7c6705
Show aws-sdk-cpp log messages 2017-06-19 18:51:32 +02:00
Eelco Dolstra
1888f7889b
macOS: Ugly hack to make the tests succeed
Sandboxes cannot be nested, so if Nix's build runs inside a sandbox,
it cannot use a sandbox itself. I don't see a clean way to detect
whether we're in a sandbox, so use a test-specific hack.

https://github.com/NixOS/nix/issues/1413
2017-06-19 14:26:05 +02:00
Eelco Dolstra
b5bdfdef73
macOS: Remove flags
In particular, UF_IMMUTABLE (uchg) needs to be cleared to allow the
path to be garbage-collected or optimised.

See https://github.com/NixOS/nixpkgs/issues/25819.
+       the file from being garbage-collected.
2017-06-19 14:19:21 +02:00
Eelco Dolstra
38b7d55af1
Remove redundant debug line 2017-06-14 13:45:38 +02:00
Eelco Dolstra
88b291ffc4
canonicalisePathMetaData(): Ignore security.selinux attribute
Untested, hopefully fixes #1406.
2017-06-14 11:41:03 +02:00
Eelco Dolstra
847f19a5f7
Provide a builtin default for $NIX_SSL_CERT_FILE
This is mostly to ensure that when Nix is started on macOS via a
launchd service or sshd (for a remote build), it gets a certificate
bundle.
2017-06-12 16:44:43 +02:00
Eelco Dolstra
7f5b750b40
Don't run pre-build-hook if we don't have a derivation
This fixes a build failure on OS X when using Hydra or Nix 1.12's
build-remote (since they don't copy the derivation to the build
machine).
2017-06-12 16:07:34 +02:00
Eelco Dolstra
186571965d
Don't show flags from config settings in "nix --help" 2017-06-07 18:41:20 +02:00
Eelco Dolstra
b8283773bd
nix: Make all options available as flags
Thus, instead of ‘--option <name> <value>’, you can write ‘--<name>
<value>’. So

  --option http-connections 100

becomes

  --http-connections 100

Apart from brevity, the difference is that it's not an error to set a
non-existent option via --option, but unrecognized arguments are
fatal.

Boolean options have special treatment: they're mapped to the
argument-less flags ‘--<name>’ and ‘--no-<name>’. E.g.

  --option auto-optimise-store false

becomes

  --no-auto-optimise-store
2017-06-07 16:17:17 +02:00
Eelco Dolstra
c8cc50d46e
Disable the build user mechanism on all platforms except Linux and OS X 2017-06-06 18:52:15 +02:00
Eelco Dolstra
85e93d7b87
Always use the Darwin sandbox
Even with "build-use-sandbox = false", we now use sandboxing with a
permissive profile that allows everything except the creation of
setuid/setgid binaries.
2017-06-06 18:44:49 +02:00
Eelco Dolstra
52fec8dde8
Remove listxattr assertion
It appears that sometimes, listxattr() returns a different value for
the query case (i.e. when the buffer size is 0).
2017-05-31 20:43:47 +02:00
Eelco Dolstra
c96e8cd097
OS X sandbox: Improve builtin sandbox profile
Also, add rules to allow fixed-output derivations to access the
network.

These rules are sufficient to build stdenvDarwin without any
__sandboxProfile magic.
2017-05-31 17:25:51 +02:00
Eelco Dolstra
44f3f8048f OS X sandbox: Don't use a deterministic $TMPDIR
This doesn't work because the OS X sandbox cannot bind-mount
path to a different location.
2017-05-31 14:09:00 +02:00
Eelco Dolstra
c740c3ce50 OS X sandbox: Store .sb file in $TMPDIR rather than the Nix store
The filename used was not unique and owned by the build user, so
builds could fail with

error: while setting up the build environment: cannot unlink ‘/nix/store/99i210ihnsjacajaw8r33fmgjvzpg6nr-bison-3.0.4.drv.sb’: Permission denied
2017-05-31 14:09:00 +02:00
Eelco Dolstra
acc889c821
Darwin sandbox: Use sandbox-defaults.sb
Issue #759.

Also, remove nix.conf from the sandbox since I don't really see a
legitimate reason for builders to access the Nix configuration.
2017-05-30 17:40:12 +02:00
Eelco Dolstra
53a1644187
Darwin sandbox: Disallow creating setuid/setgid binaries
Suggested by Daiderd Jordan.
2017-05-30 17:17:17 +02:00
Eelco Dolstra
fe08d17934
Fix seccomp build failure on clang
Fixes

  src/libstore/build.cc:2321:45: error: non-constant-expression cannot be narrowed from type 'int' to 'scmp_datum_t' (aka 'unsigned long') in initializer list [-Wc++11-narrowing]
2017-05-30 14:37:24 +02:00
Eelco Dolstra
d552d38758
Shut up some clang warnings 2017-05-30 14:35:50 +02:00
Eelco Dolstra
2ac99a32da
Add a seccomp rule to disallow setxattr() 2017-05-30 13:59:24 +02:00
Eelco Dolstra
d798349ede
canonicalisePathMetaData(): Remove extended attributes / ACLs
EAs/ACLs are not part of the NAR canonicalisation. Worse, setting an
ACL allows a builder to create writable files in the Nix store. So get
rid of them.

Closes #185.
2017-05-30 13:47:41 +02:00
Eelco Dolstra
ff6becafa8
Require seccomp only in multi-user setups 2017-05-30 12:37:04 +02:00
Eelco Dolstra
cf93397d3f
Fix seccomp initialisation on i686-linux 2017-05-29 16:14:10 +02:00
Eelco Dolstra
6cc6c15a2d
Add a seccomp filter to prevent creating setuid/setgid binaries
This prevents builders from setting the S_ISUID or S_ISGID bits,
preventing users from using a nixbld* user to create a setuid/setgid
binary to interfere with subsequent builds under the same nixbld* uid.

This is based on aszlig's seccomp code
(47f587700d).

Reported by Linus Heckemann.
2017-05-29 16:14:10 +02:00
Eelco Dolstra
6e01ecd112
Fix nix-copy-closure test
Fixes

  client# error: size mismatch importing path ‘/nix/store/ywf5fihjlxwijm6ygh6s0a353b5yvq4d-libidn2-0.16’; expected 0, got 120264

This is mostly an artifact of the NixOS VM test environment, where the
Nix database doesn't contain hashes/sizes.

http://hydra.nixos.org/build/53537471
2017-05-29 16:08:56 +02:00
Eelco Dolstra
63145be2a5
Fix typo 2017-05-29 15:52:36 +02:00
Eelco Dolstra
edbb105e98
Merge branch 'nar-accessor-tree' of https://github.com/bennofs/nix 2017-05-24 11:04:43 +02:00
Eelco Dolstra
b01d62285c
Improve progress indicator 2017-05-16 16:09:57 +02:00
Benno Fünfstück
a1f428b13b nar-accessor.cc: remove unused member NarIndexer::currentName 2017-05-15 19:41:59 +02:00
Benno Fünfstück
5ee06e612a nar-accessor: non-recursive NarMember::find
This avoids a possible stack overflow if directories are very deeply nested.
2017-05-15 19:34:18 +02:00
Eelco Dolstra
e80257f122
Simplify fixed-output check 2017-05-15 18:50:54 +02:00
Eelco Dolstra
a2d92bb20e
Add --with-sandbox-shell configure flag
And add a 116 KiB ash shell from busybox to the release build. This
helps to make sandbox builds work out of the box on non-NixOS systems
and with diverted stores.
2017-05-15 17:36:32 +02:00
Eelco Dolstra
b30f5784d0
Linux sandbox: Don't barf on invalid paths
This is useful when we're using a diverted store (e.g. "--store
local?root=/tmp/nix") in conjunction with a statically-linked sh from
the host store (e.g. "sandbox-paths =/bin/sh=/nix/store/.../bin/busybox").
2017-05-15 17:36:32 +02:00
Benno Fünfstück
75a1d9849d nar-accessor: use tree, fixes readDirectory missing children
Previously, if a directory `foo` existed and a file `foo-` (where `-` is any character that is sorted before `/`), then  `readDirectory` would return an empty list.

To fix this, we now use a tree where we can just access the children of the node, and do not need to rely on sorting behavior to list the contents of a directory.
2017-05-15 10:23:16 +02:00
Eelco Dolstra
510bc1735b
Add an option for extending the user agent header
This is useful e.g. for distinguishing traffic to a binary cache
(e.g. certain machines can use a different tag in the user agent).
2017-05-11 15:10:22 +02:00
Eelco Dolstra
62d476c7ee
Fix typo 2017-05-11 14:02:03 +02:00
Eelco Dolstra
ea65ae0f9c
Tweak error message 2017-05-11 13:59:47 +02:00
Eelco Dolstra
6f245bf24a
Change the meaning of info.ultimate
It now means "paths that were built locally". It no longer includes
paths that were added locally. For those we don't need info.ultimate,
since we have the content-addressability assertion (info.ca).
2017-05-11 13:31:23 +02:00
Eelco Dolstra
45d7b1a9e9
LocalStore::addToStore(): Check info.narSize
It allowed the client to specify bogus narSize values. In particular,
Downloader::downloadCached wasn't setting narSize at all.
2017-05-11 13:26:03 +02:00
Eelco Dolstra
7689181e4f
Minor cleanup 2017-05-08 15:56:52 +02:00
Eelco Dolstra
00b286275c
Linux sandbox: Fix compatibility with older kernels 2017-05-08 15:42:59 +02:00
Eelco Dolstra
0a97eb6bd7
Remove superfluous #ifdef 2017-05-08 11:27:20 +02:00
Eelco Dolstra
bb50c89319
Make the location of the build directory in the sandbox configurable
This is mostly for use in the sandbox tests, since if the Nix store is
under /build, then we can't use /build as the build directory.
2017-05-05 17:45:22 +02:00
Eelco Dolstra
eba840c8a1
Linux sandbox: Use /build instead of /tmp as $TMPDIR
There is a security issue when a build accidentally stores its $TMPDIR
in some critical place, such as an RPATH. If
TMPDIR=/tmp/nix-build-..., then any user on the system can recreate
that directory and inject libraries into the RPATH of programs
executed by other users. Since /build probably doesn't exist (or isn't
world-writable), this mitigates the issue.
2017-05-04 16:57:03 +02:00
Eelco Dolstra
c5bea16611
LocalStoreAccessor: Fix handling of diverted stores 2017-05-02 15:46:09 +02:00
Eelco Dolstra
7f6837a0f6
Replace $NIX_REMOTE_SYSTEMS with an option "builder-files"
Also, to unify with hydra-queue-runner, allow it to be a list of
files.
2017-05-02 15:46:09 +02:00
Eelco Dolstra
cd4d2705ec
build-remote: Fix fallback to other machines when connecting fails
Opening an SSHStore or LegacySSHStore does not actually establish a
connection, so the try/catch block here did nothing. Added a
Store::connect() method to test whether a connection can be
established.
2017-05-02 15:46:09 +02:00
Eelco Dolstra
1a68710d4d
Add an option for specifying remote builders
This is useful for one-off situations where you want to specify a
builder on the command line instead of having to mess with
nix.machines. E.g.

  $ nix-build -A hello --argstr system x86_64-darwin \
    --option builders 'root@macstadium1 x86_64-darwin'

will perform the specified build on "macstadium1".

It also removes the need for a separate nix.machines file since you
can specify builders in nix.conf directly. (In fact nix.machines is
yet another hack that predates the general nix.conf configuration
file, IIRC.)

Note: this option is supported by the daemon for trusted users. The
fact that this allows trusted users to specify paths to SSH keys to
which they don't normally have access is maybe a bit too much trust...
2017-05-02 15:42:43 +02:00
Eelco Dolstra
ebc9f36a81
Factor out machines.conf parsing
This allows hydra-queue-runner to use it.
2017-05-02 13:17:37 +02:00
Eelco Dolstra
174b68a2a2
build-hook: If there are no machines defined, quit permanently 2017-05-02 12:16:29 +02:00
Eelco Dolstra
feefcb3a98
build-remote: Ugly hackery to get build logs to work
The build hook mechanism expects build log output to go to file
descriptor 4, so do that.
2017-05-02 12:02:23 +02:00
Eelco Dolstra
3a5f04f48c
build-remote: Don't require signatures
This restores the old behaviour.
2017-05-01 20:03:25 +02:00
Eelco Dolstra
031d70e500
Support arbitrary store URIs in nix.machines
For backwards compatibility, if the URI is just a hostname, ssh://
(i.e. LegacySSHStore) is prepended automatically.

Also, all fields except the URI are now optional. For example, this is
a valid nix.machines file:

  local?root=/tmp/nix

This is useful for testing the remote build machinery since you don't
have to mess around with ssh.
2017-05-01 17:35:30 +02:00
Eelco Dolstra
deac171925
Implement LegacySSHStore::buildDerivation()
This makes LegacySSHStore usable by build-remote and
hydra-queue-runner.
2017-05-01 17:30:16 +02:00
Eelco Dolstra
3f5b98e65a
Chomp log output from the build hook 2017-05-01 17:30:16 +02:00
Eelco Dolstra
d7653dfc6d
Remove $NIX_BUILD_HOOK and $NIX_CURRENT_LOAD
This is to simplify remote build configuration. These environment
variables predate nix.conf.

The build hook now has a sensible default (namely build-remote).

The current load is kept in the Nix state directory now.
2017-05-01 17:30:16 +02:00
Eelco Dolstra
ca9f589a93
build-remote: Don't copy the .drv closure
Since build-remote uses buildDerivation() now, we don't need to copy
the .drv file anymore. This greatly reduces the set of input paths
copied to the remote side (e.g. from 392 to 51 store paths for GNU
hello on x86_64-darwin).
2017-05-01 17:30:16 +02:00
Eelco Dolstra
b986c7f8b1
Pass verbosity level to build hook 2017-05-01 14:43:14 +02:00
Eelco Dolstra
227a48f86f
Reduce severity of EMLINK warnings
Fixes #1357.
2017-05-01 14:36:56 +02:00
Eelco Dolstra
0dddcf867a
Add a dummy Store::buildPaths() method
This default implementation of buildPaths() does nothing if all
requested paths are already valid, and throws an "unsupported
operation" error otherwise. This fixes a regression introduced by
c30330df6f in binary cache and legacy
SSH stores.
2017-05-01 13:43:34 +02:00
Eelco Dolstra
a1a5e63e14
Fix brainfart 2017-04-28 16:21:54 +02:00
Shea Levy
4bc00760f9 Add Store nesting to fix import-from-derivation within filterSource 2017-04-26 14:15:47 -04:00
Eelco Dolstra
1bb87c0487
Remove debug statement 2017-04-24 15:01:28 +02:00
Eelco Dolstra
749696e71c
Detect lsof
Also, don't use lsof on Linux since it's not needed.

Fixes #1328.
2017-04-20 19:11:45 +02:00
Eelco Dolstra
efa4bdbfcd
Improve nix show-config --json
In particular, show descriptions. This could be used for manpage
generation etc.
2017-04-20 17:34:47 +02:00
Eelco Dolstra
4410e9d995
Setting: Remove "Tag" template argument 2017-04-20 16:52:53 +02:00
Eelco Dolstra
f05d5f89ff
Read per-user settings from ~/.config/nix/nix.conf 2017-04-20 14:58:16 +02:00
Eelco Dolstra
562585e901
binary-caches-parallel-connections -> http-connections 2017-04-20 14:04:00 +02:00
Eelco Dolstra
76cb3c702c
Reimplement trusted-substituters (aka trusted-binary-caches) 2017-04-20 13:41:29 +02:00
Eelco Dolstra
9cc8047f44
Reimplement connect-timeout
Fixes #1339.
2017-04-19 14:54:52 +02:00
Eelco Dolstra
f8a2e8a552
Shut up some warnings 2017-04-14 14:42:20 +02:00
Eelco Dolstra
dd3714f6ef
Doh 2017-04-14 14:42:08 +02:00
Eelco Dolstra
ba9ad29fdb
Convert Settings to the new config system
This makes all config options self-documenting.

Unknown or unparseable config settings and --option flags now cause a
warning.
2017-04-13 20:53:23 +02:00
Eelco Dolstra
1860070548
Merge branch 'rework-options' of https://github.com/copumpkin/nix 2017-04-13 16:15:51 +02:00
Eelco Dolstra
2040240e23
Add a Config class to simplify adding configuration settings
The typical use is to inherit Config and add Setting<T> members:

  class MyClass : private Config
  {
    Setting<int> foo{this, 123, "foo", "the number of foos to use"};
    Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"};

    MyClass() : Config(readConfigFile("/etc/my-app.conf"))
    {
      std::cout << foo << "\n"; // will print 123 unless overriden
    }
  };

Currently, this is used by Store and its subclasses for store
parameters. You now get a warning if you specify a non-existant store
parameter in a store URI.
2017-04-13 16:03:31 +02:00
Eelco Dolstra
1fe1976e0d Merge pull request #1316 from copumpkin/nix-retries-default
Default to 5 download retries
2017-04-10 15:58:57 +02:00
Dan Peebles
d1fdade755 Add CURLE_WRITE_ERROR as a transient error condition
We've observed it failing downloads in the wild and retrying the same URL
a few moments later seemed to fix it.
2017-04-10 09:28:44 -04:00
Dan Peebles
e43e8be8e7 Default to 5 download retries
This should help certain downloaders that don't request anything special
for the number of retries, like nix-channel.
2017-04-10 09:22:24 -04:00
Eelco Dolstra
105f8ffc98
Minor cleanup
Also, possible fix for #1310 on 32-bit systems.
2017-04-10 11:27:33 +02:00
Eelco Dolstra
95295482ea
Allow "auto" as a store URI
Using the empty string is likely to be ambiguous in some contexts.
2017-04-10 11:27:29 +02:00
Dan Peebles
98283915f5 Retry downloads on transient SSL errors too 2017-04-06 18:18:43 +00:00
Eelco Dolstra
ba20730b3f
Implement RemoteStore::queryMissing()
This provides a significant speedup, e.g. 64 s -> 12 s for

  nix-build --dry-run -I nixpkgs=channel:nixos-16.03 '<nixpkgs/nixos/tests/misc.nix>' -A test

on a cold local and CloudFront cache.

The alternative is to use lots of concurrent daemon connections but
that seems wasteful.
2017-04-06 18:40:19 +02:00
Eelco Dolstra
6b5e271163
Add a method to allow hydra-queue-runner to flush the path info cache 2017-04-06 15:22:37 +02:00
Eelco Dolstra
256940fc48
nix-daemon: Disable path info cache
This is useless because the client also caches path info, and can
cause problems for long-running clients like hydra-queue-runner
(i.e. it may return cached info about paths that have been
garbage-collected).
2017-04-06 14:30:31 +02:00
Eelco Dolstra
8decb07c31
Allow default sandbox paths to be overriden
E.g. you can now redirect /etc/resolv.conf to a different file.
2017-04-04 17:54:16 +02:00
Eelco Dolstra
488792a87d
Make /var/run/nscd/socket optional
Not every distribution uses nscd.
2017-04-04 17:40:50 +02:00
Eelco Dolstra
29d35805c6
Sandbox: Fix /dev/ptmx on recent kernels
This fixes "No such file or directory" when opening /dev/ptmx
(e.g. http://hydra.nixos.org/build/51094249).

The reason appears to be some changes to /dev/ptmx / /dev/pts handling
between Linux 4.4 and 4.9. See
https://patchwork.kernel.org/patch/7832531/.

The fix is to go back to mounting a proper /dev/pts instance inside
the sandbox. Happily, this now works inside user namespaces, even for
unprivileged users. So

  NIX_REMOTE=local?root=/tmp/nix nix-build \
    '<nixpkgs/nixos/tests/misc.nix>' -A test

works for non-root users.

The downside is that the fix breaks sandbox builds on older kernels
(probably pre-4.6), since mounting a devpts fails inside user
namespaces for some reason I've never been able to figure out. Builds
on those systems will fail with

  error: while setting up the build environment: mounting /dev/pts: Invalid argument

Ah well.
2017-03-31 18:20:19 +02:00
Shea Levy
d299bd710a Merge branch 'builtins.exec' 2017-03-31 11:22:39 -04:00
Eelco Dolstra
e1509adbbb
Retry curl error 16 2017-03-31 15:50:11 +02:00
Shea Levy
0bb8db257d Add exec primop behind allow-unsafe-native-code-during-evaluation.
Execute a given program with the (optional) given arguments as the
user running the evaluation, parsing stdout as an expression to be
evaluated.

There are many use cases for nix that would benefit from being able to
run arbitrary code during evaluation, including but not limited to:

* Automatic git fetching to get a sha256 from a git revision
* git rev-parse HEAD
* Automatic extraction of information from build specifications from
  other tools, particularly language-specific package managers like
  cabal or npm
* Secrets decryption (e.g. with nixops)
* Private repository fetching

Ideally, we would add this functionality in a more principled way to
nix, but in the mean time 'builtins.exec' can be used to get these
tasks done.

The primop is only available when the
'allow-unsafe-native-code-during-evaluation' nix option is true. That
flag also enables the 'importNative' primop, which is strictly more
powerful but less convenient (since it requires compiling a plugin
against the running version of nix).
2017-03-30 08:04:21 -04:00
Eelco Dolstra
c60715e937
Ignore broken "Deriver: unknown-deriver" fields in .narinfo
These were generated by a legacy tool.
2017-03-28 13:08:13 +02:00
Eelco Dolstra
895a74a814
LocalFSStore::getBuildLog(): Handle corrupted logs 2017-03-21 19:23:35 +01:00
Eelco Dolstra
e1e49c58e1
Only use cache.nixos.org when the store is /nix/store
This is consistent with the behaviour of the old
download-from-binary-cache substituter.
2017-03-21 17:59:18 +01:00
Eelco Dolstra
ecbc3fedd3
Require signatures by default
This corresponds to the NixOS default.
2017-03-21 15:10:48 +01:00
Eelco Dolstra
4bb38591e5
Restore cache.nixos.org as the default substituter
Fixes #1283.
2017-03-21 15:06:46 +01:00
Eelco Dolstra
f8b84a3b8c
Move istringstream_nocopy to a separate file 2017-03-21 14:43:03 +01:00
Eelco Dolstra
3229f85585
Honor $NIX_SSHOPTS again
NixOps needs this.
2017-03-21 14:35:50 +01:00
Eelco Dolstra
20df50d8e1
Merge branch 'darwin-s3-binary-cache-store' of https://github.com/shlevy/nix 2017-03-20 17:46:35 +01:00
Eelco Dolstra
287084d688
ssh:// -> ssh-ng://, legacy-ssh:// -> ssh:// 2017-03-16 14:19:32 +01:00
Eelco Dolstra
c5b83d8913
copyPaths(): Use queryValidPaths() to reduce SSH latency 2017-03-16 13:50:01 +01:00
Eelco Dolstra
91d67692cf
copyPaths(): Don't query path info for a path the target already has
For example, this cuts "nix-copy-closure --from" on a NixOS system
closure from 15.9s to 0.5s.
2017-03-16 12:05:51 +01:00
Eelco Dolstra
ea7fa88131
LegacySSHStore: Provide a faster implementation of computeFSClosure()
This avoids the latency of the standard implementation, which can make
a huge difference (e.g. 16.5s -> 0.5s on a NixOS system closure).
2017-03-16 11:44:52 +01:00
Eelco Dolstra
7a716ef2a5
Fix nix-copy-closure --to 2017-03-16 10:58:48 +01:00
Eelco Dolstra
0ec7f47b00
Remove "killing process <pid>" messages
They convey no useful information.
2017-03-16 10:52:28 +01:00
Eelco Dolstra
3f35612c04
More precise compression settings 2017-03-15 17:20:19 +01:00
Eelco Dolstra
25dff2b7db
runProgram(): Distinguish between empty input and no input
For example, if we call brotli with an empty input, it shouldn't read
from the caller's stdin.
2017-03-15 16:50:19 +01:00
Eelco Dolstra
042975ea8e
Compress NAR listings using the "text-compression" method
So if "text-compression=br", the .ls file in S3 will get a
Content-Encoding of "br". Brotli appears to compress better than xz
for this kind of file and is natively supported by browsers.
2017-03-15 16:50:19 +01:00
Eelco Dolstra
45c70382ac
S3BinaryCacheStore: Set Content-Type
This is necessary for serving log files to browsers.
2017-03-15 16:50:19 +01:00
Eelco Dolstra
8b1d65bebe
S3BinaryCacheStore: Support compression of narinfo and log files
You can now set the store parameter "text-compression=br" to compress
textual files in the binary cache (i.e. narinfo and logs) using
Brotli. This sets the Content-Encoding header; the extension of
compressed files is unchanged.

You can separately specify the compression of log files using
"log-compression=br". This is useful when you don't want to compress
narinfo files for backward compatibility.
2017-03-15 16:49:28 +01:00
Eelco Dolstra
e8186085e0
Add support for brotli compression
Build logs on cache.nixos.org are compressed using Brotli (since this
allows them to be decompressed automatically by Chrome and Firefox),
so it's handy if "nix log" can decompress them.
2017-03-15 16:49:06 +01:00
Eelco Dolstra
532d73d5d8
BinaryCacheStore: Implement getBuildLog()
We assume that build logs are stored under log/<drv>, e.g.

  /nix/store/q7ab198v13p0f8x8wgnd75dva7d5mip6-friday-devil-0.1.1.1.drv

maps to

  https://cache.nixos.org/log/q7ab198v13p0f8x8wgnd75dva7d5mip6-friday-devil-0.1.1.1.drv
2017-03-15 16:48:29 +01:00
Eelco Dolstra
0afeb7f51e
Store: Add a method for getting build logs
This allows various Store implementations to provide different ways to
get build logs. For example, BinaryCacheStore can get the build logs
from the binary cache.

Also, remove the log-servers option since we can use substituters for
this.
2017-03-15 16:48:29 +01:00
Shea Levy
96443e94a1 Merge branch 'configurable-aws-region' 2017-03-12 10:01:11 -04:00
Shea Levy
93f863be96 Add option to disable import-from-derivation completely, even if the drv is already realized 2017-03-08 08:46:12 -05:00
Shea Levy
3cc18d3753 Properly set the caFile for aws-sdk-cpp s3 2017-03-06 14:30:35 -05:00
Shea Levy
4fc30922cf istringstream_nocopy: Implement in a standards-compliant way.
Fixes the problem mentioned in e6a61b8da7

See #1135
2017-03-06 13:03:02 -05:00
Shea Levy
5789eaa3f4 Add aws-region param to S3 store URLs 2017-03-03 16:12:17 -05:00
Eelco Dolstra
d1158bb816
Cache connection failures 2017-03-03 19:36:01 +01:00
Eelco Dolstra
8490ee37a6
SSHMaster: Make thread-safe 2017-03-03 19:28:27 +01:00
Eelco Dolstra
d3eb1cf3bb
build-remote: Don't use a SSH master
This is unnecessary because we make only one connection.
2017-03-03 19:23:20 +01:00
Eelco Dolstra
577ebeaefb
Improve SSH handling
* Unify SSH code in SSHStore and LegacySSHStore.

* Fix a race starting the SSH master. We now wait synchronously for
  the SSH master to finish starting. This prevents the SSH clients
  from starting their own connections.

* Don't use a master if max-connections == 1.

* Add a "max-connections" store parameter.

* Add a "compress" store parameter.
2017-03-03 19:05:50 +01:00
Eelco Dolstra
2f992692e2
Fix fatal "broken pipe" error when $NIX_BUILD_HOOK is missing 2017-03-03 15:40:06 +01:00
Eelco Dolstra
fa125b9b28
TeeSink: Pre-reserve string space
When receiving a very large file, this can prevent the string from
having tobe copied, which temporarily doubles memory consumption.
2017-03-01 16:16:04 +01:00