Commit graph

5990 commits

Author SHA1 Message Date
Eelco Dolstra
cbe0bb29f4
Merge pull request #4035 from Ma27/url-attr
libfetchers/github: allow `url` attribute
2020-09-21 17:26:36 +02:00
Maximilian Bosch
56f1e0df05
libfetchers/github: rename url to host 2020-09-21 16:29:08 +02:00
Robert Hensing
ca30abb3fb Document addCAToStore/addToStoreFromDump source drainage
Also checked that all usages satisfy the requirement and
removed dead code.
2020-09-21 07:55:47 +02:00
Robert Hensing
fa08db5c4c wopAddToStore: return ValidPathInfo
A ValidPathInfo is created anyway. By returning it we can save a
roundtrip and we have a nicer interface.
2020-09-21 07:55:47 +02:00
Robert Hensing
7c68264085 wopAddToStore: add RepairFlag 2020-09-21 07:55:47 +02:00
Robert Hensing
fbf509c113 parseContentAddressMethodPrefix: use string_view
Co-authored-by: John Ericson <git@JohnEricson.me>
2020-09-21 07:55:47 +02:00
Robert Hensing
8279178b07 Move FramedSink next to FramedSource 2020-09-21 07:55:47 +02:00
Robert Hensing
ecc8088cb7 wopAddToStore: Throw to clarify unused refs
Co-authored-by: John Ericson <git@JohnEricson.me>
2020-09-21 07:55:47 +02:00
Robert Hensing
c602ebfb34 Refactor wopAddToStore to make wopAddTextToStore obsolete 2020-09-21 07:55:45 +02:00
Robert Hensing
e34fe47d0c Overhaul wopAddToStore 2020-09-21 07:54:05 +02:00
Marwan Aljubeh
e40772cd35 Lint issue: replacing tabs with spaces 2020-09-18 17:18:45 +01:00
Marwan Aljubeh
c00e078343 Add a nix.conf option for allowing a symlinked store 2020-09-18 17:10:39 +01:00
Bryan Richter
5fe375a8f1
nix-prefetch-url: Add --executable flag
pkgs.fetchurl supports an executable argument, which is especially nice
when downloading a large executable. This patch adds the same option to
nix-prefetch-url.

I have tested this to work on the simple case of prefetching a little
executable:

1. nix-prefetch-url --executable https://my/little/script
2. Paste the hash into a pkgs.fetchurl-based package, script-pkg.nix
3. Delete the output from the store to avoid any misidentified artifacts
4. Realise the package script-pkg.nix
5. Run the executable

I repeated the above while using --name, as well.

I suspect --executable would have no meaningful effect if combined with
--unpack, but I have not tried it.
2020-09-18 19:09:45 +03:00
Maximilian Bosch
2bcf8cbe7a
libfetchers/github: allow url attribute
Since 108debef6f we allow a
`url`-attribute for the `github`-fetcher to fetch tarballs from
self-hosted `gitlab`/`github` instances.

However it's not used when defining e.g. a flake-input

    foobar = {
        type = "github";
        url = "gitlab.myserver";
        /* ... */
    }

and breaks with an evaluation-error:

    error: --- Error --------------------------------------nix
    unsupported input attribute 'url'
    (use '--show-trace' to show detailed location information)

This patch allows flake-inputs to be fetched from self-hosted instances
as well.
2020-09-18 14:10:45 +02:00
Eelco Dolstra
958bf57123 nix build: find() -> get()
find() returns an iterator so "!attr" doesn't work.
2020-09-18 13:10:42 +02:00
Robert Hensing
14b30b3f3d Move FramedSource and FramedSink, extract withFramedSink 2020-09-17 20:21:04 +02:00
Robert Hensing
dfa547c6a8 Add ContentAddressMethod and parse/render it 2020-09-17 20:21:04 +02:00
Robert Hensing
29c82ccc77 Add Source.drainInto(Sink) 2020-09-17 20:21:04 +02:00
Robert Hensing
9ee3122ec7 Remove redundant import 2020-09-17 20:21:04 +02:00
Robert Hensing
3f93bc0d39 Typo 2020-09-17 20:21:04 +02:00
John Ericson
b7df353f27 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-17 16:33:10 +00:00
Eelco Dolstra
649d3aaf24
Merge pull request #3829 from obsidiansystems/remove-storetype-delegate-regStore
Remove storetype delegate reg store -- contains #3736
2020-09-17 13:55:01 +02:00
Eelco Dolstra
b94a35ef40
Merge pull request #4027 from tweag/fix-gc-of-ca-derivations
Fix garbage collection of CA derivations
2020-09-17 13:46:26 +02:00
regnat
520895b1da Fix garbage collection of CA derivations
Fix #4026
2020-09-17 13:36:58 +02:00
Eelco Dolstra
787469c7b6 Remove corepkgs/unpack-channel.nix 2020-09-17 09:41:02 +02:00
Eelco Dolstra
10d1865f5f Remove corepkgs/derivation.nix 2020-09-17 09:41:02 +02:00
John Ericson
f60b380a7f Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore 2020-09-16 22:35:24 +00:00
John Ericson
2741fffa35 Ensure resolved CA derivations are written
so we can link outputs to deriver and thus properly cache.
2020-09-16 17:57:32 +00:00
John Ericson
7fdbb377ba Start to fix floating CA + remote building 2020-09-16 17:50:48 +00:00
John Ericson
c5ccebae00 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-16 17:50:40 +00:00
Greg Hale
a303c0b6dc Fetch commits from github/gitlab using Auth header
`nix flake info` calls the github 'commits' API, which requires
authorization when the repository is private. Currently this request
fails with a 404.

This commit adds an authorization header when calling the 'commits' API.
It also changes the way that the 'tarball' API authenticates, moving the
user's token from a query parameter into the Authorization header.

The query parameter method is recently deprecated and will be disallowed
in November 2020. Using them today triggers a warning email.
2020-09-16 13:46:48 -04:00
Eelco Dolstra
5080d4e7b2 Merge branch 'document-store-options' of https://github.com/tweag/nix 2020-09-16 17:02:30 +02:00
Eelco Dolstra
0066ef6c59 Fix doc generation 2020-09-16 16:56:28 +02:00
Eelco Dolstra
2eacc1bc00 builtins.toFile: Fix indentation 2020-09-16 14:18:46 +02:00
Théophane Hufschmitt
77a0e2c5be
Remove useless exception copy
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-09-16 14:00:21 +02:00
regnat
d72927aa7a Fix the s3 store
Add some necessary casts in the initialisation of the store's config
2020-09-16 13:53:09 +02:00
regnat
e0817cbcdc Don't include nlohmann/json.hpp in config.hh
Instead make a separate header with the template implementation of
`BaseSetting<T>::toJSONObj` that can be included where needed
2020-09-16 13:53:09 +02:00
regnat
93c0e14a30 Include the full nlohmann/json header in config.hh
It is apparently required for using `toJSONObject()`, which we do inside
the header file (because it's in a template).

This was accidentally working when building Nix itself (presumably because
`config.hh` was always included after `nlohman/json.hpp`) but caused a
(pretty dirty) build failure in the perl bindings package.
2020-09-16 13:53:09 +02:00
regnat
fc2d31c423 Add (StoreConfig*) casts to work around a GCC bug
Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80431 that was
already there in the code but was accidentally removed in the last
commits
2020-09-16 13:53:09 +02:00
regnat
a1e82ba450 fixup! Add a default value for the settings 2020-09-16 13:53:09 +02:00
regnat
888f7afe9f Fix build issues with gcc 2020-09-16 13:53:09 +02:00
regnat
634cb2a5ae Add a markdown output to nix describe-stores 2020-09-16 13:53:09 +02:00
regnat
b73adacc1e Add a name to the stores
So that it can be printed by `nix describe-stores`
2020-09-16 13:53:09 +02:00
regnat
f24f0888f9 Document the new store hierarchy 2020-09-16 13:53:09 +02:00
regnat
d65962db4d Make uri schemes grammar more RFC-compliant
Allow `-` and `.` in the RFC schemes as stated by
[RFC3986](https://tools.ietf.org/html/rfc3986#section-3.1).

Practically, this is needed so that `ssh-ng` is a valid URI scheme
2020-09-16 13:53:09 +02:00
regnat
29a632386e fixup! Make the store plugins more introspectable 2020-09-16 13:53:09 +02:00
regnat
1129913c4e fixup! Correctly call all the parent contructors of the stores 2020-09-16 13:53:09 +02:00
regnat
7f103dcddd Properly filter the stores according to their declared uriSchemes
When opening a store, only try the stores whose `uriSchemes()` include
the current one
2020-09-16 13:53:09 +02:00
regnat
5895184df4 Correctly call all the parent contructors of the stores
Using virtual inheritance means that only the default constructors of
the parent classes will be called, which isn't what we want
2020-09-16 13:53:09 +02:00
regnat
dae39f0a7a Make nix describe-stores functional
Using the `*Config` class hierarchy
2020-09-16 13:53:08 +02:00
regnat
22afa8fb4d Separate store configs from the implems
Rework the `Store` hierarchy so that there's now one hierarchy for the
store configs and one for the implementations (where each implementation
extends the corresponding config). So a class hierarchy like

```
StoreConfig-------->Store
    |                 |
    v                 v
SubStoreConfig----->SubStore
    |                 |
    v                 v
SubSubStoreConfig-->SubSubStore
```

(with virtual inheritance to prevent DDD).

The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-16 13:53:08 +02:00
regnat
aa4eac3788 fixup! Separate the instantiation and initialisation of the stores 2020-09-16 13:53:08 +02:00
regnat
35042c9623 Add a default value for the settings
The default value is initialized when creating the setting and unchanged
after that
2020-09-16 13:53:08 +02:00
regnat
3c525d1590 Complete the toJSON instance for Setting<T>
Don't let it just contain the value, but also the other fields of the
setting (description, aliases, etc..)
2020-09-16 13:53:08 +02:00
regnat
3b57181f8e Separate the instantiation and initialisation of the stores
Add a new `init()` method to the `Store` class that is supposed to
handle all the effectful initialisation needed to set-up the store.
The constructor should remain side-effect free and just initialize the
c++ data structure.

The goal behind that is that we can create “dummy” instances of each
store to query static properties about it (the parameters it accepts for
example)
2020-09-16 13:53:08 +02:00
regnat
fa32560169 Fix the registration of stores 2020-09-16 13:53:08 +02:00
regnat
7d5bdf8b56 Make the store plugins more introspectable
Directly register the store classes rather than a function to build an
instance of them.
This gives the possibility to introspect static members of the class or
choose different ways of instantiating them.
2020-09-16 13:53:08 +02:00
John Ericson
3a5cdd737c Rename Derivation::pathOpt to Derivation::path
We no longer need the `*Opt` to disambiguate.
2020-09-15 15:21:39 +00:00
John Ericson
6387550d58 Get rid of confusing std::optional<bool> for validity 2020-09-15 15:19:45 +00:00
John Ericson
c4bf219b55 Don't link deriver until after any delayed exception is thrown
Otherwise, we will associate fixed-output derivations with outputs that
they did indeed produce, but which had the wrong hash. That's no good.
2020-09-15 14:28:06 +00:00
John Ericson
3ba552b245 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-15 14:17:06 +00:00
John Ericson
c08c9f08c7 Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore 2020-09-15 14:08:35 +00:00
Eelco Dolstra
2a80170920
Merge pull request #4014 from tweag/fix-queryDrvOutputMaps-for-old-daemons
gracefully handle old daemon versions
2020-09-15 11:25:19 +02:00
regnat
057c6203b5 gracefully handle old daemon versions
Add a fallback path in `queryPartialDerivationOutputMap` for daemons
that don't support it.

Also upstreams a couple methods from `SSHStore` to `RemoteStore` as this
is needed to handle the fallback path.
2020-09-15 09:58:01 +02:00
regnat
250f8a4bba Escape ${ in strings when printing Nix expressions
Otherwise the result of the printing can't be parsed back correctly by
Nix (because the unescaped `${` will be parsed as the begining of an
anti-quotation).

Fix #3989
2020-09-14 17:19:25 +02:00
Eelco Dolstra
a59e77d9e5 nix-daemon: Lower verbosity of restricted setting warning
Fixes #3992.
2020-09-14 13:48:51 +02:00
Eelco Dolstra
35ba092019
Merge pull request #4005 from Infinisil/fix-autoArgs
Fix auto argument passing for more auto arguments than formals
2020-09-14 13:31:23 +02:00
aszlig
525b38eee8
Fix unspecified behaviour in readStorePathCAMap
When deploying a Hydra instance with current Nix master, most builds
would not run because of errors like this:

  queue monitor: error: --- Error --- hydra-queue-runner
  error: --- UsageError --- nix-daemon
  not a content address because it is not in the form '<prefix>:<rest>': /nix/store/...-somedrv

The last error message is from parseContentAddress, which expects a
colon-separated string, however what we got here is a store path.

Looking at the worker protocol, the following message sent to the Nix
daemon caused the error above:

  0x1E -> wopQuerySubstitutablePathInfos
  0x01 -> Number of paths
  0x16 -> Length of string
  "/nix/store/...-somedrv"
  0x00 -> Length of string
  ""

Looking at writeStorePathCAMap, the store path is indeed the first field
that's transmitted. However, readStorePathCAMap expects it to be the
*second* field *on my machine*, since expression evaluation order is a
classic form of unspecified behaviour[1] in C++.

This has been introduced in https://github.com/NixOS/nix/pull/3689,
specifically in commit 66a62b3189.

[1]: https://en.wikipedia.org/wiki/Unspecified_behavior#Order_of_evaluation_of_subexpressions

Signed-off-by: aszlig <aszlig@nix.build>
2020-09-13 02:40:51 +02:00
Silvan Mosberger
8dbd57a6a5
Fix auto argument passing for more auto arguments than formals
The change in 626200713b didn't account
for when the number of auto arguments is bigger than the number of
formal arguments. This causes the following:

  $ nix-instantiate --eval -E '{ ... }@args: args.foo' --argstr foo foo
  nix-instantiate: src/libexpr/attr-set.hh:55: void nix::Bindings::push_back(const nix::Attr&): Assertion `size_ < capacity_' failed.
  Aborted (core dumped)
2020-09-11 20:41:51 +02:00
Gabriel Gonzalez
ee5906243a Add nix-shell support for preserving PS1
Fixes https://github.com/NixOS/nix/issues/1268

`nix-shell` will now preserve `PS1` if the `NIX_SHELL_PRESERVE_PROMPT`
environment variable is set.
2020-09-04 20:05:43 -07:00
John Ericson
98dfd7531d Fix querying outputs for CA derivations some more
If we resolve using the known path of a derivation whose output we
didn't have, we previously blew up. Now we just fail gracefully,
returning the map of all outputs unknown.
2020-09-04 18:33:58 +00:00
John Ericson
075d399e3f Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-09-04 16:04:35 +00:00
John Ericson
5aed6f9b25 Document mkOutputString 2020-09-04 15:58:42 +00:00
John Ericson
e9fad3006b Fix some of the issues raised by @edolstra
- More and better comments

 - The easier renames
2020-09-04 15:18:25 +00:00
John Ericson
e86dd59dcc
Apply suggestions from code review
Thanks!

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-09-04 10:48:50 -04:00
John Ericson
c9f1ed912c Don't chmod symlink before moving outputs around
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-09-04 14:41:53 +00:00
John Ericson
aad4abcc9c Fix floating CA tests
We will sometimes try to query the outputs of derivations we can't
resolve. That's fine; it just means we don't know what those outputs are
yet.
2020-09-04 01:17:38 +00:00
John Ericson
975a47f7fe Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-09-03 22:09:04 +00:00
John Ericson
e7d93e7ece Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-03 15:43:17 +00:00
Eelco Dolstra
8a945d6ddb
Remove gc-check-reachability 2020-09-03 11:30:15 +02:00
Eelco Dolstra
82b77a7726
addPermRoot(): Remove indirect flag 2020-09-03 11:29:34 +02:00
Eelco Dolstra
00d25e8457
Remove the --indirect flag
All GC roots are now indirect.
2020-09-03 11:22:00 +02:00
Eelco Dolstra
b07167be5a
createGeneration(): Always create an indirect root
This means profiles outside of /nix/var/nix/profiles don't get
garbage-collected. It also means we don't need to scan
/nix/var/nix/profiles for GC roots anymore, except for compatibility
with previously existing generations.
2020-09-03 11:13:53 +02:00
Eelco Dolstra
b74f5cdd23
createGeneration(): Take a StorePath 2020-09-03 11:06:56 +02:00
Eelco Dolstra
94a043ff3b
EvalCache: Fix caching of strings
This was broken in 50f13b06fb. Once
again it turns out that putting a bool in a std::variant is a bad
idea, since pointers get silently cast to them...
2020-09-02 14:16:44 +02:00
Domen Kožar
dd4b56c87f
Allow HTTP binary cache to request absolute uris 2020-09-01 21:35:48 +02:00
John Ericson
ef278d00f9 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-01 18:01:48 +00:00
Eelco Dolstra
6d7f7efb89
github: Use access token when calling .../commits API 2020-09-01 15:29:04 +02:00
Eelco Dolstra
84f5cabbea Merge remote-tracking branch 'origin/master' into markdown 2020-08-31 14:24:26 +02:00
John Ericson
02e0001fc0 Merge branch 'ca-floating-upstream' of github.com:obsidiansystems/nix into ca-floating-upstream 2020-08-28 21:23:09 +00:00
John Ericson
421ed527c7
Update src/libstore/build.cc
Thanks for catching, @regnat.
2020-08-28 17:22:57 -04:00
John Ericson
48e8828050 Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-08-28 20:38:25 +00:00
John Ericson
8017fe7487 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-28 19:59:14 +00:00
Eelco Dolstra
f15651303f nix develop: Add convenience flags for running specific phases
For example, for building the Nix flake, you would do:

  $ nix develop --configure
  $ nix develop --install
  $ nix develop --installcheck
2020-08-28 19:24:29 +02:00
Eelco Dolstra
3156560d41 nix develop: Set output paths to writable locations
Currently, they're set to $(pwd)/outputs/$outputName. This allows
commands like 'make install' to work.
2020-08-28 18:16:03 +02:00
Eelco Dolstra
691a1bd717 Merge branch 'minimal-logger' of https://github.com/Ma27/nix 2020-08-28 10:50:52 +02:00
Griffin Smith
626200713b Pass all args when auto-calling a function with an ellipsis
The command line options --arg and --argstr that are used by a bunch of
CLI commands to pass arguments to top-level functions in files go
through the same code-path as auto-calling top-level functions with
their default arguments - this, however, was only passing the arguments
that were *explicitly* mentioned in the formals of the function - in the
case of an as-pattern with an ellipsis (eg args @ { ... }) extra passed
arguments would get omitted. This fixes that to instead pass *all*
specified auto args in the case that our function has an ellipsis.

Fixes #598
2020-08-27 12:38:25 -04:00
Eelco Dolstra
eb75282b8d
Merge pull request #3434 from Ericson2314/derivation-header-include-order
Revise division of labor in deserialization of derivations
2020-08-27 16:39:28 +02:00
Eelco Dolstra
e915fd6d2a Typo 2020-08-27 14:51:50 +02:00
Eelco Dolstra
a0f19d9f3a RemoteStore::addToStore(): Fix race between stderrThread and NAR writer
As pointed out by @B4dM4n, the call to to.flush() on stderrThread is
unsafe because the NAR writer thread is also writing to 'to'.

Fixes #3943.
2020-08-27 14:50:51 +02:00
Eelco Dolstra
c3efef9275 Remove obsolete comment 2020-08-26 09:28:10 +02:00
Eelco Dolstra
4bf5faf416 Merge remote-tracking branch 'origin/master' into markdown 2020-08-25 19:47:34 +02:00
Eelco Dolstra
7a02865b94
Move import docs 2020-08-25 14:06:01 +02:00
Eelco Dolstra
f53b5f1058
Add getDoc() function 2020-08-25 13:31:11 +02:00
Eelco Dolstra
2a2121d264
Use RegisterPrimOp for some undocumented primops 2020-08-25 11:25:01 +02:00
Eelco Dolstra
b8416779e3
Document some primops 2020-08-25 11:16:45 +02:00
Eelco Dolstra
24b1c2c66b
Fix tests 2020-08-25 10:51:14 +02:00
Eelco Dolstra
b42789f013
Fix clang build 2020-08-24 21:13:39 +02:00
John Ericson
8eb73a8724 CA derivations that depend on other CA derivations
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-08-24 19:01:15 +00:00
John Ericson
e0b0e18905 Add constructor for BasicDerivation -> Derivation 2020-08-24 19:01:07 +00:00
John Ericson
59979e7053 Fix bad debug format string 2020-08-24 19:01:07 +00:00
Eelco Dolstra
d9a8619762
Don't barf if corepkgs is in the store but not a valid path
This can happen when using a dummy store (or indeed any non-local store).
2020-08-24 19:15:17 +02:00
Eelco Dolstra
6a67e57019
Add DummyStore (dummy://)
DummyStore does not allow building or adding paths. This is useful for
evaluation tests when you don't want to initialize a "proper" store.
2020-08-24 18:54:16 +02:00
Eelco Dolstra
d0690bc311
nix repl ':doc': Render using lowdown 2020-08-24 18:10:33 +02:00
Eelco Dolstra
0f314f3c25
Generate builtins section of the manual 2020-08-24 14:49:30 +02:00
Eelco Dolstra
a990f063ff
Move primop docs inline
This makes them available to 'nix repl'.
2020-08-24 14:31:10 +02:00
Eelco Dolstra
33b1679d75
Allow primops to have Markdown documentation 2020-08-24 13:16:02 +02:00
Eelco Dolstra
88d5c9ec58
Fix tests 2020-08-24 10:37:10 +02:00
Eelco Dolstra
2530cc6827
Merge pull request #3950 from obsidiansystems/tabs-spaces
tabs -> spaces
2020-08-24 10:20:28 +02:00
John Ericson
b0b59fd05a Merge remote-tracking branch 'obsidian/write-derivation-borrow' into HEAD 2020-08-23 15:11:10 +00:00
John Ericson
35e6288be1 writeDerivation just needs a plain store reference 2020-08-23 15:01:11 +00:00
John Ericson
3a7b330b64 "Downstream placeholders" should not be store paths
Insead they should be opaque `/<hash>` like the placeholders we already
have.
2020-08-21 19:35:35 +00:00
John Ericson
422affe102 tabs -> spaces
Sorry I let the tab sneak in there in the first place.
2020-08-21 19:19:24 +00:00
John Ericson
c265e0ea82 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-08-20 20:24:22 +00:00
John Ericson
27a3f82c0b Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-20 18:28:17 +00:00
John Ericson
45a2f1baab Rename drv output querying functions, like master
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
 - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap
2020-08-20 18:14:12 +00:00
Eelco Dolstra
4d77513d97
Merge pull request #3859 from obsidiansystems/drv-outputs-map-allow-missing
`queryDerivationOutputMap` no longer assumes all outputs have a mapping
2020-08-20 16:49:23 +02:00
John Ericson
9a9d834dc7 Rename drv output querying functions
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
 - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap`
2020-08-20 14:12:51 +00:00
Eelco Dolstra
25ecfffdc3
Remove PrimOp constructor 2020-08-20 12:34:04 +02:00
Eelco Dolstra
dc2f278c95
Allow 'nix' subcommands to provide docs in Markdown format 2020-08-20 12:21:46 +02:00
Eelco Dolstra
3c4f8c9175
List deprecated option aliases in the docs 2020-08-20 11:13:17 +02:00
Eelco Dolstra
acb99f03f9
Config: Use nlohmann/json 2020-08-20 11:02:16 +02:00
John Ericson
3df78858f2 Fix max fd calc and add test 2020-08-20 05:08:50 +00:00
Ryan Mulligan
f36793c7b9
fix spelling 2020-08-19 20:31:01 -07:00
John Ericson
a83694c7a1 Use RemoteStore to open connection for proxying daemon
Removes duplicate websocket opening code, and also means we should be
able to to ssh-ssh-... daemon relays, not just uds-uds-... ones.
2020-08-19 19:34:47 +00:00
Eelco Dolstra
c8fa39324a
Generate the nix.conf docs from the source code
This means we don't have two (divergent) sets of option descriptions
anymore.
2020-08-19 18:28:04 +02:00
John Ericson
d5af5763cf Merge branch 'master' of github.com:NixOS/nix into remove-storetype-delegate-regStore 2020-08-19 15:54:50 +00:00
Eelco Dolstra
34b22e0123
Change option descriptions to Markdown 2020-08-19 14:21:27 +02:00
John Ericson
be0d429b95 Merge branch 'master' of github.com:NixOS/nix into templated-daemon-protocol 2020-08-19 03:17:41 +00:00
John Ericson
950ddfdb82 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-08-18 14:36:44 +00:00
Eelco Dolstra
1c8b550e34
Merge pull request #3917 from obsidiansystems/output-env-var-unconditional
Simplify code as output env vars are unconditional
2020-08-18 16:21:17 +02:00
Eelco Dolstra
dfeb76dbf9
Merge pull request #3930 from obsidiansystems/legacy-ssh-build-paths
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths`
2020-08-18 16:07:40 +02:00
Eelco Dolstra
069340179e
Improve nix.1 manpage generator 2020-08-18 15:15:35 +02:00
Eelco Dolstra
6f19c776db
Start generation of the nix.1 manpage 2020-08-17 19:33:18 +02:00
Eelco Dolstra
a72a20d68f
Add 'nix dump-args' to dump all commands/flags for manpage generation 2020-08-17 17:44:52 +02:00
Eelco Dolstra
7cdc739ece
Merge remote-tracking branch 'origin/master' into markdown 2020-08-17 13:43:39 +02:00
Eelco Dolstra
e849b19872
Merge pull request #3932 from chkno/no-show-signature
Don't try to parse signature check as commit timestamp
2020-08-17 11:00:24 +02:00
Eelco Dolstra
847a5392f4 Merge branch 'ca-no-need-trust' of https://github.com/obsidiansystems/nix 2020-08-17 10:57:02 +02:00
Eelco Dolstra
3c619f6290 Merge branch 'test-RemoteStore-buildDerivation' of https://github.com/obsidiansystems/nix 2020-08-17 10:53:18 +02:00
John Ericson
6f7ac5e865 Remove extra closing paren 2020-08-14 21:59:31 +00:00
John Ericson
f899a7c6d7 Work around clang bug 2020-08-14 18:51:31 +00:00
John Ericson
3c8b5b6219 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-14 17:00:13 +00:00
John Ericson
4b571ea321
Update src/libstore/daemon.cc
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-08-14 11:52:37 -04:00
Eelco Dolstra
13e49be660
Merge pull request #3875 from obsidiansystems/new-interface-for-path-pathOpt
Offer a safer interface for path and pathOpt
2020-08-14 17:19:19 +02:00
Eelco Dolstra
7714d9a943
Merge pull request #3924 from obsidiansystems/features-per-store
Make `system-features` a store setting
2020-08-14 17:13:07 +02:00
Eelco Dolstra
9b9d5297ba
Merge pull request #3909 from matthewbauer/readd-hashed-mirrors
Add hashed-mirrors back
2020-08-14 17:05:47 +02:00
Eelco Dolstra
d81f13f7cb
Merge pull request #3899 from obsidiansystems/make-narHash-not-optional
Make narHash in ValidPathInfo not optional
2020-08-14 17:00:18 +02:00
Chuck
ed026f7206 Don't try to parse signature check as commit timestamp
When the log.showSignature git setting is enabled, the output of
"git log" contains signature verification information in addition to the
timestamp GitInputScheme::fetch wants:

  $ git log -1 --format=%ct
  gpg: Signature made Sat 07 Sep 2019 02:02:03 PM PDT
  gpg:                using RSA key 0123456789ABCDEF0123456789ABCDEF01234567
  gpg:                issuer "user@example.com"
  gpg: Good signature from "User <user@example.com>" [ultimate] 1567890123
  1567890123

For folks that had log.showSignature set, this caused all nix operations
on flakes to fail:

  $ nix build
  error: stoull
2020-08-13 17:44:42 -07:00
John Ericson
e1308b1211 Define LegacySSHStore::buildPaths using cmdBuildPaths
Evidentally this was never implemented because Nix switched to using
`buildDerivation` exclusively before `build-remote.pl` was rewritten.

The `nix-copy-ssh` test (already) tests this.
2020-08-13 21:27:55 +00:00
John Ericson
5ccd94501d Allow trustless building of CA derivations
Include a long comment explaining the policy. Perhaps this can be moved
to the manual at some point in the future.

Also bump the daemon protocol minor version, so clients can tell whether
`wopBuildDerivation` supports trustless CA derivation building. I hope
to take advantage of this in a follow-up PR to support trustless remote
building with the minimal sending of derivation closures.
2020-08-13 18:15:57 +00:00
John Ericson
85aacbee64 Use TeeSink and TeeSouce in a few more places 2020-08-13 14:51:17 +00:00
John Ericson
d2f2be0f70 Test RemoteStore::buildDerivation
Fix `wopNarFromPath` which needed a `toRealPath`.
2020-08-13 04:07:14 +00:00
John Ericson
5d67f18c86 Merge branch 'daemon-auth-cleanup' of github.com:obsidiansystems/nix into HEAD 2020-08-12 18:22:31 +00:00
John Ericson
4720853129 Make system-features a store setting
This seems more correct. It also means one can specify the features a
store should support with --store and remote-store=..., which is useful.
I use this to clean up the build remotes test.
2020-08-12 18:13:00 +00:00
John Ericson
8d4162ff9e Separate auth and logic for the daemon
Before, processConnection wanted to know a user name and user id, and
`nix-daemon --stdio`, when it isn't proxying to an underlying daemon,
would just assume "root" and 0. But `nix-daemon --stdio` (no proxying)
shouldn't make guesses about who holds the other end of its standard
streams.

Now processConnection takes an "auth hook", so `nix-daemon` can provide
the appropriate policy and daemon.cc doesn't need to know or care what
it is.
2020-08-12 15:22:33 +00:00
John Ericson
5f80aea795 Break out lambda so output can be matched just once
This is much better.
2020-08-12 02:23:31 +00:00
John Ericson
18834f7764 Recheck path validity after acquiring lock
It might have changed, and in any event this is how the cod used to work
so let's just keep it.
2020-08-11 23:44:02 +00:00
John Ericson
2de201254e Don't assume a total output map in two places in build.cc
Thanks @regnat for catching one of them. The other follows for many of
the same reasons. I'm find fixing others on a need-to-fix basis,
provided their are no regressions.
2020-08-11 23:07:50 +00:00
John Ericson
4c6aac8fdf Clarify comment on sandbox and temp fresh paths 2020-08-11 22:46:05 +00:00
John Ericson
6d57139050 Clarify outputReferences variable with self-describing type
Thanks for the idea, @Regnat!
2020-08-11 22:34:09 +00:00
John Ericson
8a068bd025 Remove redundant equality check 2020-08-11 21:25:40 +00:00
John Ericson
07e3466eb4 Float comment to out describe gaveUpOnSubstitution in general 2020-08-11 21:16:14 +00:00
John Ericson
d0f6e338dd
Apply suggestions from code review
Thanks!!

Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-08-11 16:49:10 -04:00
Ben Burdette
1a281ec07f demote remote build message to Info 2020-08-11 10:29:43 -06:00
Eelco Dolstra
7a911b6783
Merge pull request #3914 from obsidiansystems/small-drv-serialize-cleanup
Two small derivation serialization cleanups
2020-08-11 07:14:38 +02:00
John Ericson
d3fa8c04c6 Simplify code as output env vars are unconditional
Since the jsonObject unique ptr is reset to flush the string to make
`__json`, all these `!jsonObject` conditions will always be true.
2020-08-11 01:13:26 +00:00
John Ericson
2a0902634e Fix error in merge breaking floating CA drvs
Forgot to add this hunk!
2020-08-11 00:13:19 +00:00
Maximilian Bosch
5f8ae16c8b
Always reset ANSI colors in progress-bar line
When having a message like `waiting for a machine to build X` and
building with `nix build -L`, the log-prefix is always colored yellow[1]
on a small terminal-width as everything (including the ANSI color-reset) is
stripped away.

To work around that problem, this patch explicitly adds an `ANSI_NORMAL`
to the end of the line.

[1] https://imgur.com/a/FjtJOk3
2020-08-10 17:44:17 +02:00
John Ericson
1b5c24662b Merge branch 'small-drv-serialize-cleanup' of github.com:obsidiansystems/nix into single-ca-drv-build 2020-08-10 01:57:54 +00:00
John Ericson
581183d4d5 Deduplicate parsing and reading derivations 2020-08-10 01:40:50 +00:00
John Ericson
bcd0629c2e Remove name parameter from writeDerivation
The name is now stored with the derivation itself.
2020-08-10 01:35:59 +00:00
John Ericson
e913a2989f Squashed get CA derivations building 2020-08-07 19:51:55 +00:00
Eelco Dolstra
edfd676e05 Fix .ls file names in binary caches
These are not supposed to include the 'name' part of the store
path. This was broken by 759947bf72.
2020-08-07 21:18:29 +02:00
John Ericson
8f92bb5ad9 Merge branch 'drv-outputs-map-allow-missing' of github.com:obsidiansystems/nix into templated-daemon-protocol 2020-08-07 18:51:01 +00:00
John Ericson
f7ba16f9cb Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing 2020-08-07 17:46:39 +00:00
John Ericson
47644e49ca Specialize std::optional<StorePath> so this is backwards compatible
While I am cautious to break parametricity, I think it's OK in this
cases---we're not about to try to do some crazy polymorphic protocol
anytime soon.
2020-08-07 17:05:14 +00:00
Eelco Dolstra
3c75ddc16b nix build (and others): Force re-evaluation of cached errors
Fixes #3872.

This is a bit hacky. Ideally we would automatically re-evaluate the
failed attribute iff we need to print the error message (so in
commands like 'nix search' we wouldn't re-evaluate because we're
suppressing errors).
2020-08-07 14:47:23 +02:00
Eelco Dolstra
2ffc058950 Make --no-eval-cache a global setting 2020-08-07 14:13:24 +02:00
Matthew Bauer
96c158d6e1 Fix build 2020-08-06 21:04:31 -05:00
Matthew Bauer
8abc577cc2 Merge remote-tracking branch 'origin/master' into readd-hashed-mirrors 2020-08-06 18:58:59 -05:00
Carlo Nucera
46f9dd56da Fix bug due to non-deterministic arg eval order 2020-08-06 19:30:05 -04:00
Matthew Bauer
641c950701 Add hashed-mirrors back
Some users have their own hashed-mirrors setup, that is used to mirror
things in addition to what’s available on tarballs.nixos.org. Although
this should be feasable to do with a Binary Cache, it’s not always
easy, since you have to remember what "name" each of the tarballs has.
Continuing to support hashed-mirrors is cheap, so it’s best to leave
support in Nix. Note that NIX_HASHED_MIRRORS is also supported in
Nixpkgs through fetchurl.nix.

Note that this excludes tarballs.nixos.org from the default, as in
\#3689. All of these are available on cache.nixos.org.
2020-08-06 18:19:09 -05:00
Carlo Nucera
9ab07e99f5 Use template structs instead of phantoms 2020-08-06 18:04:13 -04:00
Carlo Nucera
3d8240c32e Remove leftover commented code 2020-08-06 16:04:18 -04:00
Carlo Nucera
f795f0fabc Merge branch 'drv-outputs-map-allow-missing-namespace' of github.com:obsidiansystems/nix into templated-daemon-protocol 2020-08-06 15:53:09 -04:00
John Ericson
e89b5bd0bf Minimize the usage of Hash::dummy 2020-08-06 18:31:48 +00:00
John Ericson
5e59b25a23 Merge remote-tracking branch 'upstream/master' into make-narHash-not-optional 2020-08-06 15:34:15 +00:00
Eelco Dolstra
6146447842
Merge pull request #3856 from obsidiansystems/buildable-variant
Make `Buildable` a `std::variant`
2020-08-06 14:14:06 +02:00
Eelco Dolstra
59067f0f58 repl.cc: Check for HAVE_BOEHMGC
Fixes #3906.
2020-08-06 11:40:41 +02:00
Carlo Nucera
8b175f58d1 Simplify the namespace 2020-08-05 17:57:07 -04:00