Commit graph

3752 commits

Author SHA1 Message Date
Eelco Dolstra
5176b072ed Cleanup 2021-10-12 16:43:00 +02:00
Eelco Dolstra
3a778ea8a0 Merge branch 'nix-repl-download-interruption' of https://github.com/Ma27/nix 2021-10-12 16:34:38 +02:00
Tom Bereknyei
03bb8f84e0 Add compression level for NARs
Based off on @dtzWill's #2276
2021-10-12 02:14:36 -04:00
regnat
7466048d39 (partially) Revert "Don't copy in rethrow"
This reverts some parts of commit
8430a8f086 which was trying to rethrow
some exceptions while we weren’t in the context of a `catch` block,
causing some weird “terminate called without an active exception”
errors.

Fix #5368
2021-10-11 10:51:22 +02:00
Sergei Trofimovich
d7d6fe44d6 libstore-tests: add libutil dependency (fix static link failure)
In https://github.com/NixOS/nix/pull/5350 we noticed link failures
pkgsStatic.nixUnstable. Adding explicit dependency on libutil fixes
libstore-tests linking.
2021-10-08 07:16:12 +01:00
Maximilian Bosch
0872659002
nix repl: properly deal with interruptions
When I stop a download with Ctrl-C in a `nix repl` of a flake, the REPL
refuses to do any other downloads:

    nix-repl> builtins.getFlake "nix-serve"
    [0.0 MiB DL] downloading 'https://api.github.com/repos/edolstra/nix-serve/tarball/e9828a9e01a14297d15ca41 error: download of 'e9828a9e01' was interrupted
    [0.0 MiB DL]
    nix-repl> builtins.getFlake "nix-serve"
    error: interrupted by the user
    [0.0 MiB DL]

To fix this issue, two changes were necessary:

* Reset the global `_isInterrupted` variable: only because a single
  operation was aborted, it should still be possible to continue the
  session.
* Recreate a `fileTransfer`-instance if the current one was shut down by
  an abort.
2021-10-07 23:58:02 +02:00
Eelco Dolstra
66c4b20d8b Typo 2021-10-07 13:34:04 +02:00
Eelco Dolstra
c6718a9d95 Don't reset the logger in a vfork
9c766a40cb broke logging from the
daemon, because commonChildInit is called when starting the build hook
in a vfork, so it ends up resetting the parent's logger. So don't
vfork.

It might be best to get rid of vfork altogether, but that may cause
problems, e.g. when we call an external program like git from the
evaluator.
2021-10-06 13:54:59 +02:00
Eelco Dolstra
fd57e7074f
Merge pull request #5335 from edolstra/socket-paths
Support arbitrary-length socket paths
2021-10-05 17:36:07 +02:00
Eelco Dolstra
3b7f4c7d9d Add FIXME about ptsname 2021-10-05 11:04:46 +02:00
Eelco Dolstra
43d4d75e22 Connect/bind Unix domain sockets in a child process
In the child process, we can do a chdir() and avoid the problem of the
path not fitting into sockaddr_un.
2021-10-05 10:44:59 +02:00
Eelco Dolstra
77ebbc9f54 Add a test for RefScanSink and clean up the code
Issue #5322.
2021-10-04 14:29:42 +02:00
Eelco Dolstra
ef34fd0656 scanForReferences(): Use a StorePathSet 2021-10-04 13:47:38 +02:00
John Ericson
13b6b64589 Merge remote-tracking branch 'upstream/master' into path-info 2021-10-01 17:12:54 +00:00
Sergei Trofimovich
621aa65325 local-derivation-goal.cc: downgrade "warning: rewriting hashes in..." down to debug
Before the changes when building the whole system with
`contentAddressedByDefault = true;` we get many noninformative messages:

    $ nix build -f nixos system --keep-going
    ...
    warning: rewriting hashes in '/nix/store/...-clang-11.1.0.drv.chroot/nix/store/...-11.1.0'; cross fingers
    warning: rewriting hashes in '/nix/store/...-clang-11.1.0.drv.chroot/nix/store/...-11.1.0-dev'; cross fingers
    warning: rewriting hashes in '/nix/store/...-clang-11.1.0.drv.chroot/nix/store/...-11.1.0-python'; cross fingers
    error: 2 dependencies of derivation '/nix/store/...-hub-2.14.2.drv' failed to build
    warning: rewriting hashes in '/nix/store/...-subversion-1.14.1.drv.chroot/nix/store/...-subversion-1.14.1-dev'; cross fingers
    warning: rewriting hashes in '/nix/store/...-subversion-1.14.1.drv.chroot/nix/store/...-subversion-1.14.1-man'; cross fingers
    ...

Let's downgrade these messages down to debug().
2021-10-01 17:41:44 +01:00
Eelco Dolstra
7cc220825d
Merge pull request #5167 from Ma27/keep-failed-on-ssh-remote-build
nix-store --serve: pass on `settings.keepFailed` from SSH store
2021-10-01 16:35:02 +02:00
John Ericson
f4f3203aa7 Merge remote-tracking branch 'upstream/master' into path-info 2021-09-30 22:41:53 +00:00
John Ericson
242f9bf3dc std::visit by reference
I had started the trend of doing `std::visit` by value (because a type
error once mislead me into thinking that was the only form that
existed). While the optomizer in principle should be able to deal with
extra coppying or extra indirection once the lambdas inlined, sticking
with by reference is the conventional default. I hope this might even
improve performance.
2021-09-30 21:35:09 +00:00
Eelco Dolstra
6b5efeafd1
Merge pull request #5295 from Ma27/bmcheck-fod-mismatch
build: also throw hash-mismatch errors if `buildMode == bmCheck`
2021-09-29 10:13:45 +02:00
Maximilian Bosch
bb1a851bcf
build: also throw hash-mismatch errors if buildMode == bmCheck
This actually bit me quite recently in `nixpkgs` because I assumed that
`nix-build --check` would also error out if hashes don't match anymore[1]
and so I wrongly assumed that I couldn't reproduce the mismatch error.

The fix is rather simple, during the output registration a so-called
`delayedException` is instantiated e.g. if a FOD hash-mismatch occurs.
However, in case of `nix-build --check` (or `--rebuild` in case of `nix
build`), the code-path where this exception is thrown will never be
reached.

By adding that check to the if-clause that causes an early exit in case
of `bmCheck`, the issue is gone. Also added a (previously failing)
test-case to demonstrate the problem.

[1] https://github.com/NixOS/nixpkgs/pull/139238, the underlying issue
    was that `nix-prefetch-git` returns different hashes than `fetchgit`
    because the latter one fetches submodules by default.
2021-09-27 15:44:39 +02:00
Eelco Dolstra
8430a8f086 Don't copy in rethrow 2021-09-27 14:38:10 +02:00
Eelco Dolstra
362d8f925e
Merge pull request #5253 from edolstra/flake-ifd
Don't allow IFD in flakes by default
2021-09-24 10:48:23 +02:00
Eelco Dolstra
ea9df6fe51 Shut down write side before draining the read side
This is important if the remote side *does* execute
nix-store/nix-daemon successfully, but stdout is polluted
(e.g. because the remote user's bashrc script prints something to
stdout). In that case we have to shutdown the write side to force the
remote nix process to exit.
2021-09-23 18:01:04 +02:00
Eelco Dolstra
994348e9e0 SSHStore / LegacySSHStore: Show a better error message if the remote is "nologin"
Instead of

  error: serialised integer 7161674624452356180 is too large for type 'j'

we now get

  error: 'nix-store --serve' protocol mismatch from 'sshtest@localhost', got 'This account is currently not available.'

Fixes https://github.com/NixOS/nixpkgs/issues/37287.
2021-09-23 17:50:29 +02:00
Eelco Dolstra
60642aa5e2 Remove risky char * 2021-09-23 12:07:50 +02:00
Eelco Dolstra
198fa786a1 Add some more instrumentation 2021-09-23 11:42:33 +02:00
Eelco Dolstra
d8c10028d9 Make setDefault() typed 2021-09-22 14:15:35 +02:00
Eelco Dolstra
bcd73ebf60 Add missing #include
Fixes #5282.
2021-09-22 12:14:50 +02:00
Michael Bishop
374fe49ff7 set the PER_LINUX32 personality flag, when building for armv6l-linux or armv7l-linux
this prevents 32bit builds from detecting a 64bit kernel and picking the wrong target
2021-09-19 23:07:10 -03:00
Gregor Kleen
fd67a0f927 Fix scheme argument to s3 cache 2021-09-17 10:45:19 +02:00
Matthew Kenigsberg
3b82c1a5fe
Merge remote-tracking branch 'upstream/master' into auto-uid-allocation 2021-09-15 11:58:06 -05:00
Emily
56025ad3b1 sandbox: allow Rosetta 2 on Darwin
This allows sandboxed x86_64-darwin builds on aarch64-darwin.
2021-09-15 02:02:03 +01:00
Eelco Dolstra
4b738fc7a9 Add 'nix profile wipe-history' command 2021-09-14 20:35:41 +02:00
Eelco Dolstra
f359b9981b Generations -> profile versions 2021-09-14 19:57:45 +02:00
Eelco Dolstra
817562e694 Add "nix profile rollback" command 2021-09-14 19:32:33 +02:00
Eelco Dolstra
d589a6aa8a
Merge pull request #5247 from edolstra/stablize-ca-references
Remove the 'ca-references' feature check
2021-09-14 15:42:50 +02:00
Eelco Dolstra
01a4fa3b6e Advise using --extra-experimental-features instead of --experimental-features
Fixes #3737.
2021-09-14 13:56:36 +02:00
Eelco Dolstra
761ac9d584 Remove the 'ca-references' feature check
Fixes #3422.
Fixes #4425.
2021-09-14 13:53:20 +02:00
Eelco Dolstra
76e368a3b4 Fix macOS build 2021-09-14 10:07:31 +02:00
Eelco Dolstra
4ed66735b6 RunOptions: Use designated initializers
Also get rid of _killStderr because it wasn't actually checked
anywhere.
2021-09-13 23:31:04 +02:00
Eelco Dolstra
ff09acd1f9 Remove tabs 2021-09-10 11:00:50 +02:00
Eelco Dolstra
1e43bc6dc0
Merge pull request #5224 from baloo/baloo/5089/force-nss_dns-load
preloadNSS: fixup nss_dns load
2021-09-09 10:37:41 +02:00
Arthur Gautier
3b72741f23 preloadNSS: load NSS before threads are started
preloadNSS is not thread-safe, this commit moves it before we start the
first thread.

Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-09-08 18:29:31 +00:00
Arthur Gautier
0b42a0f781 preloadNSS: fixup nss_dns load
Before this commit, the dns lookup in preloadNSS would still go through
nscd. This did not have the effect of loading the nss_dns.so as expected
(nss_dns.so being out of reach from within the sandbox).

Should LOCALDOMAIN environment variable be defined, nss will completely
avoid nscd and will do its dns resolution on its own.

By temporarly setting LOCALDOMAIN variable before calling in NSS, we can
force NSS to load the shared libraries as expected.

Fixes #5089

Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-09-06 23:13:48 +00:00
regnat
497225b07d Don’t create lockfiles with an invalid path name
Store paths are only allowed to contain a limited subset of the
alphabet, which doesn’t include `!`. So don’t create lockfiles that
contain this `!` character as that would otherwise confuse (and break)
the gc.

Fix #5176
2021-09-02 09:57:41 +02:00
Eelco Dolstra
f73d911628 Style 2021-09-01 21:46:25 +02:00
Eelco Dolstra
2cb7a502a4 Merge branch 'short-circuit-subst' of https://github.com/Kha/nix 2021-09-01 21:45:04 +02:00
Maximilian Bosch
50edbc4ddf
nix-store --serve: pass on settings.keepFailed from SSH store
When doing e.g.

    nix-build -A package --keep-failed --option \
      builders \
      'ssh://mfhydra?remote-store=/home/bosch/store x86_64-linux - 10 4 big-parallel'

this doesn't work properly because this build-setting is ignored.

I changed this behavior by passing the `settings.keepFailed` through the
serve-protocol to remote machines to make sure that I can introspect the
build-directory (which is particularly helpful when I have to look at a
`config.log` from a failed build for instance).
2021-08-31 13:11:46 +02:00
Eelco Dolstra
f3b8b4040e
Merge pull request #5071 from NixOS/dont-send-xp-features-to-the-daemon
Don’t send the experimental-features to the daemon
2021-08-31 11:55:39 +02:00
Eelco Dolstra
eda0fee160
Merge pull request #5175 from Pamplemousse/make
Don't overwrite user provided `lib*_LDFLAGS`
2021-08-30 12:44:29 +02:00
regnat
b2bce915ad Don’t accept experimental features from the client
If the client sends an “experimental features” setting, just ignore it
2021-08-30 10:48:05 +02:00
Pamplemousse
a4c6d319a8 Don't overwrite user provided lib*_LDFLAGS
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-08-25 08:59:19 -07:00
Magic_RB
8438114399
Add ignored_acls setting
Signed-off-by: Magic_RB <magic_rb@redalder.org>
2021-08-14 21:46:09 +02:00
Piotr Szubiakowski
9f13cb31e8
Install pkg-config files in the correct location
Use `$(libdir)` while installing .pc files looks like a more generic
solution. For example, it will work for distributions like RHEL or
Fedora where .pc files are installed in `/usr/lib64/pkgconfig`.
2021-08-13 21:08:58 +00:00
Pamplemousse
0da416f820 15f4d4f follow up
* libstore: `bz2` should not be linked
  * libutil: `zlib.h` should not be included

Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-08-10 16:04:53 -07:00
Eelco Dolstra
8943e3176d
Merge pull request #5111 from Pamplemousse/clean
Minor maintenance cleaning
2021-08-09 20:05:03 +02:00
Sebastian Ullrich
a0f97b1f54 Expand docstring of Store::querySubstitutablePathInfos 2021-08-09 15:58:50 +02:00
Eelco Dolstra
4c5ce2a345
Merge pull request #5104 from andir/refscan-race
Fix potential race-condition in reference scanning code
2021-08-09 14:49:24 +02:00
Andreas Rammhold
a9cb1ca32c libstore: use set instead of list for waiter list
This replaces the O(n) search complexity in our insert code with a
lookup of O(log n). It also makes removing waitees easier as we can use
the extract method provided by the set class.
2021-08-08 14:05:38 +02:00
Sebastian Ullrich
12f50b6510 Short-circuit querying substituters on success 2021-08-08 13:24:13 +02:00
Andreas Rammhold
b2d3976163 Fix potential race-condition in reference scanning code
Previously the code ensures that the isBase32 array would only be
initialised once in a single-threaded context. If two threads happen to
call the function before the initialisation was completed both of them
would have completed the initialization step. This allowed for a
race-condition where one thread might be done with the initialization
but the other thread sets all the fields to false again. For a brief
moment the base32 detection would then produce false-negatives.
2021-08-07 19:10:25 +02:00
Pamplemousse
d64cb33e90 Remove unused struct in libstore
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-07-30 10:34:28 -07:00
regnat
d9ba3385a9 Don’t send the experimental-features to the daemon
The experimental features are, well, experimental, and shouldn’t be
carelessly and transparently enabled.
Besides, some (`ca-derivations` at least) need to be enabled at startup
in order to work properly.
So it’s better to just require that daemon be started with the right
`experimental-features` option.

Fix #5017
2021-07-30 10:34:50 +02:00
Eelco Dolstra
72c5bac39d Revert no longer necessary change 2021-07-26 13:50:18 +02:00
Eelco Dolstra
fe1f34fa60 Low-latency closure copy
This adds a new store operation 'addMultipleToStore' that reads a
number of NARs and ValidPathInfos from a Source, allowing any number
of store paths to be copied in a single call. This is much faster on
high-latency links when copying a lot of small files, like .drv
closures.

For example, on a connection with an 50 ms delay:

Before:

  $ nix copy --to 'unix:///tmp/proxy-socket?root=/tmp/dest-chroot' \
    /nix/store/90jjw94xiyg5drj70whm9yll6xjj0ca9-hello-2.10.drv \
    --derivation --no-check-sigs
  real    0m57.868s
  user    0m0.103s
  sys     0m0.056s

After:

  real    0m0.690s
  user    0m0.017s
  sys     0m0.011s
2021-07-26 13:31:09 +02:00
Eelco Dolstra
9957315ce0 RemoteStore::buildPaths(): Handle evalStore 2021-07-22 22:50:48 +02:00
Eelco Dolstra
a7b7fcfb16 Remove redundant RealisedPath::closure() call 2021-07-22 22:43:18 +02:00
Eelco Dolstra
eb6db4fd38 buildPaths(): Add an evalStore argument
With this, we don't have to copy the entire .drv closure to the
destination store ahead of time (or at all). Instead, buildPaths()
reads .drv files from the eval store and copies inputSrcs to the
destination store if it needs to build a derivation.

Issue #5025.
2021-07-22 09:59:51 +02:00
Eelco Dolstra
668abd3e57 copyPaths: Pass store by reference 2021-07-22 09:59:51 +02:00
Eelco Dolstra
8d9f7048cd Use eval-store in more places
In particular, this now works:

  $ nix path-info --eval-store auto --store https://cache.nixos.org nixpkgs#hello

Previously this would fail as it would try to upload the hello .drv to
cache.nixos.org. Now the .drv is instantiated in the local store, and
then we check for the existence of the outputs in cache.nixos.org.
2021-07-22 09:59:51 +02:00
Eelco Dolstra
95e915a993 DummyStore: Remove redundant method 2021-07-22 09:59:51 +02:00
Eelco Dolstra
e9848beca7 nix-build: Copy drv closure between eval store and build store 2021-07-22 09:59:51 +02:00
Eelco Dolstra
3d9de41a5b Hacky fast closure copying mechanism 2021-07-22 09:59:51 +02:00
Eelco Dolstra
dfda499326 Downgrade warning message
If a store doesn't support GC, then we don't need to warn about the
inability to create roots.
2021-07-22 09:58:50 +02:00
Eelco Dolstra
3bb8667a17 Tweak --no/use-registries doc strings 2021-07-21 14:27:37 +02:00
Eelco Dolstra
fc248cf59e Merge branch 'balsoft/no-registries' of https://github.com/serokell/nix 2021-07-21 10:11:40 +02:00
regnat
8707773965 Properly lock the builds of CA derivations
Make sure that we can’t build the same derivation twice at the same
time.

Fix https://github.com/NixOS/nix/issues/5029
2021-07-20 06:57:56 +02:00
Alexander Bantyev
3e57e3480b
Add use-registries config option (and deprecate --no-registries flag)
Some people want to avoid using registries at all on their system; Instead
of having to add --no-registries to every command, this commit allows to
set use-registries = false in the config. --no-registries is still allowed
everywhere it was allowed previously, but is now deprecated.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-07-16 23:26:04 +03:00
regnat
a4ec6cb1da Be more lenient when realisations have a conflicting dependency set
- This can legitimately happen (for example because of a non-determinism
  causing a build-time dependency to be kept or not as a runtime
  reference)
- Because of older Nix versions, it can happen that we encounter a
  realisation with an (erroneously) empty set of dependencies, in which
  case we don’t want to fail, but just warn the user and try to fix it.
2021-07-16 11:48:41 +02:00
regnat
9b1f3cbc13 Forward the whole Nix config to the post-build-hook
Fill `NIX_CONFIG` with the value of the current Nix configuration before
calling the post-build-hook.
That way the whole configuration (including the possible
`experimental-features`, a possibly `--store` option or whatever) will
be made available to the hook
2021-07-15 18:41:56 +02:00
Eelco Dolstra
eb4788954d
Merge pull request #5006 from illustris/nscd
fixed-output derivations: fix incorrect responses for getpwuid
2021-07-13 17:20:37 +02:00
illustris
7bc17a903b fixed output derivations: fix incorrect responses for getpwuid
Passing nscd socket into the build environment causes unexpected behavior in programs that make getpwuid and other related calls.

relevant threads:
- https://github.com/NixOS/nix/issues/4991
- https://discourse.nixos.org/t/haunted-nix-build-breaks-isolation/13869
2021-07-13 15:34:14 +05:30
Eelco Dolstra
02dff9e529 Style 2021-07-12 17:32:23 +02:00
Eelco Dolstra
e06c272c12 Merge branch 'structured-attrs-shell' of https://github.com/Ma27/nix 2021-07-12 17:13:05 +02:00
Maximilian Bosch
04cd2da84c
Merge branch 'master' into structured-attrs-shell
Conflicts:
        src/nix/develop.cc
        src/nix/get-env.sh
        tests/shell.nix
2021-07-12 15:49:39 +02:00
Eelco Dolstra
91d2e8d5ad Error -> UsageError 2021-07-12 15:04:46 +02:00
Eelco Dolstra
8d97030bfd Merge branch 'days' of https://github.com/nielsegberts/nix 2021-07-12 15:02:24 +02:00
Niels Egberts
2cf14db857 Throw on unexpected input for --delete-older-than
'--delete-older-than 10' deletes the generations older than a single day, and '--delete-older-than 12m' deletes all generations older than 12 days.

This changes makes it throw on those invalid inputs, and gives an example of a valid input.
2021-07-11 11:24:43 +01:00
Eelco Dolstra
ceda58d112 Formatting 2021-07-09 14:03:51 +02:00
Eelco Dolstra
2172e60f7a
Merge pull request #4935 from alyssais/host_os
Apply OS checks to host platform, not build
2021-07-08 12:40:54 +02:00
Eelco Dolstra
cee426cc01
Merge pull request #4938 from tomcur/store-uri-parsing
Improve machine store URI parsing
2021-07-05 14:00:35 +02:00
Eelco Dolstra
6182ae6898
Merge pull request #4942 from NixOS/ca/remove-lock-files
Eventually delete the CA paths lock files
2021-06-28 16:12:19 +02:00
Eelco Dolstra
f5320299dd
Merge pull request #4937 from NixOS/ca/make-the-tests-useful
Make the CA tests actually test something
2021-06-28 16:06:49 +02:00
regnat
8b6fba2b63 Eventually delete the CA paths lock files
Mark the lockfiles as having to eventually be deleted so that they don’t
stay laying around in the store at the end of the build

Fix #4936
2021-06-24 15:45:05 +02:00
Maximilian Bosch
644415d391
Perform input rewrites only in LocalDerivationGoal 2021-06-24 15:25:21 +02:00
regnat
7746cb13dc Make CA derivations compatible with recursive Nix
Add an access-control list to the realisations in recursive-nix (similar
to the already existing one for store paths), so that we can build
content-addressed derivations in the restricted store.

Fix #4353
2021-06-24 14:53:10 +02:00
regnat
be7a4a6a13 Make the post-build-hook also run for unresolved CA derivations
Fix #4837
2021-06-24 11:41:57 +02:00
Thomas Churchman
e3d11f9a9c Improve machine store URI parsing 2021-06-23 22:09:22 +01:00
Alyssa Ross
4f80464645
Apply OS checks to host platform, not build
Previously, the build system used uname(1) output when it wanted to
check the operating system it was being built for, which meant that it
didn't take into-account cross-compilation when the build and host
operating systems were different.

To fix this, instead of consulting uname output, we consult the host
triple, specifically the third "kernel" part.

For "kernel"s with stable ABIs, like Linux or Cygwin, we can use a
simple ifeq to test whether we're compiling for that system, but for
other platforms, like Darwin, FreeBSD, or Solaris, we have to use a
more complicated check to take into account the version numbers at the
end of the "kernel"s.  I couldn't find a way to just strip these
version numbers in GNU Make without shelling out, which would be even
more ugly IMO.  Because these checks differ between kernels, and the
patsubst ones are quite fiddly, I've added variables for each host OS
we might want to check to make them easier to reuse.
2021-06-23 15:00:36 +00:00
regnat
c878cee895 Assert that compatible realisations have the same dependencies
Should always hold, but that’s not necessarily obvious, so better
enforce it
2021-06-23 11:27:16 +02:00
regnat
16fb7d8d95 Display the diverging paths in case of a realisation mismatch 2021-06-23 11:27:16 +02:00
regnat
40f925b2da Fix indentation 2021-06-23 11:27:16 +02:00
regnat
d32cf0c17a Gracefully ignore a substituter if it holds an incompatible realisation 2021-06-23 11:27:16 +02:00
regnat
b8f7177a7b Properly fail when trying to register an incoherent realisation 2021-06-23 11:27:14 +02:00
Eelco Dolstra
4a5aa1dbf6
Merge pull request #4838 from NixOS/ca/recursively-substitute-realisations
Recursively substitute the realisations
2021-06-23 10:33:25 +02:00
Eelco Dolstra
26d2c62225
Merge pull request #4906 from NixOS/collect-garbage-ca
Make `computeFSClosure` ca-aware
2021-06-23 10:07:08 +02:00
Eelco Dolstra
d9a43d3137
Merge pull request #4905 from NixOS/ca-derivations-machine-feature
Add a ca-derivations required machine feature
2021-06-23 10:05:53 +02:00
Eelco Dolstra
4f9508c3b5
Merge pull request #4836 from NixOS/ca/track-drvoutput-dependencies-2-le-retour
Track the dependencies of CA realisations
2021-06-23 10:03:31 +02:00
regnat
ed0e21a88d Fix indentation 2021-06-23 08:16:34 +02:00
regnat
7c96a76dd7 Reformat the sql statements 2021-06-23 08:16:34 +02:00
Théophane Hufschmitt
8d09a4f9a0 Remove a useless string split
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-06-23 08:16:34 +02:00
Théophane Hufschmitt
c13d7d0b97 Pass more values by reference
Rather than copying them around everywhere

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-06-23 08:16:34 +02:00
Maximilian Bosch
6f206549ba
Move writeStructuredAttrsShell out of ParsedDerivation class 2021-06-22 21:14:20 +02:00
Maximilian Bosch
27ce722638
Prefix env vars for attrs.* files with NIX_ 2021-06-22 19:45:08 +02:00
Maximilian Bosch
a92245b110
Remove now-obsolete typedef 2021-06-22 19:15:57 +02:00
Maximilian Bosch
f1e281c4fe
Split shell & json creation for build environments with structured attrs 2021-06-22 19:15:57 +02:00
Maximilian Bosch
3944a120ec
Set environment variables for .attrs.json & .attrs.sh
This way no derivation has to expect that these files are in the `cwd`
during the build. This is problematic for `nix-shell` where these files
would have to be inserted into the nix-shell's `cwd` which can become
problematic with e.g. recursive `nix-shell`.

To remain backwards-compatible, the location inside the build sandbox
will be kept, however using these files directly should be deprecated
from now on.
2021-06-22 19:15:57 +02:00
Maximilian Bosch
3b5429aec1
Source complete env in nix-shell with __structuredAttrs = true;
This is needed to push the adoption of structured attrs[1] forward. It's
now checked if a `__json` exists in the environment-map of the derivation
to be openend in a `nix-shell`.

Derivations with structured attributes enabled also make use of a file
named `.attrs.json` containing every environment variable represented as
JSON which is useful for e.g. `exportReferencesGraph`[2]. To
provide an environment similar to the build sandbox, `nix-shell` now
adds a `.attrs.json` to `cwd` (which is mostly equal to the one in the
build sandbox) and removes it using an exit hook when closing the shell.

To avoid leaking internals of the build-process to the `nix-shell`, the
entire logic to generate JSON and shell code for structured attrs was
moved into the `ParsedDerivation` class.

[1] https://nixos.mayflower.consulting/blog/2020/01/20/structured-attrs/
[2] https://nixos.org/manual/nix/unstable/expressions/advanced-attributes.html#advanced-attributes
2021-06-22 19:15:57 +02:00
regnat
dcabb46124 Shorten a stupidly long sql query name 2021-06-21 16:28:06 +02:00
Eelco Dolstra
610baf359a
Merge pull request #4833 from NixOS/ca/json-realisations-in-worker-protocol
Always send the realisations as JSON
2021-06-21 16:12:17 +02:00
regnat
3784c66a46 Remove a possible existing store path when building CA derivations
In case a previous interrupted build left a garbage path laying around,
remove it before trying to move the path to its final location.

Fix #4858
2021-06-21 15:29:15 +02:00
Eelco Dolstra
4202a3bc4e UDSRemoteStore: Support the 'root' store parameter
Useful when we're using a daemon with a chroot store, e.g.

  $ NIX_DAEMON_SOCKET_PATH=/tmp/chroot/nix/var/nix/daemon-socket/socket nix-daemon --store /tmp/chroot

Then the client can now connect with

  $ nix build --store unix:///tmp/chroot/nix/var/nix/daemon-socket/socket?root=/tmp/chroot nixpkgs#hello
2021-06-18 17:04:11 +02:00
regnat
96d7170e12 Don’t check the deriver field on computeFSClosure
That doesn’t really make sense with CA derivations (and wasn’t even
really correct before because of FO derivations, though that probably
didn’t matter much in practice)
2021-06-12 12:24:53 +02:00
regnat
7ac038fa4b Make computeFSClosure ca-aware
Fix #4820 by preventing nix-collect garbage from crashing if
`keep-outputs` or `keep-derivations` is true
2021-06-11 09:26:49 +02:00
regnat
7c077d2a0f Add a ca-derivations required machine feature
Make ca-derivations require a `ca-derivations` machine feature, and
ca-aware builders expose it.

That way, a network of builders can mix ca-aware and non-ca-aware
machines, and the scheduler will send them in the right place.
2021-06-11 09:12:53 +02:00
regnat
5985b8b527 Check the CA hash when importing stuff in the local store
When adding a path to the local store (via `LocalStore::addToStore`),
ensure that the `ca` field of the provided `ValidPathInfo` does indeed
correspond to the content of the path.
Otherwise any untrusted user (or any binary cache) can add arbitrary
content-addressed paths to the store (as content-addressed paths don’t
need a signature).
2021-06-01 15:09:24 +02:00
Eelco Dolstra
48396d940e
Merge pull request #4866 from alyssais/libdl
Only link with libdl on Linux
2021-06-01 11:50:50 +02:00
Eelco Dolstra
caef6f4314
Merge pull request #4734 from p01arst0rm/fix-s3-ifdef
unified macro style for ENABLE_S3
2021-06-01 11:30:21 +02:00
Alyssa Ross
c57ab17687
Only link with libdl on Linux
Linux is (as far as I know) the only mainstream operating system that
requires linking with libdl for dlopen.  On BSD, libdl doesn't exist,
so on non-FreeBSD BSDs linking will currently fail.  On macOS, it's
apparently just a symlink to libSystem (macOS libc), presumably
present for compatibility with things that assume Linux.

So the right thing to do here is to only add -ldl on Linux, not to add
it for everything that isn't FreeBSD.
2021-06-01 08:05:21 +00:00
regnat
a22755721b Recursively substitute the realisations
Make sure that whenever we substitute a realisation, we also substitute
its entire closure
2021-05-26 18:44:17 +02:00
regnat
1f3ff0d193 Aso track the output path of the realisation dependencies 2021-05-26 17:09:21 +02:00
regnat
cb46d70794 Add a db migration script 2021-05-26 16:59:09 +02:00
regnat
63ebfc73c5 Make copyPaths copy the whole realisations closure
Otherwise registering the realisations on the remote side might fail as
it now expects a complete closure
2021-05-26 16:59:09 +02:00
regnat
8c30acc3e8 Properly track the drvoutput references when building 2021-05-26 16:59:09 +02:00
regnat
af3afd25ea Add a method to compute the closure of a realisation
Only considers the closure in term of `Realisation`, ignores all the
opaque inputs.

Dunno whether that’s the nicest solution, need to think it through a bit
2021-05-26 16:59:09 +02:00
regnat
eca6ff06d6 Store the realisation deps on the local store 2021-05-26 16:59:09 +02:00
regnat
7ce0441d80 Add a dependencies field to DrvOutputInfo
Currently never used, nor set but will be useful shortly
2021-05-26 16:59:09 +02:00
regnat
7616268812 Always send the realisations as JSON
Align all the worker protocol with `buildDerivation` which inlines the
realisations as one opaque json blob.
That way we don’t have to bother changing the remote store protocol
when the definition of `Realisation` changes, as long as we keep the
json backwards-compatible
2021-05-26 16:59:09 +02:00
regnat
129384bcf3 Remove the remaining occurenceses of a NarHash modulo 2021-05-26 09:39:29 +02:00
regnat
79ae9e4558 Make the Nar hash non modulo
It makes much more sense to have the Nar hash be a plain straight hash
rather than a hash modulo
2021-05-25 10:58:43 +02:00
regnat
a8416866cf Always send the realisations as JSON
Align all the worker protocol with `buildDerivation` which inlines the
realisations as one opaque json blob.
That way we don’t have to bother changing the remote store protocol
when the definition of `Realisation` changes, as long as we keep the
json backwards-compatible
2021-05-19 11:45:16 +02:00
regnat
184558834a Extract a generic computeClosure function
Move the `closure` logic of `computeFSClosure` to its own (templated) function.

This doesn’t bring much by itself (except for the ability to properly
test the “closure” functionality independently from the rest), but it
allows reusing it (in particular for the realisations which will require
a very similar closure computation)
2021-05-19 11:44:58 +02:00
Eelco Dolstra
6849ae82de
Merge pull request #4818 from NixOS/ca/cli-use-builtpaths
Enforce the use of properly built paths in libcmd
2021-05-17 16:15:40 +02:00
regnat
2105084645 Enfore the use of properly built paths in libcmd
Replace `DerivedPathWithHints` by a new `BuiltPath` type that serves as
a proof that the corresponding path has been built.
2021-05-17 08:45:08 +02:00
Emily
559a504da7 sandbox: allow SystemVersionCompat.plist on Darwin
For whatever reason, many programs trying to access SystemVersion.plist
also open SystemVersionCompat.plist; this includes Python code and
coreutils’ `cat(1)` (but not the native macOS `/bin/cat`). Illustratory
`dtruss(1m)` output:

    open("/System/Library/CoreServices/SystemVersion.plist\0", 0x0, 0x0)		 = 3 0
    open("/System/Library/CoreServices/SystemVersionCompat.plist\0", 0x0, 0x0)		 = 4 0

I assume this is a Big Sur change relating to the 10.16.x/11.x
version compatibility divide and that it’s something along the lines of
a hook inside libSystem.

Fixes a lot of sandboxed package builds under Big Sur.
2021-05-17 01:24:31 +01:00
Eelco Dolstra
340f831ebe
Merge pull request #4798 from matthewbauer/relock-wait-for-build-slot-goals
Relock wait for build slot goals
2021-05-13 13:07:35 +02:00
Eelco Dolstra
89a4ede92b
Merge pull request #4801 from matthewbauer/fix-tokenize-output-names
Fix tokenize output names in DerivedPath
2021-05-13 13:06:29 +02:00
Matthew Bauer
8c7e043de2 Fix tokenize output names in drv
This should fix the issue described in
https://discourse.nixos.org/t/derivation-does-not-have-wanted-outputs-dev-out/12905.

Specifically, we get an error of

  error: derivation '/nix/store/_.drv' does not have wanted outputs 'dev,out'

when a path like /nix/store/_.drv!dev,out is sent to the daemon.
2021-05-12 21:40:28 -05:00
Matthew Bauer
9a14335845 Relock wait for build slot goals
When we don’t have enough free job slots to run a goal, we put it in
the waitForBuildSlot list & unlock its output locks. This will
continue from where we left off (tryLocalBuild). However, we need the
locks to get reacquired when/if the goal ever restarts. So, we need to
send it back through tryToBuild to get reqacquire those locks.

I think this bug was introduced in
https://github.com/NixOS/nix/pull/4570. It leads to some builds
starting without proper locks.
2021-05-12 12:15:32 -05:00
regnat
ec613603ba DerivedPathWithHints -> BuiltPath
Just a renaming for now
2021-05-12 16:19:51 +02:00
Lorenzo Manacorda
4029f4b05b doc: mention Priority for substituters 2021-05-12 13:27:05 +02:00
Eelco Dolstra
7f9759b18d
Merge pull request #4781 from NixOS/locally_cache_the_remote_realisations
Add a realisations disk cache
2021-05-10 20:37:57 +02:00
regnat
d5d19582ef Simplify the realisations disk cache 2021-05-10 17:47:14 +02:00
regnat
ab96c1ee50 Remove useless parents
I never remember the exact syntax of the `switch` statement
2021-05-10 17:36:49 +02:00
regnat
b66234134f Add a realisations disk cache
Similar to the nar-info disk cache (and using the same db).
This makes rebuilds muuch faster.

- This works regardless of the ca-derivations experimental feature.
  I could modify the logic to not touch the db if the flag isn’t there,
  but given that this is a trash-able local cache, it doesn’t seem to be
  really worth it.
- We could unify the `NARs` and `Realisation` tables to only have one
  generic kv table. This is left as an exercise to the reader.
- I didn’t update the cache db version number as the new schema just
  adds a new table to the previous one, so the db will be transparently
  migrated and is backwards-compatible.

Fix #4746
2021-05-06 17:38:20 +02:00
regnat
bf485dcf46 Properly normalize the content-addressed paths
Make sure that their timestamp are always normalized.
Otherwise, strange − and non-deterministic − things might happen, like
https://github.com/NixOS/nixpkgs/issues/121813

Fix #4775
2021-05-05 21:00:08 +02:00
regnat
dadfbce318 Fix the double-slash in the realisations path
Make sure that we always access the realisations under
`binaryCacheUrl/realisations` and not `binaryCacheUrl//realisations`

Fix #4766
2021-05-04 10:35:34 +02:00
Eelco Dolstra
40378fbcba
Merge pull request #4761 from emilazy/issue-4658-mark-impure-host-deps-as-optional
Mark `__impureHostDeps` paths as optional
2021-05-03 16:13:36 +02:00
Maximilian Bosch
e5951a6b2f
Bump version number for DerivedPath changes
I guess I misunderstood John's initial explanation about why wildcards
for outputs are sent to older stores[1]. My `nix-daemon` from 2021-03-26
also has version 1.29, but misses the wildcard[2]. So bumping seems to
be the right call.

[1] https://github.com/NixOS/nix/pull/4759#issuecomment-830812464
[2] 255d145ba7
2021-05-03 01:12:23 +02:00
Emily
c4355a52fa Mark __impureHostDeps paths as optional
Starting in macOS 11, the on-disk dylib bundles are no longer available,
but nixpkgs needs to be able to keep compatibility with older versions
that require `/usr/lib/libSystem.B.dylib` in `__impureHostDeps`. Allow
it to keep backwards compatibility with these versions by marking these
dependencies as optional.

Fixes #4658.
2021-05-02 05:30:50 +01:00
John Ericson
e023c985d5 Merge remote-tracking branch 'upstream/master' into auto-uid-allocation 2021-04-27 19:06:58 +00:00
Eelco Dolstra
fe2bf464cf Merge branch 'remove-trailing-spaces' of github.com:NixOS/nix 2021-04-23 15:09:44 +02:00
regnat
31313d1401 Replace the trailing markdown spaces by a backslash
They are equivalent according to
<https://spec.commonmark.org/0.29/#hard-line-breaks>,
and the trailing spaces tend to be a pain (because the make git
complain, editors tend to want to remove them − the `.editorconfig`
actually specifies that − etc..).
2021-04-23 14:37:21 +02:00
p01arst0rm
45473d02c9 unified macro style for ENABLE_S3 2021-04-23 07:30:05 +01:00
regnat
9161e02039 Always register the realisations of input-addressed drvs
Fix #4725
2021-04-22 20:07:02 +02:00
Alyssa Ross
9ac6534f7c
Include sys/wait.h everywhere WIFEXITED etc is used
This is required on NetBSD, and I think FreeBSD too.
2021-04-19 18:31:58 +00:00
Eelco Dolstra
6fb7582413 Merge branch 'libarchive-decompress' of https://github.com/serokell/nix 2021-04-15 13:39:04 +02:00
regnat
ed29610cc6 Fix registerDrvOutput with the daemon
Resolve a protocol issue that caused the daemon to endlessly wait for
some information that the client doesn't ever send
2021-04-13 13:58:22 +02:00
Domen Kožar
c2dfda007e
add tests for zstd compression 2021-04-09 23:13:08 +02:00
Eelco Dolstra
9b9e703df4 restoreSignals() + restoreAffinity() -> restoreProcessContext() 2021-04-07 13:10:02 +02:00
Eelco Dolstra
8a29052cb2 PathSubstitutionGoal: Clean up pipe
If there were many top-level goals (which are not destroyed until the
very end), commands like

  $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' \
    /run/current-system --no-check-sigs --substitute-on-destination

could fail with "Too many open files". So now we do some explicit
cleanup from amDone(). It would be cleaner to separate goals from
their temporary internal state, but that would be a bigger refactor.
2021-04-07 12:21:31 +02:00
John Ericson
1b6cf0d5f5 Merge remote-tracking branch 'upstream/master' into path-info 2021-04-05 18:47:33 -04:00
John Ericson
d5cef6c33a Merge commit '9dfb97c987d8b9d6a3d15f016e40f22f91deb764' into path-info 2021-04-05 18:40:30 -04:00
John Ericson
e12308dd63 Merge branch 'master' into path-info 2021-04-05 18:29:52 -04:00
John Ericson
125a824228 Document the derived path types. 2021-04-05 10:56:48 -04:00
John Ericson
d8fa7517fa buildable.{cc,hh} -> derived-path.{cc,hh} 2021-04-05 10:33:28 -04:00
John Ericson
179582872d Make DerivedPathWithHints a newtype
This allows us to namespace its constructors under it.
2021-04-05 10:05:21 -04:00
John Ericson
9b805d36ac Rename Buildable 2021-04-05 09:52:25 -04:00
John Ericson
9dfb97c987 "newtype" BuildableReq
This makes for better types errors and allows us to give it methods.
2021-04-05 09:35:55 -04:00
John Ericson
4fe41c6db3 No templates for Buildable and BuildableReq 2021-04-05 09:15:25 -04:00
John Ericson
255d145ba7 Use BuildableReq for buildPaths and ensurePath
This avoids an ambiguity where the `StorePathWithOutputs { drvPath, {}
}` could mean "build `brvPath`" or "substitute `drvPath`" depending on
context.

It also brings the internals closer in line to the new CLI, by
generalizing the `Buildable` type is used there and makes that
distinction already.

In doing so, relegate `StorePathWithOutputs` to being a type just for
backwards compatibility (CLI and RPC).
2021-04-05 08:33:00 -04:00
John Ericson
32f4454b9f Move StorePathWithOutput utilities out of store class
These are by no means part of the notion of a store, but rather are
things that happen to use stores. (Or put another way, there's no way
we'd make them virtual methods any time soon.) It's better to move them
out of that too-big class then.

Also, this helps us remove StorePathWithOutputs from the Store interface
altogether next commit.
2021-04-05 08:31:37 -04:00
John Ericson
7a2b566dc8 Move StorePathWithOutputs into its own header/file
In the following commits it will become less prevalent.
2021-04-05 08:31:37 -04:00
John Ericson
f7d9f7c338 Pull out Buildable into its own file/header in libnixstore 2021-04-05 08:31:37 -04:00
Eelco Dolstra
dd77f71afe LocalBinaryCacheStore::upsertFile(): Fix race
When multiple threads try to upsert the same file, this could fail.

Fixes #4667.
2021-03-26 17:10:15 +01:00
Eelco Dolstra
4638bcfb2c Fix some typos
Fixes #4671.
2021-03-26 16:14:38 +01:00
Eelco Dolstra
1581c3e8ef
Merge pull request #4603 from DavHau/davhau-improve-nix-conf-help-builders
improve man page for nix.conf (builders)
2021-03-23 16:13:02 +01:00
DavHau
0f40561c78 nix.conf builders: refer to manual page 2021-03-23 10:19:00 +07:00
John Ericson
f44206e719 Fix typos in the last PR #4656 2021-03-22 15:18:48 +00:00
John Ericson
9d309de0de Clean up serialization for BuildResult
A few versioning mistakes were corrected:

- In 27b5747ca7, Daemon protocol had some
  version `>= 0xc` that should have been `>= 0x1c`, or `28` since the
  other conditions used decimal.

- In a2b69660a9, legacy SSH gated new CAS
  info on version 6, but version 5 in the server. It is now 6
  everywhere.

Additionally, legacy ssh was sending over more metadata than the daemon
one was. The daemon now sends that data too.

CC @regnat

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2021-03-22 14:57:41 +00:00
Robert Hensing
a61112aadf Remove unimplemented hashAlgoOpt
It was in the header but never implemented.
2021-03-17 11:27:11 +01:00
Eelco Dolstra
ccb8a403ee
Merge pull request #4587 from obsidiansystems/derivation-goal-detect-invalid-output
Throw error for derivation goal with bogus wanted output
2021-03-15 16:49:44 +01:00
Eelco Dolstra
c0073f6268
Merge pull request #4580 from obsidiansystems/restore-test-build-remote-ca-fixed
Restore now-working build-remote-content-addressed-fixed test
2021-03-15 16:48:28 +01:00
regnat
703c98c6cb Properly sign the unresolved drvs
Don't let them inherit the signature from the parent one (because it
makes no sense to do so), but re-sign them after they have been built
2021-03-15 16:35:17 +01:00
regnat
54ced9072b Check the signatures when copying store paths around
Broken atm
2021-03-15 16:35:14 +01:00
regnat
3e6017f911 pathInfoIsTrusted -> pathInfoIsUntrusted
I guess the rationale behind the old name wath that
`pathInfoIsTrusted(info)` returns `true` iff we would need to `blindly`
trust the path (because it has no valid signature and `requireSigs` is
set), but I find it to be a really confusing footgun because it's quite
natural to give it the opposite meaning.
2021-03-15 16:34:49 +01:00
regnat
826877cabf Add some logic for signing realisations
Not exposed anywhere, but built realisations are now signed (and this
should be forwarded when copy-ing them around)
2021-03-15 16:34:49 +01:00
Eelco Dolstra
306c154632
Merge pull request #4592 from NixOS/ca/remote-cache
Substitute content-addressed derivations
2021-03-15 16:22:42 +01:00
Yorick van Pelt
8a0c00b856
Use libarchive for all compression 2021-03-10 22:34:29 +01:00
John Ericson
6212e89bf6 Avoid some StorePath -> Path -> StorePath roundtrips
There were done when StorePath was defined in Rust and there were some
FFI issues. This is no longer an issue.
2021-03-05 00:49:46 +00:00
DavHau
e16431b466 improve man page for nix.conf (builders) 2021-03-04 16:14:23 +07:00
John Ericson
7ce10924c7 Fix bad wanted output error as requested
- UsageError -> Error

- include drv path too
2021-03-01 15:07:09 +00:00
regnat
df9d4f88d5 Allow substituting drv outputs when building 2021-03-01 14:00:17 +01:00
regnat
5d1c05b075 SubstitutionGoal -> PathSubstitutionGoal
To prepare for the upcoming DrvOutputSubstitutionGoal
2021-03-01 14:00:17 +01:00
Eelco Dolstra
e64cf8e0a3
Merge pull request #4574 from grahamc/libstore-ssh-host-key
libstore: support passing a builder's public SSH host key
2021-03-01 13:12:18 +01:00
John Ericson
4bbd80c536 Throw error for derivation goal with bogus wanted output 2021-02-28 00:19:35 +00:00
John Ericson
f0ad29acc1 Merge remote-tracking branch 'upstream/master' into path-info 2021-02-27 05:42:13 +00:00
John Ericson
ae1441e548 Fix testing fixed-output derivations in double sandboxes
What happened was that Nix was trying to unconditionally mount these
paths in fixed-output derivations, but since the outer derivation was
pure, those paths did not exist. The solution is to only mount those
paths when they exist.
2021-02-27 05:23:14 +00:00
Puck Meerburg
7241fdc3d2 Properly propagate libseccomp linker flags 2021-02-26 23:01:16 +00:00
John Ericson
553b79f8c9 Remove unused redirectedBadOutputs 2021-02-26 16:10:54 +00:00
John Ericson
d560311f76 Remove temporary #if 0...#endif from previous commit 2021-02-26 16:10:52 +00:00
John Ericson
68f4c728ec Split {,local-}derivation-goal.{cc,hh}
This separates the scheduling logic (including simple hook pathway) from
the local-store needing code.

This should be the final split for now. I'm reasonably happy with how
it's turning out, even before I'm done moving code into
`local-derivation-goal`. Benefits:

1. This will help "witness" that the hook case is indeed a lot simpler,
   and also compensate for the increased complexity that comes from
   content-addressed derivation outputs.

2. It also moves us ever so slightly towards a world where we could use
   off-the-shelf storage or sandboxing, since `local-derivation-goal`
   would be gutted in those cases, but `derivation-goal` should remain
   nearly the same.

The new `#if 0` in the new files will be deleted in the following
commit. I keep it here so if it turns out more stuff can be moved over,
it's easy to do so in a way that preserves ordering --- and thus
prevents conflicts.

N.B.
```sh
git diff HEAD^^ --color-moved --find-copies-harder --patience --stat
```
makes nicer output.
2021-02-26 16:10:26 +00:00
John Ericson
05cc5a8587 Copy {,local-}derivation-goal.{cc,h}
Doing this prior to splitting, so we get better diff with default
options (e.g. on GitHub).
2021-02-26 16:01:47 +00:00
Eelco Dolstra
94637cd7e5
Merge pull request #4477 from NixOS/ca/build-remote
Build ca derivations remotely
2021-02-26 16:54:44 +01:00
regnat
f54976d77b Simplify the case where the drv is a purely input-addressed one 2021-02-26 16:35:05 +01:00
Eelco Dolstra
076d2b04da
Update src/libstore/build/derivation-goal.cc 2021-02-26 16:30:12 +01:00
Eelco Dolstra
20ea1de77d Use std::make_unique 2021-02-26 12:35:29 +01:00
Eelco Dolstra
73daffb81b Merge remote-tracking branch 'origin/deduplicate-static-hashes-computation' 2021-02-26 12:30:25 +01:00
John Ericson
ca0994819d Merge remote-tracking branch 'upstream/master' into path-info 2021-02-25 21:51:05 +00:00
regnat
c43f446f4e Make nix copy work without the ca-derivations flag
The experimental feature was by mistake required for `nix copy` to work
at oll
2021-02-25 17:21:51 +01:00
Théophane Hufschmitt
c182aac98a Apply @edolstra stylistic suggestions
Mostly removing useless comments and adding spaces before `&`

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-02-25 17:21:51 +01:00
regnat
3b76f8f252 Ensure that the ca-derivations bit is set when copying realisations
This should already hold, but better ensure it for future-proof-nees
2021-02-25 17:19:59 +01:00
regnat
f67ff1f575 Don't crash when copying realisations to a non-ca remote
Rather throw a proper exception, and catch&log it on the client side
2021-02-25 17:19:59 +01:00
regnat
2e199673a5 Use RealisedPaths in copyPaths
That way we can copy the realisations too (in addition to the store
paths themselves)
2021-02-25 17:18:48 +01:00
Graham Christensen
1130b28824
distributed builds: load remote builder host key from the machines file
This is already used by Hydra, and is very useful when materializing
a remote builder list from service discovery. This allows the service
discovery tool to only sync one file instead of two.
2021-02-25 09:17:34 -05:00
Shea Levy
98d1b64400
Initialize plugins after handling initial command line flags
This is technically a breaking change, since attempting to set plugin
files after the first non-flag argument will now throw an error. This
is acceptable given the relative lack of stability in a plugin
interface and the need to tie the knot somewhere once plugins can
actually define new subcommands.
2021-02-24 08:22:17 -05:00
Shea Levy
ec3497c1d6
Bail if plugin-files is set after plugins have been loaded.
We know the flag will be ignored but the user wants it to take effect.
2021-02-24 08:20:48 -05:00
Eelco Dolstra
a878c448d8
Merge pull request #4551 from danieldk/system-features-compute-level
Add x86_64 compute levels as system features
2021-02-23 14:30:10 +01:00
regnat
ba1a256d08 Make DerivationGoal::drv a full Derivation
This field used to be a `BasicDerivation`, but this `BasicDerivation`
was downcasted to a `Derivation` when needed (implicitely or not), so we
might as well make it a full `Derivation` and upcast it when needed.

This also allows getting rid of a weird duplication in the way we
compute the static output hashes for the derivation. We had to
do it differently and in a different place depending on whether the
derivation was a full derivation or just a basic drv, but we can now do
it unconditionally on the full derivation.

Fix #4559
2021-02-23 14:15:45 +01:00
regnat
527da73690 Properly bypass the registering step when all outputs are present
There was already some logic for that, but it didn't handle the case of
content-addressed outputs, so extend it a bit for that
2021-02-23 08:04:03 +01:00
regnat
8c385d16ee Also send ca outputs to the build hook
Otherwise they don't get registered, triggering an assertion failure
at some point later
2021-02-23 08:04:03 +01:00
regnat
27b5747ca7 RemoteStore: Send back the new realisations
To allow it to build ca derivations remotely
2021-02-23 08:04:03 +01:00
regnat
a2b69660a9 LegacySSHStore: Send back the new realisations
To allow it to build ca derivations remotely
2021-02-23 08:04:03 +01:00
regnat
5687564a27 LocalStore: Send back the new realisations
To allow it to build ca derivations remotely
2021-02-23 08:04:03 +01:00
regnat
6fbf3fe636 Make the build-hook work with ca derivations
- Pass it the name of the outputs rather than their output paths (as
  these don't exist for ca derivations)
- Get the built output paths from the remote builder
- Register the new received realisations
2021-02-23 08:04:03 +01:00
Shea Levy
35205e2e92
Warn about instability of plugin API 2021-02-22 17:10:55 -05:00
Daniël de Kok
2de232d2b3 Add x86_64 compute levels as additional system types
When performing distributed builds of machine learning packages, it
would be nice if builders without the required SIMD instructions can
be excluded as build nodes.

Since x86_64 has accumulated a large number of different instruction
set extensions, listing all possible extensions would be unwieldy.
AMD, Intel, Red Hat, and SUSE have recently defined four different
microarchitecture levels that are now part of the x86-64 psABI
supplement and will be used in glibc 2.33:

https://gitlab.com/x86-psABIs/x86-64-ABI
https://lwn.net/Articles/844831/

This change uses libcpuid to detect CPU features and then uses them to
add the supported x86_64 levels to the additional system types. For
example on a Ryzen 3700X:

$ ~/aps/bin/nix -vv --version | grep "Additional system"
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
2021-02-22 09:11:15 +01:00
regnat
f483b623e9 Remove the drv resolution caching mechanism
It isn't needed anymore now that don't need to eagerly resolve
everything like we used to do. So we can safely get rid of it
2021-02-19 15:48:31 +01:00
regnat
4bc28c44f2 Store the output hashes in the initialOutputs of the drv goal
That way we
1. Don't have to recompute them several times
2. Can compute them in a place where we know the type of the parent
  derivation, meaning that we don't need the casting dance we had before
2021-02-19 15:48:31 +01:00
Théophane Hufschmitt
0bfbd04369 Don't expose the "bang" drvoutput syntax
It's not fixed nor useful atm, so better keep it hidden

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-02-19 15:48:31 +01:00
Théophane Hufschmitt
93d9eb78a0 Syntactic fixes
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-02-19 15:48:31 +01:00
regnat
87c8d3d702 Register the realisations for unresolved drvs
Once a build is done, get back to the original derivation, and register
all the newly built outputs for this derivation.

This allows Nix to work properly with derivations that don't have all
their build inputs available − thus allowing garbage collection and
(once it's implemented) binary substitution
2021-02-19 15:48:31 +01:00
Graham Christensen
f2245091d0
Revert "narinfo: Change NAR URLs to be addressed on the NAR hash instead of the compressed hash" 2021-02-09 12:26:41 -05:00
Eelco Dolstra
ee3846b587
Merge pull request #4464 from tweag/nar-narhash-addressed
narinfo: Change NAR URLs to be addressed on the NAR hash instead of the compressed hash
2021-02-09 14:47:39 +01:00
Eelco Dolstra
480426a364 Add more instrumentation for #4270 2021-02-05 15:57:33 +01:00
Eelco Dolstra
d7c27f21ab
Merge pull request #4372 from tweag/ca/drvoutputs-commands
Add a new Cmd type working on RealisedPaths
2021-02-05 13:03:50 +01:00
Eelco Dolstra
0187838e2e Add a trace to readLine() failures
Hopefully this helps to diagnose 'error: unexpected EOF reading a
line' on macOS.
2021-02-05 12:18:11 +01:00
regnat
e69cfdebb0 Remove the visit machinery in RealisedPath
In addition to being some ugly template trickery, it was also totally
useless as it was used in only one place where I could replace it by
just a few extra characters
2021-02-05 11:42:33 +01:00
regnat
d2091af231 Move the GENERATE_CMP macro to its own file
Despite being an ugly hack, it can probably be useful in a couple extra
places
2021-02-05 11:42:33 +01:00
Théophane Hufschmitt
43d409f669
Fix a whitespace issue
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-02-04 14:47:56 +01:00
Eelco Dolstra
b19aec7eeb
Merge pull request #4461 from NixOS/ca/error-logging-fixes
Fix some logging with ca derivations
2021-01-29 16:12:50 +01:00
regnat
9355ecd543 Add a new Cmd type working on RealisedPaths
Where a `RealisedPath` is a store path with its history, meaning either
an opaque path for stuff that has been directly added to the store, or a
`Realisation` for stuff that has been built by a derivation

This is a low-level refactoring that doesn't bring anything by itself
(except a few dozen extra lines of code :/ ), but raising the
abstraction level a bit is important on a number of levels:

- Commands like `nix build` have to query for the realisations after the
  build is finished which is fragile (see
  27905f12e4a7207450abe37c9ed78e31603b67e1 for example). Having them
  oprate directly at the realisation level would avoid that
- Others like `nix copy` currently operate directly on (built) store
  paths, but need a bit more information as they will need to register
  the realisations on the remote side
2021-01-28 09:38:44 +01:00
Eelco Dolstra
965dc6070a Drop trailing whitespace 2021-01-27 14:04:49 +01:00
regnat
9da11bac57 Fix the error message when a dep is missing
Fix a mismatch in the errors thrown when a needed output was missing
from an input derivation that was leading to a wrong and quite misleading error
message
2021-01-26 14:49:23 +01:00
regnat
d3c4284133 Make the error message for missing outputs more useful
Don't only show the name of the output, but also the derivation to which
this output belongs (as otherwise it's very hard to track back what went
wrong)
2021-01-26 14:49:23 +01:00
Eelco Dolstra
3ba98ba8f0 Tell user to run 'nix log' to get full build logs 2021-01-25 17:15:38 +01:00
Eelco Dolstra
488a826842
Merge pull request #4467 from edolstra/error-formatting
Improve error formatting
2021-01-25 12:50:57 +01:00
Eelco Dolstra
c5b42c5a42
Merge pull request #4470 from matthewbauer/fix-4469
Handle missing etag in 304 Not Modified response
2021-01-25 12:50:13 +01:00
Eelco Dolstra
680d8a5b86
Merge pull request #4387 from obsidiansystems/non-local-store-build
Make `nix-build --store whatever` work
2021-01-25 12:24:23 +01:00
Matthew Bauer
a766824660 Handle missing etag in 304 Not Modified response
GitHub now omits the etag, but 304 implies it matches the one we
provided. Just use that one to avoid having an etag-less resource.

Fixes #4469
2021-01-22 14:47:45 -06:00
John Ericson
53a709535b Apply suggestions from code review
Thanks!

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-01-22 15:58:58 +00:00
John Ericson
8c07ed1dda Improve documentation and test and requested 2021-01-22 15:58:58 +00:00
Eelco Dolstra
8d4268d190 Improve error formatting
Changes:

* The divider lines are gone. These were in practice a bit confusing,
  in particular with --show-trace or --keep-going, since then there
  were multiple lines, suggesting a start/end which wasn't the case.

* Instead, multi-line error messages are now indented to align with
  the prefix (e.g. "error: ").

* The 'description' field is gone since we weren't really using it.

* 'hint' is renamed to 'msg' since it really wasn't a hint.

* The error is now printed *before* the location info.

* The 'name' field is no longer printed since most of the time it
  wasn't very useful since it was just the name of the exception (like
  EvalError). Ideally in the future this would be a unique, easily
  googleable error ID (like rustc).

* "trace:" is now just "…". This assumes error contexts start with
  something like "while doing X".

Example before:

  error: --- AssertionError ---------------------------------------------------------------------------------------- nix
  at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

       6|
       7|   x = assert false; 1;
        |       ^
       8|

  assertion 'false' failed
  ----------------------------------------------------- show-trace -----------------------------------------------------
  trace: while evaluating the attribute 'x' of the derivation 'hello-2.10'
  at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix

     191|         // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) {
     192|           name = "${attrs.pname}-${attrs.version}";
        |           ^
     193|         } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {

Example after:

  error: assertion 'false' failed

         at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

              6|
              7|   x = assert false; 1;
               |       ^
              8|

         … while evaluating the attribute 'x' of the derivation 'hello-2.10'

         at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix

            191|         // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) {
            192|           name = "${attrs.pname}-${attrs.version}";
               |           ^
            193|         } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {
2021-01-21 11:02:09 +01:00
adisbladis
144cad9069
narinfo: Change NAR URLs to be addressed on the NAR hash instead of the compressed hash
This change is to simplify [Trustix](https://github.com/tweag/trustix) indexing and makes it possible to reconstruct this URL regardless of the compression used.

In particular this means that 7c2e9ca597/contrib/nix/nar/nar.go (L61-L71) can be removed and only the bits that are required to establish trust needs to be published in the Trustix build logs.
2021-01-21 10:32:56 +01:00
Eelco Dolstra
ea756b3654 --refresh: Imply setting .narinfo disk cache TTL to 0 2021-01-18 14:38:31 +01:00
regnat
9432c170e7 Fix the drv output map for non ca derivations
With the `ca-derivation` experimental features, non-ca derivations used
to have their output paths returned as unknown as long as they weren't
built (because of a mistake in the code that systematically erased the
previous value)
2021-01-18 11:01:39 +01:00
John Ericson
7af743470c Make public keys and requireSigs local-store specific again
Thanks @regnat and @edolstra for catching this and comming up with the
solution.

They way I had generalized those is wrong, because local settings for
non-local stores is confusing default. And due to the nature of C++
inheritance, fixing the defaults is more annoying than it should be.
Additionally, I thought we might just drop the check in the substitution
logic since `Store::addToStore` is now streaming, but @regnat rightfully
pointed out that as it downloads dependencies first, that would still be
too late, and also waste effort on possibly unneeded/unwanted
dependencies.

The simple and correct thing to do is just make a store method for the
boolean logic, keeping all the setting and key stuff the way it was
before. That new method is both used by `LocalStore::addToStore` and the
substitution goal check. Perhaps we might eventually make it fancier,
e.g. sending the ValidPathInfo to remote stores for them to validate,
but this is good enough for now.
2021-01-15 16:37:41 +00:00
John Ericson
0027b05a15 Merge remote-tracking branch 'upstream/master' into non-local-store-build 2021-01-15 02:01:24 +00:00
Eelco Dolstra
61216d32e1 Add 'nix store repair' command 2021-01-13 23:27:39 +01:00
Eelco Dolstra
4e9cec79bf
Merge pull request #4444 from matthewbauer/unset-curproc-arch-affinity
Set kern.curproc_arch_affinity=0 to escape Rosetta
2021-01-13 12:16:53 +01:00
Rickard Nilsson
0ca1a50132 Remove a redundant condition in DerivationGoal::tryLocalBuild() 2021-01-13 10:13:51 +01:00
Matthew Bauer
f69820417f Set kern.curproc_arch_affinity=0 to escape Rosetta
By default, once you enter x86_64 Rosetta 2, macOS will try to run
everything in x86_64. So an x86_64 Nix will still try to use x86_64
even when system = aarch64-darwin. To avoid this we can set
kern.curproc_arch_affinity sysctl. With kern.curproc_arch_affinity=0,
we ignore this preference.

This is based on how
https://opensource.apple.com/source/system_cmds/system_cmds-880.40.5/arch.tproj/arch.c.auto.html
works. Completely undocumented, but seems to work!

Note, you can verify this works with this impure Nix expression:

```
  {
    a = derivation {
      name = "a";
      system = "aarch64-darwin";
      builder = "/bin/sh";
      args = [ "-e" (builtins.toFile "builder" ''
        [ "$(/usr/bin/arch)" = arm64 ]
        [ "$(/usr/bin/arch -arch x86_64 /bin/sh -c /usr/bin/arch)" = i386 ]
        [ "$(/usr/bin/arch -arch arm64 /bin/sh -c /usr/bin/arch)" = arm64 ]
        /usr/bin/touch $out
      '') ];
    };

    b = derivation {
      name = "b";
      system = "x86_64-darwin";
      builder = "/bin/sh";
      args = [ "-e" (builtins.toFile "builder" ''
        [ "$(/usr/bin/arch)" = i386 ]
        [ "$(/usr/bin/arch -arch x86_64 /bin/sh -c /usr/bin/arch)" = i386 ]
        [ "$(/usr/bin/arch -arch arm64 /bin/sh -c /usr/bin/arch)" = arm64 ]
        /usr/bin/touch $out
      '') ];
    };
  }
```
2021-01-11 22:40:21 -06:00
Rickard Nilsson
44fd7a05b6 Don't let 'preferLocalBuild' override 'max-jobs=0'
This resolves #3810 by changing the behavior of `max-jobs = 0`, so
that specifying the option also avoids local building of derivations
with the attribute `preferLocalBuild = true`.
2021-01-12 01:28:00 +01:00
Eelco Dolstra
6548b89cc4 string2Int(): Return std::optional 2021-01-08 12:22:21 +01:00
Eelco Dolstra
0813350349 Add 'nix store prefetch-{file,tarball}'
These replace nix-prefetch-url and nix-prefetch-url --unpack,
respectively.
2021-01-07 21:51:46 +01:00
Eelco Dolstra
0df69d96e0 Make sodium a required dependency 2021-01-06 17:56:53 +01:00
Eelco Dolstra
555152ffe8 crypto.cc: API cleanup and add generate() / to_string() methods 2021-01-06 17:04:46 +01:00
Eelco Dolstra
3edcb198e5
Merge pull request #4310 from matthewbauer/rosetta2-extra-platforms
Add x86_64-darwin and aarch64 to "extra-platforms" automatically when Rosetta2 is detected
2021-01-06 11:31:13 +01:00
Eelco Dolstra
8af4f886e2 Fix deadlock in LocalStore::addSignatures()
Fixes #4367.
2021-01-05 11:47:29 +01:00
Eelco Dolstra
6262a70363
scanForReferences: Remove misleading comment
References have always been determined only by the hash part, not the
name or the store prefix.

Fixes #4396.
2020-12-28 17:21:19 +01:00
John Ericson
fed1237246 Test nix-build with non-local-store --store
Just a few small things needed fixing!
2020-12-23 22:42:06 +00:00
John Ericson
85f2e9e8fa Expose schedule entrypoints to all stores
Remote stores still override so the other end schedules.
2020-12-23 22:42:06 +00:00
John Ericson
450c3500f1 Crudely make worker only provide a Store, not LocalStore
We downcast in a few places, this will be refactored to be better later.
2020-12-23 22:42:06 +00:00
John Ericson
57062179ce Move some PKI stuff from LocalStore to Store 2020-12-23 22:42:06 +00:00
Eelco Dolstra
8927cba62f
Merge pull request #4366 from NixOS/readInvalidDerivation-on-remote-caches
Use the fs accessor for readInvalidDerivation
2020-12-23 11:55:52 +01:00
John Ericson
1a1af75338 Overhaul store subclassing
We embrace virtual the rest of the way, and get rid of the
`assert(false)` 0-param constructors.

We also list config base classes first, so the constructor order is
always:

  1. all the configs
  2. all the stores

Each in the same order
2020-12-20 15:47:14 +00:00
regnat
4d45839499 Fix the detection of already built drv outputs
PRs #4370 and #4348 had a bad interaction in that the second broke the fist
one in a not trivial way.

The issue was that since #4348 the logic for detecting whether a
derivation output is already built requires some logic that was specific
to the `LocalStore`.

It happens though that most of this logic could be upstreamed to any `Store`,
which is what this commit does.
2020-12-17 11:35:24 +01:00
Eelco Dolstra
ae3c3e3bb2
Merge pull request #4370 from NixOS/ca/more-precise-build-noop
Better detect when `buildPaths` would be a no-op
2020-12-16 14:54:04 +01:00
regnat
cac8d5b742 Don't ignore an absent drv file in queryPartialDrvOutputMap
This ignore was here because `queryPartialDrvOutputMap` was used both
1. as a cache to avoid having to re-read the derivation (when gc-ing for
example), and
2. as the source of truth for ca realisations

The use-case 2. required it to be able to work even when the derivation
wasn't there anymore (see https://github.com/NixOS/nix/issues/4138).
However, this use-case is now handled by `queryRealisation`, meaning
that we can safely error out if the derivation isn't there anymore
2020-12-16 13:36:17 +01:00
Eelco Dolstra
3765174691
Merge pull request #4348 from NixOS/ca/use-hashmodulo
Use the hash modulo in the derivation outputs
2020-12-16 12:48:44 +01:00
regnat
962b82ef25 Fix BinaryCacheStore::registerDrvOutput
Was crashing because coercing a json document into a string is only
valid if the json is a string, otherwise we need to call `.dump()`
2020-12-16 10:54:36 +01:00
regnat
6e899278d3 Better detect when buildPaths would be a no-op
`buildPaths` can be called even for stores where it's not defined in case it's
bound to be a no-op.
The “no-op detection” mechanism was only detecting the case wher `buildPaths`
was called on a set of (non-drv) paths that were already present on the store.

This commit extends this mechanism to also detect the case where `buildPaths`
is called on a set of derivation outputs which are already built on the store.

This only works with the ca-derivations flag. It could be possible to
extend this to also work without it, but it would add quite a bit of
complexity, and it's not used without it anyways.
2020-12-16 10:36:16 +01:00
regnat
7080321618 Use the fs accessor for readInvalidDerivation
Extend `FSAccessor::readFile` to allow not checking that the path is a
valid one, and rewrite `readInvalidDerivation` using this extended
`readFile`.

Several places in the code use `readInvalidDerivation`, either because
they need to read a derivation that has been written in the store but
not registered yet, or more generally to prevent a deadlock because
`readDerivation` tries to lock the state, so can't be called from a
place where the lock is already held.
However, `readInvalidDerivation` implicitely assumes that the store is a
`LocalFSStore`, which isn't always the case.

The concrete motivation for this is that it's required for `nix copy
--from someBinaryCache` to work, which is tremendously useful for the
tests.
2020-12-15 20:10:46 +01:00
regnat
44c3fbc6e0 Fix addTextToStore for binary caches
Because of a too eager refactoring, `addTextToStore` used to throw an
error because the input wasn't a valid nar.

Partially revert that refactoring to wrap the text into a proper nar
(using `dumpString`) to make this method work again
2020-12-15 09:38:19 +01:00
Eelco Dolstra
f2f60bf5d6
Merge pull request #4330 from NixOS/ca/properly-store-outputs
Properly store the outputs of CA derivations − take 2
2020-12-14 15:01:23 +01:00
Eelco Dolstra
92438c70d2
Merge pull request #4352 from jonringer/allow-private-caches
treat s3 permission errors as file-not-found
2020-12-13 13:39:20 +01:00
regnat
e9b39f6004 Restrict the operations on drv outputs in recursive Nix
There's currently no way to properly filter them, so disallow them
altogether instead.
2020-12-11 21:17:25 +01:00
regnat
bab1cda0e6 Use the hash modulo in the derivation outputs
Rather than storing the derivation outputs as `drvPath!outputName` internally,
store them as `drvHashModulo!outputName` (or `outputHash!outputName` for
fixed-output derivations).

This makes the storage slightly more opaque, but enables an earlier
cutoff in cases where a fixed-output dependency changes (but keeps the
same output hash) − same as what we already do for input-addressed
derivations.
2020-12-11 21:17:23 +01:00
regnat
8914e01e37 Store the realisations as JSON in the binary cache
Fix #4332
2020-12-11 21:05:09 +01:00
regnat
3ac9d74eb1 Rework the db schema for derivation outputs
Add a new table for tracking the derivation output mappings.

We used to hijack the `DerivationOutputs` table for that, but (despite its
name), it isn't a really good fit:

- Its entries depend on the drv being a valid path, making it play badly with
  garbage collection and preventing us to copy a drv output without copying
  the whole drv closure too;
- It dosen't guaranty that the output path exists;

By using a different table, we can experiment with a different schema better
suited for tracking the output mappings of CA derivations.
(incidentally, this also fixes #4138)
2020-12-11 20:41:32 +01:00
regnat
58cdab64ac Store metadata about drv outputs realisations
For each known realisation, store:
- its output
- its output path

This comes with a set of needed changes:

- New `realisations` module declaring the types needed for describing
  these mappings
- New `Store::registerDrvOutput` method registering all the needed informations
  about a derivation output (also replaces `LocalStore::linkDeriverToPath`)
- new `Store::queryRealisation` method to retrieve the informations for a
  derivations

This introcudes some redundancy on the remote-store side between
`wopQueryDerivationOutputMap` and `wopQueryRealisation`.
However we might need to keep both (regardless of backwards compat)
because we sometimes need to get some infos for all the outputs of a
derivation (where `wopQueryDerivationOutputMap` is handy), but all the
stores can't implement it − because listing all the outputs of a
derivation isn't really possible for binary caches where the server
doesn't allow to list a directory.
2020-12-11 20:41:32 +01:00
Michael Bishop
63b3536f50
treat s3 permission errors as file-not-found
Signed-off-by: Jonathan Ringer <jonringer117@gmail.com>
2020-12-11 09:49:24 -08:00
Eelco Dolstra
253571e4ec
Merge pull request #4342 from tweag/fix-remote-build-hook
fix remote build hook
2020-12-09 12:40:00 +01:00
Maximilian Bosch
93a8a005de
libstore/openStore: fix stores with IPv6 addresses
In `nixStable` (2.3.7 to be precise) it's possible to connect to stores
using an IPv6 address:

  nix ping-store --store ssh://root@2001:db8::1

This is also useful for `nixops(1)` where you could specify an IPv6
address in `deployment.targetHost`.

However, this behavior is broken on `nixUnstable` and fails with the
following error:

  $ nix store ping --store ssh://root@2001:db8::1
  don't know how to open Nix store 'ssh://root@2001:db8::1'

This happened because `openStore` from `libstore` uses the `parseURL`
function from `libfetchers` which expects a valid URL as defined in
RFC2732. However, this is unsupported by `ssh(1)`:

  $ nix store ping --store 'ssh://root@[2001:db8::1]'
  cannot connect to 'root@[2001:db8::1]'

This patch now allows both ways of specifying a store (`root@2001:db8::1`) and
also `root@[2001:db8::1]` since the latter one is useful to pass query
parameters to the remote store.

In order to achieve this, the following changes were made:

* The URL regex from `url-parts.hh` now allows an IPv6 address in the
  form `2001:db8::1` and also `[2001:db8::1]`.

* In `libstore`, a new function named `extractConnStr` ensures that a
  proper URL is passed to e.g. `ssh(1)`:

  * If a URL looks like either `[2001:db8::1]` or `root@[2001:db8::1]`,
    the brackets will be removed using a regex. No additional validation
    is done here as only strings parsed by `parseURL` are expected.

  * In any other case, the string will be left untouched.

* The rules above only apply for `LegacySSHStore` and `SSHStore` (a.k.a
  `ssh://` and `ssh-ng://`).

Unresolved questions:

* I'm not really sure whether we want to allow both variants of IPv6
  addresses in the URL parser. However it should be noted that both seem
  to be possible according to RFC2732:

  > This document incudes an update to the generic syntax for Uniform
  > Resource Identifiers defined in RFC 2396 [URL].  It defines a syntax
  > for IPv6 addresses and allows the use of "[" and "]" within a URI
  > explicitly for this reserved purpose.

* Currently, it's not supported to specify a port number behind the
  hostname, however it seems as this is not really supported by the URL
  parser. Hence, this is probably out of scope here.
2020-12-09 12:23:29 +01:00
regnat
c87267c2a4 Store the final drv outputs in memory when building remotely
The `DerivationGoal` has a variable storing the “final” derivation
output paths that is used (amongst other things) to fill the environment
for the post build hook. However this variable wasn't set when the
build-hook is used, causing a crash when both hooks are used together.

Fix this by setting this variable (from the informations in the db) after a run
of the post build hook.
2020-12-09 10:45:12 +01:00
regnat
6758e65612 Revert "Re-query for the derivation outputs in the post-build-hook"
This reverts commit 1b1e076033.

Using `queryPartialDerivationOutputMap` assumes that the derivation
exists locally which isn't the case for remote builders.
2020-12-09 09:44:07 +01:00
Eelco Dolstra
82e5511594
Merge pull request #4325 from tweag/hide-local-store-sql-statements
Hide the sqlite statements declarations for the local store
2020-12-08 14:04:16 +01:00
regnat
c0f21f08f8 Hide the sqlite statements declarations for the local store
These have no need to be in the public interface and it causes spurious
rebuilds each time one wants to add or remove a new statement.
2020-12-08 13:29:13 +01:00
regnat
1b1e076033 Re-query for the derivation outputs in the post-build-hook
We can't assume that the runtime state knows about them as they might have
been built remotely, in which case we must query the db again to get
them.
2020-12-08 11:11:02 +01:00
Matthew Bauer
aa07502009 Always default to cache.nixos.org even when different nix store dir
Since 0744f7f, it is now useful to have cache.nixos.org in substituers
even if /nix/store is not the Nix Store Dir. This can always be
overridden via configuration, though.
2020-12-06 23:04:42 -06:00
Eelco Dolstra
1d1a85eb0a
Merge pull request #4318 from matthewbauer/add-slash-to-trusted-binary-cache
Canonicalize binary caches with ‘/’ when one is missing
2020-12-05 10:17:17 +01:00
Matthew Bauer
b9a00fd15b Canonicalize binary caches with ‘/’ when one is missing
This checks if there is a trusted substituter with a slash, so
trusting https://cache.nixos.org also implies https://cache.nixos.org/
is trusted.
2020-12-04 22:17:19 -06:00
Matthew Bauer
3c9b7029ba Use com.apple.oahd.plist for rosetta 2 detection 2020-12-04 13:26:53 -06:00
Stéphan Kochen
e20a3ec756 Fix compatibility with newer AWS SDKs
Tested against AWS SDK 1.8.99. Fixes #3201.
2020-12-04 19:36:09 +01:00
Matthew Bauer
4b9acf4e21 Use posix_spawn_setbinpref_np to advise which architecture to run
When running universal binaries like /bin/bash, Darwin XNU will choose
which architecture of the binary to use based on "binary preferences".
This change sets that to the current platform for aarch64 and x86_64
builds. In addition it now uses posix_spawn instead of the usual
execve. Note, that this does not prevent the other architecture from
being run, just advises which to use.

Unfortunately, posix_spawnattr_setbinpref_np does not appear to be
inherited by child processes in x86_64 Rosetta 2 translations, meaning
that this will not always work as expected.

For example:

  {
    arm = derivation {
      name = "test";
      system = "aarch64-darwin";
      builder = "/bin/bash";
      args = [ "-e" (builtins.toFile "test" ''
        set -x
        /usr/sbin/sysctl sysctl.proc_translated
        /usr/sbin/sysctl sysctl.proc_native
        [ "$(/usr/bin/arch)" = arm64 ]
        /usr/bin/touch $out
      '') ];
    };
    rosetta = derivation {
      name = "test";
      system = "x86_64-darwin";
      builder = "/bin/bash";
      args = [ "-e" (builtins.toFile "test" ''
        set -x
        /usr/sbin/sysctl sysctl.proc_translated
        /usr/sbin/sysctl sysctl.proc_native
        [ "$(/usr/bin/arch)" = i386 ]
        echo It works!
        /usr/bin/touch $out
      '') ];
    };
  }

`arm' fails on x86_64-compiled Nix, but `arm' and `rosetta' succeed on
aarch64-compiled Nix. I suspect there is a way to fix this since:

  $ /usr/bin/arch -arch x86_64 /bin/bash \
    -c '/usr/bin/arch -arch arm64e /bin/bash -c /usr/bin/arch'
  arm64

seems to work correctly. We may need to wait for Apple to update
system_cmds in opensource.apple.com to find out how though.
2020-12-03 15:41:59 -06:00
Matthew Bauer
9b1824ecbd Add extraPlatforms for Rosetta 2 macOS
macOS systems with ARM64 can utilize a translation layer at
/Library/Apple/usr/libexec/oah to run x86_64 binaries. This change
makes Nix recognize that and it to "extra-platforms". Note that there
are two cases here since Nix could be built for either x86_64 or
aarch64. In either case, we can switch to the other architecture.
Unfortunately there is not a good way to prevent aarch64 binaries from
being run in x86_64 contexts or vice versa - programs can always
execute programs for the other architecture.
2020-12-03 15:41:43 -06:00
regnat
0afab668fa Don't fail early when -j0 is passed
If the build closure contains some CA derivations, then we can't know
ahead-of-time that we won't build anything as early-cutoff might come-in
at a laster stage
2020-12-03 13:24:36 +01:00
Greg Hale
d8fc1bb7b0 fix tokens documentation 2020-12-02 10:15:18 -05:00
Eelco Dolstra
1b79b5b983 read(): Use char * instead of unsigned char *
This gets rid of some pointless casts.
2020-12-02 14:17:27 +01:00
Eelco Dolstra
faa31f4084 Sink: Use std::string_view 2020-12-02 14:17:27 +01:00
Eelco Dolstra
e5cf501c77
Merge pull request #4284 from tweag/fixed-output-depending-on-ca
Allow fixed-output derivations to depend on (floating) content-addressed ones
2020-12-01 20:25:41 +01:00
regnat
438977731c shut up clang warnings
- Fix some class/struct discrepancies
- Explicit the overloading of `run` in the `Cmd*` classes
- Ignore a warning in the generated lexer
2020-12-01 15:04:03 +01:00
Eelco Dolstra
3b7e00ce22 Move primeCache() to Worker::run()
We need the missing path info to communicate the worker's remaining
goals to the progress bar.
2020-12-01 13:44:48 +01:00
regnat
9bd8184f1f Allow fixed-output derivations to depend on (floating) content-addressed ones
Fix an overlook of https://github.com/NixOS/nix/pull/4056
2020-11-27 15:39:24 +01:00
regnat
13c557fe82 fix the hash rewriting for ca-derivations 2020-11-25 11:33:00 +01:00
John Ericson
2113ae2d85 Make drv hash modulo memo table thread-safe
Let's get one step closer to the daemon not needing to fork.
2020-11-19 16:50:06 +00:00
Eelco Dolstra
df5c69a94e
Merge pull request #4180 from Ma27/ssh-ng-substitute
Allow substituting paths when building remotely using `ssh-ng://`
2020-11-17 14:01:04 +01:00
Eelco Dolstra
e6b7c7b79c Cleanup 2020-11-17 13:58:55 +01:00
Eelco Dolstra
bccff827dc Fix deadlock in IFD through the daemon
Fixes #4235.
2020-11-17 13:50:36 +01:00
regnat
7de21f6664 Make the sql debug statements more useful
Print the expanded sql query (with the variables bound to their value) rather
than the original one in case of error
2020-11-17 10:05:45 +01:00
Jake Waksbaum
01db455733 Fix deadlock in nix-store when max-connections=1
This fixes a bug I encountered where `nix-store -qR` will deadlock when
the `--include-outputs` flag is passed and `max-connections=1`.

The deadlock occurs because `RemoteStore::queryDerivationOutputs` takes
the only connection from the connection pool and uses it to check the
daemon version. If the version is new enough, it calls
`Store::queryDerivationOutputs`, which eventually calls
`RemoteStore::queryPartialDerivationOutputMap`, where we take another
connection from the connection pool to check the version again. Because
we still haven't released the connection from the caller, this waits for
a connection to be available, causing a deadlock.

This diff solves the issue by using `getProtocol` to check the protocol
version in the caller `RemoteStore::queryDerivationOutputs`, which
immediately frees the connection back to the pool before returning the
protocol version. That way we've already freed the connection by the
time we call `RemoteStore::queryPartialDerivationOutputMap`.
2020-11-16 02:35:50 -05:00
Maximilian Bosch
3a63fc6cd5
Allow substituting paths when building remotely using ssh-ng://
Until now, it was not possible to substitute missing paths from e.g.
`https://cache.nixos.org` on a remote server when building on it using
the new `ssh-ng` protocol.

This is because every store implementation except legacy `ssh://`
ignores the substitution flag passed to `Store::queryValidPaths` while
the `legacy-ssh-store` substitutes the remote store using
`cmdQueryValidPaths` when the remote store is opened with `nix-store
--serve`.

This patch slightly modifies the daemon protocol to allow passing an
integer value suggesting whether to substitute missing paths during
`wopQueryValidPaths`. To implement this on the daemon-side, the
substitution logic from `nix-store --serve` has been moved into a
protected method named `Store::substitutePaths` which gets currently
called from `LocalStore::queryValidPaths` and `Store::queryValidPaths`
if `maybeSubstitute` is `true`.

Fixes #2770
2020-11-05 20:12:37 +01:00
Eelco Dolstra
e8c379555f LocalStore: Get rid of recursive_mutex 2020-11-03 14:45:24 +01:00
Eelco Dolstra
797a52e31d Add FIXME 2020-11-03 14:45:24 +01:00
Eelco Dolstra
7cf874c17d Don't use readDerivation() in addValidPath()
readDerivation() requires a valid path.

Fixes #4210.
2020-11-02 18:46:44 +01:00
Eelco Dolstra
ab2ef851b6
Merge pull request #4207 from hercules-ci/fix-RemoteStore-filterSource-deadlock
Fix RemoteStore pool deadlock in filterSource etc
2020-11-02 14:46:10 +01:00
Eelco Dolstra
db5424bf09 Don't send eval-related settings to the daemon 2020-11-02 13:57:58 +01:00
Robert Hensing
e8a45d07bc Restore RestrictedStore.addToStoreFromDump implementation
It was accidentally removed in commit ca30abb3fb
2020-10-31 23:56:03 +01:00
Robert Hensing
2192cac634 Fix RemoteStore pool deadlock in filterSource etc 2020-10-30 21:47:34 +01:00
Eelco Dolstra
ff4dea63c9 Generalize extra-* settings
This removes the extra-substituters and extra-sandbox-paths settings
and instead makes every array setting extensible by setting
"extra-<name> = <value>" in the configuration file or passing
"--<name> <value>" on the command line.
2020-10-29 18:17:39 +01:00
stev
869c0321ff Alter "wanted:" to "specified:" in hash mismatch output
This makes it even clearer which of the two hashes was specified in the
nix files. Some may think that "wanted" and "got" is obvious, but:
"got" could mean "got in nix file" and "wanted" could mean "want to see in nix file".
2020-10-29 00:33:14 +01:00
Eelco Dolstra
02a1facbdc
Merge pull request #4056 from tweag/non-ca-depending-on-ca
Allow non-CA derivations to depend on CA ones
2020-10-27 17:38:29 +01:00
regnat
bc081bcd81 Inline unkownHashes
See https://github.com/NixOS/nix/pull/4056#discussion_r493661632
2020-10-27 07:29:25 +01:00
regnat
c092fa4702 Allow non-CA derivations to depend on CA derivations 2020-10-27 07:29:23 +01:00
Christian Kampka
461cf2b856
Add NIX_CONFIG env var for applying nix.conf overrides 2020-10-21 13:41:26 +02:00
Eelco Dolstra
fda835b231
Merge pull request #4143 from obsidiansystems/typed-goal-maps
Properly type the derivation and substitution goal maps
2020-10-18 18:12:21 +02:00
Robert Hensing
bd9eb5c743 DerivationGoal: only retry if output closure incomplete is only problem 2020-10-18 14:26:37 +02:00
Robert Hensing
94f1e4a441 Typo 2020-10-18 14:26:37 +02:00
John Ericson
7ed46c1574 Explain that upcast_goal is still a static cast 2020-10-17 21:50:12 +00:00
John Ericson
57d0432b39 Just use auto in two places. 2020-10-17 21:47:52 +00:00
John Ericson
e6f8ae56d8 tab -> space 2020-10-17 21:45:31 +00:00
John Ericson
d3cfc14e3a Merge remote-tracking branch 'upstream/master' into auto-uid-allocation 2020-10-17 19:31:13 +00:00
John Ericson
801e6d96d8 Merge commit 'd334fd48824b41b57e267cd2926fa9619b7718e3' into auto-uid-allocation 2020-10-17 19:31:06 +00:00
John Ericson
2c9a8e7421 Merge commit '97ffc1e0139e124b7e36b5d1a62b90300f231118' into auto-uid-allocation 2020-10-17 19:20:18 +00:00
John Ericson
81c5f754a7 Merge commit '6aa64627c8e431c3b187f7bb44c943d06e39b929' into auto-uid-allocation 2020-10-17 19:12:59 +00:00
John Ericson
2546c63373 Merge commit 'f66bbd8c7bb1472facf8917e58e3cd4f6ddfa1b5' into auto-uid-allocation 2020-10-17 17:25:17 +00:00
John Ericson
10e81bf871 Fix conditions for ca-references 2020-10-15 02:21:28 +00:00
John Ericson
0fefc2a439 Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-14 20:49:01 +00:00
Eelco Dolstra
11882d7c7c Create /etc/passwd *after* figuring out the sandbox uid/gid
Fixes build failures like

  # nix log /nix/store/gjaa0psfcmqvw7ivggsncx9w364p3s8s-sshd.conf-validated.drv
  No user exists for uid 30012
2020-10-14 12:20:58 +02:00
John Ericson
55592b253f Add some more docs 2020-10-13 18:04:24 +00:00
John Ericson
13804f126e Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-13 18:02:32 +00:00
John Ericson
34b3bfac25 Merge remote-tracking branch 'upstream/master' into path-info 2020-10-13 15:36:20 +00:00
Eelco Dolstra
2653801939 Merge branch 'split_build_cc' of https://github.com/obsidiansystems/nix 2020-10-13 15:36:55 +02:00
John Ericson
39c11c5c01 Organize content-address.hh a bit better 2020-10-13 03:43:13 +00:00
John Ericson
a0f369aa3f Merge remote-tracking branch 'upstream/master' into path-info 2020-10-12 20:48:35 +00:00
John Ericson
1b8ebe92dc Merge remote-tracking branch 'obsidian/split_build_cc' into typed-goal-maps 2020-10-12 20:47:22 +00:00
John Ericson
a73959e6be Merge remote-tracking branch 'upstream/master' into split_build_cc 2020-10-12 17:20:46 +00:00
John Ericson
d334fd4882 Merge branches 'split_build_hh-0', 'split_build_hh-1', 'split_build_hh-2', 'split_build_hh-3' and 'split_build_hh-4' into split_build_cc 2020-10-12 17:20:12 +00:00
John Ericson
542972f029 Trim worker.hh 2020-10-12 17:16:48 +00:00
John Ericson
0e2306204a Rename to hand-hold git (worker.hh) 2020-10-12 17:16:48 +00:00
John Ericson
3ffa3546bd Trim substitution-goal.hh 2020-10-12 17:16:25 +00:00
John Ericson
e77a2344d5 Rename to hand-hold git (substitution-goal.hh) 2020-10-12 17:16:25 +00:00
John Ericson
d585b4c54f Trim hook-instance.hh 2020-10-12 17:16:13 +00:00
John Ericson
10b749a156 Rename to hand-hold git (hook-instance.hh) 2020-10-12 17:16:13 +00:00
John Ericson
8067d32f2a Trim goal.hh 2020-10-12 17:16:00 +00:00
John Ericson
0d0e345cdc Rename to hand-hold git (goal.hh) 2020-10-12 17:16:00 +00:00
John Ericson
2ce726947a Trim derivation-goal.hh 2020-10-12 17:15:32 +00:00
John Ericson
4eb8c69853 Rename to hand-hold git (derivation-goal.hh) 2020-10-12 17:15:32 +00:00
John Ericson
f7099965bf Change .cc files to use split build headers 2020-10-12 17:08:52 +00:00
John Ericson
5a97621d6d Prepare for build/*.hh headers 2020-10-12 17:07:51 +00:00
volth
eee18f88dd
Handle amount of disk space saved by hard linking being negative
Fixes bogus messages like "currently hard linking saves 17592186044416.00 MiB".
2020-10-12 16:06:38 +00:00
John Ericson
15fdb7cc6b Split out uds-remote-store.{cc.hh} 2020-10-11 17:37:05 +00:00
John Ericson
38e3897162 Copy {uds-,}remote-store.{cc,hh}
This prepares for the splitting that happens in the next commit.
2020-10-11 17:18:26 +00:00
John Ericson
5c74a6147b Properly type the derivation and substitution goal maps
As a bonus, Worker::removeGoal is less inefficient.
2020-10-11 17:07:14 +00:00
John Ericson
8cc510fb79 Merge branches 'build-1', 'build-2', 'build-3', 'build-4', 'build-5', 'build-6', 'build-7' and 'build-8' into split_build_cc 2020-10-11 16:44:34 +00:00
John Ericson
bcb67e1ed8 Trim lock.cc 2020-10-11 16:44:19 +00:00
John Ericson
e0be04129b Rename to hand-hold git (lock.cc) 2020-10-11 16:44:14 +00:00
John Ericson
dbc588651c Trim lock.hh 2020-10-11 16:44:07 +00:00
John Ericson
d0004bfcab Rename to hand-hold git (lock.hh) 2020-10-11 16:43:59 +00:00
John Ericson
eed53ed87a Trim build/worker.cc 2020-10-11 16:43:51 +00:00
John Ericson
904e315dae Rename to hand-hold git (build/worker.cc) 2020-10-11 16:43:42 +00:00
John Ericson
d24ffe0eb1 Trim build/substitution-goal.cc 2020-10-11 16:43:24 +00:00
John Ericson
3633b3572b Rename to hand-hold git (build/substitution-goal.cc) 2020-10-11 16:43:18 +00:00
John Ericson
4bdff7d1b0 Trim build/local-store-build.cc 2020-10-11 16:43:12 +00:00
John Ericson
dc5225cde5 Rename to hand-hold git (build/local-store-build.cc) 2020-10-11 16:43:05 +00:00
John Ericson
159054f730 Trim build/hook-instance.cc 2020-10-11 16:42:35 +00:00
John Ericson
f0b8987299 Rename to hand-hold git (build/hook-instance.cc) 2020-10-11 16:42:08 +00:00
John Ericson
819fe848ac Trim build/goal.cc 2020-10-11 16:41:58 +00:00
John Ericson
184bfc301e Rename to hand-hold git (build/goal.cc) 2020-10-11 16:41:18 +00:00
John Ericson
3bab1c5bb0 Trim build/derivation-goal.cc 2020-10-11 16:41:11 +00:00
John Ericson
9629290eda Rename to hand-hold git (build/derivation-goal.cc) 2020-10-11 16:40:52 +00:00
John Ericson
a4f0fecb03 Trim build.hh 2020-10-11 16:40:34 +00:00
John Ericson
fc72cb0760 Rename to hand-hold git (build.hh) 2020-10-11 16:40:14 +00:00
John Ericson
428536fd75 Prepare for build/* files 2020-10-11 16:39:08 +00:00
John Ericson
aef44cbaa9 Split out commonChildInit 2020-10-11 16:38:46 +00:00
John Ericson
6cc1541782 Split out local-fs-store.hh
This matches the already-existing `local-fs-store.cc`.
2020-10-09 20:18:08 +00:00
Eelco Dolstra
636ec17139
Remove stray DerivationOutputsAndPaths type 2020-10-09 15:41:24 +02:00
Eelco Dolstra
97ffc1e013 Dynamically disable user namespaces if CLONE_NEWUSER fails
This makes builds work inside nixos-enter.

Fixes #3145.
2020-10-07 22:46:01 +02:00
Eelco Dolstra
6aa64627c8 Support user namespaces being disabled
If max_user_namespaces is set to 0, then don't run the build in a user
namespace.

Fixes #4092.
2020-10-07 22:02:36 +02:00
Eelco Dolstra
f66bbd8c7b Doh 2020-10-07 21:25:06 +02:00
Eelco Dolstra
e705c24294 Tweak error messages 2020-10-07 17:28:43 +02:00
Eelco Dolstra
c43e882f54 Serialize exceptions from the daemon to the client 2020-10-07 17:13:54 +02:00
John Ericson
f8d562c0a7 Use PathReferences more widely 2020-10-07 15:00:10 +00:00
Eelco Dolstra
be149acfda Serialize exceptions from the sandbox process to the parent
Fixes #4118.
2020-10-07 16:34:03 +02:00
John Ericson
57d960dcd1 Remove generic std::optional<T> suppport from worker proto
See comment for rational; I think it's good to leave a comment lest
anyone is tempted to add such a sum-type instance again.

Fixes #4113
2020-10-07 12:50:37 +00:00
Eelco Dolstra
d761485010 Prevent a deadlock when user namespace setup fails
Observed on Centos 7 when user namespaces are disabled:
DerivationGoal::startBuilder() throws an exception, ~DerivationGoal()
waits for the child process to exit, but the child process hangs
forever in drainFD(userNamespaceSync.readSide.get()) in
DerivationGoal::runChild(). Not sure why the SIGKILL doesn't get
through.

Issue #4092.
2020-10-06 18:57:23 +02:00
Eelco Dolstra
ad143c5b3b Shut up some clang warnings 2020-10-06 14:52:30 +02:00
Eelco Dolstra
85c8be6286 Remove static variable name clashes
This was useful for an experiment with building Nix as a single
compilation unit. It's not very useful otherwise but also doesn't
hurt...
2020-10-06 13:49:20 +02:00
Eelco Dolstra
6691256e79 Factor out common showBytes() 2020-10-06 10:40:49 +02:00
Eelco Dolstra
d0bb544128 Add missing #pragma once 2020-10-06 10:40:07 +02:00
John Ericson
dae4409071 Merge remote-tracking branch 'upstream/master' into path-info 2020-10-06 04:20:44 +00:00
Eelco Dolstra
88a667e49e
Fix s3:// store
Fixes https://github.com/NixOS/nixos-org-configurations/issues/123.
2020-10-05 17:53:30 +02:00
Eelco Dolstra
51c299213b
Merge pull request #3935 from obsidiansystems/binary-cache-addToStoreFromDump
Get rid of Hash::dummy from BinaryCacheStore
2020-10-05 14:41:28 +02:00
Eelco Dolstra
f3aba88737
Merge pull request #3895 from obsidiansystems/templated-daemon-protocol
More templated STL support for the daemon protocol
2020-10-05 14:40:27 +02:00
Eelco Dolstra
20a1e20d91 Style 2020-09-30 13:35:56 +02:00
John Ericson
69afaeace3 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-30 00:42:28 +00:00
John Ericson
b759701652 nix::worker_proto -> worker_proto 2020-09-30 00:41:18 +00:00
John Ericson
45a0ed82f0 Revert "Use template structs instead of phantoms"
This reverts commit 9ab07e99f5.
2020-09-30 00:39:06 +00:00
Kevin Quick
5e7838512e
Remove github-access-token in favor of access-token. 2020-09-29 16:26:34 -07:00
Kevin Quick
66c3959e8c
Merge branch 'master' into access-tokens 2020-09-29 08:32:06 -07:00
Eelco Dolstra
5999978a05 Make Headers an optional argument 2020-09-29 13:05:19 +02:00
Eelco Dolstra
cebd2fc35d Merge branch 'github-api-token' of https://github.com/imalsogreg/nix 2020-09-29 12:17:00 +02:00
John Ericson
00135e13f4 Clarify comment a bit 2020-09-28 18:19:10 +00:00
John Ericson
80e335bb58 Use drvPath2 and give it a better name 2020-09-28 15:43:56 +00:00
John Ericson
10202bbf29 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-28 15:39:11 +00:00
John Ericson
6c31297d80
Update src/libstore/binary-cache-store.cc
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-09-28 11:32:58 -04:00
Eelco Dolstra
649c465873
Merge pull request #4064 from serokell/balsoft/fix-max-jobs
Fix max-jobs option
2020-09-28 10:42:02 +02:00
John Ericson
25fffdda86
Remove redundant nar hash and size setting
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2020-09-26 10:17:30 -04:00
John Ericson
1832436526 Fix up BinaryCacheStore::addToStore taking a path 2020-09-26 04:56:29 +00:00
John Ericson
5db83dd771 BinaryCacheStore::addTextToStore include CA field 2020-09-26 03:21:36 +00:00
John Ericson
c40c832f19 Merge remote-tracking branch 'obsidian/master' into binary-cache-addToStoreFromDump 2020-09-26 00:21:45 +00:00
Gregory Hale
faa5607f54 Merge remote-tracking branch 'origin/master' into github-api-token 2020-09-25 12:10:58 -04:00
Eelco Dolstra
7d81582488
Merge pull request #3953 from obsidiansystems/basic-derivation-goal-outputs
Deduplicate basic derivation goals too
2020-09-25 17:21:19 +02:00
Kevin Quick
5a35cc29bf
Re-add support for github-access-token, but mark as deprecated. 2020-09-25 08:09:56 -07:00
Kevin Quick
ef2a14be19
Fix reference to older name for access-tokens config value. 2020-09-25 08:08:27 -07:00
Kevin Quick
8fba2a8b54
Update to use access-tokens configuration for github/gitlab access.
This change provides support for using access tokens with other
instances of GitHub and GitLab beyond just github.com and
gitlab.com (especially company-specific or foundation-specific
instances).

This change also provides the ability to specify the type of access
token being used, where different types may have different handling,
based on the forge type.
2020-09-24 22:49:44 -07:00
Alexander Bantyev
ed218e1d6c
Fix max-jobs option
After 0ed946aa61, max-jobs setting (-j/--max-jobs)
stopped working.

The reason was that nrLocalBuilds (which compared to maxBuildJobs to figure
out whether the limit is reached or not) is not incremented yet when tryBuild
is started; So, the solution is to move the check to tryLocalBuild.

Closes https://github.com/nixos/nix/issues/3763
2020-09-25 00:07:42 +03:00
John Ericson
45ca7c3e4b Merge remote-tracking branch 'upstream/master' into path-info 2020-09-23 22:35:41 +00:00
Eelco Dolstra
4ce8a3ed45 Hopefully fix EPERM on macOS 2020-09-23 21:29:10 +02:00
Eelco Dolstra
9a24ece122 Fix exception 2020-09-23 20:21:08 +02:00
Eelco Dolstra
236d9ee7f7 lstat() cleanup 2020-09-23 19:17:28 +02:00
Eelco Dolstra
688bd4fb50 After rewriting a path, make it read-only 2020-09-23 19:10:16 +02:00
Eelco Dolstra
cec9473871 DerivationGoal::registerOutputs(): Don't canonicalize twice
Fixes #4021.
2020-09-23 18:21:22 +02:00
Eelco Dolstra
d4f8163d10 canonicalisePathMetaData_(): Change assertion to error message 2020-09-23 18:21:22 +02:00
Eelco Dolstra
e8f0b1e996 DerivationGoal::registerOutputs(): Fix bad format string 2020-09-23 18:21:22 +02:00
John Ericson
3f226f71c1 Return more info from BinaryCacheStore::addToStoreCommon
We don't need it yet, but we could/should in the future, and it's a
cost-free change since we already have the reference. I like it.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2020-09-23 14:40:41 +00:00
John Ericson
412b3a54fb
Clarify FIXME in BinaryCacheStore::addToStoreCommon
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2020-09-23 10:36:55 -04:00
John Ericson
9fbc31a65b Get rid of Hash::dummy from BinaryCacheStore 2020-09-23 04:56:04 +00:00
John Ericson
993229cdaf Deduplicate basic derivation goals too
See comments for security concerns.

Also optimize goal creation by not traversing map twice.
2020-09-22 17:13:59 +00:00
John Ericson
e9fc2031f0 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-22 14:18:31 +00:00
regnat
67cc356bb3 Merge remote-tracking branch 'origin/master' into non-ca-depending-on-ca 2020-09-22 16:11:25 +02:00
Eelco Dolstra
980edd1f3a RemoteStore::addCAToStore(): Don't hold connection while calling queryPathInfo()
This leads to a deadlock if we're at the connection limit.
2020-09-22 15:28:20 +02:00
Eelco Dolstra
5b107f2c5f
Merge pull request #4038 from maljub01/master
Add a nix.conf option for allowing a symlinked store
2020-09-22 13:19:22 +02:00
Eelco Dolstra
35a0ac1838 Style fixes 2020-09-22 11:40:19 +02:00
Eelco Dolstra
92ac8df0ec Merge branch 'add-ca-to-store' of https://github.com/hercules-ci/nix 2020-09-22 11:31:33 +02:00
John Ericson
3786a801c3 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-22 04:15:55 +00:00
John Ericson
b92d3b2edd Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-22 00:45:55 +00:00
Cole Helbling
ba37299a03
Serialize SandboxMode enum to string for JSON
Rather than showing an integer as the default, instead show the boolean
referenced in the description.

The nix.conf.5 manpage used to show "default: 0", which is unnecessarily
opaque and confusing (doesn't 0 mean false, even though the default is
true?); now it properly shows that the default is true.
2020-09-21 10:36:45 -07:00
Eelco Dolstra
0716adaa8b abstractsettingtojson.hh -> abstract-setting-to-json.hh 2020-09-21 18:49:43 +02:00
Eelco Dolstra
340ca382c4 Don't include nlohmann/json.hpp in globals.hh
This reduces compilation time by 207s.

Issue #4045.
2020-09-21 18:47:18 +02:00
Eelco Dolstra
d51ba43047 Move Callback into its own header
This gets rid of the inclusion of <future> in util.hh, cutting
compilation time by ~20s (CPU time).

Issue #4045.
2020-09-21 18:42:21 +02:00
Marwan Aljubeh
f80ffeb8c9 Update the variable name accordingly 2020-09-21 17:29:08 +01:00
Eelco Dolstra
e8e1d420f3 Don't include <regex> in header files
This reduces compilation time by ~15 seconds (CPU time).

Issue #4045.
2020-09-21 18:22:45 +02:00
Marwan Aljubeh
4e1a04733d Use a better name for the config option 2020-09-21 16:32:22 +01:00
Robert Hensing
ca30abb3fb Document addCAToStore/addToStoreFromDump source drainage
Also checked that all usages satisfy the requirement and
removed dead code.
2020-09-21 07:55:47 +02:00
Robert Hensing
fa08db5c4c wopAddToStore: return ValidPathInfo
A ValidPathInfo is created anyway. By returning it we can save a
roundtrip and we have a nicer interface.
2020-09-21 07:55:47 +02:00
Robert Hensing
7c68264085 wopAddToStore: add RepairFlag 2020-09-21 07:55:47 +02:00
Robert Hensing
fbf509c113 parseContentAddressMethodPrefix: use string_view
Co-authored-by: John Ericson <git@JohnEricson.me>
2020-09-21 07:55:47 +02:00
Robert Hensing
8279178b07 Move FramedSink next to FramedSource 2020-09-21 07:55:47 +02:00
Robert Hensing
ecc8088cb7 wopAddToStore: Throw to clarify unused refs
Co-authored-by: John Ericson <git@JohnEricson.me>
2020-09-21 07:55:47 +02:00
Robert Hensing
c602ebfb34 Refactor wopAddToStore to make wopAddTextToStore obsolete 2020-09-21 07:55:45 +02:00
Robert Hensing
e34fe47d0c Overhaul wopAddToStore 2020-09-21 07:54:05 +02:00
Marwan Aljubeh
e40772cd35 Lint issue: replacing tabs with spaces 2020-09-18 17:18:45 +01:00
Marwan Aljubeh
c00e078343 Add a nix.conf option for allowing a symlinked store 2020-09-18 17:10:39 +01:00
Robert Hensing
14b30b3f3d Move FramedSource and FramedSink, extract withFramedSink 2020-09-17 20:21:04 +02:00
Robert Hensing
dfa547c6a8 Add ContentAddressMethod and parse/render it 2020-09-17 20:21:04 +02:00
Robert Hensing
9ee3122ec7 Remove redundant import 2020-09-17 20:21:04 +02:00
John Ericson
b7df353f27 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-17 16:33:10 +00:00
Eelco Dolstra
649d3aaf24
Merge pull request #3829 from obsidiansystems/remove-storetype-delegate-regStore
Remove storetype delegate reg store -- contains #3736
2020-09-17 13:55:01 +02:00
Eelco Dolstra
b94a35ef40
Merge pull request #4027 from tweag/fix-gc-of-ca-derivations
Fix garbage collection of CA derivations
2020-09-17 13:46:26 +02:00
regnat
520895b1da Fix garbage collection of CA derivations
Fix #4026
2020-09-17 13:36:58 +02:00
John Ericson
f60b380a7f Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore 2020-09-16 22:35:24 +00:00
John Ericson
2741fffa35 Ensure resolved CA derivations are written
so we can link outputs to deriver and thus properly cache.
2020-09-16 17:57:32 +00:00
John Ericson
7fdbb377ba Start to fix floating CA + remote building 2020-09-16 17:50:48 +00:00
John Ericson
c5ccebae00 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-16 17:50:40 +00:00
Greg Hale
a303c0b6dc Fetch commits from github/gitlab using Auth header
`nix flake info` calls the github 'commits' API, which requires
authorization when the repository is private. Currently this request
fails with a 404.

This commit adds an authorization header when calling the 'commits' API.
It also changes the way that the 'tarball' API authenticates, moving the
user's token from a query parameter into the Authorization header.

The query parameter method is recently deprecated and will be disallowed
in November 2020. Using them today triggers a warning email.
2020-09-16 13:46:48 -04:00
Théophane Hufschmitt
77a0e2c5be
Remove useless exception copy
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-09-16 14:00:21 +02:00
regnat
d72927aa7a Fix the s3 store
Add some necessary casts in the initialisation of the store's config
2020-09-16 13:53:09 +02:00
regnat
e0817cbcdc Don't include nlohmann/json.hpp in config.hh
Instead make a separate header with the template implementation of
`BaseSetting<T>::toJSONObj` that can be included where needed
2020-09-16 13:53:09 +02:00
regnat
fc2d31c423 Add (StoreConfig*) casts to work around a GCC bug
Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80431 that was
already there in the code but was accidentally removed in the last
commits
2020-09-16 13:53:09 +02:00
regnat
888f7afe9f Fix build issues with gcc 2020-09-16 13:53:09 +02:00
regnat
b73adacc1e Add a name to the stores
So that it can be printed by `nix describe-stores`
2020-09-16 13:53:09 +02:00
regnat
f24f0888f9 Document the new store hierarchy 2020-09-16 13:53:09 +02:00
regnat
29a632386e fixup! Make the store plugins more introspectable 2020-09-16 13:53:09 +02:00
regnat
1129913c4e fixup! Correctly call all the parent contructors of the stores 2020-09-16 13:53:09 +02:00
regnat
7f103dcddd Properly filter the stores according to their declared uriSchemes
When opening a store, only try the stores whose `uriSchemes()` include
the current one
2020-09-16 13:53:09 +02:00
regnat
5895184df4 Correctly call all the parent contructors of the stores
Using virtual inheritance means that only the default constructors of
the parent classes will be called, which isn't what we want
2020-09-16 13:53:09 +02:00
regnat
22afa8fb4d Separate store configs from the implems
Rework the `Store` hierarchy so that there's now one hierarchy for the
store configs and one for the implementations (where each implementation
extends the corresponding config). So a class hierarchy like

```
StoreConfig-------->Store
    |                 |
    v                 v
SubStoreConfig----->SubStore
    |                 |
    v                 v
SubSubStoreConfig-->SubSubStore
```

(with virtual inheritance to prevent DDD).

The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-16 13:53:08 +02:00
regnat
aa4eac3788 fixup! Separate the instantiation and initialisation of the stores 2020-09-16 13:53:08 +02:00
regnat
3c525d1590 Complete the toJSON instance for Setting<T>
Don't let it just contain the value, but also the other fields of the
setting (description, aliases, etc..)
2020-09-16 13:53:08 +02:00
regnat
3b57181f8e Separate the instantiation and initialisation of the stores
Add a new `init()` method to the `Store` class that is supposed to
handle all the effectful initialisation needed to set-up the store.
The constructor should remain side-effect free and just initialize the
c++ data structure.

The goal behind that is that we can create “dummy” instances of each
store to query static properties about it (the parameters it accepts for
example)
2020-09-16 13:53:08 +02:00
regnat
fa32560169 Fix the registration of stores 2020-09-16 13:53:08 +02:00
regnat
7d5bdf8b56 Make the store plugins more introspectable
Directly register the store classes rather than a function to build an
instance of them.
This gives the possibility to introspect static members of the class or
choose different ways of instantiating them.
2020-09-16 13:53:08 +02:00
John Ericson
3a5cdd737c Rename Derivation::pathOpt to Derivation::path
We no longer need the `*Opt` to disambiguate.
2020-09-15 15:21:39 +00:00
John Ericson
6387550d58 Get rid of confusing std::optional<bool> for validity 2020-09-15 15:19:45 +00:00
John Ericson
c4bf219b55 Don't link deriver until after any delayed exception is thrown
Otherwise, we will associate fixed-output derivations with outputs that
they did indeed produce, but which had the wrong hash. That's no good.
2020-09-15 14:28:06 +00:00
John Ericson
3ba552b245 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-15 14:17:06 +00:00
John Ericson
c08c9f08c7 Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore 2020-09-15 14:08:35 +00:00
regnat
057c6203b5 gracefully handle old daemon versions
Add a fallback path in `queryPartialDerivationOutputMap` for daemons
that don't support it.

Also upstreams a couple methods from `SSHStore` to `RemoteStore` as this
is needed to handle the fallback path.
2020-09-15 09:58:01 +02:00
Eelco Dolstra
a59e77d9e5 nix-daemon: Lower verbosity of restricted setting warning
Fixes #3992.
2020-09-14 13:48:51 +02:00
aszlig
525b38eee8
Fix unspecified behaviour in readStorePathCAMap
When deploying a Hydra instance with current Nix master, most builds
would not run because of errors like this:

  queue monitor: error: --- Error --- hydra-queue-runner
  error: --- UsageError --- nix-daemon
  not a content address because it is not in the form '<prefix>:<rest>': /nix/store/...-somedrv

The last error message is from parseContentAddress, which expects a
colon-separated string, however what we got here is a store path.

Looking at the worker protocol, the following message sent to the Nix
daemon caused the error above:

  0x1E -> wopQuerySubstitutablePathInfos
  0x01 -> Number of paths
  0x16 -> Length of string
  "/nix/store/...-somedrv"
  0x00 -> Length of string
  ""

Looking at writeStorePathCAMap, the store path is indeed the first field
that's transmitted. However, readStorePathCAMap expects it to be the
*second* field *on my machine*, since expression evaluation order is a
classic form of unspecified behaviour[1] in C++.

This has been introduced in https://github.com/NixOS/nix/pull/3689,
specifically in commit 66a62b3189.

[1]: https://en.wikipedia.org/wiki/Unspecified_behavior#Order_of_evaluation_of_subexpressions

Signed-off-by: aszlig <aszlig@nix.build>
2020-09-13 02:40:51 +02:00
John Ericson
98dfd7531d Fix querying outputs for CA derivations some more
If we resolve using the known path of a derivation whose output we
didn't have, we previously blew up. Now we just fail gracefully,
returning the map of all outputs unknown.
2020-09-04 18:33:58 +00:00
John Ericson
075d399e3f Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-09-04 16:04:35 +00:00
John Ericson
e9fad3006b Fix some of the issues raised by @edolstra
- More and better comments

 - The easier renames
2020-09-04 15:18:25 +00:00
John Ericson
e86dd59dcc
Apply suggestions from code review
Thanks!

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-09-04 10:48:50 -04:00
John Ericson
c9f1ed912c Don't chmod symlink before moving outputs around
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-09-04 14:41:53 +00:00
John Ericson
aad4abcc9c Fix floating CA tests
We will sometimes try to query the outputs of derivations we can't
resolve. That's fine; it just means we don't know what those outputs are
yet.
2020-09-04 01:17:38 +00:00
John Ericson
975a47f7fe Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-09-03 22:09:04 +00:00
John Ericson
e7d93e7ece Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-03 15:43:17 +00:00
Eelco Dolstra
8a945d6ddb
Remove gc-check-reachability 2020-09-03 11:30:15 +02:00
Eelco Dolstra
82b77a7726
addPermRoot(): Remove indirect flag 2020-09-03 11:29:34 +02:00
Eelco Dolstra
b07167be5a
createGeneration(): Always create an indirect root
This means profiles outside of /nix/var/nix/profiles don't get
garbage-collected. It also means we don't need to scan
/nix/var/nix/profiles for GC roots anymore, except for compatibility
with previously existing generations.
2020-09-03 11:13:53 +02:00
Eelco Dolstra
b74f5cdd23
createGeneration(): Take a StorePath 2020-09-03 11:06:56 +02:00
Domen Kožar
dd4b56c87f
Allow HTTP binary cache to request absolute uris 2020-09-01 21:35:48 +02:00
John Ericson
ef278d00f9 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-01 18:01:48 +00:00
Eelco Dolstra
84f5cabbea Merge remote-tracking branch 'origin/master' into markdown 2020-08-31 14:24:26 +02:00
John Ericson
02e0001fc0 Merge branch 'ca-floating-upstream' of github.com:obsidiansystems/nix into ca-floating-upstream 2020-08-28 21:23:09 +00:00
John Ericson
421ed527c7
Update src/libstore/build.cc
Thanks for catching, @regnat.
2020-08-28 17:22:57 -04:00
John Ericson
48e8828050 Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-08-28 20:38:25 +00:00
John Ericson
8017fe7487 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-28 19:59:14 +00:00
Eelco Dolstra
eb75282b8d
Merge pull request #3434 from Ericson2314/derivation-header-include-order
Revise division of labor in deserialization of derivations
2020-08-27 16:39:28 +02:00
Eelco Dolstra
a0f19d9f3a RemoteStore::addToStore(): Fix race between stderrThread and NAR writer
As pointed out by @B4dM4n, the call to to.flush() on stderrThread is
unsafe because the NAR writer thread is also writing to 'to'.

Fixes #3943.
2020-08-27 14:50:51 +02:00
Eelco Dolstra
4bf5faf416 Merge remote-tracking branch 'origin/master' into markdown 2020-08-25 19:47:34 +02:00
John Ericson
8eb73a8724 CA derivations that depend on other CA derivations
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-08-24 19:01:15 +00:00
John Ericson
e0b0e18905 Add constructor for BasicDerivation -> Derivation 2020-08-24 19:01:07 +00:00
John Ericson
59979e7053 Fix bad debug format string 2020-08-24 19:01:07 +00:00
Eelco Dolstra
6a67e57019
Add DummyStore (dummy://)
DummyStore does not allow building or adding paths. This is useful for
evaluation tests when you don't want to initialize a "proper" store.
2020-08-24 18:54:16 +02:00
Eelco Dolstra
2530cc6827
Merge pull request #3950 from obsidiansystems/tabs-spaces
tabs -> spaces
2020-08-24 10:20:28 +02:00
John Ericson
b0b59fd05a Merge remote-tracking branch 'obsidian/write-derivation-borrow' into HEAD 2020-08-23 15:11:10 +00:00
John Ericson
35e6288be1 writeDerivation just needs a plain store reference 2020-08-23 15:01:11 +00:00
John Ericson
3a7b330b64 "Downstream placeholders" should not be store paths
Insead they should be opaque `/<hash>` like the placeholders we already
have.
2020-08-21 19:35:35 +00:00
John Ericson
422affe102 tabs -> spaces
Sorry I let the tab sneak in there in the first place.
2020-08-21 19:19:24 +00:00
John Ericson
c265e0ea82 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-08-20 20:24:22 +00:00
John Ericson
27a3f82c0b Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-20 18:28:17 +00:00
John Ericson
45a2f1baab Rename drv output querying functions, like master
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
 - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap
2020-08-20 18:14:12 +00:00
Eelco Dolstra
4d77513d97
Merge pull request #3859 from obsidiansystems/drv-outputs-map-allow-missing
`queryDerivationOutputMap` no longer assumes all outputs have a mapping
2020-08-20 16:49:23 +02:00
John Ericson
9a9d834dc7 Rename drv output querying functions
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
 - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap`
2020-08-20 14:12:51 +00:00
Eelco Dolstra
acb99f03f9
Config: Use nlohmann/json 2020-08-20 11:02:16 +02:00
John Ericson
a83694c7a1 Use RemoteStore to open connection for proxying daemon
Removes duplicate websocket opening code, and also means we should be
able to to ssh-ssh-... daemon relays, not just uds-uds-... ones.
2020-08-19 19:34:47 +00:00
Eelco Dolstra
c8fa39324a
Generate the nix.conf docs from the source code
This means we don't have two (divergent) sets of option descriptions
anymore.
2020-08-19 18:28:04 +02:00
John Ericson
d5af5763cf Merge branch 'master' of github.com:NixOS/nix into remove-storetype-delegate-regStore 2020-08-19 15:54:50 +00:00
John Ericson
be0d429b95 Merge branch 'master' of github.com:NixOS/nix into templated-daemon-protocol 2020-08-19 03:17:41 +00:00
John Ericson
950ddfdb82 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-08-18 14:36:44 +00:00
Eelco Dolstra
dfeb76dbf9
Merge pull request #3930 from obsidiansystems/legacy-ssh-build-paths
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths`
2020-08-18 16:07:40 +02:00
Eelco Dolstra
7cdc739ece
Merge remote-tracking branch 'origin/master' into markdown 2020-08-17 13:43:39 +02:00
Eelco Dolstra
847a5392f4 Merge branch 'ca-no-need-trust' of https://github.com/obsidiansystems/nix 2020-08-17 10:57:02 +02:00
Eelco Dolstra
3c619f6290 Merge branch 'test-RemoteStore-buildDerivation' of https://github.com/obsidiansystems/nix 2020-08-17 10:53:18 +02:00
John Ericson
6f7ac5e865 Remove extra closing paren 2020-08-14 21:59:31 +00:00
John Ericson
3c8b5b6219 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-14 17:00:13 +00:00
John Ericson
4b571ea321
Update src/libstore/daemon.cc
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-08-14 11:52:37 -04:00
Eelco Dolstra
13e49be660
Merge pull request #3875 from obsidiansystems/new-interface-for-path-pathOpt
Offer a safer interface for path and pathOpt
2020-08-14 17:19:19 +02:00
Eelco Dolstra
7714d9a943
Merge pull request #3924 from obsidiansystems/features-per-store
Make `system-features` a store setting
2020-08-14 17:13:07 +02:00
Eelco Dolstra
9b9d5297ba
Merge pull request #3909 from matthewbauer/readd-hashed-mirrors
Add hashed-mirrors back
2020-08-14 17:05:47 +02:00
Eelco Dolstra
d81f13f7cb
Merge pull request #3899 from obsidiansystems/make-narHash-not-optional
Make narHash in ValidPathInfo not optional
2020-08-14 17:00:18 +02:00
John Ericson
e1308b1211 Define LegacySSHStore::buildPaths using cmdBuildPaths
Evidentally this was never implemented because Nix switched to using
`buildDerivation` exclusively before `build-remote.pl` was rewritten.

The `nix-copy-ssh` test (already) tests this.
2020-08-13 21:27:55 +00:00
John Ericson
5ccd94501d Allow trustless building of CA derivations
Include a long comment explaining the policy. Perhaps this can be moved
to the manual at some point in the future.

Also bump the daemon protocol minor version, so clients can tell whether
`wopBuildDerivation` supports trustless CA derivation building. I hope
to take advantage of this in a follow-up PR to support trustless remote
building with the minimal sending of derivation closures.
2020-08-13 18:15:57 +00:00
John Ericson
85aacbee64 Use TeeSink and TeeSouce in a few more places 2020-08-13 14:51:17 +00:00
John Ericson
d2f2be0f70 Test RemoteStore::buildDerivation
Fix `wopNarFromPath` which needed a `toRealPath`.
2020-08-13 04:07:14 +00:00
John Ericson
5d67f18c86 Merge branch 'daemon-auth-cleanup' of github.com:obsidiansystems/nix into HEAD 2020-08-12 18:22:31 +00:00
John Ericson
4720853129 Make system-features a store setting
This seems more correct. It also means one can specify the features a
store should support with --store and remote-store=..., which is useful.
I use this to clean up the build remotes test.
2020-08-12 18:13:00 +00:00
John Ericson
8d4162ff9e Separate auth and logic for the daemon
Before, processConnection wanted to know a user name and user id, and
`nix-daemon --stdio`, when it isn't proxying to an underlying daemon,
would just assume "root" and 0. But `nix-daemon --stdio` (no proxying)
shouldn't make guesses about who holds the other end of its standard
streams.

Now processConnection takes an "auth hook", so `nix-daemon` can provide
the appropriate policy and daemon.cc doesn't need to know or care what
it is.
2020-08-12 15:22:33 +00:00
John Ericson
5f80aea795 Break out lambda so output can be matched just once
This is much better.
2020-08-12 02:23:31 +00:00
John Ericson
18834f7764 Recheck path validity after acquiring lock
It might have changed, and in any event this is how the cod used to work
so let's just keep it.
2020-08-11 23:44:02 +00:00
John Ericson
2de201254e Don't assume a total output map in two places in build.cc
Thanks @regnat for catching one of them. The other follows for many of
the same reasons. I'm find fixing others on a need-to-fix basis,
provided their are no regressions.
2020-08-11 23:07:50 +00:00
John Ericson
4c6aac8fdf Clarify comment on sandbox and temp fresh paths 2020-08-11 22:46:05 +00:00
John Ericson
6d57139050 Clarify outputReferences variable with self-describing type
Thanks for the idea, @Regnat!
2020-08-11 22:34:09 +00:00
John Ericson
8a068bd025 Remove redundant equality check 2020-08-11 21:25:40 +00:00
John Ericson
07e3466eb4 Float comment to out describe gaveUpOnSubstitution in general 2020-08-11 21:16:14 +00:00
John Ericson
d0f6e338dd
Apply suggestions from code review
Thanks!!

Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-08-11 16:49:10 -04:00
John Ericson
2a0902634e Fix error in merge breaking floating CA drvs
Forgot to add this hunk!
2020-08-11 00:13:19 +00:00
John Ericson
1b5c24662b Merge branch 'small-drv-serialize-cleanup' of github.com:obsidiansystems/nix into single-ca-drv-build 2020-08-10 01:57:54 +00:00
John Ericson
581183d4d5 Deduplicate parsing and reading derivations 2020-08-10 01:40:50 +00:00
John Ericson
bcd0629c2e Remove name parameter from writeDerivation
The name is now stored with the derivation itself.
2020-08-10 01:35:59 +00:00
John Ericson
e913a2989f Squashed get CA derivations building 2020-08-07 19:51:55 +00:00
Eelco Dolstra
edfd676e05 Fix .ls file names in binary caches
These are not supposed to include the 'name' part of the store
path. This was broken by 759947bf72.
2020-08-07 21:18:29 +02:00
John Ericson
8f92bb5ad9 Merge branch 'drv-outputs-map-allow-missing' of github.com:obsidiansystems/nix into templated-daemon-protocol 2020-08-07 18:51:01 +00:00
John Ericson
f7ba16f9cb Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing 2020-08-07 17:46:39 +00:00
John Ericson
47644e49ca Specialize std::optional<StorePath> so this is backwards compatible
While I am cautious to break parametricity, I think it's OK in this
cases---we're not about to try to do some crazy polymorphic protocol
anytime soon.
2020-08-07 17:05:14 +00:00
Matthew Bauer
96c158d6e1 Fix build 2020-08-06 21:04:31 -05:00
Matthew Bauer
8abc577cc2 Merge remote-tracking branch 'origin/master' into readd-hashed-mirrors 2020-08-06 18:58:59 -05:00
Carlo Nucera
46f9dd56da Fix bug due to non-deterministic arg eval order 2020-08-06 19:30:05 -04:00
Matthew Bauer
641c950701 Add hashed-mirrors back
Some users have their own hashed-mirrors setup, that is used to mirror
things in addition to what’s available on tarballs.nixos.org. Although
this should be feasable to do with a Binary Cache, it’s not always
easy, since you have to remember what "name" each of the tarballs has.
Continuing to support hashed-mirrors is cheap, so it’s best to leave
support in Nix. Note that NIX_HASHED_MIRRORS is also supported in
Nixpkgs through fetchurl.nix.

Note that this excludes tarballs.nixos.org from the default, as in
\#3689. All of these are available on cache.nixos.org.
2020-08-06 18:19:09 -05:00
Carlo Nucera
9ab07e99f5 Use template structs instead of phantoms 2020-08-06 18:04:13 -04:00
Carlo Nucera
3d8240c32e Remove leftover commented code 2020-08-06 16:04:18 -04:00
Carlo Nucera
f795f0fabc Merge branch 'drv-outputs-map-allow-missing-namespace' of github.com:obsidiansystems/nix into templated-daemon-protocol 2020-08-06 15:53:09 -04:00
John Ericson
e89b5bd0bf Minimize the usage of Hash::dummy 2020-08-06 18:31:48 +00:00
John Ericson
5e59b25a23 Merge remote-tracking branch 'upstream/master' into make-narHash-not-optional 2020-08-06 15:34:15 +00:00
Carlo Nucera
8b175f58d1 Simplify the namespace 2020-08-05 17:57:07 -04:00
Carlo Nucera
0739d428e0 Solve template deduction problem
We had to predeclare our template functions
2020-08-05 17:49:45 -04:00
John Ericson
6c66331d5c WIP: Put the worker protocol read and write in a namespace to disambig 2020-08-05 20:37:48 +00:00
Carlo Nucera
1d2e80ddd6 Merge branch 'master' of github.com:NixOS/nix into new-interface-for-path-pathOpt 2020-08-05 15:45:33 -04:00
John Ericson
ed96e603e1 Proxy -> Phantom to match Rust
Sorry, Haskell.
2020-08-05 19:44:08 +00:00
Carlo Nucera
8241e660ba Remove Hash::operator bool ()
Since the hash is not optional anymore
2020-08-05 15:30:38 -04:00
Carlo Nucera
be6e1c6457 Merge branch 'master' of github.com:NixOS/nix into make-narHash-not-optional 2020-08-05 15:14:47 -04:00
Carlo Nucera
1ad6394b33 Add Hash::dummy to signal default value
We did this in the same spirit of the dummy value that's present in
libstore/path.hh
2020-08-05 15:11:49 -04:00
Carlo Nucera
1d71028f4d Remove optionality in ValidPathInfo::narInfo 2020-08-05 14:42:48 -04:00
John Ericson
7302761f64 Merge remote-tracking branch 'obsidian/drv-outputs-map-allow-missing' into templated-daemon-protocol 2020-08-05 17:53:24 +00:00
Ben Burdette
f1a47a96b6 error messages for issue 2238 2020-08-05 10:58:00 -06:00
John Ericson
cf939055c8 Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing 2020-08-05 16:43:30 +00:00
John Ericson
0559ff3d8b Merge branch 'master' of github.com:NixOS/nix into derivation-primop-floating-output 2020-08-05 14:56:40 +00:00
John Ericson
92ad550e96 Merge remote-tracking branch 'obsidian/misc-ca' into derivation-primop-floating-output 2020-08-05 14:51:41 +00:00
Eelco Dolstra
790b694be7 Style fix 2020-08-05 16:51:06 +02:00
Eelco Dolstra
e48f944e9d Merge branch 'misc-ca' of https://github.com/obsidiansystems/nix 2020-08-05 16:50:05 +02:00
John Ericson
b9ebe373bb Sed some names to perhaps avoid conflicts 2020-08-05 14:49:25 +00:00
Eelco Dolstra
25f7912156 Style fix 2020-08-05 16:47:48 +02:00
John Ericson
e561a13a58 Reanme DerivationType::Regular defintion too
This is the one non-prefixed occurence
2020-08-05 14:45:56 +00:00
John Ericson
e7b0847f2d Make names more consistent 2020-08-05 14:44:39 +00:00
John Ericson
839f0fe095 Merge remote-tracking branch 'upstream/master' into misc-ca 2020-08-05 14:40:01 +00:00
John Ericson
03f4fafc27 Merge remote-tracking branch 'upstream/master' into misc-ca 2020-08-05 14:36:25 +00:00
Eelco Dolstra
b91dc7ebad
Merge pull request #3730 from obsidiansystems/better-ca-parse-errors
Improve hash parsing and errors
2020-08-05 16:33:07 +02:00
Eelco Dolstra
75f220a595
Merge pull request #3864 from obsidiansystems/more-topo-sort
Abstract out topo sorting logic
2020-08-05 16:07:29 +02:00
John Ericson
e61061c88e Remove stray tabs 2020-08-04 23:17:11 +00:00
John Ericson
36dd7920a2 Merge remote-tracking branch 'upstream/master' into path-info 2020-08-04 23:16:18 +00:00
John Ericson
16c98bf57c Get rid of some unneeded temporaries 2020-08-04 22:36:31 +00:00
John Ericson
1dfcbebc95 Organize and format code a bit 2020-08-04 22:28:10 +00:00
John Ericson
45b6fdb22b Remove unused functions 2020-08-04 22:10:13 +00:00
John Ericson
1bab8a321f Remove unneeded definition
Template instantiations will cover this case fine.
2020-08-04 21:56:42 +00:00
John Ericson
2f2ae993dc WIP systematize more of the worker protocol
This refactor should *not* change the wire protocol.
2020-08-04 19:02:05 +00:00
John Ericson
fbeb8695fb Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing 2020-08-04 18:35:33 +00:00
John Ericson
d3452a5ed6 Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors 2020-08-04 16:13:20 +00:00
Eelco Dolstra
4e7f1c7f11 S3BinaryCacheStore: Fix size determination 2020-08-04 16:01:13 +02:00
Eelco Dolstra
dfe66420e7 Revert "Remove putBytes"
This reverts commit b8eea7e81a.
2020-08-04 15:56:10 +02:00
Eelco Dolstra
327b1bf378 BinaryCacheStore: Explicitly flush file sink
The file sink is also flushed in its destructor, but we ignore any
exceptions in the destructor.

Issue #3886.
2020-08-04 14:54:07 +02:00
Eelco Dolstra
ed52cf632b
Merge pull request #3892 from obsidiansystems/path-info-header
Move ValidPathInfo to its own header
2020-08-04 10:44:47 +02:00
John Ericson
062533f7cd Merge remote-tracking branch 'upstream/master' into path-info-header 2020-08-03 21:02:28 +00:00
Eelco Dolstra
24e07c428f Delete compressed NARs
Fixes #3891.
2020-08-03 18:34:52 +02:00
John Ericson
54281f3ac1 addToStore in terms of addToStoreFromDump is not local-store-specific 2020-08-03 04:13:45 +00:00
John Ericson
9357512d73 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-08-01 19:38:35 +00:00
John Ericson
0e9e599b00 Merge remote-tracking branch 'upstream/master' into path-info 2020-08-01 00:06:24 +00:00
Carlo Nucera
b6d97fdbf4 Merge branch 'master' of github.com:NixOS/nix into drv-outputs-map-allow-missing 2020-07-31 13:12:51 -04:00