Commit graph

2913 commits

Author SHA1 Message Date
Eelco Dolstra
34b12bad59
NarInfoDiskCache: Handle SQLite busy errors 2017-02-28 13:44:11 +01:00
Eelco Dolstra
80027144ae
In SQLite errors, include the database path
This is necessary because we have multiple SQLite databases (e.g. the
binary cache cache).
2017-02-28 13:20:12 +01:00
Eelco Dolstra
7251d048fa
Support auto-configuration of build-max-jobs
"build-max-jobs" and the "-j" option can now be set to "auto" to use
the number of CPUs in the system. (Unlike build-cores, it doesn't use
0 to imply auto-configuration, because a) magic values are a bad idea
in general; b) 0 is a legitimate value used to disable local
building.)

Fixes #1198.
2017-02-28 12:54:50 +01:00
Eelco Dolstra
3fab1f04a7
_SC_NPROCESSORS_ONLN -> std:🧵:hardware_concurrency() 2017-02-27 16:01:54 +01:00
Eelco Dolstra
3cb0387d3f
Retry downloads on HTTP/2 stream errors
Issue #1254.
2017-02-27 14:36:09 +01:00
Eelco Dolstra
d3e1aad421
nix-shell: Better error message when the shell can't be started 2017-02-24 17:25:00 +01:00
Eelco Dolstra
e76df9bd52
Register content-addressability assertion for fixed outputs 2017-02-24 16:39:53 +01:00
Eelco Dolstra
1c718f80d3
Verify content-addressability assertions at registration time 2017-02-24 16:38:14 +01:00
Eelco Dolstra
668fef2e4f
nix-shell: Overwrite environment variables
Need to remember that std::map::insert() and emplace() don't overwrite
existing entries...

This fixes a regression relative to 1.11 that in particular triggers
in nested nix-shells.

Before:

  $ nativeBuildInputs=/foo nix-shell -p hello --run 'hello'
  build input /foo does not exist

After:

  $ nativeBuildInputs=/foo nix-shell -p hello --run 'hello'
  Hello, world!
2017-02-24 13:31:46 +01:00
Eelco Dolstra
f023f64f40
RemoteStore::addToStore(): Pass content-addressability assertion
... and use this in Downloader::downloadCached(). This fixes

  $ nix-build https://nixos.org/channels/nixos-16.09-small/nixexprs.tar.xz -A hello
  error: cannot import path ‘/nix/store/csfbp1s60dkgmk9f8g0zk0mwb7hzgabd-nixexprs.tar.xz’ because it lacks a valid signature
2017-02-22 16:58:00 +01:00
Eelco Dolstra
8b1b5f9a12
Handle CURLE_RECV_ERROR as a transient error
This fixes

  unable to download ‘https://cache.nixos.org/nar/077h8ji74y9b0qx7rjk71xd80vjqp6q5gy137r553jlvdlxdcdlk.nar.xz’: HTTP error 200 (curl error: Failure when receiving data from the peer)
2017-02-21 16:04:47 +01:00
Eelco Dolstra
99bbddedb1
Fix building without S3 support
http://hydra.nixos.org/build/49031196/nixlog/2/raw
2017-02-21 13:15:07 +01:00
Eelco Dolstra
df66d346df
Log AWS retries 2017-02-21 11:50:31 +01:00
Eelco Dolstra
302386f775
Support netrc in <nix/fetchurl.nix>
This allows <nix/fetchurl.nix> to fetch private Git/Mercurial
repositories, e.g.

  import <nix/fetchurl.nix> {
    url = https://edolstra@bitbucket.org/edolstra/my-private-repo/get/80a14018daed.tar.bz2;
    sha256 = "1mgqzn7biqkq3hf2697b0jc4wabkqhmzq2srdymjfa6sb9zb6qs7";
  }

where /etc/nix/netrc contains:

  machine bitbucket.org
  login edolstra
  password blabla...

This works even when sandboxing is enabled.

To do: add unpacking support (i.e. fetchzip functionality).
2017-02-16 15:51:50 +01:00
Eelco Dolstra
cde4b60919
Move netrcFile to Settings
Also get rid of Settings::processEnvironment(), it appears to be
useless.
2017-02-16 14:50:41 +01:00
Eelco Dolstra
40f0e3b366
Include "curl" in the User-Agent header
Some sites (e.g. BitBucket) give a helpful 401 error when trying to
download a private archive if the User-Agent contains "curl", but give
a redirect to a login page otherwise (so for instance
"nix-prefetch-url" will succeed but produce useless output).
2017-02-16 13:55:43 +01:00
Eelco Dolstra
d1139ff36b Merge pull request #1215 from k0001/netrc-1
Add netrc-file support
2017-02-16 12:45:44 +01:00
Eelco Dolstra
9ff9c3f2f8
Add support for s3:// URIs
This adds support for s3:// URIs in all places where Nix allows URIs,
e.g. in builtins.fetchurl, builtins.fetchTarball, <nix/fetchurl.nix>
and NIX_PATH. It allows fetching resources from private S3 buckets,
using credentials obtained from the standard places (i.e. AWS_*
environment variables, ~/.aws/credentials and the EC2 metadata
server). This may not be super-useful in general, but since we already
depend on aws-sdk-cpp, it's a cheap feature to add.
2017-02-14 14:20:00 +01:00
Eelco Dolstra
62ff5ad424 Merge pull request #1224 from dezgeg/configh
Unbreak 32-bit builds by always implicitly including config.h
2017-02-13 14:44:58 +01:00
Tuomas Tynkkynen
649a81bcd6 nix-daemon: Don't splice with len=SIZE_MAX
Currently, 'nix-daemon --stdio' is always failing for me, due to the
splice call always failing with (on a 32-bit host):

splice(0, NULL, 3, NULL, 4294967295, SPLICE_F_MOVE) = -1 EINVAL (Invalid argument)

With a bit of ftracing (and luck) the problem seems to be that splice()
always fails with EINVAL if the len cast as ssize_t is negative:
http://lxr.free-electrons.com/source/fs/read_write.c?v=4.4#L384

So use SSIZE_MAX instead of SIZE_MAX.
2017-02-13 15:14:44 +02:00
Renzo Carbonara
e2257d4eeb Documentation. 2017-02-09 18:16:09 +01:00
Tuomas Tynkkynen
2cd468874f Include config.h implicitly with '-include config.h' in CFLAGS
Because config.h can #define things like _FILE_OFFSET_BITS=64 and not
every compilation unit includes config.h, we currently compile half of
Nix with _FILE_OFFSET_BITS=64 and other half with _FILE_OFFSET_BITS
unset. This causes major havoc with the Settings class on e.g. 32-bit ARM,
where different compilation units disagree with the struct layout.

E.g.:

diff --git a/src/libstore/globals.cc b/src/libstore/globals.cc
@@ -166,6 +166,8 @@ void Settings::update()
     _get(useSubstitutes, "build-use-substitutes");
+    fprintf(stderr, "at Settings::update(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes);
     _get(buildUsersGroup, "build-users-group");
diff --git a/src/libstore/remote-store.cc b/src/libstore/remote-store.cc
+++ b/src/libstore/remote-store.cc
@@ -138,6 +138,8 @@ void RemoteStore::initConnection(Connection & conn)
 void RemoteStore::setOptions(Connection & conn)
 {
+    fprintf(stderr, "at RemoteStore::setOptions(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes);
     conn.to << wopSetOptions

Gave me:

at Settings::update(): &useSubstitutes = 0xb6e5c5cb
at RemoteStore::setOptions(): &useSubstitutes = 0xb6e5c5c7

That was not a fun one to debug!
2017-02-08 21:51:02 +02:00
Eelco Dolstra
4724903c78
nix-copy-closure: Use computeFSClosure() and LegacySSHStore 2017-02-07 20:55:47 +01:00
Eelco Dolstra
6f4682ad36
Merge branch 'nix-copy-closure-c++' of https://github.com/shlevy/nix 2017-02-07 20:47:45 +01:00
Eelco Dolstra
caa5793b4a
Add a LegacySSHStore that uses nix-store --serve
This is useful for nix-copy-closure.
2017-02-07 19:29:21 +01:00
Eelco Dolstra
f38224e924
copyStorePath(): Don't require signatures for "trusted" stores
For example, SSH stores could be trusted.
2017-02-07 19:29:21 +01:00
Eelco Dolstra
fa07558a06
Provide default implementations for a couple of Store methods 2017-02-07 19:29:21 +01:00
Eelco Dolstra
ddb5577f2e
Move SavingSourceAdapter to serialise.hh 2017-02-07 19:29:20 +01:00
Eelco Dolstra
ce4d8e3ef8
Remove unnecessary call to topoSortPaths()
exportPaths() already does this.
2017-02-07 19:20:41 +01:00
Eelco Dolstra
7a58ad0ef5
SSHStore: uri -> host 2017-02-07 19:20:15 +01:00
Eelco Dolstra
612aeb2df5
Cleanup 2017-02-07 19:16:25 +01:00
Eelco Dolstra
c54814b175
Remove download-via-ssh
Replaced by SSHStore.
2017-02-07 18:54:33 +01:00
Renzo Carbonara
e6e74f987f Add netrc-file support 2017-02-02 13:24:20 +01:00
Eelco Dolstra
1351b0df87
exportReferencesGraph: Only export in JSON format when in structured mode
This prevents breaking compatibility with builders that read
"closure.*", since they would accidentally pick up the new JSON files.
2017-02-02 12:20:28 +01:00
Eelco Dolstra
7a65b2470e
Restore default signal handling in child processes
In particular, this fixes Ctrl-C in nix-shell sessions.
2017-02-01 13:00:21 +01:00
Eelco Dolstra
cb1951e746
Periodically purge binary-cache.sqlite 2017-01-27 15:19:33 +01:00
Eelco Dolstra
211bc7f0e6
Implement TTL for binary cache lookups 2017-01-27 13:17:08 +01:00
Eelco Dolstra
f57a38b109
Remove unused NARExistence table 2017-01-27 12:57:49 +01:00
Eelco Dolstra
c2b0d8749f
exportReferencesGraph: Export more complete info in JSON format
This writes info about every path in the closure in the same format as
‘nix path-info --json’. Thus it also includes NAR hashes and sizes.

Example:

  [
    {
      "path": "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10",
      "narHash": "sha256:0ckdc4z20kkmpqdilx0wl6cricxv90lh85xpv2qljppcmz6vzcxl",
      "narSize": 197648,
      "references": [
        "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10",
        "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24"
      ],
      "closureSize": 20939776
    },
    {
      "path": "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24",
      "narHash": "sha256:1nfn3m3p98y1c0kd0brp80dn9n5mycwgrk183j17rajya0h7gax3",
      "narSize": 20742128,
      "references": [
        "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24"
      ],
      "closureSize": 20742128
    }
  ]

Fixes #1134.
2017-01-26 20:41:08 +01:00
Eelco Dolstra
6de33a9c67
Add support for passing structured data to builders
Previously, all derivation attributes had to be coerced into strings
so that they could be passed via the environment. This is lossy
(e.g. lists get flattened, necessitating configureFlags
vs. configureFlagsArray, of which the latter cannot be specified as an
attribute), doesn't support attribute sets at all, and has size
limitations (necessitating hacks like passAsFile).

This patch adds a new mode for passing attributes to builders, namely
encoded as a JSON file ".attrs.json" in the current directory of the
builder. This mode is activated via the special attribute

  __structuredAttrs = true;

(The idea is that one day we can set this in stdenv.mkDerivation.)

For example,

  stdenv.mkDerivation {
    __structuredAttrs = true;
    name = "foo";
    buildInputs = [ pkgs.hello pkgs.cowsay ];
    doCheck = true;
    hardening.format = false;
  }

results in a ".attrs.json" file containing (sans the indentation):

  {
    "buildInputs": [],
    "builder": "/nix/store/ygl61ycpr2vjqrx775l1r2mw1g2rb754-bash-4.3-p48/bin/bash",
    "configureFlags": [
      "--with-foo",
      "--with-bar=1 2"
    ],
    "doCheck": true,
    "hardening": {
      "format": false
    },
    "name": "foo",
    "nativeBuildInputs": [
      "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10",
      "/nix/store/4jnvjin0r6wp6cv1hdm5jbkx3vinlcvk-cowsay-3.03"
    ],
    "propagatedBuildInputs": [],
    "propagatedNativeBuildInputs": [],
    "stdenv": "/nix/store/f3hw3p8armnzy6xhd4h8s7anfjrs15n2-stdenv",
    "system": "x86_64-linux"
  }

"passAsFile" is ignored in this mode because it's not needed - large
strings are included directly in the JSON representation.

It is up to the builder to do something with the JSON
representation. For example, in bash-based builders, lists/attrsets of
string values could be mapped to bash (associative) arrays.
2017-01-26 20:40:33 +01:00
Eelco Dolstra
54801ed6ad
Bindings: Add a method for iterating in lexicographically sorted order 2017-01-26 20:40:33 +01:00
Eelco Dolstra
b1f001538e
Fix assertion failure when a path is locked
Fixes:

  nix-store: src/libstore/build.cc:3649: void nix::Worker::run(const Goals&): Assertion `!awake.empty()' failed.
2017-01-26 20:40:33 +01:00
Eelco Dolstra
83ae6503e8
Fix interrupt handling 2017-01-26 20:40:33 +01:00
Eelco Dolstra
951357e5fb
UserLock: Fix multi-threaded access to a global variable 2017-01-26 20:40:33 +01:00
Eelco Dolstra
a55f589720
openLockFile: Return an AutoCloseFD 2017-01-26 20:40:33 +01:00
Eelco Dolstra
c0f2f4eeef
UserLock: Make more RAII-ish 2017-01-26 20:40:33 +01:00
Eelco Dolstra
a529c740d2
Moving more code out of DerivationGoal::startBuilder() 2017-01-26 20:40:33 +01:00
Eelco Dolstra
e8c43abd9a
On HTTP errors, also show the curl error
This is a hopefully temporary measure to diagnose the intermittent
"HTTP error 200" failures.
2017-01-26 20:40:32 +01:00
Eelco Dolstra
4425a5c547
Move exportReferencesGraph into a separate method
startBuilder() is getting rather obese.
2017-01-26 20:40:32 +01:00
Eelco Dolstra
4e6a2fbc56
Revert "Propagate path context via builtins.readFile"
This reverts commit f7f0116dd7.

Issue #1174.
2017-01-24 15:31:54 +01:00