Commit graph

2178 commits

Author SHA1 Message Date
Shea Levy
584f8a62de Implement nar-based addToStore for remote-store 2016-09-02 14:33:58 -04:00
Shea Levy
ecba88de93 Add ssh store implementation 2016-09-02 14:31:38 -04:00
Shea Levy
b4b5e9ce2f Add narFromPath op to nix daemon 2016-09-02 14:26:02 -04:00
Shea Levy
a705e8ce0a Factor a general remote FS accessor out of BinaryCacheStore 2016-09-02 14:24:34 -04:00
Shea Levy
0f39633290 Factor out the unix domain socket-specific code from RemoteStore 2016-09-02 14:15:04 -04:00
Shea Levy
53b27ddce2 Factor a function to get the store type from a URI out of the main RegisterStoreImplementation 2016-09-02 06:39:29 -04:00
Shea Levy
a91954f0c6 Merge openStore and openStoreAt with default arguments 2016-09-02 06:35:48 -04:00
Shea Levy
dfe0938614 download.hh: Fix conflicts from nix-channel-c++ merge 2016-08-31 09:57:56 -04:00
Shea Levy
572aba284a Merge branch 'nix-channel-c++' 2016-08-31 09:49:24 -04:00
Eelco Dolstra
aa1ea0d1e4 Merge pull request #1030 from pikajude/rsd-cc
Implement resolve-system-dependencies in C++
2016-08-31 15:19:10 +02:00
Eelco Dolstra
6631a6e1a1 Increase the sleep time between download retries 2016-08-30 15:48:24 +02:00
Eelco Dolstra
97b1af1cbe Fix --fallback
This fixes an assertion failure in "assert(goal);" in
Worker::waitForInput() after a substitution goal is cancelled by the
termination of another goal. The problem was the line

  //worker.childTerminated(shared_from_this()); // FIXME

in the SubstitutionGoal destructor. This was disabled because
shared_from_this() obviously doesn't work from a destructor. So we now
use a real pointer for object identity.
2016-08-30 15:45:39 +02:00
Eelco Dolstra
a9d4f2b303 Fix queryPathFromHashPart()
The inner lambda was returning a SQLite-internal char * rather than a
std::string, leading to Hydra errors liks

  Caught exception in Hydra::Controller::Root->narinfo "path ‘ø˜£â€™ is not in the Nix store at /nix/store/6mvvyb8fgwj23miyal5mdr8ik4ixk15w-hydra-0.1.1234.abcdef/libexec/hydra/lib/Hydra/Controller/Root.pm line 352."
2016-08-24 14:53:33 +02:00
Eelco Dolstra
22d6e31fc6 Add a mechanism for derivation attributes to reference the derivation's outputs
For example, you can now say:

  configureFlags = "--prefix=${placeholder "out"} --includedir=${placeholder "dev"}";

The strings returned by the ‘placeholder’ builtin are replaced at
build time by the actual store paths corresponding to the specified
outputs.

Previously, you had to work around the inability to self-reference by doing stuff like:

  preConfigure = ''
    configureFlags+=" --prefix $out --includedir=$dev"
  '';

or rely on ad-hoc variable interpolation semantics in Autoconf or Make
(e.g. --prefix=\$(out)), which doesn't always work.
2016-08-17 17:19:32 +02:00
Shea Levy
d52d391164 builtins.fetch{url,tarball}: Allow name attribute 2016-08-15 07:37:11 -04:00
Jude Taylor
596e4a5693 remove old traces of resolve-system-dependencies 2016-08-13 15:27:49 -07:00
Shea Levy
59124228b3 nix-channel: implement in c++ 2016-08-11 11:34:43 -04:00
Eelco Dolstra
66adbdfd97 HttpBinaryCacheStore: Retry on transient HTTP errors
This makes us more robust against 500 errors from CloudFront or S3
(assuming the 500 error isn't cached by CloudFront...).
2016-08-10 18:08:23 +02:00
Eelco Dolstra
9204ea7294 Show HTTP error codes 2016-08-10 18:05:35 +02:00
Eelco Dolstra
f294623d1d SQLite:: Add some convenience 2016-08-10 18:05:35 +02:00
Eelco Dolstra
6cb4bdf152 FSAccessor: Throw InvalidPath 2016-08-10 18:05:35 +02:00
Eelco Dolstra
f43823f676 NarAccessor: Fix handling of non-executable files 2016-08-10 18:05:35 +02:00
Eelco Dolstra
d961c29c9c Mark content-addressed paths in the Nix database and in .narinfo
This allows such paths to be imported without signatures.
2016-08-10 18:05:35 +02:00
Eelco Dolstra
36a51ecab3 LocalFSStore::narFromPath(): Handle the diverted store case 2016-08-10 18:05:35 +02:00
Eelco Dolstra
3eb6217508 Add a "root" parameter to local stores
This makes it easier to create a diverted store, i.e.

  NIX_REMOTE="local?root=/tmp/root"

instead of

  NIX_REMOTE="local?real=/tmp/root/nix/store&state=/tmp/root/nix/var/nix" NIX_LOG_DIR=/tmp/root/nix/var/log
2016-08-10 18:05:35 +02:00
Eelco Dolstra
2fad86f361 Remove $NIX_DB_DIR
This variable has no reason to exist, given $NIX_STATE_DIR.
2016-08-10 18:05:35 +02:00
Eelco Dolstra
be64fbb501 RemoteStore / nix-daemon: Drop support for Nix < 1.0 2016-08-10 18:05:35 +02:00
Eelco Dolstra
ee22a91ab8 makeFixedOutputPath(): Drop superfluous HashType argument 2016-07-26 21:25:52 +02:00
Eelco Dolstra
06bbfb6004 builtins.{fetchurl,fetchTarball}: Support a sha256 attribute
Also, allow builtins.{fetchurl,fetchTarball} in restricted mode if a
hash is specified.
2016-07-26 21:16:52 +02:00
Shea Levy
ee3032e4de Merge branch 'find-runtime-roots-c++' 2016-07-24 07:33:49 -04:00
Shea Levy
18b0808475 Respect --keep-going when a substituter fails.
Fixes #977
2016-07-23 09:16:08 -04:00
Eelco Dolstra
e19c90fc6b Fix random failures caused by CurlDownloader modifying a shared string 2016-07-21 18:35:31 +02:00
Eelco Dolstra
ea8e8df6c7 NarInfo::NarInfo(): Ensure that we get a NAR size/hash 2016-07-21 18:35:12 +02:00
Eelco Dolstra
d57981bac4 Store::queryMissing(): Use a thread pool
For one particular NixOS configuration, this cut the runtime of
"nix-store -r --dry-run" from 6m51s to 3.4s. It also fixes a bug in
the size calculation that was causing certain paths to be counted
twice, e.g. before:

  these paths will be fetched (1249.98 MiB download, 2995.74 MiB unpacked):

and after:

  these paths will be fetched (1219.56 MiB download, 2862.17 MiB unpacked):
2016-07-21 17:50:39 +02:00
Eelco Dolstra
a728780fbd Store::computeFSClosure(): Use thread pool
This speeds up queries against the binary cache.
2016-07-21 14:13:35 +02:00
Shea Levy
3c46fe62b8 find-runtime-roots: fold into gc.cc 2016-07-21 07:04:41 -04:00
Shea Levy
cb5e7254b6 Modernize AutoCloseFD 2016-07-11 15:44:44 -04:00
Eelco Dolstra
74dd603495 Re-implement negative binary cache lookup caching 2016-06-20 17:39:05 +02:00
Eelco Dolstra
c68e5913c7 Run builds in a user namespace
This way, all builds appear to have a uid/gid of 0 inside the
chroot. In the future, this may allow using programs like
systemd-nspawn inside builds, but that will require assigning a larger
UID/GID map to the build.

Issue #625.
2016-06-09 18:30:39 +02:00
Eelco Dolstra
202683a4fc Use O_CLOEXEC in most places 2016-06-09 16:37:08 +02:00
Eelco Dolstra
9bdd949cfd Fix "creating statement: table ValidPaths has no column named ultimate" 2016-06-03 16:06:42 +02:00
Eelco Dolstra
5e51ffb1c2 Support sandbox builds by non-root users
This allows an unprivileged user to perform builds on a diverted store
(i.e. where the physical store location differs from the logical
location).

Example:

  $ NIX_LOG_DIR=/tmp/log NIX_REMOTE="local?real=/tmp/store&state=/tmp/var" nix-build -E \
    'with import <nixpkgs> {}; runCommand "foo" { buildInputs = [procps nettools]; } "id; ps; ifconfig; echo $out > $out"'

will do a build in the Nix store physically in /tmp/store but
logically in /nix/store (and thus using substituters for the latter).
2016-06-03 15:59:28 +02:00
Eelco Dolstra
2f8b0e557b Consistent quotes 2016-06-03 14:03:09 +02:00
Eelco Dolstra
080d607975 Don't call shared_from_this() from a destructor 2016-06-03 14:02:44 +02:00
Eelco Dolstra
30c8c3ba2d Fix some more references to storeDir 2016-06-03 14:02:44 +02:00
Eelco Dolstra
a8dfdc52b8 Merge pull request #925 from abbradar/master
Show both cycle ends
2016-06-03 13:58:51 +02:00
Nikolay Amiantov
c87a56f4d0 Show both cycle ends 2016-06-03 14:25:37 +03:00
Eelco Dolstra
75361b6dce Fix narrowing conversion from int64_t to bool
http://hydra.nixos.org/build/36613774
2016-06-03 13:20:46 +02:00
Eelco Dolstra
d64e0c1b64 Make derivationFromPath work on diverted stores 2016-06-02 18:43:36 +02:00
Eelco Dolstra
a24f2c9b84 nix run: Mount the Nix store in a private namespace
This is a convenience command to allow users who are not privileged to
create /nix/store to use Nix with regular binary caches. For example,

  $ NIX_REMOTE="local?state=$HOME/nix/var&real=/$HOME/nix/store" nix run firefox bashInteractive

will download Firefox and bash from cache.nixos.org, then start a
shell in which $HOME/nix/store is mounted on /nix/store.
2016-06-02 16:51:43 +02:00
Eelco Dolstra
4494000e04 LocalStore: Allow the physical and logical store directories to differ
This is primarily to subsume the functionality of the
copy-from-other-stores substituter. For example, in the NixOS
installer, we can now do (assuming we're in the target chroot, and the
Nix store of the installation CD is bind-mounted on /tmp/nix):

  $ nix-build ... --option substituters 'local?state=/tmp/nix/var&real=/tmp/nix/store'

However, unlike copy-from-other-stores, this also allows write access
to such a store. One application might be fetching substitutes for
/nix/store in a situation where the user doesn't have sufficient
privileges to create /nix, e.g.:

  $ NIX_REMOTE="local?state=/home/alice/nix/var&real=/home/alice/nix/store" nix-build ...
2016-06-02 16:02:48 +02:00
Eelco Dolstra
064816ab98 Respect build-use-substitutes 2016-06-02 16:02:48 +02:00
Eelco Dolstra
812c0dfbe2 Allow setting the state directory as a store parameter
E.g. "local?store=/tmp/store&state=/tmp/var".
2016-06-02 16:02:48 +02:00
Eelco Dolstra
f2682e6e18 <nix/fetchurl.nix>: Use tarballs.nixos.org
This ensures that the disappearance of upstream bootstrap tarballs
(e.g. https://github.com/NixOS/nixpkgs-channels/pull/1) doesn't break
stdenv rebuilds.
2016-06-01 17:11:51 +02:00
Eelco Dolstra
dead8189ac Typo 2016-06-01 16:56:22 +02:00
Eelco Dolstra
e24e2caaaf Skip substituters with an incompatible store directory 2016-06-01 16:40:49 +02:00
Eelco Dolstra
cf198952d0 HttpBinaryCacheStore: Fix caching of WantMassQuery
Also, test HttpBinaryCacheStore in addition to LocalBinaryCacheStore.
2016-06-01 16:24:17 +02:00
Eelco Dolstra
7850d3d279 Make the store directory a member variable of Store 2016-06-01 16:24:17 +02:00
Eelco Dolstra
a9fa5e050a Shut up some clang warnings 2016-05-31 13:31:04 +02:00
Eelco Dolstra
10f3a2e5f2 Fix clang build failure
Apparently opinion is divided on whether [[noreturn]] is allowed on a
lambda: http://stackoverflow.com/questions/26888805/how-to-declare-a-lambdas-operator-as-noreturn

http://hydra.nixos.org/build/36462100
2016-05-31 13:23:54 +02:00
Eelco Dolstra
c2d27d30cf nix-copy-closure / build-remote.pl: Disable signature checking
This restores the Nix 1.11 behaviour.
2016-05-31 11:48:05 +02:00
Eelco Dolstra
cd35f0280c Fix build 2016-05-30 16:12:50 +02:00
Eelco Dolstra
57d33013ce Check signatures before downloading the substitute 2016-05-30 15:18:12 +02:00
Eelco Dolstra
bac123ddd9 Test the NAR info cache 2016-05-30 15:18:12 +02:00
Eelco Dolstra
3593c8285d Re-implement binary cache signature checking
This is now done in LocalStore::addToStore(), rather than in the
binary cache substituter (which no longer exists).
2016-05-30 15:18:12 +02:00
Eelco Dolstra
12ddbad458 LocalStore::addToStore: Verify hash of the imported path 2016-05-30 15:18:12 +02:00
Eelco Dolstra
e222484401 Re-implement the WantMassQuery property of binary caches 2016-05-30 15:18:12 +02:00
Eelco Dolstra
b66ab6cdbc Fix repair during substitution 2016-05-30 15:18:12 +02:00
Eelco Dolstra
3be2e71ab3 BinaryCacheStore: Remove buildPaths() / ensurePath() 2016-05-30 15:18:12 +02:00
Eelco Dolstra
75d2492f20 Make the aws-cpp-sdk dependency optional 2016-05-04 17:16:48 +02:00
Eelco Dolstra
6c75cf69c3 Cleanup: Remove singleton() 2016-05-04 16:16:53 +02:00
Eelco Dolstra
0d4a10e910 Do compression in a sink 2016-05-04 16:16:53 +02:00
Eelco Dolstra
c6a21aed07 More GC fixes 2016-05-04 16:16:53 +02:00
Eelco Dolstra
538a64e8c3 Add a Store::addToStore() variant that accepts a NAR
As a side effect, this ensures that signatures are propagated when
copying paths between stores.

Also refactored import/export to make use of this.
2016-05-04 16:15:54 +02:00
Eelco Dolstra
f435f82475 Remove OpenSSL-based signing 2016-05-04 11:01:48 +02:00
Eelco Dolstra
dfebfc835f Add a copyStorePath() utility function 2016-05-04 11:01:48 +02:00
Eelco Dolstra
38539b943a Add fetchgit builtin
The function builtins.fetchgit fetches Git repositories at evaluation
time, similar to builtins.fetchTarball. (Perhaps the name should be
changed, being confusing with respect to Nixpkgs's fetchgit function,
with works at build time.)

Example:

  (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello

or

  (import (builtins.fetchgit {
    url = git://github.com/NixOS/nixpkgs-channels;
    rev = "nixos-16.03";
  }) {}).hello

Note that the result does not contain a .git directory.
2016-04-29 20:47:36 +02:00
Eelco Dolstra
4dde0b0562 BinaryCacheStore: Support bzip2 compression 2016-04-29 17:43:37 +02:00
Eelco Dolstra
6963de2091 nix verify --all: Support local binary caches 2016-04-29 17:34:31 +02:00
Eelco Dolstra
0dd988d2e3 Better error message 2016-04-29 17:33:22 +02:00
Eelco Dolstra
5acb691402 BinaryCacheStore: Support "none" compression method 2016-04-29 17:02:57 +02:00
Eelco Dolstra
8e065c6b3e BinaryCacheStore: Make the signing key a parameter 2016-04-29 16:47:20 +02:00
Eelco Dolstra
f6aee2f477 BinaryCacheStore::queryPathInfo: Don't check signatures
Other stores don't do this either. It's up to the caller to check
signatures.
2016-04-29 16:28:57 +02:00
Eelco Dolstra
95d20dfde9 Allow parameters in store URIs
This is to allow store-specific configuration,
e.g. s3://my-cache?compression=bzip2&secret-key=/path/to/key.
2016-04-29 16:26:16 +02:00
Eelco Dolstra
aa3bc3d5dc Eliminate the substituter mechanism
Substitution is now simply a Store -> Store copy operation, most
typically from BinaryCacheStore to LocalStore.
2016-04-29 13:57:08 +02:00
Eelco Dolstra
21e9d183cc Really handle carriage return 2016-04-28 14:27:00 +02:00
Eelco Dolstra
ce5776758d Fix error message 2016-04-28 14:12:10 +02:00
Eelco Dolstra
cc804d0dc6 Handle carriage return 2016-04-25 19:18:45 +02:00
Eelco Dolstra
5761827d5b Show the log tail when a build fails
If --no-build-output is given (which will become the default for the
"nix" command at least), show the last 10 lines of the build output if
the build fails.
2016-04-25 19:18:45 +02:00
Eelco Dolstra
6e1b099279 Remove --print-build-trace
This was added to support Hydra, but Hydra no longer uses it.
2016-04-25 19:18:45 +02:00
Eelco Dolstra
9eba2c3945 Fix "path is not in the Nix store" during GC 2016-04-25 19:18:45 +02:00
Eelco Dolstra
41633f9f73 Improved logging abstraction
This also gets rid of --log-type, since the nested log type isn't
useful in a multi-threaded situation, and nobody cares about the
"pretty" log type.
2016-04-25 19:18:45 +02:00
Eelco Dolstra
0207272b28 BinaryCacheStore: When adding a path, ensure the references are valid
This prevents copying a partial closure to a binary cache.
2016-04-22 12:15:06 +02:00
Eelco Dolstra
21ef1670b3 Fix test failures 2016-04-21 18:21:25 +02:00
Eelco Dolstra
7d14f5c331 Implement S3BinaryCacheStore::queryAllValidPaths()
This allows commands like "nix verify --all" or "nix path-info --all"
to work on S3 caches.

Unfortunately, this requires some ugly hackery: when querying the
contents of the bucket, we don't want to have to read every .narinfo
file. But the S3 bucket keys only include the hash part of each store
path, not the name part. So as a special exception
queryAllValidPaths() can now return store paths *without* the name
part, and queryPathInfo() accepts such store paths (returning a
ValidPathInfo object containing the full name).
2016-04-21 17:53:47 +02:00
Eelco Dolstra
d155d80155 Move S3BinaryCacheStore from Hydra
This allows running arbitrary Nix commands against an S3 binary cache.

To do: make this a compile time option to prevent a dependency on
aws-sdk-cpp.
2016-04-21 16:08:51 +02:00
Eelco Dolstra
ddea253ff8 RemoteStore: Propagate InvalidPath exceptions from the daemon 2016-04-20 15:28:07 +02:00
Eelco Dolstra
c0c4ddcd9c BinaryCacheStore: Insert new paths into the disk cache 2016-04-20 15:27:48 +02:00
Eelco Dolstra
451ebf24ce Cache path info lookups in SQLite
This re-implements the binary cache database in C++, allowing it to be
used by other Store backends, in particular the S3 backend.
2016-04-20 14:12:38 +02:00
Eelco Dolstra
e0204f8d46 Move path info caching from BinaryCacheStore to Store
Caching path info is generally useful. For instance, it speeds up "nix
path-info -rS /run/current-system" (i.e. showing the closure sizes of
all paths in the closure of the current system) from 5.6s to 0.15s.

This also eliminates some APIs like Store::queryDeriver() and
Store::queryReferences().
2016-04-19 18:52:53 +02:00
Dan Peebles
608b0265e1 Print out all bad references/requisites at once
Also updates tests to check for new information. Fixes #799
2016-04-16 19:58:26 -04:00
Eelco Dolstra
0423787086 Make the .narinfo cache bigger 2016-04-15 15:39:48 +02:00
Eelco Dolstra
a7d8eaba54 BinaryCacheStore: Do negative caching of .narinfo lookups 2016-04-15 15:39:48 +02:00
Eelco Dolstra
d1b0909894 BinaryCacheStore::readFile(): Return a shared_ptr to a string
This allows readFile() to indicate that a file doesn't exist, and
might eliminate some large string copying.
2016-04-15 15:39:48 +02:00
Eelco Dolstra
c045630522 Support channel:<channel-name> URIs
For convenience, you can now say

  $ nix-env -f channel:nixos-16.03 -iA hello

instead of

  $ nix-env -f https://nixos.org/channels/nixos-16.03/nixexprs.tar.xz -iA hello

Similarly,

  $ nix-shell -I channel:nixpkgs-unstable -p hello
  $ nix-build channel:nixos-15.09 -A hello

Abstracting over the NixOS/Nixpkgs channels location also allows us to
use a more efficient transport (e.g. Git) in the future.
2016-04-14 17:26:57 +02:00
Eelco Dolstra
3c1c6b8f00 Set RLIMIT_CORE to 0, infinity in builders
This prevents the builder from being affected by whatever the host
system limits happen to be.
2016-04-14 13:39:14 +02:00
Eelco Dolstra
96515b0c0d Fix std::atomic_flag build failure
http://hydra.nixos.org/build/34453794
2016-04-14 12:50:01 +02:00
Eelco Dolstra
867967265b Remove manifest support
Manifests have been superseded by binary caches for years. This also
gets rid of nix-pull, nix-generate-patches and bsdiff/bspatch.
2016-04-11 16:20:15 +02:00
Eelco Dolstra
8cffec8485 Remove failed build caching
This feature was implemented for Hydra, but Hydra no longer uses it.
2016-04-08 18:19:04 +02:00
Eelco Dolstra
f398949b40 Make LocalStore thread-safe
Necessary for multi-threaded commands like "nix verify-paths".
2016-04-08 18:07:13 +02:00
Eelco Dolstra
05fbc606fc nix verify-paths: Add ‘--sigs-needed <N>’ flag
This specifies the number of distinct signatures required to consider
each path "trusted".

Also renamed ‘--no-sigs’ to ‘--no-trust’ for the flag that disables
verifying whether a path is trusted (since a path can also be trusted
if it has no signatures, but was built locally).
2016-04-07 15:16:57 +02:00
Eelco Dolstra
6b2ae52808 Use secret-key-files for verifying 2016-04-07 15:16:57 +02:00
Eelco Dolstra
e39999ed48 Sign locally-built paths
Locally-built paths are now signed automatically using the secret keys
specified by the ‘secret-key-files’ option.
2016-04-07 15:16:57 +02:00
Eelco Dolstra
dc82160164 Fix "tar: This does not look like a tar archive" with fetchTarball / -I http://
The 304 Not Modified was not handled correctly, so the empty result
from the conditional request would overwrite the previous tarball.
2016-04-06 16:57:20 +02:00
Eelco Dolstra
b654381eb3 Add "nix sign-paths" command
E.g.

  $ nix sign-paths -k ./secret -r $(type -p geeqie)

signs geeqie and all its dependencies using the key in ./secret.
2016-04-05 16:39:29 +02:00
Eelco Dolstra
d0f5719c2a Add "nix copy-sigs" command
This imports signatures from one store into another. E.g.

  $ nix copy-sigs -r /run/current-system -s https://cache.nixos.org/
  imported 595 signatures
2016-04-05 16:39:23 +02:00
Eelco Dolstra
80da7a6375 Probably fix SQLITE_BUSY errors 2016-04-05 15:29:56 +02:00
Eelco Dolstra
37a337bcec throwSQLiteError(): Check for SIGINT so we don't loop forever 2016-04-04 15:08:08 +02:00
Eelco Dolstra
9cee600c88 LocalStore: Keep track of ultimately trusted paths
These are content-addressed paths or outputs of locally performed
builds. They are trusted even if they don't have signatures, so "nix
verify-paths" won't complain about them.
2016-03-30 17:35:48 +02:00
Eelco Dolstra
3d119f0a3b Improve the SQLite wrapper API
In particular, this eliminates a bunch of boilerplate code.
2016-03-30 15:50:45 +02:00
Eelco Dolstra
d9c5e3bbf0 Factour out SQLite handling 2016-03-30 13:27:25 +02:00
Eelco Dolstra
2ae43ced9a Turn retrying SQLite transactions into a higher-order function 2016-03-30 12:04:27 +02:00
Eelco Dolstra
88541569a2 HttpBinaryCacheStore: Treat 403 errors as 404 2016-03-30 11:17:51 +02:00
Eelco Dolstra
de88004a9d CurlDownloader: Fix HTTP error processing 2016-03-30 11:17:35 +02:00
Eelco Dolstra
784ee35c80 Add "nix verify-paths" command
Unlike "nix-store --verify-path", this command verifies signatures in
addition to store path contents, is multi-threaded (especially useful
when verifying binary caches), and has a progress indicator.

Example use:

$ nix verify-paths --store https://cache.nixos.org -r $(type -p thunderbird)
...
[17/132 checked] checking ‘/nix/store/rawakphadqrqxr6zri2rmnxh03gqkrl3-autogen-5.18.6’
2016-03-29 16:37:16 +02:00
Eelco Dolstra
374198ad6d Move signature support from NarInfo to ValidPathInfo 2016-03-24 11:41:00 +01:00
Eelco Dolstra
11525377e1 Typos 2016-03-24 11:27:58 +01:00
Eelco Dolstra
8b7839b608 HttpBinaryCacheStore: Make thread-safe 2016-03-24 11:10:05 +01:00
Eelco Dolstra
c7d44bad00 Drop support for daemon socket path >= 108 characters
Doing a chdir() is a bad idea in multi-threaded programs, leading to
failures such as

  error: cannot connect to daemon at ‘/nix/var/nix/daemon-socket/socket’: No such file or directory

Since Linux doesn't have a connectat() syscall like FreeBSD, there is
no way we can support this in a race-free way.
2016-03-23 17:16:16 +01:00
Eelco Dolstra
056b3ecfa4 LocalStoreAccessor::stat: Handle ENOTDIR
Closes https://github.com/NixOS/hydra/pull/286.
2016-03-23 11:17:46 +01:00
Eelco Dolstra
bb1034316d Don't overload dumpPath() 2016-03-22 14:21:45 +01:00
Eelco Dolstra
712b616a84 Move signatures from NarInfo to ValidPathInfo
This allows queryPathInfo() to return signatures.
2016-03-21 18:05:47 +01:00
Eelco Dolstra
1c5f73f529 Add Store::dumpPath() method
This allows applying nix-store --verify-path to binary cache stores:

  NIX_REMOTE=https://cache.nixos.org nix-store --verify-path /nix/store/s5c7...
2016-03-21 17:55:57 +01:00
Eelco Dolstra
87295b9844 Drop support for upgrading from Nix <= 0.12 2016-03-21 15:09:03 +01:00
Eelco Dolstra
02654f782f Fix Darwin build
http://hydra.nixos.org/build/33279996
2016-03-15 12:11:27 +01:00
Eelco Dolstra
00a75b1cd2 Fix another mismatched tag
http://hydra.nixos.org/build/33279570
2016-03-14 15:32:34 +01:00
Eelco Dolstra
e7c76f7274 BinaryCacheStore::isValidPath(): Use .narinfo cache
If a path is in the .narinfo cache, obviously it's valid.
2016-03-14 12:33:06 +01:00
Eelco Dolstra
b4e0335d4d Add option binary-cache-secret-key-file for signing binary caches 2016-03-04 17:45:22 +01:00
Eelco Dolstra
af7cdb1096 BinaryCacheStore: Remove publicKeyFile argument
The public key can be derived from the secret key, so there's no need
for the user to supply it separately.
2016-03-04 17:45:22 +01:00
Eelco Dolstra
42bc395b63 Eliminate some large string copying 2016-03-04 16:49:56 +01:00
Eelco Dolstra
7c9d7a253c Merge branch 'new-cli' 2016-03-04 15:21:23 +01:00
Eelco Dolstra
76f1ba4f3b Add file missing from 201b48de60 2016-03-03 18:03:34 +01:00
Nathan Zadoks
fe2be8f016 build.cc: fs.h doesn't appear to be necessary anymore 2016-03-03 14:11:00 +01:00
Eelco Dolstra
5a8455c85e Provide function required by Hydra 2016-03-02 18:21:48 +01:00
Eelco Dolstra
201b48de60 Add an HTTP binary cache store
Allowing stuff like

  NIX_REMOTE=https://cache.nixos.org nix-store -qR /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1

or

  NIX_REMOTE=https://cache.nixos.org nix-store --export /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 | nix-store --import
2016-03-02 15:46:07 +01:00
Eelco Dolstra
0402b6398d Eliminate local-binary-cache-store.hh 2016-03-02 15:46:07 +01:00
Eelco Dolstra
68a5414982 Make store implementations pluggable
This for instance allows hydra-queue-runner to add the S3 backend
at runtime.
2016-03-02 15:46:07 +01:00
Eelco Dolstra
0a62d9b3d7 Remove bad assertion 2016-03-02 15:46:07 +01:00
Eelco Dolstra
cca4a8dc1a importPaths(): Optionally add NARs to binary cache accessor
This enables an optimisation in hydra-queue-runner, preventing a
download of a NAR it just uploaded to the cache when reading files
like hydra-build-products.
2016-03-02 15:46:07 +01:00
Eelco Dolstra
e9c50064b5 Add an HTTP binary cache store
Allowing stuff like

  NIX_REMOTE=https://cache.nixos.org nix-store -qR /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1

or

  NIX_REMOTE=https://cache.nixos.org nix-store --export /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 | nix-store --import
2016-02-29 18:15:20 +01:00
Eelco Dolstra
6170bb474b Eliminate local-binary-cache-store.hh 2016-02-29 16:14:39 +01:00
Eelco Dolstra
0b907321cc Make store implementations pluggable
This for instance allows hydra-queue-runner to add the S3 backend
at runtime.
2016-02-29 16:11:11 +01:00
Eelco Dolstra
d0344dd2c1 Remove bad assertion 2016-02-26 21:43:59 +01:00
Eelco Dolstra
bcc9943cee importPaths(): Optionally add NARs to binary cache accessor
This enables an optimisation in hydra-queue-runner, preventing a
download of a NAR it just uploaded to the cache when reading files
like hydra-build-products.
2016-02-26 15:20:10 +01:00
Eelco Dolstra
1042c10fd0 Add NAR / Store accessor abstraction
This is primary to allow hydra-queue-runner to extract files like
"nix-support/hydra-build-products" from NARs in binary caches.
2016-02-25 17:43:19 +01:00
Eelco Dolstra
152b1d6bf9 deletePath(): Succeed if path doesn't exist
Also makes it robust against concurrent deletions.
2016-02-24 17:44:55 +01:00
Eelco Dolstra
28e7e29abd Eliminate reserveSpace flag 2016-02-24 17:44:55 +01:00
Eelco Dolstra
5a64e66268 LocalStore::addTextToStore(): Don't read the path we just wrote
This eliminates some unnecessary (presumably cached) I/O.
2016-02-24 17:11:31 +01:00
Eelco Dolstra
9ccbd55c5b BinaryCacheStore: Implement addToStore()
So now you can do

  $ NIX_REMOTE=file:///tmp/binary-cache nix-instantiate '<nixpkgs>' -A hello

and lots of other operations.
2016-02-24 16:52:28 +01:00
Eelco Dolstra
30e9d01516 openStoreAt(): Initialise the binary cache 2016-02-24 16:52:07 +01:00
Eelco Dolstra
ba0a81d14f Show a proper error message for unimplemented binary cache operations 2016-02-24 14:57:30 +01:00
Eelco Dolstra
263187a2ec Move BinaryCacheStore / LocalBinaryCacheStore from Hydra
So you can now do:

  $ NIX_REMOTE=file:///tmp/binary-cache nix-store -qR /nix/store/...
2016-02-24 14:48:16 +01:00
Eelco Dolstra
5f862658c3 Remove bad daemon connections from the pool
This is necessary for long-running processes like hydra-queue-runner:
if a nix-daemon worker is killed, we need to stop reusing that
connection.
2016-02-24 11:39:56 +01:00
Eelco Dolstra
d5626bf4c1 Pool<T>: Allow a maximum pool size 2016-02-23 16:40:16 +01:00
Eelco Dolstra
e292144d46 RemoteStore: Make thread-safe
This allows a RemoteStore object to be used safely from multiple
threads concurrently. It will make multiple daemon connections if
necessary.

Note: pool.hh and sync.hh have been copied from the Hydra source tree.
2016-02-23 15:00:59 +01:00
Eelco Dolstra
d361901bfe curl: Set CURLOPT_NOSIGNAL
Otherwise using curl is not safe in multi-threaded applications
because it installs a SIGALRM handler.
2016-02-22 15:51:14 +01:00
Eelco Dolstra
b49d323ce2 Fix build without sodium
http://hydra.nixos.org/build/32085949
2016-02-17 12:42:14 +01:00
Eelco Dolstra
c4d22997f3 Add C++ functions for .narinfo processing / signing
This is currently only used by the Hydra queue runner rework, but like
eff5021eaa it presumably will be useful
for the C++ rewrite of nix-push and
download-from-binary-cache. (@shlevy)
2016-02-16 16:38:44 +01:00
Eelco Dolstra
5ac27053e9 Rename ValidPathInfo::hash -> narHash for consistency 2016-02-16 11:49:12 +01:00
Eelco Dolstra
c8f4d89a34 Expose the export magic value and move LocalStore::queryReferences to Store 2016-02-15 15:01:26 +01:00
Eelco Dolstra
d089372565 Add function to extract hash part of a store path 2016-02-15 15:01:26 +01:00
Eelco Dolstra
ae4a3cfa03 Move addPermRoot into Store 2016-02-11 16:14:42 +01:00
Eelco Dolstra
fd205fb6f8 ref<T>: Add cast operator 2016-02-11 15:32:48 +01:00
Eelco Dolstra
fa7cd5369b StoreAPI -> Store
Calling a class an API is a bit redundant...
2016-02-04 14:48:42 +01:00
Eelco Dolstra
c10c61449f Eliminate the "store" global variable
Also, move a few free-standing functions into StoreAPI and Derivation.

Also, introduce a non-nullable smart pointer, ref<T>, which is just a
wrapper around std::shared_ptr ensuring that the pointer is never
null. (For reference-counted values, this is better than passing a
"T&", because the latter doesn't maintain the refcount. Usually, the
caller will have a shared_ptr keeping the value alive, but that's not
always the case, e.g., when passing a reference to a std::thread via
std::bind.)
2016-02-04 14:28:26 +01:00
Eelco Dolstra
d367b8e787 Add build mode to compute fixed-output derivation hashes
For example,

  $ nix-build --hash -A nix-repl.src

will build the fixed-output derivation nix-repl.src (a fetchFromGitHub
call), but instead of *verifying* the hash given in the Nix
expression, it prints out the resulting hash, and then moves the
result to its content-addressed location in the Nix store. E.g

  build produced path ‘/nix/store/504a4k6zi69dq0yjc0bm12pa65bccxam-nix-repl-8a2f5f0607540ffe56b56d52db544373e1efb980-src’ with sha256 hash ‘0cjablz01i0g9smnavhf86imwx1f9mnh5flax75i615ml71gsr88’

The goal of this is to make all nix-prefetch-* scripts unnecessary: we
can just let Nix run the real thing (i.e., the corresponding fetch*
derivation).

Another example:

  $ nix-build --hash -E 'with import <nixpkgs> {}; fetchgit { url = "https://github.com/NixOS/nix.git"; sha256 = "ffffffffffffffffffffffffffffffffffffffffffffffffffff"; }'
  ...
  git revision is 9e7c1a4bbd
  ...
  build produced path ‘/nix/store/gmsnh9i7x4mb7pyd2ns7n3c9l90jfsi1-nix’ with sha256 hash ‘1188xb621diw89n25rifqg9lxnzpz7nj5bfh4i1y3dnis0dmc0zp’

(Having to specify a fake sha256 hash is a bit annoying...)
2016-01-31 12:18:09 +01:00
Eelco Dolstra
9e7c1a4bbd Use the daemon when we don't have write access to the Nix database 2016-01-31 10:28:48 +01:00
Eelco Dolstra
786046cf13 --option build-repeat: Keep the differing output if -K is given
Similar to 00903fa799. Regardless of -K,
we now also print which output differs.
2016-01-12 18:26:24 +01:00
Eelco Dolstra
8906eda2f9 Canonicalize gids to 0
Previously files in the Nix store were owned by root or by nixbld,
depending on whether they were created by a substituter or by a
builder. This doesn't matter much, but causes spurious diffoscope
differences. So use root everywhere.
2016-01-12 17:27:40 +01:00
Eelco Dolstra
00903fa799 --check: Keep the differing output if -K is given
This makes it easier to investigate the non-determinism, e.g.

  $ nix-build pkgs/stdenv/linux -A stage1.pkgs.zlib --check -K
  error: derivation ‘/nix/store/l54i8wlw22656i4pk05c52ngv9rpl39q-zlib-1.2.8.drv’ may not be deterministic: output ‘/nix/store/11a27shh6n2ivi4a7s964i65ql80cf27-zlib-1.2.8’ differs from ‘/nix/store/11a27shh6n2ivi4a7s964i65ql80cf27-zlib-1.2.8-check’

  $ diffoscope /nix/store/11a27shh6n2ivi4a7s964i65ql80cf27-zlib-1.2.8 /nix/store/11a27shh6n2ivi4a7s964i65ql80cf27-zlib-1.2.8-check
  ...
  ├── lib/libz.a
  │   ├── metadata
  │   │ @@ -1,15 +1,15 @@
  │   │ -rw-r--r-- 30001/30000   3096 Jan 12 15:20 2016 adler32.o
  ...
  │   │ +rw-r--r-- 30001/30000   3096 Jan 12 15:28 2016 adler32.o
  ...
2016-01-12 16:44:26 +01:00
Eelco Dolstra
0cad1f8049 --check: Fix "failed to produce output path"
This occured when sandbox building is disabled, at least one output
exists, and at least one other output does not.
2016-01-12 15:13:47 +01:00
Eelco Dolstra
1c57ab8b31 --check: Fix assertion failure when some outputs are missing
E.g.

  $ nix-build pkgs/stdenv/linux/ -A stage1.pkgs.perl --check
  nix-store: src/libstore/build.cc:1323: void nix::DerivationGoal::tryToBuild(): Assertion `buildMode != bmCheck || validPaths.size() == drv->outputs.size()' failed.

when perl.out exists but perl.man doesn't. The fix is to only check
the outputs that exist. Note that "nix-build -A stage1.pkgs.all
--check" will still give a (proper) error in this case.
2016-01-12 14:54:39 +01:00
Eelco Dolstra
458711e4ee Fix "Bad address" executing build hook
This was observed in the deb_debian7x86_64 build:
http://hydra.nixos.org/build/29973215

Calling c_str() on a temporary should be fine because the temporary
shouldn't be destroyed until after the execl() call, but who knows...
2016-01-07 15:10:14 +01:00
Eelco Dolstra
9aac1861f7 Fix some signedness warnings 2016-01-07 14:37:39 +01:00
Eelco Dolstra
02a66b3fd7 nix-store --repair-path: Rebuild if there is no substituter 2016-01-06 22:07:59 +01:00
Eelco Dolstra
caaaff3954 Fix --repair failure on multiple-output derivations
If repair found a corrupted/missing path that depended on a
multiple-output derivation, and some of the outputs of the latter were
not present, it failed with a message like

  error: path ‘/nix/store/cnfn9d5fjys1y93cz9shld2xwaibd7nn-bash-4.3-p42-doc’ is not valid
2016-01-06 21:49:32 +01:00
Eelco Dolstra
743e310046 Fix non-Darwin build 2016-01-05 13:31:15 +01:00
Tuomas Tynkkynen
f770b9e6c5 libstore: mmap() returns MAP_FAILED, not NULL on failure 2016-01-05 13:26:35 +01:00
Eelco Dolstra
71a93a5f0e Don't allow sandbox profile except in relaxed mode
This makes Darwin consistent with Linux: Nix expressions can't break
out of the sandbox unless relaxed sandbox mode is enabled.

For the normal sandbox mode this will require fixing #759 however.
2016-01-04 20:01:13 +01:00
Eelco Dolstra
77ad443bd1 ~PathLocks(): Handle exceptions
Otherwise, since the call to write a "d" character to the lock file
can fail with ENOSPC, we can get an unhandled exception resulting in a
call to terminate().
2016-01-04 11:34:36 +01:00
Eelco Dolstra
b8258a4475 Fix regression in passAsFile
Caused by 8063fc497a. If tmpDir !=
tmpDirInSandbox (typically when there are multiple concurrent builds
with the same name), the *Path attribute would not point to an
existing file. This caused Nixpkgs' writeTextFile to write an empty
file. In particular this showed up as hanging VM builds (because it
would run an empty run-nixos-vm script and then wait for it to finish
booting).
2015-12-29 15:28:20 +01:00
Eelco Dolstra
52120123a5 Handle /tmp being a symlink
Hopefully fixes Darwin sandbox regression introduced in
8063fc497a.
2015-12-22 17:16:17 +01:00
Eelco Dolstra
f696af0fab Fix bad error message in Darwin chroots 2015-12-22 17:05:29 +01:00
Eelco Dolstra
8f67325a7c Build sandbox support etc. unconditionally on Linux
Also, use "#if __APPLE__" instead of "#if SANDBOX_ENABLED" to prevent
ambiguity.
2015-12-10 11:47:17 +01:00
Bjørn Forsman
65bd82d42a Clarify error message for hash mismatches (again)
This is arguably nitpicky, but I think this new formulation is even
clearer. My thinking is that it's easier to comprehend when the
calculated hash value is displayed close to the output path. (I think it
is somewhat similar to eliminating double negatives in logic
statements.)

The formulation is inspired / copied from the OpenEmbedded build tool,
bitbake.
2015-12-08 19:50:25 +01:00
Ludovic Courtès
d1e3bf01bc daemon: Add 'buildMode' parameter to 'buildPaths' RPC 2015-12-02 18:14:49 +01:00
Eelco Dolstra
8063fc497a Use deterministic $TMPDIR in sandbox
Rather than using $<host-TMPDIR>/nix-build-<drvname>-<number>, the
temporary directory is now always /tmp/nix-build-<drvname>-0. This
improves bitwise-exact reproducibility for builds that store $TMPDIR
in their build output. (Of course, those should still be fixed...)
2015-12-02 15:04:00 +01:00
Eelco Dolstra
0ab4d905e7 Merge branch 'p/sandbox-rename-minimal' of https://github.com/vcunat/nix 2015-11-25 14:53:42 +01:00
Eelco Dolstra
a0f0733413 Fix build failure introduced by #704
Also, make the FreeBSD checks conditional on FreeBSD.
2015-11-25 14:41:19 +01:00
Eelco Dolstra
cad40adce5 Merge pull request #704 from ysangkok/freebsd-support
FreeBSD support with knowledge about Linux emulation
2015-11-24 19:24:21 +01:00
Jude Taylor
279fa8f618 reintroduce host deps in tandem with sandbox profiles 2015-11-21 15:57:06 -08:00
Shea Levy
e0bd114e09 Revert "remove sandbox-defaults.sb"
As discussed in NixOS/nixpkgs#11001, we still need some of the old
sandbox mechanism.

This reverts commit d760c2638c.
2015-11-21 16:40:24 -05:00
Jude Taylor
b9b7bb1806 re-fix permissions for GHC 2015-11-19 16:06:21 -08:00
Shea Levy
5deb7fbdfb Merge branch 'sandbox-profiles' of git://github.com/pikajude/nix
Temporarily allow derivations to describe their full sandbox profile.
This will be eventually scaled back to a more secure setup, see the
discussion at #695
2015-11-19 17:44:11 -05:00
Peter Simons
6ad10591ce src/libstore/build.cc: clarify error message for hash mismatches
Nix reports a hash mismatch saying:

  output path ‘foo’ should have sha256 hash ‘abc’, instead has ‘xyz’

That message is slightly ambiguous and some people read that statement
to mean the exact opposite of what it is supposed to mean. After this
patch, the message will be:

  Nix expects output path ‘foo’ to have sha256 hash ‘abc’, instead it has ‘xyz’
2015-11-19 12:42:37 +01:00
janus
8a74a125bc FreeBSD can build Linux 32-bit binaries 2015-11-17 14:16:08 +00:00
Shea Levy
58d2fac91d AutoDelete: Add default constructor with deletion disabled 2015-11-16 05:53:10 -05:00
Shea Levy
4390142315 Use AutoDelete for sandbox profile file 2015-11-15 06:08:50 -05:00
Jude Taylor
bd09a4c967 simplify build.cc using modern C++ features 2015-11-14 14:11:03 -08:00
Jude Taylor
4876bb012e simplify build permissions 2015-11-14 14:11:03 -08:00
Jude Taylor
d760c2638c remove sandbox-defaults.sb 2015-11-14 14:11:03 -08:00
Jude Taylor
8a7f0dfd68 use per-derivation sandbox profiles 2015-11-14 14:10:43 -08:00
Vladimír Čunát
b39622a487 rename chroot to sandbox (fixes #656, close #682)
- rename options but leav old names as lower-priority aliases,
  also "-dirs" -> "-paths" to get closer to the meaning
- update docs to reflect the new names (old aliases are not documented),
  including a new file with release notes
- tests need an update after corresponding changes to nixpkgs
- __noChroot is left as it is (after discussion on the PR)
2015-11-10 22:32:51 +01:00
Eelco Dolstra
8fdd156a65 Add option to verify build determinism
Passing "--option build-repeat <N>" will cause every build to be
repeated N times. If the build output differs between any round, the
build is rejected, and the output paths are not registered as
valid. This is primarily useful to verify build determinism. (We
already had a --check option to repeat a previously succeeded
build. However, with --check, non-deterministic builds are registered
in the DB. Preventing that is useful for Hydra to ensure that
non-deterministic builds don't end up getting published at all.)
2015-11-09 23:16:24 +01:00
Eelco Dolstra
96c2ebf004 Revert "Allow using /bin and /usr/bin as impure prefixes on non-darwin by default"
This reverts commit 79ca503332. Ouch,
never noticed this. We definitely don't want to allow builds to have
arbitrary access to /bin and /usr/bin, because then they can (for
instance) bring in a bunch of setuid programs. Also, we shouldn't be
encouraging the use of impurities in the default configuration.
2015-11-09 21:28:38 +01:00
Eelco Dolstra
4384bbd2e1 optimizePath(): Detect some .links corruption
If automatic store optimisation is enabled, and a hard-linked file in
the store gets corrupted, then the corresponding .links entry will
also be corrupted. In that case, trying to repair with --repair or
--repair-path won't work, because the new "good" file will be replaced
by a hard link to the corrupted file. We can catch most of these cases
by doing a sanity-check on the file sizes.
2015-11-09 21:28:25 +01:00
Jude Taylor
c10ef2d134 fix syntax error 2015-11-03 09:03:32 -08:00
Jude Taylor
3b52e54e31 darwin: allow reading system locale and zoneinfo 2015-11-03 08:43:09 -08:00
Jude Taylor
d6bd51ffbe allow reading ICU data 2015-10-31 02:52:42 -07:00
Jude Taylor
95c66cc950 add special devices to sandbox-defaults 2015-10-30 20:32:11 -07:00
Eelco Dolstra
1f735a3440 <nix/fetchurl.nix>: Support xz-compressed NARs 2015-10-30 12:34:30 +01:00
Eelco Dolstra
dae5dc7ade <nix/fetchurl.nix>: Support downloading and unpacking NARs
This removes the need to have multiple downloads in the stdenv
bootstrap process (like a separate busybox binary for Linux, or
curl/mkdir/sh/bzip2 for Darwin). Now all those files can be combined
into a single NAR.
2015-10-30 11:27:47 +01:00
Eelco Dolstra
5c28943e8f int2String() -> std::to_string() 2015-10-29 13:26:55 +01:00
Jude Taylor
9155fbd386 use nixDataDir instead of appending /share to PREFIX 2015-10-21 14:44:44 -07:00
Jude Taylor
992cda1b11 clarifying comment 2015-10-21 14:39:16 -07:00
Jude Taylor
76f3ba42fd move preBuildHook defaulting to globals.cc 2015-10-21 14:39:08 -07:00
Jude Taylor
2dc8e19a17 restore old DEFAULT_ALLOWED_IMPURE_PREFIXES 2015-10-21 14:38:49 -07:00
Jude Taylor
ff6953cb03 Add resolve-system-dependencies.pl 2015-10-21 12:38:52 -07:00
Jude Taylor
15c7a79ed3 remove usr paths from allowed inputs 2015-10-21 12:38:52 -07:00
Jude Taylor
5bf4085000 allow access to SystemVersion for python builders 2015-10-21 12:38:52 -07:00
Jude Taylor
6a8cee19b3 fix line reading in preBuildHook 2015-10-21 12:38:52 -07:00
Jude Taylor
1fc0fe1baf remove sandbox defaults into a new file 2015-10-21 12:38:52 -07:00
Jude Taylor
d4cac051f7 restore allowed impure prefixes 2015-10-21 12:38:52 -07:00
Jude Taylor
4b8c71b4ef remove an unneeded default impure-dep 2015-10-21 12:38:52 -07:00
Jude Taylor
e770f941d6 make sandbox builds more permissive 2015-10-21 12:38:52 -07:00
Jude Taylor
1ea2aa9e99 add a few more permissions 2015-10-21 12:38:52 -07:00
Eelco Dolstra
f5b32ff47d Allow builtin fetchurl regardless of the derivation's system attribute 2015-10-21 15:14:42 +02:00
Eelco Dolstra
01615b5f63 Show progress indicator for builtin fetchurl 2015-10-21 15:14:42 +02:00
Eelco Dolstra
5db358d4d7 Disable TLS verification for builtin fetchurl
This makes it consistent with the Nixpkgs fetchurl and makes it work
in chroots. We don't need verification because the hash of the result
is checked anyway.
2015-10-21 15:14:42 +02:00
Eelco Dolstra
357d31b339 Fix segfault in builtin fetchurl
The stack allocated for the builder was way too small (32 KB). This is
sufficient for normal derivations, because they just do some setup and
then exec() the actual builder. But for the fetchurl builtin
derivation it's not enough. Also, allocating the stack on the caller's
stack was fishy business.
2015-10-21 15:14:42 +02:00
Tuomas Tynkkynen
2075ec83e3 Allow building ARMv6 stuff on ARMv7
This allows building a Raspberry Pi image on modern, faster boards.
2015-10-08 13:32:28 +02:00
Eelco Dolstra
14f7e1fc42 Revert to CURLOPT_PROGRESSFUNCTION
CURLOPT_XFERINFOFUNCTION isn't widely supported yet.

http://hydra.nixos.org/build/26679495
2015-10-08 13:15:21 +02:00
Eelco Dolstra
21ecd106ed Show progress during downloads 2015-10-07 17:31:50 +02:00
Manuel Jacob
cbc177ca01 Use pkg-config-provided LDFLAGS for libsqlite3 and libcurl.
Previously, pkg-config was already queried for libsqlite3's and
libcurl's link flags.  However they were not used, but hardcoded
instead.  This commit replaces the hardcoded LDFLAGS by the ones
provided by pkg-config in a similar pattern as already used for
libsodium.
2015-10-06 22:28:30 +02:00
Eelco Dolstra
1abda8e173 nix-store --serve: Implement log size limit 2015-10-06 17:33:30 +02:00
Eelco Dolstra
4ba6bc184c Shut up clang warnings 2015-09-18 01:22:35 +02:00
Eelco Dolstra
71a5161365 Implement buildDerivation() via the daemon 2015-09-03 12:56:59 +02:00
Eelco Dolstra
73332fd342 Filter build-chroot-dirs entries that conflict with derivation outputs
Fixes https://github.com/NixOS/nixpkgs/issues/9504.

Note that this means we may have a non-functional /bin/sh in the
chroot while rebuilding Bash or one of its dependencies. Ideally those
packages don't rely on /bin/sh though.
2015-09-02 14:54:12 +02:00
Eelco Dolstra
d711fe8f5e Remove unused variable 2015-09-02 14:46:04 +02:00
Eelco Dolstra
e12cf82782 Prevent .chroot from being GC'ed when using LocalStore::buildDerivation()
Fixes #616.
2015-08-24 11:13:31 +02:00
Eelco Dolstra
eadb86f447 nix-collect-garbage: Revive --max-freed
Fixes #609.
2015-08-21 13:57:53 +02:00
Manolis Ragkousis
26221e44eb Remove unneeded HAVE_UNSHARE.
* src/libstore/build.cc (CHROOT_ENABLED): Remove HAVE_UNSHARE.
2015-08-19 14:11:15 +03:00
Eelco Dolstra
34dfbd9394 Don't include <iostream> before config.h
This breaks the build on 32-bit systems.

http://hydra.nixos.org/build/24373658
2015-08-04 11:12:31 +02:00
Eelco Dolstra
30d19a2bdc Handle debug messages from runChild()
Turns out that "nix-build -vvv" with chroots enabled has been broken
for some time, because some debug message got interpreted as an error.
2015-08-03 18:04:32 +02:00
Eelco Dolstra
f3dda728a4 Remove unnecessary parentheses 2015-07-26 12:07:14 +02:00
Eelco Dolstra
16c9935fa9 Fix fetchurl of executable file
Pointed out by @cstrahan, thanks!
2015-07-23 22:25:04 +02:00
Eelco Dolstra
1993b10d11 Fix Darwin build
Turns out getgrouplist() is not POSIX.

http://hydra.nixos.org/build/23881243
2015-07-21 14:45:24 +02:00
Eelco Dolstra
0a2bee307b Make <nix/fetchurl.nix> a builtin builder
This ensures that 1) the derivation doesn't change when Nix changes;
2) the derivation closure doesn't contain Nix and its dependencies; 3)
we don't have to rely on ugly chroot hacks.
2015-07-20 04:38:46 +02:00
Eelco Dolstra
eda2f36c2a Provide more detailed info about build status to hydra-queue-runner
In particular, hydra-queue-runner can now distinguish between remote
build / substitution / already-valid. For instance, if a path already
existed on the remote side, we don't want to store a log file.
2015-07-20 03:20:03 +02:00
Eelco Dolstra
ccf31dbc25 nix-copy-closure: Add -v flag
And make exportPath() less spammy by default.
2015-07-20 01:52:07 +02:00
Eelco Dolstra
b3491c781c More cleanup 2015-07-20 01:16:16 +02:00
Eelco Dolstra
6bd2c7bb38 OCD: foreach -> C++11 ranged for 2015-07-17 20:13:56 +02:00
Eelco Dolstra
1511aa9f48 Allow remote builds without sending the derivation closure
Previously, to build a derivation remotely, we had to copy the entire
closure of the .drv file to the remote machine, even though we only
need the top-level derivation. This is very wasteful: the closure can
contain thousands of store paths, and in some Hydra use cases, include
source paths that are very large (e.g. Git/Mercurial checkouts).

So now there is a new operation, StoreAPI::buildDerivation(), that
performs a build from an in-memory representation of a derivation
(BasicDerivation) rather than from a on-disk .drv file. The only files
that need to be in the Nix store are the sources of the derivation
(drv.inputSrcs), and the needed output paths of the dependencies (as
described by drv.inputDrvs). "nix-store --serve" exposes this
interface.

Note that this is a privileged operation, because you can construct a
derivation that builds any store path whatsoever. Fixing this will
require changing the hashing scheme (i.e., the output paths should be
computed from the other fields in BasicDerivation, allowing them to be
verified without access to other derivations). However, this would be
quite nice because it would allow .drv-free building (e.g. "nix-env
-i" wouldn't have to write any .drv files to disk).

Fixes #173.
2015-07-17 17:57:40 +02:00
Eelco Dolstra
dd48c06bb6 Typo 2015-07-02 00:30:16 +02:00
Ludovic Courtès
9aed117395 Preserve supplementary groups of build users
The following patch is an attempt to address this bug (see
<http://bugs.gnu.org/18994>) by preserving the supplementary groups of
build users in the build environment.

In practice, I would expect that supplementary groups would contain only
one or two groups: the build users group, and possibly the “kvm” group.

[Changed &at(0) to data() and removed tabs - Eelco]
2015-07-01 14:57:48 +02:00
Eelco Dolstra
ff4de4cb27 GC: Handle ENOSPC creating/moving to the trash directory
Issue #564.
2015-06-30 21:41:26 +02:00
Eelco Dolstra
2bc9c84327 Use posix_fallocate to create /nix/var/nix/db/reserved 2015-06-22 15:54:55 +02:00
Eelco Dolstra
ba63ec6f39 Make /nix/var/nix/db/reserved bigger
Issue #564.
2015-06-22 15:47:40 +02:00
Eelco Dolstra
b65875f859 Export outputPaths function
This is useful for the new hydra-queue-runner.
2015-06-10 16:17:06 +02:00
Eelco Dolstra
7c4501886d Use std::vector::data() 2015-06-09 10:54:46 +02:00
Eelco Dolstra
b64988bb35 Allow substitutes for builds that have preferLocalBuild set
Not substituting builds with "preferLocalBuild = true" was a bad idea,
because it didn't take the cost of dependencies into account. For
instance, if we can't substitute a fetchgit call, then we have to
download/build git and all its dependencies.

Partially reverts 5558652709 and adds a
new derivation attribute "allowSubstitutes" to specify whether a
derivation may be substituted.
2015-06-04 16:30:22 +02:00
Eelco Dolstra
07d7e7df84 Chown files created for passAsFile
Nixpkgs' writeTextAsFile does this:

  mv "$textPath" "$n"

Since $textPath was owned by root, if $textPath is on the same
filesystem as $n, $n will be owned as root. As a result, the build
result was rejected as having suspicious ownership.

http://hydra.nixos.org/build/22836807
2015-06-04 14:07:43 +02:00
Ludovic Courtès
b755752f76 Add a ‘verifyStore’ RPC
Hello!

The patch below adds a ‘verifyStore’ RPC with the same signature as the
current LocalStore::verifyStore method.

Thanks,
Ludo’.

>From aef46c03ca77eb6344f4892672eb6d9d06432041 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Mon, 1 Jun 2015 23:17:10 +0200
Subject: [PATCH] Add a 'verifyStore' remote procedure call.
2015-06-02 02:14:24 +02:00
Eelco Dolstra
4ca5a9dcfd nix-collect-garbage: Don't call nix-env
Also, make sure --delete-older-than doesn't delete the current
generation.
2015-05-21 16:28:30 +02:00
Eelco Dolstra
22b1a8d43f Move profiles.{cc,hh} to libstore 2015-05-21 15:42:59 +02:00
Rok Garbas
000de699e9 cygwin: explicitly include required c headers 2015-05-13 09:37:12 +02:00
Shea Levy
71083f9e5e Don't try to map /bin/sh to a store path on non-Linux 2015-05-12 16:36:15 -04:00
Shea Levy
4d652875bd Add the pre-build hook.
This hook can be used to set system-specific per-derivation build
settings that don't fit into the derivation model and are too complex or
volatile to be hard-coded into nix. Currently, the pre-build hook can
only add chroot dirs/files through the interface, but it also has full
access to the chroot root.

The specific use case for this is systems where the operating system ABI
is more complex than just the kernel-support system calls. For example,
on OS X there is a set of system-provided frameworks that can reliably
be accessed by any program linked to them, no matter the version the
program is running on. Unfortunately, those frameworks do not
necessarily live in the same locations on each version of OS X, nor do
their dependencies, and thus nix needs to know the specific version of
OS X currently running in order to make those frameworks available. The
pre-build hook is a perfect mechanism for doing just that.
2015-04-18 16:56:02 -04:00
Shea Levy
fd6774e285 Revert "Add the pre-build hook."
Going to reimplement differently.

This reverts commit 1e4a4a2e9f.
2015-04-18 14:59:58 -04:00
Eelco Dolstra
035aeb9547 Fix using restricted mode with chroots 2015-04-16 18:46:17 +02:00
Shea Levy
1e4a4a2e9f Add the pre-build hook.
This hook can be used to set system specific per-derivation build
settings that don't fit into the derivation model and are too complex or
volatile to be hard-coded into nix. Currently, the pre-build hook can
only add chroot dirs/files.

The specific use case for this is systems where the operating system ABI
is more complex than just the kernel-supported system calls. For
example, on OS X there is a set of system-provided frameworks that can
reliably be accessed by any program linked to them, no matter the
version the program is running on. Unfortunately, those frameworks do
not necessarily live in the same locations on each version of OS X, nor
do their dependencies, and thus nix needs to know the specific version
of OS X currently running in order to make those frameworks available.
The pre-build hook is a perfect mechanism for doing just that.
2015-04-12 12:56:38 -04:00
Eelco Dolstra
c1f04fae35 Implement a TTL on cached fetchurl/fetchTarball results
This is because we don't want to do HTTP requests on every evaluation,
even though we can prevent a full redownload via the cached ETag. The
default is one hour.
2015-04-09 11:55:36 +02:00
Eelco Dolstra
1711679ea5 Revert /nix/store permission back to 01775
This broke NixOS VM tests.

Mostly reverts 27b7b94923,
5ce50cd99e,
afa433e58c.
2015-04-07 13:21:26 +02:00
Eelco Dolstra
afa433e58c Chroot builds: Provide world-readable /nix/store
This was causing NixOS VM tests to fail mysteriously since
5ce50cd99e. Nscd could (sometimes) no
longer read /etc/hosts:

open("/etc/hosts", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)

Probably there was some wacky interaction between the guest kernel and
the 9pfs implementation in QEMU.
2015-04-02 16:59:40 +02:00
Eelco Dolstra
7ea6ecf855 addToStore(): Take explicit name argument 2015-03-25 17:06:12 +01:00
Eelco Dolstra
5114a07d95 Improve setting the default chroot dirs 2015-03-24 11:57:46 +01:00
Eelco Dolstra
fd89f97be9 Add the closure of store paths to the chroot
Thus, for example, to get /bin/sh in a chroot, you only need to
specify /bin/sh=${pkgs.bash}/bin/sh in build-chroot-dirs. The
dependencies of sh will be added automatically.
2015-03-24 11:52:34 +01:00
Eelco Dolstra
5ce50cd99e Tighten permissions on chroot directories 2015-03-24 11:35:53 +01:00
Daniel Hahler
e659978ced Fix typos: s/the the/the/ 2015-03-06 16:43:22 +01:00
Eelco Dolstra
75ede65e3d Don't use vfork() before clone()
I'm seeing hangs in Glibc's setxid_mark_thread() again. This is
probably because the use of an intermediate process to make clone()
safe from a multi-threaded program (see
524f89f139) is defeated by the use of
vfork(), since the intermediate process will have a copy of Glibc's
threading data structures due to the vfork(). So use a regular fork()
again.
2015-03-04 15:13:10 +01:00
Shea Levy
c2699be93b Merge branch 'allow-system-library' of git://github.com/copumpkin/nix
Make the default impure prefix include all of /System/Library
2015-03-03 15:01:09 -05:00
Dan Peebles
336c4270c6 Make the default impure prefix (not actual allowed impurities!) include all of /System/Library, since we also want PrivateFrameworks from there and (briefly) TextEncodings, and who knows what else. Yay infectious impurities? 2015-03-02 23:01:24 -05:00
Dan Peebles
66d612f1da Allow local networking in the darwin sandbox to appease tests 2015-03-02 22:55:42 -05:00
Eelco Dolstra
885bebf13b More graceful fallback for chroots on Linux < 2.13 2015-02-23 15:54:31 +01:00
Eelco Dolstra
99897f6979 Use chroots for all derivations
If ‘build-use-chroot’ is set to ‘true’, fixed-output derivations are
now also chrooted. However, unlike normal derivations, they don't get
a private network namespace, so they can still access the
network. Also, the use of the ‘__noChroot’ derivation attribute is
no longer allowed.

Setting ‘build-use-chroot’ to ‘relaxed’ gives the old behaviour.
2015-02-23 15:54:31 +01:00
Eelco Dolstra
bd91064150 Use $<attr>Path instead of $<attr> for passAsFile 2015-02-17 16:42:54 +01:00
Eelco Dolstra
a70d275f3d Allow passing attributes via files instead of environment variables
Closes #473.
2015-02-17 14:42:15 +01:00
Harald van Dijk
5451b8db9d Use pivot_root in addition to chroot when possible
chroot only changes the process root directory, not the mount namespace root
directory, and it is well-known that any process with chroot capability can
break out of a chroot "jail". By using pivot_root as well, and unmounting the
original mount namespace root directory, breaking out becomes impossible.

Non-root processes typically have no ability to use chroot() anyway, but they
can gain that capability through the use of clone() or unshare(). For security
reasons, these syscalls are limited in functionality when used inside a normal
chroot environment. Using pivot_root() this way does allow those syscalls to be
put to their full use.
2015-02-16 12:18:19 +01:00
Shea Levy
d66d9e8425 Require linux 3.13 or later for chroot
Fixes #453
2015-02-04 18:15:56 +01:00
Daniel Peebles
f46e329a13 Make inputs writeable in the sandbox (builds still can’t actually write due to user permissions) 2015-01-18 23:25:29 -05:00
Shea Levy
79ca503332 Allow using /bin and /usr/bin as impure prefixes on non-darwin by default
These directories are generally world-readable anyway, and give us the two
most common linux impurities (env and sh)
2015-01-13 15:41:46 +01:00
Eelco Dolstra
fcf57aad27 SysError -> Error 2015-01-13 11:17:56 +01:00
Eelco Dolstra
100961e370 Don't resolve symlinks while checking __impureHostDeps
Since these come from untrusted users, we shouldn't do any I/O on them
before we've checked that they're in an allowed prefix.
2015-01-13 11:16:32 +01:00
Daniel Peebles
f1151a3373 Add basic Apple sandbox support 2015-01-12 12:00:01 +01:00
Eelco Dolstra
57d64d24aa Doh^2 2015-01-08 16:59:22 +01:00
Eelco Dolstra
57b82256b0 Doh 2015-01-08 16:49:31 +01:00
Eelco Dolstra
27b7b94923 Set /nix/store permission to 1737
I.e., not readable to the nixbld group. This improves purity a bit for
non-chroot builds, because it prevents a builder from enumerating
store paths (i.e. it can only access paths it knows about).
2015-01-08 16:39:07 +01:00
Eelco Dolstra
df05f49dcd Fix building on Darwin
Fixes #433.
2015-01-06 10:49:44 +01:00
Shea Levy
3d97b8d1e7 LocalStore initialization: Don't die if build-users-group doesn't exist
See NixOS/nixpkgs@9245516
2014-12-29 14:40:13 +01:00
Eelco Dolstra
bd0f362d2f Revive running builds in a PID namespace 2014-12-23 17:25:06 +01:00
Eelco Dolstra
4e0607369e Pedantry 2014-12-14 01:51:14 +01:00
Eelco Dolstra
8bdff8c100 Merge branch 'cygwin-master' of https://github.com/ternaris/nix 2014-12-14 01:49:14 +01:00
Eelco Dolstra
fa2063ca35 Better error message 2014-12-13 16:54:40 +01:00
Eelco Dolstra
b77037b8fd Silence some warnings on GCC 4.9 2014-12-12 17:14:28 +01:00
Eelco Dolstra
46f3eb6fdd Shut up a Valgrind warning 2014-12-12 15:10:02 +01:00
Eelco Dolstra
f52b6c944e Fix some memory leaks 2014-12-12 15:01:16 +01:00
Eelco Dolstra
28f22b4653 Ensure we're writing to stderr in the builder
http://hydra.nixos.org/build/17862041
2014-12-12 14:35:44 +01:00
Eelco Dolstra
4acca1a5b9 Remove chatty message
This broke building with "-vv", because the builder is not allowed to
write to stderr at this point.
2014-12-12 13:43:59 +01:00
Eelco Dolstra
ad790022fd Doh 2014-12-12 13:41:00 +01:00
Eelco Dolstra
851b47bd7d Don't do vfork in conjunction with setuid 2014-12-10 18:01:01 +01:00
Eelco Dolstra
b5ed5b6e66 Rename function 2014-12-10 17:25:38 +01:00
Eelco Dolstra
e529823635 Don't wait for PID -1
The pid field can be -1 if forking the substituter process failed.
2014-12-10 13:53:04 +01:00
Marko Durkovic
629e8da3aa Explicitly include required C headers 2014-12-09 13:00:59 +01:00
Eelco Dolstra
b7b6e3ddec Build derivations in a more predictable order
Derivations are now built in order of derivation name, so a package
named "aardvark" is built before "baboon".

Fixes #399.
2014-11-24 16:50:46 +01:00
Eelco Dolstra
9e3389c337 Don't create unnecessary substitution goals for derivations 2014-11-24 16:50:46 +01:00
Eelco Dolstra
8299aaf079 Disable vacuuming the DB after garbage collection
Especially in WAL mode on a highly loaded machine, this is not a good
idea because it results in a WAL file of approximately the same size
ad the database, which apparently cannot be deleted while anybody is
accessing it.
2014-11-19 18:14:24 +01:00
Eelco Dolstra
1256ab3b44 Clean up temp roots in a more C++ way 2014-11-19 17:07:29 +01:00
Eelco Dolstra
35aad73bb6 Fix message 2014-11-17 01:00:39 +01:00
Eelco Dolstra
8cfe939b0f Don't use ADDR_LIMIT_3GB
This gives 32-bit builds on x86_64-linux more memory.
2014-11-14 14:16:20 +01:00
Eelco Dolstra
bab8d9b52a Make ~DerivationGoal more reliable 2014-11-12 11:35:53 +01:00
Eelco Dolstra
06a86aee15 nix-store --gc: Don't warn about missing manifests directory 2014-11-04 10:41:29 +01:00
Eelco Dolstra
e389f4ea55 Improve error message if the daemon worker fails to start 2014-10-31 09:36:09 +01:00
Eelco Dolstra
11aef17a77 Remove comments claiming we use a private PID namespace
This is no longer the case since
524f89f139.
2014-10-29 15:49:34 +01:00
Eelco Dolstra
2a4dbcff74 Improved error message when encountering unsupported file types
Fixes #269.
2014-10-14 10:51:19 +02:00
Eelco Dolstra
3f8576a6ab Remove some duplicate code 2014-10-03 22:37:51 +02:00
Eelco Dolstra
d4a71ec3bf Update spec file
http://hydra.nixos.org/build/14344391
2014-09-18 15:42:01 +02:00
Eelco Dolstra
8be9990cdb Install some pkgconfig files 2014-09-18 12:00:40 +02:00
Eelco Dolstra
d98bfcbf81 On Linux, disable address space randomization 2014-09-17 17:21:13 +02:00
Eelco Dolstra
d37d012774 Settings: Add bool get() 2014-09-17 15:18:13 +02:00
Ludovic Courtès
8fb8c26b6d Add an 'optimiseStore' remote procedure call. 2014-09-01 23:53:26 +02:00
Eelco Dolstra
b72e93bca8 Add disallowedReferences / disallowedRequisites
For the "stdenv accidentally referring to bootstrap-tools", it seems
easier to specify the path that we don't want to depend on, e.g.

  disallowedRequisites = [ bootstrapTools ];
2014-08-28 18:57:13 +02:00
Gergely Risko
fd61069a42 Introduce allowedRequisites feature 2014-08-28 18:23:55 +02:00
Joel Taylor
b224ac1520 fix disappearing bash arguments 2014-08-21 23:17:50 +02:00
Eelco Dolstra
524f89f139 Use unshare() instead of clone()
It turns out that using clone() to start a child process is unsafe in
a multithreaded program. It can cause the initialisation of a build
child process to hang in setgroups(), as seen several times in the
build farm:

The reason is that Glibc thinks that the other threads of the parent
exist in the child, so in setxid_mark_thread() it tries to get a futex
that has been acquired by another thread just before the clone(). With
fork(), Glibc runs pthread_atfork() handlers that take care of this
(in particular, __reclaim_stacks()). But clone() doesn't do that.

Fortunately, we can use fork()+unshare() instead of clone() to set up
private namespaces.

See also https://www.mail-archive.com/lxc-devel@lists.linuxcontainers.org/msg03434.html.
2014-08-21 14:08:09 +02:00
Eelco Dolstra
11849a320e Use proper quotes everywhere 2014-08-20 18:03:48 +02:00
Eelco Dolstra
373fad75e1 Add some color 2014-08-20 16:50:17 +02:00
Eelco Dolstra
954188af27 Filter Nix-specific ANSI escape sequences from stderr
The Nixpkgs stdenv prints some custom escape sequences to denote
nesting and stuff like that. Most terminals (e.g. xterm, konsole)
ignore them, but some do not (e.g. xfce4-terminal). So for the benefit
of the latter, filter them out.
2014-08-20 14:30:01 +02:00
Eelco Dolstra
029424d17d Make hook shutdown more reliable 2014-08-19 17:47:30 +02:00
Eelco Dolstra
632f989b51 Doh 2014-08-18 11:35:50 +02:00
Eelco Dolstra
c160ead82f Reduce verbosity 2014-08-17 19:11:50 +02:00
Eelco Dolstra
42e9ad8fd1 Propagate remote timeouts properly 2014-08-17 19:09:03 +02:00
Eelco Dolstra
9367046fef Use regular file GC roots if possible
This makes hydra-eval-jobs create roots as regular files. See
1c208f2b7e.
2014-08-13 17:44:41 +02:00
Eelco Dolstra
98873ff7de Remove unnecessary call to addTempRoot() 2014-08-05 16:41:42 +02:00
Eelco Dolstra
f02c52e3d4 Doh 2014-08-05 10:19:57 +02:00
Eelco Dolstra
988bf59421 Move some options out of globals 2014-08-04 18:13:14 +02:00
Eelco Dolstra
daccd68999 Refactor 2014-08-04 18:02:29 +02:00
Eelco Dolstra
d5a076c36f Add option ‘build-extra-chroot-dirs’
This is useful for extending (rather than overriding) the default set
of chroot paths.
2014-08-04 18:00:00 +02:00
Eelco Dolstra
4d73e2e893 Get rid of "killing <pid>" message for unused build hooks 2014-08-04 17:27:45 +02:00
Eelco Dolstra
bb45092f72 Make chroot builds easier to set up
By default, we now include /bin/sh as a bind-mount of bash.
2014-08-04 17:09:26 +02:00
Eelco Dolstra
51485dcda2 Remove ugly hack for detecting build environment setup errors 2014-08-01 19:38:21 +02:00
Eelco Dolstra
4c4b219c07 Call commonChildInit() before doing chroot init
This ensures that daemon clients see error messages from the chroot
setup.
2014-08-01 19:29:03 +02:00
Eelco Dolstra
e0d7d0e45c findRoots(): Prevent a call to lstat()
This means that getting the roots from /nix/var/nix/.../hydra-roots
doesn't need any I/O other than reading the directory.
2014-08-01 17:20:25 +02:00
Eelco Dolstra
daf3f2c11f Make readDirectory() return inode / file type 2014-08-01 17:14:28 +02:00
Eelco Dolstra
1c208f2b7e Allow regular files as GC roots
If a root is a regular file, then its name must denote a store
path. For instance, the existence of the file

  /nix/var/nix/gcroots/per-user/eelco/hydra-roots/wzc3cy1wwwd6d0dgxpa77ijr1yp50s6v-libxml2-2.7.7

would cause

  /nix/store/wzc3cy1wwwd6d0dgxpa77ijr1yp50s6v-libxml2-2.7.7

to be a root.

This is useful because it involves less I/O (no need for a readlink()
call) and takes up less disk space (the symlink target typically takes
up a full disk block, while directory entries are packed more
efficiently). This is particularly important for hydra.nixos.org,
which has hundreds of thousands of roots, and where reading the roots
can take 25 minutes.
2014-08-01 17:14:28 +02:00
Eelco Dolstra
50dc1f5b71 Restore default SIGPIPE handler before invoking ‘man’
Fixes NixOS/nixpkgs#3410.
2014-07-31 10:31:17 +02:00
Eelco Dolstra
67937907ca nix-daemon: Pass on the user's $SSH_AUTH_SOCK to the SSH substituter 2014-07-25 18:02:48 +02:00
Eelco Dolstra
71a20d4d95 Change the default for use-ssh-substituter to ‘true’
Now you only have to pass ‘--option ssh-substituter-hosts
nix-ssh@bla’ to enable SSH substitution.
2014-07-25 12:57:12 +02:00
Eelco Dolstra
d3c61d83be Remove some dead code 2014-07-24 00:00:53 +02:00
Eelco Dolstra
5989966ed3 Remove dead code 2014-07-23 14:46:28 +02:00
Eelco Dolstra
ee3c5d7916 Revert old useBuildHook behaviour 2014-07-19 02:25:47 +02:00
Eelco Dolstra
049c0eb49c nix-daemon: Add trusted-users and allowed-users options
‘trusted-users’ is a list of users and groups that have elevated
rights, such as the ability to specify binary caches. It defaults to
‘root’. A typical value would be ‘@wheel’ to specify all users in the
wheel group.

‘allowed-users’ is a list of users and groups that are allowed to
connect to the daemon. It defaults to ‘*’. A typical value would be
‘@users’ to specify the ‘users’ group.
2014-07-17 16:57:07 +02:00
Eelco Dolstra
276a40b31f Handle case collisions on case-insensitive systems
When running NixOps under Mac OS X, we need to be able to import store
paths built on Linux into the local Nix store. However, HFS+ is
usually case-insensitive, so if there are directories with file names
that differ only in case, then importing will fail.

The solution is to add a suffix ("~nix~case~hack~<integer>") to
colliding files. For instance, if we have a directory containing
xt_CONNMARK.h and xt_connmark.h, then the latter will be renamed to
"xt_connmark.h~nix~case~hack~1". If a store path is dumped as a NAR,
the suffixes are removed. Thus, importing and exporting via a
case-insensitive Nix store is round-tripping. So when NixOps calls
nix-copy-closure to copy the path to a Linux machine, you get the
original file names back.

Closes #119.
2014-07-16 16:02:05 +02:00
Eelco Dolstra
fa13d3f4f3 build-remote.pl: Fix building multiple output derivations
We were importing paths without sorting them topologically, leading to
"path is not valid" errors.

See e.g. http://hydra.nixos.org/build/12451761
2014-07-14 12:19:27 +02:00
Eelco Dolstra
a5c6347ff0 build-remote.pl: Use ‘nix-store --serve’ on the remote side
This makes things more efficient (we don't need to use an SSH master
connection, and we only start a single remote process) and gets rid of
locking issues (the remote nix-store process will keep inputs and
outputs locked as long as they're needed).

It also makes it more or less secure to connect directly to the root
account on the build machine, using a forced command
(e.g. ‘command="nix-store --serve --write"’). This bypasses the Nix
daemon and is therefore more efficient.

Also, don't call nix-store to import the output paths.
2014-07-11 16:22:24 +02:00
Eelco Dolstra
e196eecbe6 Allow $NIX_BUILD_HOOK to be relative to Nix libexec directory 2014-07-11 13:55:06 +02:00
Eelco Dolstra
edbfe2232e Replace message "importing path <...>" with "exporting path <...>"
This causes nix-copy-closure to show what it's doing before rather
than after.
2014-07-10 21:30:22 +02:00
Eelco Dolstra
43b64f5038 Remove tabs 2014-07-10 17:32:21 +02:00
Eelco Dolstra
8e9140cfde Refactoring: Move all fork handling into a higher-order function
C++11 lambdas ftw.
2014-07-10 16:58:09 +02:00
Eelco Dolstra
7911e4c27a Remove maybeVfork 2014-07-10 13:35:44 +02:00
Eelco Dolstra
8504e7d604 allow-arbitrary-code-during-evaluation -> allow-unsafe-native-code-during-evaluation 2014-06-27 11:20:16 +02:00
Shea Levy
d62f46e500 Only add the importNative primop if the allow-arbitrary-code-during-evaluation option is true (default false) 2014-06-24 10:50:03 -04:00
Eelco Dolstra
b1beed97a0 Report daemon OOM better
When copying a large path causes the daemon to run out of memory, you
now get:

  error: Nix daemon out of memory

instead of:

  error: writing to file: Broken pipe
2014-06-10 13:45:50 +02:00
Eelco Dolstra
829af22759 Print a warning when loading a large path into memory
I.e. if you have a derivation with

  src = ./huge-directory;

you'll get a warning that this is not a good idea.
2014-06-10 13:30:09 +02:00
Eelco Dolstra
54a34119f3 Use std::unordered_set 2014-05-26 17:53:17 +02:00
Eelco Dolstra
9f9080e2c0 nix-store -l: Fetch build logs from the Internet
If a build log is not available locally, then ‘nix-store -l’ will now
try to download it from the servers listed in the ‘log-servers’ option
in nix.conf. For instance, if you have:

  log-servers = http://hydra.nixos.org/log

then it will try to get logs from http://hydra.nixos.org/log/<base
name of the store path>. So you can do things like:

  $ nix-store -l $(which xterm)

and get a log even if xterm wasn't built locally.
2014-05-21 17:19:36 +02:00
Eelco Dolstra
8d5f472f2c lvlInfo -> lvlTalkative 2014-05-15 11:37:44 +02:00
Eelco Dolstra
84813af5b9 nix-store --optimise: Remove bogus statistics 2014-05-15 11:33:46 +02:00
Eelco Dolstra
690adeb03d Remove tab 2014-05-15 11:19:16 +02:00
Wout Mertens
3b9ea8452f Shortcut store files before lstat
readdir() already returns the inode numbers, so we don't need to call
lstat to know if a file was already linked or not.
2014-05-15 09:02:22 +02:00
Wout Mertens
d73ffc552f Use the inodes given by readdir directly 2014-05-14 22:52:10 +02:00
Wout Mertens
e974f20c98 Preload linked hashes to speed up lookups
By preloading all inodes in the /nix/store/.links directory, we can
quickly determine of a hardlinked file was already linked to the hashed
links.
This is tolerant of removing the .links directory, it will simply
recalculate all hashes in the store.
2014-05-13 23:10:06 +02:00
wmertens
a84f503d86 Shortcut already-hardlinked files
If an inode in the Nix store has more than 1 link, it probably means that it was linked into .links/ by us. If so, skip.

There's a possibility that something else hardlinked the file, so it would be nice to be able to override this.

Also, by looking at the number of hardlinks for each of the files in .links/, you can get deduplication numbers and space savings.
2014-05-10 15:53:01 +02:00
Eelco Dolstra
696f960dee Set up directories and permissions for multi-user install automatically
This automatically creates /nix/var/nix/profiles/per-user and sets the
permissions/ownership on /nix/store to 1775 and root:nixbld.
2014-05-02 14:31:15 +02:00
Eelco Dolstra
de4cdd0d47 Set build-max-jobs to the number of available cores by default
More zero configuration.
2014-05-02 12:51:43 +02:00
Eelco Dolstra
ada3e3fa15 When running as root, use build users by default
This removes the need to have a nix.conf, and prevents people from
accidentally running Nix builds as root.
2014-05-02 12:46:03 +02:00
Eelco Dolstra
dfa2f77d2e If a .drv cannot be parsed, show its path
Otherwise you just get ‘expected string `Derive(['’ which isn't very helpful.
2014-04-08 19:24:29 +02:00
Danny Wilson
ae6b631dc4 Fix compile errors on Illumos 2014-04-03 17:39:57 +02:00
Ludovic Courtès
e7720aa10a Make sure /dev/pts/ptmx is world-writable
While running Python 3’s test suite, we noticed that on some systems
/dev/pts/ptmx is created with permissions 0 (that’s the case with my
Nixpkgs-originating 3.0.43 kernel, but someone with a Debian-originating
3.10-3 reported not having this problem.)

There’s still the problem that people without
CONFIG_DEVPTS_MULTIPLE_INSTANCES=y are screwed (as noted in build.cc),
but I don’t see how we could work around it.
2014-04-03 13:42:45 +02:00
Eelco Dolstra
1c2550a2ae boost::shared_ptr -> std::shared_ptr 2014-03-30 00:49:23 +01:00
Eelco Dolstra
acb8facbbc Fix potential segfault in waitForInput()
Since the addition of build-max-log-size, a call to
handleChildOutput() can result in cancellation of a goal.  This
invalidated the "j" iterator in the waitForInput() loop, even though
it was still used afterwards.  Likewise for the maxSilentTime
handling.

Probably fixes #231.  At least it gets rid of the valgrind warnings.
2014-03-29 22:14:11 +01:00
Eelco Dolstra
90dc50b07c restoreSIGPIPE(): Fill in sa_mask
Issue #231.
2014-03-29 20:20:14 +01:00
Ludovic Courtès
24cb65efc3 Make /dev/kvm optional
The daemon now creates /dev deterministically (thanks!).  However, it
expects /dev/kvm to be present.

The patch below restricts that requirement (1) to Linux-based systems,
and (2) to systems where /dev/kvm already exists.

I’m not sure about the way to handle (2).  We could special-case
/dev/kvm and create it (instead of bind-mounting it) in the chroot, so
it’s always available; however, it wouldn’t help much since most likely,
if /dev/kvm missing, then KVM support is missing.
2014-03-21 17:27:19 +01:00
Eelco Dolstra
1017bd68ea Set up a private /dev/pts in the chroot 2014-02-27 23:35:23 +01:00
Eelco Dolstra
3fd01b171a Set up a minimal /dev in chroots
Not bind-mounting the /dev from the host also solves the problem with
/dev/shm being a symlink to something not in the chroot.
2014-02-27 23:17:53 +01:00
Eelco Dolstra
29cde917fe Fix deadlock in SubstitutionGoal
We were relying on SubstitutionGoal's destructor releasing the lock,
but if a goal is a top-level goal, the destructor won't run in a
timely manner since its reference count won't drop to zero.  So
release it explicitly.

Fixes #178.
2014-02-27 13:34:13 +01:00
Eelco Dolstra
dcaea042fc Only start download-via-ssh if it's enabled 2014-02-26 13:41:03 +01:00
Shea Levy
df5de9dfd7 Add use-ssh-substituter setting.
It defaults to false and can be overridden by RemoteStore.

Untested currently, just quickly put this together
2014-02-26 13:32:47 +01:00
Eelco Dolstra
8129cf33d9 Slight simplification 2014-02-18 10:46:30 +01:00
Eelco Dolstra
1aa19b24b2 Add a flag ‘--check’ to verify build determinism
The flag ‘--check’ to ‘nix-store -r’ or ‘nix-build’ will cause Nix to
redo the build of a derivation whose output paths are already valid.
If the new output differs from the original output, an error is
printed.  This makes it easier to test if a build is deterministic.
(Obviously this cannot catch all sources of non-determinism, but it
catches the most common one, namely the current time.)

For example:

  $ nix-build '<nixpkgs>' -A patchelf
  ...
  $ nix-build '<nixpkgs>' -A patchelf --check
  error: derivation `/nix/store/1ipvxsdnbhl1rw6siz6x92s7sc8nwkkb-patchelf-0.6' may not be deterministic: hash mismatch in output `/nix/store/4pc1dmw5xkwmc6q3gdc9i5nbjl4dkjpp-patchelf-0.6.drv'

The --check build fails if not all outputs are valid.  Thus the first
call to nix-build is necessary to ensure that all outputs are valid.

The current outputs are left untouched: the new outputs are either put
in a chroot or diverted to a different location in the store using
hash rewriting.
2014-02-18 01:01:14 +01:00
Eelco Dolstra
b6def5b542 Make --repair work on Darwin
Mac OS X doesn't allow renaming a read-only directory.

http://hydra.nixos.org/build/9113895
2014-02-17 23:09:48 +01:00
Eelco Dolstra
dfbcb7c403 Refactoring 2014-02-17 23:04:52 +01:00
Eelco Dolstra
71adb090f0 When using a build hook, only copy missing paths 2014-02-17 22:58:21 +01:00
Eelco Dolstra
69fe6c58fa Move some code around
In particular, do replacing of valid paths during repair later.  This
prevents us from replacing a valid path after the build fails.
2014-02-17 22:25:15 +01:00
Eelco Dolstra
00d30496ca Heuristically detect if a build may have failed due to a full disk
This will allow Hydra to detect that a build should not be marked as
"permanently failed", allowing it to be retried later.
2014-02-17 14:15:56 +01:00
Eelco Dolstra
dba33d4018 Minor style fixes 2014-02-14 11:48:42 +01:00
Shea Levy
38c3beac1a Move StoreApi::serve into opServe
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-10 06:52:48 -05:00
Shea Levy
1614603165 Pass in params by const ref
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-10 06:49:37 -05:00
Shea Levy
64e23d0a38 Add download-via-ssh substituter
This substituter connects to a remote host, runs nix-store --serve
there, and then forwards substituter commands on to the remote host and
sends their results to the calling program. The ssh-substituter-hosts
option can be specified as a list of hosts to try.

This is an initial implementation and, while it works, it has some
limitations:

* Only the first host is used
* There is no caching of query results (all queries are sent to the
  remote machine)
* There is no informative output (such as progress bars)
* Some failure modes may cause unhelpful error messages
* There is no concept of trusted-ssh-substituter-hosts

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:33 -05:00
Shea Levy
5671188eb2 nix-store --serve: Flush out after every loop
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:33 -05:00
Shea Levy
73874629ef nix-store --serve: Use dump instead of export
Also remove signing support

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:33 -05:00
Shea Levy
188f96500b nix-store --serve: Don't fail if asked for info about non-valid path
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:33 -05:00
Shea Levy
9488447594 nix-store --serve: Don't loop forever
nix-store --export takes a tmproot, which can only release by exiting.
Substituters don't currently work in a way that could take advantage of
the looping, anyway.

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:32 -05:00
Shea Levy
3a38d0f356 Add the nix-store --serve command
This is essentially the substituter API operating on the local store,
which will be used by the ssh substituter. It runs in a loop rather than
just taking one command so that in the future nix will be able to keep
one connection open for multiple instances of the substituter.

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:32 -05:00
Eelco Dolstra
d210cdc435 Fix assertion failure in ‘nix-store --load-db’
Namely:

  nix-store: derivations.cc:242: nix::Hash nix::hashDerivationModulo(nix::StoreAPI&, nix::Derivation): Assertion `store.isValidPath(i->first)' failed.

This happened because of the derivation output correctness check being
applied before the references of a derivation are valid.
2014-02-03 22:36:07 +01:00
Eelco Dolstra
d6582c04c1 Give a friendly error message if the DB directory is not writable
Previously we would say "error: setting synchronous mode: unable to
open database file" which isn't very helpful.
2014-02-01 16:57:38 +01:00
Eelco Dolstra
6ef32bddc1 Fix "make dist" 2014-02-01 14:38:12 +01:00
Eelco Dolstra
0c6d62cf27 Remove Automakefiles 2014-02-01 13:54:38 +01:00
Eelco Dolstra
16e7d69209 Update Makefile variable names 2014-02-01 13:54:38 +01:00
Eelco Dolstra
e0234dfddc Rename Makefile -> local.mk 2014-01-30 12:11:06 +01:00
Eelco Dolstra
94f9c14d52 Fix some clang warnings 2014-01-21 18:29:55 +01:00
Eelco Dolstra
81628a6ccc Merge branch 'master' into make
Conflicts:
	src/libexpr/eval.cc
2014-01-21 15:30:01 +01:00
Eelco Dolstra
b1db599dd0 Generate schema.sql.hh 2014-01-09 22:10:35 +01:00
Eelco Dolstra
b4c684e0f9 Update Makefiles 2014-01-09 16:53:47 +01:00
Eelco Dolstra
11cb4bfb25 Fix checking of NAR hashes
*headdesk*
*headdesk*
*headdesk*

So since commit 22144afa8d, Nix hasn't
actually checked whether the content of a downloaded NAR matches the
hash specified in the manifest / NAR info file.  Urghhh...
2014-01-08 17:35:49 +01:00
Domen Kožar
485f4740ee wording 2014-01-06 11:38:24 +01:00
Eelco Dolstra
a6add93d73 Garbage collector: Release locks on temporary root files
This allows processes waiting for such locks to proceed during the
trash deletion phase of the garbage collector.
2013-12-10 13:13:59 +01:00
Eelco Dolstra
c5b8fe3151 Print a trace message if a build fails due to the platform being unknown 2013-12-05 14:31:57 -05:00
Eelco Dolstra
7ce0e05ad8 Rename Makefile.new -> Makefile 2013-11-25 15:25:13 +00:00
Eelco Dolstra
2bd0fcc966 Use libnix as a prefix for all Nix libraries
In particular "libutil" was always a problem because it collides with
Glibc's libutil.  Even if we install into $(libdir)/nix, the linker
sometimes got confused (e.g. if a program links against libstore but
not libutil, then ld would report undefined symbols in libstore
because it was looking at Glibc's libutil).
2013-11-23 23:53:41 +00:00
Eelco Dolstra
90dfb37f14 Allow (dynamic) libraries to depend on other libraries 2013-11-23 20:11:02 +00:00
Eelco Dolstra
6b5f89f2cf Drop the dependency on Automake 2013-11-22 19:30:24 +00:00
Eelco Dolstra
754c05ed6c Rename $(here) to $(d) for brevity, and remove trailing slash 2013-11-22 16:45:52 +00:00
Eelco Dolstra
62e35cc3a8 Add ‘make dist’ support 2013-11-22 16:42:25 +01:00
Eelco Dolstra
b8e9efc476 New non-recursive, plain Make-based build system 2013-11-22 15:54:18 +01:00
Eelco Dolstra
709cbe4e76 Include <cstring> for memset
This should fix building on Illumos.
2013-11-22 10:00:43 +00:00
Eelco Dolstra
a478e8a7bb Remove nix-setuid-helper
AFAIK, nobody uses it, it's not maintained, and it has no tests.
2013-11-14 11:57:37 +01:00
Eelco Dolstra
89e6781cc5 Make function calls show up in stack traces again
Note that adding --show-trace prevents functions calls from being
tail-recursive, so an expression that evaluates without --show-trace
may fail with a stack overflow if --show-trace is given.
2013-11-12 12:51:59 +01:00
Eelco Dolstra
c086183843 For auto roots, show the intermediate link
I.e. "nix-store -q --roots" will now show (for example)

  /home/eelco/Dev/nixpkgs/result

rather than

  /nix/var/nix/gcroots/auto/53222qsppi12s2hkap8dm2lg8xhhyk6v
2013-10-22 11:39:10 +02:00
Eelco Dolstra
a737f51fd9 Retry all SQLite operations
To deal with SQLITE_PROTOCOL, we also need to retry read-only
operations.
2013-10-16 15:58:20 +02:00
Eelco Dolstra
ff02f5336c Fix a race in registerFailedPath()
Registering the path as failed can fail if another process does the
same thing after the call to hasPathFailed().  This is extremely
unlikely though.
2013-10-16 14:55:53 +02:00
Eelco Dolstra
4bd5282573 Convenience macros for retrying a SQLite transaction 2013-10-16 14:46:35 +02:00
Eelco Dolstra
bce14d0f61 Don't wrap read-only queries in a transaction
There is no risk of getting an inconsistent result here: if the ID
returned by queryValidPathId() is deleted from the database
concurrently, subsequent queries involving that ID will simply fail
(since IDs are never reused).
2013-10-16 14:36:53 +02:00
Eelco Dolstra
7cdefdbe73 Print a distinct warning for SQLITE_PROTOCOL 2013-10-16 14:27:36 +02:00
Eelco Dolstra
d05bf04444 Treat SQLITE_PROTOCOL as SQLITE_BUSY
In the Hydra build farm we fairly regularly get SQLITE_PROTOCOL errors
(e.g., "querying path in database: locking protocol").  The docs for
this error code say that it "is returned if some other process is
messing with file locks and has violated the file locking protocol
that SQLite uses on its rollback journal files."  However, the SQLite
source code reveals that this error can also occur under high load:

  if( cnt>5 ){
    int nDelay = 1;                      /* Pause time in microseconds */
    if( cnt>100 ){
      VVA_ONLY( pWal->lockError = 1; )
      return SQLITE_PROTOCOL;
    }
    if( cnt>=10 ) nDelay = (cnt-9)*238;  /* Max delay 21ms. Total delay 996ms */
    sqlite3OsSleep(pWal->pVfs, nDelay);
  }

i.e. if certain locks cannot be not acquired, SQLite will retry a
number of times before giving up and returing SQLITE_PROTOCOL.  The
comments say:

  Circumstances that cause a RETRY should only last for the briefest
  instances of time.  No I/O or other system calls are done while the
  locks are held, so the locks should not be held for very long. But
  if we are unlucky, another process that is holding a lock might get
  paged out or take a page-fault that is time-consuming to resolve,
  during the few nanoseconds that it is holding the lock.  In that case,
  it might take longer than normal for the lock to free.
  ...
  The total delay time before giving up is less than 1 second.

On a heavily loaded machine like lucifer (the main Hydra server),
which often has dozens of processes waiting for I/O, it seems to me
that a page fault could easily take more than a second to resolve.
So, let's treat SQLITE_PROTOCOL as SQLITE_BUSY and retry the
transaction.

Issue NixOS/hydra#14.
2013-10-16 14:19:59 +02:00
Eelco Dolstra
936f9d45ba Don't apply the CPU affinity hack to nix-shell (and other Perl programs)
As discovered by Todd Veldhuizen, the shell started by nix-shell has
its affinity set to a single CPU.  This is because nix-shell connects
to the Nix daemon, which causes the affinity hack to be applied.  So
we turn this off for Perl programs.
2013-09-06 16:36:56 +02:00
Eelco Dolstra
b29d3f4aee Only show trace messages when tracing is enabled 2013-09-02 12:01:04 +02:00
Eelco Dolstra
efe4289464 Add an option to limit the log output of builders
This is mostly useful for Hydra to deal with builders that get stuck
in an infinite loop writing data to stdout/stderr.
2013-09-02 11:58:18 +02:00
Ivan Kozik
34bb806f74 Fix typos, especially those that end up in the Nix manual 2013-08-26 11:15:22 +02:00
Gergely Risko
c6c024ca6f Fix personality switching from x86_64 to i686
On Linux, Nix can build i686 packages even on x86_64 systems.  It's not
enough to recognize this situation by settings.thisSystem, we also have
to consult uname().  E.g. we can be running on a i686 Debian with an
amd64 kernel.  In that situation settings.thisSystem is i686-linux, but
we still need to change personality to i686 to make builds consistent.
2013-08-26 11:12:35 +02:00
Eelco Dolstra
a583a2bc59 Run the daemon worker on the same CPU as the client
On a system with multiple CPUs, running Nix operations through the
daemon is significantly slower than "direct" mode:

$ NIX_REMOTE= nix-instantiate '<nixos>' -A system
real    0m0.974s
user    0m0.875s
sys     0m0.088s

$ NIX_REMOTE=daemon nix-instantiate '<nixos>' -A system
real    0m2.118s
user    0m1.463s
sys     0m0.218s

The main reason seems to be that the client and the worker get moved
to a different CPU after every call to the worker.  This patch adds a
hack to lock them to the same CPU.  With this, the overhead of going
through the daemon is very small:

$ NIX_REMOTE=daemon nix-instantiate '<nixos>' -A system
real    0m1.074s
user    0m0.809s
sys     0m0.098s
2013-08-07 14:02:04 +02:00
Eelco Dolstra
a4921b8ceb Revert "build-remote.pl: Enforce timeouts locally"
This reverts commit 69b8f9980f.

The timeout should be enforced remotely.  Otherwise, if the garbage
collector is running either locally or remotely, if will block the
build or closure copying for some time.  If the garbage collector
takes too long, the build may time out, which is not what we want.
Also, on heavily loaded systems, copying large paths to and from the
remote machine can take a long time, also potentially resulting in a
timeout.
2013-07-18 12:52:29 +02:00
Shea Levy
16591eb3cc Allow bind-mounting regular files into the chroot
mount(2) with MS_BIND allows mounting a regular file on top of a regular
file, so there's no reason to only bind directories. This allows finer
control over just which files are and aren't included in the chroot
without having to build symlink trees or the like.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-07-15 16:01:33 +02:00
Eelco Dolstra
aeb810b01e Garbage collector: Don't follow symlinks arbitrarily
Only indirect roots (symlinks to symlinks to the Nix store) are now
supported.
2013-07-12 14:03:36 +02:00
Eelco Dolstra
7ccd946407 Don't set $preferLocalBuild and $requiredSystemFeatures in builders
With C++ std::map, doing a comparison like ‘map["foo"] == ...’ has the
side-effect of adding a mapping from "foo" to the empty string if
"foo" doesn't exist in the map.  So we ended up setting some
environment variables by accident.
2013-06-20 18:07:27 +00:00
Eelco Dolstra
5558652709 Don't substitute derivations that have preferLocalBuild set
In particular this means that "trivial" derivations such as writeText
are not substituted, reducing the number of GET requests to the binary
cache by about 200 on a typical NixOS configuration.
2013-06-20 19:26:31 +02:00
Eelco Dolstra
1906cce6fc Increase SQLite's auto-checkpoint interval
Common operations like instantiating a NixOS system config no longer
fitted in 8192 pages, leading to more fsyncs.  So increase this limit.
2013-06-20 14:01:33 +00:00
Eelco Dolstra
9b11165aec Disable the copy-from-other-stores substituter
This substituter basically cannot work reliably since we switched to
SQLite, since SQLite databases may need write access to open them even
just for reading (and in WAL mode they always do).
2013-06-20 12:01:33 +02:00
Eelco Dolstra
22144afa8d Don't keep "disabled" substituters running
For instance, it's pointless to keep copy-from-other-stores running if
there are no other stores, or download-using-manifests if there are no
manifests.  This also speeds things up because we don't send queries
to those substituters.
2013-06-20 11:55:15 +02:00
Eelco Dolstra
1b6ee8f4c7 Allow hard links between the outputs of a derivation 2013-06-13 17:29:56 +02:00
Eelco Dolstra
cd49ee0897 Fix a security bug in hash rewriting
Before calling dumpPath(), we have to make sure the files are owned by
the build user.  Otherwise, the build could contain a hard link to
(say) /etc/shadow, which would then be read by the daemon and
rewritten as a world-readable file.

This only affects systems that don't have hard link restrictions
enabled.
2013-06-13 17:12:24 +02:00
Eelco Dolstra
1e2c7c04b1 Fix assertion failure in canonicalisePathMetaData() after hash rewriting
The assertion in canonicalisePathMetaData() failed because the
ownership of the path already changed due to the hash rewriting.  The
solution is not to check the ownership of rewritten paths.

Issue #122.
2013-06-13 17:12:06 +02:00
Eelco Dolstra
6cc2a8f8ed computeFSClosure: Only process the missing/corrupt paths
Issue #122.
2013-06-13 16:43:20 +02:00
Eelco Dolstra
f9ff67e948 In repair mode, update the hash of rebuilt paths
Otherwise subsequent invocations of "--repair" will keep rebuilding
the path.  This only happens if the path content differs between
builds (e.g. due to timestamps).
2013-06-13 14:46:07 +02:00
Eelco Dolstra
ca70fba0bf Remove obsolete EOF checks 2013-06-07 15:10:23 +02:00
Eelco Dolstra
5959c591a0 Process stderr from substituters while doing have/info queries 2013-06-07 15:02:14 +02:00
Eelco Dolstra
c5f9d0d080 Buffer reads from the substituter
This greatly reduces the number of system calls.
2013-06-07 14:00:23 +02:00
Eelco Dolstra
b09b87321c nix-store --export: Export paths in topologically sorted order
Fixes #118.
2013-05-23 14:55:36 -04:00
Eelco Dolstra
2ee9da9e22 In trace messages, don't print the output path
This doesn't work if there is no output named "out".  Hydra didn't use
it anyway.
2013-05-10 00:24:33 +02:00
Eelco Dolstra
6eba05613a Communicate build timeouts to Hydra 2013-05-09 18:39:04 +02:00
Eelco Dolstra
69b8f9980f build-remote.pl: Enforce timeouts locally
Don't pass --timeout / --max-silent-time to the remote builder.
Instead, let the local Nix process terminate the build if it exceeds a
timeout.  The remote builder will be killed as a side-effect.  This
gives better error reporting (since the timeout message from the
remote side wasn't properly propagated) and handles non-Nix problems
like SSH hangs.
2013-05-09 17:17:17 +02:00
Eelco Dolstra
470553bd05 Don't let stderr writes in substituters cause a deadlock 2013-05-01 13:21:39 +02:00
Eelco Dolstra
0374d94437 addAdditionalRoots(): Check each path only once 2013-04-26 12:07:25 +02:00
Eelco Dolstra
772b70952f Fix --timeout
I'm not sure if it has ever worked correctly.  The line "lastWait =
after;" seems to mean that the timer was reset every time a build
produced log output.

Note that the timeout is now per build, as documented ("the maximum
number of seconds that a builder can run").
2013-04-23 18:04:59 +02:00
Eelco Dolstra
934cf2d1f4 Nix daemon: respect build timeout from the client 2013-04-23 16:59:06 +02:00
Eelco Dolstra
258897c265 Complain if /homeless-shelter exists 2013-04-04 11:16:26 +02:00
Shea Levy
cc63db1dd5 makeStoreWritable: Ask forgiveness, not permission
It is surprisingly impossible to check if a mountpoint is a bind mount
on Linux, and in my previous commit I forgot to check if /nix/store was
even a mountpoint at all. statvfs.f_flag is not populated with MS_BIND
(and even if it were, my check was wrong in the previous commit).

Luckily, the semantics of mount with MS_REMOUNT | MS_BIND make both
checks unnecessary: if /nix/store is not a mountpoint, then mount will
fail with EINVAL, and if /nix/store is not a bind-mount, then it will
not be made writable. Thus, if /nix/store is not a mountpoint, we fail
immediately (since we don't know how to make it writable), and if
/nix/store IS a mountpoint but not a bind-mount, we fail at first write
(see below for why we can't check and fail immediately).

Note that, due to what is IMO buggy behavior in Linux, calling mount
with MS_REMOUNT | MS_BIND on a non-bind readonly mount makes the
mountpoint appear writable in two places: In the sixth (but not the
10th!) column of mountinfo, and in the f_flags member of struct statfs.
All other syscalls behave as if the mount point were still readonly (at
least for Linux 3.9-rc1, but I don't think this has changed recently or
is expected to soon). My preferred semantics would be for MS_REMOUNT |
MS_BIND to fail on a non-bind mount, as it doesn't make sense to remount
a non bind-mount as a bind mount.
2013-03-25 19:00:16 +01:00
Shea Levy
2c9cf50746 makeStoreWritable: Use statvfs instead of /proc/self/mountinfo to find out if /nix/store is a read-only bind mount
/nix/store could be a read-only bind mount even if it is / in its own filesystem, so checking the 4th field in mountinfo is insufficient.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-03-25 19:00:16 +01:00
Eelco Dolstra
bdd4646338 Revert "Prevent config.h from being clobbered"
This reverts commit 28bba8c44f.
2013-03-08 01:24:59 +01:00
Eelco Dolstra
28bba8c44f Prevent config.h from being clobbered 2013-03-07 23:55:55 +01:00
Eelco Dolstra
8057a192e3 Handle systems without lutimes() or lchown() 2013-02-28 19:55:09 +01:00
Eelco Dolstra
f45c731cd7 Handle symlinks properly
Now it's really brown paper bag time...
2013-02-28 14:51:08 +01:00
Eelco Dolstra
0111ba98ea Handle hard links to other files in the output 2013-02-27 17:18:41 +01:00
Eelco Dolstra
b008674e46 Refactoring: Split off the non-recursive canonicalisePathMetaData()
Also, change the file mode before changing the owner.  This prevents a
slight time window in which a setuid binary would be setuid root.
2013-02-27 16:42:19 +01:00
Eelco Dolstra
5526a282b5 Security: Don't allow builders to change permissions on files they don't own
It turns out that in multi-user Nix, a builder may be able to do

  ln /etc/shadow $out/foo

Afterwards, canonicalisePathMetaData() will be applied to $out/foo,
causing /etc/shadow's mode to be set to 444 (readable by everybody but
writable by nobody).  That's obviously Very Bad.

Fortunately, this fails in NixOS's default configuration because
/nix/store is a bind mount, so "ln" will fail with "Invalid
cross-device link".  It also fails if hard-link restrictions are
enabled, so a workaround is:

  echo 1 > /proc/sys/fs/protected_hardlinks

The solution is to check that all files in $out are owned by the build
user.  This means that innocuous operations like "ln
${pkgs.foo}/some-file $out/" are now rejected, but that already failed
in chroot builds anyway.
2013-02-26 02:30:19 +01:00
Ludovic Courtès
3e067ac11c Add `Settings::nixDaemonSocketFile'. 2013-02-19 10:19:18 +01:00
Ludovic Courtès
5ea138dc4b Enable chroot support on old glibc versions. 2013-02-19 10:19:11 +01:00
Eelco Dolstra
5e9c3da412 Only warn about SQLite being busy once
No need to get annoying.
2013-01-23 16:45:10 +01:00
Eelco Dolstra
536c85ea49 Store build logs in /nix/var/log/nix/drvs/<XX>
...where <XX> is the first two characters of the derivation.
Otherwise /nix/var/log/nix/drvs may become so large that we run into
all sorts of weird filesystem limits/inefficiences.  For instance,
ext3/ext4 filesystems will barf with "ext4_dx_add_entry:1551:
Directory index full!" once you hit a few million files.
2013-01-17 15:47:26 +01:00
Eelco Dolstra
e42df686f3 Delete a left-over trash directory before doing a GC 2013-01-05 00:13:29 +01:00
Eelco Dolstra
92926be2fe Fix "0 store paths deleted" message 2013-01-04 15:17:19 +01:00
Eelco Dolstra
b424d29d1b Open the database after removing immutable bits 2013-01-03 13:29:17 +01:00
Eelco Dolstra
def5160b61 Clear any immutable bits in the Nix store
Doing this once makes subsequent operations like garbage collecting
more efficient since we don't have to call makeMutable() first.
2013-01-03 12:59:23 +01:00
Eelco Dolstra
299141ecbd If a substitute closure is incomplete, build dependencies, then retry the substituter
Issue #77.
2013-01-02 12:38:28 +01:00
Eelco Dolstra
1b3a78a459 Automatically fall back if the references of a substitute are not substitutable
Fixes #77.
2013-01-02 12:00:26 +01:00
Eelco Dolstra
12f9129f60 nix-build: Support talking to old daemons
Fixes #76.
2012-12-29 23:21:46 +01:00
Eelco Dolstra
b7629778ef Allow mounting a path in a different location in the chroot
Fixes #24.
2012-12-29 23:04:02 +01:00
Eelco Dolstra
2754a07ead nix-store -q --roots: Respect the gc-keep-outputs/gc-keep-derivations settings
So if a path is not garbage solely because it's reachable from a root
due to the gc-keep-outputs or gc-keep-derivations settings, ‘nix-store
-q --roots’ now shows that root.
2012-12-20 18:41:44 +01:00
Eelco Dolstra
06f62defe6 Yet another rewrite of the garbage collector
But this time it's *obviously* correct!  No more segfaults due to
infinite recursions for sure, etc.

Also, move directories to /nix/store/trash instead of renaming them to
/nix/store/bla-gc-<pid>.  Then we can just delete /nix/store/trash at
the end.
2012-12-20 17:32:15 +01:00
Eelco Dolstra
e775d4d84f If gc-keep-derivations is set, only keep the actual deriver
This prevents zillions of derivations from being kept, and fixes an
infinite recursion in the garbage collector (due to an obscure cycle
that can occur with fixed-output derivations).
2012-12-19 15:33:09 +01:00
Eelco Dolstra
fbf0b2fa45 Kill the build hook rather than shutting it down cleanly
Waiting for the hook to shut down cleanly sometimes seems to lead to
hangs.
2012-12-19 11:47:42 +01:00
Eelco Dolstra
228ea7c2f9 Revert brain fart
This reverts commit cc511fd65b.
2012-12-18 20:43:47 +01:00
Eelco Dolstra
cc511fd65b Check for potential infinite select() loops when building 2012-12-18 18:50:05 +01:00
Eelco Dolstra
772778c0ec On SQLITE_BUSY, wait a random amount of time
If all contending processes wait a fixed amount of time (100 ms),
there is a good probability that they'll just collide again.
2012-12-11 11:49:42 +01:00
Eelco Dolstra
8b8ee53bc7 Add builtin constants ‘langVersion’ and ‘nixVersion’
The integer constant ‘langVersion’ denotes the current language
version.  It gets increased every time a language feature is
added/changed/removed.  It's currently 1.

The string constant ‘nixVersion’ contains the current Nix version,
e.g. "1.2pre2980_9de6bc5".
2012-11-27 13:29:55 +01:00
Eelco Dolstra
5943f41b8b queryMissing(): Handle partially valid derivations 2012-11-26 18:25:12 +01:00
Eelco Dolstra
8d8d47abd2 Only substitute wanted outputs of a derivation
If a derivation has multiple outputs, then we only want to download
those outputs that are actuallty needed.  So if we do "nix-build -A
openssl.man", then only the "man" output should be downloaded.
Likewise if another package depends on ${openssl.man}.

The tricky part is that different derivations can depend on different
outputs of a given derivation, so we may need to restart the
corresponding derivation goal if that happens.
2012-11-26 17:15:09 +01:00
Eelco Dolstra
46a369ad95 Make "nix-build -A <derivation>.<output>" do the right thing
For example, given a derivation with outputs "out", "man" and "bin":

  $ nix-build -A pkg

produces ./result pointing to the "out" output;

  $ nix-build -A pkg.man

produces ./result-man pointing to the "man" output;

  $ nix-build -A pkg.all

produces ./result, ./result-man and ./result-bin;

  $ nix-build -A pkg.all -A pkg2

produces ./result, ./result-man, ./result-bin and ./result-2.
2012-11-26 15:39:10 +01:00
Eelco Dolstra
f794465ca8 Disable use of vfork()
vfork() is just too weird.  For instance, in this build:

  http://hydra.nixos.org/build/3330487

the value fromHook.writeSide becomes corrupted in the parent, even
though the child only reads from it.  At -O0 the problem goes away.
Probably the child is overriding some spilled temporary variable.

If I get bored I may implement using posix_spawn() instead.
2012-11-15 19:35:18 +01:00
Eelco Dolstra
8541d27fce Don't use std::cerr in a few places
Slightly scared of using std::cerr in a vforked process...
2012-11-15 15:01:02 +01:00
Eelco Dolstra
ea89df2b76 Use vfork() instead of fork() if available
Hopefully this reduces the chance of hitting ‘unable to fork: Cannot
allocate memory’ errors.  vfork() is used for everything except
starting builders.
2012-11-09 18:00:33 +01:00
Eelco Dolstra
48c19c4633 Remove definition of non-existant function 2012-11-09 17:59:41 +01:00
Eelco Dolstra
198dbe7fa1 Remove some redundant close() calls
They are unnecessary because we set the close-on-exec flag.
2012-11-09 16:58:51 +01:00
Eelco Dolstra
10dcee99ed Remove the quickExit function 2012-11-09 16:42:10 +01:00
Eelco Dolstra
4c9e3fa641 Remove a Darwin hack that should no longer be needed 2012-11-09 16:35:42 +01:00
Eelco Dolstra
f581ce0b0c Don't start copy-from-other-stores if $NIX_OTHER_STORES is unset
Slight optimisation.
2012-11-09 14:58:10 +01:00
Eelco Dolstra
91ef4d9a81 Remove unnecessary call to closeMostFDs()
We have close-on-exec on all FDs now, and there is no security risk in
passing open FDs to substituters anyway.
2012-11-09 14:43:47 +01:00
Shea Levy
d0fc615af6 canonicalizePathMetaData: Fall-back to utimes if lutimes fails due to ENOSYS 2012-11-06 11:29:59 +01:00
Eelco Dolstra
4c34d384e6 If hashes do not match, print them in base-32 for SHA-1/SHA-256
Fixes #57.
2012-10-23 18:05:50 +02:00
Eelco Dolstra
e345182055 Shorten the names of temporary build directories 2012-10-11 14:03:06 -04:00
Eelco Dolstra
904f50412c nix-store --verify: Continue on errors 2012-10-04 10:20:23 -04:00
Eelco Dolstra
90b8a34f82 Fix regular expression
http://hydra.nixos.org/build/3123177
2012-10-04 09:46:10 -04:00
Eelco Dolstra
522ecab9b8 Drop support for running nix-worker in "slave" mode
AFAIK nobody uses this, setuid binaries are evil, and there is no good
reason why people can't just run the daemon.
2012-10-03 17:30:45 -04:00
Eelco Dolstra
7586095504 Remove bin2c 2012-10-03 16:59:28 -04:00
Eelco Dolstra
2bbc4a214e nix-env: Support ‘--repair’ flag 2012-10-03 15:35:42 -04:00
Eelco Dolstra
0a7084567f Add a ‘--repair’ flag to nix-instantiate
This allows repairing corrupted derivations and other source files.
2012-10-03 15:09:18 -04:00
Eelco Dolstra
a807edfae8 Handle repairing paths that are in build-chroot-dirs 2012-10-03 11:20:16 -04:00
Eelco Dolstra
a3f205b249 When repairing a derivation, check and repair the entire output closure
If we find a corrupted path in the output closure, we rebuild the
derivation that produced that particular path.
2012-10-03 10:38:09 -04:00
Eelco Dolstra
2001895f3d Add a --repair flag to ‘nix-store -r’ to repair derivation outputs
With this flag, if any valid derivation output is missing or corrupt,
it will be recreated by using a substitute if available, or by
rebuilding the derivation.  The latter may use hash rewriting if
chroots are not available.
2012-10-02 17:13:46 -04:00
Eelco Dolstra
8e3a7bd712 nix-store --verify: Add an option ‘--repair’ to repair all missing/corrupt paths
Also, return a non-zero exit code if errors remain after
verifying/repairing.
2012-10-02 15:12:56 -04:00
Eelco Dolstra
9958bd6992 Add operation ‘nix-store --repair-path’
This operation allows fixing corrupted or accidentally deleted store
paths by redownloading them using substituters, if available.

Since the corrupted path cannot be replaced atomically, there is a
very small time window (one system call) during which neither the old
(corrupted) nor the new (repaired) contents are available.  So
repairing should be used with some care on critical packages like
Glibc.
2012-10-02 14:08:59 -04:00
Eelco Dolstra
e666e1156f Handle octal escapes in /proc/self/mountinfo 2012-09-28 21:39:30 -04:00
Eelco Dolstra
f406288cc7 Print a more descriptive error message if setting up the build environment fails 2012-09-28 21:37:02 -04:00
Eelco Dolstra
d534f137f0 Make the store writable before creating /nix/store/.links 2012-09-25 16:30:08 -04:00
Eelco Dolstra
e464b0247d Merge branch 'readonly-store' 2012-09-25 15:38:00 -04:00
Eelco Dolstra
28bf183d2d Include <sys/types.h> for off_t
Reported by "gio" on IRC.
2012-09-25 13:00:19 -04:00
Eelco Dolstra
b9c2b4d5b4 Remove setting of the immutable bit
Using the immutable bit is problematic, especially in conjunction with
store optimisation.  For instance, if the garbage collector deletes a
file, it has to clear its immutable bit, but if the file has
additional hard links, we can't set the bit afterwards because we
don't know the remaining paths.

So now that we support having the entire Nix store as a read-only
mount, we may as well drop the immutable bit.  Unfortunately, we have
to keep the code to clear the immutable bit for backwards
compatibility.
2012-09-19 16:17:54 -04:00
Eelco Dolstra
b9124a5c33 Support having /nix/store as a read-only bind mount
It turns out that the immutable bit doesn't work all that well.  A
better way is to make the entire Nix store a read-only bind mount,
i.e. by doing

  $ mount --bind /nix/store /nix/store
  $ mount -o remount,ro,bind /nix/store

(This would typically done in an early boot script, before anything
from /nix/store is used.)

Since Nix needs to be able to write to the Nix store, it now detects
if /nix/store is a read-only bind mount and then makes it writable in
a private mount namespace.
2012-09-19 15:45:29 -04:00
Eelco Dolstra
76e88871b2 Templatise tokenizeString() 2012-09-19 15:43:23 -04:00
Eelco Dolstra
00092b2d35 Keep build directory if not all expected outputs were produced
Fixes issue #123 in Nixpkgs.
2012-09-18 10:11:42 -04:00
Eelco Dolstra
8fbe96cb31 RemoteStore::connectToDaemon(): Set close-on-exec flag
This ensures that "nix-build --run-env" doesn't keep a connection to
the worker open, preventing it from exiting.
2012-09-13 14:46:37 -04:00
Eelco Dolstra
e6e495649c Vacuum the SQLite DB after running the garbage collector 2012-09-13 14:33:41 -04:00
Eelco Dolstra
2923b55f9d Delete paths in a component in topologically sorted order
The outputs of a derivation can refer to each other (even though they
cannot have cycles), so they have to be deleted in the right order.

http://hydra.nixos.org/build/3026118
2012-09-13 13:08:27 -04:00
Eelco Dolstra
6c4ac29917 Disable auto store optimisation for now
I've seen operations like "nix-store --import" take much longer on one
system.  So default to off until I've investigated this a bit further.
2012-09-13 10:28:20 -04:00
Eelco Dolstra
e56f71edaf In startBuilder(), only print the new paths we're building 2012-09-13 10:06:24 -04:00
Eelco Dolstra
4fca02077c Handle gc-keep-outputs and gc-keep-derivations both enabled
If the options gc-keep-outputs and gc-keep-derivations are both
enabled, you can get a cycle in the liveness graph.  There was a hack
to handle this, but it didn't work with multiple-output derivations,
causing the garbage collector to fail with errors like ‘error: cannot
delete path `...' because it is in use by `...'’.  The garbage
collector now handles strongly connected components in the liveness
graph as a unit and decides whether to delete all or none of the paths
in an SCC.
2012-09-12 18:49:35 -04:00
Eelco Dolstra
479e9172b3 Build hook: Pass the timeout to the remote builder
Note that this will only work if the client has a very recent Nix
version (post 15e1b2c223), otherwise the
--option flag will just be ignored.

Fixes #50.
2012-09-12 12:18:14 -04:00
Eelco Dolstra
5e2ffd0b8a Fix "non-zero padding" error
Probably it's not a good idea to pass a temporary object to
StringSource.
2012-09-11 19:09:01 -04:00
Eelco Dolstra
8cf672091e Support building a derivation if some outputs are already valid (non-chroot case)
This uses scary hash rewriting.

Fixes #21.
2012-09-11 18:39:22 -04:00
Eelco Dolstra
9dbda2b3fe Remove debug line 2012-09-11 17:05:35 -04:00
Eelco Dolstra
a2785b7391 Support building a derivation if some outputs are already valid
This handles the chroot and build hook cases, which are easy.
Supporting the non-chroot-build case will require more work (hash
rewriting!).

Issue #21.
2012-09-11 16:59:59 -04:00
Eelco Dolstra
295027f533 Include config.h before any other header
"config.h" must be included first, because otherwise the compiler
might not see the right value of _FILE_OFFSET_BITS.  We've had this
before; see 705868a8a9.  In this case,
GCC would compute a different address for ‘settings.useSubstitutes’ in
misc.cc because of the off_t in ‘settings’.

Reverts 3854fc9b42.

http://hydra.nixos.org/build/3016700
2012-09-11 14:45:42 -04:00
Eelco Dolstra
cfd968dd94 Fix stupid type error in calling std::max 2012-08-27 14:17:13 -04:00
Eelco Dolstra
e94806d030 Merge branch 'master' into no-manifests 2012-08-27 11:09:07 -04:00
Eelco Dolstra
d950cfe70b Check if MS_PRIVATE is defined
http://hydra.nixos.org/build/2955671
2012-08-20 15:55:49 -04:00
Eelco Dolstra
56e30e161c In the chroot, make all mounted filesystems private
This is required on systemd, which mounts filesystems as "shared"
subtrees.  Changes to shared trees in a private mount namespace are
propagated to the outside world, which is bad.
2012-08-20 15:27:30 -04:00
Eelco Dolstra
f0eab0636b Don't bind-mount /proc since we mount our own 2012-08-20 15:27:00 -04:00
Eelco Dolstra
862c4c5ec5 Fix 1755 permission on temporary directories left behind by ‘-K’ 2012-08-19 16:32:42 -04:00
Eelco Dolstra
325d1cfebf Don't warn about maximum link count exceeded on 0-byte files 2012-08-07 16:22:54 -04:00
Eelco Dolstra
d025142f52 Handle amount of disk space saved by hard linking being negative
Fixes bogus messages like "currently hard linking saves
17592186044416.00 MiB".
2012-08-05 21:45:27 -04:00
Eelco Dolstra
b6c989b801 Fix race condition when two processes create a hard link to a file in .links
This is a problem because one process may set the immutable bit before
the second process has created its link.

Addressed random Hydra failures such as:

error: cannot rename `/nix/store/.tmp-link-17397-1804289383' to
`/nix/store/rsvzm574rlfip3830ac7kmaa028bzl6h-nixos-0.1pre-git/upstart-interface-version':
Operation not permitted
2012-08-05 21:41:44 -04:00
Eelco Dolstra
108e14bb18 Fix race condition when two processes create the same link in /nix/store/.links 2012-08-05 18:17:55 -04:00
Eelco Dolstra
6763084ae5 Count bytes freed deleting unused links 2012-08-01 22:43:03 -04:00
Eelco Dolstra
01d56c1eec Drop the block count in the garbage collector 2012-08-01 22:34:46 -04:00
Eelco Dolstra
967d066d8e nix-store --gc: Make ‘--max-freed 0’ do the right thing
That is, delete almost nothing (it will still remove unused links from
/nix/store/.links).
2012-08-01 19:14:58 -04:00
Eelco Dolstra
1df702d347 removeUnusedLinks(): Print stats on disk space saved by hard linking 2012-08-01 19:01:50 -04:00
Eelco Dolstra
9c2decaa19 Merge branch 'master' into no-manifests 2012-08-01 18:03:49 -04:00
Eelco Dolstra
234ce610e0 Doh 2012-08-01 16:09:47 -04:00
Eelco Dolstra
7b10562370 Make ‘nix-store --optimise’ interruptible 2012-08-01 16:06:49 -04:00
Eelco Dolstra
c770a2422a Report substituter errors to clients of the Nix daemon 2012-08-01 11:19:24 -04:00
Eelco Dolstra
eb7849e3a2 Prevent an injection attack in passing untrusted options to substituters 2012-07-31 18:50:32 -04:00
Eelco Dolstra
90d9c58d4d Pass all --option flags to the daemon 2012-07-31 18:19:44 -04:00
Eelco Dolstra
89a8207029 Add an option ‘build-fallback’ (equivalent to the --fallback flag) 2012-07-31 17:56:02 -04:00
Eelco Dolstra
97421eb5ec Refactor settings processing
Put all Nix configuration flags in a Settings object.
2012-07-30 19:55:41 -04:00
Eelco Dolstra
d50d7a2874 Whitespace 2012-07-30 17:13:25 -04:00
Eelco Dolstra
d059bf48e4 Pass configuration settings to the substituters
Previously substituters could read nix.conf themselves, but this
didn't take --option flags into account.
2012-07-30 16:09:54 -04:00
Eelco Dolstra
f9613da180 Remove unused variables 2012-07-30 15:43:37 -04:00
Eelco Dolstra
6183cf2f19 Fix whitespace 2012-07-30 15:42:18 -04:00
Eelco Dolstra
73acb8b836 Let build.cc verify the expected hash of a substituter's output
Since SubstitutionGoal::finished() in build.cc computes the hash
anyway, we can prevent the inefficiency of computing the hash twice by
letting the substituter tell Nix about the expected hash, which can
then verify it.
2012-07-27 12:16:02 -04:00
Eelco Dolstra
fbf59d95f6 Remove more tabs 2012-07-27 10:56:33 -04:00
Eelco Dolstra
b4ea83249b Remove trailing whitespace / tabs 2012-07-27 09:59:18 -04:00
Eelco Dolstra
50395b71a9 Fix the substituter tests 2012-07-26 17:36:07 -04:00
Eelco Dolstra
8c79100839 Merge branch 'master' into no-manifests 2012-07-26 15:14:33 -04:00
Eelco Dolstra
3a4623afbb Set permissions on temporary build directories to 0700
Fixes #39.
2012-07-26 15:04:40 -04:00
Eelco Dolstra
e98c029717 Handle platforms that don't support linking to a symlink
E.g. Darwin doesn't allow this.
2012-07-23 18:42:18 -04:00
Eelco Dolstra
fd63c8bfcd Unlink the right file 2012-07-23 18:06:37 -04:00
Eelco Dolstra
680ab6f83d Garbage collect unused links in /nix/store/.links
Incremental optimisation requires creating links in /nix/store/.links
to all files in the store.  However, this means that if we delete a
store path, no files are actually deleted because links in
/nix/store/.links still exists.  So we need to check /nix/store/.links
for files with a link count of 1 and delete them.
2012-07-23 17:14:16 -04:00
Eelco Dolstra
6193105710 Automatically optimise the Nix store when a new path is added
Auto-optimisation is enabled by default.  It can be turned off by
setting auto-optimise-store to false in nix.conf.
2012-07-23 17:14:15 -04:00
Eelco Dolstra
564fb7d9fa optimiseStore(): Use a content-addressed file store in /nix/store/.links
optimiseStore() now creates persistent, content-addressed hard links
in /nix/store/.links.  For instance, if it encounters a file P with
hash H, it will create a hard link

  P' = /nix/store/.link/<H>

to P if P' doesn't already exist; if P' exist, then P is replaced by a
hard link to P'.  This is better than the previous in-memory map,
because it had the tendency to unnecessarily replace hard links with a
hard link to whatever happened to be the first file with a given hash
it encountered.  It also allows on-the-fly, incremental optimisation.
2012-07-23 17:14:15 -04:00
Eelco Dolstra
6852289c46 Use lutimes() if available to canonicalise the timestamp of symlinks
Also use utimes() instead of utime() if lutimes() is not available.
2012-07-23 16:52:25 -04:00
Eelco Dolstra
b7fd2c2822 Use "#pragma once" to prevent repeated header file inclusion 2012-07-18 14:59:03 -04:00
Eelco Dolstra
fe241ece29 Merge branch 'master' into no-manifests 2012-07-18 10:47:59 -04:00
Eelco Dolstra
ccc52adfb2 Add function queryPathFromHashPart()
To implement binary caches efficiently, Hydra needs to be able to map
the hash part of a store path (e.g. "gbg...zr7") to the full store
path (e.g. "/nix/store/gbg...kzr7-subversion-1.7.5").  (The binary
cache mechanism uses hash parts as a key for looking up store paths to
ensure privacy.)  However, doing a search in the Nix store for
/nix/store/<hash>* is expensive since it requires reading the entire
directory.  queryPathFromHashPart() prevents this by doing a cheap
database lookup.
2012-07-17 18:55:39 -04:00
Eelco Dolstra
3a9fdf2747 Return an exit code of 100 for cached failed builds
Exit code 100 should be returned for all permanent failures.  This
includes cached failures.

Fixes #34.
2012-07-17 15:55:30 -04:00
Eelco Dolstra
6c01fb4d68 Update Nix 1.1 release notes 2012-07-17 10:06:20 -04:00
Eelco Dolstra
53b24f3518 Allow disabling log compression 2012-07-17 09:40:12 -04:00
Eelco Dolstra
f2bdc87595 Update the other substituters 2012-07-11 18:52:09 -04:00
Eelco Dolstra
09a6321aeb Replace hasSubstitutes() with querySubstitutablePaths()
querySubstitutablePaths() takes a set of paths, so this greatly
reduces daemon <-> client latency.
2012-07-11 17:52:18 -04:00
Eelco Dolstra
58ef4d9a95 Add a function queryValidPaths()
queryValidPaths() combines multiple calls to isValidPath() in one.
This matters when using the Nix daemon because it reduces latency.
For instance, on "nix-env -qas \*" it reduces execution time from 5.7s
to 4.7s (which is indistinguishable from the non-daemon case).
2012-07-11 11:08:47 -04:00
Eelco Dolstra
667d5f1936 Rename queryValidPaths() to queryAllValidPaths() 2012-07-11 10:49:04 -04:00
Eelco Dolstra
eb3036da87 Implement querySubstitutablePathInfos() in the daemon
Also removed querySubstitutablePathInfo().
2012-07-11 10:43:24 -04:00
Eelco Dolstra
425cc612ad build.cc: Don't use hasSubstitute()
Instead make a single call to querySubstitutablePathInfo() per
derivation output.  This is faster and prevents having to implement
the "have" function in the binary cache substituter.
2012-07-08 18:39:24 -04:00
Eelco Dolstra
11800e6198 download-from-binary-cache: parallelise fetching of NAR info files
Getting substitute information using the binary cache substituter has
non-trivial latency overhead.  A package or NixOS system configuration
can have hundreds of dependencies, and in the worst case (when the
local info cache is empty) we have to do a separate HTTP request for
each of these.  If the ping time to the server is t, getting N info
files will take tN seconds; e.g., with a ping time of 0.1s to
nixos.org, sequentially downloading 1000 info files (a typical NixOS
config) will take at least 100 seconds.

To fix this problem, the binary cache substituter can now perform
requests in parallel.  This required changing the substituter
interface to support a function querySubstitutablePathInfos() that
queries multiple paths at the same time, and rewriting queryMissing()
to take advantage of parallelism.  (Due to local caching,
parallelising queryMissing() is sufficient for most use cases, since
it's almost always called before building a derivation and thus fills
the local info cache.)

For example, parallelism speeds up querying all 1056 paths in a
particular NixOS system configuration from 116s to 2.6s.  It works so
well because the eccentricity of the top-level derivation in the
dependency graph is only 9.  So we only need 10 round-trips (when
using an unlimited number of parallel connections) to get everything.

Currently we do a maximum of 150 parallel connections to the server.
Thus it's important that the binary cache server (e.g. nixos.org) has
a high connection limit.  Alternatively we could use HTTP pipelining,
but WWW::Curl doesn't support it and libcurl has a hard-coded limit of
5 requests per pipeline.
2012-07-06 19:08:20 -04:00
Eelco Dolstra
1aba0bf0fa nix-store -r: do substitutions in parallel
I.e. when multiple non-derivation arguments are passed to ‘nix-store
-r’ to be substituted, do them in parallel.
2012-06-27 16:58:15 -04:00
Eelco Dolstra
42f5a2fc29 Mount an empty /dev/shm tmpfs in the chroot
This ensures that whatever the builder writes in /dev/shm is
automatically cleaned up.
2012-06-27 09:52:27 -04:00
Eelco Dolstra
3ee208516f Check the return code of the clone() call 2012-06-27 09:52:06 -04:00
Eelco Dolstra
1db38ae81b When using chroots, use a private PID namespace
In a private PID namespace, processes have PIDs that are separate from
the rest of the system.  The initial child gets PID 1.  Processes in
the chroot cannot see processes outside of the chroot.  This improves
isolation between builds.  However, processes on the outside can see
processes in the chroot and send signals to them (if they have
appropriate rights).

Since the builder gets PID 1, it serves as the reaper for zombies in
the chroot.  This might turn out to be a problem.  In that case we'll
need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 15:45:16 -04:00
Eelco Dolstra
5489086456 Use a private UTS namespace to provide a deterministic host/domain name to builders
In chroot builds, set the host name to "localhost" and the domain name
to "(none)" (the latter being the kernel's default).  This improves
determinism a bit further.

P.S. I have to idea what UTS stands for.
2012-06-25 14:12:17 -04:00
Eelco Dolstra
723a68c826 Improve error message 2012-06-23 00:57:14 -04:00
Eelco Dolstra
7ffa523fd1 In chroot builds, use a private SysV IPC namespace
This improves isolation a bit further, and it's just one extra flag in
the unshare() call.

P.S. It would be very cool to use CLONE_NEWPID (to put the builder in
a private PID namespace) as well, but that's slightly more risky since
having a builder start as PID 1 may cause problems.
2012-06-23 00:51:40 -04:00
Eelco Dolstra
df716c98d2 In chroot builds, use a private network namespace
On Linux it's possible to run a process in its own network namespace,
meaning that it gets its own set of network interfaces, disjunct from
the rest of the system.  We use this to completely remove network
access to chroot builds, except that they get a private loopback
interface.  This means that:

- Builders cannot connect to the outside network or to other processes
  on the same machine, except processes within the same build.

- Vice versa, other processes cannot connect to processes in a chroot
  build, and open ports/connections do not show up in "netstat".

- If two concurrent builders try to listen on the same port (e.g. as
  part of a test), they no longer conflict with each other.

This was inspired by the "PrivateNetwork" flag in systemd.
2012-06-23 00:28:35 -04:00
Eelco Dolstra
f5398d374b Compress build logs on the fly using bzip2 2012-05-30 10:12:29 -04:00
Eelco Dolstra
4bc4da331a Reserve some disk space for the garbage collector
We can't open a SQLite database if the disk is full.  Since this
prevents the garbage collector from running when it's most needed, we
reserve some dummy space that we can free just before doing a garbage
collection.  This actually revives some old code from the Berkeley DB
days.

Fixes #27.
2012-05-29 22:59:12 -04:00
Eelco Dolstra
2c26985835 Add option ‘build-keep-log’ to enable/disable writing of build logs
Fixes #26.
2012-05-29 16:42:05 -04:00
Eelco Dolstra
db5b86ef13 * Add an option ‘build-use-substitutes’, which can be set to ‘false’
to disable use of substitutes; i.e., force building from source.
  Fixes Nix/221.
2012-04-30 19:15:34 -04:00
Eelco Dolstra
46cdc6ad51 Handle EPERM when creating a hard link for the chroot
There is a race condition when doing parallel builds with chroots and
the immutable bit enabled.  One process may call makeImmutable()
before the other has called link(), in which case link() will fail
with EPERM.  We could retry or wrap the operation in a lock, but since
this condition is rare and I'm lazy, we just use the existing copy
fallback.

Fixes #9.
2012-04-30 10:58:04 -04:00
Eelco Dolstra
6387830c3b Merge pull request #2 from viric/master
Again, adding the sync option
2012-04-23 03:29:03 -07:00
Eelco Dolstra
1132dd27ea Fix obscure race condition in GC root creation
This should fix rare Hydra errors of the form:

error: symlinking `/nix/var/nix/gcroots/per-user/hydra/hydra-roots/7sfhs5fdmjxm8sqgcpd0pgcsmz1kq0l0-nixos-iso-0.1pre33785-33795' to `/nix/store/7sfhs5fdmjxm8sqgcpd0pgcsmz1kq0l0-nixos-iso-0.1pre33785-33795': File exists
2012-04-16 18:47:01 +02:00
Eelco Dolstra
31e34fcf93 Close almost all file descriptors in the builder
This regression was accidentally introduced in
35355fc1fc.
2012-04-15 00:20:32 +02:00
Eelco Dolstra
f2467eb6a0 On Linux, pretend we're building on Linux 2.6
Setting the UNAME26 personality causes "uname" to return "2.6.x",
regardless of the kernel version.  This improves determinism in
a few misbehaved packages.
2012-04-05 13:03:19 +02:00
Eelco Dolstra
446b827bae Mac OS X fix 2012-03-26 23:18:36 +02:00
Eelco Dolstra
480dda0e42 Delete non-directory valid paths right away
It's unlikely that rename() is faster than unlink() on a regular file
or symlink, so don't bother.
2012-03-26 20:56:30 +02:00
Eelco Dolstra
117670be57 Make the garbage collector more concurrent
Make the garbage collector more concurrent by deleting valid paths
outside the region where we're holding the global GC lock.  This
should greatly reduce the time during which new builds are blocked,
since the deletion accounts for the vast majority of the time spent in
the GC.

To ensure that this is safe, the valid paths are invalidated and
renamed to some arbitrary path while we're holding the lock.  This
ensures that we when we finally delete the path, it's not a (newly)
valid or locked path.
2012-03-26 20:43:33 +02:00
Eelco Dolstra
8be1979f1a Remove the --max-links GC option
We don't need this anymore now that current filesystems support more
than 32,000 files in a directory.
2012-03-26 20:00:02 +02:00
Lluís Batlle i Rossell
85799bf89e Fixing the default of sync-before-registering
Setting 'false' as default, as suggested by Eelco.

I also added a comment about the setting in the code.
2012-03-22 20:05:54 +01:00
Lluís Batlle i Rossell
25de80e2b6 Adding a nix option to sync before registering a path, for non-ext*
filesystems.
2012-03-22 19:57:42 +01:00
Eelco Dolstra
19d9762ad5 nix-store --clear-failed-paths: Clear derivation outputs
If the argument to ‘nix-store --clear-failed-paths’ is a derivation,
then clear the failed state of its outputs.
2012-03-20 18:23:26 +01:00
Eelco Dolstra
b006a31d52 Drop the externals directory
Nix now requires SQLite and bzip2 to be pre-installed.  SQLite is
detected using pkg-config.  We required DBD::SQLite anyway, so
depending on SQLite is not a big problem.

The --with-bzip2, --with-openssl and --with-sqlite flags are gone.
2012-03-18 23:54:57 +01:00
Eelco Dolstra
bd50c01972 Ensure that Perl processes delete their entry in the temproots directory
By moving the destructor object to libstore.so, it's also run when
download-using-manifests and nix-prefetch-url exit.  This prevents
them from cluttering /nix/var/nix/temproots with stale files.
2012-03-13 17:07:49 +01:00
Eelco Dolstra
35355fc1fc Set the close-on-exec flag on file descriptors 2012-03-05 20:29:00 +01:00
Eelco Dolstra
7b22bec252 Don't leak a file descriptor in commonChildInit() 2012-03-05 20:28:09 +01:00
Eelco Dolstra
2b4964f319 Restore progress indication during nix-copy-closure 2012-03-05 18:13:44 +01:00
Eelco Dolstra
8afd28a922 Remove dependency on sqlite3_table_column_metadata
Not all SQLite builds have the function sqlite3_table_column_metadata.
We were only using it in a schema upgrade check for compatibility with
databases that were probably never seen in the wild.  So remove it.
2012-03-01 16:06:49 +01:00
Eelco Dolstra
1c94524458 Fix an uninitialised variable
The variable ‘useChroot’ was not initialised properly.  This caused
random failures if using the build hook.  Seen on Mac OS X 10.7 with Clang.
Thanks to KolibriFX for finding this :-)
2012-03-01 16:06:49 +01:00
Eelco Dolstra
6fe13e6aba Fix chroots builds
Chroots are initialised by hard-linking inputs from the Nix store to
the chroot.  This doesn't work if the input has its immutable bit set,
because it's forbidden to create hard links to immutable files.  So
temporarily clear the immutable bit when creating and destroying the
chroot.

Note that making regular files in the Nix store immutable isn't very
reliable, since the bit can easily become cleared: for instance, if we
run the garbage collector after running ‘nix-store --optimise’.  So
maybe we should only make directories immutable.
2012-02-18 01:23:52 +01:00
Eelco Dolstra
bd013b6f98 On Linux, make the Nix store really read-only by using the immutable bit
I was bitten one time too many by Python modifying the Nix store by
creating *.pyc files when run as root.  On Linux, we can prevent this
by setting the immutable bit on files and directories (as in ‘chattr
+i’).  This isn't supported by all filesystems, so it's not an error
if setting the bit fails.  The immutable bit is cleared by the garbage
collector before deleting a path.  The only tricky aspect is in
optimiseStore(), since it's forbidden to create hard links to an
immutable file.  Thus optimiseStore() temporarily clears the immutable
bit before creating the link.
2012-02-15 01:31:56 +01:00
Eelco Dolstra
5e57047d87 Fix a broken guard around utime()
Because of an outdated check for a timestamp of 0, we were calling
utime() even when it wasn't necessary.
2012-02-15 01:00:30 +01:00
Eelco Dolstra
58ac7a17a4 Don't use deletePath() to delete a single file 2012-02-15 00:28:01 +01:00
Eelco Dolstra
d5a5a83ad4 Use data() instead of c_str() where appropriate 2012-02-09 18:27:45 +01:00
Eelco Dolstra
a71d02440b * Oops. 2011-12-30 17:13:25 +00:00
Eelco Dolstra
6f5e3326ce * Move topoSortPaths() out of gc.cc. 2011-12-30 15:02:50 +00:00
Eelco Dolstra
b1004f40f7 * Reject a build if there is a cycle among the outputs. This is
necessary because existing code assumes that the references graph is
  acyclic.
2011-12-30 14:47:14 +00:00
Eelco Dolstra
254b3399ba * Sync with the trunk. 2011-12-30 13:08:14 +00:00
Eelco Dolstra
8c42a8c8ff * Make sure that lock files are cleaned up properly when building
through the build hook.
2011-12-25 16:38:37 +00:00
Eelco Dolstra
524fa8a4f1 * Oops. 2011-12-22 16:27:03 +00:00
Eelco Dolstra
b33da599c5 * In the garbage collector, delete invalid paths before deleting
unreachable paths.  This matters when using --max-freed etc.:
  unreachable paths could become reachable again, so it's nicer to
  keep them if there is "real" garbage to be deleted.  Also, don't use
  readDirectory() but read the Nix store and delete invalid paths in
  parallel.  This reduces GC latency on very large Nix stores.
2011-12-22 15:55:53 +00:00
Eelco Dolstra
58d974336c * Drop unnecessary call to canonPath() (nixStore is already canonical). 2011-12-22 14:33:34 +00:00
Eelco Dolstra
2aac7cd021 * Another case of lock file permissions being too liberal. 2011-12-21 19:17:45 +00:00
Shea Levy
b4cee3f816 Revert previous commit
It doesn't detect indirect references
2011-12-21 17:31:34 +00:00
Shea Levy
f3c88f297d Detect and reject mutually-referential outputs
There is probably a more efficient way to do this.
2011-12-21 17:14:28 +00:00
Eelco Dolstra
b5363810bb * Fix the build. 2011-12-20 16:37:01 +00:00
Eelco Dolstra
194d21f9f6 * Sync with the trunk. 2011-12-16 23:33:01 +00:00
Eelco Dolstra
273b288a7e * importPath() -> importPaths(). Because of buffering of the input
stream it's now necessary for the daemon to process the entire
  sequence of exported paths, rather than letting the client do it.
2011-12-16 22:31:25 +00:00
Eelco Dolstra
e0bd307802 * Make the import operation through the daemon much more efficient
(way fewer roundtrips) by allowing the client to send data in bigger
  chunks.
* Some refactoring.
2011-12-16 19:44:13 +00:00
Eelco Dolstra
5a1b9ed0aa * Refactoring: move sink/source buffering into separate classes.
* Buffer the HashSink.  This speeds up hashing a bit because it
  prevents lots of calls to the hash update functions (e.g. nix-hash
  went from 9.3s to 8.7s of user time on the closure of my
  /var/run/current-system).
2011-12-15 16:19:53 +00:00
Eelco Dolstra
3a48282b06 * Buffer writes in FdSink. This significantly reduces the number of
system calls / context switches when dumping a NAR and in the worker
  protocol.
2011-12-14 23:30:06 +00:00
Eelco Dolstra
c8c0380744 * Remove unnecessary quotes. showPaths() already adds quotes. 2011-12-05 21:04:20 +00:00
Eelco Dolstra
000160f5b9 * In ‘nix-store --verify --check-contents’, repair missing hashes
rather than complain about them.
2011-12-02 17:52:18 +00:00
Eelco Dolstra
24f863d86b * When doing "nix-store --add-fixed" without "--recursive" via the Nix
daemon (which is an error), print a nicer error message than
  "Connection reset by peer" or "broken pipe".
* In the daemon, log errors that occur during request parameter
  processing.
2011-12-01 13:48:48 +00:00
Eelco Dolstra
b1eb8f4249 * Get rid of some superfluous error messages if a substituter fails.
* Say "fetch" instead of "substitute".
2011-11-29 13:00:41 +00:00
Eelco Dolstra
5bbd693cae * Add an API function exportPaths() that provides the functionality of
‘nix-store --export’.
* Add a Perl module that provides the functionality of
  ‘nix-copy-closure --to’.  This is used by build-remote.pl so it no
  longer needs to start a separate nix-copy-closure process.  Also, it
  uses the Perl API to do the export, so it doesn't need to start a
  separate nix-store process either.  As a result, nix-copy-closure
  and build-remote.pl should no longer fail on very large closures due
  to an "Argument list too long" error.  (Note that having very many
  dependencies in a single derivation can still fail because the
  environment can become too large.  Can't be helped though.)
2011-11-23 15:13:37 +00:00
Eelco Dolstra
993fa94fb4 * Move initialisation of variables like nixConfDir from libmain to
libstore so that the Perl bindings can use it as well.  It's vital
  that the Perl bindings use the configuration file, because otherwise
  nix-copy-closure will fail with a ‘database locked’ message if the
  value of ‘use-sqlite-wal’ is changed from the default.
2011-11-22 17:28:41 +00:00
Rob Vermaas
4e1ea17052 nix: add /etc/hosts with localhost entry to chroot builds. 2011-11-21 15:19:51 +00:00
Eelco Dolstra
c0b706213d * Boldly make SQLite WAL mode the default again. Hopefully the
intermittent problems are gone by now.  WAL mode is preferrable
  because it does way fewer fsyncs.
2011-11-07 21:11:59 +00:00
Shea Levy
af2e53fd48 Include all outputs of derivations in the closure of explicitly-passed derivation paths
This required adding a queryOutputDerivationNames function in the store API
2011-11-06 06:28:20 +00:00
Eelco Dolstra
13114daa3e * Ouch. A store upgrade could cause a substituter to be triggered,
causing a deadlock.
2011-09-12 09:07:43 +00:00
Eelco Dolstra
93227ff65c * Eliminate all uses of the global variable ‘store’ from libstore.
This should also fix:

    nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed.

  which was caused by hashDerivationModulo() calling the ‘store’
  object (during store upgrades) before openStore() assigned it.
2011-08-31 21:11:50 +00:00
Eelco Dolstra
b2027f70d9 * Fix a huuuuge security hole in the Nix daemon. It didn't check that
derivations added to the store by clients have "correct" output
  paths (meaning that the output paths are computed by hashing the
  derivation according to a certain algorithm).  This means that a
  malicious user could craft a special .drv file to build *any*
  desired path in the store with any desired contents (so long as the
  path doesn't already exist).  Then the attacker just needs to wait
  for a victim to come along and install the compromised path.

  For instance, if Alice (the attacker) knows that the latest Firefox
  derivation in Nixpkgs produces the path

    /nix/store/1a5nyfd4ajxbyy97r1fslhgrv70gj8a7-firefox-5.0.1

  then (provided this path doesn't already exist) she can craft a .drv
  file that creates that path (i.e., has it as one of its outputs),
  add it to the store using "nix-store --add", and build it with
  "nix-store -r".  So the fake .drv could write a Trojan to the
  Firefox path.  Then, if user Bob (the victim) comes along and does

    $ nix-env -i firefox
    $ firefox

  he executes the Trojan injected by Alice.

  The fix is to have the Nix daemon verify that derivation outputs are
  correct (in addValidPath()).  This required some refactoring to move
  the hash computation code to libstore.
2011-07-20 18:10:47 +00:00
Ludovic Courtès
5c9e9f732d Add support for the build-timeout' and --timeout' options. 2011-06-30 15:19:13 +00:00
Eelco Dolstra
8c69dac8a1 * Handle error messages from the Nix worker containing the `%'
character.  (Nix/216)
2011-04-19 10:44:44 +00:00
Eelco Dolstra
c931a7aec5 * Do a short sleep after SQLITE_BUSY. 2010-12-17 17:23:15 +00:00
Eelco Dolstra
b1eb252172 * Propagate the "100" exit status for failed builds through the Nix
daemon.
2010-12-17 11:28:26 +00:00
Eelco Dolstra
3dd02580e3 * I forgot to catch SQLiteBusy in registerValidPaths(). So
registerValidPaths() now handles busy errors and registerValidPath()
  is simply a wrapper around it.
2010-12-14 13:25:20 +00:00
Eelco Dolstra
d787285af9 * nix-instantiate: return exit status 100 to denote a permanent build
failure.  The build hook can use this to distinguish between
  transient and permanent failures on the remote side.
2010-12-13 16:53:23 +00:00
Eelco Dolstra
5833243c92 * Create /nix/var/nix/db if it's missing. 2010-12-13 13:42:34 +00:00
Eelco Dolstra
d7ca6f44eb * Update some comments. 2010-12-13 13:19:46 +00:00
Eelco Dolstra
e4720b1a79 * Ignore the result of sqlite3_reset(). 2010-12-08 18:19:15 +00:00
Eelco Dolstra
8062d3af30 * `nix-store --verify --check-contents': don't hold the global GC lock
while checking the contents, since this operation can take a very
  long time to finish.  Also, fill in missing narSize fields in the DB
  while doing this.
2010-12-06 15:29:38 +00:00
Eelco Dolstra
de79d23f76 * Retry a transaction if SQLite returns SQLITE_BUSY. This can happen
even with a very long busy timeout, because SQLITE_BUSY is also
  returned to resolve deadlocks.  This should get rid of random
  "database is locked" errors.  This is kind of hard to test though.
* Fix a horrible bug in deleteFromStore(): deletePathWrapped() should
  be called after committing the transaction, not before, because the
  commit might not succeed.
2010-12-05 18:23:19 +00:00
Eelco Dolstra
bdf089f463 * Before a build, show the disk space that the downloaded store paths
will approximately require.
2010-11-17 14:31:42 +00:00
Eelco Dolstra
e60c962fb8 * Add an operation `nix-store -q --size'. 2010-11-17 12:40:52 +00:00
Eelco Dolstra
1db6259076 * Implement RemoteStore::queryPathInfo(). 2010-11-17 12:08:01 +00:00
Eelco Dolstra
a3883cbd28 * Store the size of a store path in the database (to be precise, the
size of the NAR serialisation of the path, i.e., `nix-store --dump
  PATH').  This is useful for Hydra.
2010-11-16 17:11:46 +00:00
Eelco Dolstra
fb9368b5a0 * Sync with the trunk. 2010-11-16 12:49:47 +00:00
Eelco Dolstra
64fd29855a * Wrap deleteFromStore() in a transaction. Otherwise there might be a
race with other processes that add new referrers to a path,
  resulting in the garbage collector crashing with "foreign key
  constraint failed".  (Nix/4)
* Make --gc --print-dead etc. interruptible.
2010-10-14 15:55:51 +00:00
Eelco Dolstra
705868a8a9 * Make sure that config.h is included before the system headers,
because it defines _FILE_OFFSET_BITS.  Without this, on 
  OpenSolaris the system headers define it to be 32, and then 
  the 32-bit stat() ends up being called with a 64-bit "struct 
  stat", or vice versa.

  This also ensures that we get 64-bit file sizes everywhere.

* Remove the redundant call to stat() in parseExprFromFile().
  The file cannot be a symlink because that's the exit condition
  of the loop before.
2010-10-04 17:55:38 +00:00
Eelco Dolstra
e4907411c2 * Only do "pragma journal_mode = ..." if the current journal mode
differs from the desired mode.  There is an open SQLite ticket
  `Executing "PRAGMA journal_mode" may delete journal file while it is
  in use.'
2010-09-01 11:36:22 +00:00
Eelco Dolstra
bf0dde9597 * Always print hook output on stderr, even if --no-build-output is
set.
* In the build hook, print a trace message to allow Hydra to pick up
  the name of the remote machine used for the build.
2010-08-31 12:36:24 +00:00
Eelco Dolstra
e2e168f7c2 `nix-store --verify' improvements:
* If a path has disappeared, check its referrers first, and don't try
  to invalidate paths that have valid referrers.  Otherwise we get a
  foreign key constraint violation.
* Read the whole Nix store directory instead of statting each valid
  path, which is slower.
* Acquire the global GC lock.
2010-08-31 11:47:31 +00:00
Eelco Dolstra
80e722278c * When using the build hook, distinguish between the stderr of the
hook script proper, and the stdout/stderr of the builder.  Only the
  latter should be saved in /nix/var/log/nix/drvs.
* Allow the verbosity to be set through an option.
* Added a flag --quiet to lower the verbosity level.
2010-08-30 14:53:03 +00:00
Eelco Dolstra
20acd43c25 * Disable the GC reachability check for now (when creating new roots),
as it's hopelessly inefficient.
2010-08-30 14:11:57 +00:00
Eelco Dolstra
766f708418 * Experimental feature: allow a derivation to tell the build hook that
it requires a certain feature on the build machine, e.g.

    requiredSystemFeatures = [ "kvm" ];

  We need this in Hydra to make sure that builds that require KVM
  support are forwarded to machines that have KVM support.  Probably
  this should also be enforced for local builds.
2010-08-27 13:18:13 +00:00
Eelco Dolstra
e437b08250 * Made the build hook mechanism more efficient. Rather than starting
the hook every time we want to ask whether we can run a remote build
  (which can be very often), we now reuse a hook process for answering
  those queries until it accepts a build.  So if there are N
  derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
Eelco Dolstra
034f608e00 * Don't complain if the stored hash of a path is zero (unknown). 2010-08-24 14:25:33 +00:00
Eelco Dolstra
f58f51f380 * Handle the unlikely case where a derivation has no dependencies at
all.
2010-08-24 11:45:44 +00:00
Eelco Dolstra
955d11aae7 * Sync with the trunk. 2010-08-24 09:56:28 +00:00
Eelco Dolstra
6846ed8b44 * Make --cores work when building through the Nix daemon. 2010-08-12 09:21:50 +00:00
Eelco Dolstra
587dc8aa00 * Sync with the trunk. 2010-08-04 17:48:29 +00:00
Eelco Dolstra
fd9c77dfc7 * Use SQLite 3.7.0's write-ahead logging (WAL mode). This is a lot
faster than the old mode when fsyncs are enabled, because it only
  performs an fsync() when doing a checkpoint, rather than at every
  commit.  Some timings for doing a "nix-instantiate /etc/nixos/nixos
  -A system" after modifying the stdenv setup script:

    42.5s - SQLite 3.6.23 with truncate mode and fsync
     3.4s - SQLite 3.6.23 with truncate mode and no fsync
    32.1s - SQLite 3.7.0 with truncate mode and fsync
    16.8s - SQLite 3.7.0 with WAL mode and fsync, auto-checkpoint
            every 1000 pages
     8.3s - SQLite 3.7.0 with WAL mode and fsync, auto-checkpoint
            every 8192 pages
     1.7s - SQLite 3.7.0 with WAL mode and no fsync

  The default is now to use WAL mode with fsyncs.  Because WAL doesn't
  work on remote filesystems such as NFS (as it uses shared memory),
  truncate mode can be re-enabled by setting the "use-sqlite-wal"
  option to false.
2010-08-04 17:35:59 +00:00
Eelco Dolstra
7f893b7a43 * Allow derivations to hint that they should not be built remotely
using the build hook mechanism, by setting the derivation attribute
  "preferLocalBuild" to true.  This has a few use cases:

  - The user environment builder.  Since it just creates a bunch of
    symlinks without much computation, there is no reason to do it
    remotely.  In fact, doing it remotely requires the entire closure
    of the user environment to be copied to the remote machine, which
    is extremely wasteful.

  - `fetchurl'.  Performing the download on a remote machine and then
    copying it to the local machine involves twice as much network
    traffic as performing the download locally, and doesn't save any
    CPU cycles on the local machine.
2010-08-04 12:13:58 +00:00
Peter Simons
a17071fef1 Include <cstring> to ensure that strcpy(), strlen(), and memset() are declared.
An "using namespace std" was added locally in those functions that refer to
names from <cstring>. That is not pretty, but it's a very portable solution,
because strcpy() and friends will be found in both the 'std' and in the global
namespace.
2010-06-24 17:51:04 +00:00
Peter Simons
8b7f8b56f1 Added support for passing an (impure) NIX_BUILD_CORES variable to build expressions.
This patch adds the configuration file variable "build-cores" and the
command line argument "--cores". These settings specify the number of
CPU cores to utilize for parallel building within a job, i.e. by passing
an appropriate "-j" flag to GNU Make. The default value is 1, which
means that parallel building is *disabled*. If the number of build cores
is specified as 0 (synonymously: "guess" or "auto"), then the actual
value is supposed to be auto-detected by builders at run-time, i.e by
calling the nproc(1) utility from coreutils.

The environment variable $NIX_BUILD_CORES is available to builders, but
the contents of that variable does *not* influence the hash that goes
into the $out store path, i.e. the number of build cores to be utilized
can be changed at will without requiring any re-builds.
2010-06-23 14:34:08 +00:00
Eelco Dolstra
3e5e0faf9c * Okay, putting a lock on the temporary directory used by importPath()
doesn't work because the garbage collector doesn't actually look at
  locks.  So r22253 was stupid.  Use addTempRoot() instead.  Also,
  locking the temporary directory in exportPath() was silly because it
  isn't even in the store.
2010-06-21 11:08:09 +00:00
Eelco Dolstra
bf87cc44b4 * Sync with the trunk. 2010-06-21 07:55:38 +00:00
Eelco Dolstra
b57189174f * In importPath() and exportPath(), lock the temporary directory to
prevent it from being deleted by the garbage collector.
2010-06-14 08:34:48 +00:00
Eelco Dolstra
d1f6c0cbe3 * Replacing ValidPath rows doesn't work because it causes a constraint
violation of the Refs table.  So don't do that.
2010-06-08 13:38:28 +00:00
Eelco Dolstra
07ca66cf24 * Applied a patch from David Brown to prevent `nix-store --optimise'
from failing on rename() on BtrFS.
2010-06-04 13:56:11 +00:00
Eelco Dolstra
89865da76d * Turn build errors during evaluation into EvalErrors. 2010-06-01 11:19:32 +00:00
Eelco Dolstra
a443c7573b * Hack to allow derivations to disable chroot builds by setting the
attribute "__noChroot = true" (requested by Rob).
2010-06-01 10:01:14 +00:00
Eelco Dolstra
aa45027818 * Sync with the trunk. 2010-05-12 22:13:09 +00:00
Eelco Dolstra
f92c9a0ac5 * Allow unprivileged users to do `nix-store --clear-failed-paths' and
`nix-store --query-failed-paths'.
2010-05-04 10:45:10 +00:00
Eelco Dolstra
6199f9b93e * Added a command `nix-store --clear-failed-paths <PATHS>' to clear
the "failed" status of the given store paths.  The special value `*'
  clears all failed paths.
2010-04-26 12:56:42 +00:00
Eelco Dolstra
2398af13c5 * Add an command `nix-store --query-failed-paths' to list the cached
failed paths (when using the `build-cache-failure' option).
2010-04-26 12:43:42 +00:00
Eelco Dolstra
6bbfe95e30 * Don't use an ostringstream in unparseDerivation(), because it's
slow.  A `nix-env -qa --drv-path \*' on Nixpkgs was spending 40% of
  its time in unparseDerivation() because of this (now 11%).
2010-04-21 19:25:50 +00:00
Eelco Dolstra
f3b8833a48 * Drop the dependency on the ATerm library. 2010-04-19 14:51:58 +00:00
Eelco Dolstra
efc7a579e8 * Don't use the ATerm library for parsing/printing .drv files. 2010-04-19 13:46:58 +00:00
Eelco Dolstra
e020d80e4e * Sync with the trunk. 2010-03-11 15:45:05 +00:00
Ludovic Courtès
05e15049a5 Show the build user's group in /etc/group in chroots.
* src/libstore/build.cc (nix::DerivationGoal::startBuilder): Create
  /etc/group showing the build user's group.
2010-03-11 14:47:04 +00:00
Eelco Dolstra
070057c1b9 * Sync with the trunk. 2010-03-11 10:52:52 +00:00
Ludovic Courtès
c752c9f41a Fix thinko in r20547.
* src/libstore/build.cc (nix::DerivationGoal::startBuilder): Fix the GID
  of the build user in /etc/passwd.
2010-03-11 10:33:04 +00:00
Ludovic Courtès
2e8eaca573 Clear supplementary groups of `nixbld' in /etc/passwd in chroots.
* src/libstore/build.cc (nix::DerivationGoal::startBuilder): Don't
  display any supplementary groups for `nixbld' in /etc/passwd.
2010-03-11 10:21:23 +00:00
Eelco Dolstra
1a65142ec4 * Remove a debug statement. 2010-03-10 12:46:25 +00:00
Eelco Dolstra
4c356acd04 * In `nix-store --export', abort if the contents of a path has
changed.  This prevents corrupt paths from spreading to other
  machines.  Note that checking the hash is cheap because we're
  hashing anyway (because of the --sign feature).
2010-03-09 14:32:03 +00:00
Eelco Dolstra
44f6e6de77 * Set gc-keep-outputs' or gc-keep-derivations' to false with
`--delete --ignore-liveness'.
2010-03-08 21:31:42 +00:00
Eelco Dolstra
2e4ef03aa3 * Increase the sqlite timeout. 2010-03-08 10:35:45 +00:00
Eelco Dolstra
bc6f7fc139 * Remove some unused functions. 2010-03-04 13:03:26 +00:00
Eelco Dolstra
3f9e647ae8 * checkInterrupt() shouldn't be called from a destructor. 2010-03-02 19:04:17 +00:00
Eelco Dolstra
594eaddd11 * When using the included sqlite/aterm libraries, build with
--enable-shared.
* In libutil/libstore/libexpr etc., link against sqlite and aterm.
* Some more header file hygiene.
2010-03-02 15:58:13 +00:00
Eelco Dolstra
24035b98b1 * Implement RemoteStore::queryValidPaths(). 2010-02-26 12:05:01 +00:00
Eelco Dolstra
e42401ee7b * Implement RemoteStore::queryDerivationOutputs(). 2010-02-25 15:52:22 +00:00
Eelco Dolstra
af565c348a * Support read-only access to the database. 2010-02-24 16:44:43 +00:00
Eelco Dolstra
e33f67ff0b * Refactor the upgrade / database initialisation logic a bit. 2010-02-24 16:30:20 +00:00
Eelco Dolstra
84a4dd5ff0 * Don't use fdatasync since it doesn't work on Snow Leopard.
* Don't refer to config.h in util.hh, because config.h is not
  installed (http://hydra.nixos.org/build/303053).
2010-02-24 15:46:06 +00:00
Eelco Dolstra
cfe742cfc5 * A function to query just the database id of a valid path. 2010-02-24 15:07:23 +00:00
Eelco Dolstra
6baa2a2f5e 2010-02-24 14:39:52 +00:00
Eelco Dolstra
9fd85c94de * Use `truncate' journal mode, which should be a bit faster. 2010-02-24 14:22:34 +00:00
Eelco Dolstra
a3c63d0d6c * Disable fsync() in SQLite if the fsync-metadata option is set to
false.
* Change the default for `fsync-metadata' to true.
* Disable `fsync-metadata' in `make check'.
2010-02-24 13:12:57 +00:00
Eelco Dolstra
90b6352d0a * Do registerValidPaths() in one transaction, which is much faster.
E.g. it cuts the runtime of the referrers test from 50s to 23s.
2010-02-24 12:48:00 +00:00
Eelco Dolstra
fae0427324 * ADDITIONAL_NETWORK_LIBS -> LIBS. 2010-02-24 12:25:48 +00:00
Eelco Dolstra
fa6a4fcb11 * Add ${sqlite_lib} everywhere. Just adding it in `libstore' doesn't
work on x86_64 when sqlite is compiled statically.
2010-02-24 12:18:48 +00:00
Eelco Dolstra
462bd50aef * Use normal (rather than full) synchronous mode, which I gather from
the description at http://www.sqlite.org/atomiccommit.html should be
  safe enough.
2010-02-24 10:57:57 +00:00
Eelco Dolstra
63b09c5e41 2010-02-23 22:31:38 +00:00
Eelco Dolstra
b4e6d98fc3 * configure: flag --with-sqlite. 2010-02-23 22:12:46 +00:00
Eelco Dolstra
2b20318b0e 2010-02-22 14:24:37 +00:00
Eelco Dolstra
9cda616949 * The database needs a trigger to get rid of self-references to
prevent a foreign key constraint violation on the Refs table when
  deleting a path.
2010-02-22 14:18:55 +00:00
Eelco Dolstra
c4d388add4 * Get derivation outputs from the database instead of the .drv file,
which requires more I/O.
2010-02-22 12:44:36 +00:00
Eelco Dolstra
103cfee056 * Revert r19650 (implement gc-keep-outputs by looking for derivations
with the same name as the output) and instead use the
  DerivationOutputs table in the database, which is the correct way to
  to do things.
2010-02-22 11:44:17 +00:00
Eelco Dolstra
299ff64812 * Put the derivation outputs in the database. This is useful for the
garbage collector.
2010-02-22 11:15:50 +00:00
Eelco Dolstra
1930570ad9 * Foreign key support in SQLite is not a persistent setting, so enable
it at startup.
* Implement negative caching.  Now `make check' passes.
2010-02-19 17:15:22 +00:00
Eelco Dolstra
9c9a88e9e2 * Implement more stuff. 2010-02-19 16:43:25 +00:00
Eelco Dolstra
762cee72cc * Implement registerValidPath(). 2010-02-19 16:04:51 +00:00
Eelco Dolstra
268f9aaf28 * Implemented queryValidPaths() and verifyStore(). 2010-02-18 16:51:27 +00:00
Eelco Dolstra
836e5b6f57 * Implemented queryReferrers(). 2010-02-18 16:21:59 +00:00
Eelco Dolstra
77cb9e3fb1 * Implement queryPathInfo(). 2010-02-18 15:52:57 +00:00
Eelco Dolstra
885e22b16e * Implement isValidPath(). 2010-02-18 15:11:08 +00:00
Eelco Dolstra
cfb09e0fad * Automatically abort transactions if they go out of scope without
committing.
2010-02-18 14:40:07 +00:00
Eelco Dolstra
e0305bb7a8 * Some wrapper objects to ensure that SQLite objects are properly
destroyed.
2010-02-18 14:30:42 +00:00
Eelco Dolstra
a053d2d8e5 * Add the deriver to the ValidPaths table. In principle we could now
store all the derivers of a path efficiently.  But that opens a big
  can of worms with respect to garbage collector semantics.
2010-02-18 13:48:18 +00:00
Eelco Dolstra
dbddac0fe9 * Assign an integer id to every row in the ValidPaths table in order
to make the Refs table more space-efficient.  For instance, this
  reduces the size of the database on my laptop from 93 MiB to 18
  MiB.  (It was 72 MiB with the old schema on an ext3 disk with a 1
  KiB block size.)
2010-02-18 13:40:46 +00:00
Eelco Dolstra
c1a07f9445 * Convert the Nix database to SQLite. 2010-02-18 13:16:59 +00:00
Eelco Dolstra
4e17be7981 * Revert r19797, and use a simpler solution: just don't monitor build
hooks for silence.  It's unnecessary because the remote nix-store
  command is already monitoring the real build.
2010-02-03 21:38:41 +00:00
Eelco Dolstra
f859a8d3c3 * While waiting for a lock, print a sign of life every 5 minutes.
This prevents remote builders from being killed by the
  `max-silent-time' inactivity monitor while they are waiting for a
  long garbage collection to finish.  This happens fairly often in the
  Hydra build farm.
2010-02-03 21:22:57 +00:00
Eelco Dolstra
4bbbe25802 * Remove most Cygwin-specific code. Cygwin 1.7 implements advisory
POSIX locks, and simulates Unix-style file deletion semantics
  sufficiently.  Note that this means that Nix won't work on Cygwin
  1.5 anymore.
2010-02-02 15:28:36 +00:00
Eelco Dolstra
07ffdc2862 * Added an option "fsync-metadata" to fsync() changes to
/nix/var/nix/db.
* Removed the function writeStringToFile since it does (almost) the
  same thing as writeFile.
2010-01-29 12:22:58 +00:00
Eelco Dolstra
ad529fb89f * Don't consider a store path valid if its info file exists but is
zero bytes long.  That makes Nix more robust in case of crashes
  (especially on ext4).
2010-01-29 11:53:58 +00:00
Eelco Dolstra
fdcaf37361 * Made `nix-store -qR --include-outputs' much faster if there are
multiple paths specified on the command line (from O(n * m) to O(n +
  m), where n is the number of arguments and m is the size of the
  closure).
2010-01-25 17:18:44 +00:00
Eelco Dolstra
5388944e8d * Make the garbage collector do the right thing when `gc-keep-outputs'
is enabled by not depending on the deriver.
2010-01-25 16:04:32 +00:00
Eelco Dolstra
ef92a14bfe * Include config.h before the C library headers, because it defines
_FILE_OFFSET_BITS=64.  Without it, functions like stat() fail on
  large file sizes.  This happened with a Nix store on squashfs:

  $ nix-store --dump /tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds > /dev/null
  error: getting attributes of path `/tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds': Value too large for defined data type

  $ stat /tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds
  File: `/tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds'
  Size: 0               Blocks: 36028797018963968 IO Block: 1024   regular empty file

  (This is a bug in squashfs or mksquashfs, but it shouldn't cause Nix
  to fail.)
2009-12-17 14:12:44 +00:00
Eelco Dolstra
945d8218fb * Build correctly against newer ATerm releases. Fixes "error: 'union'
tag used in naming 'struct _ATerm'".
2009-12-16 15:29:50 +00:00
Eelco Dolstra
bcd6cdf0d8 * Give a better error message when trying to build something and
readOnlyMode is set.
2009-12-09 17:45:22 +00:00
Eelco Dolstra
f9e766db98 * Randomise the order in which we delete entries to make the collector
less biased towards deleting paths that come alphabetically first
  (e.g. /nix/store/000...).  This matters when using --max-freed etc.
2009-11-24 09:53:18 +00:00
Eelco Dolstra
ca50c83fbb 2009-11-23 21:21:29 +00:00
Eelco Dolstra
ae6bf87273 * `nix-store --gc --print-roots': also print the path of the actual
root symlink, not just its target.  E.g.:

  /nix/var/nix/profiles/system-99-link -> /nix/store/76kwf88657nq7wgk1hx3l1z5q91zb9wd-system
2009-11-23 17:23:12 +00:00
Eelco Dolstra
c364d5d1e3 * Made the garbage collector a lot faster. It no longer computes the
complete set of live and dead paths before starting the actual
  deletion, but determines liveness on demand.  I.e. for any path in
  the store, it first tries to delete all the referrers, and then the
  path itself.  This means that the collector can start deleting paths
  almost immediately.
2009-11-23 16:34:24 +00:00
Eelco Dolstra
8824d60fe5 * Remove the --use-atime / --max-atime garbage collector flags. Many
(Linux) machines no longer maintain the atime because it's too
  expensive, and on the machines where --use-atime is useful (like the
  buildfarm), reading the atimes on the entire Nix store takes way too
  much time to make it practical.
2009-11-20 17:12:38 +00:00
Eelco Dolstra
327a232c85 * Remove support for old (before Nix 0.12pre12020) databases. 2009-11-06 01:15:44 +00:00
Eelco Dolstra
8520542071 * When building in a chroot, make a copy of a file if hard-linking
fails.  This is likely to happen after a `nix-store --optimise',
  because some files may have 32000 links (NIX-111).
2009-10-22 08:28:33 +00:00
Eelco Dolstra
6b9f6b0222 * Remove a prototype for a function that no longer exists. 2009-10-22 08:12:38 +00:00
Sander van der Burg
53a4981fa2 Added optional parameter which adds -lnsl -lsocket to make the Nix package manager work on OpenSolaris 2009-10-08 14:50:37 +00:00
Eelco Dolstra
c7057fc1f2 * And some more. 2009-09-24 07:39:55 +00:00
Eelco Dolstra
193f59e077 * Fix a build failure on Fedora 11. rename() needs <stdio.h>. 2009-09-24 07:21:29 +00:00
Eelco Dolstra
676e07902e * Darwin hack. 2009-09-23 18:04:55 +00:00
Eelco Dolstra
64e89980e8 * Create some state directories automatically as a convenience. 2009-09-23 17:05:51 +00:00
Rob Vermaas
48b58617e9 * include wait.h for WEXITSTATUS 2009-09-23 12:57:15 +00:00
Eelco Dolstra
14bc3ce3d6 * Canonicalise timestamps in the Nix store to 1 (1970-01-01 00:00:01
UTC) rather than 0 (00:00:00).  1 is a better choice because some
  programs use 0 as a special value.  For instance, the Template
  Toolkit uses a timestamp of 0 to denote the non-existence of a file,
  so it barfs on files in the Nix store (see
  template-toolkit-nix-store.patch in Nixpkgs).  Similarly, Maya 2008
  fails to load script directories with a timestamp of 0 and can't be
  patched because it's closed source.

  This will also shut up those "implausibly old time stamp" GNU tar
  warnings.
2009-06-13 16:30:58 +00:00
Eelco Dolstra
d5eab2fc82 * Use foreach in a lot of places. 2009-04-21 11:52:16 +00:00
Eelco Dolstra
8f1bf28505 * nix-store --verify: don't bail out if a referenced path is missing.
(It can't fix it though.)
2009-04-21 11:06:27 +00:00
Eelco Dolstra
4e646b0ddb * Fix a few "comparison is always false/true due to limited range of
data type" warnings on 64-bit platforms.  The one in parser.y is
  likely to be a real bug.
2009-04-16 12:03:17 +00:00
Eelco Dolstra
dfb863f333 * Don't cache transient build hook problems. 2009-04-15 06:25:02 +00:00
Eelco Dolstra
351bf658f9 * Do a substitution even if --max-jobs == 0. 2009-03-31 21:14:07 +00:00
Eelco Dolstra
cff2b2a13a * Doh! 2009-03-31 15:50:03 +00:00
Eelco Dolstra
160a60d663 2009-03-30 19:35:55 +00:00
Eelco Dolstra
805144b705 * Make the poll interval configurable. 2009-03-30 11:34:03 +00:00
Eelco Dolstra
7377195297 * With `--max-jobs 0', print a nicer error message than "Assertion
`!awake.empty()' failed."
2009-03-29 18:06:00 +00:00
Eelco Dolstra
737423a89c * Use polling to wait for a remote build slot when using a build hook
(that is, call the build hook with a certain interval until it
  accepts the build).
* build-remote.pl was totally broken: for all system types other than
  the local system type, it would send all builds to the *first*
  machine of the appropriate type.
2009-03-29 18:00:45 +00:00
Eelco Dolstra
6e946c8e72 * Scan for references and compute the SHA-256 hash of the output in
one pass.  This halves the amount of I/O.
2009-03-28 20:51:33 +00:00
Eelco Dolstra
c7152c8f97 * Don't use the non-standard __gnu_cxx::stdio_filebuf class. 2009-03-28 19:41:53 +00:00
Eelco Dolstra
3a2bbe7f8a * Simplify communication with the hook a bit (don't use file
descriptors 3/4, just use stdin/stderr).
2009-03-28 19:29:55 +00:00
Eelco Dolstra
f54e800366 * The `fixedOutput' variable didn't get initialised when using a build
hook, causing negative caching to fail randomly.
2009-03-28 16:12:27 +00:00
Eelco Dolstra
cb85bc396e * Create /nix/store if it doesn't exist. 2009-03-27 14:19:04 +00:00
Eelco Dolstra
5dd8fb2069 * Don't use ULLONG_MAX in maxFreed - use 0 to mean "no limit".
18446744073709551615ULL breaks on GCC 3.3.6 (`integer constant is
  too large for "long" type').
2009-03-26 11:02:07 +00:00
Eelco Dolstra
92f525ecf4 * Negative caching, i.e. caching of build failures. Disabled by
default.  This is mostly useful for Hydra.
2009-03-25 21:05:42 +00:00
Eelco Dolstra
a046858a22 * Doh. 2009-03-24 23:40:24 +00:00
Eelco Dolstra
12c8c64efa 2009-03-24 14:07:37 +00:00
Eelco Dolstra
cacff1be88 * No longer block while waiting for a lock on a store path. Instead
poll for it (i.e. if we can't acquire the lock, then let the main
  select() loop wait for at most a few seconds and then try again).
  This improves parallelism: if two nix-store processes are both
  trying to build a path at the same time, the second one shouldn't
  block; it should first see if it can build other goals.  Also, it
  prevents the deadlocks that have been occuring in Hydra lately,
  where a process waits for a lock held by another process that's
  waiting for a lock held by the first.

  The downside is that polling isn't really elegant, but POSIX doesn't
  provide a way to wait for locks in a select() loop.  The only
  solution would be to spawn a thread for each lock to do a blocking
  fcntl() and then signal the main thread, but that would require
  pthreads.
2009-03-23 01:05:54 +00:00
Eelco Dolstra
58969fa2bf * Refactoring. 2009-03-22 23:53:05 +00:00
Eelco Dolstra
77d272623f * NAR archives: handle files larger than 2^32 bytes. Previously it
would just silently store only (fileSize % 2^32) bytes.
* Use posix_fallocate if available when unpacking archives.
* Provide a better error message when trying to unpack something that
  isn't a NAR archive.
2009-03-22 17:36:43 +00:00
Eelco Dolstra
2897286487 * Unify exportReferencesGraph and exportBuildReferencesGraph, and make
sure that it works as expected when you pass it a derivation.  That
  is, we have to make sure that all build-time dependencies are built,
  and that they are all in the input closure (otherwise remote builds
  might fail, for example).  This is ensured at instantiation time by
  adding all derivations and their sources to inputDrvs and inputSrcs.
2009-03-18 17:36:42 +00:00
Eelco Dolstra
c183ee5c79 * Acquire the locks on the output paths before trying to run the build
hook.  This fixes a problem with log files being partially or
  completely filled with 0's because another nix-store process
  truncates the log file.  It should also be more efficient.
2009-03-18 14:48:42 +00:00
Eelco Dolstra
33ecb42991 * Cleanup. 2009-03-17 11:42:55 +00:00
Eelco Dolstra
60cb7de336 * Allow options from the Nix config file to be overriden from the
command line (e.g. "--option build-use-chroot true").
2009-02-27 11:04:41 +00:00
Eelco Dolstra
824b154ce8 * Release output locks as soon as possible, not when the destructor of
the DerivationGoal runs.  Otherwise, if a goal is a top-level goal,
  then the lock won't be released until nix-store finishes.  With
  --keep-going and lots of top-level goals, it's possible to run out
  of file descriptors (this happened sometimes in the build farm for
  Nixpkgs).  Also, for failed derivation, it won't be possible to
  build it again  until the lock is released.
  
* Idem for locks on build users: these weren't released in a timely
  manner for failed top-level derivation goals.  So if there were more
  than (say) 10 such failed builds, you would get an error about
  having run out of build users.
2009-02-16 09:24:20 +00:00
Eelco Dolstra
b682fae9d9 * Build hooks: use nix-store --import. This prevents a redundant
scan for runtime dependencies (i.e. the local machine shouldn't do a
  scan that the remote machine has already done).  Also pipe directly
  into `nix-store --import': don't use a temporary file.
2009-02-02 17:24:10 +00:00
Eelco Dolstra
061141e632 * Make it compile on Debian 4.0 (which doesn't define PER_LINUX32_3GB
in sys/personality.h).
2009-01-27 13:36:59 +00:00
Eelco Dolstra
019176137f * When using a build hook, distinguish between transient failures
(e.g. an SSH connection problem) and permanent failures (i.e. the
  builder failed).  This matters to Hydra (it wants to know whether it
  makes sense to retry a build).
2009-01-13 11:39:09 +00:00
Eelco Dolstra
c504d90c11 * Support i686-linux builds directly on x86_64-linux Nix
installations.  This is implemented using the personality() syscall,
  which causes uname to return "i686" in child processes.
2009-01-12 16:30:32 +00:00
Eelco Dolstra
8fce03e0ad * nix-store --verify: repair bad hash fields in the metadata file. 2008-12-16 13:28:18 +00:00
Eelco Dolstra
60ec75048a * Pass --use-atime / --max-atime to the daemon. 2008-12-16 12:23:35 +00:00
Eelco Dolstra
6f6bb1fdea * Delete the chroot if it already exists (e.g. left over from an
interrupted build).
2008-12-15 23:55:11 +00:00
Eelco Dolstra
92cb7c4dfe * Put chroots in the Nix store. This ensures that we can create hard
links to the inputs.
2008-12-12 17:14:57 +00:00
Eelco Dolstra
0008b0006d * Simplify deleting .lock files in /nix/store: just don't delete them
if they belong a path that's currently being built.  This gets rid
  of some Cygwin-specific code.
2008-12-12 17:03:18 +00:00
Eelco Dolstra
2b70a8e7c9 * Detect whether unshare() is available. 2008-12-12 13:41:36 +00:00
Eelco Dolstra
6776a52bb3 * Use a PathSet for the chroot directories so that we don't
accidentally bind-mount a directory twice.
2008-12-12 11:49:42 +00:00
Eelco Dolstra
d86bd22d24 * Define _GNU_SOURCE. Hopefully this fixes the build on Debian 4.0
(http://hydra.nixos.org/nixlog/384/1).
2008-12-12 10:20:19 +00:00
Eelco Dolstra
ac5478eb52 * Don't provide the whole Nix store in the chroot, but only the
closure of the inputs.  This really enforces that there can't be any
  undeclared dependencies on paths in the store.  This is done by
  creating a fake Nix store and creating bind-mounts or hard-links in
  the fake store for all paths in the closure.  After the build, the
  build output is moved from the fake store to the real store.  TODO:
  the chroot has to be on the same filesystem as the Nix store for
  this to work, but this isn't enforced yet.  (I.e. it only works
  currently if /tmp is on the same FS as /nix/store.)
2008-12-11 18:57:10 +00:00
Eelco Dolstra
652817046b * Revert r13150: now that we use private namespaces for the chroot, we
don't have to put the chroot in /nix/var/nix/chroots anymore.
  They're back in /tmp now.
2008-12-11 17:52:34 +00:00
Eelco Dolstra
5a569509b4 * Provide a minimal /etc/passwd in the chroot to keep some builders
happy.
2008-12-11 17:44:02 +00:00
Eelco Dolstra
7c54f1603f * Do chroot builds in a private namespace. This means that all the
bind-mounts we do are only visible to the builder process and its
  children.  So accidentally doing "rm -rf" on the chroot directory
  won't wipe out /nix/store and other bind-mounted directories
  anymore.  Also, the bind-mounts in the private namespace disappear
  automatically when the builder exits.
2008-12-11 17:00:12 +00:00
Eelco Dolstra
07cdfb09fb * Open the connection to the daemon lazily (on demand) so that
read-only operations (like nix-env -qa) work properly when the
  daemon isn't running.
2008-12-11 14:30:25 +00:00
Eelco Dolstra
a0766eca27 * Build on Debian 4.0 (GCC 4.1.1). 2008-12-04 21:07:29 +00:00
Eelco Dolstra
9ac3f5df9c * Propagate --max-silent-time to remote machines. 2008-12-04 16:51:16 +00:00
Eelco Dolstra
909fbb9de1 * When using build hooks, for any nix-store -r build operation, it is
necessary that at least one build hook doesn't return "postpone",
  otherwise nix-store will barf ("waiting for a build slot, yet there
  are no running children").  So inform the build hook when this is
  the case, so that it can start a build even when that would exceed
  the maximum load on a machine.
2008-12-04 14:29:41 +00:00
Eelco Dolstra
82ae85de27 * addToStore() in nix-worker: don't write the NAR dump received from
the client to a temporary directory, as that is highly inefficient.
2008-12-03 18:05:14 +00:00
Eelco Dolstra
ff762fb499 * Pass HashType values instead of strings. 2008-12-03 16:10:17 +00:00
Eelco Dolstra
1307b22223 * Made addToStore() a lot more efficient: it no longer reads the path
being copied 3 times in the worst case.  It doesn't run in constant space,
  but it didn't do that anyway.
2008-12-03 15:51:17 +00:00
Eelco Dolstra
64519cfd65 * Unify the treatment of sources copied to the store, and recursive
SHA-256 outputs of fixed-output derivations.  I.e. they now produce
  the same store path:

  $ nix-store --add x
  /nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x

  $ nix-store --add-fixed --recursive sha256 x
  /nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x

  the latter being the same as the path that a derivation

    derivation {
      name = "x";
      outputHashAlgo = "sha256";
      outputHashMode = "recursive";
      outputHash = "...";
      ...
    };

  produces.

  This does change the output path for such fixed-output derivations.
  Fortunately they are quite rare.  The most common use is fetchsvn
  calls with SHA-256 hashes.  (There are a handful of those is
  Nixpkgs, mostly unstable development packages.)
  
* Documented the computation of store paths (in store-api.cc).
2008-12-03 15:06:30 +00:00
Eelco Dolstra
a55113411f * Nix daemon: reload the configuration file after forking (NIX-100). 2008-11-20 12:25:11 +00:00
Eelco Dolstra
e13da525a7 * Files in the info directory starting with "." are temporary files
and don't indicate path validity.
2008-11-19 16:27:07 +00:00
Eelco Dolstra
6fedb7aa0f * Restore SIGPIPE to SIG_DFL when running the builder. This prevents
subtle and often hard-to-reproduce bugs where programs in pipes
  either barf with a "Broken pipe" message or not, depending on the
  exact timing conditions.  This particularly happened in GNU M4 (and
  Bison, which uses M4).
2008-11-14 15:46:45 +00:00
Eelco Dolstra
a519bb0635 * Some somewhat ad hoc mechanism to allow the build farm to monitor
build progress.
2008-11-12 11:08:27 +00:00
Eelco Dolstra
96598e7b06 * Pass the --no-build-output flag to the daemon. 2008-11-11 15:11:10 +00:00
Eelco Dolstra
4166b11a53 * Add /dev/pts to the default nix.conf. 2008-11-11 14:59:20 +00:00
Eelco Dolstra
709b55ee02 * Put the chroots under /nix/var/nix/chroots to reduce the risk of
disasters involving `rm -rf' on bind mounts.  Will try the
  definitive fix (per-process mounts, apparently possible via the
  CLONE_NEWNS flag in clone()) some other time.
2008-10-29 15:34:48 +00:00
Ludovic Courtès
c98ea254dc libstore: Always mount `/dev/pts' individually.
This fixes problems such as Tcl's PTY handling:

  ERROR: The system has no more ptys.  Ask your system administrator to
  create more.
2008-10-16 21:04:32 +00:00
Eelco Dolstra
9d6d50269b * Bug fix for building on some old installations (contributed by Pjotr). 2008-10-16 14:16:03 +00:00
Eelco Dolstra
f32fef1b07 * GC option `--max-atime' that specifies an upper limit to the last
accessed time of paths that may be deleted.  Anything more recently
  used won't be deleted.  The time is specified in time_t,
  e.g. seconds since 1970-01-01 00:00:00 UTC; use `date +%s' to
  convert to time_t from the command line. 

  Example: to delete everything that hasn't been used in the last two
  months:

  $ nix-store --gc -v --max-atime $(date +%s -d "2 months ago")
2008-09-17 14:52:35 +00:00
Eelco Dolstra
4af2fdba6d * Typo. 2008-09-17 13:00:55 +00:00
Eelco Dolstra
c987061aa4 * Some refactoring. Better output with `-v' for --use-atime. 2008-09-17 12:54:07 +00:00
Eelco Dolstra
7ab68961e4 * Garbage collector: added an option `--use-atime' to delete paths in
order of ascending last access time.  This is useful in conjunction
  with --max-freed or --max-links to prefer deleting non-recently used
  garbage, which is good (especially in the build farm) since garbage
  may become live again.

  The code could easily be modified to accept other criteria for
  ordering garbage by changing the comparison operator used by the
  priority queue in collectGarbage().
2008-09-17 10:02:55 +00:00
Eelco Dolstra
7718b19389 * Explicitly set PWD to prevent problems with chroot builds. In
particular, dietlibc cannot figure out the cwd because the inode of
  the current directory doesn't appear in .. (because getdents returns
  the inode of the mount point).
2008-08-27 17:20:25 +00:00
Eelco Dolstra
9cc0da8453 * Create a /tmp with 1777 permission in the chroot. Some builders
need a writable /tmp (they don't respect $TMPDIR).
2008-08-27 16:03:03 +00:00
Eelco Dolstra
d06be428f6 * Disable chroot builds for fixed-output derivations so that we don't
need /etc in the chroot (in particular, /etc/resolv.conf for
  fetchurl).  Not having /etc/resolv.conf in the chroot is a good
  thing, since we don't want normal derivations to download files.
2008-08-25 15:49:22 +00:00
Eelco Dolstra
b428adc267 * Strip off the `.nix' suffix from the attribute name for files in
~/.nix-defexpr,  otherwise the attribute cannot be selected with the
  `-A' option.  Useful if you want to stick a Nix expression directly
  in ~/.nix-defexpr.
2008-08-25 13:31:57 +00:00
Eelco Dolstra
db4f4a8425 * Backward compatibility check to prevent nixos-rebuild from barfing
when upgrading Nix.
2008-08-14 13:02:19 +00:00
Eelco Dolstra
c03b729319 * Increase the sleep periods a bit to make the test less likely to
fail on slow machines.  Of course it would be better if this test
  wasn't timing dependent...
2008-08-14 09:26:30 +00:00
Eelco Dolstra
98b07466fb * Better error checking of the data from the substituters. 2008-08-05 10:57:53 +00:00
Eelco Dolstra
5adbb0aabe * build.cc: only use a substituter if it returns info for a path. 2008-08-04 13:15:35 +00:00
Eelco Dolstra
c4f98941ed * nix-env --dry-run: show the total size of the substituter
downloads.
2008-08-04 12:29:04 +00:00
Eelco Dolstra
03427e76f1 * querySubstitutablePathInfo: work properly when run via the daemon.
* --dry-run: print the paths that we don't know how to build/substitute.
2008-08-04 11:44:50 +00:00
Eelco Dolstra
3c92ea399d * Make nix-env --dry-run print the paths to be substituted correctly
again.  (After the previous substituter mechanism refactoring I
  didn't update the code that obtains the references of substitutable
  paths.)  This required some refactoring: the substituter programs
  are now kept running and receive/respond to info requests via
  stdin/stdout.
2008-08-02 12:54:35 +00:00
Eelco Dolstra
989176c56e * Allow read-only access to the store (e.g., non-root users on NixOS
can do operations like "nix-store -qR <path>" even without the Nix
  daemon).
2008-07-18 15:34:46 +00:00
Eelco Dolstra
6c8641a542 2008-06-18 19:17:05 +00:00
Eelco Dolstra
d3aa183beb * Garbage collector: option `--max-freed' to stop after at least N
bytes have been freed, `--max-links' to stop when the Nix store
  directory has fewer than N hard links (the latter being important
  for very large Nix stores on filesystems with a 32000 subdirectories
  limit).
2008-06-18 14:20:16 +00:00
Eelco Dolstra
a8f3b02092 * `nix-store --optimise': handle files with >= 32000 hard links.
(There can easily be more than 32000 occurrences of the empty file.)
2008-06-18 14:13:00 +00:00
Eelco Dolstra
a72709afd8 * Some refactoring: put the GC options / results in separate structs.
* The garbage collector now also prints the number of blocks freed.
2008-06-18 09:34:17 +00:00
Eelco Dolstra
94fd46fa1c * Note. 2008-06-14 16:02:31 +00:00
Eelco Dolstra
826b271d9a * Garbage collector: don't do a complete topological sort of the Nix
store under the reference relation, since that means that the
  garbage collector will need a long time to start deleting paths.
  Instead just delete the referrers of a path first.
2008-06-13 18:25:24 +00:00
Eelco Dolstra
30c9f909b2 * Print some progress info during the early GC stages. 2008-06-13 17:21:20 +00:00
Eelco Dolstra
b0e92f6d47 * Merged the no-bdb branch (-r10900:HEAD
https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2008-06-09 13:52:45 +00:00
Eelco Dolstra
bd955e15e1 * GCC 4.3.0 (Fedora 9) compatibility fixes. Reported by Gour and
Armijn Hemel.
2008-05-21 11:17:31 +00:00
Eelco Dolstra
72034ab35d * sockaddr_un doesn't allow path names of more than 108 characters.
This isn't usually a problem, except that it causes tests to fail
  when performed in a directory with a very long path name.  So chdir
  to the socket directory and use a relative path name.
2008-04-09 05:57:01 +00:00
Eelco Dolstra
329025253d * Use /tmp/nix-build-<drvpath>-<counter> instead of
/tmp/nix-<pid>-<counter> for temporary build directories.  This
  increases purity a bit: many packages store the temporary build path
  in their output, causing (generally unimportant) binary differences.
2008-03-27 13:45:17 +00:00
Eelco Dolstra
f789ea1d09 * A transaction is unnecessary since the path cannot be
garbage-collected (it's a temporary root).
2008-03-01 21:05:33 +00:00
Eelco Dolstra
66c51dc215 * nix-store --dump-db / --load-db to dump/load the Nix DB.
* nix-store --register-validity: option to supply the content hash of
  each path.
* Removed compatibility with Nix <= 0.7 stores.
2008-01-29 18:17:36 +00:00
Michael Raskin
5eb5c23447 Fixed exportBuildReferenceGraph 2008-01-15 04:32:08 +00:00
Michael Raskin
2196cb67c5 Added a way to include derivation with its outputs and dependencies - exportBuildReferenceGraph 2007-12-30 09:30:56 +00:00
Eelco Dolstra
633518628f * nix-env -e: support uninstalling by path, so that one can say
$ nix-env -e $(which firefox)

  or

    $ nix-env -e /nix/store/nywzlygrkfcgz7dfmhm5xixlx1l0m60v-pan-0.132

* nix-env -i: if an argument contains a slash anywhere, treat it as a
  path and follow it through symlinks into the Nix store.  This allows
  things like

    $ nix-build -A firefox
    $ nix-env -i ./result

* nix-env -q/-i/-e: don't complain when the `*' selector doesn't match
  anything.  In particular, `nix-env -q \*' doesn't fail anymore on an
  empty profile.
2007-11-29 16:18:24 +00:00
Eelco Dolstra
c370755583 * Flag `--no-build-hook' to disable distributed builds.
* queryDeriver in daemon mode: don't barf if the other side returns an
  empty string (which means there is no deriver).
2007-11-16 16:15:26 +00:00
Eelco Dolstra
149e273023 * Don't need gc.hh. 2007-11-15 15:07:27 +00:00
Michael Raskin
6fc60e2060 Added possibility to specify garbage collection levels for store paths; so packages can share intermediate results of compilation and GC will collect it automatically while never touching tarballs, for example. 2007-11-15 03:47:12 +00:00
Eelco Dolstra
42d80d1170 * On FreeBSD, sys/mount.h needs sys/param.h. 2007-10-29 10:03:07 +00:00
Eelco Dolstra
d91cd30563 * Detect whether chroot / bind-mount support is available. 2007-10-27 16:51:55 +00:00
Eelco Dolstra
dc6f373842 * Delete the chroot directory automatically.
* Removed some debug messages.
2007-10-27 16:06:38 +00:00
Eelco Dolstra
9397cd30c8 * Support for doing builds in a chroot under Linux. The builder is
executed in a chroot that contains just the Nix store, the temporary
  build directory, and a configurable set of additional directories
  (/dev and /proc by default).  This allows a bit more purity
  enforcement: hidden build-time dependencies on directories such as
  /usr or /nix/var/nix/profiles are no longer possible.  As an added
  benefit, accidental network downloads (cf. NIXPKGS-52) are prevented
  as well (because files such as /etc/resolv.conf are not available in
  the chroot).

  However the usefulness of chroots is diminished by the fact that
  many builders depend on /bin/sh, so you need /bin in the list of
  additional directories.  (And then on non-NixOS you need /lib as
  well...)
2007-10-27 00:46:59 +00:00
Eelco Dolstra
d04291cfab * Doh! Don't change the permissions on /nix/store. 2007-10-10 13:52:38 +00:00
Eelco Dolstra
315183f194 * nix-store --optimise: flag "--dry-run" to just query what the disk
savings would be.
2007-10-10 13:43:04 +00:00
Eelco Dolstra
a8629de827 * New command `nix-store --optimise' to reduce Nix store disk space
usage by finding identical files in the store and hard-linking them
  to each other.  It typically reduces the size of the store by
  something like 25-35%.  This is what the optimise-store.pl script
  did, but the new command is faster and more correct (it's safe wrt
  garbage collection and concurrent builds).
2007-10-09 22:14:27 +00:00
Eelco Dolstra
f3441e6122 * Pass various options to the worker so that flags like -K or -j work
in multi-user Nix (NIX-72).
* Client/worker: exchange a protocol version number for future
  compatibility.
2007-09-18 09:11:20 +00:00
Eelco Dolstra
0d65fc08e2 * Create the Nix daemon socket in a separate directory
(/nix/var/nix/daemon-socket).  This allows access to the Nix daemon
  to be restricted by setting the mode/ownership on that directory as
  desired, e.g.

    $ chmod 770 /nix/var/nix/daemon-socket
    $ chown root.wheel /nix/var/nix/daemon-socket

  to allow only users in the wheel group to use Nix.

  Setting the ownership on a socket is much trickier, since the socket
  must be deleted and recreated every time the daemon is started
  (which would require additional Nix configuration file directives to
  specify the mode/ownership, and wouldn't support arbitrary ACLs),
  some BSD variants appear to ignore permissions on sockets, and it's
  not clear whether the umask is respected on every platform when
  creating sockets.
2007-08-30 09:50:44 +00:00
Eelco Dolstra
cb1c1004cd * When there are multiple substituters, make sure to release the
lock on the output path after trying each.  Otherwise the
  pathIsLockedByMe() test gets confused.
2007-08-28 16:22:08 +00:00
Eelco Dolstra
c970b28ba0 * Fix a race condition with parallel builds where multiple
fixed-output derivations or substitutions try to build the same
  store path at the same time.  Locking generally catches this, but
  not between multiple goals in the same process.  This happened
  especially often (actually, only) in the build farm with fetchurl
  downloads of the same file being executed on multiple machines and
  then copied back to the main machine where they would clobber each
  other (NIXBF-13).

  Solution: if a goal notices that the output path is already locked,
  then go to sleep until another goal finishes (hopefully the one
  locking the path) and try again.
2007-08-28 11:36:17 +00:00
Eelco Dolstra
e0d7e47862 * PathLocks::lockPaths: don't allow reacquiring a lock we already
hold.
2007-08-28 09:39:03 +00:00
Eelco Dolstra
3757ee589f * Bump the Nix database schema version number; delete the substitutes
table.
2007-08-13 11:37:39 +00:00
Eelco Dolstra
59afc1a15c * Backwards compatibility. 2007-08-13 11:15:13 +00:00
Eelco Dolstra
9e975458b4 * Get rid of the substitutes database table (NIX-47). Instead, if we
need any info on substitutable paths, we just call the substituters
  (such as download-using-manifests.pl) directly.  This means that
  it's no longer necessary for nix-pull to register substitutes or for
  nix-channel to clear them, which makes those operations much faster
  (NIX-95).  Also, we don't have to worry about keeping nix-pull
  manifests (in /nix/var/nix/manifests) and the database in sync with
  each other.

  The downside is that there is some overhead in calling an external
  program to get the substitutes info.  For instance, "nix-env -qas"
  takes a bit longer.

  Abolishing the substitutes table also makes the logic in
  local-store.cc simpler, as we don't need to store info for invalid
  paths.  On the downside, you cannot do things like "nix-store -qR"
  on a substitutable but invalid path (but nobody did that anyway).

* Never catch interrupts (the Interrupted exception).
2007-08-12 00:29:28 +00:00
Eelco Dolstra
6d1a1191b0 * Support queryDeriver() in multi-user installations. 2007-06-12 16:53:44 +00:00
Eelco Dolstra
9bff7ad728 * Check against creation of GC roots in the store. Those roots don't
work, because findRoots() stops when it encounters a symlink to the
  store.  And of course the store is supposed to be read-only.
2007-06-11 11:36:22 +00:00
Eelco Dolstra
7046c35843 * Typo (reported by Marc Weber). 2007-05-14 12:16:41 +00:00
Eelco Dolstra
e5da9c8803 * Create the database directory if it doesn't exist. 2007-05-07 16:59:19 +00:00
Eelco Dolstra
cbfac2fdcc * Set a terminate() handler to ensure that we leave the BDB
environment cleanly even when an exception is thrown from a
  destructor.  We still crash, but we don't take all other Nix
  processes with us.
2007-05-01 15:16:17 +00:00
Eelco Dolstra
644946feed * Make --verify more interruptable. 2007-05-01 13:21:05 +00:00
Eelco Dolstra
ae7990cc88 * Work around a bug in Apple's GCC preprocessor. 2007-03-30 13:24:35 +00:00
Eelco Dolstra
803cb6e3b9 * Override the setuid helper using NIX_SETUID_HELPER. 2007-03-20 22:04:25 +00:00
Eelco Dolstra
8ab229ddf2 * Terminate build hooks and substitutes with a TERM signal, not a KILL
signal.  This is necessary because those processes may have joined
  the BDB environment, so they have to be given a chance to clean up.
  (NIX-85)
2007-03-19 12:48:45 +00:00
Eelco Dolstra
b2b6cf3fc8 * Undocumented option `gc-check-reachability' to allow reachability
checking to be turned off on machines with way too many roots.
2007-03-19 09:16:47 +00:00
Eelco Dolstra
917e06bf63 * Delete the output paths before invoking the build hook. 2007-03-07 15:53:11 +00:00
Eelco Dolstra
db1973d012 * Look for the openssl program at compile time. If not found, call
openssl through $PATH at runtime.
2007-03-01 13:30:46 +00:00
Eelco Dolstra
b4a040e52b * Don't check the signature unless we have to. 2007-03-01 12:30:24 +00:00
Eelco Dolstra
2ea3bebc23 * Doh! The deriver can be empty. 2007-02-27 23:18:57 +00:00
Eelco Dolstra
9da367b7d5 * `nix-store -qR' and friends: print the paths sorted topologically
under the references relation.  This is useful for commands that
  want to copy paths to another Nix store in the right order.
2007-02-21 22:45:10 +00:00
Eelco Dolstra
65f195f4c7 * Check that the file containing the secret key is secret. 2007-02-21 17:51:10 +00:00
Eelco Dolstra
bdadb98de8 * `nix-store --import' now also works in remote mode. The worker
always requires a signature on the archive.  This is to ensure that
  unprivileged users cannot add Trojan horses to the Nix store.
2007-02-21 17:34:02 +00:00
Eelco Dolstra
0f5da8a83c * Support exportPath() in remote mode. 2007-02-21 16:34:00 +00:00
Eelco Dolstra
dc7d594776 * importPath(): set the deriver.
* exportPath(): lock the path, use a transaction.
2007-02-21 16:23:25 +00:00
Eelco Dolstra
43c4d18c6a * nix-store --import': import an archive created by nix-store
--export' into the Nix store, and optionally check the cryptographic
  signatures against /nix/etc/nix/signing-key.pub.  (TODO: verify
  against a set of public keys.)
2007-02-21 15:45:32 +00:00
Eelco Dolstra
46e0919ced * `nix-store --export --sign': sign the Nix archive using the RSA key
in /nix/etc/nix/signing-key.sec
2007-02-21 14:31:42 +00:00
Eelco Dolstra
b824a1daee * Start of `nix-store --export' operation for serialising a store
path.  This is like `nix-store --dump', only it also dumps the
  meta-information of the store path (references, deriver).  Will add
  a `--sign' flag later to add a cryptographic signature, which we
  will use for exchanging store paths between build farm machines in a
  secure manner.
2007-02-20 23:17:20 +00:00
Eelco Dolstra
46605fb4f5 * Fix 64-bit compiler warnings. 2007-02-06 20:03:53 +00:00
Eelco Dolstra
f52de527c7 * Doh! 2007-01-29 15:55:49 +00:00
Eelco Dolstra
b618fa6eb6 * computeStorePathForText: take the references into account when
computing the store path (NIX-77).  This is an important security
  property in multi-user Nix stores.

  Note that this changes the store paths of derivations (since the
  derivation aterms are added using addTextToStore), but not most
  outputs (unless they use builtins.toFile).
2007-01-29 15:51:37 +00:00
Eelco Dolstra
fac63d6416 * exportReferencesGraph: work on paths within store paths as well. 2007-01-23 16:57:43 +00:00
Eelco Dolstra
63f3ce6d9a * `nix-store --verify': revive checking the referrers table. This is
important to get garbage collection to work if there is any
  inconsistency in the database (because the referrer table is used to
  determine whether it is safe to delete a path).
* `nix-store --verify': show some progress.
2007-01-14 17:28:30 +00:00
Eelco Dolstra
8f67b35886 * Make the garbage collector more resilient to certain consistency
errors: in-use paths now cause a warning, not a fatal error.
2007-01-14 16:24:49 +00:00
Eelco Dolstra
a3e6415ba8 * New primop builtins.filterSource, which can be used to filter files
from a source directory.  All files for which a predicate function
  returns true are copied to the store.  Typical example is to leave
  out the .svn directory:

    stdenv.mkDerivation {
      ...
      src = builtins.filterSource
        (path: baseNameOf (toString path) != ".svn")
        ./source-dir;
      # as opposed to
      #   src = ./source-dir;
    }

  This is important because the .svn directory influences the hash in
  a rather unpredictable and variable way.
2006-12-12 23:05:01 +00:00
Eelco Dolstra
b17677462c * Use lchown() instead of chown() in canonicalisePathMetaData(). This
matters when running as root, since then we don't use the setuid
  helper (which already used lchown()).
  
* Also check for an obscure security problem on platforms that don't
  have lchown.  Then we can't change the ownership of symlinks, which
  doesn't matter *except* when the containing directory is writable by
  the owner (which is the case with the top-level Nix store directory).
2006-12-09 20:02:27 +00:00
Eelco Dolstra
5f681988f2 * Use deletePathWrapped() in more places. 2006-12-09 00:26:24 +00:00
Eelco Dolstra
fa33303146 * Goal cancellation inside the waitForInput() loop needs to be handled
very carefully, since it can invalidate iterators into the
  `children' map.
2006-12-08 18:41:48 +00:00
Eelco Dolstra
06c4929958 * Some refactoring.
* Throw more exceptions as BuildErrors instead of Errors.  This
  matters when --keep-going is turned on.  (A BuildError is caught
  and terminates the goal in question, an Error terminates the
  program.)
2006-12-08 17:26:21 +00:00
Eelco Dolstra
9dbfe242e3 * Kill a build if it has gone for more than a certain number of
seconds without producing output on stdout or stderr (NIX-65).  This
  timeout can be specified using the `--max-silent-time' option or the
  `build-max-silent-time' configuration setting.  The default is
  infinity (0).

* Fix a tricky race condition: if we kill the build user before the
  child has done its setuid() to the build user uid, then it won't be
  killed, and we'll potentially lock up in pid.wait().  So also send a
  conventional kill to the child.
2006-12-08 15:44:00 +00:00
Eelco Dolstra
d3fe6ab024 * Also for convenience, change the ownership of the build output even
in case of failure.
2006-12-08 00:19:50 +00:00
Eelco Dolstra
096194ab29 * Remove ancient terminology. 2006-12-07 23:58:36 +00:00
Eelco Dolstra
6833e8bbe8 * When keeping the temporary build directory (-K), change the owner
back to the Nix account.
2006-12-07 23:27:40 +00:00
Eelco Dolstra
2819eb36a4 * Be less verbose. 2006-12-07 21:43:35 +00:00
Eelco Dolstra
c3286ec020 * Don't count on the Pid deconstructor to kill the child process,
since if we're running a build user in non-root mode, we can't.  Let
  the setuid helper do it.
2006-12-07 17:52:58 +00:00
Eelco Dolstra
f76fdb6d42 * If not running as root, let the setuid helper kill the build user's
processes before and after the build.
2006-12-07 16:33:31 +00:00
Eelco Dolstra
ec23ecc64d * In the garbage collector, if deleting a path fails, try to fix its
ownership, then try again.
2006-12-07 15:54:52 +00:00
Eelco Dolstra
a0a43c3206 * When not running as root, call the setuid helper to change the
ownership of the build result after the build.
2006-12-07 15:18:14 +00:00
Eelco Dolstra
6a07ff1ec0 * Change the ownership of store paths to the Nix account before
deleting them using the setuid helper.
2006-12-07 14:14:35 +00:00
Eelco Dolstra
7d8cf316ee * Pass the actual build user to the setuid helper. 2006-12-07 11:27:32 +00:00
Eelco Dolstra
a45c498e4e * If Nix is not running as root, call the setuid helper to start the
builder under the desired build user.
2006-12-07 00:42:30 +00:00
Eelco Dolstra
813a7c65c9 * Sanity check. 2006-12-07 00:19:27 +00:00
Eelco Dolstra
6a8e60913a * Move killUser() to libutil so that the setuid helper can use it. 2006-12-07 00:16:07 +00:00
Eelco Dolstra
6e5ec1029a * Get rid of `build-users'. We'll just take all the members of
`build-users-group'.  This makes configuration easier: you can just
  add users in /etc/group.
2006-12-06 20:00:15 +00:00
Eelco Dolstra
9f0efa6611 * Start of the setuid helper (the program that performs the operations
that have to be done as root: running builders under different uids,
  changing ownership of build results, and deleting paths in the store
  with the wrong ownership).
2006-12-06 01:24:02 +00:00
Eelco Dolstra
2b558843a2 * Be less chatty. 2006-12-05 19:01:19 +00:00
Eelco Dolstra
44cad9630f * Urgh. Do setgid() before setuid(), because the semantics of setgid()
changes completely depending on whether you're root...
2006-12-05 18:28:15 +00:00
Eelco Dolstra
4c1c37d0b6 * FreeBSD returns ESRCH when there are no processes to kill. 2006-12-05 18:07:46 +00:00
Eelco Dolstra
62b0497c0f * Better message. 2006-12-05 16:17:01 +00:00
Eelco Dolstra
a9c4f66cfb * Allow unprivileged users to run the garbage collector and to do
`nix-store --delete'.  But unprivileged users are not allowed to
  ignore liveness.
* `nix-store --delete --ignore-liveness': ignore the runtime roots as
  well.
2006-12-05 02:18:46 +00:00
Eelco Dolstra
29cf434a35 * The determination of the root set should be made by the privileged
process, so forward the operation.
* Spam the user about GC misconfigurations (NIX-71).
* findRoots: skip all roots that are unreadable - the warnings with
  which we spam the user should be enough.
2006-12-05 01:31:45 +00:00
Eelco Dolstra
8623256f48 * findRoots: return a map from the symlink (outside of the store) to
the store path (inside the store).
2006-12-05 00:48:36 +00:00
Eelco Dolstra
d27a73b1a9 * In addPermRoot, check that the root that we just registered can be
found by the garbage collector.  This addresses NIX-71 and is a
  particular concern in multi-user stores.
2006-12-05 00:34:42 +00:00
Eelco Dolstra
74033a844f * Add indirect root registration to the protocol so that unprivileged
processes can register indirect roots.  Of course, there is still
  the problem that the garbage collector can only read the targets of
  the indirect roots when it's running as root...
2006-12-04 23:29:16 +00:00
Eelco Dolstra
7751160e9f * Don't redirect stderr. 2006-12-04 19:10:23 +00:00
Eelco Dolstra
40c3529909 * Handle exceptions and stderr for all protocol functions.
* SIGIO -> SIGPOLL (POSIX calls it that).
* Use sigaction instead of signal to register the SIGPOLL handler.
  Sigaction is better defined, and a handler registered with signal
  appears not to interrupt fcntl(..., F_SETLKW, ...), which is bad.
2006-12-04 17:55:14 +00:00
Eelco Dolstra
0130ef88ea * Daemon mode (`nix-worker --daemon'). Clients connect to the server
via the Unix domain socket in /nix/var/nix/daemon.socket.  The
  server forks a worker process per connection.
* readString(): use the heap, not the stack.
* Some protocol fixes.
2006-12-04 17:17:13 +00:00
Eelco Dolstra
4740baf3a6 * When NIX_REMOTE=daemon, connect to /nix/var/nix/daemon.socket
instead of forking a worker.
2006-12-04 14:21:39 +00:00
Eelco Dolstra
f5f0cf423f * Refactoring. 2006-12-04 13:28:14 +00:00
Eelco Dolstra
052b6fb149 * Pass the verbosity level to the worker. 2006-12-04 13:15:29 +00:00
Eelco Dolstra
1e16d20655 * Install the worker in bindir, not libexecdir.
* Allow the worker path to be overriden through the NIX_WORKER
  environment variable.
2006-12-04 13:09:16 +00:00
Eelco Dolstra
f4279bcde0 * Don't run setuid root when build-users is empty.
* Send startup errors to the client.
2006-12-03 16:25:19 +00:00
Eelco Dolstra
35247c4c9f * Removed `build-allow-root'.
* Added `build-users-group', the group under which builds are to be
  performed.
* Check that /nix/store has 1775 permission and is owner by the
  build-users-group.
2006-12-03 15:32:38 +00:00
Eelco Dolstra
4251f94b32 * Use a Unix domain socket instead of pipes. 2006-12-03 02:36:44 +00:00
Eelco Dolstra
8c76df93e6 * Better error message if the worker doesn't start. 2006-12-03 02:22:04 +00:00
Eelco Dolstra
7951c3c546 * Some hackery to propagate the worker's stderr and exceptions to the
client.
2006-12-03 02:08:13 +00:00
Eelco Dolstra
e25fad691a * Move addTempRoot() to the store API, and add another function
syncWithGC() to allow clients to register GC roots without needing
  write access to the global roots directory or the GC lock.
2006-12-02 16:41:36 +00:00
Eelco Dolstra
9c9cdb06d0 * Remove SwitchToOriginalUser, we're not going to need it anymore. 2006-12-02 14:34:14 +00:00
Eelco Dolstra
8ba5d32769 * Remove queryPathHash().
* Help for nix-worker.
2006-12-02 14:27:24 +00:00
Eelco Dolstra
fcd9900d74 * Replace read-only calls to addTextToStore. 2006-12-01 21:00:39 +00:00
Eelco Dolstra
a824d58b56 * Merge addToStore and addToStoreFixed.
* addToStore now adds unconditionally, it doesn't use readOnlyMode.
  Read-only operation is up to the caller (who can call
  computeStorePathForPath).
2006-12-01 20:51:18 +00:00
Eelco Dolstra
b0d8e05be1 * More operations.
* addToStore() and friends: don't do a round-trip to the worker if
  we're only interested in the path (i.e., in read-only mode).
2006-12-01 18:00:01 +00:00
Eelco Dolstra
0565b5f2b3 * More remote operations.
* Added new operation hasSubstitutes(), which is more efficient than
  querySubstitutes().size() > 0.
2006-11-30 22:43:55 +00:00
Eelco Dolstra
aac547a8b3 * Doh. 2006-11-30 21:32:46 +00:00
Eelco Dolstra
0263279071 * More operations. 2006-11-30 20:45:20 +00:00
Eelco Dolstra
a711689368 * First remote operation: isValidPath(). 2006-11-30 20:13:59 +00:00
Eelco Dolstra
765bdfe542 * When NIX_REMOTE is set to "slave", fork off nix-worker in slave
mode.  Presumably nix-worker would be setuid to the Nix store user.
  The worker performs all operations on the Nix store and database, so
  the caller can be completely unprivileged.

  This is already much more secure than the old setuid scheme, since
  the worker doesn't need to do Nix expression evaluation and so on.
  Most importantly, this means that it doesn't need to access any user
  files, with all resulting security risks; it only performs pure
  store operations.

  Once this works, it is easy to move to a daemon model that forks off
  a worker for connections established through a Unix domain socket.
  That would be even more secure.
2006-11-30 19:54:43 +00:00
Eelco Dolstra
40b3f64b55 * Skeleton of the privileged worker program.
* Some refactoring: put the NAR archive integer/string serialisation
  code in a separate file so it can be reused by the worker protocol
  implementation.
2006-11-30 19:19:59 +00:00
Eelco Dolstra
9adc074dc3 * Oops. 2006-11-30 18:35:50 +00:00
Eelco Dolstra
9cf1948993 * Skeleton of remote store implementation. 2006-11-30 18:35:36 +00:00
Eelco Dolstra
6ecb840fd1 * Put building in the store API. 2006-11-30 18:02:04 +00:00
Eelco Dolstra
e2ef5e07fd * Refactoring. There is now an abstract interface class StoreAPI
containing functions that operate on the Nix store.  One
  implementation is LocalStore, which operates on the Nix store
  directly.  The next step, to enable secure multi-user Nix, is to
  create a different implementation RemoteStore that talks to a
  privileged daemon process that uses LocalStore to perform the actual
  operations.
2006-11-30 17:43:04 +00:00
Eelco Dolstra
c6a97e3b74 * Doh! Path sizes need to be computed recursively of course.
(NIX-70)
2006-11-24 20:24:14 +00:00
Eelco Dolstra
a76efaeb3f * Dead files. 2006-11-24 20:07:30 +00:00
Eelco Dolstra
f459a5bb3a * Remove the undocumented `noscan' feature. It's no longer necessary
now that reference scanning is sufficiently streamy.
2006-11-13 18:19:05 +00:00
Eelco Dolstra
e2a70b7ec0 * Magic attribute `exportReferencesGraph' that allows the references
graph to be passed to a builder.  This attribute should be a list of
  pairs [name1 path1 name2 path2 ...].  The references graph of each
  `pathN' will be stored in a text file `nameN' in the temporary build
  directory.  The text files have the format used by `nix-store
  --register-validity'.  However, the deriver fields are left empty.

  `exportReferencesGraph' is useful for builders that want to do
  something with the closure of a store path.  Examples: the builders
  that make initrds and ISO images for NixOS.

  `exportReferencesGraph' is entirely pure.  It's necessary because
  otherwise the only way for a builder to get this information would
  be to call `nix-store' directly, which is not allowed (though
  unfortunately possible).
2006-11-13 18:18:13 +00:00
Eelco Dolstra
ae6fb27f18 * `nix-store --read-log / -l PATH' shows the build log of PATH, if
available.  For instance,

    $ nix-store -l $(which svn) | less

  lets you read the build log of the Subversion instance in your
  profile.

* `nix-store -qb': if applied to a non-derivation, take the deriver.
2006-10-28 16:33:54 +00:00
Eelco Dolstra
6a67556f71 * Special derivation attribute `allowedReferences' that causes Nix to
check that the references of the output of a derivation are in the
  specified set.  For instance,

    allowedReferences = [];

  specifies that the output cannot have any references.  (This is
  useful, for instance, for the generation of bootstrap binaries for
  stdenv-linux, which must not have any references for purity).  It
  could also be used to guard against undesired runtime dependencies,
  e.g.,

    {gcc, dynlib}: derivation {
      ...
      allowedReferences = [dynlib];
    }

  says that the output can refer to the path of `dynlib' but not
  `gcc'.  A `forbiddedReferences' attribute would be more useful for
  this, though.
2006-10-19 16:09:24 +00:00
Eelco Dolstra
d7efd76394 * Big cleanup of the semantics of paths, strings, contexts, string
concatenation and string coercion.  This was a big mess (see
  e.g. NIX-67).  Contexts are now folded into strings, so that they
  don't cause evaluation errors when they're not expected.  The
  semantics of paths has been clarified (see nixexpr-ast.def).
  toString() and coerceToString() have been merged.

  Semantic change: paths are now copied to the store when they're in a
  concatenation (and in most other situations - that's the
  formalisation of the meaning of a path).  So

    "foo " + ./bla

  evaluates to "foo /nix/store/hash...-bla", not "foo
  /path/to/current-dir/bla".  This prevents accidental impurities, and
  is more consistent with the treatment of derivation outputs, e.g.,
  `"foo " + bla' where `bla' is a derivation.  (Here `bla' would be
  replaced by the output path of `bla'.)
2006-10-16 15:55:34 +00:00
Eelco Dolstra
37c8a664f3 * A helpful message. 2006-10-13 11:49:55 +00:00
Eelco Dolstra
e4af398681 * Don't crash when upgrading the Berkeley DB environment. 2006-10-13 11:15:53 +00:00
Eelco Dolstra
666babbbfa * Use a bounded amount of memory in scanForReferences() by not reading
regular files into memory all at once.
2006-09-22 13:10:30 +00:00