Commit graph

217 commits

Author SHA1 Message Date
Eelco Dolstra
9958bd6992 Add operation ‘nix-store --repair-path’
This operation allows fixing corrupted or accidentally deleted store
paths by redownloading them using substituters, if available.

Since the corrupted path cannot be replaced atomically, there is a
very small time window (one system call) during which neither the old
(corrupted) nor the new (repaired) contents are available.  So
repairing should be used with some care on critical packages like
Glibc.
2012-10-02 14:08:59 -04:00
Eelco Dolstra
e94806d030 Merge branch 'master' into no-manifests 2012-08-27 11:09:07 -04:00
Eelco Dolstra
a9e6752bbd Include the output name in the GC root link
Output names are now appended to resulting GC symlinks, e.g. by
nix-build.  For backwards compatibility, if the output is named "out",
nothing is appended.  E.g. doing "nix-build -A foo" on a derivation
that produces outputs "out", "bin" and "dev" will produce symlinks
"./result", "./result-bin" and "./result-dev", respectively.
2012-08-24 16:58:11 -04:00
Eelco Dolstra
01d56c1eec Drop the block count in the garbage collector 2012-08-01 22:34:46 -04:00
Eelco Dolstra
967d066d8e nix-store --gc: Make ‘--max-freed 0’ do the right thing
That is, delete almost nothing (it will still remove unused links from
/nix/store/.links).
2012-08-01 19:14:58 -04:00
Eelco Dolstra
97421eb5ec Refactor settings processing
Put all Nix configuration flags in a Settings object.
2012-07-30 19:55:41 -04:00
Eelco Dolstra
8c79100839 Merge branch 'master' into no-manifests 2012-07-26 15:14:33 -04:00
Eelco Dolstra
564fb7d9fa optimiseStore(): Use a content-addressed file store in /nix/store/.links
optimiseStore() now creates persistent, content-addressed hard links
in /nix/store/.links.  For instance, if it encounters a file P with
hash H, it will create a hard link

  P' = /nix/store/.link/<H>

to P if P' doesn't already exist; if P' exist, then P is replaced by a
hard link to P'.  This is better than the previous in-memory map,
because it had the tendency to unnecessarily replace hard links with a
hard link to whatever happened to be the first file with a given hash
it encountered.  It also allows on-the-fly, incremental optimisation.
2012-07-23 17:14:15 -04:00
Eelco Dolstra
667d5f1936 Rename queryValidPaths() to queryAllValidPaths() 2012-07-11 10:49:04 -04:00
Eelco Dolstra
1aba0bf0fa nix-store -r: do substitutions in parallel
I.e. when multiple non-derivation arguments are passed to ‘nix-store
-r’ to be substituted, do them in parallel.
2012-06-27 16:58:15 -04:00
Eelco Dolstra
881beb170d "nix-store -l": support compressed logs 2012-05-30 00:00:02 -04:00
Eelco Dolstra
4bc4da331a Reserve some disk space for the garbage collector
We can't open a SQLite database if the disk is full.  Since this
prevents the garbage collector from running when it's most needed, we
reserve some dummy space that we can free just before doing a garbage
collection.  This actually revives some old code from the Berkeley DB
days.

Fixes #27.
2012-05-29 22:59:12 -04:00
Eelco Dolstra
8be1979f1a Remove the --max-links GC option
We don't need this anymore now that current filesystems support more
than 32,000 files in a directory.
2012-03-26 20:00:02 +02:00
Eelco Dolstra
d5a5a83ad4 Use data() instead of c_str() where appropriate 2012-02-09 18:27:45 +01:00
Eelco Dolstra
4e624849b6 * Added a command ‘nix-store --print-env $drvpath’ that prints out the
environment of the given derivation in a format that can be sourced
  by the shell, e.g.

  $ eval "$(nix-store --print-env $(nix-instantiate /etc/nixos/nixpkgs -A pkg))"
  $ NIX_BUILD_TOP=/tmp
  $ source $stdenv/setup

  This is especially useful to reproduce the environment used to build
  a package outside of its builder for development purposes.

  TODO: add a nix-build option to do the above and fetch the
  dependencies of the derivation as well.
2012-01-17 23:07:22 +00:00
Eelco Dolstra
93b56acb2d * Support multiple outputs in nix-store (specifically the ‘--query’
and ‘--realise’ actions).
2011-12-30 17:25:19 +00:00
Eelco Dolstra
273b288a7e * importPath() -> importPaths(). Because of buffering of the input
stream it's now necessary for the daemon to process the entire
  sequence of exported paths, rather than letting the client do it.
2011-12-16 22:31:25 +00:00
Eelco Dolstra
49f59dceca * Move parseHash16or32 into libutil, and use in nix-hash. 2011-12-02 11:47:06 +00:00
Eelco Dolstra
5bbd693cae * Add an API function exportPaths() that provides the functionality of
‘nix-store --export’.
* Add a Perl module that provides the functionality of
  ‘nix-copy-closure --to’.  This is used by build-remote.pl so it no
  longer needs to start a separate nix-copy-closure process.  Also, it
  uses the Perl API to do the export, so it doesn't need to start a
  separate nix-store process either.  As a result, nix-copy-closure
  and build-remote.pl should no longer fail on very large closures due
  to an "Argument list too long" error.  (Note that having very many
  dependencies in a single derivation can still fail because the
  environment can become too large.  Can't be helped though.)
2011-11-23 15:13:37 +00:00
Eelco Dolstra
e6cb3d0a0d * Added a command ‘nix-store --verify-paths PATHS’ to check whether
the contents of any of the given store paths have been modified.
  E.g.

    $ nix-store --verify-path $(nix-store -qR /var/run/current-system)
    path `/nix/store/m2smyiwbxidlprfxfz4rjlvz2c3mg58y-etc' was modified! expected hash `fc87e271c5fdf179b47939b08ad13440493805584b35e3014109d04d8436e7b8', got `20f1a47281b3c0cbe299ce47ad5ca7340b20ab34246426915fce0ee9116483aa'

  All paths are checked; the exit code is 1 if any path has been
  modified, 0 otherwise.
2011-09-06 12:06:30 +00:00
Eelco Dolstra
93227ff65c * Eliminate all uses of the global variable ‘store’ from libstore.
This should also fix:

    nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed.

  which was caused by hashDerivationModulo() calling the ‘store’
  object (during store upgrades) before openStore() assigned it.
2011-08-31 21:11:50 +00:00
Eelco Dolstra
e60c962fb8 * Add an operation `nix-store -q --size'. 2010-11-17 12:40:52 +00:00
Eelco Dolstra
a3883cbd28 * Store the size of a store path in the database (to be precise, the
size of the NAR serialisation of the path, i.e., `nix-store --dump
  PATH').  This is useful for Hydra.
2010-11-16 17:11:46 +00:00
Eelco Dolstra
fb9368b5a0 * Sync with the trunk. 2010-11-16 12:49:47 +00:00
Eelco Dolstra
705868a8a9 * Make sure that config.h is included before the system headers,
because it defines _FILE_OFFSET_BITS.  Without this, on 
  OpenSolaris the system headers define it to be 32, and then 
  the 32-bit stat() ends up being called with a 64-bit "struct 
  stat", or vice versa.

  This also ensures that we get 64-bit file sizes everywhere.

* Remove the redundant call to stat() in parseExprFromFile().
  The file cannot be a symlink because that's the exit condition
  of the loop before.
2010-10-04 17:55:38 +00:00
Eelco Dolstra
bf87cc44b4 * Sync with the trunk. 2010-06-21 07:55:38 +00:00
Ludovic Courtès
8bcdd36f10 Add XML output to `nix-store'.
* src/nix-store/Makefile.am (nix_store_SOURCES): Add `xmlgraph.cc' and
  `xmlgraph.hh'.

* src/nix-store/help.txt (Operations): Document `--xml'.

* src/nix-store/nix-store.cc (opQuery): Handle `--xml'.

* src/nix-store/xmlgraph.cc, src/nix-store/xmlgraph.hh: New files.
2010-05-31 16:36:24 +00:00
Eelco Dolstra
f92c9a0ac5 * Allow unprivileged users to do `nix-store --clear-failed-paths' and
`nix-store --query-failed-paths'.
2010-05-04 10:45:10 +00:00
Eelco Dolstra
6199f9b93e * Added a command `nix-store --clear-failed-paths <PATHS>' to clear
the "failed" status of the given store paths.  The special value `*'
  clears all failed paths.
2010-04-26 12:56:42 +00:00
Eelco Dolstra
2398af13c5 * Add an command `nix-store --query-failed-paths' to list the cached
failed paths (when using the `build-cache-failure' option).
2010-04-26 12:43:42 +00:00
Eelco Dolstra
fefd467539 * `helpText' is now zero-terminated. 2010-02-24 13:24:27 +00:00
Eelco Dolstra
fdcaf37361 * Made `nix-store -qR --include-outputs' much faster if there are
multiple paths specified on the command line (from O(n * m) to O(n +
  m), where n is the number of arguments and m is the size of the
  closure).
2010-01-25 17:18:44 +00:00
Eelco Dolstra
9b8fda796b * Templatise getIntArg / string2Int. 2009-11-24 12:26:25 +00:00
Eelco Dolstra
3d55f1eb57 * A command `nix-store --query --roots <paths>' to find the garbage
collector roots that point (directly or indirectly) to the given
  paths.
2009-11-23 18:16:25 +00:00
Eelco Dolstra
ae6bf87273 * `nix-store --gc --print-roots': also print the path of the actual
root symlink, not just its target.  E.g.:

  /nix/var/nix/profiles/system-99-link -> /nix/store/76kwf88657nq7wgk1hx3l1z5q91zb9wd-system
2009-11-23 17:23:12 +00:00
Eelco Dolstra
8824d60fe5 * Remove the --use-atime / --max-atime garbage collector flags. Many
(Linux) machines no longer maintain the atime because it's too
  expensive, and on the machines where --use-atime is useful (like the
  buildfarm), reading the atimes on the entire Nix store takes way too
  much time to make it practical.
2009-11-20 17:12:38 +00:00
Eelco Dolstra
5dd8fb2069 * Don't use ULLONG_MAX in maxFreed - use 0 to mean "no limit".
18446744073709551615ULL breaks on GCC 3.3.6 (`integer constant is
  too large for "long" type').
2009-03-26 11:02:07 +00:00
Eelco Dolstra
7a57b2920b * Better error message when nix-store --import is applied to garbage
(previously it would likely say "implementation cannot deal with >
  32-bit integers").
2009-03-22 17:51:45 +00:00
Eelco Dolstra
77d272623f * NAR archives: handle files larger than 2^32 bytes. Previously it
would just silently store only (fileSize % 2^32) bytes.
* Use posix_fallocate if available when unpacking archives.
* Provide a better error message when trying to unpack something that
  isn't a NAR archive.
2009-03-22 17:36:43 +00:00
Eelco Dolstra
0008b0006d * Simplify deleting .lock files in /nix/store: just don't delete them
if they belong a path that's currently being built.  This gets rid
  of some Cygwin-specific code.
2008-12-12 17:03:18 +00:00
Eelco Dolstra
ff762fb499 * Pass HashType values instead of strings. 2008-12-03 16:10:17 +00:00
Eelco Dolstra
64519cfd65 * Unify the treatment of sources copied to the store, and recursive
SHA-256 outputs of fixed-output derivations.  I.e. they now produce
  the same store path:

  $ nix-store --add x
  /nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x

  $ nix-store --add-fixed --recursive sha256 x
  /nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x

  the latter being the same as the path that a derivation

    derivation {
      name = "x";
      outputHashAlgo = "sha256";
      outputHashMode = "recursive";
      outputHash = "...";
      ...
    };

  produces.

  This does change the output path for such fixed-output derivations.
  Fortunately they are quite rare.  The most common use is fetchsvn
  calls with SHA-256 hashes.  (There are a handful of those is
  Nixpkgs, mostly unstable development packages.)
  
* Documented the computation of store paths (in store-api.cc).
2008-12-03 15:06:30 +00:00
Eelco Dolstra
5d250ad1ea * nix-store --dump-db: be more streamy. 2008-11-19 16:26:34 +00:00
Eelco Dolstra
f32fef1b07 * GC option `--max-atime' that specifies an upper limit to the last
accessed time of paths that may be deleted.  Anything more recently
  used won't be deleted.  The time is specified in time_t,
  e.g. seconds since 1970-01-01 00:00:00 UTC; use `date +%s' to
  convert to time_t from the command line. 

  Example: to delete everything that hasn't been used in the last two
  months:

  $ nix-store --gc -v --max-atime $(date +%s -d "2 months ago")
2008-09-17 14:52:35 +00:00
Eelco Dolstra
77afd97a99 * nix-store --gc / --delete: show how many store paths were deleted. 2008-09-17 12:53:33 +00:00
Eelco Dolstra
7ab68961e4 * Garbage collector: added an option `--use-atime' to delete paths in
order of ascending last access time.  This is useful in conjunction
  with --max-freed or --max-links to prefer deleting non-recently used
  garbage, which is good (especially in the build farm) since garbage
  may become live again.

  The code could easily be modified to accept other criteria for
  ordering garbage by changing the comparison operator used by the
  priority queue in collectGarbage().
2008-09-17 10:02:55 +00:00
Eelco Dolstra
a1d310b6b5 * `nix-store --realise': print what paths will be built/downloaded,
just like nix-env.
* `nix-store --realise': --dry-run option.
2008-08-04 13:44:46 +00:00
Eelco Dolstra
d3aa183beb * Garbage collector: option `--max-freed' to stop after at least N
bytes have been freed, `--max-links' to stop when the Nix store
  directory has fewer than N hard links (the latter being important
  for very large Nix stores on filesystems with a 32000 subdirectories
  limit).
2008-06-18 14:20:16 +00:00
Eelco Dolstra
a72709afd8 * Some refactoring: put the GC options / results in separate structs.
* The garbage collector now also prints the number of blocks freed.
2008-06-18 09:34:17 +00:00
Eelco Dolstra
b0e92f6d47 * Merged the no-bdb branch (-r10900:HEAD
https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2008-06-09 13:52:45 +00:00
Eelco Dolstra
66c51dc215 * nix-store --dump-db / --load-db to dump/load the Nix DB.
* nix-store --register-validity: option to supply the content hash of
  each path.
* Removed compatibility with Nix <= 0.7 stores.
2008-01-29 18:17:36 +00:00
Eelco Dolstra
633518628f * nix-env -e: support uninstalling by path, so that one can say
$ nix-env -e $(which firefox)

  or

    $ nix-env -e /nix/store/nywzlygrkfcgz7dfmhm5xixlx1l0m60v-pan-0.132

* nix-env -i: if an argument contains a slash anywhere, treat it as a
  path and follow it through symlinks into the Nix store.  This allows
  things like

    $ nix-build -A firefox
    $ nix-env -i ./result

* nix-env -q/-i/-e: don't complain when the `*' selector doesn't match
  anything.  In particular, `nix-env -q \*' doesn't fail anymore on an
  empty profile.
2007-11-29 16:18:24 +00:00
Eelco Dolstra
c05783ad67 * nix-store --import: add a flag --require-signature. 2007-11-16 16:10:27 +00:00
Eelco Dolstra
14e47e9c2c * Doh. 2007-10-10 14:58:00 +00:00
Eelco Dolstra
315183f194 * nix-store --optimise: flag "--dry-run" to just query what the disk
savings would be.
2007-10-10 13:43:04 +00:00
Eelco Dolstra
a8629de827 * New command `nix-store --optimise' to reduce Nix store disk space
usage by finding identical files in the store and hard-linking them
  to each other.  It typically reduces the size of the store by
  something like 25-35%.  This is what the optimise-store.pl script
  did, but the new command is faster and more correct (it's safe wrt
  garbage collection and concurrent builds).
2007-10-09 22:14:27 +00:00
Eelco Dolstra
9e975458b4 * Get rid of the substitutes database table (NIX-47). Instead, if we
need any info on substitutable paths, we just call the substituters
  (such as download-using-manifests.pl) directly.  This means that
  it's no longer necessary for nix-pull to register substitutes or for
  nix-channel to clear them, which makes those operations much faster
  (NIX-95).  Also, we don't have to worry about keeping nix-pull
  manifests (in /nix/var/nix/manifests) and the database in sync with
  each other.

  The downside is that there is some overhead in calling an external
  program to get the substitutes info.  For instance, "nix-env -qas"
  takes a bit longer.

  Abolishing the substitutes table also makes the logic in
  local-store.cc simpler, as we don't need to store info for invalid
  paths.  On the downside, you cannot do things like "nix-store -qR"
  on a substitutable but invalid path (but nobody did that anyway).

* Never catch interrupts (the Interrupted exception).
2007-08-12 00:29:28 +00:00
Eelco Dolstra
6d1a1191b0 * Support queryDeriver() in multi-user installations. 2007-06-12 16:53:44 +00:00
Eelco Dolstra
7f6161ab3a * Flush cout to show progress. 2007-02-21 23:08:55 +00:00
Eelco Dolstra
0db450024d * Export/import many paths in one go. 2007-02-21 23:00:31 +00:00
Eelco Dolstra
9da367b7d5 * `nix-store -qR' and friends: print the paths sorted topologically
under the references relation.  This is useful for commands that
  want to copy paths to another Nix store in the right order.
2007-02-21 22:45:10 +00:00
Eelco Dolstra
881feb9698 * Flag --print-invalid' in nix-store --check-validity' to print out
which paths specified on the command line are invalid (i.e., don't
  barf when encountering an invalid path, just print it).  This is
  useful for build-remote.pl to figure out which paths need to be
  copied to a remote machine.  (Currently we use rsync, but that's
  rather inefficient.)
2007-02-21 17:57:59 +00:00
Eelco Dolstra
43c4d18c6a * nix-store --import': import an archive created by nix-store
--export' into the Nix store, and optionally check the cryptographic
  signatures against /nix/etc/nix/signing-key.pub.  (TODO: verify
  against a set of public keys.)
2007-02-21 15:45:32 +00:00
Eelco Dolstra
46e0919ced * `nix-store --export --sign': sign the Nix archive using the RSA key
in /nix/etc/nix/signing-key.sec
2007-02-21 14:31:42 +00:00
Eelco Dolstra
b824a1daee * Start of `nix-store --export' operation for serialising a store
path.  This is like `nix-store --dump', only it also dumps the
  meta-information of the store path (references, deriver).  Will add
  a `--sign' flag later to add a cryptographic signature, which we
  will use for exchanging store paths between build farm machines in a
  secure manner.
2007-02-20 23:17:20 +00:00
Eelco Dolstra
afe23b5f38 * nix-pack-closure: store the top-level store paths in the closure.
* nix-unpack-closure: extract the top-level paths from the closure and
  print them on stdout.  This allows them to be installed, e.g.,
  "nix-env -i $(nix-unpack-closure)".  (NIX-64)
2007-01-13 19:50:42 +00:00
Eelco Dolstra
4ca01065c3 * Rename all those main.cc files. 2006-12-07 20:47:30 +00:00
Renamed from src/nix-store/main.cc (Browse further)