nix-super/src/libstore
Eelco Dolstra 0abb3ad537 Allow content-addressable paths to have references
This adds a command 'nix make-content-addressable' that rewrites the
specified store paths into content-addressable paths. The advantage of
such paths is that 1) they can be imported without signatures; 2) they
can enable deduplication in cases where derivation changes do not
cause output changes (apart from store path hashes).

For example,

  $ nix make-content-addressable -r nixpkgs.cowsay
  rewrote '/nix/store/g1g31ah55xdia1jdqabv1imf6mcw0nb1-glibc-2.25-49' to '/nix/store/48jfj7bg78a8n4f2nhg269rgw1936vj4-glibc-2.25-49'
  ...
  rewrote '/nix/store/qbi6rzpk0bxjw8lw6azn2mc7ynnn455q-cowsay-3.03+dfsg1-16' to '/nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16'

We can then copy the resulting closure to another store without
signatures:

  $ nix copy --trusted-public-keys '' ---to ~/my-nix /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16

In order to support self-references in content-addressable paths,
these paths are hashed "modulo" self-references, meaning that
self-references are zeroed out during hashing. Somewhat annoyingly,
this means that the NAR hash stored in the Nix database is no longer
necessarily equal to the output of "nix hash-path"; for
content-addressable paths, you need to pass the --modulo flag:

  $ nix path-info --json /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16  | jq -r .[].narHash
  sha256:0ri611gdilz2c9rsibqhsipbfs9vwcqvs811a52i2bnkhv7w9mgw

  $ nix hash-path --type sha256 --base32 /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16
  1ggznh07khq0hz6id09pqws3a8q9pn03ya3c03nwck1kwq8rclzs

  $ nix hash-path --type sha256 --base32 /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16 --modulo iq6g2x4q62xp7y7493bibx0qn5w7xz67
  0ri611gdilz2c9rsibqhsipbfs9vwcqvs811a52i2bnkhv7w9mgw
2019-10-21 17:47:24 +02:00
..
builtins OCD performance fix: {find,count}+insert => insert 2019-10-09 16:06:29 +02:00
binary-cache-store.cc BinaryCacheStore: Add index-debug-info option 2019-09-04 19:28:26 +02:00
binary-cache-store.hh BinaryCacheStore: Add index-debug-info option 2019-09-04 19:28:26 +02:00
build.cc Allow content-addressable paths to have references 2019-10-21 17:47:24 +02:00
builtins.hh Make <nix/buildenv.nix> a builtin builder 2018-03-20 17:28:09 +01:00
crypto.cc binary-cache-public-keys -> trusted-public-keys 2017-11-20 17:32:34 +01:00
crypto.hh Revert "Get rid of unicode quotes (#1140)" 2016-11-26 00:38:01 +01:00
derivations.cc Support allowSubstitutes attribute in structured attribute derivations 2019-09-03 16:03:49 +02:00
derivations.hh Support allowSubstitutes attribute in structured attribute derivations 2019-09-03 16:03:49 +02:00
download.cc Ensure that Callback is called only once 2019-09-03 13:45:35 +02:00
download.hh Shut up some warnings 2019-09-22 21:57:05 +02:00
export-import.cc Replace Unicode quotes in user-facing strings by ASCII 2017-07-30 12:32:45 +01:00
fs-accessor.hh Shut up some warnings 2019-09-22 21:57:05 +02:00
gc.cc Merge pull request #2338 from bobvanderlinden/pr-cannot-delete-alive-why 2019-09-30 14:06:52 +02:00
globals.cc Add experimental-features setting 2019-10-21 13:34:44 +02:00
globals.hh Add experimental-features setting 2019-10-21 13:34:44 +02:00
http-binary-cache-store.cc Add some noexcepts 2019-09-03 13:45:35 +02:00
legacy-ssh-store.cc Add some noexcepts 2019-09-03 13:45:35 +02:00
local-binary-cache-store.cc BinaryCacheStore: Add index-debug-info option 2019-09-04 19:28:26 +02:00
local-fs-store.cc Replace Unicode quotes in user-facing strings by ASCII 2017-07-30 12:32:45 +01:00
local-store.cc Allow content-addressable paths to have references 2019-10-21 17:47:24 +02:00
local-store.hh Remove world-writability from per-user directories 2019-10-09 23:34:48 +02:00
local.mk Handle empty sandbox_shell 2019-10-08 23:12:54 -04:00
machines.cc getMachines(): Cache result 2019-04-01 21:09:35 +02:00
machines.hh Replace $NIX_REMOTE_SYSTEMS with an option "builder-files" 2017-05-02 15:46:09 +02:00
misc.cc OCD performance fix: {find,count}+insert => insert 2019-10-09 16:06:29 +02:00
nar-accessor.cc Provide random access to cached NARs 2017-12-07 01:07:07 +01:00
nar-accessor.hh Fix build on gcc 7 2017-12-16 00:49:31 +01:00
nar-info-disk-cache.cc Fix assertion failure in NarInfoDiskCache 2018-12-12 17:13:22 +01:00
nar-info-disk-cache.hh HttpBinaryCacheStore: Fix caching of WantMassQuery 2016-06-01 16:24:17 +02:00
nar-info.cc Replace Unicode quotes in user-facing strings by ASCII 2017-07-30 12:32:45 +01:00
nar-info.hh Make the store directory a member variable of Store 2016-06-01 16:24:17 +02:00
nix-store.pc.in pkg-config files: Use c++17 2019-03-14 14:11:12 +01:00
optimise-store.cc Fix optimise-store.cc to skip more paths on macOS 2018-06-14 12:34:55 -04:00
parsed-derivations.cc Support allowSubstitutes attribute in structured attribute derivations 2019-09-03 16:03:49 +02:00
parsed-derivations.hh Support allowSubstitutes attribute in structured attribute derivations 2019-09-03 16:03:49 +02:00
pathlocks.cc pathlocks: add include to fcntl.h for O_CLOEXEC 2019-08-07 07:41:22 -05:00
pathlocks.hh Use BSD instead of POSIX file locks 2019-08-02 18:39:16 +02:00
profiles.cc only delete things older than current gen and update logic in doc as 2018-03-01 22:59:00 -05:00
profiles.hh this updates issues that were addressed by people in pr 2018-03-01 21:47:57 -05:00
references.cc Allow content-addressable paths to have references 2019-10-21 17:47:24 +02:00
references.hh Allow content-addressable paths to have references 2019-10-21 17:47:24 +02:00
remote-fs-accessor.cc Provide random access to cached NARs 2017-12-07 01:07:07 +01:00
remote-fs-accessor.hh Provide random access to cached NARs 2017-12-07 01:07:07 +01:00
remote-store.cc ssh-ng: Don't set CPU affinity on the remote 2019-10-11 18:49:46 +02:00
remote-store.hh ssh-ng: Don't set CPU affinity on the remote 2019-10-11 18:49:46 +02:00
s3-binary-cache-store.cc Add a 5 seconds of timeouts to connect to S3. 2019-03-15 13:23:58 +01:00
s3-binary-cache-store.hh Fix getS3Stats() 2016-10-06 17:00:52 +02:00
s3.hh s3: make scheme configurable 2018-12-07 23:55:38 +01:00
sandbox-defaults.sb Remove extraneous comment 2017-11-08 16:17:32 +01:00
sandbox-minimal.sb Always use the Darwin sandbox 2017-06-06 18:44:49 +02:00
sandbox-network.sb OS X sandbox: Improve builtin sandbox profile 2017-05-31 17:25:51 +02:00
schema.sql Mark content-addressed paths in the Nix database and in .narinfo 2016-08-10 18:05:35 +02:00
serve-protocol.hh Make adding paths via nix-store --serve run in constant memory 2018-08-03 21:21:16 +02:00
sqlite.cc Use extended SQLite error codes 2018-05-30 13:25:03 +02:00
sqlite.hh Use boost::format from the boost package 2018-03-14 19:24:04 +01:00
ssh-store.cc ssh-ng: Don't set CPU affinity on the remote 2019-10-11 18:49:46 +02:00
ssh.cc Revert "Restore parent mount namespace before executing a child process" 2019-02-05 10:49:19 +01:00
ssh.hh SSHMaster: Bypass SSH when connecting to localhost 2018-08-03 21:20:38 +02:00
store-api.cc Allow content-addressable paths to have references 2019-10-21 17:47:24 +02:00
store-api.hh Allow content-addressable paths to have references 2019-10-21 17:47:24 +02:00
worker-protocol.hh Make 'nix copy --to daemon' run in constant memory (daemon side) 2018-05-30 13:26:48 +02:00