Commit graph

2785 commits

Author SHA1 Message Date
Eelco Dolstra
198d0338be Add a primop ‘concatLists’
This can serve as a generic efficient list builder.  For instance, the
function ‘catAttrs’ in Nixpkgs can be rewritten from

  attr: l: fold (s: l: if hasAttr attr s then [(getAttr attr s)] ++ l else l) [] l

to

  attr: l: builtins.concatLists (map (s: if hasAttr attr s then [(getAttr attr s)] else []) l)

Statistics before:

  time elapsed: 1.08683
  size of a value: 24
  environments allocated: 1384376 (35809568 bytes)
  list elements: 6946783 (55574264 bytes)
  list concatenations: 37434
  values allocated: 1760440 (42250560 bytes)
  attribute sets allocated: 392040
  right-biased unions: 186334
  values copied in right-biased unions: 591137
  symbols in symbol table: 18273
  number of thunks: 1297673
  number of thunks avoided: 1380759
  number of attr lookups: 430802
  number of primop calls: 628912
  number of function calls: 1333544

Statistics after (including new catAttrs):

  time elapsed: 0.959854
  size of a value: 24
  environments allocated: 1010198 (26829296 bytes)
  list elements: 1984878 (15879024 bytes)
  list concatenations: 30488
  values allocated: 1589760 (38154240 bytes)
  attribute sets allocated: 392040
  right-biased unions: 186334
  values copied in right-biased unions: 591137
  symbols in symbol table: 18274
  number of thunks: 1040925
  number of thunks avoided: 1038428
  number of attr lookups: 438419
  number of primop calls: 474844
  number of function calls: 959366
2012-08-13 01:53:10 -04:00
Eelco Dolstra
b9e5b908ed Provide an efficient implementation of ‘elem’
The one in Nixpkgs is O(n^2), this one is O(n).  Big reduction in the
number of list allocations.

Statistics before (on a NixOS system config):

  time elapsed: 1.17982
  size of a value: 24
  environments allocated: 1543334 (39624560 bytes)
  list elements: 9612638 (76901104 bytes)
  list concatenations: 37434
  values allocated: 1854933 (44518392 bytes)
  attribute sets allocated: 392040
  right-biased unions: 186334
  values copied in right-biased unions: 591137
  symbols in symbol table: 18272
  number of thunks: 1392467
  number of thunks avoided: 1507311
  number of attr lookups: 430801
  number of primop calls: 691600
  number of function calls: 1492502

Statistics after:

  time elapsed: 1.08683
  size of a value: 24
  environments allocated: 1384376 (35809568 bytes)
  list elements: 6946783 (55574264 bytes)
  list concatenations: 37434
  values allocated: 1760440 (42250560 bytes)
  attribute sets allocated: 392040
  right-biased unions: 186334
  values copied in right-biased unions: 591137
  symbols in symbol table: 18273
  number of thunks: 1297673
  number of thunks avoided: 1380759
  number of attr lookups: 430802
  number of primop calls: 628912
  number of function calls: 1333544
2012-08-13 01:05:35 -04:00
Eelco Dolstra
4ccd48ce24 Add a "filter" primop
Evaluation of a NixOS configuration spends quite a lot of time in the
"filter" function in Nixpkgs.  As implemented in Nixpkgs, this is a
O(n^2) operation, so it's a good candidate for providing a more
efficient (i.e. primop) implementation.  Using it gives a ~10% speed
increase and a significant reduction in the number of evaluations.

Statistics before (on a NixOS system config):

  time elapsed: 1.3258
  size of a value: 24
  environments allocated: 1980939 (50127080 bytes)
  list elements: 14679308 (117434464 bytes)
  list concatenations: 50828
  values allocated: 2098938 (50374512 bytes)
  attribute sets allocated: 392040
  right-biased unions: 186334
  values copied in right-biased unions: 591137
  symbols in symbol table: 18271
  number of thunks: 1645752
  number of thunks avoided: 1921196
  number of attr lookups: 430798
  number of primop calls: 838807
  number of function calls: 1930107

Statistics after:

  time elapsed: 1.17982
  size of a value: 24
  environments allocated: 1543334 (39624560 bytes)
  list elements: 9612638 (76901104 bytes)
  list concatenations: 37434
  values allocated: 1854933 (44518392 bytes)
  attribute sets allocated: 392040
  right-biased unions: 186334
  values copied in right-biased unions: 591137
  symbols in symbol table: 18272
  number of thunks: 1392467
  number of thunks avoided: 1507311
  number of attr lookups: 430801
  number of primop calls: 691600
  number of function calls: 1492502
2012-08-13 00:28:08 -04:00
Eelco Dolstra
62f72eb9e1 Add some more evaluations stats 2012-08-12 23:41:48 -04:00
Eelco Dolstra
e82767910c Add some basic profiling support to the evaluator
Setting the environment variable NIX_COUNT_CALLS to 1 enables some
basic profiling in the evaluator.  It will count calls to functions
and primops as well as evaluations of attributes.

For example, to see where evaluation of a NixOS configuration spends
its time:

$ NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 ./src/nix-instantiate/nix-instantiate '<nixos>' -A system --readonly-mode
...
calls to 39 primops:
    239532 head
    233962 tail
    191252 hasAttr
...
calls to 1595 functions:
    224157 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/lists.nix:17:19'
    221767 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/lists.nix:17:14'
    221767 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/lists.nix:17:10'
...
evaluations of 7088 attributes:
    167377 undefined position
    132459 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/attrsets.nix:119:41'
     47322 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/attrsets.nix:13:21'
...
2012-08-12 23:29:28 -04:00
Eelco Dolstra
325d1cfebf Don't warn about maximum link count exceeded on 0-byte files 2012-08-07 16:22:54 -04:00
Eelco Dolstra
d025142f52 Handle amount of disk space saved by hard linking being negative
Fixes bogus messages like "currently hard linking saves
17592186044416.00 MiB".
2012-08-05 21:45:27 -04:00
Eelco Dolstra
b6c989b801 Fix race condition when two processes create a hard link to a file in .links
This is a problem because one process may set the immutable bit before
the second process has created its link.

Addressed random Hydra failures such as:

error: cannot rename `/nix/store/.tmp-link-17397-1804289383' to
`/nix/store/rsvzm574rlfip3830ac7kmaa028bzl6h-nixos-0.1pre-git/upstart-interface-version':
Operation not permitted
2012-08-05 21:41:44 -04:00
Eelco Dolstra
108e14bb18 Fix race condition when two processes create the same link in /nix/store/.links 2012-08-05 18:17:55 -04:00
Eelco Dolstra
6763084ae5 Count bytes freed deleting unused links 2012-08-01 22:43:03 -04:00
Eelco Dolstra
01d56c1eec Drop the block count in the garbage collector 2012-08-01 22:34:46 -04:00
Eelco Dolstra
967d066d8e nix-store --gc: Make ‘--max-freed 0’ do the right thing
That is, delete almost nothing (it will still remove unused links from
/nix/store/.links).
2012-08-01 19:14:58 -04:00
Eelco Dolstra
1df702d347 removeUnusedLinks(): Print stats on disk space saved by hard linking 2012-08-01 19:01:50 -04:00
Eelco Dolstra
234ce610e0 Doh 2012-08-01 16:09:47 -04:00
Eelco Dolstra
7b10562370 Make ‘nix-store --optimise’ interruptible 2012-08-01 16:06:49 -04:00
Eelco Dolstra
157170059d Manual: Remove reference to non-existent -I option 2012-07-30 22:29:42 -04:00
Eelco Dolstra
f15083c10a Document the --option flag
Pointed out by Daniel Santa Cruz on IRC.
2012-07-29 14:37:40 -04:00
Eelco Dolstra
3a4623afbb Set permissions on temporary build directories to 0700
Fixes #39.
2012-07-26 15:04:40 -04:00
Eelco Dolstra
2605f4f4e6 nix-profile.sh: Don't set NIX_REMOTE on single user installations
Commit 6a214f3e06 reused the NixOS
environment initialisation for nix-profile.sh, but this is
inappropriate on systems that don't have multi-user support enabled.
2012-07-25 17:06:09 -04:00
Eelco Dolstra
477b0fbeca Subscribe to the Nixpkgs rather than NixOS channel 2012-07-25 16:56:56 -04:00
Shea Levy
f5954e2d94 prim_import: When importing .drvs, allocate the intermediate attrset on the heap just in case it escapes the stack frame. 2012-07-25 09:30:17 -04:00
Shea Levy
1ef2d5765b Turn tests back on 2012-07-25 09:30:17 -04:00
Shea Levy
b1112bbef1 import: If the path is a valid .drv file, parse it and generate a derivation attrset.
The generated attrset has drvPath and outPath with the right string context, type 'derivation', outputName with
the right name, all with a list of outputs, and an attribute for each output.

I see three uses for this (though certainly there may be more):

* Using derivations generated by something besides nix-instantiate (e.g. guix)

* Allowing packages provided by channels to be used in nix expressions. If a channel installed a valid deriver
  for each package it provides into the store, then those could be imported and used as dependencies or installed
  in environment.systemPackages, for example.

* Enable hydra to be consistent in how it treats inputs that are outputs of another build. Right now, if an
  input is passed as an argument to the job, it is passed as a derivation, but if it is accessed via NIX_PATH
  (i.e. through the <> syntax), then it is a path that can be imported. This is problematic because the build
  being depended upon may have been built with non-obvious arguments passed to its jobset file. With this
  feature, hydra can just set the name of that input to the path to its drv file in NIX_PATH
2012-07-25 09:30:17 -04:00
Shea Levy
566a30c007 Disable tests temporarily 2012-07-25 09:30:17 -04:00
Eelco Dolstra
e98c029717 Handle platforms that don't support linking to a symlink
E.g. Darwin doesn't allow this.
2012-07-23 18:42:18 -04:00
Eelco Dolstra
fd63c8bfcd Unlink the right file 2012-07-23 18:06:37 -04:00
Eelco Dolstra
0f65793f94 Add a test for Nix store optimisation 2012-07-23 17:40:23 -04:00
Eelco Dolstra
680ab6f83d Garbage collect unused links in /nix/store/.links
Incremental optimisation requires creating links in /nix/store/.links
to all files in the store.  However, this means that if we delete a
store path, no files are actually deleted because links in
/nix/store/.links still exists.  So we need to check /nix/store/.links
for files with a link count of 1 and delete them.
2012-07-23 17:14:16 -04:00
Eelco Dolstra
6193105710 Automatically optimise the Nix store when a new path is added
Auto-optimisation is enabled by default.  It can be turned off by
setting auto-optimise-store to false in nix.conf.
2012-07-23 17:14:15 -04:00
Eelco Dolstra
564fb7d9fa optimiseStore(): Use a content-addressed file store in /nix/store/.links
optimiseStore() now creates persistent, content-addressed hard links
in /nix/store/.links.  For instance, if it encounters a file P with
hash H, it will create a hard link

  P' = /nix/store/.link/<H>

to P if P' doesn't already exist; if P' exist, then P is replaced by a
hard link to P'.  This is better than the previous in-memory map,
because it had the tendency to unnecessarily replace hard links with a
hard link to whatever happened to be the first file with a given hash
it encountered.  It also allows on-the-fly, incremental optimisation.
2012-07-23 17:14:15 -04:00
Eelco Dolstra
ed59bf7a18 nix-build: Support the ‘-’ argument to build an expression from stdin 2012-07-23 17:11:12 -04:00
Eelco Dolstra
6852289c46 Use lutimes() if available to canonicalise the timestamp of symlinks
Also use utimes() instead of utime() if lutimes() is not available.
2012-07-23 16:52:25 -04:00
Eelco Dolstra
1832ab71db Bump version 2012-07-18 17:17:23 -04:00
Eelco Dolstra
98193bb440 Remove RPM builds that don't evaluate 2012-07-18 16:50:56 -04:00
Eelco Dolstra
b7fd2c2822 Use "#pragma once" to prevent repeated header file inclusion 2012-07-18 14:59:03 -04:00
Eelco Dolstra
58337e0e61 Set release date 2012-07-18 11:51:27 -04:00
Eelco Dolstra
ccc52adfb2 Add function queryPathFromHashPart()
To implement binary caches efficiently, Hydra needs to be able to map
the hash part of a store path (e.g. "gbg...zr7") to the full store
path (e.g. "/nix/store/gbg...kzr7-subversion-1.7.5").  (The binary
cache mechanism uses hash parts as a key for looking up store paths to
ensure privacy.)  However, doing a search in the Nix store for
/nix/store/<hash>* is expensive since it requires reading the entire
directory.  queryPathFromHashPart() prevents this by doing a cheap
database lookup.
2012-07-17 18:55:39 -04:00
Eelco Dolstra
220818f758 queryPathInfo(): return hash in base-32 if desired
Cherry-picked from the no-manifests branch.
2012-07-17 16:55:45 -04:00
Eelco Dolstra
3a9fdf2747 Return an exit code of 100 for cached failed builds
Exit code 100 should be returned for all permanent failures.  This
includes cached failures.

Fixes #34.
2012-07-17 15:55:30 -04:00
Eelco Dolstra
1217204c81 Remove dead code 2012-07-17 14:07:52 -04:00
Eelco Dolstra
51d71ad3d7 Manual: Don't claim we support Cygwin 2012-07-17 11:49:47 -04:00
Eelco Dolstra
6c01fb4d68 Update Nix 1.1 release notes 2012-07-17 10:06:20 -04:00
Eelco Dolstra
53b24f3518 Allow disabling log compression 2012-07-17 09:40:12 -04:00
Eelco Dolstra
a7a43adb79 builtins.storePath: resolve symlinks
Needed for Charon/Hydra interaction.
2012-07-12 18:25:01 -04:00
Eelco Dolstra
eae802459d Pass --insecure to curl so that https works 2012-07-09 15:49:20 -04:00
Eelco Dolstra
2dd3117c27 Inline fetchurl.sh 2012-07-09 15:48:55 -04:00
Eelco Dolstra
51f9f9924b Add a test for the fetchurl function 2012-07-09 15:41:43 -04:00
Shea Levy
035aa11403 Remove obsolete comment 2012-07-09 15:29:49 -04:00
Shea Levy
a2865f6b3d corepkgs/fetchurl: Build locally and outside of the chroot 2012-07-09 15:29:49 -04:00
Shea Levy
53f52c2111 corepkgs/fetchurl: the 'system' argument can be optional 2012-07-09 15:29:49 -04:00