Commit graph

16570 commits

Author SHA1 Message Date
Eelco Dolstra
06be819b89
Merge pull request #9485 from edolstra/tarball-cache
Add a Git-based content-addressed tarball cache
2024-02-15 22:37:57 +01:00
Eelco Dolstra
78b8db72b5 Remove tarball-cache.{hh,cc}
TarballInfo is only used in github.cc, and getTarballCache() is a bit
too trivial to have its own file.
2024-02-15 21:58:08 +01:00
Eelco Dolstra
9b04ce16fc Merge branch 'tarball-cache' of github.com:Ericson2314/nix into tarball-cache 2024-02-15 21:48:10 +01:00
Eelco Dolstra
4b40fc5b46 Merge remote-tracking branch 'origin/master' into tarball-cache 2024-02-15 21:46:36 +01:00
Eelco Dolstra
27b8d5d920
Merge pull request #10013 from edolstra/libgit2-lookup
GitInputAccessor: Speed up lookup()
2024-02-15 17:30:22 +01:00
John Ericson
ed24baaec4 Finish separating concerns with tarball cache
There is no longer an `importTarball` method. Instead, there is a
`unpackTarfileToSink` function (back in libutil). The caller can use
thisw with the `getParseSink` method we added in the last commit easily
enough.

In addition, tarball cache functionality is separated from `git-utils`
and moved into `tarball-cache`. This ensures we are separating mechanism
and policy.
2024-02-15 10:34:01 -05:00
John Ericson
ba6a5f06ee Split GitRepoImpl::importTarball
There is now a separation of:

1. A `FileSystemObjectSink` for writing to git repos

2. Adapting libarchive to use that parse sink.

The prepares a proper separation of concerns.
2024-02-15 10:27:54 -05:00
Théophane Hufschmitt
bc9471d032
Merge pull request #9966 from NixOS/mention-OPTIMIZE-in-hacking-guide
Mention `OPTIMIZE=0` more prominently in the hacking guide
2024-02-15 09:33:15 +01:00
Eelco Dolstra
67a6d34448 GitInputAccessor: Speed up lookup()
A command like

  rm -rf ~/.cache/nix/tarball-cache/ ~/.cache/nix/fetcher-cache-v1.sqlite*; nix flake metadata 'git+file:///home/eelco/Dev/nixpkgs?rev=9463103069725474698139ab10f17a9d125da859'

was spending about 84% of its runtime in lookup(), specifically in
git_tree_entry_bypath(). (The reading of blobs is less than 3%.)

It appears libgit2 doesn't do a lot of caching of trees, so we now
make sure that when we look up a path, we add all its parents, and all
the immediate children of the parents (since we have them in memory
anyway), to our own cache.

This speed up the command above from 17.2s to 7.8s on my machine.

Fixes (or at least should improve a lot) #9684.
2024-02-14 19:25:23 +01:00
Rebecca Turner
6d2b446e2b
Add release notes for "Pretty print values in nix repl" 2024-02-14 08:58:54 -08:00
Eelco Dolstra
54354eaecf Merge remote-tracking branch 'origin/master' into tarball-cache 2024-02-14 14:45:19 +01:00
Théophane Hufschmitt
d857914e1a
Merge pull request #9931 from 9999years/pretty-printer
Pretty-print values in the REPL
2024-02-14 13:32:58 +01:00
John Ericson
ee9f6d022f
Merge pull request #10001 from abathur/fix_macos_daemon_perms
install-darwin: fix symbolic perms for install cmd
2024-02-13 17:39:11 -05:00
John Ericson
ef0b82a267
Merge pull request #9986 from nbelakovski/patch-1
Add a note about lists values.md
2024-02-13 17:10:32 -05:00
Eelco Dolstra
2b946aad8e
Merge pull request #10006 from edolstra/fix-impure-fetchurl
<nix/fetchurl.nix>: Restore support for "impure = true"
2024-02-13 22:50:55 +01:00
Rebecca Turner
a694cfb7bd
Fix "Failed tcsetattr(TCSADRAIN)" when nix repl is not a TTY
Before:
```
$ echo builtins.nixVersion | nix repl
Welcome to Nix 2.18.1. Type :? for help.

Failed tcsetattr(TCSADRAIN): Inappropriate ioctl for device
"2.18.1"

Failed tcsetattr(TCSADRAIN): Inappropriate ioctl for device
```

After:
```
$ echo builtins.nixVersion | nix repl
Nix 2.21.0pre20240131_dirty
Type :? for help.
"2.21.0pre20240131_dirty"
```
2024-02-13 11:09:12 -08:00
John Ericson
5b26c66a8c
Merge pull request #10009 from obsidiansystems/ca-type-names
Proper `parse` and `render` functions for `FileIngestionMethod` and `ContentAddressMethod`
2024-02-13 13:28:52 -05:00
John Ericson
9d64613dca
Update src/libutil/file-content-address.cc 2024-02-13 12:50:10 -05:00
Alois Wohlschlager
89e21ab4bd
Restore builtins.pathExists behavior on broken symlinks
Commit 83c067c0fa changed `builtins.pathExists`
to resolve symlinks before checking for existence. Consequently, if the path
refers to a symlink itself, existence of the target of the symlink (instead of
the symlink itself) was checked. Restore the previous behavior by skipping
symlink resolution in the last component.
2024-02-13 18:09:55 +01:00
John Ericson
db41a0616a Use ContentAddressMethod::render in one more place
Good to deduplicate the code.
2024-02-13 11:14:49 -05:00
Eelco Dolstra
4254cdabf2
Merge pull request #10008 from edolstra/config-status-dep
*.in files: Depend on config.status
2024-02-13 16:49:41 +01:00
John Ericson
f49efb32fb
Merge pull request #10005 from fricklerhandwerk/local-store-glossary
update glossary entry on store types
2024-02-13 10:45:36 -05:00
John Ericson
41dd9857c7 Proper parse and render functions for FileIngestionMethod and ContentAddressMethod
No outward facing behavior is changed.

Older methods with same names that operate on on method + algo pair (for
old-style `<method>:algo`) are renamed to `*WithAlgo`.)

The functions are unit-tested in the same way the names for the hash
algorithms are tested.
2024-02-13 10:30:16 -05:00
Valentin Gagarin
f27205f743 redirect local and chroot store to main page 2024-02-13 16:25:07 +01:00
John Ericson
fb5a438dca
Merge pull request #10007 from NixOS/split-test
Split `hash.sh` test in two
2024-02-13 10:22:12 -05:00
Eelco Dolstra
bb21892351 *.in files: Depend on config.status 2024-02-13 16:10:56 +01:00
Valentin Gagarin
5b69409f6b only refer to the local store page 2024-02-13 16:08:44 +01:00
John Ericson
8bebf9607c Split hash.sh test in two
Converting hashes and hashing files are pretty separate tasks, and more
test parallelism is better.
2024-02-13 09:44:14 -05:00
Eelco Dolstra
b439b1fc66
Merge pull request #9993 from edolstra/builtin-paths
builtin:{unpack-channel,buildenv}: Get output path from the derivation
2024-02-13 14:57:13 +01:00
Valentin Gagarin
39c353f6fa
reword description of the cores setting (#9522)
* reword description of the `cores` setting

- be precise about the `builder` executable
- clearly distinguish between `builder` and job parallelism
- clarify the role of `mkDerivation` in the example
- remove prose for the default, it's shown programmatically
- mention relation to `max-jobs`
2024-02-13 13:52:51 +00:00
Eelco Dolstra
bb63bd50e6 <nix/fetchurl.nix>: Restore support for "impure = true" 2024-02-13 14:14:20 +01:00
Valentin Gagarin
fd82ba0985
extract reference documentation on remote builds (#9526)
- move all reference documentation to the `builders` configuration setting
- reword documentation on machine specification, add examples
- disable showing the default value, as it rendered as `@/dummy/machines`, which is wrong
- highlight the examples
- link to the configuration docs for distributed builds
- builder -> build machine

Co-authored-by: Janik H <janik@aq0.de>
2024-02-13 14:13:56 +01:00
Valentin Gagarin
e37d502895
add instructions to wipe the substituter lookup cache (#9498)
* add instructions to wipe the substituter lookup cache
2024-02-13 13:34:51 +01:00
Valentin Gagarin
ce19338f9f update glossary entry on store types
the interesting information is on the proper pages, and is now presented
a bit more prominently.

the paragraph was a bit confusing to read, also because an anchor link
to an inline definition was in the middle of the sentence. "local store"
now has its own glossary entry.
2024-02-13 13:30:17 +01:00
Théophane Hufschmitt
9d87f25673
Merge pull request #9997 from pg83/patch-1
Update nar-info-disk-cache.cc
2024-02-13 08:55:57 +01:00
Travis A. Everett
64a076fe06 install-darwin: fix symbolic perms for install cmd
The symbolic form in use here doesn't seem to have an effect
in either the BSD or coreutils install commands, leaving the
daemon plist with empty permissions. This seems to cause its
own problems.

I think I've got the right symbolic syntax now :)
2024-02-13 01:18:08 -06:00
Anton Samokhvalov
64cbd4c05a
Update nar-info-disk-cache.cc
fix case when asserts are no-op, like in release build
2024-02-12 23:37:40 +03:00
Robert Hensing
bdb6f56c90
Merge pull request #9295 from NixOS/store-path-complete-construction
Include store path exact spec in the docs
2024-02-12 19:02:03 +01:00
John Ericson
f29d2a9d11 Small EBNF fix 2024-02-12 12:30:28 -05:00
John Ericson
898fd1e48d
Update doc/manual/src/protocols/store-path.md 2024-02-12 12:04:37 -05:00
John Ericson
5169f5f4d9
Apply suggestions from code review
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-02-12 12:01:54 -05:00
John Ericson
c873a140d7
Apply suggestions from code review
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-02-12 11:34:54 -05:00
John Ericson
4c3e4d6d71 Sections, EBNF tweaks 2024-02-12 11:33:06 -05:00
John Ericson
0862d7ce57 Move around non-terminals 2024-02-12 11:26:08 -05:00
John Ericson
eb76b35efa Merge remote-tracking branch 'upstream/master' into store-path-complete-construction 2024-02-12 11:22:54 -05:00
John Ericson
30f6b0f9c5 pre -> fingerprint in store path grammar
As suggested by @roberth in
https://github.com/NixOS/nix/pull/9295#discussion_r1486402040.

Thanks!
2024-02-12 11:22:08 -05:00
John Ericson
95190e68ed Mention the use of Extended Backus–Naur form 2024-02-12 11:21:54 -05:00
John Ericson
ac1301ddfd Convert store path "grammar" to EBNF 2024-02-12 11:16:53 -05:00
John Ericson
91557df4a7
Apply suggestions from code review
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-02-12 11:16:12 -05:00
Eelco Dolstra
a9b69b2fff builtin:{unpack-channel,buildenv}: Get output path from the derivation
Similar to 1ee42c5b88, get the "out"
path from the derivation (and complain if it doesn't exist), rather
than getting it from the environment.
2024-02-12 16:34:59 +01:00