There is no longer an `importTarball` method. Instead, there is a
`unpackTarfileToSink` function (back in libutil). The caller can use
thisw with the `getParseSink` method we added in the last commit easily
enough.
In addition, tarball cache functionality is separated from `git-utils`
and moved into `tarball-cache`. This ensures we are separating mechanism
and policy.
There is now a separation of:
1. A `FileSystemObjectSink` for writing to git repos
2. Adapting libarchive to use that parse sink.
The prepares a proper separation of concerns.
A command like
rm -rf ~/.cache/nix/tarball-cache/ ~/.cache/nix/fetcher-cache-v1.sqlite*; nix flake metadata 'git+file:///home/eelco/Dev/nixpkgs?rev=9463103069725474698139ab10f17a9d125da859'
was spending about 84% of its runtime in lookup(), specifically in
git_tree_entry_bypath(). (The reading of blobs is less than 3%.)
It appears libgit2 doesn't do a lot of caching of trees, so we now
make sure that when we look up a path, we add all its parents, and all
the immediate children of the parents (since we have them in memory
anyway), to our own cache.
This speed up the command above from 17.2s to 7.8s on my machine.
Fixes (or at least should improve a lot) #9684.
As discussed in the last Nix team meeting (2024-02-95), this method
doesn't belong because `CanonPath` is a virtual/ideal absolute path
format, not used in file systems beyond the native OS format for which a
"current working directory" is defined.
Progress towards #9205
Add several tests for git fetching:
- shallow-cache-separation: can fetch the same repo shallowly and non-shallowly
- shallow-ignore-ref: ensure that ref gets ignored when shallow=true is set
- ssh-shallow: can fetch a git repo via ssh using shallow=1
libgit2 is not capable of using git-credentials helpers yet.
This prevents private repositories from being used.
Based on code that was replaced in https://github.com/NixOS/nix/pull/9240
(Introduce libgit2); hence:
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Also fingerprint and some preparatory improvements.
Testing is still not up to scratch because lots of logic is duplicated
between the workdir and commit cases.
Enabled for fetchGit, which historically had this behavior,
among other behaviors we do not want in fetchGit.
fetchTree disables this parameter by default. It can choose the
simpler behavior, as it is still experimental.
I am not confident that the filtering implementation is future
proof. It should reuse a source filtering wrapper, which I believe
Eelco has already written, but not merged yet.
Also move `SourcePath` into `libutil`.
These changes allow `error.hh` and `error.cc` to access source path and
position information, which we can use to produce better error messages
(for example, we could consider omitting filenames when two or more
consecutive stack frames originate from the same file).
This uses `git check-ignore` to determine if files are ignored before
attempting to add them in `putFile`.
We also add a condition to the `fetchFromWorkdir` filter to always add
the `flake.lock` file, even if it's not tracked. This is necessary to
resolve inputs.
This fixes#8854 without `git add --force`.
GitArchiveInputScheme now streams tarballs into a Git repository. This
deduplicates data a lot, e.g. when you're fetching different revisions
of the Nixpkgs repo. It also warns if the tree hash returned by GitHub
doesn't match the tree hash of the imported tarball.
We occasionnally commit to git repositories (like with `nix flake update --commit-lock-file`).
This shells out to `git commit`, which might wait for user input (for a signing key passphrase for instance).
Disable the progress bar while this is running to make sure that the
user can enter it.
This avoids repeated copying of the same source tree between Nix
invocations. It requires the accessor to have a "fingerprint" (e.g. a
Git revision) that uniquely determines its contents.
