Commit graph

1677 commits

Author SHA1 Message Date
John Ericson
20b95d6223 Git object hashing in libutil
This is the core functionality but just unit-tested and not yet made
part of the store layer. This is because there is some tech debt around
(a) repeated boilerplate hashing objects (b) better integration of the
new `SourceAccessor` type that needs to be cleaned up first.

Part of RFC 133

Co-Authored-By: Matthew Bauer <mjbauer95@gmail.com>
Co-Authored-By: Carlo Nucera <carlo.nucera@protonmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Florian Klink <flokli@flokli.de>
2023-11-10 11:02:44 -05:00
John Ericson
3d9d5dc189 Create MemorySink
This is for writing to a `MemorySourceAccessor`.
2023-11-10 11:02:37 -05:00
Robert Hensing
12953b942c
Fixup docs 2023-11-09 07:08:56 +01:00
John Ericson
f0adb72c23 Mark fetchTree as unstable again
As discussed in our last meeting, we need a bit more time, but we are
"time boxing" the work left to do to ensure there is not unbounded
delay.

Rather than putting it back underneath `flakes`, though, put it
underneath its own `fetch-tree` experimental feature (which `flakes`
includes/implies). This signals our commitment to the plan to stabilize
it first without waiting to go through the rest of Flakes, and also will
give users a "release candidate" when we get closer to stabilization.

This reverts commit 4112dd1fc9.
2023-11-08 23:23:56 -05:00
John Ericson
d854e8696b Specify the size of the experimental feature array in a more robust way
See doc comment for details.
2023-11-08 22:58:10 -05:00
Robert Hensing
ab69dc4da3 Test parseShebangContent round trip 2023-11-07 13:24:01 -05:00
Robert Hensing
589d338776 parseShebangs: Make strings with backtick sequences representable 2023-11-07 13:24:01 -05:00
Robert Hensing
ffd414eb75 Fix nix shebang interaction with #8131 overhaul completions 2023-11-07 13:24:01 -05:00
Robert Hensing
198bc22e3b nix: Add command baseDir to parse --expr relative to shebang script 2023-11-07 13:24:01 -05:00
Robert Hensing
20ff61ab25 nix: Reserve shebang line syntax and only parse double backtick quotes
Being restrictive about syntax leaves opportunity to improve the
syntax and functionality later.
2023-11-07 13:24:01 -05:00
Tom Bereknyei
bbeddf0602 fix: refactor parseCmdline interface 2023-11-07 13:24:01 -05:00
Tom Bereknyei
06f3583b1c feat: break out of shebang processing for non-comments 2023-11-07 13:24:01 -05:00
Tom Bereknyei
01f61cefcb Read file incrementally 2023-11-07 13:24:01 -05:00
Eelco Dolstra
5f9b5758b6 src/libutil/util.hh: Formatting 2023-11-07 13:24:01 -05:00
Tom Bereknyei
74210c12fe Shellbang support with flakes
Enables shebang usage of nix shell. All arguments with `#! nix` get
added to the nix invocation. This implementation does NOT set any
additional arguments other than placing the script path itself as the
first argument such that the interpreter can utilize it.

Example below:

```
    #!/usr/bin/env nix
    #! nix shell --quiet
    #! nix nixpkgs#bash
    #! nix nixpkgs#shellcheck
    #! nix nixpkgs#hello
    #! nix --ignore-environment --command bash
    # shellcheck shell=bash
    set -eu
    shellcheck "$0" || exit 1
    function main {
        hello
        echo 0:"$0" 1:"$1" 2:"$2"
    }
    "$@"
```

fix: include programName usage

EDIT: For posterity I've changed shellwords to shellwords2 in order
      not to interfere with other changes during a rebase.
      shellwords2 is removed in a later commit. -- roberth
2023-11-07 13:24:01 -05:00
Robert Hensing
1fa87c6aa8
Merge pull request #9283 from obsidiansystems/mememory-source-accessor
Factor out `MemorySourceAccessor`, implement missing features
2023-11-06 19:51:02 +01:00
John Ericson
dea63bb810
Merge pull request #9299 from tfc/config-improvements
Improvements in src/libutil/config.*
2023-11-06 13:03:54 -05:00
John Ericson
937e02e7b9 Shuffle ValidPathInfo JSON rendering
`Store::pathInfoToJSON` was a rather baroque functions, being full of
parameters to support both parsed derivations and `nix path-info`. The
common core of each, a simple `dValidPathInfo::toJSON` function, is
factored out, but the rest of the logic is just duplicated and then
specialized to its use-case (at which point it is no longer that
duplicated).

This keeps the human oriented CLI logic (which is currently unstable)
and the core domain logic (export reference graphs with structured
attrs, which is stable), separate, which I think is better.
2023-11-06 11:06:31 -05:00
Jacek Galowicz
a4b7df7bfa More const, scope reductions, move fixes 2023-11-06 16:00:25 +01:00
Jacek Galowicz
f404e9b3b3 Make toJSONObject const 2023-11-06 16:00:25 +01:00
Jacek Galowicz
ad385f9ec4 Minor improvements 2023-11-06 16:00:25 +01:00
Jacek Galowicz
07ac53732b Fix moves in appendOrSet 2023-11-06 16:00:25 +01:00
John Ericson
ac89bb064a Split up util.{hh,cc}
All OS and IO operations should be moved out, leaving only some misc
portable pure functions.

This is useful to avoid copious CPP when doing things like Windows and
Emscripten ports.

Newly exposed functions to break cycles:

 - `restoreSignals`
 - `updateWindowSize`
2023-11-05 12:20:02 -05:00
John Ericson
2678b51b31 Narrower scope for nativeSystem
I don't think we need a CPP defininition and a header entry, and this
way allows constant expression elimination.
2023-11-05 12:18:28 -05:00
John Ericson
9b880e3e29 Factor out MemorySourceAccessor, implement missing features
The new `MemorySourceAccessor` rather than being a slightly lossy flat
map is a complete in-memory model of file system objects.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-11-04 15:24:20 -04:00
BootRhetoric
6df32889a5
Add git commit verification input attributes
This implements the git input attributes `verifyCommit`, `keytype`,
`publicKey` and `publicKeys` as experimental feature
`verified-fetches`. `publicKeys` should be a json string.
This representation was chosen because all attributes must be of type bool,
int or string so they can be included in flake uris (see definition of
fetchers::Attr).
2023-11-03 20:15:12 +01:00
John Ericson
d15c3a33e6 Don't use std::invocable C++ concept yet
It s not supported on all platforms yet. Can revert this once it is.
2023-11-02 12:06:29 -04:00
John Ericson
b107431816 Systematize characterization tests a bit more
Deduplicating code moreover enforcing the pattern means:

 - It is easier to write new characterization tests because less boilerplate

 - It is harder to mess up new tests because there are fewer places to
   make mistakes.

Co-authored-by: Jacek Galowicz <jacek@galowicz.de>
2023-11-02 12:06:08 -04:00
John Ericson
90de958637
Merge pull request #9265 from obsidiansystems/better-parse-sink
Make `ParseSink` a bit better
2023-11-02 09:28:29 -04:00
Eelco Dolstra
2f5c1a27dc LocalStoreAccessor: Reuse PosixSourceAccessor 2023-11-01 17:22:25 +01:00
Eelco Dolstra
1a902f5fa7 Merge FSAccessor into SourceAccessor 2023-11-01 17:09:28 +01:00
Eelco Dolstra
581693bdea fmt(): Handle std::string_view 2023-11-01 16:33:22 +01:00
Eelco Dolstra
cdb27c1519 SourceAccessor: Change the main interface from lstat() to maybeLstat() 2023-11-01 15:26:07 +01:00
John Ericson
1093d6585f Make ParseSink a bit better
I wouldn't call it *good* yet, but this will do for now.

- `RetrieveRegularNARSink` renamed to `RegularFileSink` and moved
  accordingly because it actually has nothing to do with NARs in
  particular.

  - its `fd` field is also marked private

- `copyRecursive` introduced to dump a `SourceAccessor` into a
  `ParseSink`.

- `NullParseSink` made so `ParseSink` no longer has sketchy default
  methods.

This was done while updating #8918 to work with the new
`SourceAccessor`.
2023-11-01 02:36:43 -04:00
Felix Uhl
f282ef5a56 fix: segfault in positional arg completion
Adding the inputPath as a positional feature uncovered this bug.
As positional argument forms were discarded from the `expectedArgs`
list, their closures were not. When the `.completer` closure was then
called, part of the surrounding object did not exist anymore.

This didn't cause an issue before, but with the new call to
`getEvalState()` in the "inputs" completer in nix/flake.cc, a segfault
was triggered reproducibly on invalid memory access to the `this`
pointer, which was always 0.

The solution of splicing the argument forms into a new list to extend
their lifetime is a bit of a hack, but I was unable to get the "nicer"
iterator-based solution to work.
2023-10-31 15:33:57 +01:00
Kirill Trofimov
90e3ed06f8 fix: Use default destructor. 2023-10-23 18:07:57 +03:00
Kirill Trofimov
a31fc5cc86 fix: Use using instead of typedef for type aliasing.
Since C++ 11 we shouldn't use c-style `typedefs`. In addition, `using` can be templated.
2023-10-23 18:07:17 +03:00
Kirill Trofimov
b205da16ef fix: Explicitly pass lambda scope variables.
Default capture implicitly also capture *this, which would automatically be used if for example you referenced a method from the enclosing scope.
2023-10-23 18:06:15 +03:00
Kirill Trofimov
c82066cf73 fix: Declare constructor as default 2023-10-23 16:59:19 +03:00
John Ericson
b461cac21a
Overhaul completions, redo #6693 (#8131)
As I complained in
https://github.com/NixOS/nix/pull/6784#issuecomment-1421777030 (a
comment on the wrong PR, sorry again!), #6693 introduced a second
completions mechanism to fix a bug. Having two completion mechanisms
isn't so nice.

As @thufschmitt also pointed out, it was a bummer to go from `FlakeRef`
to `std::string` when collecting flake refs. Now it is `FlakeRefs`
again.

The underlying issue that sought to work around was that completion of
arguments not at the end can still benefit from the information from
latter arguments.

To fix this better, we rip out that change and simply defer all
completion processing until after all the (regular, already-complete)
arguments have been passed.

In addition, I noticed the original completion logic used some global
variables. I do not like global variables, because even if they save
lines of code, they also obfuscate the architecture of the code.

I got rid of them  moved them to a new `RootArgs` class, which now has
`parseCmdline` instead of `Args`. The idea is that we have many argument
parsers from subcommands and what-not, but only one root args that owns
the other per actual parsing invocation. The state that was global is
now part of the root args instead.

This did, admittedly, add a bunch of new code. And I do feel bad about
that. So I went and added a lot of API docs to try to at least make the
current state of things clear to the next person.

--

This is needed for RFC 134 (tracking issue #7868). It was very hard to
modularize `Installable` parsing when there were two completion
arguments. I wouldn't go as far as to say it is *easy* now, but at least
it is less hard (and the completions test finally passed).

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-10-23 15:03:11 +02:00
Eelco Dolstra
955bbe53c5
Merge pull request #9177 from edolstra/input-accessors
Backport FSInputAccessor and MemoryInputAccessor from lazy-trees
2023-10-23 11:42:04 +02:00
Eelco Dolstra
34a42f0d0a Move PosixSourceAccessor into its own file 2023-10-23 11:05:50 +02:00
Eelco Dolstra
57db3be9e4 SourceAccessor::readFile(): Support reading into a sink 2023-10-20 16:36:41 +02:00
Eelco Dolstra
bacceaea91 Move getLastModified(), remove setPathDisplay() 2023-10-20 12:40:46 +02:00
Eelco Dolstra
df10dc630f
Doxygen
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-10-20 12:36:18 +02:00
Jacek Galowicz
b0f4ac29d3 libutil: Use c++ style cast 2023-10-19 18:26:49 +01:00
Jacek Galowicz
55f06b6f30 libutil: Remove non-needed constructor 2023-10-19 18:26:49 +01:00
Jacek Galowicz
87c4f4a972 libutil: Move some non-template implememntations from config.hh to
config.cc
2023-10-19 18:26:49 +01:00
Eelco Dolstra
50156302c0 Deduplicate FSSourceAccessor and FSInputAccessor 2023-10-19 15:20:10 +02:00
Eelco Dolstra
9f572eb0e3 Unify the two implementations of dumpPath() 2023-10-19 15:07:56 +02:00