nix-community / nurl Goto Github PK

View Code? Open in Web Editor NEW

425.0 6.0 7.0 678 KB

Generate Nix fetcher calls from repository URLs [maintainer=@figsoda]

Home Page: https://discourse.nixos.org/t/24374

License: Mozilla Public License 2.0

Rust 95.65% Nix 4.35%

fetch github gitlab mercurial nix prefetch sourcehut git hg bitbucket

nurl's Introduction

nurl

Generate Nix fetcher calls from repository URLs

$ nurl https://github.com/nix-community/patsh v0.2.0 2>/dev/null
fetchFromGitHub {
  owner = "nix-community";
  repo = "patsh";
  rev = "v0.2.0";
  hash = "sha256-7HXJspebluQeejKYmVA7sy/F3dtU1gc4eAbKiPexMMA=";
}

If you want to generate Nix packages, feel free check out nix-init, which builds on top of nurl

Supported Fetchers

builtins.fetchGit
fetchCrate
fetchFromBitbucket
fetchFromGitHub
fetchFromGitLab
fetchFromGitea
fetchFromGitiles
fetchFromRepoOrCz
fetchFromSourcehut
fetchHex
fetchPypi
fetchgit
fetchhg
fetchsvn

Usage

Usage: nurl [OPTIONS] [URL] [REV]

Arguments:
  [URL]  URL to the repository to be fetched
  [REV]  The revision or reference to be fetched

Options:
  -S, --submodules[=<SUBMODULES>]      Fetch submodules instead of using the fetcher's default [possible
                                       values: true, false]
  -f, --fetcher <FETCHER>              Specify the fetcher function instead of inferring from the
                                       URL [possible values: builtins.fetchGit, fetchCrate,
                                       fetchFromBitbucket, fetchFromGitHub, fetchFromGitLab,
                                       fetchFromGitea, fetchFromGitiles, fetchFromRepoOrCz,
                                       fetchFromSourcehut, fetchHex, fetchPypi, fetchgit, fetchhg,
                                       fetchsvn]
  -F, --fallback <FALLBACK>            The fetcher to fall back to when nurl fails to infer it from
                                       the URL [default: fetchgit] [possible values:
                                       builtins.fetchGit, fetchCrate, fetchFromBitbucket,
                                       fetchFromGitHub, fetchFromGitLab, fetchFromGitea,
                                       fetchFromGitiles, fetchFromRepoOrCz, fetchFromSourcehut,
                                       fetchHex, fetchPypi, fetchgit, fetchhg, fetchsvn]
  -n, --nixpkgs <NIXPKGS>              Path to nixpkgs (in nix) [default: <nixpkgs>]
  -i, --indent <INDENT>                Extra indentation (in number of spaces) [default: 0]
  -H, --hash                           Only output the hash
  -j, --json                           Output in json format
  -p, --parse                          Parse the url without fetching the hash, output in json
                                       format
  -a, --arg <NAME> <EXPR>              Additional arguments to pass to the fetcher
  -A, --arg-str <NAME> <STRING>        Same as --arg, but accepts strings instead Nix expressions
  -o, --overwrite <NAME> <EXPR>        Overwrite arguments in the final output, not taken into
                                       consideration when fetching the hash
  -O, --overwrite-str <NAME> <STRING>  Same as --overwrite, but accepts strings instead Nix
                                       expressions
  -e, --expr <EXPR>                    Instead of fetching a URL, get the hash of a fixed-output
                                       derivation, implies --hash and ignores all other options
  -l, --list-fetchers                  List all available fetchers
  -L, --list-possible-fetchers         List all fetchers that can be generated without --fetcher
  -s, --list-sep <SEPARATOR>           Print out the listed fetchers with the specified separator,
                                       only used when --list-fetchers or --list-possible-fetchers is
                                       specified
  -h, --help                           Print help
  -V, --version                        Print version

Comparison to nix-prefetch

nurl infers the fetcher from the URL. For nix-prefetch, you need to pick the fetcher and supply the arguments manually.
nix-prefetch relies on FOD which is slow, nurl tries to use alternatives when possible.
nix-prefetch is more configurable and supports file attributes.
nix-prefetch has an interface similar to nix-build.
nurl has some nice features dedicated to generated packages (--indent, --list-possible-fetchers).

Changelog

See CHANGELOG.md

nurl's People

Contributors

Stargazers

Watchers

Forkers

kranzes sno2wman the-furry-hubofeverything helenakew inayet teto matt1432

nurl's Issues

resolve revision when using submodules

I have started using nurl in luarocks-nix.

$ nurl https://github.com/teto/luabitop master -S

fetchFromGitHub {
  owner = "teto";
  repo = "luabitop";
  rev = "master";
  hash = "sha256-yLE198w1Ry4y9kflK/QHeWlmi0T7zJSie00UVgR/l54=";
  fetchSubmodules = true;
}

ofc it doesn't build

       last 13 log lines:
       > exporting https://github.com/teto/luabitop.git (rev master) into /nix/store/20zhsmkxm7nndlprrqbchknghjlz4cv1-source
       > Initialized empty Git repository in /nix/store/20zhsmkxm7nndlprrqbchknghjlz4cv1-source/.git/
       > fatal: couldn't find remote ref refs/tags/master
       > remote: Enumerating objects: 33, done.
       > remote: Counting objects: 100% (33/33), done.
       > remote: Compressing objects: 100% (29/29), done.
       > remote: Total 33 (delta 2), reused 22 (delta 2), pack-reused 0
       > Unpacking objects: 100% (33/33), 37.73 KiB | 402.00 KiB/s, done.
       > From https://github.com/teto/luabitop
       >  * branch            HEAD       -> FETCH_HEAD
       > fatal: Not a valid object name
       > Unrecognized git object type:
       > Unable to checkout refs/tags/master from https://github.com/teto/luabitop.git.
       For full logs, run 'nix log /nix/store/lmr3pxlqnd82169xlr7sdsb5ddq5nbvl-source.drv'.

would be nice if nurl could infer the revision. Related to #5

Allow fetching the wheel hash from pypi

❯ nurl https://pypi.org/project/home-assistant-frontend/ 20230309.0
$ nix-prefetch-url https://pypi.org/packages/source/h/home-assistant-frontend/home-assistant-frontend-20230309.0.tar.gz
error: unable to download 'https://pypi.org/packages/source/h/home-assistant-frontend/home-assistant-frontend-20230309.0.tar.gz': HTTP error 404
Error: command exited with exit code exit status: 1

https://pypi.org/project/home-assistant-frontend/#files

FetchGitHub with submodules

When using this tool to update package that is using fetchFromGitHub with fetchSubmodules set to true.
The hash would mismatch since this tool is probably not fetching the submodules.
nurl https://github.com/WerWolv/ImHex-Patterns ImHex-v1.26.2

It would be nice if there was option for nurl to fetch with submodules maybe with command line flag?

Include the timestamp in the output

Hi, I'm using nurl in one of my scripts to fetch information about the latest revision of a repository. I would like to request to include the timestamp of the latest revision in the output, so I can use that to generate an "unstable-" version field from my script.
Hopefully you will consider including this. Thank you!

Infer revision from URL

Infer revision from URLs like

https://github.com/nix-community/nurl/commit/39b02af7295726268b73e91dca9f80433527b4b4
https://github.com/nix-community/nurl/tree/main

Originally posted by @amesgen in #3 (comment)

nurl gives different hash than Nixpkgs

nix run pkgs#nurl https://github.com/nix-community/nixd/archive/refs/tags/2.2.3.zip

(nurl 0.3.13) Gives me

$ nix flake prefetch --extra-experimental-features 'nix-command flakes' --json github:nix-community/nixd/38560011298ab5f955036ef0ff453d387a746044
fetchFromGitHub {
  owner = "nix-community";
  repo = "nixd";
  rev = "38560011298ab5f955036ef0ff453d387a746044";
  hash = "sha256-xi6iFd1DFR0g8TyJCOg5pkLHmXP0oyB4rMx5GvqwyKU=";
}

At the same time, by removing the hash (hash = "") from nixd and refetching, it gives me

error: hash mismatch in fixed-output derivation '/nix/store/nab810qn7l15xvsq1mhpr6d4gzw2zfx4-source.drv':
         specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
            got:    sha256-1i4U012vqw84NuD5cjgfVX/he4vW9MyEeIetJ55g280=

I'm on Nixpkgs 74d38fdb308da734ea497aae3b98640b4d8265ca. I alternated between these two operations repeatedly and got the same results.

Fall back to nixpkgs from the flake registry if `<nixpkgs>` is unavailable

Generate rev with version substituted

It would be nice to get rev = "v${version}"; with an option.

Latest revision support for all flake-based fetchers

Support for fallback revisions

Sometimes a script may know it needs version 1.2.3, but it doesn't know whether the upstream repo prepends v on their version tags or not. In such cases a list of fallback revisions to try would be great. Example

nurl repo-url v1.2.3 1.2.3

would try repo-url/v1.2.3 first, if it doesn't exit then try repo-url/1.2.3.

Especially pessimistic scripts could then even fallback on main or master.

Final newline

~ +$ nurl https://github.com/nix-community/nurl
$ nix flake prefetch --experimental-features 'nix-command flakes' --json github:nix-community/nurl/2ba2570404d1465abed87ad3b975a9b18f124a1a
fetchFromGitHub {
  owner = "nix-community";
  repo = "nurl";
  rev = "2ba2570404d1465abed87ad3b975a9b18f124a1a";
  hash = "sha256-xHhvT6fDlMyHEdr1MzF/Xkxm2YSoRJeg+UlrfG72pgs=";
}~ +$

Please output a final newline, at least when printing to a terminal.

Github: support private repository

Trying to use nurl on a private repository, I get an error:

HTTP error 404

       response body:

       {
         "message": "Not Found",
         "documentation_url": "https://docs.github.com/rest/commits/commits#get-a-commit"
       }
Error: command exited with exit status: 1

I see in the source code that no token are set.
Would it be possible to read optional token from env variable for example GITHUB_TOKEN. Or to be able to specify it via CLI, for example via --token?

For those interested, my current workaround it to use (this is based on some comments I found in a github issue, but I forgot to write down the author name, I adapted it to fix the deprecation warning):

nix-shell -p nix-prefetch-git jq --run "nix hash convert --hash-algo sha256 --to sri \$(nix-prefetch-git --url https://github.com/USER/REPO --quiet --rev TAG | jq -r '.sha256')"

Of course this just gives me the sha256, which is just what I need for now.

Have a nice day.

Hash prefetching commands (sometimes?) need the flakes experimental feature

Try running

nurl --hash -f fetchPypi https://pypi.org/project/semgrep 1.32.0 -A format wheel -A dist cp37.cp38.cp39.cp310.cp311.py37.py38.py39.py310.py311 -A python cp37.cp38.cp39.cp310.cp311.py37.py38.py39.py310.py311 -A platform macosx_11_0_arm64

on Nix 2.15 or 2.16.1, and you'll see a message saying that the flakes feature needs to be enabled, like this:

error:
       … while calling the 'import' builtin

         at «string»:1:2:

            1| (import(<nixpkgs>){}).fetchPypi{pname="semgrep";version="1.32.0";hash="sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";format="wheel";dist="cp37.cp38.cp39.cp310.cp311.py37.py38.py39.py310.py311";python="cp37.cp38.cp39.cp310.cp311.py37.py38.py39.py310.py311";platform="macosx_11_0_arm64";}
             |  ^

       … while calling the 'findFile' builtin

         at «string»:1:9:

            1| (import(<nixpkgs>){}).fetchPypi{pname="semgrep";version="1.32.0";hash="sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";format="wheel";dist="cp37.cp38.cp39.cp310.cp311.py37.py38.py39.py310.py311";python="cp37.cp38.cp39.cp310.cp311.py37.py38.py39.py310.py311";platform="macosx_11_0_arm64";}
             |         ^

       error: experimental Nix feature 'flakes' is disabled; use '--extra-experimental-features flakes' to override
       ```
       
Since I already have flakes enabled, changing the `nix-build` invocation to `--extra-experimental-features` instead of `--experimental-features` fixes the problem for me.

Make revision optional

Add support for fetching the latest revision for most fetchers, this will also allow fetchzip and fetchurl to be implemented