msteen / nix-prefetch Goto Github PK

nix-prefetch fails on nur-packages

error: Could not find Nixpkgs path: /tmp/nur-packages/pkgs/top-level

to reproduce

cd /tmp
git clone https://github.com/milahu/nur-packages
cd nur-packages
git checkout 670a6637e9488bbd2f8bc00e3c8dc58a9fe1f1bc
./pkgs/tools/archivers/sevenzip/update.sh

update.sh fails at

NEW_FREE_HASH=$(nix-prefetch -f "$NIXPKGS_ROOT" -E "$attr_path.src" --url "$URL")

which evals to

nix-prefetch -f /tmp/nur-packages -E sevenzip.src --url https://7-zip.org/a/7z2200-src.tar.xz

ideally, nix-prefetch should not rely on the internal structure of nixpkgs (pkgs/top-level)
and only use default.nix or flake.nix as entry point

just like nix-build

nix-build /tmp/nur-packages -A sevenzip

workaround:

- nix-prefetch -f "$NIXPKGS_ROOT" -E "$attr_path.src" --url "$URL"
+ nix-prefetch -f "<nixpkgs>" -E "(import $NIXPKGS_ROOT {}).$attr_path.src" --url "$URL"

Currently when trying to use nix-prefetch with something that requires flakes it will fail, it is testing for flake but only adds --experimental-features nix-command:
https://github.com/msteen/nix-prefetch/blob/master/src/main.sh#L163

I have a flake that provides a packages that is using fixed output hashes (go-modules vendor stuff). I would like to do something like:

nix-prefetch '{ sha256 }: (builtins.getFlake (toString ./.)).packages.x86_64-linux.terraform-provider-aws.go-modules.overrideAttrs (_: { vendorSha256 = sha256; })'

This commands will refuse to evaluate due to flakes feature not being enabled.

Would it make sense to add this, or is it possible to provide extra flags to the nix command via the nix-prefetch CLI?

Hey y'all,
thanks for your work on nix-prefetch.

I'm trying to fetch some mercurial from a private repo server,
pretty straight forward call like

nix-prefetch '{fetchhg}: fetchhg {
          url = "https://{url omitted}";
          rev = "ee5d52c08d9a0d9c89003931cae116af1895e7b6";
          sha256 = "sha256-I9QPJiur0MmKT3RzRWV7l9H4ZqO4POTKxBEL5Nvexfk=";
        }

and I end up with

abort: error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate

My cursory debugging suggests that the nix-prefetch derived derivation uses 'secure-hg' while
my own flakes derivation (which does work) uses plain mercurial?

Thanks for having a look!

When nix-prefetch launched with both --verbose and --compute-hash hash will not appear

0.3.1 is 5-months-old now. On nixpkgs it's also patched (for #8) since there's no later release.

I've found some peculiar behavior on macOS in which the fetchurl prefetcher works when invoked through nix run and nix-shell, but fails when invoked directly.

Namely, the following works:

❯ nix run 'nixpkgs#nix-prefetch' -- fetchurl --url https://pypi.io/packages/source/f/feeluown/feeluown-3.7.11.tar.gz
warning: unknown setting 'experimental-features'
warning: unknown setting 'experimental-features'
The fetcher will be called as follows:
> fetchurl {
>   sha256 = "sha256:0000000000000000000000000000000000000000000000000000";
>   url = "https://pypi.io/packages/source/f/feeluown/feeluown-3.7.11.tar.gz";
> }

sha256-cXN6W6LV0QnTgQTEFYvXYPgL3YggUdB7gh2TvjXnkHg=

But this doesn't:

❯ nix path-info 'nixpkgs#nix-prefetch'
/nix/store/8anr2v3g5xz5m93746732zckqbq2yz5w-nix-prefetch-0.4.1
❯ /nix/store/8anr2v3g5xz5m93746732zckqbq2yz5w-nix-prefetch-0.4.1/bin/nix-prefetch fetchurl --url https://pypi.io/packages/source/f/feeluown/feeluown-3.7.11.tar.gz
warning: unknown setting 'experimental-features'
warning: unknown setting 'experimental-features'
error: Could not determine the fetcher used by the source.

After further investigation, I've pinpointed the problem to the value of the TMPDIR environment variable. For some reason, nix-prefetch doesn't seem to like the default value of TMPDIR provided by the macOS system. Setting TMPDIR (or XDG_RUNTIME_DIR) to /tmp seems to fix the problem:

❯  TMPDIR=/tmp /nix/store/8anr2v3g5xz5m93746732zckqbq2yz5w-nix-prefetch-0.4.1/bin/nix-prefetch fetchurl --url https://pypi.io/packages/source/f/feeluown/feeluown-3.7.11.tar.gz
warning: unknown setting 'experimental-features'
warning: unknown setting 'experimental-features'
The fetcher will be called as follows:
> fetchurl {
>   sha256 = "sha256:0000000000000000000000000000000000000000000000000000";
>   url = "https://pypi.io/packages/source/f/feeluown/feeluown-3.7.11.tar.gz";
> }

sha256-cXN6W6LV0QnTgQTEFYvXYPgL3YggUdB7gh2TvjXnkHg=

Running nix-prefetch through nix run or nix-shell worked because Nix sets TMPDIR to /tmp, thus avoiding the problem.

I'm not sure how best to deal with this problem, but maybe nix-prefetch should ignore TMPDIR on macOS?

The tag for the current version (0.4.1) is out of this source tree, which is quite confusing.

I'm not sure if there's a way to "fix" the issue aside from another in-tree tag for the next version.

Some fetchers (like fetchrepoproject) take lists, which I cannot seem to figure out how to pass to nix-prefetch.

Using --input nix or --input json fails with jq errors, regular command line arguments seem to turn the list into a string.

I tried some more things, but I doubt they are relevant. I would love to be able to use nix-prefetch, but without being able to pass lists to it somehow, it sadly does not fulfill my requirements.

Hello, I'm getting these errors when I try to run on MacOS with v0.3.1:

$ nix-prefetch hello
error: Could not determine the runtime directory (i.e. XDG_RUNTIME_DIR).

$ XDG_RUNTIME_DIR=$HOME/.cache nix-prefetch hello
error: Unsupported input type ''.

# Specifying an input type
$ XDG_RUNTIME_DIR=$HOME/.cache nix-prefetch hello --input nix
error: Unsupported input type 'nix'.

Ran across what I think is a bug while debugging a invalidPath error with cachix. From what I understand, nix-prefetch works by building with a fake sha256, letting the build fail, and then extracting the correct sha256. This has the side effect of leaving invalid paths in the store. Take the following for example (clean install).

> ls /nix/store | grep versions.json
# nothing
> nix-prefetch pkgs.fetchurl --url https://julialang-s3.julialang.org/bin/versions.json > /dev/null
> ls /nix/store | grep versions.json
7rdyclssbgcfwvhmcwp3aqcqyfmbb9ys-versions.json.lock
ha94j211qk3vh124irxc9qc801754hzf-versions.json
ha94j211qk3vh124irxc9qc801754hzf-versions.json.lock
> nix store verify /nix/store/ha94j211qk3vh124irxc9qc801754hzf-versions.json
> nix store verify /nix/store/7rdyclssbgcfwvhmcwp3aqcqyfmbb9ys-versions.json
error: path '/nix/store/7rdyclssbgcfwvhmcwp3aqcqyfmbb9ys-versions.json' is not valid

So it looks like the 7rdyclssbgcfwvhmcwp3aqcqyfmbb9ys-versions.json.lock is from the failed build with the fake sha256. Cachix throws an error when it sees this because, well, it's invalid. They should probably read paths from nix-store --dump-db but is deleting/preventing the invalid store path within the scope of nix-prefetch?

nix-prefetch-{git,url,etc} don't leave invalid paths behind, for comparison.

When running the following on a system without any channels in Nix path, it fails:

$ nix-prefetch hello.src
warning: unknown setting 'experimental-features'
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)

With more people starting to use flakes exclusively, it would be useful not to rely on NIX_PATH.

I'm on current nixpkgs master.
Command: nix-prefetch cargo-flamegraph.cargoDeps --fetcher '<nixpkgs/pkgs/build-support/rust/fetchcargo.nix>'
Outputs:

warning: unknown setting 'experimental-features'
warning: unknown setting 'flake-registry'
warning: unknown setting 'experimental-features'
warning: unknown setting 'flake-registry'
error: attribute 'override' missing, at /nix/store/0x8sqpcg4kcpvw5nqabd8vhmn93da1f2-nix-prefetch-0.3.1/lib/nix-prefetch/fetcher.nix:37:65

Given the following rust package definition:

# This works:
{ lib, fetchzip, rustPlatform }:
rustPlatform.buildRustPackage rec {
  pname = "lscolors";
  version = "0.7.1";
  src = fetchzip {
    url = "https://crates.io/api/v1/crates/lscolors/0.7.1/download#crate.tar.gz";
    sha256 = "1kli299gg3vafjj0vbrfmwcaawq23c9dw31q25i2g3n49pyfic62";
  };
  cargoSha256 = "sha256:0dxry46dizjx7j9sdmg891dh7ji29znxdj3clcf30a27ni6ydb76";
}

# This doesn't:
# { lib, fetchCrate, rustPlatform }:
# rustPlatform.buildRustPackage rec {
#   pname = "lscolors";
#   version = "0.7.1";
#   src = fetchCrate {
#     inherit pname version;
#     sha256 = "1kli299gg3vafjj0vbrfmwcaawq23c9dw31q25i2g3n49pyfic62";
#   };
#   cargoSha256 = "sha256:0dxry46dizjx7j9sdmg891dh7ji29znxdj3clcf30a27ni6ydb76";
# }

running

nix-prefetch '{ sha256 }: (import ./default.nix {}).lscolors2.cargoDeps.overrideAttrs (_: { cargoSha256 = sha256; })'

works just fine when using fetchzip but when using fetchcrate (the commented-out block above), I get the following error:

error: attribute 'overrideAttrs' missing, at /nix/store/blqi2h6ijjanswmv20gwqx39232224ax-nix-prefetch-0.4.0/lib/nix-prefetch/prefetcher.nix:87:10

This appears to be unique to the fetchcrate fetcher. I'm guessing this is related to how fetchcrate is defined (using overrideDerivation)?

First of all, thank you for this great project!

nix-prefetch has already become mission-critical in the Nix ecosystem. However, the development seems halted, and Nixpkgs maintainer(s) sometimes need to bump the version with out-of-tree commit(s) (#33). It would be great if this project can be put under a shared-ownership environment, so that more people can join force with its development and maintenance.

The most direct way to integrate deeper with Nixpkgs is to merge into it. This way, all the Nixpkgs committers will be able to add changes and merge related PRs, and tests can be added to ensure that it works as other part of Nixpkgs changes.

To retain the administration over the project while inviting more people to become a committer, you can also get it included as a @nix-community project. That's also how rycee's Home Manager goes (it's still under active development). Such project can still be the upstream of an Nixpkgs package, taking poetry2nix as an example.

consider:

> nix-prefetch fetchgit --url https://github.com/msteen/nix-prefetch --fetchLFS
error: attribute 'rev' missing

       at /nix/store/hg08xxk8z4sjbk3fnblwns635vm2fqy1-nixpkgs-22.05pre335095.1fbcb733eb9/nixpkgs/pkgs/applications/version-management/git-lfs/default.nix:18:112:

           17|
           18|   ldflags = [ "-s" "-w" "-X ${goPackagePath}/config.Vendor=${version}" "-X ${goPackagePath}/config.GitCommit=${src.rev}" ];
             |                                                                                                                ^
           19|
(use '--show-trace' to show detailed location information)

vs without --fetchLFS it works fine:

> nix-prefetch fetchgit --url https://github.com/msteen/nix-prefetch
The fetcher will be called as follows:
> fetchgit {
>   sha256 = "sha256:0000000000000000000000000000000000000000000000000000";
>   url = "https://github.com/msteen/nix-prefetch";
> }

sha256-QftUvzdlEHl7yDMnogfW8aqGaxASOVNGKsqeWcez73c=

It also works fine if i explicitly set --no-fetchLFS:

nix-prefetch fetchgit --url https://github.com/msteen/nix-prefetch --no-fetchLFS
The fetcher will be called as follows:
> fetchgit {
>   fetchLFS = false;
>   sha256 = "sha256:0000000000000000000000000000000000000000000000000000";
>   url = "https://github.com/msteen/nix-prefetch";
> }

sha256-QftUvzdlEHl7yDMnogfW8aqGaxASOVNGKsqeWcez73c=

version:

> nix-prefetch --version
0.4.1

-- Edit:

This is with nixpkgs 22.05 ; apparently it works with nixpkgs-unstable

Thanks for writing this tool. It would be create if it would also support modSha256 of our new buildGoModule function.
cc @kalbasit

When using nix-prefetch (both master branch and the version in nixpkgs 20.03), I seem to be getting an error like this:

[mcgibbon@xps13:~/nixcfg/pkgs]$ nix-prefetch '(import ./. {}).pypyp'
error: getting status of '/run/user/1000/nix-prefetch/tmp.5UxKFqcQb3/pkgs': No such file or directory

It appears to come up only when I have overlays configured -- if I turn off my overlays the error magically disappears. Perhaps this is user error. If so, apologies for the noise.

--output nix currently outputs only the arguments to the fetcher. It would be nice if there's an output type like nix that includes the name of the fetcher along with the arguments.

As of release 0.4.1, I observe the following behavior.

$ uname -ris
Darwin 19.6.0 MacBookPro11,1
$ nix-prefetch fetchFromGitHub --help
error: Unsupported input type ''.

$ uname -ris
Linux 3.10.0-862.14.4.el7.x86_64 x86_64
$ nix-prefetch fetchFromGitHub --help
Prefetch the fetchFromGitHub function call

Usage:
  nix-prefetch fetchFromGitHub
...

Currently nix-prefetch breaks with nixFlakes because it prints the new hashes.

NixOS/nix#3786 (comment)

Notice this also is broken if the old nix is in PATH as the message is generated by nix-daemon.

We use the same FOD output hash 0000000000000000000000000000000000000000000000000000 for prefetching.

This makes all output store paths collide with each other since FODs are content addressed. A direct result is that when doing two unrelated nix-prefetch, one would lock the result path (in nix-store --realise) and blocks others, making prefetching effectively single-threaded.

One possible solution is to make the fake hash random, or somewhat hashed from the derivation itself.

I'm using nix-prefetch through nix-update and I'm getting an error

I've seen in one of the other issues there's a rewrite in progress so I guess this might already be addressed
Or also it could be because it's using nix-prefetch 0.3.1 rather than 0.4.0

From nix-update wrapper: /nix/store/bg9jc33jylaiqkhj0jx046r3ksj7rx6v-nix-prefetch-0.3.1/bin

nix-update nerdctl
$ nix eval --json --impure --experimental-features nix-command --expr (with import ./. {};
    let
      pkg = nerdctl;
      raw_version_position = builtins.unsafeGetAttrPos "version" pkg;

      position = if pkg ? isRubyGem then
        raw_version_position
      else
        builtins.unsafeGetAttrPos "src" pkg;
    in {
      name = pkg.name;
      old_version = (builtins.parseDrvName pkg.name).version;
      inherit raw_version_position;
      filename = position.file;
      line = position.line;
      urls = pkg.src.urls or null;
      url = pkg.src.url or null;
      rev = pkg.src.url.rev or null;
      hash = pkg.src.outputHash;
      mod_sha256 = pkg.modSha256 or null;
      vendor_sha256 = pkg.vendorSha256 or null;
      cargo_sha256 = pkg.cargoSha256 or null;
      tests = pkg.passthru.tests or null;
    })
fetch https://github.com/AkihiroSuda/nerdctl/releases.atom
Update 0.6.0 -> 0.6.1 in /home/jk/projects/personal/nixpkgs/pkgs/applications/networking/cluster/nerdctl/default.nix
$ nix-prefetch (import ./. {}).nerdctl
warning: unknown setting 'experimental-features'
warning: unknown setting 'experimental-features'
The package nerdctl-0.6.1 will be fetched as follows:
> fetchFromGitHub {
>   owner = "AkihiroSuda";
>   repo = "nerdctl";
>   rev = "v0.6.1";
>   sha256 = "sha256-QhAN30ge0dbC9dGT1yP4o0VgrcS9+g+r6YJ07ZjPJtg=";
> }

A probably-wrong output hash of zeroes has been used, yet it somehow still succeeded in building.
Please report an issue at: https://github.com/msteen/nix-prefetch/issues
Traceback (most recent call last):
  File "/nix/store/lagdrmaal96m49fh9p012g58mlyvzqa4-nix-update-0.3.2/bin/.nix-update-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/lagdrmaal96m49fh9p012g58mlyvzqa4-nix-update-0.3.2/lib/python3.8/site-packages/nix_update/__init__.py", line 172, in main
    package = update(options)
  File "/nix/store/lagdrmaal96m49fh9p012g58mlyvzqa4-nix-update-0.3.2/lib/python3.8/site-packages/nix_update/update.py", line 139, in update
    update_src_hash(opts, package.filename, package.hash)
  File "/nix/store/lagdrmaal96m49fh9p012g58mlyvzqa4-nix-update-0.3.2/lib/python3.8/site-packages/nix_update/update.py", line 84, in update_src_hash
    target_hash = nix_prefetch([f"(import {opts.import_path} {{}}).{opts.attribute}"])
  File "/nix/store/lagdrmaal96m49fh9p012g58mlyvzqa4-nix-update-0.3.2/lib/python3.8/site-packages/nix_update/update.py", line 76, in nix_prefetch
    res = run(["nix-prefetch"] + cmd, extra_env=extra_env)
  File "/nix/store/lagdrmaal96m49fh9p012g58mlyvzqa4-nix-update-0.3.2/lib/python3.8/site-packages/nix_update/utils.py", line 36, in run
    return subprocess.run(
  File "/nix/store/papbnjfxwrdkmz6xfp7v2q7nxyvgcr5l-python3-3.8.6/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['nix-prefetch', '(import ./. {}).nerdctl']' returned non-zero exit status 1.

Hi, I'm using this tool to keep an overlay set of packages rev/sha256 to be up to date. However, one of my Rust packages has started updating their cargo multiple times a day, requiring manual intervention.

Could this tool learn how to handle cargoSha256 cases as well?

As mentioned by @ajs124 in #1 there is a bug with the custom overlay injection code. The error given is getting status of '/run/user/1000/nix-prefetch/prelude.nix': No such file or directory, which can only occur if either $lib (in main.sh) or . (in Nix) point to /run/user/1000/nix-prefetch. After inspecting the code this seems to be caused by this line of code. It should probably be replaced an import of $lib/overlay.nix rather than inlining its content, which contains the problematic import ./prelude.nix.

Edit: This reproduces the problem, so my assumption was correct: NIX_PATH=nixpkgs-overlays=$PWD/test.nix:$NIX_PATH nix-prefetch hello

wrong project