msteen / nix-prefetch Goto Github PK
View Code? Open in Web Editor NEWPrefetch any fetcher function call, e.g. a package source
License: MIT License
Prefetch any fetcher function call, e.g. a package source
License: MIT License
nix-prefetch fails on nur-packages
error: Could not find Nixpkgs path: /tmp/nur-packages/pkgs/top-level
to reproduce
cd /tmp
git clone https://github.com/milahu/nur-packages
cd nur-packages
git checkout 670a6637e9488bbd2f8bc00e3c8dc58a9fe1f1bc
./pkgs/tools/archivers/sevenzip/update.sh
update.sh fails at
NEW_FREE_HASH=$(nix-prefetch -f "$NIXPKGS_ROOT" -E "$attr_path.src" --url "$URL")
which evals to
nix-prefetch -f /tmp/nur-packages -E sevenzip.src --url https://7-zip.org/a/7z2200-src.tar.xz
ideally, nix-prefetch should not rely on the internal structure of nixpkgs (pkgs/top-level)
and only use default.nix or flake.nix as entry point
just like nix-build
nix-build /tmp/nur-packages -A sevenzip
workaround:
- nix-prefetch -f "$NIXPKGS_ROOT" -E "$attr_path.src" --url "$URL"
+ nix-prefetch -f "<nixpkgs>" -E "(import $NIXPKGS_ROOT {}).$attr_path.src" --url "$URL"
Currently when trying to use nix-prefetch
with something that requires flakes it will fail, it is testing for flake but only adds --experimental-features nix-command
:
https://github.com/msteen/nix-prefetch/blob/master/src/main.sh#L163
I have a flake that provides a packages that is using fixed output hashes (go-modules vendor stuff). I would like to do something like:
nix-prefetch '{ sha256 }: (builtins.getFlake (toString ./.)).packages.x86_64-linux.terraform-provider-aws.go-modules.overrideAttrs (_: { vendorSha256 = sha256; })'
This commands will refuse to evaluate due to flakes
feature not being enabled.
Would it make sense to add this, or is it possible to provide extra flags to the nix command via the nix-prefetch
CLI?
Hey y'all,
thanks for your work on nix-prefetch.
I'm trying to fetch some mercurial from a private repo server,
pretty straight forward call like
nix-prefetch '{fetchhg}: fetchhg {
url = "https://{url omitted}";
rev = "ee5d52c08d9a0d9c89003931cae116af1895e7b6";
sha256 = "sha256-I9QPJiur0MmKT3RzRWV7l9H4ZqO4POTKxBEL5Nvexfk=";
}
and I end up with
abort: error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate
My cursory debugging suggests that the nix-prefetch derived derivation uses 'secure-hg' while
my own flakes derivation (which does work) uses plain mercurial?
Thanks for having a look!
When nix-prefetch
launched with both --verbose
and --compute-hash
hash will not appear
0.3.1 is 5-months-old now. On nixpkgs it's also patched (for #8) since there's no later release.
I've found some peculiar behavior on macOS in which the fetchurl
prefetcher works when invoked through nix run
and nix-shell
, but fails when invoked directly.
Namely, the following works:
❯ nix run 'nixpkgs#nix-prefetch' -- fetchurl --url https://pypi.io/packages/source/f/feeluown/feeluown-3.7.11.tar.gz
warning: unknown setting 'experimental-features'
warning: unknown setting 'experimental-features'
The fetcher will be called as follows:
> fetchurl {
> sha256 = "sha256:0000000000000000000000000000000000000000000000000000";
> url = "https://pypi.io/packages/source/f/feeluown/feeluown-3.7.11.tar.gz";
> }
sha256-cXN6W6LV0QnTgQTEFYvXYPgL3YggUdB7gh2TvjXnkHg=
But this doesn't:
❯ nix path-info 'nixpkgs#nix-prefetch'
/nix/store/8anr2v3g5xz5m93746732zckqbq2yz5w-nix-prefetch-0.4.1
❯ /nix/store/8anr2v3g5xz5m93746732zckqbq2yz5w-nix-prefetch-0.4.1/bin/nix-prefetch fetchurl --url https://pypi.io/packages/source/f/feeluown/feeluown-3.7.11.tar.gz
warning: unknown setting 'experimental-features'
warning: unknown setting 'experimental-features'
error: Could not determine the fetcher used by the source.
After further investigation, I've pinpointed the problem to the value of the TMPDIR
environment variable. For some reason, nix-prefetch doesn't seem to like the default value of TMPDIR
provided by the macOS system. Setting TMPDIR
(or XDG_RUNTIME_DIR
) to /tmp
seems to fix the problem:
❯ TMPDIR=/tmp /nix/store/8anr2v3g5xz5m93746732zckqbq2yz5w-nix-prefetch-0.4.1/bin/nix-prefetch fetchurl --url https://pypi.io/packages/source/f/feeluown/feeluown-3.7.11.tar.gz
warning: unknown setting 'experimental-features'
warning: unknown setting 'experimental-features'
The fetcher will be called as follows:
> fetchurl {
> sha256 = "sha256:0000000000000000000000000000000000000000000000000000";
> url = "https://pypi.io/packages/source/f/feeluown/feeluown-3.7.11.tar.gz";
> }
sha256-cXN6W6LV0QnTgQTEFYvXYPgL3YggUdB7gh2TvjXnkHg=
Running nix-prefetch through nix run
or nix-shell
worked because Nix sets TMPDIR
to /tmp
, thus avoiding the problem.
I'm not sure how best to deal with this problem, but maybe nix-prefetch should ignore TMPDIR
on macOS?
Line 1 in 1113458
Some fetchers (like fetchrepoproject) take lists, which I cannot seem to figure out how to pass to nix-prefetch.
Using --input nix
or --input json
fails with jq errors, regular command line arguments seem to turn the list into a string.
I tried some more things, but I doubt they are relevant. I would love to be able to use nix-prefetch, but without being able to pass lists to it somehow, it sadly does not fulfill my requirements.
Hello, I'm getting these errors when I try to run on MacOS with v0.3.1:
$ nix-prefetch hello
error: Could not determine the runtime directory (i.e. XDG_RUNTIME_DIR).
$ XDG_RUNTIME_DIR=$HOME/.cache nix-prefetch hello
error: Unsupported input type ''.
# Specifying an input type
$ XDG_RUNTIME_DIR=$HOME/.cache nix-prefetch hello --input nix
error: Unsupported input type 'nix'.
Ran across what I think is a bug while debugging a invalidPath
error with cachix. From what I understand, nix-prefetch works by building with a fake sha256, letting the build fail, and then extracting the correct sha256. This has the side effect of leaving invalid paths in the store. Take the following for example (clean install).
> ls /nix/store | grep versions.json
# nothing
> nix-prefetch pkgs.fetchurl --url https://julialang-s3.julialang.org/bin/versions.json > /dev/null
> ls /nix/store | grep versions.json
7rdyclssbgcfwvhmcwp3aqcqyfmbb9ys-versions.json.lock
ha94j211qk3vh124irxc9qc801754hzf-versions.json
ha94j211qk3vh124irxc9qc801754hzf-versions.json.lock
> nix store verify /nix/store/ha94j211qk3vh124irxc9qc801754hzf-versions.json
> nix store verify /nix/store/7rdyclssbgcfwvhmcwp3aqcqyfmbb9ys-versions.json
error: path '/nix/store/7rdyclssbgcfwvhmcwp3aqcqyfmbb9ys-versions.json' is not valid
So it looks like the 7rdyclssbgcfwvhmcwp3aqcqyfmbb9ys-versions.json.lock is from the failed build with the fake sha256. Cachix throws an error when it sees this because, well, it's invalid. They should probably read paths from nix-store --dump-db
but is deleting/preventing the invalid store path within the scope of nix-prefetch?
nix-prefetch-{git,url,etc} don't leave invalid paths behind, for comparison.
When running the following on a system without any channels in Nix path, it fails:
$ nix-prefetch hello.src
warning: unknown setting 'experimental-features'
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)
With more people starting to use flakes exclusively, it would be useful not to rely on NIX_PATH.
I'm on current nixpkgs master.
Command: nix-prefetch cargo-flamegraph.cargoDeps --fetcher '<nixpkgs/pkgs/build-support/rust/fetchcargo.nix>'
Outputs:
warning: unknown setting 'experimental-features'
warning: unknown setting 'flake-registry'
warning: unknown setting 'experimental-features'
warning: unknown setting 'flake-registry'
error: attribute 'override' missing, at /nix/store/0x8sqpcg4kcpvw5nqabd8vhmn93da1f2-nix-prefetch-0.3.1/lib/nix-prefetch/fetcher.nix:37:65
Given the following rust package definition:
# This works:
{ lib, fetchzip, rustPlatform }:
rustPlatform.buildRustPackage rec {
pname = "lscolors";
version = "0.7.1";
src = fetchzip {
url = "https://crates.io/api/v1/crates/lscolors/0.7.1/download#crate.tar.gz";
sha256 = "1kli299gg3vafjj0vbrfmwcaawq23c9dw31q25i2g3n49pyfic62";
};
cargoSha256 = "sha256:0dxry46dizjx7j9sdmg891dh7ji29znxdj3clcf30a27ni6ydb76";
}
# This doesn't:
# { lib, fetchCrate, rustPlatform }:
# rustPlatform.buildRustPackage rec {
# pname = "lscolors";
# version = "0.7.1";
# src = fetchCrate {
# inherit pname version;
# sha256 = "1kli299gg3vafjj0vbrfmwcaawq23c9dw31q25i2g3n49pyfic62";
# };
# cargoSha256 = "sha256:0dxry46dizjx7j9sdmg891dh7ji29znxdj3clcf30a27ni6ydb76";
# }
running
nix-prefetch '{ sha256 }: (import ./default.nix {}).lscolors2.cargoDeps.overrideAttrs (_: { cargoSha256 = sha256; })'
works just fine when using fetchzip
but when using fetchcrate
(the commented-out block above), I get the following error:
error: attribute 'overrideAttrs' missing, at /nix/store/blqi2h6ijjanswmv20gwqx39232224ax-nix-prefetch-0.4.0/lib/nix-prefetch/prefetcher.nix:87:10
This appears to be unique to the fetchcrate
fetcher. I'm guessing this is related to how fetchcrate
is defined (using overrideDerivation
)?
First of all, thank you for this great project!
nix-prefetch has already become mission-critical in the Nix ecosystem. However, the development seems halted, and Nixpkgs maintainer(s) sometimes need to bump the version with out-of-tree commit(s) (#33). It would be great if this project can be put under a shared-ownership environment, so that more people can join force with its development and maintenance.
The most direct way to integrate deeper with Nixpkgs is to merge into it. This way, all the Nixpkgs committers will be able to add changes and merge related PRs, and tests can be added to ensure that it works as other part of Nixpkgs changes.
To retain the administration over the project while inviting more people to become a committer, you can also get it included as a @nix-community project. That's also how rycee's Home Manager goes (it's still under active development). Such project can still be the upstream of an Nixpkgs package, taking poetry2nix as an example.
consider:
> nix-prefetch fetchgit --url https://github.com/msteen/nix-prefetch --fetchLFS
error: attribute 'rev' missing
at /nix/store/hg08xxk8z4sjbk3fnblwns635vm2fqy1-nixpkgs-22.05pre335095.1fbcb733eb9/nixpkgs/pkgs/applications/version-management/git-lfs/default.nix:18:112:
17|
18| ldflags = [ "-s" "-w" "-X ${goPackagePath}/config.Vendor=${version}" "-X ${goPackagePath}/config.GitCommit=${src.rev}" ];
| ^
19|
(use '--show-trace' to show detailed location information)
vs without --fetchLFS
it works fine:
> nix-prefetch fetchgit --url https://github.com/msteen/nix-prefetch
The fetcher will be called as follows:
> fetchgit {
> sha256 = "sha256:0000000000000000000000000000000000000000000000000000";
> url = "https://github.com/msteen/nix-prefetch";
> }
sha256-QftUvzdlEHl7yDMnogfW8aqGaxASOVNGKsqeWcez73c=
It also works fine if i explicitly set --no-fetchLFS
:
nix-prefetch fetchgit --url https://github.com/msteen/nix-prefetch --no-fetchLFS
The fetcher will be called as follows:
> fetchgit {
> fetchLFS = false;
> sha256 = "sha256:0000000000000000000000000000000000000000000000000000";
> url = "https://github.com/msteen/nix-prefetch";
> }
sha256-QftUvzdlEHl7yDMnogfW8aqGaxASOVNGKsqeWcez73c=
version:
> nix-prefetch --version
0.4.1
-- Edit:
This is with nixpkgs 22.05 ; apparently it works with nixpkgs-unstable
Thanks for writing this tool. It would be create if it would also support modSha256
of our new buildGoModule function.
cc @kalbasit
When using nix-prefetch (both master branch and the version in nixpkgs 20.03), I seem to be getting an error like this:
[mcgibbon@xps13:~/nixcfg/pkgs]$ nix-prefetch '(import ./. {}).pypyp'
error: getting status of '/run/user/1000/nix-prefetch/tmp.5UxKFqcQb3/pkgs': No such file or directory
It appears to come up only when I have overlays configured -- if I turn off my overlays the error magically disappears. Perhaps this is user error. If so, apologies for the noise.
--output nix
currently outputs only the arguments to the fetcher. It would be nice if there's an output type like nix
that includes the name of the fetcher along with the arguments.
As of release 0.4.1, I observe the following behavior.
$ uname -ris
Darwin 19.6.0 MacBookPro11,1
$ nix-prefetch fetchFromGitHub --help
error: Unsupported input type ''.
$ uname -ris
Linux 3.10.0-862.14.4.el7.x86_64 x86_64
$ nix-prefetch fetchFromGitHub --help
Prefetch the fetchFromGitHub function call
Usage:
nix-prefetch fetchFromGitHub
...
Currently nix-prefetch breaks with nixFlakes
because it prints the new hashes.
Notice this also is broken if the old nix is in PATH as the message is generated by nix-daemon.
We use the same FOD output hash 0000000000000000000000000000000000000000000000000000
for prefetching.
Line 231 in a4773bf
This makes all output store paths collide with each other since FODs are content addressed. A direct result is that when doing two unrelated nix-prefetch
, one would lock the result path (in nix-store --realise
) and blocks others, making prefetching effectively single-threaded.
One possible solution is to make the fake hash random, or somewhat hashed from the derivation itself.
I'm using nix-prefetch
through nix-update
and I'm getting an error
I've seen in one of the other issues there's a rewrite in progress so I guess this might already be addressed
Or also it could be because it's using nix-prefetch
0.3.1
rather than 0.4.0
From nix-update
wrapper: /nix/store/bg9jc33jylaiqkhj0jx046r3ksj7rx6v-nix-prefetch-0.3.1/bin
nix-update nerdctl
$ nix eval --json --impure --experimental-features nix-command --expr (with import ./. {};
let
pkg = nerdctl;
raw_version_position = builtins.unsafeGetAttrPos "version" pkg;
position = if pkg ? isRubyGem then
raw_version_position
else
builtins.unsafeGetAttrPos "src" pkg;
in {
name = pkg.name;
old_version = (builtins.parseDrvName pkg.name).version;
inherit raw_version_position;
filename = position.file;
line = position.line;
urls = pkg.src.urls or null;
url = pkg.src.url or null;
rev = pkg.src.url.rev or null;
hash = pkg.src.outputHash;
mod_sha256 = pkg.modSha256 or null;
vendor_sha256 = pkg.vendorSha256 or null;
cargo_sha256 = pkg.cargoSha256 or null;
tests = pkg.passthru.tests or null;
})
fetch https://github.com/AkihiroSuda/nerdctl/releases.atom
Update 0.6.0 -> 0.6.1 in /home/jk/projects/personal/nixpkgs/pkgs/applications/networking/cluster/nerdctl/default.nix
$ nix-prefetch (import ./. {}).nerdctl
warning: unknown setting 'experimental-features'
warning: unknown setting 'experimental-features'
The package nerdctl-0.6.1 will be fetched as follows:
> fetchFromGitHub {
> owner = "AkihiroSuda";
> repo = "nerdctl";
> rev = "v0.6.1";
> sha256 = "sha256-QhAN30ge0dbC9dGT1yP4o0VgrcS9+g+r6YJ07ZjPJtg=";
> }
A probably-wrong output hash of zeroes has been used, yet it somehow still succeeded in building.
Please report an issue at: https://github.com/msteen/nix-prefetch/issues
Traceback (most recent call last):
File "/nix/store/lagdrmaal96m49fh9p012g58mlyvzqa4-nix-update-0.3.2/bin/.nix-update-wrapped", line 9, in <module>
sys.exit(main())
File "/nix/store/lagdrmaal96m49fh9p012g58mlyvzqa4-nix-update-0.3.2/lib/python3.8/site-packages/nix_update/__init__.py", line 172, in main
package = update(options)
File "/nix/store/lagdrmaal96m49fh9p012g58mlyvzqa4-nix-update-0.3.2/lib/python3.8/site-packages/nix_update/update.py", line 139, in update
update_src_hash(opts, package.filename, package.hash)
File "/nix/store/lagdrmaal96m49fh9p012g58mlyvzqa4-nix-update-0.3.2/lib/python3.8/site-packages/nix_update/update.py", line 84, in update_src_hash
target_hash = nix_prefetch([f"(import {opts.import_path} {{}}).{opts.attribute}"])
File "/nix/store/lagdrmaal96m49fh9p012g58mlyvzqa4-nix-update-0.3.2/lib/python3.8/site-packages/nix_update/update.py", line 76, in nix_prefetch
res = run(["nix-prefetch"] + cmd, extra_env=extra_env)
File "/nix/store/lagdrmaal96m49fh9p012g58mlyvzqa4-nix-update-0.3.2/lib/python3.8/site-packages/nix_update/utils.py", line 36, in run
return subprocess.run(
File "/nix/store/papbnjfxwrdkmz6xfp7v2q7nxyvgcr5l-python3-3.8.6/lib/python3.8/subprocess.py", line 512, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['nix-prefetch', '(import ./. {}).nerdctl']' returned non-zero exit status 1.
Hi, I'm using this tool to keep an overlay set of packages rev/sha256 to be up to date. However, one of my Rust packages has started updating their cargo multiple times a day, requiring manual intervention.
Could this tool learn how to handle cargoSha256
cases as well?
As mentioned by @ajs124 in #1 there is a bug with the custom overlay injection code. The error given is getting status of '/run/user/1000/nix-prefetch/prelude.nix': No such file or directory
, which can only occur if either $lib
(in main.sh
) or .
(in Nix) point to /run/user/1000/nix-prefetch
. After inspecting the code this seems to be caused by this line of code. It should probably be replaced an import of $lib/overlay.nix
rather than inlining its content, which contains the problematic import ./prelude.nix
.
Edit: This reproduces the problem, so my assumption was correct: NIX_PATH=nixpkgs-overlays=$PWD/test.nix:$NIX_PATH nix-prefetch hello
wrong project
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.