panamax-rs / panamax Goto Github PK
View Code? Open in Web Editor NEWMirror rustup and crates.io repositories, for offline Rust and cargo usage.
License: Apache License 2.0
Mirror rustup and crates.io repositories, for offline Rust and cargo usage.
License: Apache License 2.0
Hello! How do I configure panamax to pull down additional compilation targets? That is, things that you would normally get with something along the lines of rustup target add aarch64-linux-android
. I figured pinned_rust_versions
was the closest option, but if it is I can't seem to find the right settings (stable-aarch64-linux-android
doesn't seem to work, for instance).
Is it possible to download only speciefic crates, e.g., panamax sync my-mirror --crates="serenity,rand,songbird"
?
When trying to run the latest container image (tag 1.0.6) I get the following symbol issue.
/usr/local/bin/panamax: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.29' not found (required by /usr/local/bin/panamax
This occurs with the init, sync, and serve commands.
Right now, Panamax handles pinning previous stable versions in the form of e.g. "1.42", however it does not support other version types like "nightly-2021-09-08".
This is because Panamax grabs channel toml files from "static.rust-lang.org/dist/channel-rust-.toml" where version is "stable", "beta", "nightly", or a version like "1.42".
Unfortunately, the historical nightly versions do not follow this trend, and instead this channel file needs to be grabbed from "static.rust-lang.org/dist/2021-09-08/channel-rust-nightly.toml".
I've noticed a lot of issues with how git's handled. I originally wanted to use a bare repo to save space, however the complexity of handling it is very high, and has likely indirectly caused a lot of the other issues on this issue tracker.
This should be rewritten so instead of using a bare repo, a regular git repo is used instead, with regular file access rather than access through libgit2.
I usually set net.git-fetch-with-cli=true
inside .cargo/config.toml
. However, this does not work with panamax. Taking a look at the documentation reveals that CARGO_NET_GIT_FETCH_WITH_CLI
can also be used. But that does not work for panamax as well.
In commit 89a2d64 the list of rustup-supported target-triples began to be fetched on each panamax sync. All Unix target-triples are now defined as the inverse of the hard-coded Windows ones.
While explicitly specifying Windows targets to limit the download does work, this change prevents filtering / limiting the Unix platforms being downloaded on sync, see relevant code section.
A simple fix is provided in a PR.
Am I doing something incorrectly? I greped for ‘panamax.internal’ under ~/.cargo and panamax/mirror, no results. Why cargo still searches for this name?
At this point, the dependency versions are getting pretty aged, and there are some transitive dependencies that have even been yanked.
Update to the latest versions of everything, and change the code where applicable.
Also where possible, use rustls instead of any native openssl library.
dear keith
thanks a lot for the quick fix! i did a full re-sync and most things work as expected again!
what does not work is a fresh rust installation. i can download rust-init
but when i run it http://pamanax.internal/dist/2021-05-10/rust-src-1.52.1.tar.xz
is not found. the directory http://pamanax.internal/dist/2021-05-10/
does exist and is populated - but no rust-src-*
files are in there.
Originally posted by @panzerballett in #24 (comment)
my sync stopped working a few days ago with the message:
Merging crates.io-index repository failed: GitError(Error { code: -3, klass: 22, message: "no merge base found" })
You will need to sync again to finish this download.
the error presists; re-running panamax
does not help.
could i just cd
to crates.io-index/.git
and do a git fetch --all
(or something more advanced like git fetch origin "+refs/heads/*:refs/heads/*" "+refs/tags/*:refs/tags/*" "+branches/*:branches/*" --prune --tags --update-head-ok
)?
i just restarted a fresh sync (using the mirror.toml
file i got from panamax init
with minor changes [contact=
, base_url=
]) and only the crates.io-index
directory was updated. the crates
directory remained empty.
$ panamax --version
panamax 0.2.2
hmmm. weird. while i was re-checking that what i wrote here was accurate, i re-ran panamax sync .
and now 4 crates were mirrored: libnotcurses-sys, parametrizer, schemafy_core, schemafy_lib
.
is this all just due to maintenance on the original mirror?
this is probably an unrelated issue but i also noticed that rust-src
is not mirrored. this seems to be needed for rustup-init
to work.
thanks & best regards!
% cargo install cargo-cache
Updating `panamax` index
Installing cargo-cache v0.8.3
error: failed to compile `cargo-cache v0.8.3`, intermediate artifacts can be found at `/tmp/cargo-installGZoqCc`
Caused by:
failed to download from `http://crates.dw/3/l/log/0.4.17/log-0.4.17.crate`
Caused by:
failed to get 200 response from `http://crates.dw/3/l/log/0.4.17/log-0.4.17.crate`, got 404
I've checked index and there is an index file under /3/l/log
, but crates folder contains log crate under path 3/log
which is not correct
I've tried with dockerized version of latest commit on master
Serving panamax within a container, and attempting to fetch packages via cargo fails with response 500. I debugged and identified the container does not have git installed which is invoked in the Rust code. Its possible the Debian base image had git installed in a previous version, but has since been removed? Regardless, manually adding git resolves the issue.
Dockerfile patch:
---
RUN apt install -y libssl1.1 ca-certificates
---
+++
RUN apt install -y libssl1.1 ca-certificates git
+++
Use actix, tide, warp or another HTTP server library to serve a full Panamax mirror. This should remove our reliance on an nginx config and thus make setup much simpler.
The most complex part of this will be serving a git repository from this server. Options will need to be investigated for this, if not just straight-up implementing this in-tree.
As stated fails to install panamax.
System: Manjaro 5.13
KDE: 5.24
Hardware: AMD Ryzen 7 2700
Error message:
cargo install panamax
...
Compiling panamax v1.0.4
error[E0599]: no method named `progress_chars` found for enum `Result` in the current scope
--> /home/verontrix/.cargo/registry/src/github.com-1ecc6299db9ec823/panamax-1.0.4/src/crates.rs:127:18
|
127 | .progress_chars(" ")
| ^^^^^^^^^^^^^^ method not found in `Result<ProgressStyle, indicatif::style::TemplateError>`
error[E0308]: mismatched types
--> /home/verontrix/.cargo/registry/src/github.com-1ecc6299db9ec823/panamax-1.0.4/src/crates.rs:131:27
|
131 | pb.enable_steady_tick(10);
| ^^ expected struct `Duration`, found integer
error[E0599]: no method named `progress_chars` found for enum `Result` in the current scope
--> /home/verontrix/.cargo/registry/src/github.com-1ecc6299db9ec823/panamax-1.0.4/src/crates.rs:190:18
|
190 | .progress_chars("█▉▊▋▌▍▎▏ ")
| ^^^^^^^^^^^^^^ method not found in `Result<ProgressStyle, indicatif::style::TemplateError>`
error[E0308]: mismatched types
--> /home/verontrix/.cargo/registry/src/github.com-1ecc6299db9ec823/panamax-1.0.4/src/crates.rs:194:27
|
194 | pb.enable_steady_tick(10);
| ^^ expected struct `Duration`, found integer
error[E0599]: no method named `set_draw_rate` found for struct `ProgressBar` in the current scope
--> /home/verontrix/.cargo/registry/src/github.com-1ecc6299db9ec823/panamax-1.0.4/src/crates.rs:195:8
|
195 | pb.set_draw_rate(10);
| ^^^^^^^^^^^^^ help: there is an associated function with a similar name: `set_draw_target`
error[E0308]: mismatched types
--> /home/verontrix/.cargo/registry/src/github.com-1ecc6299db9ec823/panamax-1.0.4/src/rustup.rs:343:27
|
343 | pb.enable_steady_tick(10);
| ^^ expected struct `Duration`, found integer
error[E0308]: mismatched types
--> /home/verontrix/.cargo/registry/src/github.com-1ecc6299db9ec823/panamax-1.0.4/src/rustup.rs:663:27
|
663 | pb.enable_steady_tick(10);
| ^^ expected struct `Duration`, found integer
error[E0599]: no method named `progress_chars` found for enum `Result` in the current scope
--> /home/verontrix/.cargo/registry/src/github.com-1ecc6299db9ec823/panamax-1.0.4/src/crates_index.rs:45:18
|
45 | .progress_chars(" ")
| ^^^^^^^^^^^^^^ method not found in `Result<ProgressStyle, indicatif::style::TemplateError>`
error[E0308]: mismatched types
--> /home/verontrix/.cargo/registry/src/github.com-1ecc6299db9ec823/panamax-1.0.4/src/crates_index.rs:51:27
|
51 | pb.enable_steady_tick(10);
| ^^ expected struct `Duration`, found integer
error[E0599]: no method named `progress_chars` found for enum `Result` in the current scope
--> /home/verontrix/.cargo/registry/src/github.com-1ecc6299db9ec823/panamax-1.0.4/src/rustup.rs:264:18
|
264 | .progress_chars("█▉▊▋▌▍▎▏ ")
| ^^^^^^^^^^^^^^ method not found in `Result<ProgressStyle, indicatif::style::TemplateError>`
Some errors have detailed explanations: E0308, E0599.
For more information about an error, try `rustc --explain E0308`.
error: failed to compile `panamax v1.0.4`, intermediate artifacts can be found at `/tmp/cargo-installOtGonN`
Caused by:
could not compile `panamax` due to 10 previous errors
Cloning panamax successfully builds. However, trying to install the build using cargo install --path
fails . . Same error message.
Right now, Panamax only prints progress bars out to stdout. While this looks nice, unfortunately it's not great for automated systems.
There should be some level of logging that can be thrown into a file or into a syslog, with different levels of erroring. I'm thinking of using the tracing
crate for this.
Despite not specifying platforms_unix
or platforms_windows
, which according to the documentation it should download all the rustup versions.
rustup for the architecture mipsel-unknown-linux-musl
is failing to download any of the rustup distribution files.
Hello,
As far as I understand how to use this repo, it is possible to sync panamax 1 time for the whole crates.io registry :
# This takes a long time but gathers everything at a specific date
panamax sync /mirror
It is also possible to sync only a vendor directory :
# Fast as a few crates are downloaded
panamax sync /mirror /my/path/to/vendor
I'm currently on the second case, I now want to push another vendor directory to panamax. So I do :
# Fast as a few crates are downloaded
panamax sync /mirror /other/path/to/vendor
I want to add multiple vendor directories because I have projects using different versions for a crate and the full sync is too heavy for my use case.
But panamax returns the following result and does not update anything :
Syncing Rustup repositories...
[1/5] Syncing rustup-init files... █████2/2 [00:00:00 / 00:00:00]
[2/5] Syncing latest stable... █████ 25/25 [00:00:02 / 00:00:00]
[3/5] Skipping syncing beta.
[4/5] Skipping syncing nightly.
[5/5] Cleaning old files... █████ 0/0 [00:00:00 / 00:00:00]
Syncing Rustup repositories complete!
Syncing Crates repositories...
[1/3] Fetching crates.io-index... █████ [00:00:00]
[2/3] Syncing crates files... █████2/2 [00:00:01 / 00:00:00]
[3/3] Syncing config...
Syncing Crates repositories complete!
Can I do what I intend to do (add another vendor dir) with Panamax ? If not, why ? What's the alternative ?
Thank you,
Hi there 👋 !
First of all, well done for your work on panamax
!
So, the issue here has already been highlighted in others issues (same author):
crates.io
will reference more and more crates.I opened a pull request in January to add a feature to panamax
allowing users to check for missing crates in the synced mirror PR#57.
The check is not fetching the crates.io-index
, but just looking at the the local index state.
I closed it as I had no feedback from you. But I decided to still open an issue about it.
I couldn't find a way to trigger the problem using panamax
, but I decided to manually delete some crates from my local mirror.
Here are the steps I did (from my panamax
's fork):
/path/to/my/mirror/crates/fa/st/fastrand/
and /path/to/my/mirror/crates/ri/ng/ring/0.16.20/
cargo run -- verify /path/to/my/mirror
. The manually deleted crate are spotted by the check.cargo run -- sync /path/to/my/mirror
. Everything went find according to the execution's output.cargo run -- verify /path/to/my/mirror
. The manually deleted crate are still spotted by the check.ls /path/to/my//mirror/crates/fa/st/fastrand /path/to/my//mirror/crates/ri/ng/ring/0.16.20
just to be sure. And get No such file or directory
error for both directory.There are two things highlighted by this issue:
panamax
is not able to download them again because of the fact that the reference to download crates is a diff between the local index (master
branch) and the distant one (origin/master
branch). This makes sense that the missing crates are not downloaded.By exchanging about it around me, I've been put on the track that it could come from the fact that our local commit have been squashed on origin which result of local commit id to be absent from the tree. But I've looked into panamax
's code and you seem to handle this case. So, I can't figure a reason of this behavior.
Anyway, I hope this issue will lead to some discussion, and if you need help, I'd be glad to re-open a PR.
Cheers 😄
Currently, Panamax will download every architecture for a given release, which is a fairly heavy download. There should be an optional config parameter that allows specifying a list of triples to download.
panamax
seems to be missing some updates for some crates.
i pulled the current version (commit 0b169b0), built it and ran sync
.
when trying to build a project on the internal network, i noticed that some files where missing. the crates.io-index seems to have the correct versions - but the crates are not downloaded.
two examples:
1.0.14
, but listing crates/qu/ot/quote/
shows that the most recent version is 1.0.10
0.2.16
, listing crates/pp/v-/ppv-lite86/
shows that the most recent version is 0.2.15
.any ideas?
thanks in advance!
Hello,
I'm interested in setting up my local mirror such that I can easily enable it by setting one environment variable, or disable it by unsetting it.
I have attempted to set it up as both a registry and a source, but the issue I'm running into is that I need to somehow set source.cargo-io
's 'index' value via an environment variable. The docs say I should be able to set CARGO_SOURCE_<sourcename>_INDEX
, but I am unsure how to represent the source name here. Using a -, such as CARGO_SOURCE_CRATES-IO_INDEX
is invalid under linux, and replacing the dash with a single or double underscore, or omitting it entirely doesn't work.
How can I set the crates-io replacement key via environment variable? This would allow me to set up a source in my config but only to enable it when necessary.
Thanks.
On Ubuntu, there's a package you can install called awscli
that allows you to access many different APIs, including that of S3.
If you run
aws s3 ls --no-sign-request s3://static-rust-lang-org/rustup/dist/
Then you will receive a full list of rustup architectures, i.e.:
PRE aarch64-apple-darwin/
PRE aarch64-linux-android/
PRE aarch64-unknown-linux-gnu/
PRE aarch64-unknown-linux-musl/
PRE arm-linux-androideabi/
PRE arm-unknown-linux-gnueabi/
PRE arm-unknown-linux-gnueabihf/
PRE armv7-linux-androideabi/
PRE armv7-unknown-linux-gnueabihf/
PRE i686-apple-darwin/
PRE i686-linux-android/
PRE i686-pc-windows-gnu/
PRE i686-pc-windows-msvc/
PRE i686-unknown-linux-gnu/
PRE mips-unknown-linux-gnu/
PRE mips64-unknown-linux-gnuabi64/
PRE mips64el-unknown-linux-gnuabi64/
PRE mipsel-unknown-linux-gnu/
PRE powerpc-unknown-linux-gnu/
PRE powerpc64-unknown-linux-gnu/
PRE powerpc64le-unknown-linux-gnu/
PRE riscv64gc-unknown-linux-gnu/
PRE s390x-unknown-linux-gnu/
PRE x86_64-apple-darwin/
PRE x86_64-linux-android/
PRE x86_64-pc-windows-gnu/
PRE x86_64-pc-windows-msvc/
PRE x86_64-unknown-freebsd/
PRE x86_64-unknown-illumos/
PRE x86_64-unknown-linux-gnu/
PRE x86_64-unknown-linux-musl/
PRE x86_64-unknown-netbsd/
If we could perform a similar call when syncing rustup, we could use this list to either collect every single rustup-init, or use it to validate any filters that may be set in mirror.toml
. Ideally, this would replace the hard-coded PLATFORMS
and PLATFORMS_EXE
lists that currently exist in src/rustup.rs
.
The only unresolved question I have with regards to this, is what should be done if Panamax is grabbing from another Rust mirror, as opposed to the official repo. We probably can't assume said mirror would be placed on S3 or something with an S3-compatible API, so I think the best option is to not validate, and accept any architecture if rustup's source
in mirror.toml
is not set to https://static.rust-lang.org
.
If any history file is missing (that is, mirror-stable-history.toml
, mirror-beta-history.toml
, or mirror-nightly-history.toml
), the cleaning step returns this error:
Cleaning old files failed: Io(Os { code: 2, kind: NotFound, message: "No such file or directory" })
This is due to the get_channel_history
function in rustup.rs
not having the proper error handling involved. If a file does not exist, it should be ignored, or treated as though it's empty.
The way I'd solve this is by adding a match to the first call to get_channel_history
in the clean_old_files
function. There is a second call to get_channel_history
that handles the error properly, so it would just be a matter of copying from there.
Hi,
PR #71 added an option to use cargo vendor to download only selected packages.
The Panamax version number is not updated after this feature and the latest docker hub image (version 1.0.6) does not have this feature.
Could you create a new version and build a new docker image with this feature included?
From structopt
documentation:
As clap v3 is now out, and the structopt features are integrated into (almost as-is), structopt is now in maintenance mode: no new feature will be added.
I think it would be better to move to clap
.
I'll try to submit a PR for that.
Panamax doesn't seem to sync the files for tier 3 targets like
mips-unknown-linux-musl
mipsel-unknown-linux-musl
armv7-unknown-linux-musleabihf
It would be great to also grab these targets with a sync.
Add a field to the config that allows specific historical versions to be downloaded and kept.
This should support stable, beta, and nightly versions.
panamax version: 1.0.6
Operating system: Ubuntu 20.04 LTS in WSL2
I start panamax on a new directory, synced around 150k crates out of 588629 crates,
then I interrupt it.
When I run panamax sync
on that directory again,
the memory usage keeps increasing, reaching 30GB+,
and OOM my PC, so Windows OS kills the WSL2
It seems to have a memory leak issue.
I do not think panamax needs so much memory
Contents of mirror.html
# This is a Panamax mirror. It is a self-contained directory made to be easily copied
# to an offline network or machine via rsync, USB, or another method.
# When offline, Panamax also includes a "serve" command that can be used to serve
# rustup and cargo clients from the mirror. This will also give setup instructions
# on the homepage.
[mirror]
# Global mirror settings.
# Number of download retries before giving up.
retries = 5
# Contact information for the user agent.
# This is entirely optional, and is not required for the crates.io CDN.
# You may want to set this if you are mirroring from somewhere else.
# contact = "[email protected]"
[rustup]
# These are the configuration parameters for the rustup half of the mirror.
# This will download the rustup-init files, as well as all components needed
# to run Rust on a machine.
# Perform rustup synchronization. Set this to false if you only want to mirror crates.io.
sync = false
# Whether to mirror XZ archives. These archives are more efficiently compressed
# than the GZ archives, and rustup uses them by default.
download_xz = true
# Whether to mirror GZ archives, for further backwards compatibility with rustup.
download_gz = false
# Number of downloads that can be ran in parallel.
download_threads = 128
# Where to download rustup files from.
source = "https://rsproxy.cn"
# How many historical versions of Rust to keep.
# Setting these to 1 will keep only the latest version.
# Setting these to 2 or higher will keep the latest version, as well as historical versions.
# Setting these to 0 will stop Panamax from downloading the release entirely.
# Removing the line will keep all release versions.
keep_latest_stables = 1
keep_latest_betas = 1
keep_latest_nightlies = 1
# Pinned versions of Rust to download and keep alongside latest stable/beta/nightly
# Version specifiers should be in the rustup toolchain format:
#
# <channel>[-<date>][-<host>]
#
# <channel> = stable|beta|nightly|<major.minor>|<major.minor.patch>
# <date> = YYYY-MM-DD
# <host> = <target-triple>
#
# e.g. valid versions could be "1.42", "1.42.0", and "nightly-2014-12-18"
# Uncomment the following lines to pin extra rust versions:
#pinned_rust_versions = [
# "1.42"
#]
# UNIX platforms to include in the mirror
# Uncomment the following lines to limit which platforms get downloaded.
# This affects both rustup-inits and components.
# platforms_unix = [
# "arm-unknown-linux-gnueabi",
# "x86_64-unknown-linux-gnu",
# "x86_64-unknown-linux-musl",
# ]
# Windows platforms to include in the mirror
# Uncomment the following lines to limit which platforms get downloaded.
# This affects both rustup-inits and components.
# platforms_windows = [
# "x86_64-pc-windows-gnu",
# "x86_64-pc-windows-msvc",
# ]
# Whether to download the rustc-dev component.
# This component isn't always needed, so setting this to false can save lots of space.
download_dev = false
[crates]
# These are the configuration parameters for the crates.io half of the mirror.
# This will download the crates.io-index, as well as the crates themselves.
# Once downloaded, it will then (optionally) rewrite the config.json to point to your mirror.
# Perform crates synchronization. Set this to false if you only want to mirror rustup.
sync = true
# Number of downloads that can be ran in parallel.
download_threads = 64
# Where to download the crates from.
# The default, "https://crates.io/api/v1/crates", will actually instead use the corresponding
# url at https://static.crates.io in order to avoid a redirect and rate limiting
source = "https://rsproxy.cn/api/v1/crates"
# Where to clone the crates.io-index repository from.
source_index = "http://rsproxy.cn/crates.io-index"
# URL where this mirror's crates directory can be accessed from.
# Used for rewriting crates.io-index's config.json.
# Remove this parameter to perform no rewriting.
# If removed, the `panamax rewrite` command can be used later.
base_url = "http://127.0.0.1:3000/crates"
An update for cargo is in the works to make downloading of the index smaller and faster. This is especial important for CI jobs. It would be nice, if Panamax could support this.
Rust blog post: https://blog.rust-lang.org/2022/06/22/sparse-registry-testing.html
Tracking issue: rust-lang/cargo#9069
hello, me again...
i think the following problem started when the crates.io mirror changed its format (but i am not sure about that).
running cargo update
on an internal machine works fine - the crates info seems to be updated. but when i try to run cargo build
(or something similar) this happens:
error: failed to download from `http://panamax.internal/crates/bitflags/bitflags-1.3.2.crate`
Caused by:
failed to get 200 response from `http://panamax.internal/crates/bitflags/bitflags-1.3.2.crate`, got 404
what does exist on my local mirror is:
http://panamax.internal/crates/bi/tf/bitflags/1.3.2/bitflags-1.3.2.crate
on the external machine i built pamamax
manually, upgraded everything to the latest version and ran panamax sync
:
$ git pull origin master
$ cargo build --release
$ ./target/release/panamax sync mirror/
versions:
panamax 1.0.0
cargo 1.54
any advice?
Add functionality that lets Panamax run through a web proxy. Hopefully this shouldn't be too difficult, as reqwest is heavily used and has built-in proxy functionality.
The proxy should be settable via the HTTP_PROXY environment variable, or an optional http_proxy option in mirror.toml, under the [mirror]
section.
Having a message
Downloading crates.io-index repository failed: gitError(Error { code: -1, klass: 2, message: "failed to make directory './.': })
os: Windows10
path of mirror: D:\rust-mirror
Windows is very slow when handling many small files, and the entire crates mirror roughly totals to almost 1 million small files.
As you can imagine, that makes it extremely painful to transfer the mirror via normal filesystem operations such as copy and paste.
However, one way to circumvent this cost is to wrap the small files in a container file, such as a zip archive.
You don't even have to compress it - just having a single file to copy will make the file transfer much more efficient.
It'd be a really nice quality-of-life feature if panamax offered this in the future.
The directory tree of the crate mirror uses package names, which can mix lower and uppercase characters.
In the Cargo Book, it says that file names should be in lowercase, with a lowercase prefix (reference). So in the Git crates.io-index
also, filenames are lowercase.
Case-insensitive file systems (macOS, Windows...) break Panamax and prevent it from delivering crates.
I will certainly modify to create a lowercase directory tree, but the question of transition arises: should I move files if both exist (example: crates/An/
and crates/an/
)? If a script is acceptable, it would be much easier.
My initial sync took hours and then died, afterwards I needed to sync all files again.
Maybe it would be possible to restart from the already downloaded files so you don't need to redownload all crates again?
rustup-init files in windows platforms need .exe
extension in the download url
The sync_rustup_init
always call to sync_one_init
with is_exe=false
as parameter
I get this error when fetching the crates.io index. I didn't change the default configuration at all except disabling Rustup sync (so I could get to this error faster).
Rustup self-updates don't work on panamax, because the files needed for this are not mirrored.
In one of the new rustup versions, nightly releases appear to be skipped if a built rustfmt
doesn't exist within it. Because of this, if a mirror only keeps one/two/more nightly copies and none of them contain rustfmt, rustup will loop (and fail) through every nightly version back to 2014.
Add a parameter in the config that only purges old nightly copies once a version containing rustfmt is downloaded.
As a mirror lives, it rotates through different stable, beta, and nightly versions. It will also remove old versions, if that setting is enabled. Unfortunately, when it removes those old versions, it doesn't remove the directory itself.
Make the clean_old_files code delete the date directories in dist/
if empty.
Thiserror appears to be the more de-facto error handling library, so Panamax should be using that instead.
Luckily, as quick-error and thiserror work fairly similarly, this should be an easy change.
Hello-- I have a Panamax mirror set up and I'm able to use the rustup portion without problems. For Cargo, however, it's giving me trouble when accessing the /git/crates.io-index endpoint.
I've tried running this both with and without the net.git-fetch-with-cli config set to true, with no difference.
Panamax is running on port 80, with a fully synced mirror.
Here's the relevant log entry:
04:25:11 Step 10/10: Run x.py (PowerShell)
04:25:11 PowerShell Executable: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
04:25:11 Working directory: D:\Home\teamcity\work\Rust
04:25:11 Command: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
04:25:11 PowerShell arguments: -NoProfile, -NonInteractive, -ExecutionPolicy, ByPass, -File, D:\Home\teamcity\temp\buildTmp\powershell4071428807607952288.ps1
04:25:12 downloading http://[panamax-mirror]/dist/2022-09-22/rust-std-1.64.0-x86_64-pc-windows-msvc.tar.xz
04:25:12
04:25:12 #################### 28.3%
04:25:12 ############################################## 64.9%
04:25:12
04:25:12 ######################################################################## 100.0%
04:25:18 extracting D:\Home\teamcity\work\Rust\rust-repo\build\cache\2022-09-22\rust-std-1.64.0-x86_64-pc-windows-msvc.tar.xz
04:25:18 downloading http://[panamax-mirror]/dist/2022-09-22/rustc-1.64.0-x86_64-pc-windows-msvc.tar.xz
04:25:18
04:25:18 ####### 10.0%
04:25:18
04:25:18 ################## 25.8%
04:25:18 ############################# 41.6%
04:25:18 ######################################### 57.3%
04:25:18
04:25:18 ################################################### 72.1%
04:25:18 ############################################################## 87.1%
04:25:18 ######################################################################## 100.0%
04:25:31 extracting D:\Home\teamcity\work\Rust\rust-repo\build\cache\2022-09-22\rustc-1.64.0-x86_64-pc-windows-msvc.tar.xz
04:25:31 downloading http://[panamax-mirror]/dist/2022-09-22/cargo-1.64.0-x86_64-pc-windows-msvc.tar.xz
04:25:31
04:25:31 ######################################################################## 100.0%
04:25:32 extracting D:\Home\teamcity\work\Rust\rust-repo\build\cache\2022-09-22\cargo-1.64.0-x86_64-pc-windows-msvc.tar.xz
04:25:32 Building rustbuild
04:25:33 Updating panamax
index
04:25:33 error: Unable to update registry crates-io
04:25:33
04:25:33 Caused by:
04:25:33 failed to update replaced source registry crates-io
04:25:33
04:25:33 Caused by:
04:25:33 failed to fetch http://[panamax-mirror]/git/crates.io-index 04:25:33 04:25:33 Caused by: 04:25:33 process didn't exit successfully:
git fetch --force --update-head-ok http://[panamax-mirror]/git/crates.io-index +HEAD:refs/remotes/origin/HEAD` (exit code: 128)
04:25:33 --- stderr
04:25:33 remote: Unhandled rejection: Io(Os { code: 2, kind: NotFound, message: "No such file or directory" })
04:25:33 fatal: unable to access 'http://[panamax-mirror]/git/crates.io-index/': The requested URL returned error: 500
04:25:33 failed to run: D:\Home\teamcity\work\Rust\rust-repo\build\x86_64-pc-windows-msvc\stage0\bin\cargo.exe build --manifest-path D:\Home\teamcity\work\Rust\rust-repo\src/bootstrap/Cargo.toml
04:25:33 Build completed unsuccessfully in 0:00:20
04:25:33 Process exited with code 0
There are .asc files that exist alongside many of the rustup files. For this issue, these files don't need to be verified, just downloaded.
To add to this issue - it doesn't appear that panamax pulls in `.asc` files for rustup channels.
_Originally posted by @DrChat in https://github.com/panamax-rs/panamax/issues/7#issuecomment-915617132_
Lines 110 to 118 in 0b169b0
There are two scenarios here:
After carefully reading the code, I realised scenario 2 (do_full_scan == true
) should only happen on the initial clone, because the rewrite of the config.json adds a new commit every time which makes it always mismatch when compared to refs/remotes/origin/master
. Except that, if someone comments out base_url (so no rewrite happens, so no commit is made) and happens to update when there haven't been any changes to the index, I think they will initiate a full sync.
I feel like a good solution to this is to turn the do_full_scan
logic into an explicit flag that gets passed in. If a "first sync" is detected then this flag can be set and passed in.
Given that the download
function does check if a file already exists on disk, this would also permit passing an argument to sync
to 'recheck' the whole downloaded repository.
i tried to start
$ panamax serve /path/to/mirror
on a machine behind a VPN (which only supports IPv4) where i have completely disabled the IPv6 stack (as there is no use for that). panamax
wants to bind to [::]:8080
and therefore crashes on this machine.
is there a flag or something that lets me just bind to 127.0.0.1:8080
and ignore IPv6?
thanks in advance!
This one is a really weird one, and may not necessarily be Panamax's fault, but I can consistently reproduce this hang when using Panamax.
Panamax is able to synchronize rustup-init
files successfully and attempts to move on to the next step.
When it shuts down the progress bar worker thread to move on however, the main thread hangs when joining the progress bar thread.
In turn, the progress bar thread is hung in the following callstack:
> ntdll.dll!NtWaitForSingleObject�() Unknown
ntdll.dll!LdrpDrainWorkQueue() Unknown
ntdll.dll!LdrShutdownThread() Unknown
ntdll.dll!RtlExitUserThread�() Unknown
kernel32.dll!BaseThreadInitThunk�() Unknown
ntdll.dll!RtlUserThreadStart�() Unknown
As in, the thread has successfully received the message to exit and has attempted to exit, but hung when synchronizing with the Windows loader.
Why this is hanging is a mystery. It appears to be hung on either one of ntdll!LdrpWorkCompleteEvent
or ntdll!LdrpLoadCompleteEvent
.
Here is a relevant thread on Twitter: https://twitter.com/BruceDawson0xB/status/1263602284040089600
This may be a problem with one of Panamax's dependencies, or a Windows OS bug, but the only thing I've seen trigger this bug is the specific combination present in this code.
Steps to reproduce:
platforms_windows = [
"x86_64-pc-windows-msvc",
"x86_64-pc-windows-gnu",
]
platforms_windows = [
"x86_64-pc-windows-msvc",
]
Result: *-gnu
files are removed from my-mirror/dist
but not from my-mirror/rustup
.
i have a mirror and it worked well. but recently cargo can no longer download all crates. some work - others to not.
an example:
$ cargo build
Updating `panamax` index
Downloaded io-lifetimes v1.0.6 (registry `panamax`)
error: failed to download from `http://panamax.internal/crates/3/l/log/0.4.17/log-0.4.17.crate`
Caused by:
failed to get successful HTTP response from `http://rust.cry/crates/3/l/log/0.4.17/log-0.4.17.crate`, got 404
on the server there is a file /mnt/rust/crates/3/log/0.4.17
but not /mnt/rust/crates/3/l/log/0.4.17
(note the l
in the middle).
there are creates which are twice in the mirror (some only up to an older version). an example:
older versions:
$ ls 3/syn/
0.10.0 ... 1.0.107
the most recent version:
$ ls 3/s/syn/
0.10.0 ... 1.0.109
but this naming is not consistent:
$ ls 3/x
xcp xot
but:
$ ls -d 3/x??
3/x11 3/x1b 3/xal ... 3/xvi 3/xxv 3/xyn
for the moment i am not able to build a project containing the syn
or the log
crate. i can neither get it to work using nginx rewrite rules nor with panamax serve
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.