Code Monkey home page Code Monkey logo

ktra's People

Contributors

fmeow avatar gagbo avatar gl-yziquel avatar jbeaurivage avatar moriturus avatar roastveg avatar yodaldevoid avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ktra's Issues

ktra 0.7.0 - Failed to build with "--no-default-features --features=secure-auth,db-mongo" and "--no-default-features --features=secure-auth,db-redis"

Thank you for an excellent and very useful utility!

I have been using ktra with the default db-sled but once every few months I get errors associated with data corruption.
So, the logical step is to use another db engine with ktra that supports data integrity.

I tried to install ktra with "--no-default-features --features=secure-auth,db-mongo" and "--no-default-features --features=secure-auth,db-redis" and in both cases I get the following compilation error:

 Compiling ktra v0.7.0
error[E0382]: borrow of moved value: `db_manager`
  --> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/ktra-0.7.0/src/main.rs:86:39
   |
77 |     db_manager: Arc<RwLock<impl DbManager>>,
   |     ---------- move occurs because `db_manager` has type `Arc<tokio::sync::RwLock<impl DbManager>>`, which does not implement the `Copy` trait
...
84 |         .or(put::apis(db_manager, index_manager, dl_dir_path));
   |                       ---------- value moved here
85 |     #[cfg(not(feature = "openid"))]
86 |     let routes = routes.or(post::apis(db_manager.clone()));
   |                                       ^^^^^^^^^^^^^^^^^^ value borrowed here after move

For more information about this error, try `rustc --explain E0382`.
error: could not compile `ktra` due to previous error

I traced the source code and It looks like an easy fix. I will submit a pull request with it.

Thank you for all your hard work!

What are the recovery expectations of ktra

First: Thanks for working on this. I've been experimenting with it the last couple of days and it's great.

I don't know if this is an issue or simply expected behavior, so I thought I'd ask.

If I start ktra and then publish a few crates, my git repo is updated correctly and the various queries about which crates are available etc all work as expected.

Then: I delete my database (I'm using sled), index and my crates directory. (i.e. all contents created and managed by ktra).
Now, when I start ktra, it happily updates my index and seems to "do the right thing" in terms of populating that with the name of my crates from my git repo. I create a new user and get a token and then when I use that to ask ktra what crates are available I get an empty list.

I know deleting my database, etc... is a silly thing to do, but I sort of feel like it should be possible to recreate (most of) the contents of the ktra data based on the information that is stored in the git repo. So, that's my question really. Should it be possible?

Changing branch

First thank you for this project, this is awesome!

I had one issue though that I managed to fixed but I am opening the issue in case someone else has it too.

I am not using github so default branch name is not main but master (in my case it was bitbucket, fyi).

I found that you could change the branch name in ktra.toml file:

[index_config]
remote_url = "..."
branch = "master"

... but if you do it AFTER the first initialization, then you get stuck with git error:

src refspec 'refs/heads/master' does not match any existing object

The solution I found was to delete all existing artifacts (db, index and crates directories) before starting ktra again with the updated config.

Now it works like a charm!

cargo build fails

Just tried to compile ktra with cargo build on tag 0.7.0. Got the following log.

This is an ubuntu system. Linux 6.2.0-1010-nvidia #10-Ubuntu SMP PREEMPT_DYNAMIC x86_64 GNU/Linux

cargo tree tells me that my ssl dependency pulled by cargo is openssl-sys v0.9.90. git2 pulls version v0.9.90 and reqwest pulls version v0.10.55. This gets reconciled to version v0.9.90.

  = note: /usr/bin/ld: /home/yziquel/home/cellar/ktra/target/debug/deps/liblibgit2_sys-b724feea96647aca.rlib(openssl.o): in function `openssl_init':
          /home/yziquel/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libgit2-sys-0.12.26+1.3.0/libgit2/src/streams/openssl.c:145: undefined reference to `SSL_CTX_ctrl'
          /usr/bin/ld: /home/yziquel/home/cellar/ktra/target/debug/deps/liblibgit2_sys-b724feea96647aca.rlib(openssl.o): in function `verify_server_cert':
          /home/yziquel/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libgit2-sys-0.12.26+1.3.0/libgit2/src/streams/openssl.c:401: undefined reference to `SSL_get1_peer_certificate'
          /usr/bin/ld: /home/yziquel/home/cellar/ktra/target/debug/deps/liblibgit2_sys-b724feea96647aca.rlib(openssl.o): in function `openssl_connect':
          /home/yziquel/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libgit2-sys-0.12.26+1.3.0/libgit2/src/streams/openssl.c:531: undefined reference to `SSL_ctrl'
          /usr/bin/ld: /home/yziquel/home/cellar/ktra/target/debug/deps/liblibgit2_sys-b724feea96647aca.rlib(openssl.o): in function `openssl_certificate':
          /home/yziquel/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libgit2-sys-0.12.26+1.3.0/libgit2/src/streams/openssl.c:545: undefined reference to `SSL_get1_peer_certificate'
          /usr/bin/ld: /home/yziquel/home/cellar/ktra/target/debug/deps/liblibssh2_sys-2a9b7891975d8646.rlib(openssl.o): in function `_libssh2_ed25519_new_private_frommemory':
          /home/yziquel/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libssh2-sys-0.2.23/libssh2/src/openssl.c:1835: undefined reference to `EVP_PKEY_get_id'
          /usr/bin/ld: /home/yziquel/home/cellar/ktra/target/debug/deps/liblibssh2_sys-2a9b7891975d8646.rlib(openssl.o): in function `_libssh2_pub_priv_keyfile':
          /home/yziquel/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libssh2-sys-0.2.23/libssh2/src/openssl.c:3001: undefined reference to `EVP_PKEY_get_id'
          /usr/bin/ld: /home/yziquel/home/cellar/ktra/target/debug/deps/liblibssh2_sys-2a9b7891975d8646.rlib(openssl.o): in function `_libssh2_pub_priv_keyfilememory':
          /home/yziquel/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libssh2-sys-0.2.23/libssh2/src/openssl.c:3211: undefined reference to `EVP_PKEY_get_id'
          /usr/bin/ld: /home/yziquel/home/cellar/ktra/target/debug/deps/liblibssh2_sys-2a9b7891975d8646.rlib(mac.o): in function `mac_method_hmac_ripemd160_hash':
          /home/yziquel/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libssh2-sys-0.2.23/libssh2/src/mac.c:354: undefined reference to `EVP_ripemd160'
          /usr/bin/ld: /home/yziquel/home/cellar/ktra/target/debug/deps/liblibssh2_sys-2a9b7891975d8646.rlib(crypt.o):(.data.rel.ro.libssh2_crypt_method_blowfish_cbc+0x40): undefined reference to `EVP_bf_cbc'
          /usr/bin/ld: /home/yziquel/home/cellar/ktra/target/debug/deps/liblibssh2_sys-2a9b7891975d8646.rlib(crypt.o):(.data.rel.ro.libssh2_crypt_method_cast128_cbc+0x40): undefined reference to `EVP_cast5_cbc'
          /usr/bin/ld: /home/yziquel/home/cellar/ktra/target/debug/deps/libopenssl-cb21597defe4182a.rlib(openssl-cb21597defe4182a.openssl.f5acc07b9d67cffb-cgu.07.rcgu.o): in function `openssl::error::Error::get':
          /home/yziquel/.cargo/registry/src/index.crates.io-6f17d22bba15001f/openssl-0.10.55/src/error.rs:122: undefined reference to `ERR_get_error_all'
          /usr/bin/ld: /home/yziquel/home/cellar/ktra/target/debug/deps/libopenssl_sys-0d6d27204dddf829.rlib(openssl_sys-0d6d27204dddf829.openssl_sys.547b75131ddb7575-cgu.0.rcgu.o): in function `openssl_sys::openssl::ssl::SSL_CTX_set_mode':
          /home/yziquel/.cargo/registry/src/index.crates.io-6f17d22bba15001f/openssl-sys-0.9.90/src/./ssl.rs:246: undefined reference to `SSL_CTX_ctrl'
          /usr/bin/ld: /home/yziquel/home/cellar/ktra/target/debug/deps/libopenssl_sys-0d6d27204dddf829.rlib(openssl_sys-0d6d27204dddf829.openssl_sys.547b75131ddb7575-cgu.0.rcgu.o): in function `openssl_sys::openssl::ssl::SSL_CTX_add_extra_chain_cert':
          /home/yziquel/.cargo/registry/src/index.crates.io-6f17d22bba15001f/openssl-sys-0.9.90/src/./ssl.rs:380: undefined reference to `SSL_CTX_ctrl'
          /usr/bin/ld: /home/yziquel/home/cellar/ktra/target/debug/deps/libopenssl_sys-0d6d27204dddf829.rlib(openssl_sys-0d6d27204dddf829.openssl_sys.547b75131ddb7575-cgu.0.rcgu.o): in function `openssl_sys::openssl::tls1::SSL_set_tlsext_host_name':
          /home/yziquel/.cargo/registry/src/index.crates.io-6f17d22bba15001f/openssl-sys-0.9.90/src/./tls1.rs:24: undefined reference to `SSL_ctrl'
          /usr/bin/ld: /home/yziquel/home/cellar/ktra/target/debug/deps/libopenssl_sys-0d6d27204dddf829.rlib(openssl_sys-0d6d27204dddf829.openssl_sys.547b75131ddb7575-cgu.0.rcgu.o): in function `openssl_sys::openssl::ssl::SSL_CTX_set_min_proto_version':
          /home/yziquel/.cargo/registry/src/index.crates.io-6f17d22bba15001f/openssl-sys-0.9.90/src/./ssl.rs:455: undefined reference to `SSL_CTX_ctrl'
          /usr/bin/ld: /home/yziquel/home/cellar/ktra/target/debug/deps/libopenssl_sys-0d6d27204dddf829.rlib(openssl_sys-0d6d27204dddf829.openssl_sys.547b75131ddb7575-cgu.0.rcgu.o): in function `openssl_sys::openssl::ssl::SSL_CTX_set_max_proto_version':
          /home/yziquel/.cargo/registry/src/index.crates.io-6f17d22bba15001f/openssl-sys-0.9.90/src/./ssl.rs:464: undefined reference to `SSL_CTX_ctrl'
          collect2: error: ld returned 1 exit status
          
  = note: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)

warning: `ktra` (bin "ktra") generated 1 warning
error: could not compile `ktra` (bin "ktra") due to previous error; 1 warning emitted

Error 404 when trying to publish without a token (should be 400)

I have an empty project through cargo init --lib
Almost everything is following the tutorial

➜  test_rust git:(main) ✗ cargo publish --allow-dirty --verbose --registry ktra
    Updating `ktra` index
     Running `git fetch --force --update-head-ok 'ssh://[email protected]/xxxxxx/ktra-index' '+HEAD:refs/remotes/origin/HEAD'`
warning: manifest has no description, license, license-file, documentation, homepage or repository.
See https://doc.rust-lang.org/cargo/reference/manifest.html#package-metadata for more info.
   Packaging test_rust v0.1.0 (/Users/jack/Dev/new_trading/libraries/test_rust)
   Archiving Cargo.toml
   Archiving Cargo.toml.orig
   Archiving src/lib.rs
   Verifying test_rust v0.1.0 (/Users/jack/Dev/new_trading/libraries/test_rust)
   Compiling test_rust v0.1.0 (/Users/jack/Dev/new_trading/libraries/test_rust/target/package/test_rust-0.1.0)
     Running `rustc --crate-name test_rust --edition=2021 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C split-debuginfo=unpacked -C debuginfo=2 -C metadata=51782412979cedc2 -C extra-filename=-51782412979cedc2 --out-dir /Users/jack/Dev/new_trading/libraries/test_rust/target/package/test_rust-0.1.0/target/debug/deps -C incremental=/Users/jack/Dev/new_trading/libraries/test_rust/target/package/test_rust-0.1.0/target/debug/incremental -L dependency=/Users/jack/Dev/new_trading/libraries/test_rust/target/package/test_rust-0.1.0/target/debug/deps`
    Finished dev [unoptimized + debuginfo] target(s) in 2.74s
   Uploading test_rust v0.1.0 (/Users/jack/Dev/new_trading/libraries/test_rust)
error: failed to publish to registry at http://localhost:8000

Caused by:
  the remote server responded with an error (status 404 Not Found): resource or api is not defined

[registries]
ktra = { index = "ssh://[email protected]/xxxxxxxx/ktra-index" }

I'm not sure why is this happening.

git error: corrupted loose reference file: FETCH_HEAD when trying to mirror crates.io

Dec 25 19:12:08.256 INFO run_server: ktra: crates directory: "crates"
Dec 25 19:12:08.257 INFO run_server:new: ktra::db_manager::sled_db_manager: create and/or open database: "db"
Dec 25 19:12:08.262 DEBUG sled::pagecache::iterator: ordering before clearing tears: {0: 0}, max_header_stable_lsn: 0
Dec 25 19:12:08.262 DEBUG sled::pagecache::iterator: in clean_tail_tears, found missing item in tail: None and we'll scan segments {0: 0} above lowest lsn 0
Dec 25 19:12:08.263 DEBUG sled::pagecache::iterator: filtering out segments after detected tear at (lsn, lid) 132
Dec 25 19:12:08.263 DEBUG sled::pagecache::iterator: hit max_lsn 132 in iterator, stopping
Dec 25 19:12:08.263 DEBUG sled::pagecache::snapshot: zeroing the end of the recovered segment at lsn 0 between lids 133 and 524287
Dec 25 19:12:08.282 DEBUG sled::pagecache::blob_io: gc_blobs removing any blob with an lsn above 133
Dec 25 19:12:08.282 DEBUG sled::pagecache::segment: SA starting with tip 524288 stable -1 free {}
Dec 25 19:12:08.282 DEBUG sled::pagecache::iobuf: starting log at recovered active offset 133, recovered lsn 133
Dec 25 19:12:08.282 DEBUG sled::pagecache::iobuf: starting IoBufs with next_lsn: 133 next_lid: 133
Dec 25 19:12:08.282 DEBUG sled::pagecache: load_snapshot loading pages from 0..4
Dec 25 19:12:08.285 INFO run_server:new:clone_or_open_repository: ktra::index_manager: open index repository: "index"
Dec 25 19:12:08.302 INFO run_server:pull:fetch: ktra::index_manager: fetches latest commit from origin/main
Dec 25 19:12:11.630 DEBUG run_server: sled::pagecache::logger: IoBufs dropped
Error: git error: corrupted loose reference file: FETCH_HEAD; class=Reference (4)

Downloading ktra's crates.io mirror may fail when tried from some Cargo versions

Mar 04 19:26:20.566 ERROR request{method=GET path=/ktra/api/v1/mirror/anyhow/1.0.41/download version=HTTP/1.1 remote.addr=127.0.0.1:60092}: warp::filters::trace: unable to process request (internal error) status=500 error=Rejection([MethodNotAllowed, MethodNotAllowed, MethodNotAllowed, MethodNotAllowed, MethodNotAllowed, MethodNotAllowed, MethodNotAllowed, MethodNotAllowed, MissingHeader { name: "Authorization" }, HttpRequest(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("static.crates.io")), port: None, path: "/crates/anyhow/anyhow-1.0.41.crate", query: None, fragment: None }, source: hyper::Error(Io, Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }) })])

I'm not sure, but just retrying (or retrying with another Cargo version?) may resolve.

[Feat Req] crates.io mirroring

When setting up ktra as the default registry, if a crate is not found, get it from crates.io and cache it in ktra.

My company is looking for a solution to allow offline build from our CI servers and atm, only Meuse is capable of that but written in Clojure, which limit our capability to contribute back as none of us knows Clojure 😅

How to use SSH for Ktra config file

I am using GitLab self hosting (Got Gitlab set up on my Synology - been using it for years now)

Now I want to implement Ktra for my Rust projects.

I've got to this stage:
https://book.ktra.dev/quick_start/create_ktra_configuration_file.html
I tried:

https_username
https_password

However, because it is self-hosting I'm not using HTTPS, it just HTTP so this doesn't work.

So I tried to use SSH instead, so here is the file

[index_config]
remote_url = "ssh://git@<self-hosting-url>/crate/registry.git"
ssh_privkey_path = "~/.ssh/id_rsa.pub"
branch = "master"

When I do this:

Ktra -c ./cargo/ktra.toml

I get a message

Error: git error: username not defined

So I did the following:

git config --global user.name

I can see my username, but can't work out the problem. Am I missing any additional ssh fields in the .ktra.toml film?

No push to remote repo

Hi
Thanks for this great repo! I am running into some - what I guess- basic issues.
I managed to run locally ktra, add my user and initialize with a gitlab repo as described in the book.
As well the configuration step and pushing a project into the index works - almost.

cargo publish --allow-dirty --token --TOKEN--
Updating `http://gitlab.ngs-ai.lab/eschmid/crate_ngsai/` index
Compiling abc v0.2.3 (/home/eschmid/Projects/AgnosticBamCounter/target/package/abc-0.2.3)
    Finished dev [unoptimized + debuginfo] target(s) in 35.39s
   Uploading abc v0.2.3 (/home/eschmid/Projects/AgnosticBamCounter)

So that looks good to me and the ktra index/repo reflects as well this new addition.
What is though unexpected, is that the gitlab did not receive any commit, pushes - nada.

Any idea what prevents the proper push - but no error message at all ?

Update

Sorry just to add, if I try then to add the above as a dependency for an empty project, I get

cargo build
    Updating `http://gitlab.ngs-ai.lab/eschmid/crate_ngsai/` index
error: no matching package named `abc` found
location searched: registry `http://gitlab.ngs-ai.lab/eschmid/crate_ngsai/`
required by package `goo v0.1.0 (/home/eschmid/Projects/goo)`

So it checks in the right place, but obviously the index contains nothing because the above step only did local index changes but did not push to the remove repo

Can ktra be used as a transparent caching proxy for crates.io?

Hi, our internal CI system seems recently to be getting a number of failed downloads (see below for an example) from crates.io and anyway it's not very efficient to make lots of requests to crates.io for downloading the same crate again and again. Therefore, I was searching for a "transparent" caching proxy for crates.io and came upon ktra. I wonder if it can serve my envisioned purpose - it seems quite likely but I'd like to ask here, also to get any tips and perhaps to suggest updating the documentation with this use case in mind.

It seems like ktra may be able to perfectly fit my desired use case of a transparent caching proxy for crates.io:

  1. I configure cargo inside the CI system to access our instance of ktra and thus only ktra downloads the packages from crates.io, which it does only once.
  2. As we don't plan to host any private packages we would ideally not have to change the repositories (e.g. the Cargo.toml files) of the packages being built. Rather we would set this up in a way where it is only used for builds inside the CI system. So ktra would be "transparent" in that sense.
  3. As a further idea, perhaps ktra could delete its mirrored cache of files once per day (or week or month) to avoid accumulating gigabytes of packages which are never used. I guess ktra could alternatively track number of downloads and delete any package from the crates.io mirror which hasn't been downloaded for some specified period of time.

If this is currently possible with ktra, I would be happy to hear it and receive any pointers about how to set it up. I could try to improve this for inclusion into the book if we end up going this route.

Alternatively, I imagine this may be the domain of a traditional, "standard" HTTP caching proxy like squid. At least, perhaps it would have previously been easy before everything was HTTPS. In that case, I suppose that ktra is overkill. Nevertheless, it would be useful for people like me to know how to set that up. So a pointer for that would still be greatly appreciated.

(Also, as I write this, I have discovered that others have similar issues and suggest reverting to an older nightly may solve the errors in CI builds. Nevertheless, for efficiency's sake, the idea of caching is still interesting, even if the specific errors I am seeing can be solved with using a different version of nightly rust and cargo.)

Finally, here is an example of an error we get with crates.io (This is not bug a bug with ktra. So far we are not useing ktra.):

$ cargo build
 Downloading crates ...
  Downloaded rand_chacha v0.1.1
error: failed to download from `https://crates.io/api/v1/crates/half/1.7.1/download`
Caused by:
  [55] Failed sending data to the peer (Connection died, tried 5 times before giving up)

Ktra server commits empty files

Trying to get this to work with the following configuration:

[index_config]
remote_url = "https://github.com/******************.git"
https_username = "*********************************"
https_password = "**********************************"
branch = "master"

... but when I publish a crate the ktra server publishes empty files in the git index, instead of metadata files. This results in the package tarball verification failing when trying to pull the crate.

Changing dl /API url

In the example from Ktra book under bullet point 3 Create config.json file then commit and push it to remote repository. I did the following:

cd YOUR_INDEX
echo '{"dl":"http://localhost:8000/dl","api":"http://localhost:8000"}' > config.json
git add config.json
git commit -am "initial commit"
git push origin main

This all works on the same machine as I am running Ktra. However, I want to access my private cargo from another machine. So I edited the config.json file:

# from
{"dl":"http://localhost:8000/dl","api":"http://localhost:8000"}
# to
{"dl":"http://192.168.1.100:8000/dl","api":"http://192.168.1.100:8000"}

Ktra is currently on 192.168.1.100. When I do cargo build on another machine (I have set up my registry in .cargo/config.json) I get an error message:

failed to get 200 response from `http://192.168.1.100:8000/dl/dns_qprb/0.2.6/download`, got 404

How do i fix this? How I get other machines to pull private cargo from my own repo

Thanks

[Maintenance] Publish 0.7.0

Hello,

This issue serves as marker/reminder that even in the 0.7.0 version has been properly merged and tagged in the main branch, it hasn't been pushed to crates.io yet because I don't "own" the crate there; and the new version of the book has been merged but not deployed either as far as I understand.

To publish:

  • ktra 0.6.0 on crates.io
  • ktra 0.7.0 on crates.io
  • moriturus/ktra-book@6th_edition on book.ktra.dev

Delete versions / crates

Crates.io only supports yanking, so other projects/crates, that are locked to a specific version do not break. However, in a private environment, like ktra, it is safer to remove old versions/crates since you can control whether other crates are locked to that particular version.

This should be implemented as an extra web API, since cargo only has the yank subcommand. For extra safety, the delete API should only work on versions that are already yanked, and that are not depended on by any (non-yanked) crate in the same repository.

[Question] Does ktra needs write access to index repository?

Hi!

We are on the verge to install Ktra in my workplace and I'm reading the Ktra book on the procedure (thanks for writing it!).

I have a couple of questions though:

  • Why is config.json refering to localhost?
  • Does the user/password to the index needs write access to the index repository?

Thank you for your help!

Add auth feature to manage download crates

I wanted to try out a private registry in the company where I work to allow better way than submodules and https clone for internal dependencies. But it seems that there is no way to limit crates download thus the crates are totally public.

I found out there is talk about this feature since a year and a RFC have been accepted about it rust-lang/cargo#10474.

So this issue is about tracking the implementation of this feature in ktra.

non-`master` branches in index unsupported by `cargo`

cargo at the moment does not support indexes with a main branch as anything other than master. See rust-lang/cargo#4024 for the PR that forced this and rust-lang/cargo#7329 for an issue tracking support for other branches.

ktra's default branch is currently main and it makes it seem like main will work as a main branch for an index in it's documentation. Until cargo adds support, I think ktra should change it's default branch and all references in the documentation to master to avoid difficulty when a user is trying to set up a new registry.

Renamed packages aren't usable

My full story lives there, the short story is that the name and package values are reversed when a package is renamed in a dependency

FTR the extract from the Cargo Book, with matching permalink:

In the git repo:

        {
            // Name of the dependency.
            // If the dependency is renamed from the original package name,
            // this is the new name. The original package name is stored in
            // the `package` field.
            "name": "rand",
            // The SemVer requirement for this dependency.
            // This must be a valid version requirement defined at
            // https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html.
            "req": "^0.6",
            // Array of features (as strings) enabled for this dependency.
            "features": ["i128_support"],
            // Boolean of whether or not this is an optional dependency.
            "optional": false,
            // Boolean of whether or not default features are enabled.
            "default_features": true,
            // The target platform for the dependency.
            // null if not a target dependency.
            // Otherwise, a string such as "cfg(windows)".
            "target": null,
            // The dependency kind.
            // "dev", "build", or "normal".
            // Note: this is a required field, but a small number of entries
            // exist in the crates.io index with either a missing or null
            // `kind` field due to implementation bugs.
            "kind": "normal",
            // The URL of the index of the registry where this dependency is
            // from as a string. If not specified or null, it is assumed the
            // dependency is in the current registry.
            "registry": null,
            // If the dependency is renamed, this is a string of the actual
            // package name. If not specified or null, this dependency is not
            // renamed.
            "package": null,
        }

in the cargo publish payload:

{
            // Name of the dependency.
            // If the dependency is renamed from the original package name,
            // this is the original name. The new package name is stored in
            // the `explicit_name_in_toml` field.
            "name": "rand",
            // The semver requirement for this dependency.
            "version_req": "^0.6",
            // Array of features (as strings) enabled for this dependency.
            "features": ["i128_support"],
            // Boolean of whether or not this is an optional dependency.
            "optional": false,
            // Boolean of whether or not default features are enabled.
            "default_features": true,
            // The target platform for the dependency.
            // null if not a target dependency.
            // Otherwise, a string such as "cfg(windows)".
            "target": null,
            // The dependency kind.
            // "dev", "build", or "normal".
            "kind": "normal",
            // The URL of the index of the registry where this dependency is
            // from as a string. If not specified or null, it is assumed the
            // dependency is in the current registry.
            "registry": null,
            // If the dependency is renamed, this is a string of the new
            // package name. If not specified or null, this dependency is not
            // renamed.
            "explicit_name_in_toml": null,
        }

Self contained docker stack

I would like a way to get rid of the git repo requirement, would it be possible to set up a docker stack that contains ktra and a git server only used for ktra, the git could be read by all externals (or only with auth) and only write by ktra in the stack (no other write allowed).

This would allow to make git requirement more transparent for user of ktra.

I will maybe try to set up it myself, but I'm lack of git server knowledge so any help / advice would be nice.

nice rust-lang/cargo#10470 look like we will soon be able to remove git as a database... :p

Add https for the server

Unless I miss something ktra only support http, thus one could for now just put a proxy that handle https for ktra it would be nice to have a https features. (I will probably try to add it since it's in my skillset).

Dependency listed incorrectly if it has aliased

Hi @moriturus, thanks for open-sourcing this useful project. It works very smoothly(especially with Docker containers :D), but I believe there is a subtle bug:

When a package to be published has a dependency with package field, ktra places it into index wrongly.

Steps to reproduce:

  1. Create a dummy package(say foo)
[package]
name = "foo"
version = "0.1.0"
edition = "2018"


[dependencies]
hello-world = { version = "*", package = "hello" }
  1. Publish it
  2. Referring Cargo reference, the added index should have dependency with fields "name": "hello-world", "package": "hello", but two fields are interchanged. This causes confusing dependency resolution failure on cargo fetch(no matching package), although cargo search on ktra registry returns the exact package.

To resolve, Into<Dependency> impl of MetadataDependency should swap two fields.

ktra/src/models.rs

Lines 19 to 34 in 8a14cd2

impl Into<Dependency> for MetadataDependency {
#[tracing::instrument(skip(self))]
fn into(self) -> Dependency {
Dependency {
name: self.name,
req: self.version_req,
features: self.features,
optional: self.optional,
default_features: self.default_features,
target: self.target,
kind: self.kind,
registry: self.registry,
package: self.explicit_name_in_toml,
}
}
}

(possibly fixing) change:

        Dependency {
            name: self
                .explicit_name_in_toml
                .clone()
                .unwrap_or_else(|| self.name.clone()),
            req: self.version_req,
            features: self.features,
            optional: self.optional,
            default_features: self.default_features,
            target: self.target,
            kind: self.kind,
            registry: self.registry,
            package: if self.explicit_name_in_toml.is_some() {
                Some(self.name)
            } else {
                None
            },
        }

Thanks in advance!

Ktra openid doesn't install via cargo

When installing Ktra 0.7.0 with openid-connect support via cargo install ktra --no-default-features --features=openid, compilation fails due to several missing crates:

error[E0432]: unresolved import `argon2`
 --> /home/david/.cargo/registry/src/github.com-1ecc6299db9ec823/ktra-0.7.0/src/db_manager/sled_db_manager.rs:6:14
  |
6 | use argon2::{self, hash_encoded, verify_encoded};
  |              ^^^^ no external crate `argon2`

error[E0433]: failed to resolve: use of undeclared crate or module `rand`
 --> /home/david/.cargo/registry/src/github.com-1ecc6299db9ec823/ktra-0.7.0/src/utils.rs:7:5
  |
7 | use rand::prelude::*;
  |     ^^^^ use of undeclared crate or module `rand`

error[E0433]: failed to resolve: use of undeclared crate or module `rand`
 --> /home/david/.cargo/registry/src/github.com-1ecc6299db9ec823/ktra-0.7.0/src/utils.rs:6:5
  |
6 | use rand::distributions::Alphanumeric;
  |     ^^^^ use of undeclared crate or module `rand`

error[E0432]: unresolved imports `argon2`, `rand::distributions::Alphanumeric`
 --> /home/david/.cargo/registry/src/github.com-1ecc6299db9ec823/ktra-0.7.0/src/db_manager/utils.rs:3:14
  |
3 | use argon2::{self, ThreadMode, Variant};
  |              ^^^^ no external crate `argon2`
  |
 ::: /home/david/.cargo/registry/src/github.com-1ecc6299db9ec823/ktra-0.7.0/src/utils.rs:6:5
  |
6 | use rand::distributions::Alphanumeric;
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error[E0433]: failed to resolve: use of undeclared crate or module `argon2`
  --> /home/david/.cargo/registry/src/github.com-1ecc6299db9ec823/ktra-0.7.0/src/error.rs:41:12
   |
41 |     Argon2(argon2::Error),
   |            ^^^^^^ use of undeclared crate or module `argon2`

error[E0433]: failed to resolve: use of undeclared crate or module `rand`
  --> /home/david/.cargo/registry/src/github.com-1ecc6299db9ec823/ktra-0.7.0/src/utils.rs:31:9
   |
31 |         rand::thread_rng()
   |         ^^^^ use of undeclared crate or module `rand`

Announcement: About the development of Ktra in future

Dear all,

As you all know Ktra is now under inactive development.
I have not give it up but it is true that I cannot make time to develop for some time because of my main job is too busy.

Thus I decided to look for co-developers to continue with Ktra development.
However, it is not fixed that to make Ktra be an organized project or invite some people.
This is the first attempt for me, it might puzzle you, please let me know you think.

Regards,
Henrique Sasaki Yuya

Store crates on S3

Hello,

Thank for this project !
Do you have planned to store assets on S3 storage ?
Where data are stored using Docker container ?

Thanks
Marc-Antoine

Recovering lost "accounts"

I am looking at using ktra as an internal registry at work, and while I don't think I will loose my password, there is the possibility for others to do so. It would be nice to have a way to delete user accounts or forcibly set an account's password to allow for recovery.

Add verbosity option to `ktra` command?

Hello, thanks for the project!

Background: I've followed the directions. When I run ktra for the first time, it appears to hang. (In this issue, I'm not asking for direct help on this part.)

My ask for this issue: I'd like to get more visibility about the steps being performed. How about --verbose and -v command line options?

Ktra panics when using Mongo db backend

Ktra panics when trying to use the MongoDB driver with Ktra. This seems to stem from an incompatibility between tokio versions (ktra - tokio 1.1, mongodb - tokio 0.2). Here is a backtrace -

Finished dev [unoptimized + debuginfo] target(s) in 1m 32s
     Running `target/debug/ktra --mongodb-url 'mongodb://ktra:password@localhost:27017/'`
thread 'main' panicked at 'there is no timer running, must be called from the context of a Tokio 0.2.x runtime', /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.25/src/time/driver/handle.rs:24:32
stack backtrace:
   0: rust_begin_unwind
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/panicking.rs:142:14
   2: core::panicking::panic_display
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/panicking.rs:72:5
   3: core::panicking::panic_str
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/panicking.rs:56:5
   4: core::option::expect_failed
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/option.rs:1874:5
   5: core::option::Option<T>::expect
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/option.rs:738:21
   6: tokio::time::driver::handle::Handle::current
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.25/src/time/driver/handle.rs:24:9
   7: tokio::time::driver::registration::Registration::new
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.25/src/time/driver/registration.rs:18:22
   8: tokio::time::delay::Delay::new_timeout
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.25/src/time/delay.rs:72:28
   9: tokio::time::timeout::timeout
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.25/src/time/timeout.rs:53:17
  10: mongodb::runtime::AsyncRuntime::timeout::{{closure}}
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/mongodb-1.2.5/src/runtime/mod.rs:139:13
  11: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/future/mod.rs:91:19
  12: mongodb::sdam::message_manager::TopologyMessageSubscriber::wait_for_message::{{closure}}
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/mongodb-1.2.5/src/sdam/message_manager.rs:66:55
  13: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/future/mod.rs:91:19
  14: mongodb::client::Client::select_server::{{closure}}
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/mongodb-1.2.5/src/client/mod.rs:280:60
  15: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/future/mod.rs:91:19
  16: mongodb::client::executor::<impl mongodb::client::Client>::get_session_support_status::{{closure}}
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/mongodb-1.2.5/src/client/executor.rs:350:76
  17: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/future/mod.rs:91:19
  18: mongodb::client::executor::<impl mongodb::client::Client>::start_implicit_session::{{closure}}
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/mongodb-1.2.5/src/client/executor.rs:326:48
  19: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/future/mod.rs:91:19
  20: mongodb::client::executor::<impl mongodb::client::Client>::execute_operation::{{closure}}
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/mongodb-1.2.5/src/client/executor.rs:48:68
  21: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/future/mod.rs:91:19
  22: mongodb::coll::Collection<T>::estimated_document_count::{{closure}}
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/mongodb-1.2.5/src/coll/mod.rs:223:44
  23: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/future/mod.rs:91:19
  24: <ktra::db_manager::mongo_db_manager::MongoDbManager as ktra::db_manager::traits::DbManager>::new::{{closure}}::{{closure}}::{{closure}}
             at ./src/db_manager/mongo_db_manager.rs:75:57
  25: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/future/mod.rs:91:19
  26: <F as futures_core::future::TryFuture>::try_poll
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.21/src/future.rs:82:9
  27: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.21/src/future/try_future/into_future.rs:34:9
  28: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.21/src/future/future/map.rs:55:37
  29: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.21/src/lib.rs:91:13
  30: <futures_util::future::try_future::MapErr<Fut,F> as core::future::future::Future>::poll
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.21/src/lib.rs:91:13
  31: <ktra::db_manager::mongo_db_manager::MongoDbManager as ktra::db_manager::traits::DbManager>::new::{{closure}}::{{closure}}
             at ./src/db_manager/mongo_db_manager.rs:89:42
  32: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/future/mod.rs:91:19
  33: <ktra::db_manager::mongo_db_manager::MongoDbManager as ktra::db_manager::traits::DbManager>::new::{{closure}}
             at ./src/db_manager/mongo_db_manager.rs:58:5
  34: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/future/mod.rs:91:19
  35: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/future/future.rs:124:9
  36: ktra::run_server::{{closure}}::{{closure}}
             at ./src/main.rs:137:60
  37: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/future/mod.rs:91:19
  38: ktra::run_server::{{closure}}
             at ./src/main.rs:107:1
  39: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/future/mod.rs:91:19
  40: ktra::main::{{closure}}
             at ./src/main.rs:360:23
  41: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/future/mod.rs:91:19
  42: tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.0/src/park/thread.rs:267:54
  43: tokio::coop::with_budget::{{closure}}
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.0/src/coop.rs:102:9
  44: std::thread::local::LocalKey<T>::try_with
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/thread/local.rs:445:16
  45: std::thread::local::LocalKey<T>::with
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/thread/local.rs:421:9
  46: tokio::coop::with_budget
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.0/src/coop.rs:95:5
  47: tokio::coop::budget
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.0/src/coop.rs:72:5
  48: tokio::park::thread::CachedParkThread::block_on
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.0/src/park/thread.rs:267:31
  49: tokio::runtime::enter::Enter::block_on
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.0/src/runtime/enter.rs:152:13
  50: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.0/src/runtime/scheduler/multi_thread/mod.rs:79:9
  51: tokio::runtime::Runtime::block_on
             at /home/wearable-avionics/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.0/src/runtime/mod.rs:492:44
  52: ktra::main
             at ./src/main.rs:360:5
  53: core::ops::function::FnOnce::call_once
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/ops/function.rs:248:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.