Code Monkey home page Code Monkey logo

tame-gcs's Introduction

๐Ÿ“‚ tame-gcs

Embark Embark Crates.io Docs dependency status Build Status

tame-gcs is a crate with a limited set of Google Cloud Storage(GCS) operations that follows the sans-io approach.

Why?

  • You want to control how you actually make HTTP requests against GCS.
  • You want to have more control over your dependencies, and not be bottlenecked by sticking to a particular version, or quickly upgrading, your HTTP related crates.

Why not?

  • This crate only supports some operations.
  • There are several other GCS crates available that have many more features and are easier to work with, if you don't care about what HTTP clients they use.
  • This crate requires more boilerplate to work with.

Example

For example usage, see the gsutil crate, which reimplements parts of the official gsutil tool.

Contributing

Contributor Covenant

We welcome community contributions to this project.

Please read our Contributor Guide for more information on how to get started.

License

Licensed under either of

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

tame-gcs's People

Contributors

davidpdrsn avatar deanbdean avatar jake-shadle avatar lpil avatar quettabit avatar repi avatar shikhar avatar soniasingla avatar xampprocky avatar yiwu-arbug avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

tame-gcs's Issues

Support custom endpoints

Is your feature request related to a problem? Please describe.

tame-gcs currently hardcodes the request URLs

"https://www.googleapis.com/upload/storage/v1/b/{}/o?uploadType=multipart",

"https://www.googleapis.com/storage/v1/b/{}/o/{}?alt=json",

If we want to use change the endpoints to use other GCS-compatible services e.g. fake-gcs-server, we will need to manually replace the URLs in the generated http::Request, which is awkward.

Note that the official GCS libraries in all other languages support providing custom endpoints:

Describe the solution you'd like

Have an API specifying which endpoint to use.

Describe alternatives you've considered

Additional context

BucketName rejects valid bucket names

Describe the bug

Bucket names may contain periods, but the BucketName type rejects all strings with periods.

This comment correctly links to these requirements for bucket names. Importantly, bucket names may contain dots (.) - as long as the bucket creator has demonstrated control over the domain containing the bucket name, when the bucket name is interpreted as a domain name.

Note that this means a domain name is always a valid domain name until it hits the API layer (i.e. whenever it's in Rust). I might not be able to create the my-gcs-bucket.embark-studios.com - but it's OK for y'all to create it, or for me to read from it (if/when it exists). That is: within my program('s type system), it's a valid BucketName; it may or may not be valid for a particular API call from a particular user.

To Reproduce

#[test]
fn try_bucket() {
  let bn = BucketName::try_from("my-gcs-bucket.embark-studios.com").unwrap();
}

Compile error with ring 0.16.16

Seems to be a breaking change from ring 0.16.15 -> 0.16.16

error[E0599]: no method named `description_` found for struct `ring::error::KeyRejected` in the current scope
   --> C:\Users\repi\.cargo\registry\src\github.com-1ecc6299db9ec823\tame-gcs-0.8.0\src\error.rs:162:31
    |
162 |         Error::KeyRejected(re.description_())
    |                               ^^^^^^^^^^^^ method not found in `ring::error::KeyRejected`

Ah was indeed a removal of deprecated functionality briansmith/ring@f06811a

Add signed url support

Signed urls are a useful way to get access to data for a limited time that is otherwise private and inaccessible on the general internet.

Implementation details can be found here for the latest version.

Upgrade to http 0.2

We need to upgrade this crate to the recently released 0.2 version of the http crate as that is what the rest of the ecosystem has moved to also with tokio and hyper. Ran into some problems with this crate when testing to upgrade them to the latest git versions

As the http crate is part of the public interface for this crate this requires a 0.x.0 version bump

Support resumable upload

Is your feature request related to a problem? Please describe.

Currently tame-gcs only exposed API for uploading a file atomically (insert_multipart and insert_simple).

For huge files it is more robust to use resumable upload (insert with uploadType=resumable) to upload the file in chunks.

Describe the solution you'd like

Add a method to support the resumable upload API.

Describe alternatives you've considered

Add a method to support the compose API (or both).

Additional context

Update to v0.3 of Embark's standard lints

Embark uses a standard set of lints across all repositories. We just upgraded
to version 0.3 of these lints, and this repository needs to be updated to
the new set of lints with any warnings fixed.

Steps

  1. Copy the contents of the lints.rs file from
    EmbarkStudios/rust-ecosystem and replace the old list of lints in the
    src/lib.rs and/or src/main.rs files with the new list.
  2. Run cargo clippy
  3. Update and fix any code that now triggers a new warning.

Support futures::AsyncRead being body for multipart upload API

Is your feature request related to a problem? Please describe.
no

Describe the solution you'd like
It looks to me the body of a multipart upload can only be a io::Read, which does not support async reading like the future::AsyncRead. Do you consider supporting it? Thanks.

Describe alternatives you've considered
n/a

Additional context
n/a

Make all enum Copyable?

Is your feature request related to a problem? Please describe.
Its very annoying to manually implement a copy constructor for enum.
Some of these are very large.

Describe the solution you'd like
Copies of cheap enums should be easy to make.

Describe alternatives you've considered
Creating a large functions that are error prone.

Additional context
Add any other context or screenshots about the feature request here.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.