gwihlidal / cloudstore-rs Goto Github PK

It is important to support paging multiple groups of keys in case the list is extremely long. This is done in AWS through "continuation tokens", and adding support for them would be a good improvement for large data sets.

References:

• https://github.com/rusoto/rusoto/blob/master/integration_tests/tests/s3.rs
• https://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Types/ListObjectsV2Output.html
• https://rusoto.github.io/rusoto/rusoto_s3/struct.ListObjectsV2Request.html
• https://rusoto.github.io/rusoto/rusoto_s3/struct.ListObjectsV2Output.html

It would be good to implement support for Docker, so that the microservice can be more easily deployed in a variety of infrastructures/configurations. The Dockerfile should build the source tree using cargo, and use the server binary as the default entry point.

Currently there doesn't appear to be a clean way t return a gRPC error. The public GrpcStatus enum maps the correct error code values for gRPC, but the module itself is private:

• stepancheg/grpc-rust#102
• https://github.com/stepancheg/grpc-rust/blob/master/grpc/src/grpc.rs

Discussions on proper gRPC error handling:

• stepancheg/grpc-rust#102
• http://avi.im/grpc-errors/
• grpc/grpc#4543

It would much better to replace code like this:

return grpc::SingleResponse::err(grpc::Error::GrpcMessage(grpc::GrpcMessageError {
    grpc_status: 15,
    grpc_message: "Failed to copy object data".to_string()
}));

With correct enum entries like this:

return grpc::SingleResponse::err(grpc::Error::GrpcMessage(grpc::GrpcMessageError {
    grpc_status: grpc::grpc::GrpcStatus::DataLoss as i32,
    grpc_message: "Failed to copy object data".to_string()
}));

Currently trying to use GrpcStatus results in visibility errors:

error[E0603]: module `grpc` is private
   --> src/bin/server.rs:187:42
    |
187 |                             grpc_status: grpc::grpc::GrpcStatus::DataLoss as i32,
    |                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

error: Could not compile `cloudstore`.

There are two areas that should be tested:

1. Very large files (multiple Gb)
2. Long upload times (> 60 min - in the case of slow connection speed)

For #1, likely point of failure will be the single bytes array definition in protobuf, as gRPC doesn't automatically chunk up that array into segments that fit within the max message transfer size. The usual approach here is to explicitly chunk up the data and send multiple messages.

For #2, likely point of failure will be login credentials expiring. This may not be an issue, but validating this is warranted:
https://medium.com/@otusweb/how-to-tame-the-aws-framework-to-upload-a-large-file-to-s3-19430d245510

There are a number of hardcoded entrypoints and settings that should be removed in favor of config and environment driven settings.

let endpoint = "http://192.168.1.69:9000".to_string();

let bucket_name = "p4content".to_string();

let credentials = rusoto_credential::StaticProvider::new_minimal(
            "AJ20P3XYDOURW7WZSHJ1".to_string(),
            "EVT9XzEw/77PevdntA88wjcEBF2cANl/Duc09mkl".to_string());

server.http.set_port(8080);