This code is very inflexible and cannot support azure blob other endpoints.

I have forked the project under alexbarta/sbkn.
After installing dep prerequisites:

mkdir -p $GOPATH/src/github.com/alexbarta && cd $_
git clone https://github.com/alexbarta/skbn.git && cd skbn
make

But I get the following compilation fail:

# make
dep ensure
go fmt ./pkg/... ./cmd/...
go vet ./pkg/... ./cmd/...
# github.com/alexbarta/skbn/cmd [github.com/alexbarta/skbn/cmd.test]
cmd/skbn.go:55:23: too many arguments in call to skbn.Copy
        have (string, string, int, float64)
        want (string, string, int)

Instead compiling the original project nuvo/skbn works fine, no issue.

Hello,

is there any plan to support minio s3 endpoints?

Thanks

First of all nice application!

When copying from s3 it only takes the first 1000 items
See https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#S3.ListObjects
Returns some or all (up to 1000) of the objects in a bucket. You can use the request parameters as selection criteria to return a subset of the objects in a bucket.

Could skbn please update to modern go mod to track the dependencies? This integrates better with newer Go projects.

I am getting a strange error when upgrading the awscli Homebrew recipe during a skbn process. skbn complains that the aws executable is missing.

Does skbn not use the standard AWS SDK for Go?

https://aws.amazon.com/sdk-for-go/

Does it not support for copying local file to dst?

Trying to copy file from kubernetes pod to azure storage. But facing the above issue.

I have a requirement to use FIPS endpoints for accessing S3 buckets as described here https://aws.amazon.com/compliance/fips/#FIPS_Endpoints_by_Service. However based on docs, I can either a) not see a way to enable this endpoint usage, or b) verify that the parameter AWS_S3_ENDPOINT has any effect on the endpoint used, and overall not verify that a FIPS endpoint is being used. I was wondering if this functionality already existed, and/or if it could be built in. Thanks in advance.

Can we please implement the build system with mage? mage provides stronger cross-platform guarantees, by parsing the build steps as pure Go code rather than sh/bat interpreted code.

https://magefile.org/

Hi All ,

I am trying to install skbn on Google GKE cluster and I am facing the below error. Can you please advise to move further. Please let me know if you need any further details. thanks.

$ make
dep ensure
/home/ganesan_mg/gopath:/google/gopath/src/github.com/nuvo/skbn is not within a known GOPATH/src
Makefile:46: recipe for target 'bootstrap' failed
make: *** [bootstrap] Error 1

$git rev-parse HEAD
f3f8aee

Hi, is there any interest in adding support for Backblaze B2 to skbn? I could probably tackle this in the next week or two if no one else wants to.

Currently skbn will download a file in its whole to memory, and upload it from memory.
This poses a problem when dealing with large files (larger than the memory resource limit).

Need to investigate if there is another way to go about transferring files without using local memory (or using a pre-determined memory).
Maybe we can stream it from source to destination in some way.

When uploading large files to aws s3, large files - for example an 120Gi file is truncated to 22.3 Gi.

Expected behaviour - upload the files in full.

It would be great to include Google Cloud Storage in this offering to ensure all major cloud providers are covered by this tool.

When copying from s3 to k8 foldernames with dots will bring the process to a halt, cause everything in front of the dot is stripped from the foldername.

I am trying to restore files from S3 to a Jenkins Helm instillation.

I have skbn installed locally and get the following error. I have tried running as an admin and the Jenkins service account we have in AWS.

error in Stream: Unauthorized dst: file: default/[pod-id]/jenkins-master/var/jenkins_home/jobs/admin/config.xml

I also setup the in-cluster from the examples and get the same error.

The path in Jenkins is owned by root

If I do not add the AWS secret and just use ~/.kube/config I get the following error:

2019/09/19 10:17:15 AccessDenied: Access Denied
	status code: 403, request id: 926596AA106D4016, host id: QgFAcBC8IHi2KgMjnYvz2uL7Ozx0[...]

What could I be missing in any of the solutions I have tried?

Please support both the newer M1 Macs, as well as common Intel Macs, so that skbn runs reliably on macOS.

We have an issue whereby copying from an s3 bucket to a k8s pod the command never exits after copy is complete. Copy from k8s pod to s3 bucket works fine and the command exits after the files are copied.

Kubernetes 1.14
maorfr/skbn latest 26748c0240f8 7 months ago 50.2MB

command: ["skbn"]
       args:
       - cp
       - --src
       - s3://redacted/src/
       - --dst
       - k8s://redacted/dest/

Sadly there is no useful log output we could add to this ticket.

I'm trying use cron job to copy files from kubernetes to S3 and in kubernetes and got pod error OMMKilled
Looks like it slowly increase virtual memory and reaching pod limits.

Container limits set to: cpu 3. Memory: 5Gb
amount of copy files: 3401311
changing buffer memory didn't help
Am i doing something wrong?
or its memory leak issue?

Hello,

This tool is very good, but when i schedule it to take a backup of a certain pod, it recopies all the data and thats what is most time consuming, ideally if the data is same only the latest and the greatest should be copied like s3 sync. Can we have that capability, I am also open to contribute.

Let me know.

Please make sure to support local files as sources and targets.

Unfortunately, Kubernetes file transfer can be quite slow. So copying files first from S3 to local disk, then local disk to Kubernetes, would often be more reliable. You can restart the file transfer without having to keep requesting the same data over and over from S3, which reduces overhead for every single file transferred.

If you must have a URI scheme, then use file://, e.g. file:///Users/andrew/Downloads/coolbeans.txt