seedifferently / boto_rsync Goto Github PK

This tools is fantastic-- but it seems crash on broken symlinks. Telling it to ignore 0 byte files doesn't solve it, since the symlinks are not 0 bytes. Maybe I'm missing something...

www/boi-cms.old/taps_ctrl.sh
Traceback (most recent call last):
File "/usr/local/bin/boto-rsync", line 1150, in
main()
File "/usr/local/bin/boto-rsync", line 544, in main
file_size = os.path.getsize(fullpath)
File "/usr/lib/python2.7/genericpath.py", line 49, in getsize
return os.stat(filename).st_size
OSError: [Errno 2] No such file or directory: '/var/www/boi-cms.old/current'
Scanning for files to transfer...
shells
Muttrc
pam.conf
nanorc
legal
Traceback (most recent call last):
File "/usr/local/bin/boto-rsync", line 1150, in
main()
File "/usr/local/bin/boto-rsync", line 544, in main
file_size = os.path.getsize(fullpath)
File "/usr/lib/python2.7/genericpath.py", line 49, in getsize
return os.stat(filename).st_size
OSError: [Errno 2] No such file or directory: '/etc/blkid.tab'
ckelty@beadle:/etc/backup$ ls -al /etc/blkid.
blkid.conf blkid.tab
ckelty@beadle:/etc/backup$ ls -al /etc/blkid.tab
lrwxrwxrwx 1 root root 15 2011-07-23 16:47 /etc/blkid.tab -> /dev/.blkid.tab
ckelty@beadle:/etc/backup$ ls -al /var/www/boi-cms.old/current
lrwxrwxrwx 1 capistrano www-data 40 2012-04-12 14:20 /var/www/boi-cms.old/current -> /var/www/boi-cms/releases/20120412212119
ckelty@beadle:/etc/backup$

I have been trying to upload multiple files, and at the beginning the files were overwritten in the bucket, but now they don't, how can I overwrite files or delete them in the bucket before uploading? I have this script executed in a cron but is not usable anymore because I do not have any way to overwrite files. Thanks!

running the below gives the error, but putting the key and secret in the cli works fine.
boto-rsync --access-key /etc/awsapi-key --secret-key /etc/awsapi-key --glob /var/log/nginx/access.log s3://nginx-access-logs/

I used the example in the boto config link
[Credentials]
aws_access_key_id =
aws_secret_access_key =

Any thoughts? I'm at a loss as to the why.

Traceback (most recent call last):
File "/usr/local/bin/boto-rsync", line 1122, in
main()
File "/usr/local/bin/boto-rsync", line 312, in main
debug = int(a)
ValueError: invalid literal for int() with base 10: '/srv/htdocs/firstclass/dev/instalacoes/lojaskd/novalojaskd_R1900/prd/11000/11011'

if i use --access-key the error appear like:

ValueError: invalid literal for int() with base 10: '--access-key'

:C

Hey there.

Every time I run boto_rsync it transfers every single file from the source to the destination. The command I'm using to run it is:

boto-rsync -v --access-key=XXXXXX --secret-key=XXXXXX --endpoint="s3-eu-west-1.amazonaws.com" /local/path/folder s3://bucketname/files/remote/path/folder

I've got boto 2.4.1 installed though not sure if that's relevant.

Thanks.

I'm getting errors of this form:

Host b.egf.me.commondatastorage.googleapis.com returned an invalid certificate (remote hostname "b.egf.me.commondatastorage.googleapis.com" does not match certificate): {'notAfter': 'Jun  7 19:43:27 2013 GMT', 'subjectAltName': (('DNS', '*.googleusercontent.com'), ('DNS', '*.blogspot.com'), ('DNS', '*.commondatastorage.googleapis.com'), ('DNS', '*.doubleclickusercontent.com'), ('DNS', '*.ggpht.com'), ('DNS', '*.googledrive.com'), ('DNS', '*.googlesyndication.com'), ('DNS', '*.storage.googleapis.com'), ('DNS', 'blogspot.com'), ('DNS', 'commondatastorage.googleapis.com'), ('DNS', 'doubleclickusercontent.com'), ('DNS', 'ggpht.com'), ('DNS', 'googledrive.com'), ('DNS', 'googleusercontent.com'), ('DNS', 'static.panoramio.com.storage.googleapis.com'), ('DNS', 'storage.googleapis.com')), 'subject': ((('countryName', u'US'),), (('stateOrProvinceName', u'California'),), (('localityName', u'Mountain View'),), (('organizationName', u'Google Inc'),), (('commonName', u'*.googleusercontent.com'),))}

I really wish that wrapped...

How would I go about using ssh for the destination?

The Python team at AWS has been working hard on a boto-based CLI project for their cloud services. Among the tools included is an interface for S3 which duplicates (and in many ways supersedes) most of the functionality provided by boto-rsync.

https://github.com/aws/aws-cli
http://docs.aws.amazon.com/cli/latest/reference/s3/index.html

Unless anybody sees a good reason not to, I'll probably deprecate this project in favor of awscli. They have more resources at their disposal for maintenance, not to mention the fact that it's an "official" tool provided by the AWS team.

Please discuss here if you feel otherwise.

I am one of the maintainers of Boto. A few weeks ago we began to merge backward-compatible Python 3.3 and 3.4 support into our develop branch. Most modules now support Python 3 with passing unit and integration tests, and I am hoping to do a release soon. I am reaching out to you to ask if you would test our develop branch with boto-rsync.

Testing against Python 2.x will help to ensure that we don't accidentally break third party modules like boto-rsync. If you would like, you may also try out the Python 3 support and let us know any feedback. More information about the Python 3 support can be found in this boto-users post:

https://groups.google.com/forum/#!topic/boto-users/vcvoy8zrfP8

I appreciate any time that you can put into helping the Boto project with this.

This request is part of boto/boto#2437

1 year without commit.
No pull request merged.
Many forks...

Maybe it's time to have a new maintainer(s) if @seedifferently doesn't have enough time.

Without answer, maybe the community must do a "community" fork. What do you think?

Some times boto-rsync fail to detect that some files are already up to date, and re-download them, as if some changed occured.
This is on a Raspberry running Raspbian.

When doing a boto-rsync from one bucket to another, is the data being transferred between directly between the two, or am I downloading, then uploading? From the out put it looks direct, but I'm can't tell for sure.

Not sure if this has been looked into already, but the speed of transfers seems to be vastly improved by threads/processes running in parallel.

My experience with is is using dreamhost, it's possible the official s3 is different.

I noticed a 5x improvement in speed in calling boto-rsync on each of 5 large directories, rather than on the single parent directory. I assume an internal system for using multiple upload threads/queues would see similar improvements.

E.g., this branch commit:

eroh92@971b423