lericson / simples3 Goto Github PK

Simple, quick Amazon AWS S3 interface in Python

License: BSD 2-Clause "Simplified" License

Python 100.00%

simples3's Introduction

#simples3

##Overview A fairly simple, decently quick interface to Amazon's S3 storage service.

It grew out of frustration with other libraries that were either written too pragmatically (slow), too bloatedly, or just half-done.

The module aims for:

simplicity,
decent speed,
non-intrusiveness.

It really is designed to fit into programmer memory. The three basic operations are as easy as with dictionaries.

##Dependencies

Requires Python 2.5+ and nose for running tests. Python 3 support is not yet available. Apart from that, the code relies solely on Python standard libraries.

##Installation

pip install simples3

##Usage

Access to a bucket is done via the S3Bucket class. It has three required arguments:

from simples3.bucket import S3Bucket

s = S3Bucket(bucket,
             access_key=access_key,
             secret_key=secret_key)
 
print s  
#<S3Bucket ... at 'https://s3.amazonaws.com/...'>

To add a file, simply do

s.put("my file", "my content")

To retrieve a file do

f = s.get("my file")
print f.read()
#my content

To retrieve information about a file, do

print f.s3_info["mimetype"]
#'application/octet-stream'

print f.s3_info.keys()
#['mimetype', 'modify', 'headers', 'date', 'size', 'metadata']

To delete a file, do

del s["my file!"]

For more detailed documentation, refer here

##Contributing

###IRC #sendapatch on chat.freenode.net.

simples3's People

Contributors

Stargazers

Watchers

simples3's Issues

Add support for buckets in different regions

When trying to use simples3 to connect to a bucket hosted in eu-west-1, I get the following error:

The bucket you are attempting to access must be addressed using the specified endpoint.

Make it optional to add -x-amz to metadata during copy

I am trying to set some far future expiration on my files in a bucket on S3, using this python code:

future = 60*60*24*666
future_date = datetime.now() + timedelta(seconds=future)
future_date = future_date.strftime("%a, %d %b %Y %H:%M:%S GMT")

# and further down in file:
f = open(src)
s3.put(key, f.read())

s3.copy("%s/%s" % (bucket, key), key, metadata={
    "Cache-control": "max-age=%d" % future,
    "Expires": future_date})

However, when I investigate the files in firebug they do not have neither Expires och Cache-control. Instead there are prefixed versions of the metadata.

x-amz-meta-cache-control    max-age=57542400
x-amz-meta-expires  Thu, 15 May 2014 01:03:25 GMT

I would highly appreciate to set metadata and skip the "-x-amz-" prefixes.

UnboundLocalError on S3Bucket.listdir() method

S3Bucket.listdir() method raises UnboundLocalError. It seems to occur when its result list is empty, but I have no idea about its detail. What is next_marker attribute for?

Traceback (most recent call last):
  File "/.../app.py", line 134, in list_sizes
    keys = list(self.bucket.listdir(prefix=prefix + '/'))
  File "/home/styleshare/lib/python2.7/site-packages/simples3/bucket.py", line 395, in listdir
    for item in listing:
  File "/home/styleshare/lib/python2.7/site-packages/simples3/bucket.py", line 189, in __iter__
    self.next_marker = item[0]
UnboundLocalError: local variable 'item' referenced before assignment

Needs proxy support

The only egress access out of my network is through a proxy server that is tightly controlled. How can I use a proxy with this? Or is this a feature that could be added?

HTTP 204 returned on S3Bucket.delete

Traceback (most recent call last):
  File "./s3sync_olaf.sh", line 317, in <module>
    main() 
  File "./s3sync_olaf.sh", line 301, in main
    trim_bucket(bucket)
  File "./s3sync_olaf.sh", line 116, in trim_bucket
    del s3b[key]
  File "/usr/lib/pymodules/python2.5/simples3/bucket.py", line 157, in __delitem__
    def __delitem__(self, name): return self.delete(name)
  File "/usr/lib/pymodules/python2.5/simples3/bucket.py", line 296, in delete
    resp = self.make_request("DELETE", key=key)
  File "/usr/lib/pymodules/python2.5/simples3/bucket.py", line 259, in make_request
    raise exc_cls.from_urllib(e, key=key)
simples3.bucket.S3Error: HTTP error (code=204, key='space/ourproject-2011-04-24.tar.gz', filename='http://s3.amazonaws.com/Assembla-space-ourproject/space/ourproject-2011-04-24.tar.gz')

I use the S3Bucket.listdir API call a lot. It gives a list of files. I iterate through it and check the filedate if it is an old file I run del s3b[key]. That command is on a separate line, and is the line triggering the exception.

Basically:

for (k, modify, etag, size) in b.listdir(...):
    if predicate(modify):
        del b[k]

Bucket creation

One should be able to create buckets. Suggested API for this is either:

from simples3 import S3Bucket
b = S3Bucket("foo", ...)
b.create(acl="public-read")

or class-style (a bit better IMO):

from simples3 import S3Bucket
b = S3Bucket.create_new("foo", ..., acl="public-read")

There are other suggestions as well, including module-level functions.

progress callback doesn't work

fp = ProgressCallingFile(fp, progress)
...

TypeError: __init__() takes exactly 4 arguments (3 given)

Need put_file return status

This not an issue but a small requirement.

There is no return type for put_file() method, hence could not decide whether the object is copied to S3 properly or not.

Broken RFC 822 formatting on non-US locale

The current implementation of RFC 822 formatting uses just time.strftime() function, and the formatter for it, defined in simples3.util.rfc822_fmt, is '%a, %d %b %Y %H:%M:%S GMT'. According to the documentation of time.strftime():

%a: Locale’s abbreviated weekday name.
%b: Locale’s abbreviated month name.

As a result, the current RFC 822 formatting is broken in non-US locale (e.g. ko_KR, ja_JP). You can reproduce it easily:

>>> import time
>>> t = time.gmtime()
>>> time.strftime('%a, %d %b %Y %H:%M:%S GMT', t)
'Wed, 30 Nov 2011 08:29:14 GMT'
>>> import locale
>>> locale.setlocale(locale.LC_ALL, 'ko_KR')
'ko_KR'
>>> time.strftime('%a, %d %b %Y %H:%M:%S GMT', t)
'\xec\x88\x98, 30 11 2011 08:29:14 GMT'
>>> print _
수, 30 11 2011 08:29:14 GMT
>>> locale.setlocale(locale.LC_ALL, 'ja_JP')
'ja_JP'
>>> time.strftime('%a, %d %b %Y %H:%M:%S GMT', t)
'\xe6\xb0\xb4, 30 11 2011 08:29:14 GMT'
>>> print _
水, 30 11 2011 08:29:14 GMT

This bug also produces incorrect signatures (used for AWS authentication).

`;` as query string separator causes a trouble

S3Request seems to have a problem with handling arguments because of using ; as args separator(https://github.com/lericson/simples3/blob/master/simples3/bucket.py#L138).
It works right with & instead of ;, so I carefully guess S3 may does not support ; separator.
Did I do anything wrong?

listdir limited to 1000 entries

Simple S3 is liiting the output to 1000 file when using listdir.

http://stackoverflow.com/questions/40331681/python-simples3-listdir-limit-the-output-to-1000-files

Any way / plan to make it support more than 1000 - or an unlimited number of entries?

Edit : fixed in github but not in pypi : Bad content-length set after 2 put_file

Hi,

When you issue 2 put_file request with different file size, the header content-length is stuck at the size of the first file, causing a 400 return code from server due to the invalid content-length.

To reproduce the issue :

f_path = 'tmp'
with open(f_path, 'wb') as f:
    f.write("toto")
s3.put_file(f_path, f_path)
with open(f_path, 'wb') as f:
    f.write("toto2")
s3.put_file(f_path, f_path)

Tested against RADOS GATEWAY.

The root cause is due to the fact that is header param is empty when calling put_file, a default dict is assigned in the function definition. https://github.com/lericson/simples3/blob/master/simples3/streaming.py#L34

As a result, every subsequent call will get the same object reference (as the dict is in the class scope and not function scope).

Content-length will always be set after the first call, as a result it is not updated
https://github.com/lericson/simples3/blob/master/simples3/streaming.py#L58

Here is a fix :

class StreamingMixin(object):
    def put_file(self, key, fp, acl=None, metadata={}, progress=None,
                 size=None, mimetype=None, transformer=None, headers=None):
        """Put file-like object or filename *fp* on S3 as *key*.
        *fp* must have a read method that takes a buffer size, and must behave
        correctly with regards to seeking and telling.
        *size* can be specified as a size hint. Otherwise the size is figured
        out via ``os.fstat``, and requires that *fp* have a functioning
        ``fileno()`` method.
        *progress* is a callback that might look like ``p(current, total,
        last_read)``. ``current`` is the current position, ``total`` is the
        size, and ``last_read`` is how much was last read. ``last_read`` is
        zero on EOF.
        """
        if not headers:
            headers = {}
        do_close = False

Don't bail when python lacks ssl support

If your python is compiled without ssl support, simples3 will start to cry:

python2.5 setup.py build -b build-2.5
Traceback (most recent call last):
  File "setup.py", line 8, in <module>
    import simples3
  File "simples3/__init__.py", line 124, in <module>
    from .bucket import S3File, S3Bucket, S3Error, KeyNotFound
  File "simples3/bucket.py", line 66, in <module>
    class StreamHTTPSHandler(urllib2.HTTPSHandler):
AttributeError: 'module' object has no attribute 'HTTPSHandler'

IMO ssl should be optional and not required

lericson / simples3 Goto Github PK

simples3's Introduction

simples3's People

Contributors

Stargazers

Watchers

Forkers

simples3's Issues

Recommend Projects

Recommend Topics

Recommend Org