kobayashi / s3monkey Goto Github PK

A Python library that allows you to interact with Amazon S3 Buckets as if they are your local filesystem.

Python 100.00%

mock python s3 fs fuse monkeypatching

s3monkey's Introduction

s3monkey: Access your S3 buckets like they're native files

Platforms like Heroku don't allow for FUSE filesystem usage, so I had to get a bit creative.

Introducing, s3monkey, a library that mocks out all standard Python library system file operations, allowing you to use already–written code to interface with Amazon S3.

All standard library file operation modules are patched when using the provided context manager, including the built–in open, os, io, & pathlib.

If you're interested in financially supporting Kenneth Reitz open source, consider visiting this link. Your support helps tremendously with sustainability of motivation, as Open Source is no longer part of my day job.

Potential Use Cases

Running Jupyter Notebooks on non-persistient storage (still being worked out).
Storing user uploads for Django applications (e.g. the media folder).

Usage

AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are expected to be set:

$ AWS_ACCESS_KEY_ID=xxxxxxxxxxx
$ AWS_SECRET_ACCESS_KEY=xxxxxxxxxxx

Basic usage:

from s3monkey import S3FS

with S3FS(bucket='media.kennethreitz.com', mount_point='/app/data') as fs:

    # Create a 'test' key on S3, with the contents of 'hello'.
    with open('/app/data/test', 'w') as f:
        f.write('hello')

    # List the keys in the S3 bucket.
    print(os.listdir('/app/data'))
    # ['file1.txt', 'file2.txt', 'file2.txt', 'test', …]

Installation

$ pipenv install s3monkey

This module only supports Python 3.

s3monkey's People

Contributors

Stargazers

Watchers

Forkers

willyhakim devlord-shiv priestd09 theassyrian fpedrini 7minus2 cloudxtreme khurramhazen jayd2446 bigsoftcms munstar0s forky-mcforkface 5l1v3r1

s3monkey's Issues

The bucket doesn't exist

I keep getting an error, saying

~/anaconda/lib/python3.6/site-packages/bucketstore.py in __init__(self, name, create)
     36             else:
     37                 raise ValueError(
---> 38                     'The bucket {0!r} doesn\'t exist!'.format(self.name))
     39 
     40     def __getitem__(self, key):

ValueError: The bucket 'mybucket.s3.amazonaws.com' doesn't exist!

I've tried multiple formats:

S3FS(bucket='mybucket.s3.amazonaws.com', mount_point='/s3/test')
S3FS(bucket='mybucket.s3.eu-west-2.amazonaws.com', mount_point='/s3/test')
S3FS(bucket='s3.amazonaws.com/mybucket', mount_point='/s3/test')
S3FS(bucket='s3.eu-west-2.amazonaws.com/mybucket', mount_point='/s3/test')

All failed with the same message...

Any ideas how to overcome this?

Thanks!

[Discussion] folder from new context

Avant-Propos

Hello there !

I pretty much love this library idea, and it would fit my use case perfectly !

Now I was just wondering about a few issues I encountered while testing it.

Chain calls of context:

import s3monkey

folder = "/tmp/foldername"
fpath = f"{folder}/somefile.txt"
s3_ctx = s3monkey.S3FS(bucket="s3monkey-testing", mount_point="/tmp")

with s3_ctx as fs:
    pass

with s3_ctx as fs:
    pass

Here is the traceback:

Traceback (most recent call last):
  File "/tmp/sample.py", line 14, in <module>
    with s3_ctx as fs:
  File "/Users/gjeusel/src/s3monkey/s3monkey/core.py", line 122, in __enter__
    self.patcher.setUp()
  File "/Users/gjeusel/src/s3monkey/s3monkey/pyfakefs/fake_filesystem_unittest.py", line 403, in setUp
    temp_dir = tempfile.gettempdir()
  File "/Users/gjeusel/miniconda3/envs/squamish/lib/python3.7/tempfile.py", line 294, in gettempdir
    tempdir = _get_default_tempdir()
  File "/Users/gjeusel/miniconda3/envs/squamish/lib/python3.7/tempfile.py", line 209, in _get_default_tempdir
    with _io.open(fd, 'wb', closefd=False) as fp:
  File "/Users/gjeusel/src/s3monkey/s3monkey/pyfakefs/fake_filesystem.py", line 3995, in open
    file_path, mode, buffering, encoding, errors, newline, closefd, opener)
  File "/Users/gjeusel/src/s3monkey/s3monkey/pyfakefs/fake_filesystem.py", line 4445, in __call__
    return self.call(*args, **kwargs)
  File "/Users/gjeusel/src/s3monkey/s3monkey/pyfakefs/fake_filesystem.py", line 4559, in call
    fakefile_class, key = self.filesystem.open_callback(locals())
  File "/Users/gjeusel/src/s3monkey/s3monkey/core.py", line 154, in open_callback
    if 'w' in locals['mode'] and locals['file_'].startswith(self.mount_point):
AttributeError: 'int' object has no attribute 'startswith'

Path existence while having a folder in it:

While this works just perfectly:

import os
import s3monkey

folder = "/tmp/foldername"
fpath = f"{folder}/somefile.txt"
s3_ctx = s3monkey.S3FS(bucket="s3monkey-testing", mount_point="/tmp")

with s3_ctx as fs:
    os.mkdir(folder)

    with open(fpath, "w") as f:
        f.write("Hello")

    with open(fpath, "r") as f:
        print(f.readline())

    assert os.path.exists(fpath)

If I were to execute the following after having run the previous one
- meaning after having created this S3 object and check it does exist in my S3 bucket -,
this would fail:

import os
import s3monkey

folder = "/tmp/foldername"
fpath = f"{folder}/somefile.txt"
s3_ctx = s3monkey.S3FS(bucket="s3monkey-testing", mount_point="/tmp")

with s3_ctx as fs:
    assert os.path.exists(fpath)

Environment

Python 3.7.9
s3monkey: tip of master branch

Is this repository maintained ?

Also, if you need any more informations please don't hesitate !

Cordially,

Read file content

Is it possible to read file content as follow:

file_list = os.listdir(mount_point)
print(file_list[10])
with open(file_list[10], 'r') as f:
    print(f.read())

When I try I get exception: FileNotFoundError: [Errno 2] No such file or directory: