Code Monkey home page Code Monkey logo

splits's Introduction

Splits

Splits is a library for reading and writing files in splittable chunks. It works on any file-like object. There is built in support for writing and reading split files from S3. It also has built in support for gzip.

Installation

  $ pip install splits

Usage

from splits import SplitWriter, SplitReader
from splits.s3 import S3File, GzipS3File

if __name__ == '__main__':

    with SplitWriter('s3://test-bucket/test-multifile',
                     suffix='.txt', lines_per_file=100,
                     fileClass=GzipS3File) as w:
        w.writelines([str(x) for x in range(0, 1000)])

    with SplitReader('s3://test-bucket/test-multifile',
                     fileClass=GzipS3File) as r:
        for line in r:
            print line

Tests

 $ pip install tox

To run the tests in both Python2 and Python3 run,

 $ tox

splits's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

splits's Issues

`.decode` in `SplitReader.read` fails for some values of `num`

The decode step will sometimes raise a UnicodeDecodeError, I think because it tries to decode num bytes from the file at a time, which isn't necessarily a valid utf-8 encoded string even if the full contents of the file is a valid utf-8 encoded string.

To reproduce:

This works fine:

>>> from ripley.readers import SFReader
>>> f = SFReader('prod', 'style')
2018-04-06 14:11:31,837 [INFO] wednesday.client:18 - __init__ ServiceClient for 'staunch' in environment 'prod' with base_uri 'http://staunch.vertigo.stitchfix.com'
2018-04-06 14:11:32,289 [INFO] ripley.metadata:41 - Finding metadata for prod.style?_no_partition_=y
2018-04-06 14:11:33,279 [INFO] ripley.metadata:54 - Only hive metadata found (no s3)
>>> s = f.read()
>>>

This fails:

>>> f = SFReader('prod', 'style')
2018-04-06 14:20:02,814 [INFO] wednesday.client:18 - __init__ ServiceClient for 'staunch' in environment 'prod' with base_uri 'http://staunch.vertigo.stitchfix.com'
2018-04-06 14:20:06,387 [INFO] ripley.metadata:41 - Finding metadata for prod.style?_no_partition_=y
2018-04-06 14:20:07,349 [INFO] ripley.metadata:54 - Only hive metadata found (no s3)
>>> s = ''
>>> while True:
...     s += f.read(1)
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/Users/ahunter/.virtualenvs/aa-py2/lib/python2.7/site-packages/splits/readers.py", line 62, in read
    new_data = new_data.decode('utf-8')
  File "/Users/ahunter/.virtualenvs/aa-py2/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe2 in position 0: unexpected end of data
>>>

Data corruption bug

Exceptions thrown by the authentication layer are occasionally swallowed by the read function of SplitReader:

def read(self, num=None):
        val = ''
        try:
            while True:
                if num > 0:
                    new_data = self._get_current_file().read(num - len(val))
                else:
                    new_data = self._get_current_file().read()

                if not new_data:
                    self._current_file.close()
                else:
                    val += new_data

                if num > 0 and len(val) == num:
                    break
        except:
            pass

        return val

The naked exception handler is apparently there to ignore a StopIteration exception.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.