Code Monkey home page Code Monkey logo

export-dynamodb's Introduction

export_dynamodb

CircleCI License: GPL v3

A cli to export dynamodb. export-dynamodb on pypi.

Key Features

  • Scan table in single or parallel thread.
  • Output file can be json or csv.
  • Get list of tables from yaml file.

Assumption

  • credentials and regions are configured in ~/.aws/config file.

How To Use

$ pip install export-dynamodb
$ export-dynamodb --help

Getting Started

# Install virtual environment
$ pip3 install virtualenv 
$ virtualenv -p python3 venv
$ source venv/bin/activate

# Install dependencies
$ pip install -r requirements.txt

# test cli local
pip install -e .

# if you want to exit from development environment, use deactivate command
$ deactivate

python setup.py sdist upload -r pypi

# Upload new version
$ python setup.py sdist
$ twine upload dist/*

export-dynamodb's People

Contributors

sarjann avatar truongleswe avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

export-dynamodb's Issues

ImportError: cannot import name 'Mapping' from 'collections'

I'm using python 3.10 and installed all the requirements.
The program fails immediatly on the import error, here is the stack trace:

C:\dci\data_migration\venv\Scripts\python.exe C:/dci/data_migration/exportdynamodb/entrypoints/main.py --profile dev -t Carriers -f csv -o carriers.csv
Traceback (most recent call last):
File "C:\dci\data_migration\exportdynamodb\entrypoints\main.py", line 7, in
from boto3 import Session
File "C:\dci\data_migration\venv\lib\site-packages\boto3_init_.py", line 16, in
from boto3.session import Session
File "C:\dci\data_migration\venv\lib\site-packages\boto3\session.py", line 17, in
import botocore.session
File "C:\dci\data_migration\venv\lib\site-packages\botocore\session.py", line 25, in
import botocore.configloader
File "C:\dci\data_migration\venv\lib\site-packages\botocore\configloader.py", line 19, in
from botocore.compat import six
File "C:\dci\data_migration\venv\lib\site-packages\botocore\compat.py", line 25, in
from botocore.exceptions import MD5UnavailableError
File "C:\dci\data_migration\venv\lib\site-packages\botocore\exceptions.py", line 15, in
from botocore.vendored.requests.exceptions import ConnectionError
File "C:\dci\data_migration\venv\lib\site-packages\botocore\vendored\requests_init_.py", line 58, in
from . import utils
File "C:\dci\data_migration\venv\lib\site-packages\botocore\vendored\requests\utils.py", line 26, in
from .compat import parse_http_list as parse_list_header
File "C:\dci\data_migration\venv\lib\site-packages\botocore\vendored\requests\compat.py", line 7, in
from .packages import chardet
File "C:\dci\data_migration\venv\lib\site-packages\botocore\vendored\requests\packages_init
.py", line 3, in
from . import urllib3
File "C:\dci\data_migration\venv\lib\site-packages\botocore\vendored\requests\packages\urllib3_init_.py", line 10, in
from .connectionpool import (
File "C:\dci\data_migration\venv\lib\site-packages\botocore\vendored\requests\packages\urllib3\connectionpool.py", line 38, in
from .response import HTTPResponse
File "C:\dci\data_migration\venv\lib\site-packages\botocore\vendored\requests\packages\urllib3\response.py", line 9, in
from .collections import HTTPHeaderDict
File "C:\dci\data_migration\venv\lib\site-packages\botocore\vendored\requests\packages\urllib3_collections.py", line 1, in
from collections import Mapping, MutableMapping
ImportError: cannot import name 'Mapping' from 'collections' (C:\Users\diuppa\AppData\Local\Programs\Python\Python310\lib\collections_init
.py)

Thoughts?

Why is "cur_total" twice as much as it should be?

I was reading through the source code of this package cause I needed to copy something, when I realized that:

cur_total = (len(items) + raw_data['Count'])

Why is this defined like this? It effectively makes the count of total items retrieved twice as much as it actually is, since items = raw_data['Items'].

Can do Incremental export?

I don't know if this feature can be implemented.

I didn't find incremental export of dynamodb through google.

Numeric attribute values are exported as strings, not numbers

When I have a DynamoDB item like this:

{
  "id": "7e7e9de9-cb80-4697-bbed-b112caa34050",
  "minValue": 0,
  "maxValue": 10000,
  "readingType": "Power",
  "units": "Watts"
}

where the types of minValue and maxValue are numbers (not strings), the exported version records them as strings:

{
  "id": "7e7e9de9-cb80-4697-bbed-b112caa34050",
  "minValue": "0",
  "maxValue": "10000",
  "readingType": "Power",
  "units": "Watts"
}

all data is kept in memory before writing to file

Hi from the looks of it all data is gathered in memory before writing to a file.
This works well for small tables. However for big tables the data should be written to file/disk instead of keeping it in memory.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.