Asynchronous and Distributed Data Exporter from a Django QuerySet
DDE main goal is to get data from a QuerySet and export to an output format (currently CSV only) using Celery for task distribution.
pip install django-dde
DDE uses Celery to manage its tasks. Currently our broker defaults to Redis so be sure to install the dependencies need, for more information click here.
File management is dealt using Django's Default Storage. For more information on how to use it please click here.
We developed DDE using Django 1.11 and Python 3.5. Other versions are pending tests. Let us know if works in your virtual environment.
First of all:
pip install django-dde
Excelent! Now you must ensure that in your settings.py
you have these configurations:
EXPORTER_FROM_EMAIL = '[email protected]'
Now you have to configure a periodic task using Celery, please refere to this link.
Now in the code:
from exporter.models import Exporter
exporter = Exporter.objects.create_exporter(
queryset=Model.objects.queryset, # Change Model to your Model
email=example@mail.com, # E-mail that will receive the link to the exporter,
limit_per_task=100, # How many entries each task will proccess
)
And that's it! Simple, right?
Installing our dependencies in your virtual environment is pretty straight foward:
pip install -r requirements/development.txt
But you can use our Docker to test DDE. This method already install all dependencies on a container, creates a Redis container and you ready to go. We already configured some shortcuts to help:
make build # Builds docker image, be sure to run this first
make build-no-cache # Build/Rebuilds (cleaning cache) docker image
make tests # Run all tests
make bash # Enters Docker's Bash
make help # Shows the available commands
- 0.1
- Initial version
Stored E-commerce
Distributed under the MIT license. See LICENSE
for more information.
- Fork it (https://github.com/yourname/yourproject/fork)
- Create your feature branch (
git checkout -b feature/fooBar
) - Commit your changes (
git commit -am 'Add some fooBar'
) - Push to the branch (
git push origin feature/fooBar
) - Create a new Pull Request
Remember: Every feature created or fixed must have a test written and/or fixed.