Code Monkey home page Code Monkey logo

mass-data-generator's Introduction

mass-data-generator

Mass data generator for dev pruposes. This module generates csv data with as many records as needed, creating people, credit cards, bank accounts and/or bank transactions fake data to be used in developing, data science or any purposes.

Created data are invented and random, but some (credit cards) are generated using real validation algorithm.

When generating person data, the namefake.com API is used. Main module may request a proportional number of names and birthdays and mix queried data in order to increase performance, reducing requests to the external RESTful service.

Note: if you system is configured with ulimits, you may need to generate larger sets in more than one execution; with my cpu limit set to 180, I generate 1M people per time.

 
 

README Map

Run from command line

From the project's root directory, run the following command and follow instructions to
generate any number of registries and save them to a csv file:

python -m datagenerator

Run from python console

from datagenerator.Person import Person
for i in range(3):
    p = Person()
    p.create()
    if i == 0: print(','.join(p.info().keys()))
    print(p.csv())

from datagenerator.CreditCard import CreditCard
cc = CreditCard(person=p)
cc.info()
cc.csv()

from datagenerator.BankAccount import BankAccount
acc = BankAccount(person=p); acc.info()

from datagenerator.BankTransaction import BankTransaction
t = BankTransaction(account=acc, value=1000.00, description='Opening deposit'); t.info()
t = BankTransaction(account=acc, value=-100.00, description='Withdrawl'); t.csv()

Automatic tests

Python tests are available using unittest:

python -m unittest tests/test_*

Logs

Logs are recorded in the 'logs' directory in the project's root.
Please see docs/logs if you wish to access samples of the generated logs.

Documentation

See docs for documentation. Or run:

python -c "import datagenerator; help(datagenerator)"
python -c "import datagenerator.Person; help(datagenerator.Person)"
python -c "import datagenerator.CreditCard; help(datagenerator.CreditCard)"
python -c "import datagenerator.BankAccount; help(datagenerator.BankAccount)"
python -c "import datagenerator.BankTransaction; help(datagenerator.BankTransaction)"

To do

  • Add automatic tests.
  • Automatic rotation of log files.
  • Log classes other than Person.
  • Document main module.

mass-data-generator's People

Contributors

nandoabreu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

mass-data-generator's Issues

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.