Code Monkey home page Code Monkey logo

pydas's Introduction

Python Data Acquisition Service

A full python stack application for acquiring financial data and generating arbitrary data sets. Can be deployed as a REST API for dataset generation as a service, or run from the command line to create ad-hoc data sets for machine learning.

Installation

Currently, the simplest method for installing is from source. This guide assumes that Python 3.7 or greater is installed with pip and virtualenv available. It is assumed and recommended that a virtual environment is created, and will be denoted in the scripts shown:

$> git clone https://github.com/UVU-PFP-Research/Personal-Financial-Planner
$> cd Personal-Financial-Planner/stock-data-acquisition
(env) $> python setup.py install

Once the installation has been completed, you'll want to run the installer to setup the database. The installer can be found within the sDAS package and can be run by calling:

(env) $> python -m pydas.install -h
usage: install.py [-h] [-a ALEMBIC] [-r REQUIREMENTS] [-u USERNAME]
                  [-p PASSWORD] [-d DATABASE] [--deploy-rest]

Stock Data Acquisition Service installation script for configuring the
metadata database and REST API on a server.

optional arguments:
  -h, --help            show this help message and exit
  -a ALEMBIC, --alembic ALEMBIC
                        Path to alembic configuration file (typically
                        alembic.ini)
  -r REQUIREMENTS, --requirements REQUIREMENTS
                        Path to project dependency file (typically
                        requirements.txt)
  -u USERNAME, --user USERNAME
                        Database username for opening connections
  -p PASSWORD, --password PASSWORD
                        Database password for opening connections
  -d DATABASE, --database DATABASE
                        Database name to deploy metadata database into
  --deploy-rest         Flag indicating whether the REST API server should be
                        deployed. DO NOT USE THIS OPTION! uWSGI support is not
                        currently supported and using this flag will result in
                        undefined/unsupported behavior.

Once the installer has been executed, the sDAS metadata database will be deployed, and (optionally) the REST API deployed.

pydas's People

Contributors

bvanfleet avatar

Watchers

 avatar

pydas's Issues

Integrate Dependency Injector library

Purpose:
This platform is growing in complexity and already utilized a handful of factory objects. However, there's some awkwardness with how we use or pass the factory objects around. I'd like to integrate dependency injection to help with automating/managing this better.

Modularize security functions

The app is currently tightly coupled with Google Firebase Auth for user access. I don't like this coupling (what if firebase auth goes away). I'd like to migrate the authentication of our application to a pydas_auth module that can allow for the user to select and configure the auth on their own. We can utilize the before_request function to then engage with the auth module and validate user rights.

Reduce all unnecessary sub-packages into modules

This is dependent on #13.

Once all extensions have been removed, many sub-packages may be reduced down into modules. An example of this could be the transformers package. Once the IEX extensions have been removed, they can be reduced down into a transformers module with the Transformer class and a couple of utility functions.

There are others that may not be reducible yet, such as the formatters. However, this will be a case-by-case determination.

This will result in breaking changes to the API, as some code may rely on the sub-package architectures.

Create DB Context Manager

Reading through SQLAlchemy session documentation it was suggested to keep sessions external and separate. It was also suggested that the sessions are carefully transacted (commit on success and rollback on exception). Currently, pyDAS does neither of these.

  • Investigate using dependency-injector to inject session contexts
  • Create a context manager as shown via example in the article linked above

Doing both of these should help with better session management and handling errors.

Migrate extensions to another repository

This codebase is cluttered with extensions that are not generally useful. This causes bloat in the codebase and bloat in the dependencies. To reduce this bloat, and move this project towards a more generally useful state, these extensions should be moved.

The eventual landing place will be the pyDAS-Extensions repository.

Rename all abstract classes to remove 'Base' prefix

This is dependant on #13.

Once we've removed all extensions from the codebase, we'll further clean the codebase by removing the Base prefix on all abstract classes. This will be a breaking change that affects all extensions.

Update Installation Docs

The install docs haven't been updated in a long time, they need to be updated to address changes to the new pydas.yaml config, pending updates to the installer, and how to properly install the pyDAS system (via pip) with required extras.

New database deployments incorrectly configure key constraints

For new database installations, the foreign key constraints on the optionsbase table is set up wrong. It sets up two FK constraints, and sets UPDATE and DELETE events to restrict instead of cascade. This is a problem because the constraints conflict with the companyfeaturebase table. If there's a deletion, then the user will get a key violation error.

We need to properly configure the foreign key on optionbase to map to both columns, and cascade delete/update actions.

Fix pyDAS Installation module to properly configure server

The pyDAS.install module was created before we consolidated our configuration into one YAML. It also lacked functionality into creating a uWSGI configuration script. The former issue causes the GitHub workflow step to run the install script to fail, and has subsequently been disabled.

We should fix the script to:

  • Use the consolidated configuration script
  • (Optional) Support manually configuring the script via CLI arguments.
  • Configure the uWSGI config options (consider either creating a new INI file, or adding a uwsgi section to our pydas.yaml)
  • Deploy a new database (especially for SQLite) if required.

Change option model to support global overrides

What are you trying to do?

Currently, the option model is only supported for feature. However, this quickly becomes a pain when adding the same option to many features on a company. I'd like to change this to support company-wide or feature-wide overrides (such as adding a range modifier on all features mapped to a company).

There's a couple of options that thought about:

  1. Create additional tables for feature options and company options.
  2. Alter the existing option table to add a discriminator column to specify what type of option:
    a. company (global)
    b. feature (global)
    c. company-feature

I would prefer the latter option, as it's a smaller change, and SQL Alchemy supports model inheritance using a discriminator. The following image shows what this setup might look like:

image

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.