thegraydot / gcd_docker Goto Github PK

• auto create publishers dump with correct name (e.g., _DATE)
• Auto determine TGD output dump name (e.g., _DATE)

Potential automation:

• Load up environment
• Import dump
• Dump publishers
• Perform migration

I need the issue published date property to be a datetime type to ease filtering and processing of date related queries. Write a post processing script to be run after data insertion to convert varchar dates to datetime type.

It will be useful if some of the code is shared between gcd and tgd databases. So moving the python app code to a shared folder and mounting seems sensible.

Goal:

• move app to project root
• also move model into this folder
• come up with a suitable folder/app name

To make modifications to the original data, it makes sense to store the modified data in this repo. However, the project needs to be updated to ensure that the original GCD license is adhered to.

Goal:

• add cc4 license, which matches GCD license
• add info about contributing to GCD
• add link to GCD where needed

Add migration section to tgd section
Include schema

Goal:

• input is datetime object of last update
• method should:
• check if issue is newly created after last update
• check if issue has been modified after last update
• performance is important for this

Goal:

• issue, series, publisher models
• expand out data types to all selected columns needed

Full:

• remove date check
• remove print statements

Partial:

• insert for newly created
• patch for recently modified

I don't really use the publisher table apart from mapping the ID to name. To save a select query, dump a JSON file with ID to name mapping.

Goal:

• script to dump ID to name JSON file
• save json file in repo
• remove publisher select queries from scripts
• replace with ID lookup

When using the published_date conversion script (varchar yo datetime) the processing time is huge. A potential solution is to dump an sql file of sql statements. Future updates to this file could use the modified datetime value to only process specific issues, thus saving time.

Goal:

• implement first pass at sql output
• include table generation or column addition
• include new datetime value for published date

Only for issue, series and publisher tables for the id column

Goal:

• two folders in root directory for each DB
• shared comics py module and classes
• update makefile where needed

Start scaffolding new TGD DB with:

• docker compose solution for DB and app

Potential ideas:

• cbdb (see bee dee bee)

Where to change:

• repo name
• python project name

Goal:

• update docker compose file
• create python container
• create mysql container
• load data into mysql
• update makefile
• write python test script

This should be very simple dev stack, to mess around with. Prod stack to be implemented in other project.

Goal:

• script to run series of queries and time results
• test queries without indexing
• test queries with indexing
• check caching performance

Goal:

• check last updated or if newly created
• extract date
• auto convert to datetime
• populate in JSON, or sqlite
• use issue ID as key