Run N multiple parallel processes (fork) using queues as communication method.
Lightyear is flexible: you can create N kinds of different processes, launch N instances for each of them, and create N queues for communication.
Lightyear allows you to add specific command line options for every pipeline.
$ python main.py --help
usage: lightyear [-h] [-l {error,warning,info,debug}] [-d] {akeneo,brandquad,retailnext} ...
Pipeline runner
positional arguments:
{akeneo,brandquad,retailnext}
optional arguments:
-h, --help show this help message and exit
-l {error,warning,info,debug}, --log-level {error,warning,info,debug}
set the logging level (default: info)
-d, --dummy avoid bigquery insertion
- Python 3.8.5
python -m venv env
source env/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
source env/bin/activate
export GOOGLE_APPLICATION_CREDENTIALS={path-to-your-json-credentials-file}
docker-compose build
docker-compose run lightyear
docker exec <container-id> /bin/bash -c 'ps fax'