View Code? Open in Web Editor
NEW
Using Devcontainers to create a Apache Spark Devlopment Envrionment
Shell 0.60%
Dockerfile 2.59%
Python 96.81%
devcontainer_example's Introduction
- From vscode command palette (crtl+shirf+p) > Remote-Containers: Add Development Container Configuration File...
- Modify devcointainer.json to your liking. NB: docker-compose used.
- Example add extentions to the continer configuration
- Example run extra commands like installing requirements.txt for python app
- Example foward ports back to the host if your app for instance will launch a listening server
- From vscode command palette (crtl+shirf+p) > Remote-Containers: Rebuild and Reopen in Container
Package environment to ship to executors
- venv-pack --python-prefix "/opt/bitnami/python"
- --python-prefix indicates where to link the executable in the shipped env to the real executable on the executor
- set .env variable PYSPARK_PYTHON=./venv/bin/python (will use this environment in current working directory on executor)
devcontainer_example's People
Contributors
Watchers