This repository contains nginx and static file setup for:
- bmeg public site https://bmeg.io
- bmeg data directory https://bmeg.io/bmeg-data
- secure access to grip server https://bmeg.io/analyze/access
- secure access to recount.bio data https://recount.bio
Additionally, we maintain nginx configurations for:
- jupyterhub - for analysis https://bmeg-jupyter.ddns.net
- cadvisor - for devops statistics https://bmeg-jupyter.ddns.net/cadvisor
- gen3 - example arbitrary other content https://gen3-ohsu.ddns.net
Each site (domain) is backed by a letsencrypt certbot certificate renewal process.
- databases and docker workareas live on : /mnt/data1/...
- repositories, etl and configurations live on : /mnt/data2/...
The nginx service leverages two key technologies:
-
certbot and a shared volume
data/- provides signed valid certs for multiple sites
-
- implements google oauth, implemented for external connections to the
gripserver - see
secrets/nginx.envfor parameters, matched with 'bmeg' gcp
- implements google oauth, implemented for external connections to the
-
The
run.shscript launches nginx within its container- The script will reload the config and certs every 6 hours.
- The
DEBUGenv variable controls parameter logging
nginx ├── Dockerfile ├── etc │ └── nginx │ ├── conf.d │ │ ├── default.conf │ │ └── protected.conf │ ├── grip.conf │ ├── http.conf.d │ │ └── env.conf │ ├── lua │ │ └── nginx-google-oauth │ ├── nginx.conf │ ├── protected.conf │ ├── sites-available │ │ └── default │ └── sites-enabled │ ├── bmeg.io │ ├── bmegio.ohsu.edu │ ├── bmeg-jupyter.ddns.net │ └── gen3-ohsu.ddns.net └── usr └── share └── nginx ├── default └── gen3-ohsu.ddns.net
Each site has a file in etc/nginx/sites-enabled. Ensure that the site is included in etc/nginx/
Content is mapped to usr/share/nginx/.... The bmeg-data/ directory is mounted to /usr/share/nginx/bmeg.io.data
The bmeg-site hugo single page app is generated into nginx/bmeg-site/bmeg.io/public and mounted to /usr/share/nginx/bmeg.io
The makesite.sh script sets the base site in the hugo config and runs the docker image site-builder
├── makesite.sh
├── site-builder
│ └── Dockerfile
To update the content:
DEPLOYMENT_DIR=/mnt/data2/bmeg/deployment
cd $DEPLOYMENT_DIR
alias dc='docker-compose -f docker-compose.yml -f compose-services/docker-compose.yml -f compose-services/onprem/docker-compose.yml'
cd nginx
./makesite.sh
cd ..
$dc build nginx
$dc stop nginx
$dc rm -f nginx
$dc up -d
The init-letsencrypt.sh cli tool :
- creates temporary self signed certs
- requests the letsencrypt certificate
- maintains shared convention to
data/with thecertbotandnginxservices. See volume mapping docker-compose.yml - certbot will check renewal for each domain cert every 12 hours
init-letsencrypt.sh // cli tool
data // letsencrypt data
└── certbot
├── conf
│ ├── accounts
│ │ └── acme-v02.api.letsencrypt.org
│ ├── archive
│ │ ├── bmegio.ohsu.edu
│ │ ├── bmeg-jupyter.ddns.net
│ │ └── gen3-ohsu.ddns.net
│ ├── csr
│ │ ├── 0000_csr-certbot.pem
│ │ └── ....
│ ├── keys
│ │ ├── 0000_key-certbot.pem
│ │ └── ...
│ ├── live
│ │ ├── bmegio.ohsu.edu
│ │ ├── bmeg-jupyter.ddns.net
│ │ ├── gen3-ohsu.ddns.net
│ │ └── README
│ ├── options-ssl-nginx.conf
│ ├── renewal
│ │ ├── bmegio.ohsu.edu.conf
│ │ ├── bmeg-jupyter.ddns.net.conf
│ │ └── gen3-ohsu.ddns.net.conf
│ ├── renewal-hooks
│ │ ├── deploy
│ │ ├── post
│ │ └── pre
│ └── ssl-dhparams.pem
└── www
To create a new host, in this example bmeg.io
First, point the DNS at this host. Once that is verified, ask letsencrypt for new certs.
sudo ./init-letsencrypt.sh bmeg.io
This should respond with a "congratulations" message.
bash convenience
# stop, rebuild, start and montitor
alias dc='docker-compose '
rs() {
dc stop $1; dc rm -f $1; dc build $1; dc up -d $1; dc logs -f $1
}
# in the nginx dir, build the new site.
cd nginx/
export SERVER_NAME=bmeg.io
export BASE_URL=https://bmeg.io
export BMEG_SITE_BRANCH=master
./makesite.sh
# restart nginx
cd ..
rs nginx
Plain vanilla mongo installation. Data maintained on /mnt/data1/bmeg/mongo-data
See secrets/grip_config.yml for configuration.
It's default schema is mapped from bmeg-site/static/meta/schema.json
The nginx service leverages two key technologies:
- dockerspawner==0.11.1
- see
secrets/userlistfor whitelist of users. - see
secrets/jupyterhub_config.pyfor parameters:- each user's notebook is launched with cpu_limit = 1, mem_limit = '10G' and will be paused after 1 hr of inactivity
- see
- oauthenticator==0.8.2
- see
secrets/oauth.envfor parameters, matched with 'bmeg-jupyter' gcp
- see
The jupyterlab image is used for notebooks. Build via dc build jupyterlab
Please Note: While hub data is maintained in /mnt/data1/bmeg/jupyter-data, the lifecycle of data has NOT been tested. Please back up your work.
├── jupyterhub
│ └── Dockerfile
├── jupyterlab
│ └── Dockerfile
├── secrets
│ ├── jupyter.env
│ ├── jupyterhub_config.py
Note: exec into the grip server to drop & create graph.
The mount bmeg-etl directory, is mounted on /bmeg-etl.
├── etl // python 3.7 utility
│ ├── Dockerfile
│ ├── docker-start.sh
│ └── README.md
A streaming ui is provided at https://bmeg-jupyter.ddns.net/cadvisor.
Notes:
- the data is not persisted
- you need to log into jupyterhub before accessing
Notes: to sync from internal services -
This crontab will sync daily at 00:00
0 0 * * * rsync --checksum <source-directory>/*.* <user@host>:<target-directory>
- setup
# see commons.bmeg.io
cd xxx/deployment
git status
# On branch commons.bmeg.io
git clone [email protected]:ohsu-comp-bio/compose-services.git
checkout compose-services
git checkout commons.bmeg.io
cd ..
alias dc='docker-compose -f docker-compose.yml -f compose-services/docker-compose.yml -f compose-services/onprem/docker-compose.yml'
# build local services
cd compose-services/onprem/
make
>>>
$ docker images | grep onprem
onprem/tube latest b455e7513b9f 9 minutes ago 2.11GB
onprem/data-portal latest 552a3d09ca9c 11 minutes ago 1.22GB
onprem/hatchery latest b88a0a5e39da 15 minutes ago 954MB
onprem/s3indexer latest 1bde3a35f4cf 16 minutes ago 1.16GB
onprem/indexs3client latest 4658ee3c14dc 17 minutes ago 10.5MB
onprem/fence latest 857b63840d4a 18 minutes ago 670MB
# customize gen3 secrets
cd Secrets
grep commons.bmeg.io *.*
fence-config.yaml:BASE_URL: 'https://commons.bmeg.io/user'
peregrine_creds.json: "hostname": "commons.bmeg.io"
sheepdog_creds.json: "hostname": "commons.bmeg.io",
cd ..
# prep for elastic
sysctl -w vm.max_map_count=262144
# install yp needed by creds_setup.sh
# see https://mikefarah.gitbook.io/yq/#on-ubuntu-16-04-or-higher-from-debian-package
# generate local certs
$ bash ./creds_setup.sh commons.bmeg.io
>>>
The Subject's Distinguished Name is as follows
...
commonName :ASN.1 12:'<hostname>'
...
Write out database with 1 new entries
Data Base Updated
# configure keys
check `Secrets/fenceJwtKeys/<now() timestamp>/` fence will use most recent timestamp
# copy fence's public key to elastic Secrets/elasticsearch-open-distro/config.yml
# see jwt_auth_domain/signing_key
# see voucher-service in compose-services/onprem/docker-compose.yml
update compose-services/Secrets/.env
# slow start order (use this if you are having problems), otherwise skip to next
dc up -d postgres
dc up -d fence-service
dc up -d arborist-service
dc exec fence-service fence-create migrate
dc exec fence-service fence-create sync --arborist http://arborist-service --yaml user.yaml
# bring up gen3
dc up -d
# apply security config to elastic
### note: compare against top of elastic log file
dc exec esproxy-service \
"/usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh" -cd "/usr/share/elasticsearch/plugins/opendistro_security/securityconfig" -icl -key "/usr/share/elasticsearch/config/kirk-key.pem" -cert "/usr/share/elasticsearch/config/kirk.pem" -cacert "/usr/share/elasticsearch/config/root-ca.pem" -nhnv
### should produce
>>>
...
Done with success
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent