:bulb: [Feature] Make bind-mount locations configurable about birdhouse-deploy HOT 3 CLOSED

Thanks for submitting an issue.
Please make sure to provide enough details for us to be able to replicate your issue or understand your question.

from birdhouse-deploy.

This is slightly more tricky (or has more implications) than it looks.

In order to manage different user-workspaces and file access permissions by various services using different interfaces, some file/dir hardlink special logic is being developed by @ChaamC to be used with https://github.com/ouranosinc/cowbird that much sync corresponding resources shared between those services. Because of these hard-links, the directories must reside on the same mount drive location to support resolution within docker services. Allowing custom location overrides by users that are not accustomed to the full stack could suddenly grant access to all user-workspace private files without realizing.

Similarly, because multiple services with different APIs can share access to those files (notably items under wps_outputs and under THREDDS data stores), the specific path under which they are mounted can impact the operability of those services, not to mention that some services can have a tendency to take ownership of change permissions of some of these files. In the case of THREDDS, the path is critical, as it must be replicated inside its own configuration to properly resolve and construct the URLs of the nested file hierarchy and their various data-conversion handlers.

In the case of Weaver, the mount location must match exactly between the host side and the mounted path inside the docker image. This is because Weaver CWL execution commands are published within its docker image, but the docker referenced in the Application Package that is executed by a subscribed weaver-worker is started as a sibling container on the host. This is done to allow minimal scaling and avoid docker-in-docker issues (security access elevation and such). However, this current restriction makes it such that these containers must use ${WEAVER_WPS_OUTPUTS_DIR}:${WEAVER_WPS_OUTPUTS_DIR} mappings explicitly, and allowing more versatile customization of WEAVER_WPS_OUTPUTS_DIR (although technically still possible/override-able), could lead to security concerns or simply break entirely Weaver's processes if done by users unaware of this logic.

For cases such as ${DATA_PERSIST_ROOT}/magpie_persist, this is less critical because the directory is intended to be used only by Magpie's PostgreSQL service. However, this is already customizable using MAGPIE_PERSIST_DIR override, so it could already be stored anywhere outside DATA_PERSIST_ROOT (or even another mount point).

from birdhouse-deploy.

@fmigneault

Thanks for the insight into the details of this:

some file/dir hardlink special logic is being developed by @ChaamC to be used with https://github.com/ouranosinc/cowbird

Ok good to know, let me look into this a bit more to make sure I understand what is happening before I do anything else

In the case of Weaver, the mount location must match exactly between the host side and the mounted path inside the docker image

This bit I was aware of, and I believe its the same for the jupyter user data directories that get mounted to the jupyter containers in a similar way. I'll make sure not to affect those.

from birdhouse-deploy.