rdmorganiser / rdmo-docker-compose Goto Github PK

I am currently trying to deploy rdmo via rdmo-docker-compose. I have successfully managed to start the app via docker, however it is not usable when trying to deploy it under a subroute like example.com/rdmo.

Here is my nginx snippet for forwarding rdmo:

location /rdmo/ {
  proxy_pass http://localhost:8484/;
}

When accessing the app via example.com/rdmo/ all relative links are not generated correctly and assets are not beingloaded. As I have seen in rdmo, there is the possibility to configure the BASE_URL in config/settings/local.py.

Could you make BASE_URL configurable in the variables.env in rdmo-docker-compose? That would be a great help.

Hi @triole,

this is an issue that aims at discussing potential setups in order to further ease the use of RDMO running inside docker. Although the current implementation using local images with the help of a makefile is working very well, I think it could be further improved towards implementing docker best practices.

As a user, I would like to have a single docker-compose.yml file that contains, all configuration of the docker setup. Furthermore, as in other projects it would be great if the images are centrally provided such as in the Docker Hub so that these need to be pulled only. This setup dramatically eases upgrades by pulling and starting up new images.

I see the advantage of building the images locally in order to easily create and use the current user ID to run processes inside the docker container, especially, because of the use of the docker volumes. But is it really needed to have the VE and the RDMO-App as a docker volume? Instead, the RDMO configuration file local.py as well as a rdmo theme might be integrated using docker volumes. This allows for central provision of the docker images.
For rewriting the user ID when starting the container, a similar approach than in the Jupyter Docker images might be used: https://github.com/jupyter/docker-stacks/blob/fbd5c88596df5139e16aa3b13fbb6dddc98c66be/base-notebook/start.sh#L84-L91 Or user namespaces might be used: https://docs.docker.com/engine/security/userns-remap/

Looking forward to your ideas on this setup.

Best regards
Max

Dear rdmo developers,
I appreciate your effort to create a docker containers to make my live easier. However I am experiencing some issues. Digging in the commands necessary to build the container I have run into makemigrations is not running as expected (python manage.py makemigrations). I get the following issue:

File "manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/vol/ve/lib/python3.7/site-packages/django/core/management/init.py", line 381, in execute_from_command_line
utility.execute()
File "/vol/ve/lib/python3.7/site-packages/django/core/management/init.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/vol/ve/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "/vol/ve/lib/python3.7/site-packages/django/core/management/base.py", line 361, in execute
self.check()
File "/vol/ve/lib/python3.7/site-packages/django/core/management/base.py", line 390, in check
include_deployment_checks=include_deployment_checks,
File "/vol/ve/lib/python3.7/site-packages/django/core/management/base.py", line 377, in _run_checks
return checks.run_checks(**kwargs)
File "/vol/ve/lib/python3.7/site-packages/django/core/checks/registry.py", line 72, in run_checks
new_errors = check(app_configs=app_configs)
File "/vol/ve/lib/python3.7/site-packages/django/core/checks/urls.py", line 40, in check_url_namespaces_unique
all_namespaces = _load_all_namespaces(resolver)
File "/vol/ve/lib/python3.7/site-packages/django/core/checks/urls.py", line 57, in _load_all_namespaces
url_patterns = getattr(resolver, 'url_patterns', [])
File "/vol/ve/lib/python3.7/site-packages/django/utils/functional.py", line 80, in get
res = instance.dict[self.name] = self.func(instance)
File "/vol/ve/lib/python3.7/site-packages/django/urls/resolvers.py", line 579, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
File "/vol/ve/lib/python3.7/site-packages/django/utils/functional.py", line 80, in get
res = instance.dict[self.name] = self.func(instance)
File "/vol/ve/lib/python3.7/site-packages/django/urls/resolvers.py", line 572, in urlconf_module
return import_module(self.urlconf_name)
File "/vol/ve/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "/vol/rdmo-app/config/urls.py", line 11, in
path('api/v1/', include('rdmo.core.urls.v1')),
File "/vol/ve/lib/python3.7/site-packages/django/urls/conf.py", line 34, in include
urlconf_module = import_module(urlconf_module)
File "/vol/ve/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "/vol/ve/lib/python3.7/site-packages/rdmo/core/urls/v1.py", line 11, in
path('conditions/', include('rdmo.conditions.urls.v1')),
File "/vol/ve/lib/python3.7/site-packages/django/urls/conf.py", line 34, in include
urlconf_module = import_module(urlconf_module)
File "/vol/ve/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "/vol/ve/lib/python3.7/site-packages/rdmo/conditions/urls/v1.py", line 5, in
from ..viewsets import ConditionViewSet, RelationViewSet
File "/vol/ve/lib/python3.7/site-packages/rdmo/conditions/viewsets.py", line 2, in
from rest_framework.decorators import list_route
ImportError: cannot import name 'list_route' from 'rest_framework.decorators' (/vol/ve/lib/python3.7/site-packages/rest_framework/decorators.py)

Even though it worked smoothly a few weeks ago, something seems to have been changed recently. Any suggestion?
Thanks!

Here the list of installed packages.

certifi 2019.6.16
chardet 3.0.4
coreapi 2.3.3
coreschema 0.0.4
coverage 4.5.3
defusedxml 0.6.0
Django 2.2.3
django-allauth 0.39.1
django-appconf 1.0.3
django-compressor 2.3
django-extensions 2.2.1
django-filter 2.2.0
django-js-asset 1.2.2
django-libsass 0.7
django-mptt 0.10.0
django-rest-swagger 2.2.0
django-settings-export 1.2.1
django-test-generator 0.6.0
django-widget-tweaks 1.4.5
djangorestframework 3.10.1
drf-extensions 0.5.0
idna 2.8
iso8601 0.1.12
itypes 1.1.0
Jinja2 2.10.1
jsonfield 2.0.2
libsass 0.19.2
Markdown 3.1.1
MarkupSafe 1.1.1
oauthlib 3.0.2
openapi-codec 1.3.2
pip 19.2.1
psycopg2 2.8.3
pypandoc 1.4
python3-openid 3.1.0
pytz 2019.1
rcssmin 1.0.6
rdmo 0.14.4
requests 2.22.0
requests-oauthlib 1.2.0
rjsmin 1.1.0
rules 2.0.1
setuptools 41.0.1
simplejson 3.16.0
six 1.12.0
sqlparse 0.3.0
uritemplate 3.0.0
urllib3 1.25.3
wheel 0.33.4
WARNING: The directory '/home/rdmo/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

When deploying rdmo with a subdomain, it seems that not all paths are being rewritten correctly.

rdmo/vol/rdmo-app/config/settings/local.py
--
BASE_URL = '/rdmo'

For instance, when visiting the index page, the "Login"/"Anmelden" link in the navbar correctly routes to /rdmo/account/login.

However, other routes (for changing the locale) or routes within forms are not being prepended with the BASE_URL. For example, the link to the registration form or the POST-URL for the login form does not re-route correctly.

Although the BASE_URL is configured, the URLs point to:

• /account/login instead of /rdmo/account/login
• /account/signup instead of /rdmo/account/signup
• /i18n/en instead of /rdmo/i18n/en
  etc.

This routing issue makes the app unusable for users when trying to deploy under a server subdirectory path (e.g. example.com/rdmo).

My nginx-snippet on the host OS.

location / {
  proxy_pass http://localhost:8484/;
}

See also here: psycopg/psycopg2#1293

Django 2.2 is incompatible with psycopg >= 2.9 due to some changes with regard to handling timezone infos. This leads to RDMO being unable to login users (Error 500 in Browser, no entry in logs) or importing catalogs (Stacktrace with AssertionError: database connection isn't set to UTC)

This can be fixed by installing psycopg-2.8.6 by changing the line pip install psycopg2 in rdmo/rootfs/opt/install-rdmo.sh to pip install psycopg2==2.8.6.

Sorry to repeat the already closed issue #9 this but this is exactly what happens in my docker-compose installation:

• Docker version (docker-compose)
• Installation in subdirectory <MY_SERVER>/rdmo/
• BASE_PATH in variables.env set to /rdmo
• Apache Docker proxy directive for routing the URLs to the rdc-nginx container

The home view comes up smoothly, except for 1 problem: Most of the contained links (not all!) do not have the /rdmo/ path included. The link to the standalone sign-in form works; not working are the form action and the languag switcher and the logo link. They all do not have the BASE_PATH subdir included.

My proxy directive:

ProxyPass /rdmo http://127.0.0.1:8484/
ProxyPassReverse /rdmo http://127.0.0.1:8484/

I noticed the message "NOTE: BASE_URL does not work, will be fixed later" within the variables.env file. What is the current status for this? Might this be the reason for the problem here?

So where to go from here? Any suggestions are highly appreciated. Thanks.

Hi,
are the first two steps in the CI needed?
The Ubuntu image comes with docker and docker-compose pre-installed. Do you need a special version somehow?

For users testing the latest development version such as for RDMO v1.6 it would be very helpful to have a corresponding dev branch for this docker-compose setup that installs the development branch of RDMO including the corresponding dependencies. We would be happy to contribute these adjustments.

Furthermore, the following might be adjusted:

• nginx: docker-master might be added an apt upgrade -y as in the postgres specification; Maybe it is also worth to switch to the nginx docker-images
• add a re-build command to the makefile which pulls the latest docker-images used as a basis and re-builds the custom images

I appreciate the issue "ImportError: cannot import name 'list_route' from rest_framework.decorators" has been solved 👍. However, I am not able to make it run yet. It seems that the module "django" does not exist in the virtual enviroment.

rdc-apache | [Mon Jul 29 16:20:45.026356 2019] [wsgi:error] [pid 8:tid 140136115406592] [remote 127.0.0.1:47194] mod_wsgi (pid=8): Failed to exec Python script file '/vol/rdmo-app/config/wsgi.py'.
rdc-apache | [Mon Jul 29 16:20:45.026391 2019] [wsgi:error] [pid 8:tid 140136115406592] [remote 127.0.0.1:47194] mod_wsgi (pid=8): Exception occurred processing WSGI script '/vol/rdmo-app/config/wsgi.py'.
rdc-apache | [Mon Jul 29 16:20:45.026455 2019] [wsgi:error] [pid 8:tid 140136115406592] [remote 127.0.0.1:47194] Traceback (most recent call last):
rdc-apache | [Mon Jul 29 16:20:45.026470 2019] [wsgi:error] [pid 8:tid 140136115406592] [remote 127.0.0.1:47194] File "/vol/rdmo-app/config/wsgi.py", line 3, in
rdc-apache | [Mon Jul 29 16:20:45.026473 2019] [wsgi:error] [pid 8:tid 140136115406592] [remote 127.0.0.1:47194] from django.core.wsgi import get_wsgi_application
rdc-apache | [Mon Jul 29 16:20:45.026483 2019] [wsgi:error] [pid 8:tid 140136115406592] [remote 127.0.0.1:47194] ModuleNotFoundError: No module named 'django'

Hi! After fixing the issue #21 the postgres container fails with

initdb: error: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted

After removing the ARG UID, RUN chown -R ${UID} /var/lib/postgresql and USER ${UID} statements in the posgres Dockerfile and commenting out the args: UID: <LOCAL_UID> statement in the dc_master.yaml the postgres container starts.

As far as I know, the postgres image runs as the postgres user per default, so changing the user and permissions seems unnecessary here. If required, maybe run an Ubuntu image and install postgres during build? might also fix issue #21.

Hello
while trying to install and run the docker i realised that after doing a make fromscratch the vol/rdmo-app folder is created but is empty, this file remains ide if you leave in the conf.toml commented RDMO_INSTALL_URL=git+https://github.com/rdmorganiser/[email protected] or if you uncomment this line (i changed the version 1.8.2 by default by the latest version 2.1.2)
Is this empty folder normal? Should I manually make a git clone of rdmo-app in the vol folder?

If I understand correctly, this folder should contain the rdmo-app repository (https://github.com/rdmorganiser/rdmo-app) and therefore the manage.py file which is called via different aliases when you put it in the bashrc and which is also used to install rdmo and launch the server?

Furthermore, I'm wondering, in the event that this problem is resolved, how is the RDMO instance supposed to work? Is it supposed to start at the end of the make? or do we have to enter the container and launch it manually?

Thank you in advance
Gautier

Hi,
after cloning git clone https://github.com/rdmorganiser/rdmo-docker-compose.git test into test and issuing make in this dir I encountered the following error:

 => ERROR [rdmo  3/17] RUN rm "$(find /usr/lib/python?.??/ -regex ".*EXTERNALLY-MANAGED.*")"                   0.5s
------
 > [rdmo  3/17] RUN rm "$(find /usr/lib/python?.??/ -regex ".*EXTERNALLY-MANAGED.*")":
0.398 find: ‘/usr/lib/python?.??/’: No such file or directory
0.399 rm: cannot remove '': No such file or directory
------
failed to solve: process "/bin/sh -c rm \"$(find /usr/lib/python?.??/ -regex \".*EXTERNALLY-MANAGED.*\")\"" did not complete successfully: exit code: 1
make: *** [makefile:24: run_build] Error 17

It seems the current base image does not cotain Externally managed python libs?!
Commenting the offending RUN in the dockerfile

diff --git a/docker/rdmo/Dockerfile b/docker/rdmo/Dockerfile
index 3885915..ff67656 100644
--- a/docker/rdmo/Dockerfile
+++ b/docker/rdmo/Dockerfile
@@ -9,7 +9,7 @@ ENV LC_ALL C.UTF-8

 RUN apt update -y && apt upgrade -y && apt install -y \
   python3 python3-dev python3-pip locales
-RUN rm "$(find /usr/lib/python?.??/ -regex ".*EXTERNALLY-MANAGED.*")"
+#RUN rm "$(find /usr/lib/python?.??/ -regex ".*EXTERNALLY-MANAGED.*")"
 RUN ln -s /usr/bin/python3 /usr/bin/python
 RUN python3 -m pip install --upgrade pip

successfully builds the containers, but the won't start:
The postgres container shows:

rdc-postgres exited with code 1
rdc-postgres  | The files belonging to this database system will be owned by user "postgres".
rdc-postgres  | This user must also own the server process.
rdc-postgres  |
rdc-postgres  | initdb: error: invalid locale settings; check LANG and LC_* environment variables
rdc-postgres  | The files belonging to this database system will be owned by user "postgres".
rdc-postgres  | This user must also own the server process.

It seems that the en_US.utf locale is not installed in the container.
This can be fixed by adding RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen in the Dockerfile of the postgres container:

diff --git a/docker/postgres/Dockerfile b/docker/postgres/Dockerfile
index c5f6726..fa7e3b2 100644
--- a/docker/postgres/Dockerfile
+++ b/docker/postgres/Dockerfile
@@ -6,6 +6,10 @@ ENV PATH="${PATH}:/opt:${HOME}/sh"
 RUN apt update && apt upgrade -y && apt install -y postgresql-client vim
 RUN rm -rf /var/lib/apt/lists/*

+# Set the locale
+RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && \
+    locale-gen

After cleaning up, e.g.
make run_remove && rm -fr vol && docker volume rm rdcpostgres rdcvol
the rdc-caddy and rdc-postgres container start.
However the rdc-rdmo container complains:

rdc-rdmo      | python: can't open file '/vol/rdmo-app/manage.py': [Errno 2] No such file or directory
rdc-rdmo      | python: can't open file '/vol/rdmo-app/manage.py': [Errno 2] No such file or directory
rdc-rdmo      | python: can't open file '/vol/rdmo-app/manage.py': [Errno 2] No such file or directory
rdc-rdmo      | python: can't open file '/vol/rdmo-app/manage.py': [Errno 2] No such file or directory
rdc-rdmo      | Run gunicorn
rdc-rdmo      | gunicorn failed to start, retry in 60s

This is due to the fact, that sh/prepare.sh creates vol/rdmo-app. If this exists, the rdmo initialisation is skipped.
A fix is:

diff --git a/sh/prepare.sh b/sh/prepare.sh
index 6edbf5b..9ca0db4 100755
--- a/sh/prepare.sh
+++ b/sh/prepare.sh
@@ -61,7 +61,8 @@ cat ${dc_master} | envsubst >${dc_temp}

 mkdir -p ${VOLDIR}/log
 mkdir -p ${VOLDIR}/postgres
-mkdir -p ${VOLDIR}/rdmo-app
+# rdmo-app will be created by the rdc-rdmo container
+#mkdir -p ${VOLDIR}/rdmo-app

 docker_compose_command="docker compose -p ${GLOBAL_PREFIX}"
 if ! groups | grep docker >/dev/null 2>&1; then

With these fixes a fresh installation can be done.

I'm running into problems using the current version for a 2nd docker instance on the same machine. Another older version of the docker-compose instance on the same machine works fine.

1. Edited the variables.local
2. Run make

make then stops with the following message:

Configuring postgresql-common
-----------------------------

createcluster.conf: A new version (/tmp/postgresql-common.TeWQyR) of
configuration file /etc/postgresql-common/createcluster.conf is available, but
the version installed currently has been locally modified.

  1. install the package maintainer's version
  2. keep the local version currently installed
  3. show the differences between the versions
  4. show a side-by-side difference between the versions
  5. start a new shell to examine the situation
What do you want to do about modified configuration file createcluster.conf?

Since its running in non-interactive mode, none of the choices gets submitted. I then followed https://stackoverflow.com/a/72273217/6948765 and added DEBIAN_FRONTEND=noninteractive as env variable. This fixes the config prompt.

The postgres container gets built, but doesn't start properly. I dont know if it's related at all. docker logs shows:

initdb: error: invalid locale settings; check LANG and LC_* environment variables
chmod: changing permissions of '/var/lib/postgresql/data': Operation not permitted
chmod: changing permissions of '/var/run/postgresql': Operation not permitted
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

And this is where I'm a bit lost. The created volume under /vol/ has the owner systemd-coredump but I cant make anything of it.

Any idea? Thanks!

Hello,

I'm encountering an issue while attempting to run the RDMO Docker. After copying docker/baseconf.toml as conf.toml, I try to build the container using the make command. However, I end up in an infinite loop with the following message:

rdc-rdmo      | Run gunicorn
rdc-rdmo      | gunicorn failed to start, retry in 60s

I'm having trouble pinpointing the source of the problem. Any insights or suggestions would be greatly appreciated.

Thank you in advance!
Gautier

Thanks for providing this docker compose set up. Unfortunately I am experiencing some problems. After cloning the repo and changing port and the allowed host (adding the external IP address of the server) in the variables.local, the following error occured after running make:

rdc-apache  | [Thu Jul 18 15:06:20.038594 2019] [wsgi:error] [pid 8:tid 140556231497472] [remote 127.0.0.1:38496] mod_wsgi (pid=8): Failed to exec Python script file '/vol/rdmo-app/config/wsgi.py'.
rdc-apache  | [Thu Jul 18 15:06:20.038692 2019] [wsgi:error] [pid 8:tid 140556231497472] [remote 127.0.0.1:38496] mod_wsgi (pid=8): Exception occurred processing WSGI script '/vol/rdmo-app/config/wsgi.py'.
rdc-apache  | [Thu Jul 18 15:06:20.038970 2019] [wsgi:error] [pid 8:tid 140556231497472] [remote 127.0.0.1:38496] Traceback (most recent call last):
rdc-apache  | [Thu Jul 18 15:06:20.039066 2019] [wsgi:error] [pid 8:tid 140556231497472] [remote 127.0.0.1:38496]   File "/vol/rdmo-app/config/wsgi.py", line 7, in <module>
rdc-apache  | [Thu Jul 18 15:06:20.039080 2019] [wsgi:error] [pid 8:tid 140556231497472] [remote 127.0.0.1:38496]     application = get_wsgi_application()
rdc-apache  | [Thu Jul 18 15:06:20.039099 2019] [wsgi:error] [pid 8:tid 140556231497472] [remote 127.0.0.1:38496]   File "/vol/ve/lib/python3.5/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
rdc-apache  | [Thu Jul 18 15:06:20.039110 2019] [wsgi:error] [pid 8:tid 140556231497472] [remote 127.0.0.1:38496]     django.setup(set_prefix=False)
rdc-apache  | [Thu Jul 18 15:06:20.039127 2019] [wsgi:error] [pid 8:tid 140556231497472] [remote 127.0.0.1:38496]   File "/vol/ve/lib/python3.5/site-packages/django/__init__.py", line 24, in setup
rdc-apache  | [Thu Jul 18 15:06:20.039137 2019] [wsgi:error] [pid 8:tid 140556231497472] [remote 127.0.0.1:38496]     apps.populate(settings.INSTALLED_APPS)
rdc-apache  | [Thu Jul 18 15:06:20.039153 2019] [wsgi:error] [pid 8:tid 140556231497472] [remote 127.0.0.1:38496]   File "/vol/ve/lib/python3.5/site-packages/django/apps/registry.py", line 83, in populate
rdc-apache  | [Thu Jul 18 15:06:20.039163 2019] [wsgi:error] [pid 8:tid 140556231497472] [remote 127.0.0.1:38496]     raise RuntimeError("populate() isn't reentrant")
rdc-apache  | [Thu Jul 18 15:06:20.039205 2019] [wsgi:error] [pid 8:tid 140556231497472] [remote 127.0.0.1:38496] RuntimeError: populate() isn't reentrant

• Host-OS: Debian 9.9
• Docker: 18.09.8

What might be the problem here?

First let me say thank you for making this nice piece of software and dockerizing it!

When trying to deploy RDMO in our environment we weren't able to access our Docker host via URL as the hostname was not made available to the DNS so we tried to access it via IP address. This led to HTTP 400 errors (Bad Request) as the IP was not listed under ALLOWED_HOSTS in template_local.py:

Hi,

in order clarify the usage conditions of the repository, a license would be very helpful.

Best regards

Hi! The rdmo container exits with /drun.sh: line 22: gunicorn: command not found. It seems like an issue with the virtual environment? I ll try to figure out how to fix it and will update here.