magic-nexus / nis-backend Goto Github PK

The Docker image fails after loading . The problem seems to be with the loading of the with the FADN datasource. The file referenced (see bold in log below) does not seem to exist. There may be other datasources missing, but the container boot process does not continue to be able to find all of them.

wget -v http://ec.europa.eu/agriculture/rica/database/help/infometa.csv
--2021-12-03 10:09:55-- http://ec.europa.eu/agriculture/rica/database/help/infometa.csv
Resolving ec.europa.eu (ec.europa.eu)... 2a01:7080:14:100::666:30, 2a01:7080:24:100::666:30, 147.67.34.30, ...
Connecting to ec.europa.eu (ec.europa.eu)|2a01:7080:14:100::666:30|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://ec.europa.eu/agriculture/rica/database/help/infometa.csv [following]
--2021-12-03 10:09:55-- https://ec.europa.eu/agriculture/rica/database/help/infometa.csv
Connecting to ec.europa.eu (ec.europa.eu)|2a01:7080:14:100::666:30|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://ec.europa.eu/info/food-farming-fisheries [following]
--2021-12-03 10:09:56-- https://ec.europa.eu/info/food-farming-fisheries
Reusing existing connection to [ec.europa.eu]:443.
HTTP request sent, awaiting response... 200 OK
Length: 59596 (58K) [text/html]
Saving to: ‘infometa.csv’

infometa.csv 100%[========================================================================================================================================>] 58.20K 174KB/s in 0.3s

2021-12-03 10:09:57 (174 KB/s) - ‘infometa.csv’ saved [59596/59596]

while the file is saved, the content is of an html page resulting from the redirection,

Docker container startup log (trimmed to after the error).
[2021-12-03 18:06:19 +0000] [17] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 129, in init_process
self.load_wsgi()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
return self.load_wsgiapp()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 350, in import_app
import(module)
File "/app/nexinfosys/restful_service/service_main.py", line 135, in
nexinfosys.data_source_manager = register_external_datasources()
File "/app/nexinfosys/initialization.py", line 216, in register_external_datasources
data_engine=nexinfosys.data_engine))
File "/app/nexinfosys/ie_imports/data_sources/fadn.py", line 247, in init
self._measures = load_measures_dictionary(self._base_directory)
File "/app/nexinfosys/ie_imports/data_sources/fadn.py", line 146, in load_measures_dictionary
d[df.iloc[r, 0]] = (df.iloc[r, en_pos], formula)
File "/usr/local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1762, in getitem
return self._getitem_tuple(key)
File "/usr/local/lib/python3.7/site-packages/pandas/core/indexing.py", line 2067, in _getitem_tuple
self._has_valid_tuple(tup)
File "/usr/local/lib/python3.7/site-packages/pandas/core/indexing.py", line 703, in _has_valid_tuple
self._validate_key(k, i)
File "/usr/local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1994, in _validate_key
self._validate_integer(key, axis)
File "/usr/local/lib/python3.7/site-packages/pandas/core/indexing.py", line 2063, in _validate_integer
raise IndexError("single positional indexer is out-of-bounds")
IndexError: single positional indexer is out-of-bounds
[2021-12-03 18:06:19 +0000] [17] [INFO] Worker exiting (pid: 17)
Connecting to data server
USER: root
USER: root
download URL: http://ec.europa.eu/agriculture/rica/database/help/infometa.csv
file: /root/.local/share/nis-backend/data/fadn/measures.csv
[2021-12-03 18:06:19 +0000] [18] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 129, in init_process
self.load_wsgi()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
return self.load_wsgiapp()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 350, in import_app
import(module)
File "/app/nexinfosys/restful_service/service_main.py", line 135, in
nexinfosys.data_source_manager = register_external_datasources()
File "/app/nexinfosys/initialization.py", line 216, in register_external_datasources
data_engine=nexinfosys.data_engine))
File "/app/nexinfosys/ie_imports/data_sources/fadn.py", line 247, in init
self._measures = load_measures_dictionary(self._base_directory)
File "/app/nexinfosys/ie_imports/data_sources/fadn.py", line 146, in load_measures_dictionary
d[df.iloc[r, 0]] = (df.iloc[r, en_pos], formula)
File "/usr/local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1762, in getitem
return self._getitem_tuple(key)
File "/usr/local/lib/python3.7/site-packages/pandas/core/indexing.py", line 2067, in _getitem_tuple
self._has_valid_tuple(tup)
File "/usr/local/lib/python3.7/site-packages/pandas/core/indexing.py", line 703, in _has_valid_tuple
self._validate_key(k, i)
File "/usr/local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1994, in _validate_key
self._validate_integer(key, axis)
File "/usr/local/lib/python3.7/site-packages/pandas/core/indexing.py", line 2063, in _validate_integer
raise IndexError("single positional indexer is out-of-bounds")
IndexError: single positional indexer is out-of-bounds
[2021-12-03 18:06:19 +0000] [18] [INFO] Worker exiting (pid: 18)
Connecting to data server
USER: root
USER: root
download URL: http://ec.europa.eu/agriculture/rica/database/help/infometa.csv
file: /root/.local/share/nis-backend/data/fadn/measures.csv
[2021-12-03 18:06:19 +0000] [9] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 129, in init_process
self.load_wsgi()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
return self.load_wsgiapp()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 350, in import_app
import(module)
File "/app/nexinfosys/restful_service/service_main.py", line 135, in
nexinfosys.data_source_manager = register_external_datasources()
File "/app/nexinfosys/initialization.py", line 216, in register_external_datasources
data_engine=nexinfosys.data_engine))
File "/app/nexinfosys/ie_imports/data_sources/fadn.py", line 247, in init
self._measures = load_measures_dictionary(self._base_directory)
File "/app/nexinfosys/ie_imports/data_sources/fadn.py", line 146, in load_measures_dictionary
d[df.iloc[r, 0]] = (df.iloc[r, en_pos], formula)
File "/usr/local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1762, in getitem
return self._getitem_tuple(key)
File "/usr/local/lib/python3.7/site-packages/pandas/core/indexing.py", line 2067, in _getitem_tuple
self._has_valid_tuple(tup)
File "/usr/local/lib/python3.7/site-packages/pandas/core/indexing.py", line 703, in _has_valid_tuple
self._validate_key(k, i)
File "/usr/local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1994, in _validate_key
self._validate_integer(key, axis)
File "/usr/local/lib/python3.7/site-packages/pandas/core/indexing.py", line 2063, in _validate_integer
raise IndexError("single positional indexer is out-of-bounds")
IndexError: single positional indexer is out-of-bounds
[2021-12-03 18:06:19 +0000] [9] [INFO] Worker exiting (pid: 9)
Connecting to data server
USER: root
USER: root
download URL: http://ec.europa.eu/agriculture/rica/database/help/infometa.csv
file: /root/.local/share/nis-backend/data/fadn/measures.csv
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 210, in run
self.sleep()
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 360, in sleep
ready = select.select([self.PIPE[0]], [], [], 1.0)
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 245, in handle_chld
self.reap_workers()
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/bin/gunicorn", line 8, in
sys.exit(run())
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 61, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 223, in run
super(Application, self).run()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 232, in run
self.halt(reason=inst.reason, exit_status=inst.exit_status)
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 345, in halt
self.stop()
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 393, in stop
time.sleep(0.1)
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 245, in handle_chld
self.reap_workers()
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>