geocompx / docker Goto Github PK
View Code? Open in Web Editor NEWDockerfiles for Geocomputation
Home Page: https://github.com/geocompx/docker/pkgs/container/docker
License: GNU General Public License v3.0
Dockerfiles for Geocomputation
Home Page: https://github.com/geocompx/docker/pkgs/container/docker
License: GNU General Public License v3.0
See geocompx/geocompr#1108 and failing workflows
@Robinlovelace do I understand correctly that the geocompr images are based on rocker-org/geospatial? If so, https://github.com/rocker-org/geospatial uses very old R version and has not been updated in ~2 years. Should we do something with it?
With this error:
Encountered problems while solving:
- nothing provides requested quarto
The environment can't be solved, aborting the operation
error libmamba Could not solve for environment specs
critical libmamba UnsatisfiableError
The command '/usr/local/bin/_dockerfile_shell.sh micromamba install -y -n base -f /tmp/env.yaml && micromamba clean --all --yes' returned a non-zero code: 1
As discussed with @benz0li, we could use the b-data
stack throughout, with more consistency and good R/Python support. First we would need to think about similarities/differences and pros/cons, leaving this as an open question, I'm a bit out of my depth, so any advice / PRs welcome : )
Solution: use ubuntugis stable ppa
Context: geocompx/geocompy#52 and #26
Or similar, see error messages here:
#10 757.2 Failed to build cartopy pandas
#10 757.2 ERROR: Could not build wheels for cartopy, which is required to install pyproject.toml-based projects
------
Dockerfile:13
--------------------
11 | RUN pip3 install rasterio rioxarray netcdf4 h5netcdf
12 | # Install Python packages for visualisation
13 | >>> RUN pip3 install jupyter matplotlib hvplot pyviz geoviews
14 | # Install quarto
15 | RUN curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/etc/apt/trusted.gpg.d/githubcli-archive-keyring.gpg
--------------------
error: failed to solve: process "/bin/sh -c pip3 install jupyter matplotlib hvplot pyviz geoviews" did not complete successfully: exit code: 1
Error: buildx failed with: error: failed to solve: process "/bin/sh -c pip3 install jupyter matplotlib hvplot pyviz geoviews" did not complete successfully: exit code: 1
https://github.com/geocompr/docker/runs/5438444509?check_suite_focus=true#step:8:6181
Any ideas @anitagraser or anyone?
Heads-up @michaeldorman.
As discussed with @Nowosad currently the Docker images are rebuilt every time one of us pushes to the repo. This is not ideal. Any ideas for getting a more automated system going? It's tricky because DockerHub only triggers builds when it detects commits in this repo I think, so I was thinking of a cron job on my computer, but that is not an ideal solution. Suggestions welcome.
I think I know why: geocompy is now in geocompx/geocompy not geocompx/py
Evidence: https://github.com/geocompx/docker/actions/runs/4501197328/jobs/7921300929#step:8:553
With upstream containers building first.
Can we add GRASS, SAGA and qgisprocess to our geocompr:qgis image? This way we could harness the full potential of QGIS, and it would be also rather helpful for writing the qgisprocess vignette (see r-spatial/qgisprocess#16). For now, I have simply adjusted and rebuild your Dockerfile locally. Thinking it over again, yes, it is a good idea to add qgisprocess but for writing the vignette it makes more sense to mount my local qgisprocess repo. Hence, we would have to adjust the Dockerfile as follows (I can put in a PR if you want me to):
FROM geocompr/geocompr:qgis
RUN apt-get update \
&& apt-get install --no-install-recommends --no-install-suggests --allow-unauthenticated -y \
qgis-plugin-grass \
saga
RUN env QT_QPA_PLATFORM='offscreen' qgis_process
RUN echo "[PythonPlugins]\nprocessing=true" >> root/.local/share/QGIS/QGIS3/profiles/default/QGIS/QGIS3.ini
RUN R -e "remotes::install_github('paleolimbot/qgisprocess')"
The penultimate line is necessary to enable 3rd party providers when running qgis_process in a headless state (see qgis/QGIS#37989). And we need to execute the line before (RUN env QT_QPA_PLATFORM='offscreen' qgis_process) in order to create the QGIS3.ini file (maybe a bad hack, but it works).
FYI @paleolimbot
To include extras.
Seeing this message:
#8 53.84 Downloading Cartopy-0.18.0.tar.gz (14.4 MB)
#8 53.99 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.4/14.4 MB 92.6 MB/s eta 0:00:00
#8 54.39 Preparing metadata (setup.py): started
#8 54.63 Preparing metadata (setup.py): finished with status 'error'
#8 54.64 error: subprocess-exited-with-error
#8 54.64
#8 54.64 × python setup.py egg_info did not run successfully.
#8 54.64 │ exit code: 1
#8 54.64 ╰─> [13 lines of output]
#8 54.64 Traceback (most recent call last):
#8 54.64 File "/tmp/pip-install-xpr0hdls/cartopy_f8366d3bf5f44cd6b51d81b49e18c0f5/setup.py", line 43, in <module>
#8 54.64 import numpy as np
#8 54.64 ModuleNotFoundError: No module named 'numpy'
#8 54.64
#8 54.64 During handling of the above exception, another exception occurred:
#8 54.64
#8 54.64 Traceback (most recent call last):
#8 54.64 File "<string>", line 2, in <module>
#8 54.64 File "<pip-setuptools-caller>", line 34, in <module>
#8 54.64 File "/tmp/pip-install-xpr0hdls/cartopy_f8366d3bf5f44cd6b51d81b49e18c0f5/setup.py", line 45, in <module>
#8 54.64 raise ImportError('NumPy 1.10+ is required to install cartopy.')
#8 54.64 ImportError: NumPy 1.10+ is required to install cartopy.
#8 54.64 [end of output]
Source: https://github.com/geocompr/docker/runs/7054120092?check_suite_focus=true#step:8:736 Any ideas welcome, call for help in the dark @martinfleis!
Hot on the heels of #45
E.g. as can be seen here for proj: https://launchpad.net/~ubuntugis/+archive/ubuntu/ubuntugis-unstable?field.series_filter=focal
Plan: remove these Docker images.
I've tried a session starting with our qgis
container and get an error:
docker run -it ghcr.io/geocompx/docker:qgis /bin/bash
R
library(qgisprocess)
>>> PROBLEM encountered: couldn't build and save package cache!
Any ideas @jannes-m ?
PS C:\Users\georl_admin\github\nptscot> docker run -it ghcr.io/geocompx/docker:qgis /bin/bash
Unable to find image 'ghcr.io/geocompx/docker:qgis' locally
qgis: Pulling from geocompx/docker
e311a697a403: Pulling fs layer
926f2d6ac23b: Pulling fs layer
f85e4c4a147f: Pulling fs layer
961a38eedb06: Pulling fs layer
9c1cd9f12f56: Waiting 33ea3844e60e: Waiting f48a64097edb: Waiting
c1950ac6f0a7: Waiting
e16a99d08fab: Waiting 17f2f9344dea: Waiting 4d87aa41db2c: Waiting 22270397e270: Waiting 0b536e7d93f5: Pull complete
d1f515638bdc: Pull complete 30e84c00a654: Pull complete
7ea03b0dabd7: Pull complete c4e6ae8c57f2: Pull complete
e6d7b0cedb30: Pull complete
b70814f01360: Pull complete
0c66adfcfbfd: Pull complete 8da0ffa79364: Pull complete
e4ce9a391b17: Pull complete
a5b1049ad8b9: Pull complete
23c772569714: Pull complete 7263fc5234f6: Pull complete
0765648ea1e0: Pull complete 85d0466e54fd: Pull complete
caa1d3bf8920: Pull complete
95dfd028ee08: Pull complete
6614e611b87d: Pull complete
4e4b6b8e68ec: Pull complete
57611ebb3ea8: Pull complete
9e7b4fc17415: Pull complete
fa232696a74e: Pull complete
8f50b93b0730: Pull complete
6c62b14d42f5: Pull complete
5e1d47bbd067: Pull complete
c3c453181d49: Pull complete
9f866ee83c7e: Pull complete
c0c0ef0831f1: Pull complete
ae4d1b67ebe4: Pull complete
Digest: sha256:744b011b60b086fd5d487eaeaeab8b21d477ebda960e619f9e38ad6aca827fda
Status: Downloaded newer image for ghcr.io/geocompx/docker:qgis
root@6845cd09c1ee:/#
root@6845cd09c1ee:/#
root@6845cd09c1ee:/# R
R version 4.4.0 (2024-04-24) -- "Puppy Cup"
Copyright (C) 2024 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> qgisprocess::Attempting to load the package cache ... No cache found.
Will try to reconfigure qgisprocess and build new cache ...
Standard error message from 'qgis_process':
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
Standard error message from 'qgis_process':
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
ATTENTION: the QGIS algorithms could not be queried. You will loose some functionality.
You may want to (re)run `qgis_configure()`; see its documentation.
Error message was:
Error: parse error: premature EOF
{ "gdal_version": "3.8.5",
(right here) ------^
qgisprocess::qgis_run qgisprocess::qgis_enable_plugins
qgisprocess::qgis_result_stderr qgisprocess::qgis_get_description
qgisprocess::qgis_as_raster qgisprocess::qgis_has_provider
qgisprocess::qgis_detect_paths qgisprocess::qgis_detect_macos
qgisprocess::qgis_search_algorithms qgisprocess::qgis_run_algorithm_p
qgisprocess::has_qgis qgisprocess::qgis_outputs
qgisprocess::qgis_providers qgisprocess::qgis_use_json_input
qgisprocess::qgis_get_argument_specs qgisprocess::qgis_has_algorithm
qgisprocess::qgis_detect_windows_paths qgisprocess::qgis_result_args
qgisprocess::qgis_disable_plugins qgisprocess::qgis_algorithms
qgisprocess::qgis_path qgisprocess::qgis_as_terra
qgisprocess::qgis_run_algorithm qgisprocess::qgis_tmp_vector
qgisprocess::qgis_tmp_folder qgisprocess::qgis_function
qgisprocess::qgis_description qgisprocess::qgis_detect_macos_paths
qgisprocess::qgis_result_clean qgisprocess::qgis_extract_output
qgisprocess::qgis_arguments qgisprocess::qgis_list_input
qgisprocess::qgis_extract_output_by_class qgisprocess::qgis_dict_input
qgisprocess::qgis_plugins qgisprocess::qgis_pipe
qgisprocess::qgis_using_json_output qgisprocess::qgis_get_output_specs
qgisprocess::as_qgis_argument qgisprocess::qgis_detect_windows
qgisprocess::qgis_extract_output_by_name qgisprocess::qgis_clean_tmp
qgisprocess::qgis_tmp_raster qgisprocess::qgis_tmp_file
qgisprocess::qgis_using_json_input qgisprocess::qgis_result_single
qgisprocess::qgis_result_stdout qgisprocess::qgis_use_json_output
qgisprocess::qgis_clean_argument qgisprocess::qgis_output
qgisprocess::qgis_clean_result qgisprocess::qgis_tmp_base
qgisprocess::qgis_show_help qgisprocess::qgis_extract_output_by_position
qgisprocess::qgis_configure qgisprocess::qgis_unconfigure
qgisprocess::qgis_version qgisprocess::qgis_result_status
qgisprocess::qgis_has_plugin qgisprocess::qgis_as_brick
qgisprocess::qgis_tmp_clean
> qgisprocess::qgis_algorithms()
NULL
> qgisprocess::qgis_algorithms
qgisprocess::qgis_algorithms
> qgisprocess::qgis_plugins
qgisprocess::qgis_plugins
> qgisprocess::qgis_plugins()
# A tibble: 3 x 2
name enabled
<chr> <lgl>
1 grassprovider FALSE
2 processing TRUE
3 processing_saga_nextgen FALSE
> qgisprocess::qgis_search_algorithms("clean")
Error: result does not inherit from class data.frame
> library(qgisprocess)
>>> PROBLEM encountered: couldn't build and save package cache! <<<
The 'qgis_process' command-line utility was either not found or
did not fulfil the needs to build the package cache.
Please run `qgis_configure()` to fix this and rebuild the cache.
See its documentation if you need to preset the path of qgis_process.
If the problem persists, make sure that you correctly installed QGIS
for your operating system using the instructions at
<https://download.qgis.org>.
> qgisprocess::qgis_search_algorithms("clean")
Error: result does not inherit from class data.frame
>
As shown below:
Source: https://github.com/geocompx/docker/actions/workflows/suggests.yml
Related: geocompx/geocompr#1049
With the following message:
Dockerfile:48
--------------------
46 | RUN sh -c 'echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/qgis-archive-keyring.gpg] https://qgis.org/ubuntugis `lsb_release -c -s` main" > /etc/apt/sources.list.d/qgis.list'
47 | RUN apt-get update
48 | >>> RUN apt-get install -y qgis qgis-plugin-grass saga
49 | RUN wget -qO sagang_plugin.zip https://plugins.qgis.org/plugins/processing_saga_nextgen/version/0.0.7/download/
50 | RUN unzip -q sagang_plugin.zip -d /home/rstudio/.local/share/QGIS/QGIS3/profiles/default/python/plugins
--------------------
ERROR: failed to solve: process "/bin/sh -c apt-get install -y qgis qgis-plugin-grass saga" did not complete successfully: exit code: 100
Error: buildx failed with: ERROR: failed to solve: process "/bin/sh -c apt-get install -y qgis qgis-plugin-grass saga" did not complete successfully: exit code: 100
Source: https://github.com/geocompx/docker/actions/runs/6321057960/job/17164486628#step:8:644
Relevant lines:
Lines 48 to 50 in 00f4c90
Any ideas @jannes-m ?
I hit issues with sf
installation but this can be worked around following guidance from the Rocker maintainers here: rocker-org/rocker-versioned2#631
Not through pip
...
Context: https://github.com/rocker-org/rocker-versioned2/blob/master/scripts/install_python.sh
I need this for my own purposes but will be useful for others I think.
As documented here: https://pythonspeed.com/articles/activate-conda-dockerfile/
The README is R focussed but in fact we now have the Python images.
With the current set-up we don't do any tagging of different versions. However it would improve reproducibility if we could add tags making it easier for people to get images produced at a particular data/release. The action on which our Docker actions are based enable tagging but I wasn't sure how best to implement this https://github.com/docker/build-push-action/blob/master/docs/advanced/tags-labels.md
Any comments/suggestions on this very welcome.
Hi @Robinlovelace -- are there any plans to rename images from geocompr to geocompx?
This is an issue associated with images that build on geocompr/geocompr:minimal
:
#6 0.349 > remotes::install_github('geocompr/geocompkg', upgrade = TRUE, dependencies = TRUE)
#6 0.542 Skipping install of 'geocompkg' from a github remote, the SHA1 (6922a37b) has not changed since last install.
#6 0.542 Use `force = TRUE` to force installation
#6 0.542 >
#6 0.542 >
#6 DONE 3.0s
Source: https://github.com/geocompr/docker/runs/3776530772#step:8:270
Dailly is a bit intense.
For some reason. Logs: https://github.com/geocompr/docker/runs/5431901131?check_suite_focus=true#step:8:695
Heads-up @anitagraser plan for now is to remove movingpandas and add later. I'll keep this issue open to track it.
I may have made a silly mistake, quite likely as currently my Python dev skills amount to guesswork!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.