geocompx / docker Goto Github PK

View Code? Open in Web Editor NEW

35.0 6.0 5.0 229 KB

Dockerfiles for Geocomputation

Home Page: https://github.com/geocompx/docker/pkgs/container/docker

License: GNU General Public License v3.0

R 6.55% Dockerfile 92.31% Python 1.14%

docker-image docker

docker's Issues

Use suggests image as basis for qgis image

See geocompx/geocompr#1108 and failing workflows

rocker

@Robinlovelace do I understand correctly that the geocompr images are based on rocker-org/geospatial? If so, https://github.com/rocker-org/geospatial uses very old R version and has not been updated in ~2 years. Should we do something with it?

qgis image failing

As shown here

Mamba build failing

With this error:

Encountered problems while solving:
  - nothing provides requested quarto

The environment can't be solved, aborting the operation
error    libmamba Could not solve for environment specs
critical libmamba UnsatisfiableError
The command '/usr/local/bin/_dockerfile_shell.sh micromamba install -y -n base -f /tmp/env.yaml &&     micromamba clean --all --yes' returned a non-zero code: 1

Switch to Rocker's new host location for geospatial images

https://github.com/rocker-org/rocker-versioned2/pkgs/container/geospatial

Refactor images, start from same base

As discussed with @benz0li, we could use the b-data stack throughout, with more consistency and good R/Python support. First we would need to think about similarities/differences and pros/cons, leaving this as an open question, I'm a bit out of my depth, so any advice / PRs welcome : )

Current version uses PROJ 6.3.1

Solution: use ubuntugis stable ppa

Build python image from continuumio/miniconda3

Context: geocompx/geocompy#52 and #26

Cannot install pyviz

Or similar, see error messages here:

 #10 757.2 Failed to build cartopy pandas
#10 757.2 ERROR: Could not build wheels for cartopy, which is required to install pyproject.toml-based projects
------
Dockerfile:13
--------------------
  11 |     RUN pip3 install rasterio rioxarray netcdf4 h5netcdf
  12 |     # Install Python packages for visualisation
  13 | >>> RUN pip3 install jupyter matplotlib hvplot pyviz geoviews
  14 |     # Install quarto
  15 |     RUN curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/etc/apt/trusted.gpg.d/githubcli-archive-keyring.gpg
--------------------
error: failed to solve: process "/bin/sh -c pip3 install jupyter matplotlib hvplot pyviz geoviews" did not complete successfully: exit code: 1
Error: buildx failed with: error: failed to solve: process "/bin/sh -c pip3 install jupyter matplotlib hvplot pyviz geoviews" did not complete successfully: exit code: 1

https://github.com/geocompr/docker/runs/5438444509?check_suite_focus=true#step:8:6181

Any ideas @anitagraser or anyone?

Add rasterio and rioxarray to python images

Heads-up @michaeldorman.

Auto build Docker containers

As discussed with @Nowosad currently the Docker images are rebuilt every time one of us pushes to the repo. This is not ideal. Any ideas for getting a more automated system going? It's tricky because DockerHub only triggers builds when it detects commits in this repo I think, so I was thinking of a cron job on my computer, but that is not an ideal solution. Suggestions welcome.

mamba image failing

I think I know why: geocompy is now in geocompx/geocompy not geocompx/py

Evidence: https://github.com/geocompx/docker/actions/runs/4501197328/jobs/7921300929#step:8:553

Sequence Docker container builds

With upstream containers building first.

Add grass7, saga and qgisprocess to geocompr:qgis

Can we add GRASS, SAGA and qgisprocess to our geocompr:qgis image? This way we could harness the full potential of QGIS, and it would be also rather helpful for writing the qgisprocess vignette (see r-spatial/qgisprocess#16). For now, I have simply adjusted and rebuild your Dockerfile locally. Thinking it over again, yes, it is a good idea to add qgisprocess but for writing the vignette it makes more sense to mount my local qgisprocess repo. Hence, we would have to adjust the Dockerfile as follows (I can put in a PR if you want me to):

FROM geocompr/geocompr:qgis

RUN apt-get update \
    && apt-get install --no-install-recommends --no-install-suggests --allow-unauthenticated -y \
    qgis-plugin-grass \
    saga
RUN env QT_QPA_PLATFORM='offscreen' qgis_process
RUN echo "[PythonPlugins]\nprocessing=true" >> root/.local/share/QGIS/QGIS3/profiles/default/QGIS/QGIS3.ini
RUN R -e "remotes::install_github('paleolimbot/qgisprocess')"

The penultimate line is necessary to enable 3rd party providers when running qgis_process in a headless state (see qgis/QGIS#37989). And we need to execute the line before (RUN env QT_QPA_PLATFORM='offscreen' qgis_process) in order to create the QGIS3.ini file (maybe a bad hack, but it works).

FYI @paleolimbot

Update qgis image

To include extras.

Python package installation issues

Seeing this message:

#8 53.84   Downloading Cartopy-0.18.0.tar.gz (14.4 MB)
#8 53.99      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.4/14.4 MB 92.6 MB/s eta 0:00:00
#8 54.39   Preparing metadata (setup.py): started
#8 54.63   Preparing metadata (setup.py): finished with status 'error'
#8 54.64   error: subprocess-exited-with-error
#8 54.64   
#8 54.64   × python setup.py egg_info did not run successfully.
#8 54.64   │ exit code: 1
#8 54.64   ╰─> [13 lines of output]
#8 54.64       Traceback (most recent call last):
#8 54.64         File "/tmp/pip-install-xpr0hdls/cartopy_f8366d3bf5f44cd6b51d81b49e18c0f5/setup.py", line 43, in <module>
#8 54.64           import numpy as np
#8 54.64       ModuleNotFoundError: No module named 'numpy'
#8 54.64       
#8 54.64       During handling of the above exception, another exception occurred:
#8 54.64       
#8 54.64       Traceback (most recent call last):
#8 54.64         File "<string>", line 2, in <module>
#8 54.64         File "<pip-setuptools-caller>", line 34, in <module>
#8 54.64         File "/tmp/pip-install-xpr0hdls/cartopy_f8366d3bf5f44cd6b51d81b49e18c0f5/setup.py", line 45, in <module>
#8 54.64           raise ImportError('NumPy 1.10+ is required to install cartopy.')
#8 54.64       ImportError: NumPy 1.10+ is required to install cartopy.
#8 54.64       [end of output]

Source: https://github.com/geocompr/docker/runs/7054120092?check_suite_focus=true#step:8:736 Any ideas welcome, call for help in the dark @martinfleis!

Add rust image to README

Hot on the heels of #45

'Unstable' and 'dev-osgeo' versions are out of date

E.g. as can be seen here for proj: https://launchpad.net/~ubuntugis/+archive/ubuntu/ubuntugis-unstable?field.series_filter=focal

Plan: remove these Docker images.

Add 'netcdf4' and 'h5netcdf' to geocompr/geocompr:python

qgis container not working out of the box?

I've tried a session starting with our qgis container and get an error:

docker run -it ghcr.io/geocompx/docker:qgis /bin/bash
R
library(qgisprocess)

>>> PROBLEM encountered: couldn't build and save package cache!

Any ideas @jannes-m ?

PS C:\Users\georl_admin\github\nptscot> docker run -it ghcr.io/geocompx/docker:qgis /bin/bash
Unable to find image 'ghcr.io/geocompx/docker:qgis' locally
qgis: Pulling from geocompx/docker
e311a697a403: Pulling fs layer
926f2d6ac23b: Pulling fs layer
f85e4c4a147f: Pulling fs layer
961a38eedb06: Pulling fs layer
9c1cd9f12f56: Waiting                                                                                                   33ea3844e60e: Waiting                                                                                                   f48a64097edb: Waiting
c1950ac6f0a7: Waiting
e16a99d08fab: Waiting                                                                                                   17f2f9344dea: Waiting                                                                                                   4d87aa41db2c: Waiting                                                                                                   22270397e270: Waiting                                                                                                   0b536e7d93f5: Pull complete
d1f515638bdc: Pull complete                                                                                             30e84c00a654: Pull complete
7ea03b0dabd7: Pull complete                                                                                             c4e6ae8c57f2: Pull complete
e6d7b0cedb30: Pull complete
b70814f01360: Pull complete
0c66adfcfbfd: Pull complete                                                                                             8da0ffa79364: Pull complete
e4ce9a391b17: Pull complete
a5b1049ad8b9: Pull complete
23c772569714: Pull complete                                                                                             7263fc5234f6: Pull complete
0765648ea1e0: Pull complete                                                                                             85d0466e54fd: Pull complete
caa1d3bf8920: Pull complete
95dfd028ee08: Pull complete
6614e611b87d: Pull complete
4e4b6b8e68ec: Pull complete
57611ebb3ea8: Pull complete
9e7b4fc17415: Pull complete
fa232696a74e: Pull complete
8f50b93b0730: Pull complete
6c62b14d42f5: Pull complete
5e1d47bbd067: Pull complete
c3c453181d49: Pull complete
9f866ee83c7e: Pull complete
c0c0ef0831f1: Pull complete
ae4d1b67ebe4: Pull complete
Digest: sha256:744b011b60b086fd5d487eaeaeab8b21d477ebda960e619f9e38ad6aca827fda
Status: Downloaded newer image for ghcr.io/geocompx/docker:qgis
root@6845cd09c1ee:/#
root@6845cd09c1ee:/#
root@6845cd09c1ee:/# R

R version 4.4.0 (2024-04-24) -- "Puppy Cup"
Copyright (C) 2024 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> qgisprocess::Attempting to load the package cache ... No cache found.
Will try to reconfigure qgisprocess and build new cache ...

Standard error message from 'qgis_process':
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'



Standard error message from 'qgis_process':
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'



ATTENTION: the QGIS algorithms could not be queried. You will loose some functionality.
You may want to (re)run `qgis_configure()`; see its documentation.
Error message was:

Error: parse error: premature EOF
                                       {   "gdal_version": "3.8.5",
                     (right here) ------^



qgisprocess::qgis_run                         qgisprocess::qgis_enable_plugins
qgisprocess::qgis_result_stderr               qgisprocess::qgis_get_description
qgisprocess::qgis_as_raster                   qgisprocess::qgis_has_provider
qgisprocess::qgis_detect_paths                qgisprocess::qgis_detect_macos
qgisprocess::qgis_search_algorithms           qgisprocess::qgis_run_algorithm_p
qgisprocess::has_qgis                         qgisprocess::qgis_outputs
qgisprocess::qgis_providers                   qgisprocess::qgis_use_json_input
qgisprocess::qgis_get_argument_specs          qgisprocess::qgis_has_algorithm
qgisprocess::qgis_detect_windows_paths        qgisprocess::qgis_result_args
qgisprocess::qgis_disable_plugins             qgisprocess::qgis_algorithms
qgisprocess::qgis_path                        qgisprocess::qgis_as_terra
qgisprocess::qgis_run_algorithm               qgisprocess::qgis_tmp_vector
qgisprocess::qgis_tmp_folder                  qgisprocess::qgis_function
qgisprocess::qgis_description                 qgisprocess::qgis_detect_macos_paths
qgisprocess::qgis_result_clean                qgisprocess::qgis_extract_output
qgisprocess::qgis_arguments                   qgisprocess::qgis_list_input
qgisprocess::qgis_extract_output_by_class     qgisprocess::qgis_dict_input
qgisprocess::qgis_plugins                     qgisprocess::qgis_pipe
qgisprocess::qgis_using_json_output           qgisprocess::qgis_get_output_specs
qgisprocess::as_qgis_argument                 qgisprocess::qgis_detect_windows
qgisprocess::qgis_extract_output_by_name      qgisprocess::qgis_clean_tmp
qgisprocess::qgis_tmp_raster                  qgisprocess::qgis_tmp_file
qgisprocess::qgis_using_json_input            qgisprocess::qgis_result_single
qgisprocess::qgis_result_stdout               qgisprocess::qgis_use_json_output
qgisprocess::qgis_clean_argument              qgisprocess::qgis_output
qgisprocess::qgis_clean_result                qgisprocess::qgis_tmp_base
qgisprocess::qgis_show_help                   qgisprocess::qgis_extract_output_by_position
qgisprocess::qgis_configure                   qgisprocess::qgis_unconfigure
qgisprocess::qgis_version                     qgisprocess::qgis_result_status
qgisprocess::qgis_has_plugin                  qgisprocess::qgis_as_brick
qgisprocess::qgis_tmp_clean
> qgisprocess::qgis_algorithms()
NULL
> qgisprocess::qgis_algorithms
qgisprocess::qgis_algorithms
> qgisprocess::qgis_plugins
qgisprocess::qgis_plugins
> qgisprocess::qgis_plugins()
# A tibble: 3 x 2
  name                    enabled
  <chr>                   <lgl>
1 grassprovider           FALSE
2 processing              TRUE
3 processing_saga_nextgen FALSE
> qgisprocess::qgis_search_algorithms("clean")
Error: result does not inherit from class data.frame
> library(qgisprocess)

>>> PROBLEM encountered: couldn't build and save package cache! <<<
The 'qgis_process' command-line utility was either not found or
did not fulfil the needs to build the package cache.
Please run `qgis_configure()` to fix this and rebuild the cache.
See its documentation if you need to preset the path of qgis_process.
If the problem persists, make sure that you correctly installed QGIS
for your operating system using the instructions at
<https://download.qgis.org>.
> qgisprocess::qgis_search_algorithms("clean")
Error: result does not inherit from class data.frame
>

Add brief guide

Like this 🎉

Add rocker/geospatial:dev-osgeo based build

rocker-org/geospatial#31

Workflows disabled

As shown below:

Source: https://github.com/geocompx/docker/actions/workflows/suggests.yml

Related: geocompx/geocompr#1049

Saga plugin installation steps causing Docker build to fail

With the following message:

Dockerfile:48
--------------------
  46 |     RUN sh -c 'echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/qgis-archive-keyring.gpg] https://qgis.org/ubuntugis `lsb_release -c -s` main" > /etc/apt/sources.list.d/qgis.list'
  47 |     RUN apt-get update
  48 | >>> RUN apt-get install -y qgis qgis-plugin-grass saga
  49 |     RUN wget -qO sagang_plugin.zip https://plugins.qgis.org/plugins/processing_saga_nextgen/version/0.0.7/download/
  50 |     RUN unzip -q sagang_plugin.zip -d /home/rstudio/.local/share/QGIS/QGIS3/profiles/default/python/plugins
--------------------
ERROR: failed to solve: process "/bin/sh -c apt-get install -y qgis qgis-plugin-grass saga" did not complete successfully: exit code: 100
Error: buildx failed with: ERROR: failed to solve: process "/bin/sh -c apt-get install -y qgis qgis-plugin-grass saga" did not complete successfully: exit code: 100

Source: https://github.com/geocompx/docker/actions/runs/6321057960/job/17164486628#step:8:644

Relevant lines:

docker/qgis/Dockerfile

Lines 48 to 50 in 00f4c90

    
           RUN apt-get install -y qgis qgis-plugin-grass saga 
        
           RUN wget -qO sagang_plugin.zip https://plugins.qgis.org/plugins/processing_saga_nextgen/version/0.0.7/download/ 
        
           RUN unzip -q sagang_plugin.zip -d /home/rstudio/.local/share/QGIS/QGIS3/profiles/default/python/plugins

Any ideas @jannes-m ?

Use suggests for buildbook + maybe binder versions

Add quarto to python image

Add binder dockerfile

Use `rocker/geospatial:dev-osgeo` as basis for `qgis` image

I hit issues with sf installation but this can be worked around following guidance from the Rocker maintainers here: rocker-org/rocker-versioned2#631

How to install Python environment?

Not through pip...

Context: https://github.com/rocker-org/rocker-versioned2/blob/master/scripts/install_python.sh

replies here: https://twitter.com/robinlovelace/status/1443537000691347460

Add osgeo image

Add container with R + Rust

I need this for my own purposes but will be useful for others I think.

Make geocompy environment loaded by default in conda image

As documented here: https://pythonspeed.com/articles/activate-conda-dockerfile/

Update README to communicate that this repo doesn't just generate R based images

The README is R focussed but in fact we now have the Python images.

Tagging?

With the current set-up we don't do any tagging of different versions. However it would improve reproducibility if we could add tags making it easier for people to get images produced at a particular data/release. The action on which our Docker actions are based enable tagging but I wasn't sure how best to implement this https://github.com/docker/build-push-action/blob/master/docs/advanced/tags-labels.md

Any comments/suggestions on this very welcome.

Rename geocompr to geocompx

Hi @Robinlovelace -- are there any plans to rename images from geocompr to geocompx?

Use updated rocker osgeo recipes

rocker-org/rocker-versioned2#708

Packages not being updated if geocompkg already at latest version in image

This is an issue associated with images that build on geocompr/geocompr:minimal:

#6 0.349 > remotes::install_github('geocompr/geocompkg', upgrade = TRUE, dependencies = TRUE)
#6 0.542 Skipping install of 'geocompkg' from a github remote, the SHA1 (6922a37b) has not changed since last install.
#6 0.542   Use `force = TRUE` to force installation
#6 0.542 > 
#6 0.542 > 
#6 DONE 3.0s

Source: https://github.com/geocompr/docker/runs/3776530772#step:8:270

I may have made a silly mistake, quite likely as currently my Python dev skills amount to guesswork!

	RUN apt-get install -y qgis qgis-plugin-grass saga
	RUN wget -qO sagang_plugin.zip https://plugins.qgis.org/plugins/processing_saga_nextgen/version/0.0.7/download/
	RUN unzip -q sagang_plugin.zip -d /home/rstudio/.local/share/QGIS/QGIS3/profiles/default/python/plugins

geocompx / docker Goto Github PK

docker's Issues

Recommend Projects

Recommend Topics

Recommend Org