jvdkwast / qgis-processing-pcraster Goto Github PK

When calling the QGIS CLI program qgis_process (which we call in R package qgisprocess) to run a PCRaster algorithm, the algorithm is run and the output is written, but in the end a segfault happens (which at the R side interrupts further handling in R and throws an error). It is specific to PCRaster, not seen for other QGIS processing providers.

I compiled and installed PCRaster from source in Linux Mint, using the Ubuntu guidelines at https://jvdkwast.github.io/qgis-processing-pcraster/. A Windows R user appeared to have similar results (r-spatial/qgisprocess#159).

I'm not sure whether this is to be solved on the PCRaster side alone, or also in qgis_process. Feel free to move this as appropriate.

Example below.

$ wget -q https://github.com/pcraster/pcraster/raw/master/data/demo/dem.map
$ 
$ myJSON=$(</dev/stdin)
{
  "inputs": {
    "INPUT": "dem.map",
    "OUTPUT": "dem_slope.map"
  }
}
$ 
$ echo "$myJSON" | qgis_process run pcraster:Slope -
/usr/lib/python3/dist-packages/qgis/utils.py:888: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  mod = _builtin_import(name, globals, locals, fromlist, level)
{
  "algorithm_details": {
    "can_cancel": true,
    "deprecated": false,
    "group": "Derivatives of digital elevation models",
    "has_known_issues": false,
    "help_url": null,
    "id": "pcraster:Slope",
    "name": "slope",
    "requires_matching_crs": false,
    "short_description": null,
    "tags": []
  },
  "gdal_version": "3.6.2",
  "geos_version": "3.11.1-CAPI-1.17.1",
  "inputs": {
    "INPUT": "dem.map",
    "OUTPUT": "dem_slope.map"
  },
  "log": {},
  "proj_version": "Rel. 9.1.1, December 1st, 2022",
  "provider_details": {
    "can_be_activated": true,
    "default_raster_file_extension": "map",
    "default_vector_file_extension": "gpkg",
    "is_active": true,
    "long_name": "PCRaster",
    "name": "PCRaster",
    "supported_output_raster_extensions": [
      "map"
    ],
    "supported_output_table_extensions": [
      "gpkg",
      "shp",
      "000",
      "csv",
      "dgn",
      "dxf",
      "fgb",
      "gdb",
      "geojson",
      "geojsonl",
      "geojsons",
      "gml",
      "gpx",
      "gxt",
      "ili",
      "itf",
      "json",
      "kml",
      "ods",
      "sql",
      "sqlite",
      "tab",
      "txt",
      "xlsx",
      "xml",
      "xtf"
    ],
    "supported_output_vector_extensions": [
      "gpkg",
      "shp",
      "000",
      "csv",
      "dgn",
      "dxf",
      "fgb",
      "gdb",
      "geojson",
      "geojsonl",
      "geojsons",
      "gml",
      "gpx",
      "gxt",
      "ili",
      "itf",
      "json",
      "kml",
      "ods",
      "sql",
      "sqlite",
      "tab",
      "txt",
      "xlsx",
      "xml",
      "xtf"
    ],
    "supports_non_file_based_output": false,
    "version": "QGIS PCRaster Provider version 0.2.0"
  },
  "python_version": "3.10.6",
  "qgis_code_revision": "0992b53397",
  "qgis_version": "3.30.2-'s-Hertogenbosch",
  "qt_version": "5.15.3",
  "results": {
    "OUTPUT": "dem_slope.map"
  }
Segmentation fault (core dumped)
$ 
$ ls dem_slope.map 
dem_slope.map
$ 
$ qgis_process --version
QGIS 3.30.2-'s-Hertogenbosch ''s-Hertogenbosch' (0992b53397)
QGIS code revision 0992b53397
Qt version 5.15.3
Python version 3.10.6
GDAL/OGR version 3.6.2
PROJ version 9.1.1
EPSG Registry database version v10.076 (2022-08-31)
GEOS version 3.11.1-CAPI-1.17.1
SQLite version 3.37.2
OS Linux Mint 21.1
$ 
$ python
Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pcraster as pcr
>>> pcr.__version__
'4.5.0-dev'
>>> exit()
$ 
$ echo $PYTHONPATH
:/usr/local/python
$ 
$ which qgis_process
/usr/bin/qgis_process
$ 
$ apt policy qgis
qgis:
  Installed: 1:3.30.2+36jammy-ubuntugis
  Candidate: 1:3.30.2+36jammy-ubuntugis
  Version table:
 *** 1:3.30.2+36jammy-ubuntugis 500
        500 https://qgis.org/ubuntugis jammy/main amd64 Packages
        100 /var/lib/dpkg/status
     3.22.16+dfsg-1~jammy3 500
        500 http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu jammy/main amd64 Packages
     3.22.4+dfsg-3build1 500
        500 http://ftp.belnet.be/ubuntu jammy/universe amd64 Packages
$ 

After installing the PCRaster Tools pluging, QGIS crashes at closing. This was observed for different QGIS versions using OSGeo4W as well as Conda installations. It also occured with the original resource sharing tools. It also occurs when no tool is used.

Here's a copy of the crash report:
Crash ID: 2bba35ac54ec3f66db08369439704d5a7beb5523

Stack Trace

GDALDestroyDriver :
dal::GDALRasterDriver::deregisterGDALDrivers :
dal::Client::~Client :
calc::DataStorageId::setId :
::operator() :
__crt_seh_guarded_call::operator(), &, > :
execute_onexit_table :
calc::vs2CsfVs :
calc::vs2CsfVs :
LdrpCallInitRoutine :
LdrShutdownProcess :
RtlExitUserProcess :
ExitProcessImplementation :
exit_or_terminate_process :
common_exit :
BaseThreadInitThunk :
RtlUserThreadStart :

QGIS Info
QGIS Version: 3.22.0-Bia?owie?a
QGIS code revision: d9022691f1
Compiled against Qt: 5.15.2
Running against Qt: 5.15.2
Compiled against GDAL: 3.4.0
Running against GDAL: 3.4.0

System Info
CPU Type: x86_64
Kernel Type: winnt
Kernel Version: 10.0.19042

The lookup tool gives the error below all versions I've tested and reported by users. The result raster is created, but isn't added to the Layers panel.

The lookup tool (and several other tools) use a list of input rasters. The CRS is derived from the first raster, but this seems to cause the error:
self.set_output_crs(output_file=outputFilePath, crs=input_rasters[0].crs(), feedback=feedback, context=context)

@nyalldawson can you please check how to do this correctly?

Here's the error message:

Algorithm 'lookup' starting…
Input parameters:
{ 'INPUT' : ['Z:/Lesson4/strahler.map'], 'INPUT1' : 'Z:\Lesson_4\lookup.txt', 'INPUT2' : 2, 'OUTPUT' : 'Z:/Lesson_4/testreclassplugin.map' }

Traceback (most recent call last):
File "C:\Users/hansa/AppData/Roaming/QGIS/QGIS3\profiles\ES/python/plugins\pcraster_tools\processing\algorithms\pcraster_lookup_algorithm.py", line 137, in processAlgorithm
self.set_output_crs(output_file=outputFilePath, crs=input_rasters[0].crs(), feedback=feedback, context=context)
AttributeError: 'str' object has no attribute 'crs'

So that users who install the plugin will know why they don't have any tools available

I have installed pcraster plugin via the plugin and then used the OSGeo4W network installer.

I keep getting this error on QGIS start-up. Any suggestions?

Grant Boxer

2023-10-08T11:55:11 CRITICAL Traceback (most recent call last):
File "C:\OSGeo4W/apps/qgis/./python\qgis\utils.py", line 448, in startPlugin
plugins[packageName].initGui()
File "C:\Users/boxer/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\pcraster_tools\pcraster_plugin.py", line 104, in initGui
import pcraster # pylint: disable=import-outside-toplevel,unused-import
File "C:\OSGeo4W/apps/qgis/./python\qgis\utils.py", line 888, in _import
mod = builtin_import(name, globals, locals, fromlist, level)
File "C:\OSGeo4W\apps\Python39\lib\site-packages\pcraster_init.py", line 17, in
os.add_dll_directory(pcraster_bin_pathname)
File "C:\OSGeo4W\apps\Python39\lib\os.py", line 1111, in add_dll_directory
cookie = nt._add_dll_directory(path)
OSError: [WinError 87] The parameter is incorrect: '.'

I've been trying to test this plugin using a fresh install of QGIS 3.20.3 via the osgeo4w v2 installer and a clean profile. I've installed PCRaster 4.3.1-4 via osgeo4w, and added the PCRaster Tools plugin.

Never having used PCRaster before, I naively tried to run the "slope" tool on an elevation .tif file (test2m.zip), which promptly gave me the "QGIS unexpectedly ended" crash dialog.

I now understand that I am supposed to use the "Convert to PCRaster Format" first (and have successfully run the "slope" tool with that), but ideally the slope processing tool log would simply show an error there, rather than crashing QGIS. Or even better, the tool parameters dialog could only let me select PCRaster format layers for the input.

Thanks for considering -- looking forward to exploring PCRaster more!

Report Details

Crash ID: a573919ad860cbb9df0bd4bce11551b46e72585b

Stack Trace

calc::PosException::operator= :
calc::PosException::operator= :
calc::DataValue::operator= :
calc::DataValue::operator= :
PyArg_ParseTuple_SizeT :
PyEval_EvalFrameDefault :
PyFunction_Vectorcall :
PyFloat_FromDouble :
PyVectorcall_Call :
PyObject_Call :
PyInit__core :
QgsProcessingAlgorithm::runPrepared :
QgsProcessingAlgRunnerTask::run :
PyInit__core :
QgsTask::start :
QThreadPoolPrivate::reset :
QThread::start :
BaseThreadInitThunk :
RtlUserThreadStart :

QGIS Info
QGIS Version: 3.20.3-Odense
QGIS code revision: 495fbaec
Compiled against Qt: 5.15.2
Running against Qt: 5.15.2
Compiled against GDAL: 3.3.1
Running against GDAL: 3.3.1

System Info
CPU Type: x86_64
Kernel Type: winnt
Kernel Version: 10.0.19043

@jvdkwast are you able to edit https://github.com/jvdkwast/qgis-processing-pcraster/blob/main/pcraster_tools/metadata.txt and flesh out the description, tags, and other areas with nicer content?

The resample tool script (resample.py) is installed, but doesn't show up in the Processing Toolbox.
@nyalldawson Can you please have a look at it?

I have installed PCraster from inside Qgis also from ZIP and from OSGeo4w app but it still appear this warning PCraster is not installed, although the icon for PCraster is appearing in the processing toolbox . and when conducting an analysis for a model it says PCraster library is not installed .

Could you please support me in solving this ?

This issue was originally reported in the PCRaster repository, but is not related to PCRaster, but the plugin. Therefore continuing here @GvdDool.
pcraster/pcraster#364 (comment)

I'll check it as soon as I have time. Meanwhile check these videos to see if you're using the tool correctly:

• Create lookup tables from a Raster Attribute Table and reclassify raster using PCRaster in QGIS
• Create Lookup Tables from Raster Attribute Tables of Continuous Rasters using PCRaster in QGIS

The PCRaster format doesn't include CRS info. When a raster is converted to the PCRaster map format with the Convert to PCRaster Format tool, GDAL creates an .aux.xml file with the CRS info. Output layers from PCRaster tools should have the same .aux.xml file in order to assign the same projection that is readable by QGIS.

When I install the PCRaster plugin it tells me that the pcraster libs need to be available and to consult the plugin docs. I can't find where the docs are - perhaps they can be added to the README, plugin metadata and the warning that pops up saying your libs are missing.

i always get this error message "PCRaster Tools Plugin: PCRaster is not installed -- algorithms will not be available" when ever i launch the pcraster plugin, which makes it difficult to run an analysis on QGIS.

I am the guy that posted on the YouTube tutorial video. I appreciate your willingness to help.

So I am trying to generate catchments from outlet points across 3,500 km-sq. There are 7891 outlet points that are actually sample points with a 5 digit sample ID.

I followed your youtube tutorial for how to use PCRaster tools with the QGIS plugin and everything went well except for the col2map procedure.

With your suggestion I then bypassed the QGIS plugin and used the OSGeo4W shell to run the command.
After numerous iterations I eventually found that there is a limit of 254 entries in the csv table to be converted to pcraster format. Also an ID of '255' is not a 'legal nominal value in small cell representation' I assume this is a limitation due to the data type established for the outlet ID field (byte?).

Now as long as my csv table only includes ID's of 1-254 it will execute reliably in the shell but errors persist in QGIS with more erratic behavior. Perhaps restarting QGIS is required.

It would be ideal if the col2map could intake an ID as a field with a few more numbers (make it an int?) but I assume that means modifying the PCRaster core codebase.

I hope any of this is helpful but please reach out if any more detail is needed.

QGIS version: 3.16.14 LTR
PCRaster Plugin: 0.1.1
pcraster version: 4.3.1
Windows 10