Law of least privilege should apply to prevent breaches or abuse, and enforce rate limiting as outlined in #27. This is of particular importance to ensuring that #23 is implemented in an appropriate manner, however this process needs to be robust and properly documented so that third party vendors are capable of providing similar protections to their data and services

Check if auth integration with moodle is possible

In line with #17, it stands to reason that each "project" (see #32) may require a particular "data source" definition which needs to be managed accordingly.

For PostGIS, this might be a service definition. For a QGIS backend, this will likely be an uploaded QGIS Project file with all the appropriate layers, styling, and data source credentials (like a pgservice definition etc).

This needs to be properly modelled, and have the lifecycle of a project taken into account.

The application is expected to deliver key data collections for the initial proof of concept delivery. This possibly includes data sources from ZANGI hosted by kartoza, and possibly OSM data (via docker OSM/ overpass etc). We need to collect a list of specific layers from specific sources and identify how they might be included into the product.

One simple solution to #27 may be to create a "credit" system which attaches a balance to a user account and limits the number of operations they are able to perform based on that balance.

This scales as well, as new users might simply start with a credit balance, and manually adding new credits to an account should be trivial for administrators. Once proper billing is implemented, a more sophisticated approach to this might be used.

Although I am typically not a proponent of microtransactions, in this case I believe it may have significant merit. In addition, I feel that the typical ethical issues that I believe affect many microtransaction implementations can easily be mitigated by pegging the value of the credits to a legitimate currency/ basket and offering reimbursement options. This prevents skewing of user value perceptions and may simply be made implementation specific if necessary.

Multiple levels of metering would likely be required in order to track, bill, and limit (#27) user actions which may result in costly processing and storage operations. This is unlikely to be meaningful exclusively at an infrastructure level and most likely some level of metering information would need to be captured on each operation (such as the region size to be clipped, or the total amount of data processed in Mb etc.)

The core application backend should be exposed via an API to ensure integration capabilities outlined in #1

Pages for #44 and #45 will need to be added to the application and referenced accordingly in #46

Create processes and documentation for dev

This is expected to be a storage intensive process and managing data access by users, storage locations, and transfer between systems is a key consideration

The utility or API will require a few parameters in order to perform it's work which will need to be clearly defined and documented, including which parameters are required, optional, or what defaults might be used

Billing will likely be a critical consideration for multiple aspects of the system, as it will serve to ensure platform sustainability, promote growth, and provide incentive for maintaining platform data by vendors.

Billing is also non-trivial and has multiple points of contact beyond a simple "subscriber" type system, including managing vendor costs, storage, processing, metering, and transfers to vendors where appropriate.

Develop a frontend (select framework) and some UI components

If any data is outstanding as listed in #22 it will need to be sourced and incorporated into the corporate data store

Define the initial data processing pipeline used to develop the PoC

The user authentication framework will need to be coupled with the application to enforce ACL rules to resource access

User attributes should be clearly defined, especially in line with ensuring that storing and processing of user data is done in compliance with the expected legislations (GDPR/ POPIA etc)

In line with requirements for #12 a web address will be critical to the application exposure and brand

Data objects will need to be properly defined and ensure that their lifecycle and properties can be handled effectively.

The nomenclature should probably also be standardised (e.g. data object/ element/ source/ project/ collection etc.)

Aside from the data required from #22 it makes sense to try and identify potential data sources that might be made available within the product or framework

Ideally we should be in a position to leverage the py-qgis-wps framework for a scalable and powerful approach to backend processing, which provides a status api and provides many of the benefits of a generic WPS, along with direct support for qgis providers, plugins, scripts, and models.

A simple stack can be setup up with the following docker-compose.yaml:

version: "3"

services:
  wps:
    image: 3liz/qgis-wps:ltr-rc
    platform: linux/x86_64
    environment:
      QGSWPS_SERVER_PARALLELPROCESSES: "2"
      QGSWPS_SERVER_LOGSTORAGE: REDIS
      QGSWPS_REDIS_HOST: wpsredis
      QGSWPS_PROCESSING_PROVIDERS_MODULE_PATH: /processing
      QGSWPS_CACHE_ROOTDIR: /projects
      QGSWPS_SERVER_WORKDIR: /srv/data
      # QGSWPS_USER: 1000:1000
      QGSRV_SERVER_RESPONSE_TIMEOUT: 1800
      QGSRV_SERVER_CROSS_ORIGIN: "yes"
      QGSWPS_LOGLEVEL: DEBUG
    volumes:
      - ./projects:/projects
      - ./processing:/processing
      - ./output:/srv/data
    ports:
      - "9999:8080"

  wpsredis:
    image: redis:5-alpine

The initial script developed for testing is on my development branch under commit c42ca95, which can be downloaded directly from github:

• https://github.com/zacharlie/geodata-mart/blob/c42ca95119610059308723afa4e3c3b742e7f16a/geodata/processing/scripts/clip_project.py

The "test" directory contains a small sample data set that can be used for evaluation:

• https://github.com/zacharlie/geodata-mart/blob/c42ca95119610059308723afa4e3c3b742e7f16a/geodata/test/projects/dem.tif
• https://github.com/zacharlie/geodata-mart/blob/c42ca95119610059308723afa4e3c3b742e7f16a/geodata/test/projects/world.geojson
• https://github.com/zacharlie/geodata-mart/blob/c42ca95119610059308723afa4e3c3b742e7f16a/geodata/test/projects/sample.qgs

Note that QGIS Processing will require the script to be added to the default profile, as the processing framework does not currently support profiles. The script can be run locally with QGIS Processing, e.g.

C:\OSGeo4W\bin\qgis_process-qgis-ltr.bat run script:gdmclip --distance_units=meters --area_units=m2 --ellipsoid=EPSG:7030 --LAYERS='world, dem' --CLIP_GEOM='Polygon ((28.5 -28.0, 28.5 -29.0, 29.5 -29.0, 29.5 -28.0, 28.5 -28.0))' --OUTPUT_CRS='EPSG:4326' --BUFFER_DIST_KM=50 --PROJECT_PATH='c:/test/projects/sample.qgs' --OUTPUT='c:/test/output/geodatamart'

This produces a zip file with gpkg and associated rasters

resetting raster paths in output project seems to be a bit buggy will fix

Storing the raster to geopackage does not work within the processing framework. When using the CLI, it throws an error. When using the same processing script from the QGIS GUI (3.26) it throws an error, but writes the output raster to the geopackage. When using the processing tools driectly in QGIS, it works. The RASTER_TABLE parameter does not work properly though unless used as TABLE.

Instead of a single clip and transform with GDAL, instead a multi-step process has been used to try identify or address issues as a work around, with the geopackage output raster process being commented out and replaced with a flat file output raster to include in a zip.

Assessing the WPS capabilities, operations, input, and output requirements can be performed with OWSLib, or alternatively, with a little mor work, vanilla requests as outlined by the py-qgis-wps tests.

OWSLib is easily installed with conda and is utilised as demonstrated below:

from owslib.wps import WebProcessingService, printInputOutput

# https://geopython.github.io/OWSLib/usage.html#wps


wps = WebProcessingService(
    "http://127.0.0.1:9999/ows/?service=WPS&MAP=sample", verbose=False, skip_caps=True
)
wps.getcapabilities()
wps.identification.type
wps.identification.title
wps.identification.abstract
for operation in wps.operations:
    operation.name

for process in wps.processes:
    process.identifier, process.title

processes = [process for process in wps.processes]

for process in processes:
    print(process.identifier)
    process = wps.describeprocess(process.identifier)
    for output in process.processOutputs:
        printInputOutput(output)
    # for output in process.processOutputs:
    #     printInputOutput(output)
    # process.identifier
    # process.title
    # process.abstract
    print("----------")

Execute process:

from owslib.wps import WebProcessingService, monitorExecution
import re
import uuid

output_name = uuid.uuid4().hex
wps = WebProcessingService(
    "http://127.0.0.1:9999/ows/?service=WPS&MAP=sample", verbose=False, skip_caps=True
)
processid = "script:gdmclip"
inputs = [
    ("LAYERS", "world, dem"),
    ("CLIP_GEOM", "POLYGON((29.0 29.0,29.0 30.0,30.0 30.0,30.0 29.0,29.0 29.0))"),
    ("OUTPUT_CRS", "4326"),
    ("BUFFER_DIST_KM", "50"),
    ("OUTPUT", "OUTPUT"),
]

execution = wps.execute(processid, inputs, output=output_name)

response = str(execution.response)

id = re.search(r"uuid=(.*?)\"", response).group(1)

print(id)

monitorExecution(execution)

The result is an "internal error" which has not been resolved despite various attempted workarounds and experiments.

Using the template processing script from QGIS (simple vector buffer), works with the WPS and compose stack provided.

In the meantime, the plan is to use QGIS Processing directly in the django/ celery containers and revisit the WPS utilization later.

Aside from the data required from #22 it makes sense to try and identify potential data sources that might be made available within the product or framework

The project needs an appropriate license, such as AGPL

Outline existing tools, frameworks, utilities, and processes that may be used alongside #1 to determine product and process structure

We need a test project and set of data for development purposes

A terms and conditions document needs to be created

As a product and platform the system should have it's own brand guidelines

Define structure of backend application

Add user authentication management

Create a login/ logout UI for the frontend

This may be specific to certain data collection types or processing backends, but it is likely that certain models and application details (such as definitions of layer objects) will need to be managed internally by the application or specified by data vendors.

The backend may already accommodate for this, such as adding WFS layers to QGIS projects for processing, but it is likely that implementation specific details will need to be considered such ensuring the correct credentials are accessible by the project and the system.

An effective way to document project details for developers and users must be configured

For access to data as well as limiting the processing abilities of individual users to prevent degradation of services or abuse, it may be required to implement various rate limiting operations on different parts of the system. The strategy for these implementations should be clearly defined.

It should be decided whether external resources (such as metadata objects stored in a STAC catalog, GeoNetwork, or similar) are able to be defined on the platform, and what sort of distinguishing metadata might be attached to such items.

For any items listed in #22 access will need to be made available to the application. This includes the creation of a specific user for a backend data source, e.g. PostgreSQL

QGIS projects will need to be created for the initial data sources outlined in #22

Define the product and structure for development

Multiple processing backends are expected to be supported in order to properly support the #1 Roadmap, so we need to answer:

• What backends are supported
• How are they managed
• How is this model extended

A privacy policy would be required for storing and managing user data. In addition, if user actions are logged such as outlined in #43 it is likely that this action must be outlined in policy

Get a project template with boilerplate code and structure in place

Define the UX for the clipping tool and marketplace

Define a product development road map that outlines expected milestones, targets, and strategy for development

It would probably make a lot of sense to expose data components to external systems through industry standard APIs such as WCS or other OGC APIs where possible

Aside from typical policies such as #44 and #45 additional policies are likely required in order for the platform to function efficiently in line with its goals. Many of these would typically be required when operating a service on behalf of data vendors. The first step would be to outline required policies.

Some such examples might include:

• data removal
• data use, licensing, and relicensing
• data quality standards
• data access

Map forms will be used to create processing jobs, but an "ecommerce" type workflow and ux should be used to:

• ensure that processes are only run when desired
• ensure that operations/ orders can be inspected/ double checked against costs (e.g. what layers are to be processed)
• processing costs are accurately evaluated and available for introspection
• credit balances are correct and can be debited at runtime
• advanced validation can be done prior to processing, preventing frontend bugs from affecting backend processes
• orders can be cancelled, or stored for later processing

Data vendors will be distinct from users and will need to be handled appropriately to ensure control of data sources

It is highly probable that at some point the ability to group or otherwise categorize data will be required to provide a good UX. This includes items such as allowing users to select "layer groups" from a data source, or browsing and searching the data catalogue.

This may include some options such as layer tagging or other metadata management processes which should be outlined earlier rather than later, as each distinct UX may have implementation specific concerns.

A sign up procedure should be created which ensures that users agree to the specified terms. If the service remains a closed service, information regarding account creation and the process to create accounts should be made available

While the definition of how to manage the details for connecting to a projects data sources may be covered in #34, it is still necessary to establish a model and strategy for how this data might be exposed or published for end users.

This includes the definition of specific layers that are published, how they might be tested for consistency, definition of available extents for data or layers, and some form of access control to limit which users might be exposed to a particular data collection.

In addition, some form of tests against requests for out of bounds data might be required as well.

Create processes and documentation for prod