schemas, users and groups for PPD and LSD and other priority data maintainers to be able to load data

a generic user for readonly access?

service files?

https://docs.google.com/document/d/1WjRSopokiqeVUHmTDlQYS4stMMlvb7ZjUCU9YjYuPRI/edit#

https://docs.google.com/document/d/1_fPu8PCpy4wgiuTXEgDVgcYrRvyJIJsLpbWY9vgphgc/edit#heading=h.115jmdnefh5g

'analytics' mentioned in section 5 Deliverables table - enquire what this actually means

https://docs.google.com/document/d/1y9YGe7o1r286ZWtP1Hxy2RmG-228v_2n0XlNHmqBAHU/edit#heading=h.wc70n4w8x69s

The topo maps need to contain similar themes to the ones used by UK's Ordnance Survey agency in 1991, of which a picture is available on the first mission report

https://docs.google.com/document/d/1-tdcNiT-hWiK3QvuMJCCUGh4xTtLXV5fW1ugxN0Ln3M/edit#heading=h.rl7kdm1pjk0

We need to understand which layers are needed and how to use them in order to produce a topographic map of similar quality

All layers used in the production of the topo maps are priorities to be available in PostGIS to support topo maps in QGIS

The layers might well be published in GeoServer as well (to provide web service endpoints for general users)

https://docs.google.com/document/d/1kOnFr8ia3KV_Y7uHAt9iZsY0Cy54KoBHPvoXKezdJMo/edit#

We will be using the styling work done in topostyle as a starting point for styling the layers used in the Dominica topo maps.

For this we need to map the topostyle styles to the layers determined in #37. We will also need to tweak the styles as appropriate.

We need to implement a web interface for the public to access the generated topo maps. This might be subject to data distribution policies and/or some monetization

https://docs.google.com/document/d/1_fPu8PCpy4wgiuTXEgDVgcYrRvyJIJsLpbWY9vgphgc/edit#heading=h.115jmdnefh5g

The spreadsheet in

https://docs.google.com/spreadsheets/d/1uiuey2w4IvCNtNK0rV2Z3e5IOk0TsbGrkQit9Cdic0o/edit#gid=0

contains a Catalog sheet which contains an initial list of most datasets that were identified during previous stages of the project.

We need to fill in the blanks in the spreadsheet in order to

1. Get a description of the existing datasets and their respective custodians
2. Track the progress of data upload, validation and publishing

This sheet shall contain the following columns - be sure to create any columns listed here that may not be in the spreadsheet yet:

• Columns related to data description and owners

  • OGC Topic Category - A value coming from the fixed ISO 19115 topic categories, which are listed in the first column of the Lookups sheet
  • Dataset - Name of the dataset
  • Dataset custodian department - Department that will be responsible for owning the data
  • Dataset custodian person
  • Metadata custodian department - Department that will be responsible for maintaining the metadata of the dataset
  • Metadata custodian person - Person that will be responsible for maintaining the metadata of the dataset
  • Style custodian department - Department that will be responsible for maintaining the style for the dataset
  • Style custodian person -
  • name of dataset on old DomiNode - This means that we most likely already have the dataset (as we have a dump of old DomiNode's contents)
  • location of authoritative file - It is likely that more than one department has different copies of datasets, but we need to find which is the canonical one
  • additional details
  • dataset is to be freely accessible to the general public (YES/NO)
  • dataset type - NATIONAL BASEMAP/CORE LAYER/PROJECT LAYER - Read more about this in the DomiNode Data Assessment SOP

• Columns related to data ingestion workflow status - These are mostly Yes/No columns

  • dataset has been uploaded to staging DB or filesystem
  • dataset has been validated
  • dataset has a metadata record
  • metadata record has been validated
  • dataset has a style
  • dataset style has been validated
  • dataset has been promoted to the published DB schema or filesystem dir
  • dataset should be published via OGC webservices
  • dataset has been published via OGC webservices
  • dataset should be published via GeoNode
  • dataset has been published via GeoNode

Interact with the various project stakeholders in order to gather the information specified above.

Start with the datasets owned by the LSD and PPD departments, as these are the biggest stakeholders.

Production deployment is to be documented with step by step directions.

This document shall then be used to perform the actual deployment.

https://docs.google.com/document/d/1-Y0MhmB4eQ6VeaV8d3UrmCxRMg24hzbdRIbswzpW0kc/edit#

• back up dominode.dm (old instance)
• restore to staging GeoNode (only layers that are not in the staging server, which came from dominode.piensa.co)

https://docs.google.com/document/d/1TBWl-FhmwYhWEukfZiPhMhh-hn_WxWw7SzmiMur_tx0/edit#

The data permission matrix should identify

• Main stakeholder agencies
• Specific people inside each agency that will serve designated roles (data custodian, etc)
• Specific people inside each agency that will be given privileged access

It should mention

• Required capabilities for each database (data, styles, metadata, geonode), webservice (WFS, etc) and also to the minio storage
• Required capabilities for data publishing

Access to the matrix:

https://docs.google.com/spreadsheets/d/1uiuey2w4IvCNtNK0rV2Z3e5IOk0TsbGrkQit9Cdic0o/edit#gid=656035837

To start now with the staging system and continue through to the production release

• coordinate the update of https://docs.google.com/spreadsheets/d/1uiuey2w4IvCNtNK0rV2Z3e5IOk0TsbGrkQit9Cdic0o/edit#gid=0 by GoCD
• coordinate the implementation of the user-permission matrix that should be documented in https://docs.google.com/document/d/1TBWl-FhmwYhWEukfZiPhMhh-hn_WxWw7SzmiMur_tx0/edit#
• start with core data maintainers from PPD and LSD and ensure they have the right permissions
• coordinate and support the maintenance users to connect to PostGIS or MinIO and load their data and metadata as per the data catalog sheet

assign to Luisa

https://docs.google.com/document/d/1_fPu8PCpy4wgiuTXEgDVgcYrRvyJIJsLpbWY9vgphgc/edit#heading=h.115jmdnefh5g

in 3.5.1: The firm will also assist in developing a data request process along with a data request form hosted on the DomiNode.

more detail from Collin email 8 June:

The PCU will follow up with Lands and Surveys, ICTU as well as Physical Planning with regards to a form. A call with the Firm and those agencies to sketch out the content of this form may be useful. They will need to define if they want one form for all requests or different forms for different parties, i.e. commercial vs research requests. Parties will also need to define the process for acceptance and processing of these requests.

https://docs.google.com/document/d/1TBWl-FhmwYhWEukfZiPhMhh-hn_WxWw7SzmiMur_tx0/edit#

https://docs.google.com/document/d/197UlgpDycsKU2206Bv1sgm_GiwxuurZFGBwIpzPyZoQ/edit#heading=h.95ipcbtpz3ul

https://docs.google.com/spreadsheets/d/1a52ZJAwjK5hI-BNBaTTPSh06uc0b0NPG2eEfvDIBXz8/edit#gid=0

3.5.7 Budgeting and Recurring Costs

The firm will assist the GoCD in budgeting for the maintenance of the NSDI, and particularly the SDMP and related components. Minimizing recurring costs to a level which can be supported by the budgets of participating agencies is critical. Cloud services used under this project, for instance, must have a long term recurring cost which is possible with the budgets of participating agencies. Yearly budgets for maintenance of all hardware and software required for maintaining the processes and systems created under this project, or deemed vital for the NSDI by the GoCD, will be developed by the firm for a period of 5 years from the start of this contract.

further clarification from Collin email 8 June:

A five-year budget will be a rough estimate at best given the pace of technological change. If the Firm gives a budget before the end of your engagement, it should be fine as long as it identifies the factors which could substantially increase or lower the quoted amounts.

https://drive.google.com/open?id=1kOnFr8ia3KV_Y7uHAt9iZsY0Cy54KoBHPvoXKezdJMo

from Collin 8 June:

Please follow up with Charles and Lands and Surveys (arranging a call may be useful) with regards to the current status of the CORS.

• prep orchestration to remove Nucs from staging
• ship Nucs to Dominica
• install Nucs in Dominica data centre
• update orchestration to final staging and production nodes and clusters
• test everything
• point dominode.dm to production instance

We need to produce Dominican topo maps for the following scales:

• 1:50000
• 1:10000
• 1:5000
• 1:2500

In QGIS and based on a fork of https://github.com/kartoza/topostyle

https://docs.google.com/document/d/1TlMkh4SxcnhT8wWv83yDx--XuUVrF-ZA987rJF9815Q/edit#heading=h.k9i5gxiqwz1g

Use elements from dominode.dm to skin the new one in the current best practice way using the same logo and colour scheme.

We just want a basic skin for now, not too much effort.

3.5.6 Introduction of Cloud Based Analysis and Processing Tools

The firm will work with the LSD and PPD to determine possible cloud based tools, such as Google Earth Engine and cloud hosted Jupyter notebook, which they can use in their workflows. Other possible tools include cloud based drone imagery processing for LSD.

We want to provide users with a nice interface for performing CRUD operations on metadata. geonetwork seems like a good fit for this job.

However, we need to investigate on the current state of the integration between geonode and geonetwork:

• Is it still possible to use a geonetwork catalog as a backend to geonode?
• Is it possible to make this geonetwork catalog be read-write or read-only from the geonode side?

https://docs.google.com/document/d/1TlMkh4SxcnhT8wWv83yDx--XuUVrF-ZA987rJF9815Q/edit#heading=h.k9i5gxiqwz1g

After #33 is done, or while it is being finalized we need to deploy the tech stack into the production environment.

• - Rancher and Kubernetes cluster configuration

• - minIO

  • - Setup a mount for raster data. It should be properly structured in order to offer dedicated areas for each stakeholder, including a production and a staging area for each, as specified in #25
  • - Setup a mount for the GeoServer data dir too
  • - Setup a mount for the django media dir too (to store uploaded user profile pictures)

• - Install PostGIS data DB

  • - Configure appropriate schemas for each stakeholder, including those for production and staging as specified in #26

• - Install and configure GeoServer

• - Install and configure Geonetwork - This will also require a DB for storage of metadata records

• - Install and configure Geonode

  • - Also needs a DB
  • - Setup Geonetwork as the metadata catalogue

Write a short guide on how to connect to the staging environment

Please include:

• How to use rancher in order to access the environment
• How to use kubectl to monitor the cluster (with rancher and with a local terminal)
• How to connect via SSH
• How to access the various databases (geonode, geoserver, metadata)

Printing of Dominican national topo maps shall be done from QGIS, so we need to develop suitable QGIS map layout templates for all scales:

• 1:50000
• 1:10000
• 1:5000
• 1:2500

Write a document describing how data management workflows should be done.

This document shall describe workflows for both dataset initial ingestion and further modifications, describing the following:

• Staging area -> process -> validate -> production area
• Creation and validation of styles
• Creation and validation of metadata records
• Publishing on GeoServer and GeoNode

The document shall be created here:

https://docs.google.com/document/d/1rcoH0c1RpCguuPFLe92V-2xdipJU9ODwItOS_XPe-GM/edit

Write a document specifying the conventions for naming DomiNode resources.

The document shall be created here:

https://docs.google.com/document/d/1ttyl9iSCs3uF0HEi8az77bIsZjdIae9uDgcM5P-vsjI/edit

https://docs.google.com/document/d/197UlgpDycsKU2206Bv1sgm_GiwxuurZFGBwIpzPyZoQ/edit#heading=h.95ipcbtpz3ul

https://docs.google.com/document/d/1y9YGe7o1r286ZWtP1Hxy2RmG-228v_2n0XlNHmqBAHU/edit#heading=h.wc70n4w8x69s

The GoCD currently operates four (4) high accuracy GNSS base-stations, which report observations in RINEX, Leica, and Trimble formats. These observations are useful for the post-processing of GNSS observations by surveyors and other data collection personnel. It is the intention of the GoCD to consolidate the collection of these data and streamline access.

The firm will work with the GoCD to consolidate all CORS data on a single server and develop a web based access to this server with read only access (as detailed in Task 3.2.2 of these Terms of Reference).

To streamline the use of these data, the firm will develop an app to extend the DomiNode interface to allow for easy access to the CORS datasets. The app shall:

• Provide an interactive web-map based view of station locations (as points) and status (online/offline/under repair). Additional details such as station names, coordinates, owner and other details deemed necessary by the client may be included in a popup or sidebar for each point on the map.
• Provide the ability to select a site, then as a sidebar or popup interface provide an interface for the user to select a date range of observations to download.
• Provide the ability to control user access to the app by DomiNode administrators. Full public access will be restricted and downloads will require a user account allowed to download by administrators.
• Allow the download of data as a .zip file.
• Implement basic error handling within the app and will include messages on the interface in the case that the date range selected is invalid or data is unavailable.
• Provide warnings to the user in case they have selected a large download (over 50 MB).
• Provide a disclaimer before download and terms of use for user acceptance. The text of this disclaimer will be provided by the LSD.
• Downloading will require confirmation that the user downloading is human (Captcha or similar).
• Human readable logs will be kept of access to the app along with user logons and downloads.

Taken from slack message:

I pointed out the following to Gavin in respect of the Dominode:

• the Coat of Arms must be at the top of the page
• the link to the homepage says Geonode instead of Dominode
• Layer thumbnails do not load
• Legend thumbnails do not load
• my credentials don't get passed on to Geoserver from GeoNode (after signing in)
• clicking "View on Site" on Django User Details page sends user to an invalid page (URL is incomplete)

This is still to be better described

We need to send a patch to Charles's customization repo, in order to make it work against recent geonode.

Write a document specifying how to do versioning of DomiNode resources

The document shall be created here:

https://docs.google.com/document/d/1KhOmdEJsJYuKyOPhO-CdALWWWQvHlQqEYtJo4SuHau0/edit#

https://docs.google.com/document/d/1-Y0MhmB4eQ6VeaV8d3UrmCxRMg24hzbdRIbswzpW0kc/edit#

https://docs.google.com/document/d/1y9YGe7o1r286ZWtP1Hxy2RmG-228v_2n0XlNHmqBAHU/edit#heading=h.wc70n4w8x69s

not sure we have to implement it - not mentioned in contract

clarification from Collin email 8 June:

When the TOR was initially written, we felt that the implementation of a field surveying application would be out of scope given the available budget. So only a plan, which the government could act upon if funding became available from a different source, was requested. If doing more is within scope and the line agencies find this useful then the PCU would go ahead with a full implementation.

https://docs.google.com/document/d/197UlgpDycsKU2206Bv1sgm_GiwxuurZFGBwIpzPyZoQ/edit#heading=h.95ipcbtpz3ul

Write a short section on our internal docs detailing how to access, use and confgure the minio instance that is currently deployed to staging environment

https://docs.google.com/document/d/1eqjlYGl1s9GJiq6a7Sbh6sBotU101Jc0gtpvl3GDgxM/edit#heading=h.tza7cfsvmcox