When creating the groups for project 13505 (Swipe to help MSF in Douentza, Mali (1)) some groups (31 out of 1138) where not created correctly.

This file contains the geometries of the groups:
groups_13505.zip

As mentioned in #42 we changed our database infrastructure from mysql to psql for easier json handling.
Therefore we have to:

• setup postgres databases (dev is already done)
• add postgres credentials to cfg
• go over the scripts rewrite the db connection, change the variable naming

It looks like the version for GDAL is pretty old compared with what is available. Is there a specific reason for keeping it at this version?

Ideally, we should be able to use the exact same file in the frontend and the backend

we need to add this to the BaseFuntions.

12-20 17:28:12 WARNING === === === === ===>>> started update <<<=== === === === ===
12-20 17:28:12 WARNING ALL - get_environment - use development instance
12-20 17:32:04 WARNING === === === === ===>>> started update <<<=== === === === ===
12-20 17:32:04 WARNING ALL - get_environment - use development instance
12-20 17:32:05 ERROR ALL - run_mapswipe_workers - got exception
Traceback (most recent call last):
File "run_mapswipe_worker.py", line 63, in
b.run_update(args.modus, args.filter, args.output)
File "/python-mapswipe-workers/mapswipe_workers/basic/BaseFunctions.py", line 365, in run_update
update_users_postgres(firebase, postgres)
File "/python-mapswipe-workers/mapswipe_workers/basic/BaseFunctions.py", line 225, in update_users_postgres
for user in users:
TypeError: 'NoneType' object is not iterable
12-20 17:32:05 WARNING XXX XXX XXX XXX XXX>>> errored update <<<XXX XXX XXX XXX XXX
12-20 17:32:05 WARNING zZz zZz zZz zZz zZz>>> sleep for 900 seconds <<<zZz zZz zZz zZz zZz

Adapt this to the new structure.
We need to export:

• users and stats
• projects
• project results

Improve documentation on how to add new projects.
Document alongside the test scenario described in #57

Currently projects and all groups per project stay in firebase. Also imports stay in firebase forever. An "archive" function would help us to reduce the storage costs in firebase, increase firebase performance.

Mapswipe Workers will look at all projects in Firebase and archive them based on certain terms:

• archive flag = true (The idea is that project managers can set an archive flag to true if they decide the project is finished)

Additionally we might think about doint this automatically:

• project.progress = 100%
• No new results for e.g. 3 months

Note: Imports (projects draft) will now be deleted after successfull creation of the project

The archive function will:

• set status of a project to archived
• will delete all tasks and groups in firebase

during the import a zero division error occured in the import workflow.

In this example, not all groups are created correctly

Now that Travis.ci is setup we should add a linter to automatically run on the Python code.
Note that this will probably require a lot of formatting changes in the existing code.

To check whether a project exists in firebase, we should also check if there are groups in firebase.

create and review the update progress function for projects

While running our scripts we got this error:

google-mapswipe-workers: run_export.py: error occured
(<class 'requests.exceptions.HTTPError'>, HTTPError(HTTPError('401 Client Error: Unauthorized for url: https://msf-mapswipe.firebaseio.com/projects.json',), '{\n  "error" : "Unauthorized request."\n}\n'), <traceback object at 0x7fa90708b1c8>)

It seems is if it occurs randomly, after several minutes the error didn't occur anymore.

This seems to been a know issue with oauth2client, which us used by pyrebase: thisbejim/Pyrebase#117

In order to handle additional project types we need to restructure the mysql db.
We need a project table with attributes for:

• id(integer)
• name(text)
• objective(lookFor)(text)
• project type(integer)
  • in order to distinguish results from different project types like built area or footprint.

part of #38
For the results table base attributes are:

• Task-id
• User-Id
• Project-Id
• Timestamp

Specific per project type:
Built area project:

• result (integer of user interaction)
  Footprint project:
• result (integer)
• footprint geometry (geom)
  • We will store if necessary every task of a specific project type, for the export we can then join the result with the task geometry

we need to check that error handling is working properly.

At the moment we store the progress of projects which is the amount of tasks which have been evaluated divided by the amount of tasks which need to be evaluated in a text file on the nginx server.
In order to keep all data in one central place like mentioned in #49 we will setup an additional table with timestamp, project-, and contributors stats.

We concluded that it's worth to better differentiate the functionalities of our data stores.
Firebase is for communication with the mapswipe client app only. Right now we store everything except results there. We want to shift that storing functionality completely over to the postgres database. Firebase remains the database for active projects.

Improvements needed on postgres side:

• Setup tables for group information (right now only tasks are stored). This table provides information on which project the groups are assigned to, the group id, completed and needed count.
• At the moment we store not all project information we have in firebase
• Additionally we add an attribute to each project if it is archived

What is the Backup routine of the new server?

In addition to update the progress per project in firebase, we want to add an export to a file in order to show live statistics of the trend of progress per projects per time.

Make sure that after init() a project has all attributes

We create a kml or geojson during the import. This file contains the geometries for each project. The file should be stored in data/input_geometries folder and should use a better naming of the files.

e.g. data/input_geometries/input_geometries_{id}.kml or data/input_geometries/input_geometries_{id}.geojson

Write a script which transfers the data of the MySQL database to the new Postgres database.

Currently logging info does not follow a common format.

E.g.:

2018-10-25 23:33:52,823 - root - WARNING - got project contributors from mysql for project: 13531

2018-10-25 23:33:53,006 - root - WARNING - log contributors to file for project 13531 successful

project id should be easier to spot, as well as the corresponding function

use a single sql statement to delete information in postgres instead of several functions

come up with a mechanism to delete project data in firebase and being able to roll back.

When using docker the database tables are created by the following script: docker/postgres_create_tables.sql

Is there another use case for setup_database_tables.py?

We should remove it entirely or link to the existing sql script.

Write a script which transfers the data from the Firebase instance to the Postgres database

When many results are uploaded to the mapswipe database, we may not be able to import them quick enough.

Potential bottlenecks are:

• insert into mysql database and check for duplicates
• remove entries in firebase

Now that Travis.ci is setup we should add extensive tests and aim for an 80% test coverage.

When importing a new project, we need to adjust some things.

In project:

• add the url to the tileserver for each project

In Tasks:

• remove projectId from tasks, since it's already in the groups (project id can be generated from the task_id, and is also in the group)
• feature_id --> featureId

• we need to make sure to have an user
• the script should take an import and insert it to firebase
• the trigger the run_import function to import the project in firebase, create groups etc and do the stuff in postgres
• simulate results for the imported project
• get projects
• get group per project
• randomly generate results for tasks
• run transfer results
• run update
• run export

The Api should have a landing page with description and link to the documentation.

Let's get started with the basic structure for this new project type.

we need to define the project json, which attributes will this project type need.

How will groups look like?

What are the expected results?

• Setup a new server (Google Compute Engine)
  • Configure server backup
  • Install Python-MapSwipe-Workers (do NOT run the containers)
• data transition to new server
  • Import Firebase data into Postgres (#66)
  • import, projects, groups
  • compare amount of rows for each table firebase vs. psql
  • Stop import process on the old server
• Start import process on the new server
  • update web import formular
  • generate a test import in prodcution instance
  • test import was successfully created in production instance
• Start transfer results on the new server
  • Migrate MySQL data (results) to Postgres (#65)
  • test transfer results with results generated from the production app
  • stop transfer results on the old server
  • transfer all results from mysql to postgresql which have not been transfered yet
  • start transfer results process on the new server
• Start update on the new server
  • test update for a few old projects
  • update completed count of groups in postgres
  • update all projects in firebase using the new workers
  • stop update on the old server
  • start update process for active projects
• Start export process on the new server
  • test export for a few old projects
  • manually generate exports for all projects
  • stop export on the old server
  • start export process for all active projects
  • make sure http://api.mapswipe.org/ points to the correct ip-address
• Redirect API domain name to new server
• Shutdown old server
  • create a snapshot to backup data in mapswipe-workers compute engine
  • create a snapshot to backup data in cloud sql mapswipe-db-3
    - created dump at https://console.cloud.google.com/storage/browser/mapswipe-db-dump?project=msf-mapswipe
  • wait for 2 weeks and monitor new server and processes
  • shut down compute engine mapswipe-workers
  • shut down cloud sql mapswipe-db-3

we should create an index.html file for the nginx server
this html file will provide a link to the documentation of the mapswipe api at read the docs

and furthermore show all available data sets in the respective folders.

currently there is an api documenation for the mapswipe api in a google doc. This should be replaced with the documenation within this github repository

In addition to update the amount of contributors per project in firebase, we want to add an export to a file in order to show live statistics of the trend of users per projects per time.

this will make it easier for us to delete projects in firebase and all corresponding results in postgresql.

if the import of a project is not successful, we should delete the local files, e.g. import_1006.kml

Show all available data sets in the respective folders

+-- index.html
+-- stats.json
+-- projects.json
+-- users.json
+-- input_geometries
|     +-- input_geometries_1.kml
|     +-- input_geometries_3.geojson
+-- progress
|      +-- progress_1.json
|      +-- progress_3.json
+-- results
|      +-- results_1.json
|      +-- results_3.json
+-- tmp
|      +-- results.json
|      +-- raw_tasks.txt

Currently our error messages do not provide much detail.

ERROR    FileNotFoundError processing TIN / NPI message! In <module>, the following error happened -
 2 at line 473.

Let's change this.

We will change the import of projects, that every task is also stored in mysql.
This will help us to create exports of results joined by task specific attributes (like geometries)

We use numpy format for the docstrings

Test compatibility of MapSwipe version 1.2 and new MapSwipe Workers

@ericboucher already started to setup docker for the mapswipe workers.

Currently we are not using docker at the backend. What's the current status and what stops us from solving this issue?

Test compatibility of http://mapswipe.org/import with the new MapSwipe Workers.

currently some mapswipe data can be downloaded directly from the api:

for example:
http://api.mapswipe.org/stats.json

We should enable https for this to allow easy integration into other website, services etc.

Development and production instance of MapsSwipe Workers should not run in the same container. Therefor there is no need for a shared configuration file (no need for the fields "dev_psql" and "dev_firebase") and also not for the dev_firebase_admin_auth() function and the dev_psqlDB class in the auth module.

When running the transfer of the results from firebase to a mysql instance the following error occurs:

Unable to load configuration file at ./cfg/config.cfg. Exiting.
Traceback (most recent call last):
  File "transfer_results.py", line 12, in <module>
    from cfg.auth import firebase_admin_auth
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
  File "/data/environments/mapswipe_workers/lib/python3.4/site-packages/mapswipe_workers-0.0.1-py3.4.egg/cfg/auth.py", line 14, in <module>
FileNotFoundError: [Errno 2] No such file or directory: './cfg/config.cfg'

The same errors occurs for the import module, export module and the update module.

• basic.BaseFunction.update_users_postgres()
• basic.BaseFunction.run_delete()
• basic.BaseFunction.run_archive()
• basic.BaseFunction.run_export()
• basic.BaseGroup
• basic.BaseProject.set_groups_postgres()
• basic.BaseProject.set_import_postgres()
• basic.BaseProject.delete_tasks_postgres()
• basic.BaseProject.delete_groups_postgres()
• basic.BaseProject.delete_import_postgres()
• basic.BaseProject.update_project()
• basic.BaseProject.set_project_progress_postgres()
• basic.BaseTask
• ProjectTypes.BuildArea.BuildAreaGroup
• ProjectTypes.BuildArea.BuildAreaProject.aggregate_results()
• ProjectTypes.BuildArea.BuildAreaTask
• ProjectTypes.BuildArea.GroupingFunctions
• ProjectTypes.BuildArea.TileFunctions
• ProjectTypes.Footprint.FootprintGroup
• ProjectTypes.Footprint.FootprintProject.aggregate_results()
• ProjectTypes.Footprint.FootprintTask
• ProjectTypes.FootprintGroupingFunctions
• utils.errorhandling
• utils.slack
• utils.migrate_mysql_psql
• basic.auth.firebase_admin_auth()
• basic.auth.dev_firebase_admin_auth()
• basic.get_api_key()
• basic.get_tileserver_url()
• basic.get_submission_key()
• basic.mysqlDB
• basic.dev_mysqlDB
• basic.psqlDB
• basic.dev_psqlDB
• basic.BaseFunctions.init_import()
• basic.BaseFunctions.init_project()
• basic.BaseFunctions.project_exists()
• basic.BaseFunctions.project_exists_firebase()
• basic.BaseFunctions.project_exists_postgres()
• basic.BaseFunctions.run_transfer_results()
• basic.BaseFunctions.delete_project_firebase()
• basic.BaseFunctions.delete_project_postgres()
• basic.BaseFunctions.delete_local_files()
• basic.BaseFunctions.BaseImport.BaseImport
• basic.BaseFunctions.BaseImport.set_import_postgres()
• basic.BaseFunctions.BaseImport.set_project_postgres()
• basic.BaseFunctions.BaseImport.set_groups_postgres()
• basic.BaseFunctions.BaseImport.set_tasks_postgres()
• basic.BaseFunctions.BaseImport.set_project_firebase()
• basic.BaseFunctions.BaseImport.set_groups_firebase()