opendronemap / webodm Goto Github PK

Suggested approach since it involves communication with a third party component (node-OpenDroneMap).

1. Create python server stub from node-OpenDroneMap's OpenAPI definition (swagger.json).
2. Call client_api.py methods to test results.

File "C:\Users\piero\Desktop\WebODM\nodeodm\api_client.py", line 64, in new_task
data={'name': name, 'options': json.dumps(options)}).json()
File "C:\Users\piero\webodm\lib\site-packages\requests\api.py", line 110, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Users\piero\webodm\lib\site-packages\requests\api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\piero\webodm\lib\site-packages\requests\sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\piero\webodm\lib\site-packages\requests\sessions.py", line 596, in send
r = adapter.send(request, **kwargs)
File "C:\Users\piero\webodm\lib\site-packages\requests\adapters.py", line 473, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', TimeoutError(10060, 'A connection attempt failed because the connected party did not properly respond a

requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', TimeoutError(10060, 'A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond', None, 10060, None))

I'm currently working on designing the dashboard.

Processing the odm_data_bellus dataset with --resize-to 1800 option, the task remains stuck on running. node-OpenDroneMap successfully processed the images.

Ideally each project can be accessed between multiple users of the system (but we want to control who has access to what, maybe some users have only read only access, maybe if they are part of a group they can do certain edits, etc.).

Useful links:

http://django-guardian.readthedocs.io/en/stable/
http://www.django-rest-framework.org/api-guide/permissions/

I started a processing node using https://github.com/pierotofy/node-OpenDroneMap .
I can access it at 127.0.0.1:3000
I started WebODM at 127.0.0.1:8000 and added a processing node (127.0.0.1 , port 3000). WebODM tells me that my processing node seems to be offline.

In the next few weeks I'll dedicate some time to write the code necessary to create new ProcessingNode instances and have them communicate with https://github.com/pierotofy/node-OpenDroneMap. I will likely spend some time tweaking node-OpenDroneMap to take care of authorization issues and to make the process of setting up new `ProcessingNode instances easy.

After careful evaluation, I think I will not add Celery as discussed in chat to the mix; the additional complexity added by Celery does not seem to simplify much the problem of distributing tasks to multiple nodes. This is particularly true because the number of processing tasks that we can expect are not many, so I think a simpler cron-like lib, with some custom logic, a priority queue and basic error handling/retry will serve the purpose (hopefully) better.

A new version has been announced: http://www.django-rest-framework.org/topics/3.5-announcement/

Adds some features such as swagger docs generation and test file uploads / downloads.

Need to create a thread / cronjob that performs the following:

• Checks for tasks that need to be processed, and starts processing
• Updates processing node info

WebODM should facilitate the planning of missions.

Input:

• area(s) to cover (polygon) - if multiple areas, treat as multiple missions?
• angle(s) to take photos
• altitude
• hazards (is this a thing handled by mission planning software?)
• camera specs
• desired overlap
• ...

Output:

• 3D linestring representing flight path
• [flight planning output]
• ...

The dashboard is going to be a client side app using React that uses the API.

I'm currently working to add support for initializing a working environment in Docker. This will make it easier for people to contribute and will allow users to get started more quickly.

Will also add instructions in the readme.

Some people are experiencing issues using Docker when rebuilding images.

A script should be created to handle the following tasks:

• Start WebODM
• Stop WebODM
• Rebuild From Scratch Images, ignoring cache and removing the proper volumes.

Once library requirements have been assessed, change Python version from 2 to 3.

Some formatting issues are still present in the administration area (due to collision with the bootstrap theme and the default admin CSS).

The admin area should be checked and fixes should be applied where necessary.

Hi,

I upgraded my system to Ubuntu 16.10, and I can't get WebODM to run.
I'm getting this error :
E: Version '2.3.0+dfsg-2.pgdg80+1' for 'postgresql-9.5-postgis-2.3' was not found E: Version '2.3.0+dfsg-2.pgdg80+1' for 'postgresql-9.5-postgis-2.3-scripts' was not found E: Version '2.3.0+dfsg-2.pgdg80+1' for 'postgis' was not found ERROR: Service 'db' failed to build: The command '/bin/sh -c apt-get update && apt-get install -y --no-install-recommends postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR=$POSTGIS_VERSION postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR-scripts=$POSTGIS_VERSION postgis=$POSTGIS_VERSION && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100

Should I change something in the db/Dockerfile ?

Thank you.

I currently skipped the parts of code to handle GCP files. This will need to be added at some point.

WebODM should keep a log of operations, such as when a task started getting processed, whether a task succeeded, whether a task failed and display these information in the menu bar.

Geotiffs created using GDALRaster.transform result in large files because of lack of compression. Perhaps where necessary we should call gdal_translate -co compress=DEFLATE input.tif output.tif (or the equivalent using the python API).

Before the project proceeds any further I would like to ask the developers which version of python the would consider using?

I would opt for python 3 as in my experience I see python 2 being slowly phased out in favor of 3 and virtually all geospatial libraries now they providing support for it.

Thoughts?

Fixtures are a little problematic due to the fact that we catch signals and have first run initialization commands that do not play well with fixtures.

1. Remove test_projects and test_users
2. Create base class for tests that calls boot code
3. Make all tests inherit from such base class
4. Create test data programmatically via .create()

[What does Cluster Management mean in the context of WebODM?]

From #1:

By "cluster management" (there's probably a better term for this) I mean the management of distributing image processing tasks across multiple server instances.

This image might clarify a bit:

The Web Manager is WebODM.

The REST API is https://github.com/pierotofy/node-OpenDroneMap, http://nodeodm.masseranolabs.com/

The WebODM user interface needs to surface the following functionality:

• task management - creation and status
• model display - allow orthophotos + 3D models to be browsed
• cluster management - ???
• mission planning - is there existing FOSS for this?
• documentation

@pierotofy can you clarify what you're thinking around cluster management?

I'm also thinking that model display and mission planning might be better suited as their own applications, integrating with the WebODM API according to their needs.

Thoughts?

Throw some sort of JS error to confirm, and clean up the mess (since it's probably easier than trying to implement "re-start upload" for now).

Any React pointers for me on how this could be done, @pierotofy? Happy to take it from there ...

if the upload is quicker than you are at providing a name / clicking "Save" then you are stuck where you cannot submit the job anymore ...

It would be cool to have the ability to measure volumes directly from the map.

I think this could be approached in the following way:

1. Load 3D points, cast them to polygonZ collection, use http://postgis.net/docs/ST_DelaunayTriangles.html to generate a TIN
2. Create stored procedure that given a list of 2D points:

• generates a plane
• extrudes the TIN based on the plane boundaries
• calls http://postgis.net/docs/manual-dev/ST_Volume.html on the resulting extruded TIN

Should also add the possibility to specify a "Cut" or "Fill" property for concave/convex cases.

It might help to warn users properly if they are running 2.7 instead of 3.5.

This will avoid errors messages from popping up in docker-compose.

WebODM should facilitate the creation and management of tasks.

Tasks represent processing steps required to convert a stack of photos (or video) into various ODM outputs.

Tasks contain these pieces of metadata:

• ...
• created timestamp
• updated timestamp

Once a task is done being processed, it would be nice for the user to see a preview and orthophoto extent information directly on the dashboard panel without opening the map.

Although this is something to do toward the end, the Docker command line is still too confusing for many GIS technicians; there's going to be a need for a graphical installer.

ODM-generated orthophotos should be browsable using a Leaflet-like interface.

Needs:

• TMS endpoint as TileJSON or a standard URL template
• geographic extent (potentially in TileJSON)
• zoom extent (potentially in TileJSON)
• ...

Ciao Piero,

Iḿ trying run WebODM from start.sh, but the following errors are listed :

emilio@emilio-laptop:/opt/WebODM$ ./start.sh
Building asssets...
Hash: bcdca8dde46cb9609cf5
Version: webpack 1.13.3
Time: 52662ms
Asset Size Chunks Chunk Names
main-bcdca8dde46cb9609cf5.js 1.9 MB 0 [emitted] main
css/main.css 41.3 kB 0 [emitted] main
[0] multi main 28 bytes {0} [built]
+ 255 hidden modules

WARNING in .//leaflet-measure/dist/leaflet-measure.js
Critical dependencies:
1:113-120 This seems to be a pre-built javascript file. Though this is possible, it's not recommended. Try to require the original source to get better results.
@ .//leaflet-measure/dist/leaflet-measure.js 1:113-120
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 6 hidden modules
Child extract-text-webpack-plugin:
+ 5 hidden modules
Child extract-text-webpack-plugin:
+ 14 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Running migrations
Traceback (most recent call last):
File "manage.py", line 22, in
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 367, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 341, in execute
django.setup()
File "/usr/local/lib/python2.7/dist-packages/django/init.py", line 27, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python2.7/dist-packages/django/apps/registry.py", line 108, in populate
app_config.import_models(all_models)
File "/usr/local/lib/python2.7/dist-packages/django/apps/config.py", line 199, in import_models
self.models_module = import_module(models_module_name)
File "/usr/lib/python2.7/importlib/init.py", line 37, in import_module
import(name)
File "/opt/WebODM/app/models.py", line 23, in
from nodeodm.models import ProcessingNode
File "/opt/WebODM/nodeodm/models.py", line 10, in
from .api_client import ApiClient
File "/opt/WebODM/nodeodm/api_client.py", line 9, in
from urllib.parse import urlunparse
ImportError: No module named parse
Unhandled exception in thread started by <function wrapper at 0x7fbf61b15848>
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/django/utils/autoreload.py", line 226, in wrapper
fn(args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/runserver.py", line 113, in inner_run
autoreload.raise_last_exception()
File "/usr/local/lib/python2.7/dist-packages/django/utils/autoreload.py", line 249, in raise_last_exception
six.reraise(_exception)
File "/usr/local/lib/python2.7/dist-packages/django/utils/autoreload.py", line 226, in wrapper
fn(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/init.py", line 27, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python2.7/dist-packages/django/apps/registry.py", line 108, in populate
app_config.import_models(all_models)
File "/usr/local/lib/python2.7/dist-packages/django/apps/config.py", line 199, in import_models
self.models_module = import_module(models_module_name)
File "/usr/lib/python2.7/importlib/init.py", line 37, in import_module
import(name)
File "/opt/WebODM/app/models.py", line 23, in
from nodeodm.models import ProcessingNode
File "/opt/WebODM/nodeodm/models.py", line 10, in
from .api_client import ApiClient
File "/opt/WebODM/nodeodm/api_client.py", line 9, in
from urllib.parse import urlunparse
ImportError: No module named parse

Grazie mille.

Emilio Bruno

Installed the Docker image on Windows 10 64-bit. In the terminal window are repeated error messages:

[##mwebapp exit with code 127 (where ## is an integer in the range 31 through 36)
No such file or directory[0m /usr/ bin/env: bash
No such file or directory/usr/ bin/env: bash
No such file or directory/usr/ bin/env: bash
(repeated a total of 10 times)

Looking at the webapp container log in Kitematic, the "No such file or directory/usr/ bin/env: bash" also shows up 11 times.

Took a while, but figured out how to open up the WebApp. Am processing a simple task, and it seems to be working. So are those error messages of concern?

WebODM should implement the following endpoints:

• authorization
• list projects (Project) ☑️
  • [filtered by User] ☑️
  • [filtered by Group]
  • [filtered by geographic extent]
• list tasks (Task) ☑️
• create tasks (Task) ☑️
• update tasks (Task) ☑️
• list/add/change processing nodes ☑️
• ability to start/stop processing a Task☑️
• get project info ☑️
  • TileJSON / equivalent
  • 3D model stuff
• create project ☑️
• create mission
• get mission info
• ...

On dashboard load, check server side if there's at least one node online. It makes the user experience nicer (instead of uploading files and finding out later that the nodes are offline).

ODM-generated 3D models should be browsable using a Cesium-like interface.

Needs display and manipulation of the following:

• Point clouds
• Textured Mesh

Hi! Such a great work! I'm very interested in the project, however I can't managed to make it work yet. When I launch the web interface and add a task, system says "There are no processing nodes available. Make sure at least one of them is reachable.". How can I create a processing node? I thought this was already included with this interface, but I was mistaken. Is there any tutorial available or walktrough process from the image loading to the final orthomosaic? A little help would be nice. Thanks in advance! Cheers!

Leaving the dashboard and pressing the back button on the browser currently loses the application's state. This should be fixed with something like react-history.

Probably using https://github.com/lord/slate is a good option. Swagger is too picky and limited.

In regard to a discussion which began here: #40 (review)

Is scheduler.py code reinventing the wheel (functionality that already exists in Celery)?

In essence it [the code] is replicating some of the Celery features (it is a task processing queue). What is driving me to re-implement this mechanism is that so far, the task processing involves more than a single processing step (think input images in ---> assets out). Maybe I'm over-thinking it, but so far I found these cases where Celery does not seem like it would fare so well.

1. Input images in --> assets are not what I expected. I should be able to restart the process without re-sending the images, just by changing the processing options

2. Console output: how to retrieve real-time console output from the opendronemap process?

3. How to protect processing nodes (authentication) without making it difficult to setup? Ideally a processing node should be (optionally) protected by a static token, or a userID/password combination that generates a temporary token. The docs here hint at using public/private key exchange for security: http://docs.celeryproject.org/en/latest/userguide/security.html (I'm not sure I'd want to go down that route)

At some point it could be beneficial to create a basic infrastructure for allowing people to donate server resources to process maps.

It could be as simple as having volunteers install the processing software, having a sign-up form, a web page with the list of volunteers, and some code to check which nodes are online.

Developers need an entry point to authenticate with the Django Rest Framework and retrieve a token for use in subsequent calls. http://www.django-rest-framework.org/api-guide/authentication/

Ran into this issue while running docker-compose up

Step 13 : RUN curl --silent --location https://deb.nodesource.com/setup_6.x | sudo bash -
 ---> Running in 6537e09cc0d5
/bin/sh: 1: sudo: not found
ERROR: Service 'webapp' failed to build: The command '/bin/sh -c curl --silent --location https://deb.nodesource.com/setup_6.x | sudo bash -' returned a non-zero code: 127

My specs:

• Ubuntu 14.04 server

dmb2@OpenDroneMap:~/WebODM$ docker version
Client:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 22:00:36 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 22:00:36 2016
 OS/Arch:      linux/amd64
dmb2@OpenDroneMap:~/WebODM$ docker-compose --version
docker-compose version 1.8.1, build 878cff1

It would be cool to have the ability to display an automatically generated elevation model along with the orthophoto.

@smathermather how would you go about generating an elevation model from the current outputs of opendronemap (lastools come to mind, but they aren't OSS)?

Somewhat linked to #21, there's going to be a button for uploading images directly from the dashboard. This will in turn automatically create the first project and task. These are all shortcuts for the sake of ease of use, and a more in depth project and task management UI will be created.

Hi there
I am not sure if this error is related to webODM or the processig node

My setup is two different AWS instance (t2.medium) one running WebODM and the second one running Node-ODM, both installed with Dockers

Killed
[ERROR]   quitting cause: 
	/code/SuperBuild/install/bin/pmvs2 /var/www/data/813e7f99-16a8-4ec7-b5d8-edaa958957aa/pmvs/recon0/ option-0000
returned with code 35072.

any help will be very helpful

Users should be notified via e-mail when processing has finished.

An ability to overwrite the default settings is usefull in numerous occasions.

This can be implemented easily by appending :

try:
from .local_settings import *
except ImportError:
pass

ofcourse local_settings.py should be added in .gitignore as well

Many people are experiencing issues setting up a processing node manually using docker (because of the intricacies of docker).

I will add a docker-compose file and supporting configuration to setup automatically a processing node as part of the docker compose up command.