linksmart / historical-datastore Goto Github PK

Discuss and improve the simplified API:
https://docs.linksmart.eu/display/HDS/Simplified+SensorThings+API

Don't like the new name? Visit: https://docs.linksmart.eu/display/LSI/OGC+SersorThings+Service+Candidates

Considering the status of current OGC-ST implementations and the complexity of creating another fully-compliant OGC-ST service, we would like to propose the use of OGC-ST data model alongside a custom and simplified RESTful API. This task will focus on creating OpenAPI specs for the aforementioned service.

[LS-255] created by carvajal

Also aggregation?

Critical TODOs:

• Validate name
• Validate MQTT source (uri, qos 0-2, scheme==ssl when caFile is given, etc)

And return appropriate errors!

[LS-484] created by tavakoli

[LS-336] created by jannis.warnat

[LS-123] created by marco.jahn

HDS needs an end-to-end integration test to avoid issues such as LS-379

DoD:

• A team to support support of OGC Sensorthings API for HDS is formed
• Roles are defines for each member

[LS-286] created by carvajal

It is much more convenient to start submitting SenML data without the need to register data sources beforehand.

An optional configurable feature can enable automatic registration of data source upon first submission of a SenML entry.

Broker-based resource(thing) and service registration.

Related to:
https://docs.linksmart.eu/display/LSI/2017/10/10/Broker-based+Dataflow+Proposal

[LS-482] created by tavakoli

[LS-475] created by tavakoli

Jannis Warnat: when I register resources in HDS that require a password for the MQTT broker, these passwords are not stored in LevelDB (only *****). Do you know whether there was a conscious decision to do so for security reasons or can I change this? I understand that we probably do not want users to see the password by just checking the /registry endpoint. But it would be much easier if HDS could retrieve the passwords from LevelDB after a restart. I guess I would have to analyze where the json is "marshalled".

After the Broker fails all components must re-send their registration info.
This task may be split several tasks or a sub-task per component must be created.

Deleting measurements and retention policies sometimes create error messages like this:

Error deleting data source: Error removing historical data: Statement error: shard 1: engine is closed

Todo:

• Self-signed CA
• Client key
• Client cert

Sample code: https://github.com/eclipse/paho.mqtt.golang/blob/master/cmd/ssl/main.go

Compliance to OGCSensorThings > =1.0: IoT Device Gateway, Resource Catalogue, IoT Agents, Historicla Datastore, Sensor Platform

ToDo:

• Remove local clients and use interfaces
• Remove notifications and instead pass control functions to registry
• Remove elog nonsense and log flags, make time optional
• Cleanup main
• Remove Mongodb
• Remove multi sc support
• Improve file naming

[LS-479] created by tavakoli

[LS-373] created by tavakoli

DOD :

• The documentation for HDS  follows the Linksmart component template
• At least one person from LS reviews it
• Explain REST / MQTT APIs, supported format (senml), etc.

[LS-279] created by devasya

Create Swagger specification (either inside components or external)

DoD:

A specification for MQTT apis is defined

This issue is created to support Web of things Things description specification for storage of the metadata.
Example

{
    "@context": "https://www.w3.org/2019/wot/td/v1",
    "id": "urn:dev:ops:32473-WoTTemp-1234",
    "title": "MyLampThing",
    "securityDefinitions": {
        "basic_sc": {"scheme": "basic", "in":"header"}
    },
    "security": ["basic_sc"],
    "base": "/data/",
    "properties": {
        "source" : { 
            "type": "float",
            "forms": [{
		"href": "https://gw1.iot/status", 
	         "contentType": "application/senml+json",
		 "op": ["readproperty", "writeproperty"],
		 "security": ["basic_sc"]
	    }],
        },
        "storage" : { 
	   "readOnly" : true,
            "type": "float",
            "forms": [{"href": "/data/urn:dev:ops:32473-WoTTemp-1234","security": ["basic_sc"]}]
        }
    }
}

In the above example. properties/source property represents the original device that produced the data. On the other hand, properties/storage represents the data stored as part of historical datastore.
The sample senml:

[
  {"bn":"urn:dev:ops:32473-WoTTemp-1234","bu":"Cel","t":1.276020076e+09, "v":20},
  {"t":1.276020091e+09, "v":23.5},
]

How it is now:

{
    "bn": "string",
    "bu": "string",
    "bts": "int",
    "bte": "int",
    "e": [
{
    "n": "string",
    "ts": "int",
    "te": "int",
    "mean": "float",
    "stddev": "float",
    "sum": "float",
    "min": "float",
    "max": "float",
    "median": "float"
}
    ]
}

The aggregate could be instead part of the name. "te" is not needed. The rest can fit into SenML.

ToDo:

A wiki page comparing OGC sensorthings servers  (FROST and GOST ) is prepared

This should allow submission of data to /data endpoint without explicitly providing the ID. The ID will be retrieved based on the SenML name field.

Historical Datastore should be improved to support expected use-cases with simple and pragmatic APIs.

Use cases:

• Low dimensional time-series storage and retrieval 
• High dimensional time-series storage and retrieval
• Log storage and retrieval
• Lightweight storage on IoT devices by using embedded DBs such as Leveldb, Boltdb, Sqlite

Features:

• Retention 
• Simple aggregation (persistent) 
• HTTP API 
• MQTT connector 
• Annotation
• Query with mathematical functions

[LS-446] created by tavakoli

When two measurements are published together with the same topic in a single SenML message, HDS only subscribes once. If one of the measurements are not registered, the message will be dropped completely resulting in loss of both measurements.

[LS-112] created by marco.jahn

To optimize the backend and fix race condition issue between data and aggregation storages, retention policies can be instead configured prior to runtime and created during startup. The datasources then will only be able to use the preconfigured retention policies.

The openapi spec describes /registry/

/

/

.

The default configuration of HDS is not working
DoD:
HDS docker image runs without issues

Binary distributions should be build with Travis and published along releases.

Datasource v0.5

{
      "id": "9b1921c1-b20f-496a-a127-b5a01f0a91a5",
      "url": "/registry/9b1921c1-b20f-496a-a127-b5a01f0a91a5",
      "data": "/data/9b1921c1-b20f-496a-a127-b5a01f0a91a5",
      "resource": "13/0/49/4/power",
      "meta": {
"name": "Philip's laptop"
      },
      "connector": {
"mqtt": {
  "url": "ssl://demo.linksmart.eu:8883",
  "topic": "LS/v2/ZGW/linksmart-philip/senml",
  "qos": 1
}
      },
      "retention": "1w",
      "aggregation": [
{
  "id": "4efaf6fb",
  "interval": "1m",
  "data": "/aggr/4efaf6fb/9b1921c1-b20f-496a-a127-b5a01f0a91a5",
  "aggregates": [
    "mean"
  ],
  "retention": ""
}
      ],
      "type": "float",
      "format": "application/senml+json"
}

Problems

• two identifiers (id and resource) needed during different operations.
• resource is the name attribute in SenML
• "format" carries no important information (it's the same for all)
• url, data, aggr.data are shows as absolute paths but in fact they are relative to the public endpoint.

[LS-447] created by tavakoli

DoD:

• A table of integration test is created to measure the coverage 
• Part of these test is performed

[LS-282] created by devasya

[LS-481] created by tavakoli

Because of this bug, every time the retention period is changed, HDS will just switch to a new measurement under the new period in influx without moving the data.

Agree or update the LinkSmart specs in https://docs.linksmart.eu/pages/viewpage.action?pageId=5570896

for more see https://docs.linksmart.eu/display/LSI/Data+Flow+Proposal

The filtering API of HDS can be simplified by removing the {type} and taking the /many behaviour as default. The efficiency effects should be considered.

• Get datasource with name:
  /registry/one/name/equals/home/bedroom/temperature
• Filter all datasources with prefix home in name:
  /registry/many/name/prefix/home
• Filter all datasources based on address meta field:
  /registry/many/meta.address/contains/Bonn

[LS-430] created by tavakoli

[LS-480] created by tavakoli

The go package name should be changed from code.linksmart.eu/hds/historical-datastore to github.com/linksmart/historical-datastore.

[LS-371] created by tavakoli

[LS-249] created by tavakoli

Supporting aggregation function in HDS:

• Should support API based aggregation (Separate API for creation of aggregations)
• Support basic aggregations such as
  • Mean
  • Sum
  • Last value
  • custom with a go or python code
• This aggregation should be triggered via
  • Periodic (cron based)
  • Data based (triggered whenever a new data point is inserted)

DoD:

A registry API has nextLink way of pagination

Creating integration tests between the Service Catalog and each Linksmart® Service