MQTT Publish seems to fail randomly:

[2019-07-11 12:47:04 UTC : MQTT : DEBUG] Received: Fail "\NUL\"devices/0ff8373bo/sensors/HU/value{\"value\": \"2.21\"}" [] "string"
[2019-07-11 12:47:11 UTC : MQTT : DEBUG] Received: Fail "\NUL\"devices/0ff8373bo/sensors/TC/value{\"value\": \"6.00\"}" [] "string"
[2019-07-11 12:47:11 UTC : MQTT : DEBUG] Received: Fail "\NUL\"devices/0ff8373bo/sensors/HU/value{\"value\": \"2.21\"}" [] "string"
[2019-07-11 12:48:52 UTC : MQTT : DEBUG] Received: Fail "devices/0ff8373bo/sensors/TC/value" [] "string"
[2019-07-11 12:48:52 UTC : MQTT : DEBUG] Received: Fail "{\"value\": \"6.00\"}" [] "string"
[2019-07-11 12:48:52 UTC : MQTT : DEBUG] Received: Fail "\NUL\"" [] "string"
[2019-07-11 12:48:52 UTC : MQTT : DEBUG] Received: Fail "devices/0ff8373bo/sensors/HU/value" [] "string"
[2019-07-11 12:48:52 UTC : MQTT : DEBUG] Received: Fail "{\"value\": \"2.21\"}" [] "string"
[2019-07-11 12:49:04 UTC : MQTT : DEBUG] Received: Fail "\NUL\"devices/0ff8373bo/sensors/TC/value{\"value\": \"6.00\"}" [] "string"
[2019-07-11 12:49:04 UTC : MQTT : DEBUG] Received: Fail "\NUL\"devices/0ff8373bo/sensors/HU/value{\"value\": \"2.21\"}" [] "string"
[2019-07-11 12:49:12 UTC : MQTT : DEBUG] Received: Fail "\NUL\"" [] "string"
[2019-07-11 12:49:12 UTC : MQTT : DEBUG] Received: Fail "devices/0ff8373bo/sensors/TC/value{\"value\": \"6.00\"}" [] "string"
[2019-07-11 12:49:12 UTC : MQTT : DEBUG] Received: Fail "\NUL\"" [] "string"
[2019-07-11 12:49:12 UTC : MQTT : DEBUG] Received: Fail "devices/0ff8373bo/sensors/HU/value{\"value\": \"2.21\"}" [] "string"

The payload does not seem to be framed correctly: sometime it begins too early or too late.

Currently, the API represent the link between a device and a gateway, e.g. "a device radio data has been received by a gateway". This link is represented as a field in Device data structure:

device: 
{
  id: XX
  gateway_id: YYY
}

It's not the best because:

• finding the devices connected to a gateway takes time, because you need to scan all the devices (thousands) and also verify the authorization.
• a device could be received by several gateways, this is not representable: gateway_id field has only one gateway.

The following API call fails, most likely because of the umlaut letter "ä":

await fetch("https://api.waziup.io/api/v2/devices/MyGuestSensor/sensors/TC/value", {
	method: "POST",
	headers: {
		"Content-Type": "application/json; charset=UTF-8"
    },
    body: JSON.stringify({
		value: "Hällo",
		timestamp: new Date()
    })
});

Returns: 500 Something went wrong

Without the umlaut there are no problems.

Waziup-API might have a space leak: memory grows a lot.

For synchronization mechanisms with the Gateway, an endpoint is required that allows the upload of multiple separated data points. This can be achieved with multiple calls to the POST devices/xxx/sensors/yyy/value endpoint, what is a very inefficient solution.

Based on the /value endpoint (see the swagger spec.), the /values should be like:

Headers:

• Content-Type: application/json

Body: array of data points, can have length 0, ordered by timestamp (ascending, latest value last)
Data Point: a {Value,Timestamp} Object
Value: required, any type, not null
Timestamp: required, timestamp (JavaScript Date)

Example:

[{
  "value": 25,
  "timestamp": "2016-06-08T18:00:00.000Z"
},{
  "value": 27,
  "timestamp": "2016-06-08T19:00:00.000Z"
},{
  "value": 24,
  "timestamp": "2016-06-08T20:00:00.000Z"
}]

Note that the API implementation might require to only submit values that are newer than the existing values at the API database. This might prevent the upload of values from the past when there are newer data points present.

Retrieving data in descending order doesn't return the last datapoint.
Today is 2019-06-04 but it returns data from yesterday:

cdupont@cdupont-XPS:~$ curl -X GET "https://api.staging.waziup.io/api/v2/sensors_data?device_id=UGB-PILOTS_Sensor200&sensor_id=BV&limit=1&sort=dsc" -H  "accept: text/csv;charset=utf-8"
device_id,sensor_id,value,timestamp,date_received
UGB-PILOTS_Sensor200,BV,4.11,2019-06-03T16:15:30Z,2019-06-03T16:15:39Z

Adding just a "date_from" filter makes it return the last datapoint:

$ curl -X GET "https://api.staging.waziup.io/api/v2/sensors_data?device_id=UGB-PILOTS_Sensor200&sensor_id=BV&limit=1&sort=dsc&date_from=2019-06-04T06:43:21Z" -H  "accept: text/csv;charset=utf-8"
device_id,sensor_id,value,timestamp,date_received
UGB-PILOTS_Sensor200,BV,4.11,2019-06-04T16:07:06Z,2019-06-04T16:07:16Z

The objective is to keep compatibility with gateways that still use the V1.1 AP1.
Gateway code: https://github.com/CongducPham/LowCostLoRaGw/blob/master/gw_full_latest/CloudWAZIUP.py
API V1.1: https://api.waziup.io
API V2: https://api.staging.waziup.io

Proposed mappings:

• map api/v1/sensors/XXX/measurements/YYY to api/v2/devices/XXX/sensors/YYY
• map api/v1/sensors/XXX/measurements/YYY/values to api/v2/devices/XXX/sensors/YYY/value

Currently, GET /sensors_data need two parameters: device_id and sensor_id.
Idea is to replace them with plural devices and sensors.
Both parameters can be supplied with a list of values, or a start * for all values.

Example:

curl -X GET "https://api.waziup.io/api/v2/sensors_data?devices=Dev1,Dev2&sensors=*" -H  "accept: application/json;charset=utf-8"

The error appears when there is no gateways. In that case, Keycloak returns:

[{"scopes":["gateways:view"]}]

Which doesn't contain rsname.

• limit/offset in notifications and socials
• ontologies checking in sensors and actuators
• check existence of gateway in devices
• check existence of devices and gateways in projects
• permissions for projects, gateways, notifications
• replace q request
• domain in projects

Improvements:

• change exception package
• add messages on throw
• remove type checks in Orion
• remove warnings

Notifications triggering will call socials/batch.
This in turn requires Keycloak authorizations:

• "view users" to retrieve the user twitter/phone number
• "manage users" to write new SMS amount.

However, notifications triggers cannot carry a token, because it will be expired. How to recognize the author of the subscription, and authorize him?

A cache containing the permissions for each users can be built:

permsCache :: Map Username PermCache

PermCache :: {
  perms :: [Perm],
  retrievedTime :: UTCTime
}

Perm :: {
  resourceId :: String,
  scopes :: [Scope]
}

The cache is a map between the users and their permissions. The permissions is a list including all resources (devices, gateways, projects) and all scopes (create, delete...). There are currently 700 resources with 3-5 scopes per resources. It takes 700ms to extract all permissions from Keycloak.

The cache starts empty. At each request, the cache is tested: if the user doesn't have a cache, or if the cache is outdated, all permissions are requested from Keycloak for that user. The cache is then updated. If the user does have a valid cache, it is used instead.

Any change of resource in Keycloak (create, update visibility, delete) should update invalidate the cache. Invalidating the cache simply mean to delete the full cache and restart empty.
The cache entries have a validity of ~5min. This means that changes in the policies will be reflected after max 5 min.

When querying datapoints, the performance is bad:

$ time curl -X GET "http://localhost:800/api/v2/sensors_data?device_id=MyDevice&sensor_id=TC1&calibrated=false" -H  "accept: application/json;charset=utf-8"
[{"sensor_id":"TC1","date_received":"2019-11-06T16:31:49Z","device_id":"MyDevice","value":25,"timestamp":"2016-06-08T18:20:27.873Z"}]
real	0m2,594s
user	0m0,006s
sys	0m0,000s

There are currently 3 million datapoints in DB.

The API PUTs need header "Content-Type: application/json;charset=utf-8".
Would be better to simplify to "Content-Type: application/json".

When creating a gateway, and then a device with the same ID, the gateway resource created in Keycloak gets overwritten. This leads to authorization problems afterwards.

For example, in this image the ID of the resource is gateway-00155d00641b (in the URL), which identifies it as a gateway. But the data in the resource corresponds to a device.

curl -X POST "http://localhost:3000/api/v2/devices/MyDevice/sensors/TC1/value" -H  "accept: application/json;charset=utf-8" -H  "Content-Type: application/json;charset=utf-8" -d "{  \"value\": \"\",  \"timestamp\": \"2016-06-08T18:20:27.873Z\"}"
Something went wrong

It should be possible to run the platform without MQTT (HTTP only)

In the current implementation, the permissions are retrieved from Keycloak and stored in the API server during the MQTT CONNECT. All further actions (i.e. Publishes) will use the same permissions, until the disconnect.
This is a problem when permissions change between the Connect and Disconnect. E.g. if the user connects, creates a device, and try to publish on that device, it will not work: the new device will not appear in the permissions stored.

The code uses Control.Monad.Catch.try, but it seems that this function cannot catch exceptions thrown by throwError. Instead, the function catchError should be used.

Currently, a device contains the keycloak ID as a field.
It would be better to use the device ID directly as the resource ID in Keycloak (as is already the case for projects and gateways).
This requires a database migration:

• Each Keycloak resource need to be destroyed and re-created with the right ID
• Each Orion device need to drop the keycloak_id field.

Not providing device_id parameter will return "something went wrong".

curl "https://api.waziup.io/api/v2/sensors_data?sort=dsc&calibrated=true&limit=100"
"Something went wrong"

The field "connected" of gateways goes to false after some time.
Currently:
MQTT CONNECT -> connected = true
TCP thread dies -> connected = false

The API is currently slow. for GET /devices it can take from 1600ms to 7s when the server is loaded (rush hour).

Currently, the Twitter ID should not contain the "@". If you include it, it will not work.
Let's remove it automatically when it's present.

Currently, the message "phone ID or sms credit absent" appears. It would be better to distinguish between the two.

Currently, when a calibration is present in the "sensor" datastructure, this calibration will be used to calculate the values output by the API (e.g. in devices/XX/sensors/YY/values).
This will cause problems because the "value/values" fields of a sensor changes meaning.
For instance the synchronisation module in the gateways will think this is the raw value, while this might be the calibrated value.

curl -X PUT "http://localhost:800/api/v2/devices/MyDevice/actuators/Act1/value" -H  "accept: application/json;charset=utf-8" -H  "Content-Type: application/json;charset=utf-8" -d "444"

gives:

400
Error from Keycloak: {"error":"invalid_resource","error_description":"Resource with id [device-MyDevice] does not exist."}

It should be 404.

The visibilty in Keycloak is not changed.

Currently, MQTT subscriptions return just the value:

{"value": "3"}

However, you can subscribe to a variety or devices/sensors using wildcards:

mosquitto_sub -L "mqtt://api.waziup.io/devices/MyDevice/sensors/+/value"

So, it would be better to return also device_id and sensor_id.