Code Monkey home page Code Monkey logo

sensae-console's Introduction

Sensae Console

This document presents all the information needed to install, run and understand Sensae Console.

This project is unmaintained and discontinued, it is not ready for production and no support related to it will ever be given. This project is here solely for demonstrations purposes as it was the work for my master's thesis.

Current Version: 0.10.0

The services currently supported are:

  • Fleet Management
  • Smart Irrigation
  • Notification Management

Links

sensae-console's People

Contributors

filipemcruz avatar

Watchers

 avatar

sensae-console's Issues

Device Records can't be deleted

Describe the bug
A device record can't be deleted since the device record slave recreates it with the 'Unknown' name.

To Reproduce
Steps to reproduce the behaviour:

  1. Go to 'device records'
  2. Click on 'delete' of any device record
  3. Click on 'Reload Devices'
  4. See the entry be reloaded with the 'Unknown' name

Expected behavior
The Device Record should be deleted and disappear.

Additional context
none

Shared Library for Backend Services

Description
This document will present a new solution to deal duplicate code inside each backend service.

Reason
Right now the created environment backend containers share the same model to talk with each other. Each service has a lot of duplicate code that shouldn't exist.

Solution
A clear and concise description of what you want to happen.
Create a shared library with the duplicate concepts. Import this library as a dependency.

Concepts that could be moved to the library:

  • MessageConsumed;
  • MessageSupplied;
  • ProcessedSensorData;
  • ProcessedSensorDataWithRecord;
  • SensorDataDetails;
  • DeviceRecords;

Every DTO that is used in the sensor data flow thought the system should end up here.

Additional context

This option would require us to create an account in https://mvnrepository.com.
It cloud be good to also create an account in https://hub.docker.com/, both with the sharespot website.

Add new data types: moisture and illuminance to model

Is your feature request related to a problem? Please describe.
Currently there is no way to define moisture or illuminance in the environment.

Describe the solution you'd like
Add this two types to iot-core

Describe alternatives you've considered
none

Additional context
none

Rethink how to access Common Database

Is your feature request related to a problem? Please describe.
Common Database is currently under stress due to high number of open connections. In the near future, if more databases are create in the Common Database connections will start to fail and the application will become unresponsive.

Describe the solution you'd like
All "slave" containers should communicate only with their master. This would lower the amount of open connections by a lot.
Instead of 10 containers accessing the Common Database we could have 5:

  • device-commander
  • device-management-master
  • device-management-slave
  • data-decoder-master
  • data-decoder-slave
  • data-processor-master
  • data-processor-slave
  • identity-management-master
  • identity-management-slave
  • smart-irrigation

All communications with masters must be authenticated, or inaccessible from outside.
By authenticating communications between slave/master there is a need to handle the request, creation and refresh of jwt tokens for slaves.

Another alternative is to rely on the message broker, slaves would request X info, store data somewhere and wait for an answer.
Masters would then receive X request and send X info to slaves. This would enable a truly reactive flow of data though the system.

The second alternative was chosen due to ease of implementation and maintenance.

Describe alternatives you've considered
Create a new Database Container Instance and split the created Databases into the two Database Container Instances.

Additional context
Both options would clear the path for the optimisations referenced in #28 since none rely on database access.

Documentation for Decoders is missing

Is your feature request related to a problem? Please describe.
No decoders documentation is provided

Describe the solution you'd like
Write missing documentation

Describe alternatives you've considered
None

Additional context
None

Location Tracking - Merge history box with search box

Is your feature request related to a problem? Please describe.
There are two places where the user can search for a device, there is no need for that

Describe the solution you'd like
Merge the two boxes into a single one.

Describe alternatives you've considered
Bind both 'device name' input in search and history to the same variable so that the user only needs to write the id in a single place.
Remove the 'device name' input from the history.

Additional context
Right now the data stored and queried in QuestDB does not have the 'device records' so it is hard to search the history by 'content', something that the search provides.
It may be wise to also store this information so that the user can have a better iteration with the platform.

This may lead us to drop QuestDB since it was built for normalized data and by adding the device records we no longer have normalized data and can't leverage QuestDB fast response time.

A solution to the above issue is to stringify the device records section and insert it as a new column.

QuestDB needs to be tested due to #1 before thinking about this.

Alert & Notification Service

Is your feature request related to a problem? Please describe.

  1. Not everyone can or wants to login to a dashboard to retrieve critical information about their sensor solution.
  2. People have a mobile phone on them all the time, it is the quickest way to alert of any situation that needs attention or prompt for action.

Describe the solution you'd like
(i) SMS Alerts (critical) and (ii) SMS Notifications / Summary / Reports, short but concise, (iii) Action Items.

Additional context
(i) SMS Alerts --> When a out-of-bounds event happens that needs attention. Can include action points to take.
(ii) SMS Notification/Reports --> Examples include daily or weekly reports containing condensed information that is concise and humanly understandable.
(iii) SMS Action Items.

Examples:

  1. (ALERTS) NOT: Temperature in Fridge is 10 Degrees. YES: Fridge Door is Open, please close.
  2. (SMS Reports) Total Water consumption Week: 500 L (Average); Average Air Quality Index Week: 30 (GOOD)
  3. (SMS Action Items): Soil Moisture is low, please water.

The Key is Intelligence. People don't care about numbers, they care about human understandable alerts and actions.

In Identity Management devices are identified only by it's uuid

Is your feature request related to a problem? Please describe.
Identity Management doesn't present much information regarding devices

Describe the solution you'd like
Devices are identified only with it's uuid since no other information can be extrapolated by the data received. As an improvement we could provide a link to each Device Records details.

Describe alternatives you've considered
The Identity Management Backend could fetch device records from Device Records Master Backend but this would create a direct dependency to Device Records Master in the backend, it would be harder to verify permissions and pave the way for entanglement in the backend.

Additional context
None

Rule Engine

Is your feature request related to a problem? Please describe.
Many solutions are not only about passive sensing (downlinks), the also interact back with the sensor to take action based on trigger points (uplink).

If a condition is met a trigger is activated (If Value > (x) Then Action). A Rule Engine listens for the trigger and executes a uplink command to execute an action (open a valve etc).

Describe the solution you'd like
Rules can be triggered automatically or manually through the dashboards. This provides operational control over sensors that support downlinks.

Data Processor - `Reported At` Property Transformation should have an associated unit of time

Is your feature request related to a problem? Please describe.
Right now no unit of time is presented inReposted At Property Transformation.
This leads users to think that any unit of time is valid while it isn't.

Describe the solution you'd like
Add more information to about how this data must be presented.
E.g. simple tag with 'in unix timestamp miliseconds'.

Describe alternatives you've considered
Try to guess in what timestamp the value is presented and act according to it.

Additional context
In the future add support for more time units/formats.

Upgrade all project depedencies ahead of 0.7.0 version release

Is your feature request related to a problem? Please describe.
There are some dependencies that can be updated.

Describe the solution you'd like
Review all dependencies, check for possible regressions, if none is found update them.

Describe alternatives you've considered
None.

Additional context
None.

Fleet management UI

Before we go into the details of the Asset Tracking metrics and UI it is important to note that asset tracking is not new, it is the most widely adopted use-case of IoT. Therefore to save time and product iterations we should not reinvent the wheel. The innovation here is the connectivity even though it makes up a small part of the product, without connectivity there is no product. That said the customer will not care about that and will mainly look at the product and what value it brings.

Is your feature request related to a problem? Please describe.

  1. UI information must be concise and highlight key metrics without overcrowding the view. This might sound simple but it is very easy to over-crowd the UI with too much data making it look more like spaghetti than anything else.

Describe the solution you'd like

Dashboard

Key information:

  1. Menus: Live Map, Reports, Playback (play trajectory), Locations (geofencing), Alerts.
    image

  2. State of movement --> Active, Small Stop, Stopped --> Display as icons on map.
    image

  3. Side-bar displaying --> Asset, Movement status, last activity, Current location.
    image

  4. When clicking on a vehicle display --> Number plate, Activity Status, Time, Speed Km/h, Total Kms 24h, Current location.

image

Reports

Key metrics:

  1. Type of asset

image

  1. Safety Score (Accelerometer and speed)
    image

  2. Harsh Driving (speed)

image

  1. Engine ON/OFF

image

  1. Number of Stops

  2. Distance Traveled

image

  1. Fuel Efficiency

image

  1. Vehicle Activity

image

  1. Wasted Fuel

image

  1. Speed Violations

image

Timeline

5127b245-990c-4fb5-b3bb-0ad858204fa7

Screenshots

Below are screenshots of other products in existence that will help guide the UI development process:

nl-mapping-image-desktop

nl-device-group-fleettracking

0d08c6a6-6b4f-46a6-b54d-82c98597b3f3

5127b245-990c-4fb5-b3bb-0ad858204fa7

Version based models for sensor data

Description
This document will present a new solution to deal with different versions of models inside an environment.

Reason
Right now if two containers use different version of sharespot-core models there is no way to ensure that they both can communicate.
This forces the environment to be fully updated every time a new container is added.

Solution
By adding the sharespot-core version to the routing keys and applying semantic versioning to it we can ensure that all containers talk the same "language".

The package sharespot-core will have the following schema for its versions, Z.X.Y, where every letter represents a number. Any version with the following schema: 0.X.Y is not compatible with any other version so that some time is given for the model to mature. After that initial phase the 1.X.Y versions must ensure some stability.
When the Z param changes this implies that the internal structure or the order of the routing keys has changed.
When the X param changes this implies that a new routing key and internal structure was added to the model. Any new routing key added here must go to the end of the routing keys.
When the Y param changes this implies that some bug was fixed.

To ensure that, e.g. a project with sharespot-core:1.3.4 can exchange information with sharespot-core:1.5.6 all routing keys must end with #.
The versions and # can't be changed by the container that uses them and are added to each message so that nothing erroneous happens.

Location Tracking - History returns wrong values after X closed timed queries

Describe the bug
After 5 queries related to history in location tracking no GPS data is returned.

To Reproduce
Steps to reproduce the behavior:

  1. Go to 'location tracking'
  2. Fill 'History Box' with values
  3. Click on 'Search' many times
  4. See that no results are presented after some queries

Expected behavior
Results related to history don't disappear after a while.

Related Containers:

  • Name: sharespot-location-tracking-backend
    • Version: 0.4.0
  • Name: QuestDB
    • Version: 6.1.3

Additional context
Queries made to QuestDB from the container 'sharespot-location-tracking-backend' return all rows with null gps_data after 5 consecutive attempts.
The same query made from QuestDB's web console always returns the expected result.

The problem is either in QuesDB or in JDBC driver for postgres.
This bug occurs in both JDBC driver version: 42.2.13 and 42.3.1.

The Decoder tool has no idividual permissions

Is your feature request related to a problem? Please describe.
The Decoder tool is sharing the permissions with Data Transformation.

Describe the solution you'd like
The containers sharespot-data-decoder-master-backend and sharespot-data-decoder-frontend should use their own permissions.

Like:

data_decoder:decoders:write and data_decoder:decoders:read

Describe alternatives you've considered
None.

Additional context
None

Sensor Provisioning Tool

Is your feature request related to a problem? Please describe.
How to keep track of the deployment and management of hundreds or thousands of sensors.

Describe the solution you'd like

We need a way to facilitate sensor onboarding, deployments, linkage to the platform back-end and customer account.

Ideas:

  1. Generate printed QR codes for sensors with device info (App Keys, EUIs, Sensor type. Some sensors already come with a barcode/ QR code that can be scanned.
  2. Provisioning Mobile/Web app.

Describe alternatives you've considered
The only alternative I can think of is excel sheets, which TBH is not a good practice and very error prone. That said something like google forms might be an option like we do now with hotspot installs.

Additional context

QR Code

The purpose of the QR code is to quickly onboard a sensor onto the Helium console and the Sensae platform. By scanning the QR code and automatically associating the device ID to a customer prevents the situation where sensors get "lost" without knowing where it is located and if it is even installed or not.

Provisioning App (Mobile or Web)

The goal of the Provisioning app is to link the following components:

  1. Sensor
  2. Customer
  3. Type of asset (car, trailer, bulldozer etc)

This could be a simple web-app that scans the QR code of the sensor, then asks for the customer ID (drop-down) followed by the asset, category search, device type. It is important to know which asset is being tracked as different assets will imply different things to track.

As mentioned above a initial simple way to do this would be a google forms but a direct link to the platform might be better.

Device Records is currently Dumb regarding what domains own each device

Is your feature request related to a problem? Please describe.
The Device Records tool can't understand what devices belong to the current user domains. As such it is advised to give no permissions related to device records to any domain other than the root. With this change the Ownership Type would start to be useful, for now it has no impact.

Describe the solution you'd like
No good solution has been found.

Describe alternatives you've considered
As an alternative, the device record master could query the identity management master for all the devices a user can write or read.
This would create a direct dependency to identity management master in the backend, it could pave the way for entanglement in the backend.

As another alternative, the device record master could store domain permissions as they flow into the slaves. This would create a window of time in which a user could change a device records without permissions: Tenant A in domain R removes device X from domain S, no new data is sent by device X, tenant B in domain S change device X records.

As another alternative we could merge identity management and device records into one single tool, merging frontends, backend masters and backend slaves.

Additional context
None.

Microfrontends aren't creating their own apollo client and depend on ui-aggregator

Is your feature request related to a problem? Please describe.
Even though apollo-angular 3 is used all apollo clients are not created by importing Apollo Client in each micro frontend.

Describe the solution you'd like
Change the creation of apollo clients to the related frontend and remove it from ui-aggregator.

Describe alternatives you've considered
Follow the Migration guide from apollo angular, and then remove unneeded properties from ui-aggregator.

Additional context
ref

Documentation about IoT data structure inside the system is missing

Is your feature request related to a problem? Please describe.
No internal documentation about the message structure is provided

Describe the solution you'd like
Write missing documentation about the semantic and structural rules applied in the data events of the system. So that anyone can easily create decoders.

Describe alternatives you've considered
None

Additional context
None

Last Location - Red Color

Is your feature request related to a problem? Please describe.
When an device is not moving, its already show the last position on map, but with blue icon. Like a moving device.

Describe the solution you'd like
For stoped devices, show the icon in red color

Example:
Captura de ecrã 2022-01-21 151506

Containers related to IoT data flows not optimized for high data thoughput

Is your feature request related to a problem? Please describe.
Containers related to IoT data flows are simple spring boot applications that take too long to start and consume high resources. This will lead to higher infrastructure costs in the future.

Describe the solution you'd like
Instead of using java + spring boot, try another technology tailored for this kind of workload.

Describe alternatives you've considered
Try to build a container in rust.
Try to build a container in java + quarkus + graalvm native image.
Try to build a container in java + micronaut + graalvm native image.
Try to build a container in java + spring native + graalvm native image.

Additional context
The best case cenario is to use spring native, since there would be no need the re implement anything.
Right now quarkus + java + graalvm native image or java + micronaut + graalvm native image would be faster to develop since we don't need to reimplement the iot-core package in rust and java is a well-known programming language for me.
Rust would bring more performance to the table but it would cost much more time to develop, it's an interesting option for the future.

For now the container to be reimplemented are:

  • sharespot-data-gateway
  • sharespot-data-decoder-slave-backend
  • sharespot-data-processor-slave-backend
  • sharespot-data-validator-backend
  • sharespot-device-records-slave-backend

It is advised to start by the sharespot-data-gateway.

Progress

Quarkus and Micronaut can't detect implementations of interfaces when they are in separated maven modules.
Spring Native runs but throws an error related to serialization of domain classes with rabbittemplate.

Update `iot-core` with new data types discussed in 2022-04-11

Is your feature request related to a problem? Please describe.
The Data Model doesn't support many data types. In the 2022-04-11 meeting some more types to add and modifications for the current data model have been defined.

Describe the solution you'd like

Preform this alterations:

  • Add data.air_humidity.relative_percentage as percentage
  • Add data.water_pressure.bar for water pipes
  • Add data.battery.max_volts
  • Add data.battery.min_volts
  • Add data.ph.value
  • Add data.distance.millimeters
  • Add data.distance.max_millimeters
  • Add data.distance.min_millimeters
  • Add data.occupation.percentage
  • Add data.soil_conductivity.micro_siemens_per_cemtimeter
  • Add data.co2.ppm
  • Add data.co.ppm
  • Add data.no2.ppm
  • Add data.nh3.ppm
  • Add data.o3.ppm
  • Add data.voc.ppm
  • Add data.pm2_5.micro_grams_per_cubic_meter
  • Add data.pm10.micro_grams_per_cubic_meter
  • Change data.moisture.percentage to data.soil_moisture.relative_percentage
  • Change data.pressure.hPa to data.air_pressure.hecto_pascal air pressure
  • Change data.humidity.gramspercubicmeter to data.air_humidity.grams_per_cubic_meter
  • Change data.velocity.kmperh to data.velocity.kilometer_per_hour
  • Change data.alarm.value to data.trigger.value
  • Change data.aqi.eaqi to data.aqi.value

Describe alternatives you've considered
None.

Additional context
This work will be preform in the ior-core repo first and then integrated into this repo.

The integration with this repo requires us to:

  • Update Data Processor Frontend with new types;
  • Update Data Processor Master Backend with new types;
  • Update all other containers version of iot-core;
  • Add validators to new data types in data validator;

Update docs for upcoming version release 0.7.0

Is your feature request related to a problem? Please describe.
Documentation is outdated.

Describe the solution you'd like
Rewrite docs with new features.

Describe alternatives you've considered
None.

Additional context
None.

Device Information Frontend - Group by label

Is your feature request related to a problem? Please describe.
Device Information page is currently presenting everything unorganized.

Describe the solution you'd like
We could create a group by label X option to display device information in a more concise manner.

Describe alternatives you've considered
none.

Additional context
Idea taken from Helium console.

Sensors

GPS Sensor

Is your feature request related to a problem? Please describe.

The current GPS tracker (LGT92) albeit cheap, lacks in a number of areas:

  1. GPS chip is low accuracy.
  2. Only Uses GPS and not a combination of Glonass, BeiDou and GPS.
  3. No native support for external power/requires modification.
  4. No Odometer
  5. Limited native sensor intelligence.
  6. Prone to failure.

Describe the solution you propose:

I propose switching to a more advanced and accurate sensor, the Digital Matter G62.

The configuration and usage guide can be found here.

Although we will spare the complete details, some important features include:

  1. Tracks Runtime/Trips natively including duration.
  2. Odometer calculates trip distance traveled, which is more accurate than post-calculation with GPS data.
  3. Automatically starts/ends a trip based on accelerometer, no need to calculate it in post-processing.
  4. Ignition detection. Can detect when the car is turned ON/OFF.

Calculating these parameters on the sensor itself reduced server-side workload and are more accurate.

Summary

We should take these changes into consideration when implementing our Asset Tracking application because it will significantly reduce development time.

Smart Irrigation

Air Quality

Fire Detection

Location Tracking - Present distance made by each device in a given time span

Is your feature request related to a problem? Please describe.
There is no concise way of knowing what was the travelling distance of a device in a given time span.

Describe the solution you'd like
When someone accesses the 'Location Tacking' page and uses the 'History Box' along with the history also present the total distance travelled.

Describe alternatives you've considered
This can be implemented using Turf or QuestDB.

Additional context
QuestDB needs to be tested due to #1 before thinking about this.

Identity Access Management - Frontend

Is your feature request related to a problem? Please describe.
Right now there is no frontend to handle Identity Access Management stuff.

Describe the solution you'd like
Build a micro frontend to handle Identity Access Management stuff.

Describe alternatives you've considered
None.

Additional context
None.

Support Programmatically Remote Control Of Sensors

Is your feature request related to a problem? Please describe.
The system can't send downlinks to command devices.

Describe the solution you'd like
Add support for commands in iot-core and create a new container, device management slave, that handles commands sent though HTTP downlinks.
Commands are defined in the device-management tool

This work will be divided into the following tasks:

  • Add suppport for commands in iot-core;
  • Update all container with new iot-core version that supports commands;
  • Add downlink property to the data-processor tool;
  • Add commands notion to the device-management tool;
  • Add commands notion to smart irrigation;
  • Create new container device-commander-backend that has all needed info to dispatch downlinks;
  • Add service to open/close a valve in smart-irrigation;

Describe alternatives you've considered
Send downlinks directly though each service.

Additional context
Related to #48

Micro Frontends Generation

Description
This document will present a new solution to deal with code generation according to features requested.

Reason
The environment frontend generation is hard to deal with. When working with a big base project that will be striped down till there is only what the client asked about things can become hard due to internal dependencies.

Solution
To alleviate this problem we need to decrease the coupling between each service. By using microfrontends we can enforce this and in the end it's all about loading or not a feature.

Data Decoder

Description
This issue presents an alternative to the decoder used inside helium console.

Reason
Right now it isn't clear how to transform data with the "Data Processor". The "Data Processor" is also very limited.

Solution
This solution would provide a new page where people can submit javascript files. This files would be stored and then used to transform the incoming data into a structure used by the system.

Design
This solution is composed by 3 containers: Frontend, Master Backend and Slave Backend. The submited files are store in the file system.
The slave backend contianer, responsible for loading the script and transforming the incoming data will be built using graalvm and use graaljs to execute the scripts.

Additional context
Add any other context or screenshots about the feature request here.

Data Validator Container must validate more than GPS Data

Is your feature request related to a problem? Please describe.
Currently Data Validator only verifies GPS Data, it needs to check for all other types of sensor data.

Describe the solution you'd like
Implement verification for other sensor data:

  • temperature.celsius
  • motion.value
  • velocity.kmperh
  • aqi.eaqi
  • humidity.gramspercubicmeter
  • pressure.hPa
  • battery.volts
  • battery.percentage
  • moisture.percentage
  • illuminance.lux
  • alarm.value

Describe alternatives you've considered
None.

Additional context
This Issue regards the implementation of a "dumb" validation, in the future AI will be used to increase the effectiveness of this container.

Location Tracking - Present last GPS info from each sensor

Is your feature request related to a problem? Please describe.
There is no concise way of knowing what devices the platform manages, or when was the last time that any of them signalled something.

Describe the solution you'd like
When someone accesses the 'Location Tacking' page load the latest info about each device and present it in a different 'marker' color.
If some device sends new info overwrite the marker.

Describe alternatives you've considered
No alternatives considered.

Additional context
QuestDB needs to be tested due to #1 before thinking about this.

Allow access to the platform from emails that don't belong to the sensae domain

Is your feature request related to a problem? Please describe.
Currently only *@sensae.pt emails are allowed. The platform will be open for other organizations in the near future and they need access to it.

Describe the solution you'd like
Enable Multi tenancy in Microsoft AAD.

Describe alternatives you've considered
None.

Additional context
None.

Model Based Routing with RabbitMQ Topics

Description
This document will present a new solution to deal with messages/events routing inside the built environment.

Reason
Right now the created environment uses Fanout Exchanges to publish events and Queues to subscribe to messages.
The problem with this approach is that if, for example, we want to collect all events about unknown data (the one sent by any Data Gateway) that service would need to subscribe to n exchanges.

Solution
Changing to Topic Exchanges may be the solution to the problem above presented. Each Data Gateway would simply publish in an Exchange with the following pattern data.unknown.{sensor.id}. Now the new service would only need to subscribe to data.unknown.*.
But other problems may arise with this approach.

Topic exchanges use routing keys to know to where to send the data1.
A solution is to define the sensor data model in this routing keys so that each container subscribes only to models that can handle.

For this the following "metamodel" can be defined in routing keys as such:
<containerType>.<containerName>.data.<infoType>.<sensorId>.<channel>.<records>.<gps>.<temp_celc>

And each key can have the following types:

  • containerType: any container type, e.g. Data Gateway;
  • containerName: any, defined by the configuration;
  • infoType: encoded, decoded, processed;
  • sensorId: any, defined by the configuration;
  • channel: default and any defined in configuration (relevant for data ramify);
  • records: withRecords, withoutRecords;
  • gps: withGPSData, withoutGPSData;
  • temp_celc: withTempC, withoutTempC;

More sensor data types or other options can be added as needed.

Design
The following table will attempt to set exchanges to each service, inbound and outbound (if needed). All services that don't interact with sensor data are not presented.
There are 3 types of containers: start (publishes), middle (subscribes and publishes), end (subscribes).
This solution is viable only if we can keep flexibility while providing a way for each container to only subscribe and publish to one single exchange (without looking to special cases).

Container Subscribe Publish Notes
Data Gateway --- datagateway.{container.name}.data.{sensor.infoType}.{sensor.id}.default.withoutRecords.withoutGPSData.withoutTempC
Data Processor *.*.data.decoded.{sensor.id}.*.*.*.* dataprocessor.{container.name}.data.processed.<keep>.<keep>.withoutRecords.withGPSData.withoutTempC 1
Data Store *.*.data.*.*.*.*.*.* --- 2
Device Records Slave *.*.data.processed.*.*.withoutRecords.*.* devicerecordsslave.{container.name}.data.<keep>.<keep>.<keep>.withRecords.<keep>.<keep>
Data Decoder *.*.data.encoded.{sensor.id}.*.*.*.* datadecoder.{container.name}.data.decoded.<keep>.<keep>.withoutRecords.withGPSData.withoutTempC 4
Fast Data Store *.*.data.processed.*.*.*.withGPSData.* --- 5
Data Aggregator *.*.data.processed.*.*.*.withGPSData.withoutTempC && data.processed.*.*.*.withoutGPSData.withTempC dataaggregator.{container.name}.data.processed.<keep>.<keep>.<keep>.withGPSData.withTempC 6
Authenticator Slave TODO TODO
Sensor Viewer Slave TODO TODO
Tracking Delivery *.*.data.processed.*.*.*.withGPSData.* ---
Farm Conditions Control TODO TODO
Location Tracking *.*.data.processed.*.*.withRecords.withGPSData.* ---

Additional context
The section presents notes about specific containers.

Data Store

A data store stores sensor data, this container can be deployed as many times as needed to aggregate data as requested.
So this can aggregate all data by subscribing to all data with *.*.data.* or to more specific subsets of this data, like *.*.data.decoded.* and *.*.data.processed.gps.*.

Data Processor

As an example, the sensor.type for this processor was a simple GPS data type. That with in the example this container publishes with this routing keys.
This container can possibly subscribe to *.*.data.encoded.*.*.*.*.* if all data has the same format, e.g, they were all decoded to the same json format.

Data Decoder

It would be nice if we let, based on something, this container publish with the following keys:
*.*.data.processed.<something>

Fast Data Store

For now this container can only processed gps data but more is planed for the future.

Data Aggregator

This is an example. The idea is that the configuration defines what two data type it will aggregate. The only constrain is that the types it subscribes to can't be the same as the ones that it publishes.
E.g.: subscribe to *.*.data.processed.*.*.*.withGPSData.withoutTempC && *.*.data.processed.*.*.*.withGPSData.withTempC, publish dataaggregator.thisaggregator.data.processed.<keep>.<keep>.<keep>.withGPSData.withTempC is invalid.

Data Ramify

The idea where is that, in configuration, using a function a user can define what channel to send the information.
E.g.:

static outcome(sensor: any): Channel {
    if(sensor.data.gps.latitude > 90) {
        return Channel.of("drop")
    }
    if(sensor.data.gps.latitude < -90) {
        return Channel.of("drop")
    }
    if(sensor.data.gps.longitude > 180) {
        return Channel.of("drop")
    }
    if(sensor.data.gps.longitude < -180) {
        return Channel.of("drop")
    }
    return Channel.of("default")
}

For now the config can only subscribe to *.*.data.processed.<something> maybe we can change it to *.*.data.*.<something> in the future.

System Permissions must have a smaller scope

Is your feature request related to a problem? Please describe.
Some permissions have a wider scope then what is expected.

Describe the solution you'd like
Create new permissions by dividing old ones. E.g.:

  • Divide Device Records Write permission into Device Records Create, Device Records Change, Device Records Delete
  • Divide Fleet Management Read permission into Fleet Management See Live Data, Fleet Management See Past Data

Describe alternatives you've considered
None

Additional context
More ideas and examples may be added.

Location Tracking - Indicate when a device was not moving in history

Is your feature request related to a problem? Please describe.
There is no concise way of knowing when a device was not moving in its history.

Describe the solution you'd like
When someone accesses the 'Location Tacking' page and uses the 'History Box' the path presented could have some indication that the device wasn't moving for X time.
For this to happen 'sharespot-location-tracking-backend' must also send the timestamp related to each GPS Coordinates.
Additionally there has to be a way to 'know' when a device is standing still.

Describe alternatives you've considered
Define a 'radious' and if consecutive sent data is inside the same 'radious' the device is still.
See if there is a way for the device to send information about its current 'state'.

Additional context
QuestDB needs to be tested due to #1 before thinking about this.

Support Controllers - Devices that manage various things

Is your feature request related to a problem? Please describe.
The current iot-core version expects that all data packets come from a single sensor or thing.

Describe the solution you'd like
Upgrade iot-core model to support controllers.

Describe alternatives you've considered
None.

Additional context
None.

Metrics

Fleet Management

  • Km per device per day
  • Km per device per month
  • Km per month (secondary)
  • Active time per device per day
  • Active time per device per month
  • Active time per month (secondary)
  • Top X stop locations per device per month (by number of stops)
  • Top X stop locations per device per month (by time) (secondary)

Smart Irrigation

Indoor Air Quality

Fire Detection

Smart Greenhouse

Location Tracking - Indicate when the history of a device may be incomplete

Is your feature request related to a problem? Please describe.
There is no concise way of knowing when a device stopped sending data. This may lead to weird history data, e.g. history paths crossing an entire county.

Describe the solution you'd like
When someone accesses the 'Location Tacking' page and uses the 'History Box' the path presented could have some indication that the device wasn't sending any information between two history points. This information can be presented with a dashed line.
There is a need to define what is the 'time span' acceptable between data packets of a simple device.
E.g.: If two consecutive data packets from the same device have more than 10 minutes between then signal that history path as 'incomplete'.
This is related to distance calculation since big time frames between data packets will represent less distance travelled then what it really was.

Describe alternatives you've considered
No alternatives were considered.

Additional context
QuestDB needs to be tested due to #1 before thinking about this.

Authentication & IAM

Description
This document will present alternatives to deal with Authentication and Authorization in a micro frontend/micro service environment.

Reason

There is a high need to provide sane authentication and authorization in the system.
An Admin should be able to define who can access a service, tool or device information.
To avoid dealing with data breaches and ease the users first interaction with the environment, using OpenID Connect is preferred.
There is currently no way to restrict some one to access a resource. In the future this requirement is needed to give different persons access according their profiles.

Solution
Implement something that can deal with Authentication/Authorization of users.

Design

Internal Authorization Server

By creating an Internal Authorization Server we could have a normal and controlled flow in the environment.

The following diagram presents the normal environment flow for this alternative.

a1-UC01

But there would be a risk related to data breaches and anyone would need to register in the environment. For this reason this alternative was discarded.

External Authorization Server

By using an external Authorization Server there would be no need to store user credentials.

The following diagram presents the normal environment flow for this alternative.

a2-UC01

This approach would create a strong dependency to the auth service used.
Some of this services are:

The dependency is even more severe since all user roles would have to be managed in this auth service.

The platform Auth0 was tested and it is capable of registering user roles according to the requirements.

The dependency created would force the environment created to always be coupled to the service no matter what. For this reason this alternative was removed.

External Authorization Server with Internal Roles Server

By using an external Authorization Server there would be no need to store user credentials.

The following diagram presents the normal environment flow for this alternative.

a3-UC01

This approach would create a dependency to the auth service used and presented in the second alternative.

The dependency to the auth service is less severe compared with the second alternative since all roles would be managed internally.
This approach would require any authorized request to query the user roles backend for the user roles, and therefore linger down the performance of the environment.

External Authorization Server with Internal Oauth2 Server

By using an external Authorization Server there would be no need to store user credentials.

The following diagram presents the normal environment flow for this alternative.

a4-UC01

This approach would create a dependency to the auth service used and presented in the second alternative.

The dependency to the auth service is less severe compared with the second alternative since all roles would be managed internally.
This approach would require the system to create (and possibly refresh them) access tokens based on the id token received.

Contrary to the alternative 3 it would not create excessive pressure in some container.

Allow Anonymous Users to access the platform

Is your feature request related to a problem? Please describe.
Currently only authenticated users can access the platform. Some information/services, e.g. air quality sensor information or public transport services, may be publicly available in the near future.

Describe the solution you'd like
Assign an anonymous account that belongs to a "public" domain to users that don't authenticate in the system.
The public domain would have permissions for, e.g. access air quality service, and companies could just move sensors to that specific domain, making their sensor data public.

Describe alternatives you've considered
Instead of creating a new "public" domain we could use the "unallocated" root domain, but there would be a need to change this domain so that it could have permissions and devices.

Additional context
Idea previously discussed with Nuno Silva.

Reports

Is your feature request related to a problem? Please describe.

"Dashboards are cool but data Science and analytics to drive business decisions is what matters" The data has to mean something.

Describe the solution you'd like
Data Science and Machine Learning are future topics but we should at least condense data, extract key metrics and present it in a humanly readable/understandable fashion.

Additional context
Reports can be part of:

(i) Dashboard environment.
(ii) Email
(iii) SMS

For example weekly or monthly reports that are short and concise will enable decision making based on data.

Reports should include:

  1. Key metrics: Total distance traveled, Total Time in use etc. Cumulative/Finite values.
  2. Change: Presented in the form of graphs/charts. Distance traveled last 30 days, 1 day resolution.
  3. Anomalies: Anomalies indicate potential improvements to be made in processes. AI/ML/Data Science will in the future play a big role in anomaly detection and reporting, however, for the time being detecting and reporting out of bound occurrences will present a start.
  4. Actions: Insight into processes and operation data is important. We can do better. We can help decisions makers by advising them in decisions to make (e.g. Alert: Not enough water, Advice: Buy a bigger Water tank; Water has build-up of Algae, Put shade over water). This can be advise based on defined triggers.

Create a new Service - Smart Irrigation

Description
Create a new Service like Fleet Management but for Smart Irrigation.

Reason
It's in the roadmap

Solution
Develop a frontend and backend to deal with this service.

Design
Build a frontend in Angular + Typescript, Build a backend in Java + Spring, this services communicate with GraphQL, the backend subscribes to events with moisture, records, permissions and valid.

Additional context
It may be interesting to have two data sources in the backend, one for sensor data (in questdb) and another related to the business (in postgres).

This container will trigger the creation of the rules engine and possibly something that can communicate with devices though downlinks.

Permissions are not updated in Identity Management

Describe the bug
Permissions can 't be changed due to an error in Identity Management Database.

To Reproduce
Steps to reproduce the behaviour:

  1. Go to 'Identity Management'
  2. Open browser console
  3. Click on a sub domain
  4. Check/Uncheck a permission
  5. See error appearing in the browser console

Expected behaviour
Permissions are changed.

Additional context
none

Upgrade to the latest version of `iot-core` to enable protobufs

Is your feature request related to a problem? Please describe.
By upgrading to the latest version of iot-core some containers can be compiled using graalvm.
This containers are like:

  • data-validation
  • data-gateway

By adopting protobuf and dropping support for json the system will also be more performant.

Describe the solution you'd like
Upgrade iot-core and make the needed changes in each container.

Describe alternatives you've considered
none.

Additional context
Related to #28

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.