• Overview
  • Architecture
  • Known Limitation and Disclaimer
  • Prerequisites
• Installation
  • Microservice
  • Web App Plugin
  • Build, Deploy, Run
• Configuration MQTT connection to broker
• Definition and Deployment of MQTT mappings
  • Table of MQTT mappings
  • Define mappings from source to target format (Cumulocity REST format)
    • Wizzard to define a mapping
    • Snooping payloads on source topic
    • Enable snooping payloads on source topic
    • Define templates and substitutions for source and target payload
  • Test transformation from source to target format
  • Send transformed test message to test device in Cumulocity
  • Use snooped payloads in source templates
• Monitoring
• Setup Sample MQTT mappings

Cumulocity IoT has a MQTT endpoint but does not yet allow devices to send generic MQTT payloads. This project addresses this gap by providing the following artifcats:

• A Microservice - exposes REST endpoints, uses the PAHO MQTT Client to connect to a MQTT broker, a generic Data Mapper & Expression Language for data mapping and the Cumulocity Microservice SDK to connect to Cumulocity.
• A Frontend Plugin - uses the exposed endpoints of the microservice to configure a MQTT broker connection & to perform graphical MQTT Data Mappings within the Cumumlocity IoT UI.

Using the solution you are able to connect to any MQTT broker and map any JSON-based payload on any topic dynamically to the Cumulocity IoT Domain Model in a graphical way.

The grey components are part of this project which are:

• MQTT Client - using PAHO MQTT Client to connect and subscribe to a MQTT broker
• Data Mapper - handling of received messages via MQTT and mapping them to a target data format for Cumulocity IoT. Also includes an expression runtime JSONata to execute expressions
• C8Y Client - implements part of the Cumulocity IoT REST API to integrate data
• REST Endpoints - custom endpoints which are used by the MQTT Frontend or can be used to add mappings programmatically
• MQTT Frontend - A plugin for Cumulocity IoT to provide an UI for MQTT Configuration & Data Mapping

Please Note: A required MQTT Broker is not part of this repo and must be provided to make use of this component. In upcoming releases a Generic MQTT Broker will be part of Cumulocity IoT. If necessary we will adapt this component to work with it seamlessly!

The following diagram describes what happens in the microservice if a new MQTT mapping is added.

The following diagram describes what happens in the microservice if a new messages arrives and how the payload is transformed.

Currently this project is focussing on JSON Payload only. Any other payload sent via MQTT must be mapped programmatically. See chapter Enhance for more details.

As we already have a very good C8Y API coverage for mapping not all complex cases might be supported. Currently the following Mappings are supported:

• Inventory
• Events
• Measurements
• Alarms

Beside that complex JSON objects & arrays are supported but not fully tested.

Due to two different libraries to evaluate JSONata in:

1. frontend (nodejs): npmjs JSONata and
2. backend (java): JSONata4Java

differences in more advanced expressions can occur. Please test your expressions before you use advanced elements.

The Paho java client uses memory persistence to persit its state (used to store outbound and inbound messages while they are in flight). When the microservice restarts this information is lost. Micorservice can not use the default MqttDefaultFilePersistence.

Pull Requests adding mappings for other data formats or additional functionaly are welcomed!

In your Cumulocity IoT Tenant you must have the microservice feature subscribed. Per default this feature is not avilable and must be provided by administrators of the instance you are using.

Make sure to use an user with admin privileges in your Tenant.

You need to install two components to your Cumulocity IoT Tenant:

• Microservice
• WebApp Plugin

Both are provided as binaries in Releases. Take the binaries from the latest release and upload them to your Cumulocity IoT Tenant.

In Administration App go to Ecosystem -> Microservices and click on "Add Microservice" on the top right.

.

Select the "mqtt.mapping.service.zip". Make sure that you subscribe the microservice to your tenant when prompted

In Adminstration App go to Ecosystem -> Packages and click on "Add Application" on the top right.

NOTE: If you don't see the Packages Menu you have to add "?beta=true" in your URL. Example: {{url}}/apps/administration?beta=true

Select "mqtt-mapping.zip" and wait until it is uploaded.

NOTE: We need to clone the Administration app to add the plugin to it

After succesful upload go to "All Applications" and click on "Add Application". Select "Duplicate existing application" and afterwards "Administration".

.

Now select the cloned Administration App and go to the "Plugin" Tab. Click on "Install Plugin" and select "MQTT configuration plugin"

Make sure that Docker and Apache Maven are installed and running on your Computer.

Run mvn clean package in folder backend to build the Microservice which will create a ZIP archive you can upload to Cumulocity. Just deploy the ZIP to the Cumulocity Tenant like described here.

Run npm run build in folder frontend/mqtt-mapping to build the Front End (plugin) for the Administration which will build a plugin. Run npm run deploy in folder frontend/mqtt-mapping to deploy the Front End (plugin) to your Cumulocity istration which will build a plugin. The Frontend is build as Plugin here.

The MQTT broker configuration is persisted in the tenant options of a Cumulocity IoT Tenant and can be configured by the following UI.
Furthermore, connections to the MQTT broker can be enabled or disabled.

Once the connection to a MQTT broker is configured and successfully enabled you can start defining MQTT mappings. The MQTT mappings table is the entry point for:

1. Creating new MQTT mappings: Press button Add Mapping
2. Updating exsiting MQTT mapping: Press the pencil in the row of the relevant mapping
3. Deleting exsiting MQTT mapping: Press the "-" icon in the row of the relevant mapping to delete an existing mappings

After every change the mappings are automatically updated in the microservice.

Mappings are persisted as Managed Objects and can be easily changed, deleted or migrated.

In addition to using plain properties of the source payload, you can apply functions on the payload properties. This covers a scenario where a device name should be a combination of a generic name and an external device Id. Complex mapping expressions are supported by using JSONata.
In this case the following function could be used: $join([device_name, _DEVICE_IDENT_]). \

Further example for JSONata expressions are:

• to convert a UNIX timestamp to ISO date format use: $fromMillis($number(deviceTimestamp))
• to join substring starting at position 5 of property txt with device identifier use: $join([$substring(txt,5), "-", DEVICE_IDENT])

NOTE:

• escape properties with special characters with </code>. The property <code>customer-1</code> becomes <code>customer-1`
• function chaining using ~> is not supported, instead use function notation. The expression Account.Product.(Price * Quantity) ~> $sum() becomes $sum(Account.Product.(Price * Quantity))

The wizzard to define a mapping consists of the steps:

1. Define the properties of the topic and API to be used
2. Define the templates for the source and target, in JSON format. The soure payload can be in any custom JSON format. the target format has to follow the schemsa for Alarm, Events, Measurements or Inventory, see Cumulocity OpenAPI.
3. Test the mapping by applying the transformation and send the result to a test device.

In the first wizzard step properties for the topic are defined.

For the mappings we differentiate between a subscription topic and a template topic:

This is the topic which is actually subscribed on in the MQTT broker. It can contain wildcards, either single level "+" or multilevel "#". When you use a wildcard it signals to the mapping that you want to extract the device identifier from the topic name. In this case the additinal property _DEVICE_IDENT_is added to the source template shown in the next wizzard step. It must not be deleted when editing the JSON source template.

NOTE: Multi-level wildcards can only appear at the end of topic. The topic "/device/#/west" is not valid. Examples of valid topics are: "device/#", "device/data/#", "device/12345/data" etc.

The template topic is the key of the persisted mapping. The main difference to the subscription topic is that a template topic can have a path behind the wildcard for the reason as we can receive multiple topics on a wildcard which might be mapped differently.
Examples are: "device/+/data, "device/date/north/+/events/", "device/+"
If the template topic contains a wildcard, you have to specify which part to the topic defined the device identfier by pressing the button Device Identifier.

Very often you want to use the payloads of existing JSON messages as a sample to define the source template. This can be achieved by listening and recording - snooping- to messages on a topic.

In order to record JSON payloads on the defined topic a subscrition records the payloads and saves them for later use in a source template.

The snooping process goes through the steps ENABLED -> STARTED -> STOPPED.

If a payload is found the status moves to STARTED. This is indicated in the last column of the mappping table, where the number of payloads snooped so far is shown.

To enable snooping select ENABLED in the drop down as shown in the screenshot below. This starts the snooping process and the microservice subscribes to the related topic and records the received payloads.

Connected devices send their data using an external device identifier, e.g. IMEI, serial number, ... In this case the external id has to be mapped to the device id used by Cumulocity. To achieve this enable the switch Map Device Identifier and specify the name of the type of external id. When a payload from this device arries the external id is translated to the internal Cumulocity id.

In the second wizzard step, shown on the screenshot below the mapping is furher defined:

1. Editing the source template directly or use a snooped template by pressing button Snooped templates
2. Editing the target templatedirectly or use a sample template by pressing button Sample target template
3. Adding substitutions

In order to define a substitution ( substitute values in the target payload with values extracted at runtime from the source payload), the UI offers the following features:

1. Add mapping (button with "+" sign)
2. Show & Select already defined substitutions (button with skip symbol). A selected substitution is colored and can be deleted by pressing the button with "-" sign
3. Delete mapping (button wiht one "-" sign), the selected substitution is deleted
4. Delete all mappings (button wiht two "--" signs). In this case the substitution to define the deviceIdentifier is automatically added again. This is the case when a template topic contains a wildcard, eithe "+"- singel level or "#" - multi level

To define a new substitution the following steps have to be performed:

1. Select a property in the source JSON payload by double-click on the respective property. Then the JSONpath is appears in the field with the label Evaluate expression on source
2. Select a property in the target JSON payload by double-click on the respective property. Then the JSONpath is appears in the field with the label Substitute in target
3. Select Defines device identifier if this property defines the device identifier, i.e. it is mapped to source.id.
4. Press the add button with the + sign.

NOTE: When adding a new substitution the following two consistency rules are checked:

1. Does another substitution for the same target property exist? If so, a modal dialog appears and asks the user for confirmation to overwrite the existing substitution.
2. If the new substitution defines the device identifier, it is checked if another substitution already withe the same proprty exists. If so, a modal dialog appears and asks for confirmation to overwrite the existing substitution.

To avoid inconsistent JSON being send to the Cumulocity APIS schemas are defined for For all target payloads (Measurement, Event, Alarm, Inventory). The schemas validate if requred properties are defined and if the time is in the correct format.

In the sample below, e.g. a warning is shown since the required property c8y_IsDevice is missing in the payload.

To test the defined transformation, press the button Transform test message. The result of the transformation and any error are displayed.

To send the a transformed payload to a test device, press the button Send test message. If an error occurs this is shown in the UI.

In order to use a previously snooped payload click the button Snooped templates. Multiples activation of this button iterates over all the recorded templates.

On the monitoring tab Monitoring you can see how a specific MQTT mapping performs since the last activation in the microservice.

A script to create sample MQTT mappings can be found here.

In the folder Callbacks you can either overwrite the existing JSONCallback.class or add a new Handler in the handler folder. As an example see the SysHandler which subscribes and handles all topics for $SYS and creates Measurements in Cumulocity for the received data.

These tools are provided as-is and without warranty or support. They do not constitute part of the Software AG product suite. Users are free to use, fork and modify them, subject to the license agreement. While Software AG welcomes contributions, we cannot guarantee to include every contribution in the master project.

Contact us at TECHcommunity if you have any questions.