softwareag / cumulocity-dynamic-mapper Goto Github PK

The ultimate Mapper for building the bridge between any Message Broker and Cumulocity IoT in a zero-code approach!

License: Apache License 2.0

Java 51.76% TypeScript 37.45% HTML 8.49% CSS 1.07% Shell 0.16% Python 0.69% JavaScript 0.37% Less 0.01%

cumulocity-iot iot-analytics agent cumulocity-agent cumulocity-microservice microservice mqtt server-side-agent angular java

cumulocity-dynamic-mapper's People

Contributors

Stargazers

Watchers

Forkers

switschel merinstdl scfx mlhsouza sag-henrychen lattic tupacamaru7 arsalansoftwareag cumulocity-open-source

cumulocity-dynamic-mapper's Issues

Feature Request: Support Downstream mappings

Currently the dynamic mqtt mapper is focusing on upstream data.
It would be great if mappings could be define to send downstream data to the devices like commands & operations.

Microservice seems to fail with 1.1

Hi all,

I tried to setup the Mapping Service on a new Tenant and while the setup looked ok, it failed on every operation. The log gives the following error:

External ID MQTT_MAPPING_SERVICE not found
2022-10-26 11:11:36.719  WARN 13 --- [scheduling-1] c.c.m.context.ContextServiceImpl         : execution of task failed within tenant : t734704630 - null 
2022-10-26 11:11:36.719 ERROR 13 --- [scheduling-1] c.m.s.s.MicroserviceSubscriptionsService : null

java.lang.NullPointerException: null
	at mqtt.mapping.core.C8yAgent.lambda$initialize$1(C8yAgent.java:114)
	at com.cumulocity.microservice.subscription.service.impl.MicroserviceSubscriptionsServiceImpl.lambda$runForTenant$7(MicroserviceSubscriptionsServiceImpl.java:242)
	at com.cumulocity.microservice.context.ContextServiceImpl.callWithinContext(ContextServiceImpl.java:78)
	at com.cumulocity.microservice.subscription.service.impl.MicroserviceSubscriptionsServiceImpl.lambda$callForTenant$8(MicroserviceSubscriptionsServiceImpl.java:251)
	at java.base/java.util.Optional.map(Optional.java:265)
	at com.cumulocity.microservice.subscription.service.impl.MicroserviceSubscriptionsServiceImpl.callForTenant(MicroserviceSubscriptionsServiceImpl.java:251)
	at com.cumulocity.microservice.subscription.service.impl.MicroserviceSubscriptionsServiceImpl.runForTenant(MicroserviceSubscriptionsServiceImpl.java:241)

I checked that this is a regression on 1.1 by removing the 1.1. MS and installing 1.0 on the same tenant. This gives the message
2022-10-26 11:40:32.586 INFO 13 --- [main] mqtt.mapping.core.C8yAgent : Agent has been created with ID ID [type=com_cumulocity_model_idtype_GId, value=131847730]

After that I was able to update from 1.0 to 1.1 by uploading the new package. So it looks like the new version is not creating the ManagedObject on setup?

Mario

pom.xml installed in the repository still contained <version>${revision}</version>

The properties in the installed pom.xml were not resolved, resulting in:

<version>${revision}</version>

which lead to incorrect artifacts.

Bug v3.3.6: Outbound dispatcher will set the status of operations even there is no mapping defined

This results in interfering in other potential operation logic... when using the mapper and operations are created which should not be handled by the mapper.

Error in UI when configured custom extension was not loaded.

When an inbound mapping uses a custom extension that was not loaded, the UI
mapping-stepper.component.ts
would fail to recover from this situation.

Replace paho mqtt client

Replace paho mqtt client with a more recent maintained and stable MQTT client e.g.
https://github.com/hivemq/hivemq-mqtt-client

In the web UI simulation for multiple devices is processed correctly

E.g. if you want to create multiple devices from the following json:

{
  "device": [
    "d1_id",
    "d2_id"
  ],
  "types": {
    "type_A": "type_A",
    "type_B": "type_B"
  },
  "used_name": [
    "Pressure_d1",
    "Pressure_d2"
  ]
}

The backend process this as creating two devices. This is currently not shown correctly in the web UI.

Solution is pending.

Ability to control and reduce the costs for events being sent to the Cumulocity

The backend sends events and updated to the inventory for the monitoring:

processed mappings
connector status

This can create substantial costs on the Cumulocity tenant.

Error when trying test inventory mapping and sendTest result

Component MappingStepTestingComponentthrows an error when trying to test inventory mapping and sendTest result.

Error on creating Outbound Mappings

In the latest version in dev branch it is not possible to create output mappings:
It seems the validation fails due to templateTopics is null while publishTopic is maintained.

Microservice Error log:

2023-07-26 14:52:52.321  INFO 11 --- [http-nio-80-exec-5] m.m.rest.MQTTMappingRestController       : Add mapping: Mapping(name=Operation Test, id=8c441076-d4d7-4e04-8c7e-8fb2e8e0a864, ident=8c441076-d4d7-4e04-8c7e-8fb2e8e0a864, subscriptionTopic=, publishTopic=operation/+, templateTopic=, templateTopicSample=operation/123456, targetAPI=OPERATION, active=false, tested=false, qos=AT_LEAST_ONCE, substitutions=[MappingSubstitution(pathSource=cust_TestFragment, pathTarget=type, repairStrategy=DEFAULT, expandArray=false), MappingSubstitution(pathSource=decription, pathTarget=decription, repairStrategy=DEFAULT, expandArray=false), MappingSubstitution(pathSource=deviceId, pathTarget=_TOPIC_LEVEL_[1], repairStrategy=DEFAULT, expandArray=false)], mapDeviceIdentifier=true, createNonExistingDevice=false, updateExistingDevice=false, externalIdType=c8y_Serial, snoopStatus=NONE, mappingType=JSON, extension=null, direction=OUTBOUND, filterOutbound=custom_TestFragment, autoAckOperation=true, lastUpdate=1690383172282)
2023-07-26 14:52:52.484  WARN 11 --- [http-nio-80-exec-5] c.c.m.context.ContextServiceImpl         : execution of task failed within tenant : t14368213 - Validation errors:[ TemplateTopic_And_TemplateTopicSample_Do_Not_Have_Same_Number_Of_Levels_In_Topic_Name ]

The typescript code does not follow any coding rules.

The typescript code does not follow any coding rules:

using either ' or "
avoid using public
consistently end stmt with ;

Status of the websocket connection for Notification 2.0 subscriptions is not visible to users.

Currently the status of the websocket connection for Notification 2.0 subscriptions is not visible to users.
Therefore they cannot verify if an outbound mapping is processed correctly.

Mapping for Measurement Value not working

Hi all,

I tried the mapper with some custom Measurement mapping, but it looks like all mappings to the c8y_TemperatureMeasurement.T.value become null - although the value can be found on the incoming MQTT message (see log below for an example)
I tried to hardcode the value in the mapping definition instead of making a dynamic mapping and had the same result. I would assume the path c8y_TemperaturMeasurement.T.value can be found, otherwise it would stay at its initial value (110) (?)

I also tried to rule out any interfering rules on the type, by exchanging the fragment name and measurement type, but the result is always the same.

The mapping does work when I test it with the example message.

Thanks,
Mario Heidenreich

2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] m.mapping.processor.impl.JSONProcessor   : Message received on topic '/moneo/c2818e07-4c09-42f0-ba24-ddb712573ab5/AL1352/192168221/80/X03/VVB001StatusB/Crest'  with message [
{
"tid":"5e4bac9f-b47a-499e-8601-68fc16a9847c",
"psid":"Crest",
"devicePath":"c2818e07-4c09-42f0-ba24-ddb712573ab5_AL1352_192168221_80_X03_VVB001StatusB_Crest",
"processDataUnit":"",
"values":[
{
"value":4.6,
"timestamp":1648562285347
}
]
}
]
2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] mqtt.mapping.model.InnerNode             : Trying to resolve: '/moneo/c2818e07-4c09-42f0-ba24-ddb712573ab5/AL1352/192168221/80/X03/VVB001StatusB/Crest' in [/]
2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] mqtt.mapping.model.InnerNode             : Trying to resolve: 'moneo/c2818e07-4c09-42f0-ba24-ddb712573ab5/AL1352/192168221/80/X03/VVB001StatusB/Crest' in [moneo]
2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] mqtt.mapping.model.InnerNode             : Trying to resolve: '/c2818e07-4c09-42f0-ba24-ddb712573ab5/AL1352/192168221/80/X03/VVB001StatusB/Crest' in [/]
2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] mqtt.mapping.model.InnerNode             : Trying to resolve: 'c2818e07-4c09-42f0-ba24-ddb712573ab5/AL1352/192168221/80/X03/VVB001StatusB/Crest' in [c2818e07-4c09-42f0-ba24-ddb712573ab5]
2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] mqtt.mapping.model.InnerNode             : Trying to resolve: '/AL1352/192168221/80/X03/VVB001StatusB/Crest' in [/]
2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] mqtt.mapping.model.InnerNode             : Trying to resolve: 'AL1352/192168221/80/X03/VVB001StatusB/Crest' in [AL1352]
2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] mqtt.mapping.model.InnerNode             : Trying to resolve: '/192168221/80/X03/VVB001StatusB/Crest' in [/]
2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] mqtt.mapping.model.InnerNode             : Trying to resolve: '192168221/80/X03/VVB001StatusB/Crest' in [192168221]
2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] mqtt.mapping.model.InnerNode             : Trying to resolve: '/80/X03/VVB001StatusB/Crest' in [/]
2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] mqtt.mapping.model.InnerNode             : Trying to resolve: '80/X03/VVB001StatusB/Crest' in [80]
2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] mqtt.mapping.model.InnerNode             : Trying to resolve: '/X03/VVB001StatusB/Crest' in [/]
2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] mqtt.mapping.model.InnerNode             : Trying to resolve: 'X03/VVB001StatusB/Crest' in [X03]
2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] mqtt.mapping.model.InnerNode             : Trying to resolve: '/VVB001StatusB/Crest' in [/]
2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] mqtt.mapping.model.InnerNode             : Trying to resolve: 'VVB001StatusB/Crest' in [VVB001StatusB]
2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] mqtt.mapping.model.InnerNode             : Trying to resolve: '/Crest' in [/]
2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] mqtt.mapping.model.InnerNode             : Trying to resolve: 'Crest' in [Crest]
2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] mqtt.mapping.model.MappingNode           : Resolved mapping: Mapping(id=1, subscriptionTopic=/moneo/c2818e07-4c09-42f0-ba24-ddb712573ab5/AL1352/192168221/80/X03/VVB001StatusB/Crest, templateTopic=/moneo/c2818e07-4c09-42f0-ba24-ddb712573ab5/AL1352/192168221/80/X03/VVB001StatusB/Crest, indexDeviceIdentifierInTemplateTopic=-1, targetAPI=MEASUREMENT, active=true, tested=true, qos=AT_LEAST_ONCE, substitutions=[MappingSubstitution(pathSource=$now(), pathTarget=time, definesIdentifier=false), MappingSubstitution(pathSource=**.value, pathTarget=c8y_TemperatureMeasurement.T.value, definesIdentifier=false), MappingSubstitution(pathSource="k", pathTarget=c8y_TemperatureMeasurement.T.unit, definesIdentifier=false), MappingSubstitution(pathSource="moneo_device1", pathTarget=source.id, definesIdentifier=true)], mapDeviceIdentifier=false, createNonExistingDevice=false, updateExistingDevice=false, externalIdType=c8y_Serial, snoopStatus=STOPPED, lastUpdate=1666282647656), tp.size(): 0
2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] m.mapping.processor.impl.JSONProcessor   : Patched payload:false, [ {
  "tid" : "5e4bac9f-b47a-499e-8601-68fc16a9847c",
  "psid" : "Crest",
  "devicePath" : "c2818e07-4c09-42f0-ba24-ddb712573ab5_AL1352_192168221_80_X03_VVB001StatusB_Crest",
  "processDataUnit" : "",
  "values" : [ {
    "value" : 4.6,
    "timestamp" : 1648562285347
  } ]
} ]
2022-10-20 16:18:07.211  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] m.mapping.processor.impl.JSONProcessor   : Patched sub.pathSource: $now(), $now()
2022-10-20 16:18:07.212  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] m.mapping.processor.impl.JSONProcessor   : Evaluated substitution (pathSource, substitute): ($now(),"2022-10-20T16:18:07.212Z"), (pathTarget): (time), [ {
  "tid" : "5e4bac9f-b47a-499e-8601-68fc16a9847c",
  "psid" : "Crest",
  "devicePath" : "c2818e07-4c09-42f0-ba24-ddb712573ab5_AL1352_192168221_80_X03_VVB001StatusB_Crest",
  "processDataUnit" : "",
  "values" : [ {
    "value" : 4.6,
    "timestamp" : 1648562285347
  } ]
} ], {"c8y_TemperatureMeasurement":{"T":{"unit":"C","value":110}},"time":"2022-08-05T00:14:49.389+02:00","source":{"id":"909090"},"type":"c8y_TemperatureMeasurement"}
2022-10-20 16:18:07.212  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] m.mapping.processor.impl.JSONProcessor   : Patched sub.pathSource: **.value, **.value
2022-10-20 16:18:07.212  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] m.mapping.processor.impl.JSONProcessor   : Evaluated substitution (pathSource, substitute): (**.value,4.6), (pathTarget): (c8y_TemperatureMeasurement.T.value), [ {
  "tid" : "5e4bac9f-b47a-499e-8601-68fc16a9847c",
  "psid" : "Crest",
  "devicePath" : "c2818e07-4c09-42f0-ba24-ddb712573ab5_AL1352_192168221_80_X03_VVB001StatusB_Crest",
  "processDataUnit" : "",
  "values" : [ {
    "value" : 4.6,
    "timestamp" : 1648562285347
  } ]
} ], {"c8y_TemperatureMeasurement":{"T":{"unit":"C","value":110}},"time":"2022-08-05T00:14:49.389+02:00","source":{"id":"909090"},"type":"c8y_TemperatureMeasurement"}
2022-10-20 16:18:07.213  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] m.mapping.processor.impl.JSONProcessor   : Patched sub.pathSource: "k", "k"
2022-10-20 16:18:07.213  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] m.mapping.processor.impl.JSONProcessor   : Evaluated substitution (pathSource, substitute): ("k","k"), (pathTarget): (c8y_TemperatureMeasurement.T.unit), [ {
  "tid" : "5e4bac9f-b47a-499e-8601-68fc16a9847c",
  "psid" : "Crest",
  "devicePath" : "c2818e07-4c09-42f0-ba24-ddb712573ab5_AL1352_192168221_80_X03_VVB001StatusB_Crest",
  "processDataUnit" : "",
  "values" : [ {
    "value" : 4.6,
    "timestamp" : 1648562285347
  } ]
} ], {"c8y_TemperatureMeasurement":{"T":{"unit":"C","value":110}},"time":"2022-08-05T00:14:49.389+02:00","source":{"id":"909090"},"type":"c8y_TemperatureMeasurement"}
2022-10-20 16:18:07.213  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] m.mapping.processor.impl.JSONProcessor   : Patched sub.pathSource: "moneo_device1", "moneo_device1"
2022-10-20 16:18:07.213  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] m.mapping.processor.impl.JSONProcessor   : Evaluated substitution (pathSource, substitute): ("moneo_device1","moneo_device1"), (pathTarget): (source.id), [ {
  "tid" : "5e4bac9f-b47a-499e-8601-68fc16a9847c",
  "psid" : "Crest",
  "devicePath" : "c2818e07-4c09-42f0-ba24-ddb712573ab5_AL1352_192168221_80_X03_VVB001StatusB_Crest",
  "processDataUnit" : "",
  "values" : [ {
    "value" : 4.6,
    "timestamp" : 1648562285347
  } ]
} ], {"c8y_TemperatureMeasurement":{"T":{"unit":"C","value":110}},"time":"2022-08-05T00:14:49.389+02:00","source":{"id":"909090"},"type":"c8y_TemperatureMeasurement"}
2022-10-20 16:18:07.263  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] mqtt.mapping.processor.PayloadProcessor  : Found id 3690875 for external id: moneo_device1
2022-10-20 16:18:07.333  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] mqtt.mapping.core.C8yAgent               : New measurement posted: {"self":"http://t161291189.eu-latest.cumulocity.com/measurement/measurements/3691186","time":"2022-10-20T16:18:07.212Z","id":"3691186","source":{"self":"http://t161291189.eu-latest.cumulocity.com/inventory/managedObjects/3690875","id":"3690875"},"type":"c8y_TemperatureMeasurement","c8y_TemperatureMeasurement":{"T":{"unit":"k","value":null}}}
2022-10-20 16:18:07.333  INFO 13 --- [MQTT Call: cumulocity-mqtt-test_d1] m.mapping.processor.impl.JSONProcessor   : Added payload for sending: {"c8y_TemperatureMeasurement":{"T":{"unit":"k"}},"time":"2022-10-20T16:18:07.212Z","source":{"id":"3690875"},"type":"c8y_TemperatureMeasurement"}, MEASUREMENT, numberDevices: 1

Add roles for microservice endpoints

We should secure the REST endpoints with at least 1 or 2 roles.
ADMIN - can make configuration changes to MQTT Broker config etc.
USER - can add/edit/delete mappings

We could also make this more sophisticated but that makes it much more complex to assign the correct roles to the users.

Not appending device ID in device name

Currently the device ID will be added to the device name in C8Y.
This should be optionable. The expression language allows such things but we shouldn't do this hard coded.

Write concept for multi tenancy

Currently the generic mqtt agent only works for a single tenant.
What should happen if multiple tenants subscribe to the microservice?
If multiple tenants subscribe to the ms they all would create their own mappings.
All use one connection to the mqtt broker, so their topics couls possibly interfere and be defined twice, e.g.:

tenant 1 uses topic "myAlarms"
tenant 2 uses topic "myAlarms"

So topics need to be unique across different tenants. This seems to be a strong requirement and violates the isolations of tenant data.

NullPointerException when deleting outbound mapping

In the methods MappingComponent.deleteFromMappingCache() a NullPointerException was raised when deleting outbound mapping.

Adding chart to monitoring page.

The monitoring page now contains a simple bar chart aggregating all counts for inbound and outbound mappings.

When restarting mqtt-mapping-service the mappingStatus is current is overwritten

When the microservice mqtt-mapping-serviceis restarted the monitoring statistic of the defined mappings is loaded from C8Y managedObject (marked with the externalID GENT_ID = "MQTT_MAPPING_SERVICE").
The persisted mappingStatus is not loaded correctly upon initialisation.

Expressions in mappings

The data retrieved by MQTT might not be able to mapped directly to cumulocity.
Most likely we need to transform or calculate the data retrieved by using a simple expression language.

Suggested expressions

Strings:

substring with an index. Example. {{device.id}}.substring(0,4) -> will return the first 4 chars of the string from left.
append / concat: Append two string from different json paths: device.id + device.id.extension
split / tokenize with an delimiter returning an array with an index: Example: {{device.id}}.split("-")[2]

Numbers:

basic operations like: +,-,*,/. Example: {{sensor.value}} / 1000, ({{sensor.value}} + 100 ) / 1000
Math functions like sqrt, pow, round etc. Example: {{sensor.value}}.round(2) while the number provided is the decimal places

Dates:

get UTC formatted date from TS. {{device.ts}}.getUTCDateTime("format")

Mapping of json keys

I have a usecase, where there are many different measurements and more may also be added over time. To help that i don't want to have a mapping for each measurement, but only one dynamic mapping that covers that.

Threfore i have to map the keys of the source message (eg. /devicename/measurementname { "series-name": 12 } ) to the mqtt message format of cumulocity where the measurement name and series names are keys not values . In this example from a topic to a json key ( [ _TOPIC_LEVEL_[1] -> $keys()[3] ] ) and from a json key to a json key for the series name: [ $keys()[0] -> $keys($eval($keys()[3]))[0] ] .
To put the value into the series i used: [ $eval($keys($)[0]) -> **.value[0] ]

I know that with that kind of mapping the order of the fragments is relevant what is not json like but i think thats the only option.

I have no experience with jsonata but my queries worked in the online jsonata tester (https://try.jsonata.org/) . The testing in the c8y app seems to have failed (screenshot).

Is it in somehow possible to map the keys or is this service limited to value mapping?

UI in first step to define topic is misplaced

The option "Update Existing Device" is misplaced on the first step wizard to define topic.

Adding feature to remove a attribute in target when left side is null

The option REMOVE_IF_NULL allows to remove an attribute in the target payload, if the left side evaluates to null.
If you have payloads with varying content:

{
    "Measurementname": "Airsensor", 
    "Seriesname": "Humidity", "value": 10,
    "unit": "%"
}

and

{
    "Measurementname": "Liquidsensor", 
    "Seriesname": "Level", "value": 10,
    "unit": "l"
}

for the target payload:


{
    "Airsensor": "dummy",
    "Liquidsensor": "dummy",
    "time": "2022-08-05T00:14:49.389+02:00", "source": {
    "id": "909090" },
    "type": "c8y_measurementtype"
}

The mapping can use repair strategy "REMOVE_IF_NULL"
A mapping could now be defined as follows:

* [ Measurementname = "Airsensor" ? {Seriesname:{"value": value, "unit": unit}} : null -> Airsensor ] / RepairStrategy: REMOVE_IF_NULL
* [ Measurementname = "Liquidsensor" ? {Seriesname:{"value": value, "unit": unit}} : null -> Liquidsensor ] /RepairStrategy: REMOVE_IF_NULL

UI for configuring the broker connections allows to copy an existing connection configuration.

Thhis feature was implemented in: cecc349

Error when pulishing payload to mapping in snoop mode

When publishing a test payload to the mapping binaryEvent/berlin_01 an exception occurs:

2023-07-30 11:27:44.340  INFO 11 --- [pool-2-thread-6] m.m.p.inbound.AsynchronousDispatcher     : New message on topic: 'binaryEvent/berlin_01', wrapped message: {"message":"57652077696c6c20616c77617973206170707265636961746520746865206e6578742072656c656173652031302e313721"}
2023-07-30 11:27:44.340  INFO 11 --- [pool-2-thread-6] m.m.processor.inbound.JSONProcessor      : Patched payload: {
  "message" : "57652077696c6c20616c77617973206170707265636961746520746865206e6578742072656c656173652031302e313721",
  "_TOPIC_LEVEL_" : [ "binaryEvent", "berlin_01" ]
}
2023-07-30 11:27:44.342 ERROR 11 --- [pool-2-thread-6] m.m.processor.inbound.JSONProcessor      : Exception for: "Temp: "&$parseInteger($string("0x"&$substring(message,0,2)),"0")&" C", {
  "message" : "57652077696c6c20616c77617973206170707265636961746520746865206e6578742072656c656173652031302e313721",
  "_TOPIC_LEVEL_" : [ "binaryEvent", "berlin_01" ]
}

com.api.jsonata4java.expressions.EvaluateException: Unknown function: $parseInteger
	at com.api.jsonata4java.expressions.Expressions.evaluate(Expressions.java:160)
	at mqtt.mapping.processor.inbound.JSONProcessor.extractFromSource(JSONProcessor.java:99)
	at mqtt.mapping.processor.inbound.AsynchronousDispatcher$MappingProcessor.lambda$call$1(AsynchronousDispatcher.java:149)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at mqtt.mapping.processor.inbound.AsynchronousDispatcher$MappingProcessor.call(AsynchronousDispatcher.java:91)
	at mqtt.mapping.processor.inbound.AsynchronousDispatcher$MappingProcessor.call(AsynchronousDispatcher.java:61)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

Add repair strategy if substitute value is null

To define a mapping for flexible payloads a setting is required to remove nodes in the target. Both payload should be handled by ONE mapping.

PAYLOAD 1:

(device side)

 "Measurementname": "Airsensor",
 "Seriesename":"Humidity",
 "value":10,
 "unit":"%",
}```

map to > 

(cumulocity side)

```{
  "Airsensor": {
    "Humidity": {
      "unit": "%",
      "value": 10
    }
  }
 "time": ....
...
}```
-------
PAYLOAD 2:

(device side)
```{
 "Measurementname": "Watersensor",
 "Seriesename":"Level",
 "value":20,
 "unit":"%",
}```

map to > 

(cumulocity side)
```{
  "Watersensor": {
    "Level": {
      "unit": "%",
      "value": 20
    }
  }
 "time": ....
...
}```

Reference MQTT mappings

Example MQTT topic structure from new IIoT customer:

<plant>/<line>/<device>_<MeasurementType>

This should result in the cy8_Serial: <plant>_<line>_<device> and with MeasurementType in the measurement payload

Cumulocity IoT Edge support

When using Cumulocity IoT edge we have to disable the notification API until it is available on the edge.

Quick Fix: Add a configuration property to disable operation handling completely.
In the microservice we will check that property and ignore any Notification 2.0 Code.
Also we will provide a status for the UI so it can disable the Views for Outbound Mappings.

The UI might need to fetch the status from an endpoint and hide the downstream mappings

Capability of modifying the root of the message

Dear awesome team,

It would be really great if there's a possibility of editing the root of the message in order to create custom JSON measurements based even in the type get from a MQTT message. For one use case we do have a JSON input like:

{
"values": [
{
"key": "velocidad_cabezal",
"value": 136.34
},
{
"key":"temperature",
"value":25
}
],
"TOPIC_LEVEL": [
"v2",
"things",
"test"
]
}

And the idea is to be able to create an output like:

{
"time": "2023-10-31T11:11:23.747Z",
"source": {
"id": "9299280"
},
"type": "c8y_DynamicMeasurement",
"c8y_velocidad_cabezal": {
"Measurement": {
"value": 1211.34,
"key": "U"
}
},
"c8y_temperature": {
"Measurement": {
"value": 230,
"key": "U"
}
},
"id": "983171",
"lastUpdated": "2023-10-31T11:11:23.752Z"
}

enabling/disabling outbound mapping not possible

NullPointer Exception thrown when enabling/disabling outbound mapping in
mqtt.mapping.core.MappingComponent.java

repeated values for filterOutbound throw exception

The values for filterOutbound over all outbound mappings have to be unique.
When adding a second mapping with an already existing filterOutbound an internal exception in thrown.

Add a validation rule to prevent this.

Device ID mapping

In MQTT the device id could be either be part of the topic structure or part of the payload. We want to support both ideally.

Topic

The topics are normally separated by "/" and can contain wildcards "#". To map the device identifier we can split the received topics by "/" and then use an index pointing to the string containing the id.

Example 1: device/sample/123123123 --> in this case we subscribe on "device/sample/#" and know that the device is on the last position of the topic (this is currently supported).

Example 2: device/123123123/1/sample --> in this case we subscribe on "device/#" but we need to define that the device is on the second position of the topic structure.

To support all the examples above we need to define the position of the device identifier within the topic structure. A valid mapping could look like this: TOPIC[1] -> ext.id,type. While "[1]" is pointing to the index of split topics meaning on second position of the topic structure.
Here we should make sure that the external ID Type can be specified defaulted to "c8y_Serial".

Subscriptions without wildcards only make sense when the device id is part of the payload e.g. containing multiple device information. This could also be the case for Gateways sending their ID in the topic but child device IDs in Payload or subtopics.

Topic Mapping

One major problem is that we cannot bind only one mapping to a subscription topic but most likely to a topic template:
Coming back to this example: device/123123123/1/sample we might have "sample" but also "data" and "metadata as subtopics. For each of them we need different mappings. So the mapping must be bound to the template
"device/#/1/sample" or "device/#/1/data" while "device/#" is still the subscribed topic.

I would suggest to add a subscribe topic and a template topic while internally when defining multiple mapper on the same subscribed topics they should share the subscription but filter on the retrieved topic which mapper to be executed. If template topic is not defined it will use the subscribed topic.

Payload

When using static topics like "/device/data" the device identifier is part of the payload which can be a single device or can contain multiple devices data.

For a JSON Object we can simply define the path where the device identifier is located and map it the external id.

For a JSON Array containing multiple devices we should support iteration and concrete index.
For each entry in the array we will map a path to the external id or we can specify that the information is in the first position of an array.

Example 3: The payload looks like this:

{
	"device": {
		"identifier": {
			"serial": "123123123"
		}
	}
}

Mapping: device.identifier.serial -> ext.id,type

Example 4: The payload looks like this:

{
	"devices": [
		{
			"identifier": {
				"serial": "123123123"
			}
		}
	]
}

Mapping: devices[*].identifier.serial -> ext.id,type

Example 5: The payload looks like this:

{
	"devices": [
		"123123123",
                "DeviceType"
	]
}

Mapping: devices[1] -> ext.id,type

Active mapping are processed in the MappingProcessor

When a mapping is deactivated it should not be processed.

ExtensibleProcessorInbound was not properly created with extensions

The ExtensibleProcessorInbound was not properly created with extensions.This resulted in a bug and inbound mappings using a processor extension could not be processed properly.