edgexfoundry / device-mqtt-go Goto Github PK

Owner: Device WG

License: Apache License 2.0

Dockerfile 3.18% Makefile 3.47% Shell 2.18% Go 91.17%

device-mqtt-go's Introduction

Device MQTT Go

Warning
The main branch of this repository contains work-in-progress development code for the upcoming release, and is not guaranteed to be stable or working. It is only compatible with the main branch of edgex-compose which uses the Docker images built from the main branch of this repo and other repos.

The source for the latest release can be found at Releases.

Overview

MQTT Micro Service - device service for connecting a MQTT topic to EdgeX acting like a device/sensor feed.

Documentation

For latest documentation please visit https://docs.edgexfoundry.org/latest/microservices/device/services/device-mqtt/Purpose

Build with NATS Messaging

Currently, the NATS Messaging capability (NATS MessageBus) is opt-in at build time. This means that the published Docker images do not include the NATS messaging capability.

The following make commands will build the local binary or local Docker image with NATS messaging capability included.

make build-nats
make docker-nats

The locally built Docker image can then be used in place of the published Docker image in your compose file. See Compose Builder nat-bus option to generate compose file for NATS and local dev images.

Packaging

This component is packaged as docker images. Please refer to the Dockerfile and Docker Compose Builder scripts.

Community

Discussion: https://github.com/orgs/edgexfoundry/discussions
Mailing lists: https://lists.edgexfoundry.org/mailman/listinfo

License

Apache-2.0

device-mqtt-go's People

Contributors

Stargazers

Watchers

Forkers

wangnuannuan jpwku anonymouse64 juanmagal ahaedgar sally-yu devfallingstar sivakrishnan-intel xixidefujun34 viethoangeng tmpowers adam-intel worldmaomao kaponz lenny-goodell alihanyalcin zhuzcasd lranjbar shitou8080 venkata-lakshmi-penna zettaflowio faisaljamil2010 isgasho tonyespy super-deng ernestojeda kurokobo eno-intel jamesrgregg jbonafide623 venkata-subbareddyk mitacsitc bill-mahoney caiyuexiang vgajjala siggiskulason felixting iain-anderson isabella232 haoyuli1990 wontdoanything apps-incloud thyun4synctechno jim-wang-intel jenmwms ejlee3 jpwhitemn gpt-projects-awesome xtangram farshidtz tpiperatgod weichou1229 jhhe66 tttchinjava snowfly568790005 monicaisher stjordanis igit-cn jinlinguan vli11 dunstable h0ryz0n texasoct marcpfuller wackxu jumpingliu jackchenjc cloudxxx8 avvocato-sudo kwang1971 charithmadhuranga dummyfish56 shilianming

device-mqtt-go's Issues

Broken link in README

The link to the document (https://github.com/edgexfoundry/edgex-go/blob/master/docs/examples/Ch-ExamplesMQTTDeviceService.rst) is broken.

Update VERSION to 1.1.0 in master

1.1.0 is the version number of fuiji

device-mqtt ProtocolDriver Handlers connect to broker on each request

While debugging issue #87, I noticed that the ProtocolDriver connects to the broker on each request vs. maintaining a consistent connection to the broker while running. I'm not sure if this was an intentional design decision or not, but it seems like maintaining a persistent connection might be a better approach?

When I start the service, I see the following log message each time a new AutoEvent is triggered:

level=INFO ts=2019-08-15T20:30:57.328441876Z app=edgex-device-mqtt source=driver.go:279 msg="Create MQTT client and connection: uri=tcp://admin:[email protected]:1883 clientID=CommandPublisher "

Here are the versions of the snaps I'm using:

edgex-device-mqtt 1.0.0-20190801+fd7235a
edgexfoundry 1.0.1-20190729+c28b77d1

Add snap packaging

As per the Seoul F2F, the decision was made to support device services as snaps and one of the ones we will release/support with Edinburgh is device-mqtt, with the snap name edgex-device-mqtt. We will need to add the snapcraft.yaml file to this repo, as well as setup jenkins jobs for verifying the snap and staging/releasing the snap to the edge channel.

To do this, we should do the following things in order:

Open a PR here with the snapcraft.yaml added to the repo (but don't merge)
Deploy jenkins snap verify job for this repo to the jenkins sandbox
Open PR to ci-management with the snap verify and stage jobs, referencing the sandbox jobs for testing, etc.
Merge the ci-management PR
Close the original PR from 1, and re-open to trigger the new verify jobs
Merge the PR here

Should probably also update the README here to mention the snap

This issue can be assigned to me

Remove ReadMaxLimit from configuration

To address the change of edgexfoundry/device-sdk-go#303

Refactor reading value's handling method

The handleReadingStringValue function is too big so I want to refactor it to another file

After that, I will refactor convertReadingValueType and checkValueInRange function.

But I already refactor these functions and separate into three commits. Could I put these commits in one PR? Or create three PR?

Update the version of device-sdk-go to v1.0.0 in go.mod for edinburgh branch

The device-sdk-go is tagging the release v1.0.0, and all Device Services should use this dependency.

Change the name of the LICENSE file

Please change the license-2.0.txt file name to LICENSE per https://wiki.edgexfoundry.org/pages/viewpage.action?pageId=21823866

Update version number to 0.7.1 for Delhi dot release and create Attribution.txt

AutoEvents fail when service is enabled without a peer MQTT application running

Running the 1.0.1 of Edgex via the snap (currently in the beta channel) and then install the edgex-device-mqtt snap (version 1.0.0-20190627+29babbb), when I check the device-mqtt log, the AutoEvent configured for every 20s seems to be broken as I'm seeing the following repeated pattern of log messages:

level=DEBUG ts=2019-08-01T23:53:31.903028584Z app=edgex-device-mqtt source=executor.go:42 msg="AutoEvent - executing {"frequency":"20s","resource":"testrandfloat32"}"
level=DEBUG ts=2019-08-01T23:53:31.904818944Z app=edgex-device-mqtt source=command.go:200 msg="Handler - execReadCmd: deviceResource: randfloat32"
level=DEBUG ts=2019-08-01T23:53:31.915691844Z app=edgex-device-mqtt source=command.go:207 msg="Handler - execReadCmd: deviceResource: {"description":"device random number with Base64 encoding","name":"randfloat32","properties":{"value":{"type":"Float32","readWrite":"R","minimum":"100.00","maximum":"0.00","defaultValue":"0.00","size":"4","floatEncoding":"Base64"},"units":{"type":"String","readWrite":"R"}}}"
level=INFO ts=2019-08-01T23:53:31.923028135Z app=edgex-device-mqtt source=driver.go:279 msg="Create MQTT client and connection: uri=tcp://admin:[email protected]:1883 clientID=CommandPublisher "
level=INFO ts=2019-08-01T23:53:31.933729185Z app=edgex-device-mqtt source=driver.go:154 msg="Publish command: {"cmd":"randfloat32","method":"get","uuid":"5d437b7b098f1f0a08a7cd43"}"
level=INFO ts=2019-08-01T23:53:36.944297628Z app=edgex-device-mqtt source=driver.go:122 msg="Handle read commands failed: can not fetch command response: method=get cmd=randfloat32"
level=ERROR ts=2019-08-01T23:53:36.947428952Z app=edgex-device-mqtt source=executor.go:45 msg="AutoEvent - error occurs when reading resource testrandfloat32"

Update registry argument value in snap

The snap currently has:

device-mqtt-go/snap/snapcraft.yaml

Line 30 in 34de416

    
           command: bin/device-mqtt -confdir $SNAP_DATA/config/device-mqtt -profile res --registry

We need to change it to use the full registry specification like so:

https://github.com/edgexfoundry/edgex-go/blob/ebd3e33187214f9cfb58aa9cb02c876e37963ec2/snap/snapcraft.yaml#L284-L285

both "autoevent" and "scheduler" call "HandleReadCommands" which causes crash

Using 1.0.1 images, I add one mock-mqtt device which sends data "randfloat64" every 5s, set up a "5s" autoevent of deviceresoure "ping", and one scheduler of "testmessage" occur every 3s, then I find the edgex-device-mqtt frequently disconnect and reconnect to the broker, after several minutes the service panics with following kinds of errors.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x46f039]
goroutine 4102 [running]:
io.ReadAtLeast(0x0, 0x0, 0xc0001930c0, 0x1, 0x1, 0x1, 0xa4dea0, 0xa54f01, 0xc0001930c0)
       /usr/local/go/src/io/io.go:310 +0x59
io.ReadFull(0x0, 0x0, 0xc0001930c0, 0x1, 0x1, 0x1, 0x0, 0xc000562de8)
       /usr/local/go/src/io/io.go:329 +0x58
github.com/eclipse/paho.mqtt.golang/packets.ReadPacket(0x0, 0x0, 0x0, 0x0, 0x0, 0xa90300)
       /go/pkg/mod/github.com/eclipse/[email protected]/packets/packets.go:105 +0x9f
github.com/eclipse/paho%2emqtt%2egolang.(*client).connect(0xc000631c00, 0xc414e0)
       /go/pkg/mod/github.com/eclipse/[email protected]/client.go:399 +0xe9
github.com/eclipse/paho%2emqtt%2egolang.(*client).Connect.func1(0xc000631c00, 0xc0001140b0, 0xc0005bdb80)
       /go/pkg/mod/github.com/eclipse/[email protected]/client.go:223 +0x3c9
created by github.com/eclipse/paho%2emqtt%2egolang.(*client).Connect
       /go/pkg/mod/github.com/eclipse/[email protected]/client.go:195 +0x1eb

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x8b24de]
goroutine 589 [running]:
github.com/eclipse/paho%2emqtt%2egolang.(*client).internalConnLost(0xc0003e5c00, 0xc41480, 0xc0000baa00)
       /go/pkg/mod/github.com/eclipse/[email protected]/client.go:460 +0x5e
created by github.com/eclipse/paho%2emqtt%2egolang.errorWatch
       /go/pkg/mod/github.com/eclipse/[email protected]/net.go:332 +0x1df

the reason may be as stated in the title. both "autoevent" and "scheduler" call "HandleReadCommands" and register to mqtt broker using the same "clientID" get from "connectionInfo.ClientId", conflict arises and for some reason the service panics.
when I leave the "ClientId" in protocol to be blank, it works all fine.

Upgrade to latest device-sdk and core-contract

Upgrade the lib to use the latest device-sdk and core-contract

Update due to Device changes and Schedule* deprecation

The example configuration file needs to be updated to the replacement of Device.Addressable with Device.Protocols, and the deprecation of Schedule* objects by device-sdk-go.

The same Device configuration.toml and example driver changes that were made to device-sdk-go in commit fc71221 need to be applied.

Likewise the Schedule* configuration.toml changes (see device-sdk-go commit 2db9552 also need to be applied.

Docker version of configuration.toml doesn't set ProfilesDir

When running in docker the profiles aren't found because ProfilesDir is set to ""

CommandResponses map is not thread safe

Driver.CommandResponses is not thread safe
https://github.com/edgexfoundry/device-mqtt-go/blob/master/internal/driver/driver.go#L49

Please refer to the resolution
edgexfoundry/device-random#67

why lastconnected value of device not be updated, when use device-mqtt-go send data to edgex-go?

Update device-sdk-go and add AutoEvents to example

reference: https://github.com/edgexfoundry/device-sdk-go/blob/master/example/cmd/device-simple/res/configuration.toml#L68

Restructure configuration.toml for new Writeable section and Abstract Registry

Once the Abstract Registry integration into device-sdk-go is merged the go.mod file needs to be updated to pull latest device-sdk-go and then the following changes need to be made to the following files:

    - cmd/res/configuration.toml
    - cmd/res/docker/configuration.toml

Add the Writable section as:

[Writable]
LogLevel = 'INFO'

Modify the Logging section to be:

[Logging]
EnableRemote = false
File = "./device-simple.log"

Also add following to Registry section

Type = "consul"

Remove --registry from Dockerfile

According to change of edgexfoundry/device-sdk-go#274, once updating the device-sdk-go the latest version in go.mod, --registry has to include the registry URL.
Thus, all the Device Services should not include --registry in the command line. It is consistent with the C Device Services.

Error "Handle write commands failed: can not fetch command response"

Hello,

I am trying to follow the doc : https://docs.edgexfoundry.org/Ch-ExamplesAddingMQTTDevice.html
in order to test mqtt by sending rand number. However i am block to 6.5.8, i can't execute the put and get command.

Best Regards

Reading data type conversion problem: float64 to int64、int32....

==================================================================
device-mqtt-go/internal/dirver/incominglistener.go Line 54

func onIncomingDataReceived(client mqtt.Client, message mqtt.Message) {
	var response map[string]interface{}
	json.Unmarshal(message.Payload(), &response)
        .......

device-mqtt-go/internal/dirver/driver.go Line 296

func newResult(deviceObject models.DeviceObject, ro models.ResourceOperation, reading interface{}) (*sdkModel.CommandValue, error) {
	var result = &sdkModel.CommandValue{}
	var err error
	var resTime = time.Now().UnixNano() / int64(time.Millisecond)

	switch deviceObject.Properties.Value.Type {
	case "Bool":
		result, err = sdkModel.NewBoolValue(&ro, resTime, reading.(bool))
	case "String":
		result = sdkModel.NewStringValue(&ro, resTime, reading.(string))
	case "Uint8":
		result, err = sdkModel.NewUint8Value(&ro, resTime, reading.(uint8))
	case "Uint16":
		result, err = sdkModel.NewUint16Value(&ro, resTime, reading.(uint16))
	case "Uint32":
		result, err = sdkModel.NewUint32Value(&ro, resTime, reading.(uint32))
	case "Uint64":
		result, err = sdkModel.NewUint64Value(&ro, resTime, reading.(uint64))
	case "Int8":
		result, err = sdkModel.NewInt8Value(&ro, resTime, reading.(int8))
	case "Int16":
		result, err = sdkModel.NewInt16Value(&ro, resTime, reading.(int16))
	case "Int32":
		result, err = sdkModel.NewInt32Value(&ro, resTime, reading.(int32))
	case "Int64":
		result, err = sdkModel.NewInt64Value(&ro, resTime, reading.(int64))
	case "Float32":
		result, err = sdkModel.NewFloat32Value(&ro, resTime, reading.(float32))
	case "Float64":
		result, err = sdkModel.NewFloat64Value(&ro, resTime, reading.(float64))
	default:
		err = fmt.Errorf("return result fail, none supported value type: %v", deviceObject.Properties.Value.Type)
	}

	return result, err
}

===================================================================

I send the data {"name":"test-device","cmd":"speed","speed":12} to DataTopic, the type of response["speed"] is float64 via json.Unmarshal(message.Payload(), &response).
In this way, both int64 and int32 will be unified into float64. So, It needs to convert the reading into a data type defined in valueDescriptor.
In this case, the type of speed is int64 in valueDescriptor. So it will run case "Int64": result, err = sdkModel.NewInt64Value(&ro, resTime, reading.(int64)). Howerver, this will lead to panic : panic: interface conversion: interface {} is float64, not int64. I hope that I can pay attention and fix such bugs as soon as possible.

In order to verify that converting reading is the wrong way, I do a simple test.

package main

import (
	"encoding/json"
	"fmt"
	"reflect"
)

func main() {

	var response map[string]interface{}
	a := []byte(`{"name":"test-device","cmd":"speed","speed":12}`)
	json.Unmarshal(a, &response)
	//reading := response["speed"].(int64) // panic: interface conversion: interface {} is float64, not int64
	reading := int64(response["speed"].(float64))
	fmt.Println(reading)  // 12
	fmt.Println(reflect.TypeOf(reading)) // int64

}

Failed to create the pre-defined Devices

I use branch delhi, and in ./cmd, run go run main.go

Init: useRegistry: false profile:  confDir: 
Bypassing registration in registry...
Calling service.Start.
EnableRemote is false, using local log file
INFO: 2019/01/12 15:32:08 Check Metadata service's status ...
INFO: 2019/01/12 15:32:08 Check Data service's status ...
INFO: 2019/01/12 15:32:08 Service clients initialize successful.
DEBUG: 2019/01/12 15:32:08 Trying to find Device Service: edgex-device-mqtt
INFO: 2019/01/12 15:32:08 Device Service  doesn't exist, creating a new one
ERROR: 2019/01/12 15:32:08 AddressableForName failed: invalid ObjectId in JSON: "4481c9f5-4669-4e9e-921b-1d8697624824"
ERROR: 2019/01/12 15:32:08 makeNewAddressable failed: invalid ObjectId in JSON: "4481c9f5-4669-4e9e-921b-1d8697624824"
DEBUG: 2019/01/12 15:32:08 Device Service in Core MetaData: {"created":0,"modified":0,"origin":0,"description":"","id":null,"name":null,"lastConnected":0,"lastReported":0,"operatingState":"","labels":null,"addressable":{"created":0,"modified":0,"origin":0,"id":null,"name":null,"protocol":null,"method":null,"address":null,"port":0,"path":null,"publisher":null,"user":null,"password":null,"topic":null,"baseURL":null,"url":null},"adminState":""}
ERROR: 2019/01/12 15:32:08 Device cache initialization failed: Item not found
ERROR: 2019/01/12 15:32:08 Schedule Event cache initialization failed: Item not found
DEBUG: 2019/01/12 15:32:08 Loading pre-define Devices from configuration: [{MQTT test device Test.Device.MQTT.Profile MQTT device is created for test purpose [MQTT test] {"created":0,"modified":0,"origin":0,"id":null,"name":"Gateway address","protocol":"TCP","method":null,"address":"127.0.0.1","port":1883,"path":null,"publisher":"CommandPublisher","user":"admin","password":"public","topic":"CommandTopic","baseURL":"TCP://127.0.0.1:1883","url":"TCP://127.0.0.1:1883"}}]
DEBUG: 2019/01/12 15:32:08 Device MQTT test device doesn't exist, creating a new one
ERROR: 2019/01/12 15:32:08 Device Profile Test.Device.MQTT.Profile doesn't exist for Device {MQTT test device Test.Device.MQTT.Profile MQTT device is created for test purpose [MQTT test] {"created":0,"modified":0,"origin":0,"id":null,"name":"Gateway address","protocol":"TCP","method":null,"address":"127.0.0.1","port":1883,"path":null,"publisher":"CommandPublisher","user":"admin","password":"public","topic":"CommandTopic","baseURL":"TCP://127.0.0.1:1883","url":"TCP://127.0.0.1:1883"}}
ERROR: 2019/01/12 15:32:08 creating Device from config failed: {MQTT test device Test.Device.MQTT.Profile MQTT device is created for test purpose [MQTT test] {"created":0,"modified":0,"origin":0,"id":null,"name":"Gateway address","protocol":"TCP","method":null,"address":"127.0.0.1","port":1883,"path":null,"publisher":"CommandPublisher","user":"admin","password":"public","topic":"CommandTopic","baseURL":"TCP://127.0.0.1:1883","url":"TCP://127.0.0.1:1883"}}
error: Failed to create the pre-defined Devices
exit status 1

Convert to Go Module

This service needs to be converted to a Go module in order to eliminate the use of glide.

More information
https://github.com/golang/go/wiki/Modules#migrating-to-modules

Note that the device-sdk-go is already a module.

Recommend this be sequenced after a PR is merged to address the following issue
edgexfoundry/device-sdk-go#173

Integrate DeviceProfile Resources / Commands Name Change

Update the example device profile and device-sdk-go, core-contract.
Resources -> DeviceCommands
Commands -> CoreCommands

Read driver config from configuration.toml by sdk API

According to PR edgexfoundry/device-sdk-go#207 is merged, device-mqtt can add config in configration.tmol and read by device-sdk-go API.

Update top-level README.md configuration instructions

The top-level README.md file in the master branch needs some work, as it's out-of-date and lacks basic information about the capabilities and basic functions of this device service.

Some issues include:

It references (in poor English) a deprecated/archived page which describes how the original Java device-mqtt service worked. Instead of linking to an old document, it would make much more sense for this top-level README page to describe the high-level design of this service, and how it can be used.
It still references the deprecated configuration-driver.toml file. This has been replaced by the [Driver] section of configuration.toml. It also doesn't explain why you would do this, or how (especially when running via docker containers) you'd do this.
The "Device list" section mentions how to modify the Device list in order to automatically add devices and device profiles on service startup, yet it only describes the Device list, there's nothing about device profiles.
I know there's additional documentation in our new documentation repository. It'd be great if the top-level README file had a link to this documentation. Ideally this would be standard for all reference device services.

Error while setting up with docker-compose

While I following the official docs with MQTT device service, I got an error situation with device-mqtt.

On consul dashboard, it shows the health checking is not working properly.
After some conversation in slack, Me and @mhall119 found that the device-mqtt is not working with the docker-compose.yml script, version delhi 0.7.1, with uncommented device-mqtt section.

This is the error when I run a command docker-compose up device-mqtt after I run docker-compose up -d and got weird situation as described above.

yang-yuseong-ui-MacBookPro:edgex-device yousungyang$ docker-compose up device-mqtt
edgex-files is up-to-date
edgex-mongo is up-to-date
edgex-core-consul is up-to-date
Starting edgex-config-seed ... done
edgex-support-logging is up-to-date
edgex-core-metadata is up-to-date
edgex-core-data is up-to-date
edgex-core-command is up-to-date
Starting edgex-device-mqtt ... done
Attaching to edgex-device-mqtt
edgex-device-mqtt | Init: useRegistry: true profile: docker confDir: /res
edgex-device-mqtt | Check registry is up... http://edgex-core-consul:8500
edgex-device-mqtt | Register the Service ...
edgex-device-mqtt | Register the Health Check ...
edgex-device-mqtt | Register in registry...
edgex-device-mqtt | Calling service.Start.
edgex-device-mqtt | EnableRemote is false, using local log file
edgex-device-mqtt | INFO: 2019/02/20 16:49:55 Check Data service's status ...
edgex-device-mqtt | INFO: 2019/02/20 16:49:55 Check Metadata service's status ...
edgex-device-mqtt | INFO: 2019/02/20 16:49:55 Service clients initialize successful.
edgex-device-mqtt | INFO: 2019/02/20 16:49:55 Device Service edgex-device-mqtt exists
edgex-device-mqtt | INFO: 2019/02/20 16:49:55 Create MQTT client and connection: uri=tcp://admin:[email protected]:1883 clientID=CommandResponseSubscriber
edgex-device-mqtt | INFO: 2019/02/20 16:49:55 Create MQTT client and connection: uri=tcp://admin:[email protected]:1883 clientID=IncomingDataSubscriber
edgex-device-mqtt | INFO: 2019/02/20 16:49:55 Starting internal Scheduler
edgex-device-mqtt | INFO: 2019/02/20 16:49:55 Started internal Scheduler
edgex-device-mqtt | panic: close of nil channel
edgex-device-mqtt |
edgex-device-mqtt | goroutine 90 [running]:
edgex-device-mqtt | github.com/edgexfoundry/device-mqtt-go/vendor/github.com/eclipse/paho%2emqtt%2egolang.(*client).closeStop(0xc000059880)
edgex-device-mqtt | 	/go/src/github.com/edgexfoundry/device-mqtt-go/vendor/github.com/eclipse/paho.mqtt.golang/client.go:482 +0x12f
edgex-device-mqtt | github.com/edgexfoundry/device-mqtt-go/vendor/github.com/eclipse/paho%2emqtt%2egolang.(*client).disconnect(0xc000059880)
edgex-device-mqtt | 	/go/src/github.com/edgexfoundry/device-mqtt-go/vendor/github.com/eclipse/paho.mqtt.golang/client.go:495 +0x2f
edgex-device-mqtt | github.com/edgexfoundry/device-mqtt-go/vendor/github.com/eclipse/paho%2emqtt%2egolang.(*client).Disconnect(0xc000059880, 0x1388)
edgex-device-mqtt | 	/go/src/github.com/edgexfoundry/device-mqtt-go/vendor/github.com/eclipse/paho.mqtt.golang/client.go:439 +0x1f9
edgex-device-mqtt | github.com/edgexfoundry/device-mqtt-go/internal/driver.startCommandResponseListening(0x89ec40, 0xc00011abf0)
edgex-device-mqtt | 	/go/src/github.com/edgexfoundry/device-mqtt-go/internal/driver/responselistener.go:38 +0x54f
edgex-device-mqtt | github.com/edgexfoundry/device-mqtt-go/internal/driver.(*Driver).Initialize.func1()
edgex-device-mqtt | 	/go/src/github.com/edgexfoundry/device-mqtt-go/internal/driver/driver.go:70 +0x26
edgex-device-mqtt | created by github.com/edgexfoundry/device-mqtt-go/internal/driver.(*Driver).Initialize
edgex-device-mqtt | 	/go/src/github.com/edgexfoundry/device-mqtt-go/internal/driver/driver.go:69 +0x8f
edgex-device-mqtt exited with code 2

Ran docker run -d --rm --name emqx -p 18083:18083 -p 1883:1883 emqx/emqx for MQTT broker and device.go for sample device and there's no problems.

Refactor convertReadingValueType method

The convertReadingValueType function is too big so I want to refactor it to another file

panic when MQTT broker is not running

When running device-mqtt without a MQTT broker, a friendly message is printed, but there is still a panic somewhere in the code:

Aug 22 14:14:56 systemd[1]: Started Service for snap application edgex-device-mqtt.device-mqtt.
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: Init: useRegistry: consul://localhost:8500 profile: res confDir: /var/snap/edgex-device-mqtt/x1/config/device-mqtt
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: Check registry is up... http://localhost:8500
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: Register in registry...
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: Calling service.Start.
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: EnableRemote is false, using local log file
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: level=INFO ts=2019-08-22T19:14:56.639659055Z app=edgex-device-mqtt source=init.go:137 msg="Check Metadata service's status ..."
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: level=INFO ts=2019-08-22T19:14:56.639701876Z app=edgex-device-mqtt source=init.go:137 msg="Check Data service's status ..."
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: level=INFO ts=2019-08-22T19:14:56.646675345Z app=edgex-device-mqtt source=init.go:47 msg="Service clients initialize successful."
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: level=DEBUG ts=2019-08-22T19:14:56.649698772Z app=edgex-device-mqtt source=service.go:135 msg="Trying to find Device Service: edgex-device-mqtt"
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: level=INFO ts=2019-08-22T19:14:56.649751462Z app=edgex-device-mqtt source=loader.go:250 msg="listen for config changes from Registry"
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: level=INFO ts=2019-08-22T19:14:56.654158009Z app=edgex-device-mqtt source=service.go:149 msg="Device Service edgex-device-mqtt exists"
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: level=INFO ts=2019-08-22T19:14:56.656532727Z app=edgex-device-mqtt source=loader.go:274 msg="Writeable configuration has been updated. Setting log level to DEBUG"
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: level=DEBUG ts=2019-08-22T19:14:56.658627786Z app=edgex-device-mqtt source=service.go:152 msg="Device Service in Core MetaData: edgex-device-mqtt"
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: level=DEBUG ts=2019-08-22T19:14:56.666558198Z app=edgex-device-mqtt source=profiles.go:39 msg="created absolute path for loading pre-defined Device Profiles: /var/snap/edgex-device-mqtt/current/config/device-mqtt/res"
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: level=DEBUG ts=2019-08-22T19:14:56.672861268Z app=edgex-device-mqtt source=devices.go:22 msg="Loading pre-define Devices from configuration"
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: level=DEBUG ts=2019-08-22T19:14:56.675785868Z app=edgex-device-mqtt source=restrouter.go:21 msg="init status rest controller"
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: level=INFO ts=2019-08-22T19:14:56.675820243Z app=edgex-device-mqtt source=driver.go:279 msg="Create MQTT client and connection: uri=tcp://admin:[email protected]:1883 clientID=CommandResponseSubscriber "
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: level=INFO ts=2019-08-22T19:14:56.675843807Z app=edgex-device-mqtt source=driver.go:279 msg="Create MQTT client and connection: uri=tcp://admin:[email protected]:1883 clientID=IncomingDataSubscriber "
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: level=DEBUG ts=2019-08-22T19:14:56.678805457Z app=edgex-device-mqtt source=restrouter.go:24 msg="init command rest controller"
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: level=DEBUG ts=2019-08-22T19:14:56.681865584Z app=edgex-device-mqtt source=restrouter.go:30 msg="init callback rest controller"
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: panic: start command response Listener failed, please check MQTT broker settings are correct, Network Error : dial tcp 0.0.0.0:1883: connect: connection refused
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: goroutine 75 [running]:
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: github.com/edgexfoundry/device-mqtt-go/internal/driver.(*Driver).Initialize.func1()
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]:         /root/parts/device-mqtt/src/internal/driver/driver.go:73 +0xac
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]: created by github.com/edgexfoundry/device-mqtt-go/internal/driver.(*Driver).Initialize
Aug 22 14:14:56 edgex-device-mqtt.device-mqtt[40511]:         /root/parts/device-mqtt/src/internal/driver/driver.go:70 +0xa2
Aug 22 14:14:56 systemd[1]: snap.edgex-device-mqtt.device-mqtt.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

It would be nice if:

an ERROR level message was logged when this happens
we didn't panic and instead output a message and exit with os.Exit(1)

CommandPublisher doesn't take configured IP

I can see from broker side that the device-mqtt connects to the broker (clients CommandResponseSubscriber & IncomingDataSubscriber) but client CommandPublisher doesn't take the same IP and tries to connect to 127.0.0.1
Yet all of the IP in "configuration.yml" are set to 172.21.0.16 port 1883

Here are the logs from edgex-device-mqtt:

[SC2@localhost device-service-demo]$ docker-compose logs -f device-mqtt
Attaching to edgex-device-mqtt
edgex-device-mqtt | Init: useRegistry: consul://edgex-core-consul:8500 profile:  confDir: /custom-config
edgex-device-mqtt | Check registry is up... http://edgex-core-consul:8500
edgex-device-mqtt | Pushing configuration into Registry...
edgex-device-mqtt | Loading configuration from: /custom-config/configuration.toml
edgex-device-mqtt | 
edgex-device-mqtt | Register in registry...
edgex-device-mqtt | Calling service.Start.
edgex-device-mqtt | EnableRemote is false, using local log file
edgex-device-mqtt | level=INFO ts=2019-07-16T13:22:21.137655686Z app=edgex-device-mqtt source=init.go:137 msg="Check Metadata service's status ..."
edgex-device-mqtt | level=INFO ts=2019-07-16T13:22:21.138389147Z app=edgex-device-mqtt source=init.go:137 msg="Check Data service's status ..."
edgex-device-mqtt | level=INFO ts=2019-07-16T13:22:21.187878699Z app=edgex-device-mqtt source=init.go:47 msg="Service clients initialize successful."
edgex-device-mqtt | level=DEBUG ts=2019-07-16T13:22:21.225361794Z app=edgex-device-mqtt source=service.go:135 msg="Trying to find Device Service: edgex-device-mqtt"
edgex-device-mqtt | level=INFO ts=2019-07-16T13:22:21.225469383Z app=edgex-device-mqtt source=loader.go:237 msg="listen for config changes from Registry"
edgex-device-mqtt | level=INFO ts=2019-07-16T13:22:21.258222925Z app=edgex-device-mqtt source=service.go:149 msg="Device Service edgex-device-mqtt exists"
edgex-device-mqtt | level=INFO ts=2019-07-16T13:22:21.291542336Z app=edgex-device-mqtt source=loader.go:261 msg="Writeable configuration has been updated. Setting log level to DEBUG"
edgex-device-mqtt | level=DEBUG ts=2019-07-16T13:22:21.3015196Z app=edgex-device-mqtt source=service.go:152 msg="Device Service in Core MetaData: edgex-device-mqtt"
edgex-device-mqtt | level=DEBUG ts=2019-07-16T13:22:21.346807636Z app=edgex-device-mqtt source=profiles.go:39 msg="created absolute path for loading pre-defined Device Profiles: /custom-config"
edgex-device-mqtt | level=DEBUG ts=2019-07-16T13:22:21.428577072Z app=edgex-device-mqtt source=devices.go:22 msg="Loading pre-define Devices from configuration"
edgex-device-mqtt | level=DEBUG ts=2019-07-16T13:22:21.450101482Z app=edgex-device-mqtt source=devices.go:25 msg="Device MQ_DEVICE exists, using the existing one"
edgex-device-mqtt | level=INFO ts=2019-07-16T13:22:21.475125766Z app=edgex-device-mqtt source=driver.go:279 msg="Create MQTT client and connection: uri=tcp://:@172.21.0.16:1883 clientID=CommandResponseSubscriber "
edgex-device-mqtt | level=DEBUG ts=2019-07-16T13:22:21.47511988Z app=edgex-device-mqtt source=restrouter.go:21 msg="init status rest controller"
edgex-device-mqtt | level=INFO ts=2019-07-16T13:22:21.475113327Z app=edgex-device-mqtt source=driver.go:279 msg="Create MQTT client and connection: uri=tcp://:@172.21.0.16:1883 clientID=IncomingDataSubscriber "
edgex-device-mqtt | level=DEBUG ts=2019-07-16T13:22:21.541127479Z app=edgex-device-mqtt source=restrouter.go:24 msg="init command rest controller"
edgex-device-mqtt | level=INFO ts=2019-07-16T13:22:21.564558111Z app=edgex-device-mqtt source=responselistener.go:52 msg="[Response listener] Start command response listening. "
edgex-device-mqtt | level=INFO ts=2019-07-16T13:22:21.575396283Z app=edgex-device-mqtt source=incominglistener.go:54 msg="[Incoming listener] Start incoming data listening. "
edgex-device-mqtt | level=DEBUG ts=2019-07-16T13:22:21.599122341Z app=edgex-device-mqtt source=restrouter.go:30 msg="init callback rest controller"
edgex-device-mqtt | level=DEBUG ts=2019-07-16T13:22:21.704014923Z app=edgex-device-mqtt source=restrouter.go:33 msg="init other rest controller"
edgex-device-mqtt | level=DEBUG ts=2019-07-16T13:22:21.740087577Z app=edgex-device-mqtt source=restrouter.go:37 msg="init the metrics and config rest controller each"
edgex-device-mqtt | level=INFO ts=2019-07-16T13:22:21.764210602Z app=edgex-device-mqtt source=service.go:120 msg="*Service Start() called, name=edgex-device-mqtt, version=1.0.0"
edgex-device-mqtt | level=INFO ts=2019-07-16T13:22:21.788728996Z app=edgex-device-mqtt source=service.go:126 msg="Listening on port: 49982"
edgex-device-mqtt | level=INFO ts=2019-07-16T13:22:21.813454278Z app=edgex-device-mqtt source=service.go:127 msg="Service started in: 904.652008ms"
edgex-device-mqtt | level=DEBUG ts=2019-07-16T13:22:21.838196853Z app=edgex-device-mqtt source=service.go:129 msg="*Service Start() exit"
edgex-device-mqtt | level=DEBUG ts=2019-07-16T13:22:51.788560871Z app=edgex-device-mqtt source=executor.go:42 msg="AutoEvent - executing {\"frequency\":\"30s\",\"resource\":\"testrandnum\"}"
edgex-device-mqtt | level=DEBUG ts=2019-07-16T13:22:51.847008027Z app=edgex-device-mqtt source=command.go:200 msg="Handler - execReadCmd: deviceResource: randnum"
edgex-device-mqtt | level=DEBUG ts=2019-07-16T13:22:51.899689015Z app=edgex-device-mqtt source=command.go:207 msg="Handler - execReadCmd: deviceResource: {\"description\":\"device random number\",\"name\":\"randnum\",\"properties\":{\"value\":{\"type\":\"Float64\",\"readWrite\":\"R\",\"size\":\"4\",\"floatEncoding\":\"eNotation\"},\"units\":{\"type\":\"String\",\"readWrite\":\"R\"}}}"
edgex-device-mqtt | level=INFO ts=2019-07-16T13:22:51.930512101Z app=edgex-device-mqtt source=driver.go:279 msg="Create MQTT client and connection: uri=tcp://:@127.0.0.1:1883 clientID=CommandPublisher "
edgex-device-mqtt | level=ERROR ts=2019-07-16T13:22:51.937830592Z app=edgex-device-mqtt source=executor.go:45 msg="AutoEvent - error occurs when reading resource testrandnum"

Update to meet CommandValue and CommandRequest refactoring

Modify the related code to meet this refactoring: edgexfoundry/device-sdk-go#262

Refactor checkValueInRange method

Refactor convertReadingValueType function to another file

Readings should be timestamped in nanoseconds

According to the architecture decision in face to face meeting, we would use nanoseconds in Events and Readings to get more accuracy.
The origin field of reading is input by the specific Device Service protocol driver.

Errors in the configuration.toml file causing startup issues

In multiple places in the configuration.toml file, instead of the relevant service URL such as edgex-core-consul, "localhost" is specified. This causes the service to crash on startup.
This issue is present in the file inside the /cmd/res and /cmd/res/example folders. The file in the /cmd/res/docker folder is configured correctly.

Optimize error message when subscribe fail

The user will fail to startup MQTT device because MQTT broker does not set up correctly.
So MQTT device should throw an appropriate message.

Logging import package name change

For full context regarding this interlock, please see here edgexfoundry/go-mod-core-contracts#46

Wait for edgexfoundry/device-sdk-go#219 merged and then change the import path logging to logger

Command testrandnum not created

Hello,

I am triying to launch the command in order to find the executable command. However, i am stuck because of this error:

MQTT Docker log:
edgex-device-mqtt | level=INFO ts=2019-10-02T11:46:06.945546211Z app=edgex-device-mqtt source=driver.go:279 msg="Create MQTT client and connection: uri=tcp://:@172.19.0.1:1883 clientID=CommandPublisher "
edgex-device-mqtt | level=INFO ts=2019-10-02T11:46:06.947118137Z app=edgex-device-mqtt source=driver.go:154 msg="Publish command: {"cmd":"randnum","method":"get","uuid":"5d948dfeb8dd790001bf0c4e"}"
edgex-device-mqtt | level=INFO ts=2019-10-02T11:46:11.948379424Z app=edgex-device-mqtt source=driver.go:122 msg="Handle read commands failed: can not fetch command response: method=get cmd=randnum"
edgex-device-mqtt | level=ERROR ts=2019-10-02T11:46:11.948980258Z app=edgex-device-mqtt source=executor.go:45 msg="AutoEvent - error occurs when reading resource testrandnum"

Thank you

Command endpoints can return status=500; which doesn't exist in the device profile, nor is indicated by Core Metadata

When unexpected errors occurs, this device service will return the HTTP status code 500.

This isn't indicated in the default device profile for this device service, nor does it automatically get included in the list of possible return values returned by Core Metadata from the device endpoint:

$ curl -s http://localhost:48081/api/v1/device/name/MQTT%20test%20device

This can be reproduced by starting this device service without a corresponding MQTT peer application, and then issuing a command either directly to the device service or via Core Command:

$ curl -w "%{http_code}" http://localhost:49982/api/v1/device/21a5e108-ef33-4d48-b931-81afa739f092/testrandfloat32
Handler - execReadCmd: error for Device: MQTT test device cmd: testrandfloat32, can not fetch command response: method=get cmd=randfloat32 /api/v1/device/21a5e108-ef33-4d48-b931-81afa739f092/testrandfloat32
500

I reproduced this with the following snaps:

Here are the versions of the snaps I'm using:

edgex-device-mqtt 1.0.0-20190801+fd7235a
edgexfoundry 1.0.1-20190729+c28b77d1

Add eNotation and Base64 to device profile example for float encoding

This change can refer to edgexfoundry/device-sdk-go#265

Update the configuration in the res/docker folder

Remove configuration-driver.toml
[configurationdriver.toml] Update EnableRemote=true
[configurationdriver.toml] Add driver config

Driver should checks value type appropriate when convert reading data

Whenever device-MQTT receives an incoming async reading or executes the read command, it should check value type that is matched the device-profile defined.

Convert Properties.Value.Type to lower case at switch case to make it more flexible.

https://github.com/edgexfoundry/device-mqtt-go/blob/master/internal/driver/driver.go#L335

switch strings.ToLower(deviceObject.Properties.Value.Type) {
case "bool"

Check actual data type before convert reading data

https://github.com/edgexfoundry/device-mqtt-go/blob/master/internal/driver/driver.go#L299

Example 1 - Input is "123" and device-profile defines value is int16, then the driver should parse the string type to int64 type and check the reading in the int16's value range. If reading in the range and then cast int64 to int16.

Example 2 - Input is 123 and device-profile defines value is int8, then the driver should parse it to int64 and check the reading in the int8's value range. If reading in the range and then cast int64 to int8.

... etc.

Driver should check the data format of Payload

device-mqtt-go/internal/driver/incominglistener.go

Line 59 in e415b6d

deviceName := response["name"].(string)

func onIncomingDataReceived(client mqtt.Client, message mqtt.Message) {
	var response map[string]interface{}
	json.Unmarshal(message.Payload(), &response)

	// add check the data format by Geoffrey
	if response == nil || response["name"] == nil || response["cmd"] == nil {
		driver.Logger.Warn(fmt.Sprintf("[Incoming listener] Incoming reading ignored. Data Format Error, couldn't be converted to Map[String]interface{}  : topic=%v payload=%v", message.Topic(), string(message.Payload())))
		return
	}

	deviceName := response["name"].(string)
	cmd := response["cmd"].(string)
	reading := response[cmd]

Action： add check the data format of Payload by Geoffrey

Reason：
If the Payload is "testmest" or {"run":"100"}···, it will make the device-mqtt-goservice exit by the panicpanic: interface conversion: interface {} is nil, not string. Because these non-standard formats cannot be converted to map[string]interface{}. Even though the Payload could be converted to map[string]interface{}, it must hava map["name"] and map["cmd"]. If not, the the device-mqtt-goexit by the panicpanic: interface conversion: interface {} is nil, not string. As a DeviceService, the wrong data type can be left unprocessed, but should not cause the service to be unavailable. Therefore, we should verify the data format of Payload to prevent the device-mqtt-gofrom exiting abnormally due to data format problems. In addition, the availability of thedevice-mqtt-go` is guaranteed. Maybe the way I deal with is not elegant enough, but I hope to attract attention to this kind of problem.

By Geoffrey

Load version number from VERSION instead of hardcode

version.go should load version number from VERSION

Dockerfile should include running with Registry

All EdgeX services running in Docker are expected to run with the Registry. This has been removed for the MQTT Device service and all other device services. It should be add back as --registry=consul://edgex-core-consul:8500

Optimize error message when device service fails to connection MQTT broker

Optimize error message when device service fails to connection MQTT broker.

edgex-device-mqtt | panic: close of nil channel
edgex-device-mqtt | 
edgex-device-mqtt | goroutine 89 [running]:
edgex-device-mqtt | github.com/edgexfoundry/device-mqtt-go/vendor/github.com/eclipse/paho%2emqtt%2egolang.(*client).closeStop(0xc0003b0000)
edgex-device-mqtt | 	/go/src/github.com/edgexfoundry/device-mqtt-go/vendor/github.com/eclipse/paho.mqtt.golang/client.go:482 +0x12f
edgex-device-mqtt | github.com/edgexfoundry/device-mqtt-go/vendor/github.com/eclipse/paho%2emqtt%2egolang.(*client).disconnect(0xc0003b0000)
edgex-device-mqtt | 	/go/src/github.com/edgexfoundry/device-mqtt-go/vendor/github.com/eclipse/paho.mqtt.golang/client.go:495 +0x2f
edgex-device-mqtt | github.com/edgexfoundry/device-mqtt-go/vendor/github.com/eclipse/paho%2emqtt%2egolang.(*client).Disconnect(0xc0003b0000, 0x1388)
edgex-device-mqtt | 	/go/src/github.com/edgexfoundry/device-mqtt-go/vendor/github.com/eclipse/paho.mqtt.golang/client.go:439 +0x1f9
edgex-device-mqtt | github.com/edgexfoundry/device-mqtt-go/internal/driver.startCommandResponseListening(0x89ec40, 0xc000381210)
edgex-device-mqtt | 	/go/src/github.com/edgexfoundry/device-mqtt-go/internal/driver/responselistener.go:38 +0x54f
edgex-device-mqtt | github.com/edgexfoundry/device-mqtt-go/internal/driver.(*Driver).Initialize.func1()
edgex-device-mqtt | 	/go/src/github.com/edgexfoundry/device-mqtt-go/internal/driver/driver.go:70 +0x26
edgex-device-mqtt | created by github.com/edgexfoundry/device-mqtt-go/internal/driver.(*Driver).Initialize
edgex-device-mqtt | 	/go/src/github.com/edgexfoundry/device-mqtt-go/internal/driver/driver.go:69 +0x8f
edgex-device-mqtt exited with code 2

see #27 (comment).

receipt of non-float number data will cause a panic in newResult function

The newResult function of driver.go will always fail if the reading is something other than a string or float64. This is because the JSON unmarshaling will always interpret any number as a float64 (see https://golang.org/pkg/encoding/json/#Unmarshal). Therefore, the type of the reading in the newResult function for any number (whether int, uint, etc.) will always be a float64 and cause a panic in the checks. Best I could do was just assume that the profileValueType for the data was going to be correct and cast to the appropriate type.

func newResult(deviceObject models.DeviceObject, ro models.ResourceOperation, reading interface{}) (*sdkModel.CommandValue, error) {
var result = &sdkModel.CommandValue{}
var err error
var resTime = time.Now().UnixNano() / int64(time.Millisecond)
var profileValueType = strings.ToLower(deviceObject.Properties.Value.Type)

switch profileValueType {
case "bool":
	result, err = sdkModel.NewBoolValue(&ro, resTime, reading.(bool))
case "string":
	result = sdkModel.NewStringValue(&ro, resTime, reading.(string))
case "uint8":
	result, err = sdkModel.NewUint8Value(&ro, resTime, uint8(reading.(float64)))
case "uint16":
	result, err = sdkModel.NewUint16Value(&ro, resTime, uint16(reading.(float64)))
case "uint32":
	result, err = sdkModel.NewUint32Value(&ro, resTime, uint32(reading.(float64)))
case "uint64":
	result, err = sdkModel.NewUint64Value(&ro, resTime, uint64(reading.(float64)))
case "int8":
	result, err = sdkModel.NewInt8Value(&ro, resTime, int8(reading.(float64)))
case "int16":
	result, err = sdkModel.NewInt16Value(&ro, resTime, int16(reading.(float64)))
case "int32":
	result, err = sdkModel.NewInt32Value(&ro, resTime, int32(reading.(float64)))
case "int64":
	result, err = sdkModel.NewInt64Value(&ro, resTime, int64(reading.(float64)))
case "float32":
	result, err = sdkModel.NewFloat32Value(&ro, resTime, reading.(float32))
case "float64":
	result, err = sdkModel.NewFloat64Value(&ro, resTime, reading.(float64))
default:
	err = fmt.Errorf("return result fail, none supported value type: %v", deviceObject.Properties.Value.Type)
}

return result, err

}

Ideally, the snap should be setup such that after installing the snap, you can simply install a MQTT broker like mosquitto and be able to test it out locally by turning on the service.