Code Monkey home page Code Monkey logo

mqtt's Introduction

MQTT mqtt - MQTT for kdb+

GitHub release (latest by date) Travis (.org) branch

This interface lets you communicate with an MQTT broker from a kdb+ session. The interface follows closely the PAHO MQTT C API. Exposed functionality includes

  • connect to a MQTT broker
  • subscribe to topics
  • publish to a broker

This is part of the Fusion for kdb+ interface collection.

New to kdb+ ?

Kdb+ is the world’s fastest timeseries database, optimized for ingesting, analyzing and storing massive amounts of structured data. To get started with kdb+, please visit https://code.kx.com/q/learn/ for downloads and developer information. For general information, visit https://kx.com/

New to MQTT ?

Message Queueing Telemetry Transport (MQTT) is a machine-to-machine/IOT connectivity protocol. It is designed to provide a lightweight publish/subscribe messaging protocol.

MQTT is used commonly for constrained devices with low-bandwidth, high-latency or unreliable networks.

Use cases

This messaging protocol is used extensively where a small software footprint is required, e.g.

  • communication with edge devices/sensors
  • home automation
  • mobile applications

🌐 MQTT: Real-world applications

API Documentation

πŸ‘‰ API reference

Installation Documentation

πŸ‘‰ Install guide

Building From Source

πŸ‘‰ Building guide

Quick start

Requirements

To run the examples you will need a MQTT broker e.g.

Start Mosquitto

Start Mosquitto on default port localhost:1883.

sudo systemctl start mosquitto

Launch consumer

We assume you are in the source directory of the MQTT-kdb+ interface.

mqtt]$ cd q
q]$ q ../examples/consumer.q 

Launch producer

In the same directory as the one launching a consumer, run:

q]$ q ../examples/producer.q

Then pubish a message by a timer, say every second:

q)\t 1000

Going back to the console of consumer, now you can see received messages in .mqtt.tab:

q).mqtt.tab
topic  msg_sent                      msg_recv                      received_m..
-----------------------------------------------------------------------------..
topic1 2021.01.06D13:24:46.267356000 2021.01.06D13:24:46.267718000 topic1_3  ..
topic2 2021.01.06D13:24:46.267450000 2021.01.06D13:24:46.368412000 topic2_3  ..
topic1 2021.01.06D13:24:47.267366000 2021.01.06D13:24:47.268676000 topic1_4  ..
topic2 2021.01.06D13:24:47.267624000 2021.01.06D13:24:47.372268000 topic2_4  ..
topic1 2021.01.06D13:24:48.276623000 2021.01.06D13:24:48.277677000 topic1_5  ..
topic2 2021.01.06D13:24:48.276801000 2021.01.06D13:24:48.377926000 topic2_5  ..
topic1 2021.01.06D13:24:49.267324000 2021.01.06D13:24:49.268228000 topic1_6  ..
topic2 2021.01.06D13:24:49.267482000 2021.01.06D13:24:49.368805000 topic2_6  ..
topic1 2021.01.06D13:24:50.267684000 2021.01.06D13:24:50.268125000 topic1_7  ..
topic2 2021.01.06D13:24:50.267804000 2021.01.06D13:24:50.368889000 topic2_7  ..

Status

The interface is currently available under an Apache 2.0 license and is supported on a best-efforts basis by the KX Fusion team. The interface is currently in active development, with additional functionality released on an ongoing basis.

πŸ‘‰ Issues and feature requests

πŸ‘‰ Guide to contributing

mqtt's People

Contributors

awilson-kx avatar cmccarthy1 avatar mshimizu-kx avatar nmcdonnell-kx avatar paulgrainger85 avatar rianoc avatar sshanks-kx avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

mqtt's Issues

Windows Integration issue

While the code has been written with Windows integration in mind and the build files work to create an appropriate .dll attempts to send and receive messages had been resulting in a semaphore error.

This needs to be looked into before any support for windows can be considered

Releases missing files - ERROR: Directory 'lib' does not exist

Issue with all Linux/OSX/Windows releases, only the .so/.dll is included

./install.sh
Detected System
* OS: linux-gnu
* TYPE: x86_64
* MACHINE TYPE: x86_64-pc-linux-gnu
ERROR: Directory 'lib' does not exist. Please run from release package
$ wget -O mqtt.tar.gz https://github.com/KxSystems/mqtt/releases/download/1.1.0/mqttkdb-linux-1.1.0.tgz
$ tar -tvf mqtt.tar.gz
drwxrwxr-x travis/travis     0 2022-01-07 11:54 ./
drwxrwxr-x travis/travis     0 2022-01-07 11:54 ./lib/
-rwxrwxr-x travis/travis 22688 2022-01-07 11:54 ./lib/mqttkdb.so

This functions as a work around:

wget -O mqtt.tar.gz https://github.com/KxSystems/mqtt/archive/1.1.0.tar.gz
tar -xzf mqtt.tar.gz
cd mqtt-1.1.0
wget -O lib.tar.gz https://github.com/KxSystems/mqtt/releases/download/1.1.0/mqttkdb-linux-1.1.0.tgz
tar -xzf lib.tar.gz
./install.sh

Docs should be included with release

New docs folder wont currently be added to a release build - should be included in the future while docs are there.
.travis.yml example area
e.g.
elif [[ $TRAVIS_OS_NAME == "windows" ]]; then
7z a -tzip $FILE_NAME README.md install.bat LICENSE q examples;
elif [[ $TRAVIS_OS_NAME == "linux" || $TRAVIS_OS_NAME == "osx" ]]; then
tar -zcvf $FILE_NAME README.md install.sh LICENSE q examples;

producer.q log printing sending msgs after instructing it to stop

Running q producer.q, I type '\t 100'.
It prints sending a msg every few ms.
After a while, I type '\t 0' to stop. When I do that, it seems to suddenly prints a big batch of msgs - but using a client at the same time those batch of msgs are never received. Not sure if there were actually sent, since they seem to be printed after doing the cmd to stop the sends.

Instruction header improvement

Looks like a header with size and either 'a','b','c' sent across the connection in order to transmit actions.
Could be a struct with an enum instead
e.g. mqttaction {long size;action_type action)
were action_type is a new enum.
Will make the code easier to read/understand
NOTE: above code is just an example - don't need to use same variable names/etc

mqtt with raspbery pi - install issues

Report from user:

I have the interface working on a Raspberry Pi but had some issues following the instructions and using the supplied install.sh
Default kdb+ build from kx comes named q/l32arm . I needed to rename to q/l32
Otherwise error will be seen:
q)\l mqtt.q
'/home/pi/q/l32/mqtt.so: cannot open shared object file: No such file or directory
[2] /home/pi/q/mqtt.q:2: connX :mqtt 2:(connX;3);
^
[0] ()
Install.sh fails
Detected System

  • OS: linux-gnueabihf
  • TYPE: arm
  • MACHINE TYPE: arm-unknown-linux-gnueabihf
    ERROR: OSTYPE linux-gnueabihf not currently supported by this script
    Please view README.md for installation instructions
    Issue 2 can be bypassed with: OSTYPE="linux-gnu" ./install.sh but that shows another error:
    ERROR: Directory 'lib' does not exist. Please run from release package
    The .so is located in the cmake folder

Instead of using install.sh I just did copies manually:
cp q/mqtt.q $QHOME/
cp cmake/mqtt.so $QHOME/l32/

Connection options - allowing user to configured various connection options

Ref:
https://www.eclipse.org/paho/files/mqttdoc/MQTTClient/html/_m_q_t_t_client_8h.html#aaa8ae61cd65c9dc0846df10122d7bd4e
https://www.eclipse.org/paho/files/mqttdoc/MQTTClient/html/struct_m_q_t_t_client__connect_options.html#a9f1cdffc99659fd4e2d20e6de3c64df0

Currently theres a dict for username/password.
Could add other options and document them, that correlate to the options on the above links.

NOTE: different options can have different datatypes

Connection - support for username/password

May want to support other options.
Ref:
https://www.eclipse.org/paho/files/mqttdoc/MQTTClient/html/_m_q_t_t_client_8h.html#aaa8ae61cd65c9dc0846df10122d7bd4e
https://www.eclipse.org/paho/files/mqttdoc/MQTTClient/html/struct_m_q_t_t_client__connect_options.html#a9f1cdffc99659fd4e2d20e6de3c64df0

When I tried to connect to the solace mqtt broker, it gave me an error 4.
Temp adding in the following to the 'conn' function allowed it to work (replacing the username/password with my actual user/pass)

conn_opts.keepAliveInterval = 20;
conn_opts.cleansession = 1;
conn_opts.username = " my username ";
conn_opts.password = " my password ";

Could potentially pass in a dict to the function to allow the end user to provide a range of supported options & any addition ones in the future.

Example - expose topics

For the producer.q - may be handy to print topic being used or provide a topic
Quickly ran the example with the solace subscriber & took a min to figure out the topic I should be subscribing to, in order to see the data.

Example running, just for show:
q producer.q
(msgsent;1) (msgsent;2)
(`msgsent;3)

subscribing to topic2

Message received

payload | "2020.05.07D11:51:58.142589000;topic2_0"
dest | `topic2
isRedeliv| 0b
isDiscard| 0b
isRequest| 0b
sendTime | 2000.01.01D00:00:00.000000000

Message received

payload | "2020.05.07D11:51:58.344448000;topic2_1"
dest | `topic2
isRedeliv| 0b
isDiscard| 0b
isRequest| 0b
sendTime | 2000.01.01D00:00:00.000000000

Travis configuration file warnings

Build config validation

deploy: deprecated key skip_cleanup (not supported in dpl v2, use cleanup)
root: missing dist, using the default xenial
root: missing os, using the default linux
deploy: key api_key is an alias for token, using token
root: key matrix is an alias for jobs, using jobs

As per KxSystems/solace#156

Unable to build MQTT on Redhat7.6

When trying to build on redhat7.6 it doesn't seem possible (easy) to install cmake3

Clean install

sudo yum install libGL libGLU libX11 vim netstat bind-utils tcpdump lsof telnet
sudo yum install net-tools wget libpng fontconfig zip unzip bzip2 rsync unixODBC
sudo yum install  java-11-openjdk
sudo yum install compat-openssl10
sudo yum install libXcomposite libXcursor libXi libXtst libXrandr alsa-lib mesa-libEGL libXdamage mesa-libGL libXScrnSaver

-- installed Anaconda
./Anaconda3-2022.05-Linux-x86_64.sh

sudo yum install git gcc g++ cmake make vim wget dos2unix autoconf automake libtool

mkdir github
cd github/

sudo yum install git gcc g++ cmake make vim wget dos2unix autoconf automake libtool
sudo yum -y install openssl-devel

git clone https://github.com/KxSystems/mqtt
git clone https://github.com/eclipse/paho.mqtt.c.git
cd paho.mqtt.c/
make
sudo make install
cd ../mqtt/

mkdir build && cd build
cmake ..
(base) [kxlm@kxlmukdev01 build]$ cmake ..
CMake Error at CMakeLists.txt:4 (cmake_minimum_required):
  CMake 3.1 or higher is required.  You are running version 2.8.12.2


-- Configuring incomplete, errors occurred!
(base) [kxlm@kxlmukdev01 build]$ cmake --version
cmake version 2.8.12.2
(base) [kxlm@kxlmukdev01 build]$ sudo yum -y install cmake3
Failed to set locale, defaulting to C
Loaded plugins: langpacks, product-id, search-disabled-repos
Resolving Dependencies
--> Running transaction check
---> Package cmake3.x86_64 0:3.17.5-1.el7 will be installed
--> Processing Dependency: cmake3-data = 3.17.5-1.el7 for package: cmake3-3.17.5-1.el7.x86_64
--> Processing Dependency: librhash.so.0()(64bit) for package: cmake3-3.17.5-1.el7.x86_64
--> Processing Dependency: libuv.so.1()(64bit) for package: cmake3-3.17.5-1.el7.x86_64
--> Processing Dependency: libzstd.so.1()(64bit) for package: cmake3-3.17.5-1.el7.x86_64
--> Running transaction check
---> Package cmake3-data.noarch 0:3.17.5-1.el7 will be installed
--> Processing Dependency: /usr/bin/python3 for package: cmake3-data-3.17.5-1.el7.noarch
---> Package libuv.x86_64 1:1.44.2-1.el7 will be installed
---> Package libzstd.x86_64 0:1.5.2-1.el7 will be installed
---> Package rhash.x86_64 0:1.3.4-2.el7 will be installed
--> Processing Dependency: /usr/bin/python3 for package: cmake3-data-3.17.5-1.el7.noarch
--> Finished Dependency Resolution
Error: Package: cmake3-data-3.17.5-1.el7.noarch (epel)
           Requires: /usr/bin/python3
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

Tried various tests/installs to get python installed but still can't yum install cmake3

cmake --version
  161  sudo yum -y install python3 cmake3 ## No lnonger in 7.6
  162  sudo yum -y install rh-python36 cmake3
  163  sudo yum -y install rh-python36
  164  sudo yum -y install cmake3 ## Still fails
  165  scl enable rh-python36 bash
python3 -V
which python  ## installed in /opt/rh/rh-python36/root/usr/bin/python3

Build using more libs than required

Build using libpaho-mqtt3a, paho-mqtt3as, paho-mqtt3c, paho-mqtt3cs - shouldnt need all these.

Ref: https://github.com/eclipse/paho.mqtt.c#libraries

Running without any SSL lib installed (nor wanting to use SSL)...

q ./mqtt/examples/consumer.q
'libssl.so.1.0.0: cannot open shared object file: No such file or directory
  [5]  /mqttq/q/mqtt.q:3: connX:`mqttkdb 2:(`connX;3);

Relates to #44 mentioning use of libs

Add LICENSE file

From main org screen, noticed it wasn't showing up with a license (e.g. apache2)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.