Project Overview
This sample application provides an introduction on how to use the GridDB Kafka Connector. The main idea is to connect a GridDB database to a Kafka cluster so that messages or records received from a Kafka cluster can be received, processed, and eventually stored into GridDB. This project also uses Mosquitto (MQTT) -- another message broker widely used in IoT applications -- to receive and send messages. In this application, the MQTT messages represent sensor data being sent to the Kafka cluster to be stored in GridDB. The purpose and advantage of using these messaging brokers is that they allow different programs and processes to communicate and send data to each other.
For this application, an MQTT client program generates random data which can be used to represent sensor readings in a real-world Internet of Things (IoT) system; some examples are volt readings from an electricity sensor or light readings from a light sensor. The MQTT library is used to publish this data to a machine's local MQTT server and from there a MQTT-Kafka connector passes that data to the Kafka cluster where a GridDBSinkConnector and GridDBSinkTask will read and parse that record into a GridDB row to be inserted into a container. A client app that uses the MQTT API will publish APIs that will be received by the MQTT server, which then publishes it to listening MQTT subscribers. The Kafka Source connector is a MQTT subscriber, and publishes the message to a Kafka server. Kafka then publishes the message to its subscribers, one of which is the GridDB Sink connector which then inserts the message into GridDB. Finally, the DataViewer reads the message from GridDB. This project will also show you how to configure the MQTT-Kafka connector to allow for multiple MQTT clients from multiple machines to send data to a single, remote GridDB database through Kafka.
A diagram of the communication flow for an MQTT client on a single machine can be viewed in the image below.
Once data is successfully being inserted into GridDB, you can use the Data Viewer program to view statistics of all the sensors that are reporting to GridDB. You can view averages, standard deviations, and other aggregations from individual sensors and average readings of certain sensor types in general. In this application, there are 3 sensor types.
- Light Sensor: Records and measures Light and Sound Data
- Watt/Power Sensor: Records and measures Power and Heat Data
- Electricity Sensors: Records and measures Voltage and Amperage Data
To setup this application you will need to have the following components installed and configured on your system:
- GridDB
- MQTT server and development library
- Kafka messaging broker and server
- Some additional Java libraries in .JAR files
Instructions on how to install and set up these components can be found in the Setup Prerequisites section.
Once everything is installed you can follow the instructions in the setting up multiple MQTT servers section to learn how to configure your system(s) to allow for multiple, remote Mosquito clients.
When you are ready to test the applications with one or more MQTT clients, you can follow the instructions in the Build and Run Components section. It will teach you how to fully build and run the application and see the power and utility offered of using GridDB with messaging brokers.
Project Components
In order to leverage GridDB, Kafka, and MQTT, three more components had to be developed. For further details about each of the components, please review the Kafka and MQTT Sample Application Source Code Overview section.
GridDB Connector
We started with a Kafka-Connector-Skeleton but had to update the API for use with Kafka 0.10.2.1 by comparing the FileStream Connector within Kafka’s source code. More details on this component can be found in the Simple GridDB SinkConnector section.
GridDB Data Viewer
The GridDB Data Viewer outputs the data recorded by the GridDB Kafka Sink. The location of the source code of these components can be found in the DataViewer folder of the source code. More details on how this component works for a single sensor-type can be found in the Simple DataViewer section. Details on how this components works for multiple sensor-types can be found in the Data Viewer section.
Mqtt Client
The MQTT Client sends a JSON formatted message to the MQTT Server. It can generate data that represents readings from one or more different types of sensors. The location of the source code for these components can be found in the mqttclient folder of the source code. Details on how the MQTT client works and is setup for a single sensor-type can be found in the Simple Mqtt section. Details on the more general MQTT client that can generate data of multiple sensor-types can be found in the MqttClient section.
Source Code
Complete source code used for the GridDB Kafka and MQTT Application can be downloaded from the following:
Download: griddb-kafka-mqtt-application.tar.gz