Kakfa and MQTT Sample Application > Setup Documents

Using Multiple Servers

Now that the basic scenario is working, you can expand the scenario to include multiple MQTT servers. For example, these MQTT servers may collect data in different buildings or factories while Kafka and GridDB run on servers located in a data center or they may be in the same data center as the Kafka and GridDB instances and are exposed to the outside sensors via a DNS round robin or other load balancing mechanism.
Following the instructions from the simple scenario, you can learn how to set up Kafka, Mosquitto, Kafka MQTT Source Connector with a few exceptions:

Kafka

Do not start the Kafka and Zookeeper servers as they will cause a conflict and the MQTT Source Connector will not deliver messages to the centralized Kafka Server. The libraries and other components are still required for the MQTT Source Connector to run.

On the centralized Kafka server, make sure port 9092 is accessible from the remote MQTT instances completely disable the firewall on the centralized server.

Mqtt Source Connector

You will need to edit /opt/kafka/config/connect-standalone.properties, changing bootstrap.servers to the IP address of your Kafka Server.

bootstrap.servers=localhost:9092

To:

bootstrap.servers=KAFKA_IP_OR_HOSTNAME:9092

Verify MqttClient-to-GridDB-Sink-Connection

Now you can run the MqttClient program on all of the remote MQTT instances and be able to view the logged messages either with the kafka-console-consumer or GridDB Sink Connector.

Source Code

Complete source code used for the GridDB Kafka and MQTT Application can be downloaded from the following:

Download: griddb-kafka-mqtt-application.tar.gz