Using Fluentd to Push Data to GridDB

Fluentd is an open source data collector that allows you to easily ingest data into GridDB;
this data is most typically information generated by edge devices or other sources not in your
local network. Fluentd supports multiple input sources including HTTP, MQTT and more.
Fluentd also has many different output options, including GridDB.

The GridDB output plugin relies on the GridDB WebAPI, this post will cover the installation, configuration, and usage of Fluentd, GridDB WebAPI, and the GridDB Plugin for Fluentd.

GridDB and WebAPI Setup

If you don’t have GridDB running already, the quick start is here. To install Fluentd, follow
the guides for CentOS or Debian/Ubuntu. While following these guides, you’ll also install ruby and gem which will be required to build the GridDB plugin for Fluentd. To install the WebAPI, download and untar the release and then copy the webapi folder to $GS_HOME

$ sudo cp -a webapi /var/lib/gridstore
$ sudo chown gsadm.gridstore /var/lib/gridstore/webapi

Then edit conf/repository.json to reflect your GridDB cluster configuration. You will likely
only need to set the cluster name.

{
"clusters" : [
{
"name" : "defaultCluster",

"mode" : "MULTICAST",
"address" : "239.0.0.1",
"port" : 31999,
"jdbcAddress" : "239.0.0.1",
"jdbcPort" : 41999,
"transactionMember": "",
"sqlMember": "",
"providerUrl": null
}
]
}

Now to run WebAPI:

$ sudo su - gsadm
export CLASSPATH=:/usr/share/java/gridstore.jar
cd webapi/
java -jar lib/griddb-webapi-ce-2.1.0.jar 2>&1 /dev/null &

Check $GS_HOME/logs/gs_webapi*.log for errors, if it has successfully started, the logs should
contain:

[main] org.springframework.boot.web.embedded.tomcat.TomcatWebServer:
Tomcat started on port(s): 8080 (http) with context path ''
[main] org.springframework.boot.StartupInfoLogger: Started Application in
2.677 seconds (JVM running for 3.278)

Now that the GridDB WebAPI is installed and running, it’s time to install the GridDB Fluentd
connector. After downloading and un-tarring the release, run:

$ cd fluent-plugin-griddb-1.0.0/
/opt/td-agent/embedded/bin/gem build fluent-plugin-griddb.gemspec
/opt/td-agent/embedded/bin/gem install --force --local fluent-plugin-griddb-1.0.0.gem

Use Fluentd GridDB Plugin

The simplest GridDB configuration follows, this chunk would be added to /etc/td-agent/td-agent.conf and then the td-agent will need to be restarted.


@type griddb
host http://localhost:8080/
cluster defaultCluster
database public
container container_1
username admin
password admin

NOTE: If you are getting “failed connect to localhost:8888” errors, try running:
$ sudo systemctl start td-agent.service

Before Fluentd can write data to GridDB, you need to create the container using the GridDB
WebAPI:

$ curl -X POST --basic -u admin:admin -H "Content-type:application/json" -d '{"container_name":"container_1", "container_type":"TIME_SERIES", "rowkey":true, "columns":[{"name": "date", "type": "TIMESTAMP" }, {"name": "value", "type": "DOUBLE" },{"name": "str", "type": "STRING" }]}' http://localhost:8080/griddb/v2/defaultCluster/dbs/public/containers

With the GridDB plugin in place, you can POST data to Fluentd:

$ curl -X POST -d 'json={"date":"2018-09-20T12:08:21.112Z",
"value":"4.23", "str":"Hello World"}' http://localhost:8888/griddb

And using the GridDB WebAPI, we can see the data we POSTed to FluentD:

$ curl -X POST --basic -u admin:admin -H "Content-type:application/json" -d '{"limit":1000}' http://localhost:8080/griddb/v2/defaultCluster/dbs/public/containers/container_1/rows

This is just the beginning of what you can do with Fluentd and its GridDB plugin. For more
information, please check out the detailed documentation.