Fluentd is an open source data collector that allows you to easily ingest data into GridDB; this data is most information generated by edge devices or other sources not in your local network. Fluentd supports multiple including HTTP, MQTT and more. Fluentd also has many different output options, including GridDB.

The GridDB output plugin relies on the GridDB WebAPI, this post will cover the installation, configuration, and usage of Fluentd, GridDB WebAPI, and the GridDB Plugin for Fluentd.

GridDB and WebAPI Setup

If you don’t have GridDB running already, the quick start is here. To install Fluentd, follow the guides for CentOS or Debian/Ubuntu. While following these guides, you’ll also install ruby and gem which will be required to build the GridDB plugin for Fluentd. To install the WebAPI, download and untar the release and then copy the webapi folder to $GS_HOME

$ sudo cp -a webapi /var/lib/gridstore

$ sudo chown gsadm.gridstore /var/lib/gridstore/webapi

Then edit conf/repository.json to reflect your GridDB cluster configuration. You will likely only need to set the cluster name.

{
"clusters" : [
{
"name" : "defaultCluster",

"mode" : "MULTICAST",
"address" : "239.0.0.1",
"port" : 31999,
"jdbcAddress" : "239.0.0.1",
"jdbcPort" : 41999,
"transactionMember": "",
"sqlMember": "",
"providerUrl": null
}
]
}

Now to run WebAPI:

$ sudo su - gsadm
export CLASSPATH=:/usr/share/java/gridstore.jar
cd webapi/
java -jar lib/griddb-webapi-ce-2.1.0.jar 2>&1 /dev/null &

Check $GS_HOME/logs/gs_webapi*.log for errors, if it has successfully started, the logs should contain:

[main] org.springframework.boot.web.embedded.tomcat.TomcatWebServer:
Tomcat started on port(s): 8080 (http) with context path ''
[main] org.springframework.boot.StartupInfoLogger: Started Application in
2.677 seconds (JVM running for 3.278)

Use Fluentd GridDB Plugin

Now it’s time to install the GridDB Fluentd connector with Ruby Gems.

$ sudo /opt/td-agent/embedded/bin/gem install fluent-plugin-griddb

With the plugin installed we’ll add the simplest GridDB configuration to /etc/td-agent/td-agent.conf and then restart the td-agent.

@type griddb
host http://localhost:8080/
cluster defaultCluster
database public
container container_1
username admin
password admin

$ sudo systemctl start td-agent.service

Before Fluentd can write data to GridDB, you need to create the container using the GridDB WebAPI:

$ curl -X POST --basic -u admin:admin -H "Content-type:application/json" -d '{"container_name":"container_1", "container_type":"TIME_SERIES", "rowkey":true, "columns":[{"name": "date", "type": "TIMESTAMP" }, {"name": "value", "type": "DOUBLE" },{"name": "str", "type": "STRING" }]}' http://localhost:8080/griddb/v2/defaultCluster/dbs/public/containers

With the GridDB plugin in place, you can POST data to Fluentd:

$ curl -X POST -d 'json={"date":"2018-09-20T12:08:21.112Z",
"value":"4.23", "str":"Hello World"}' http://localhost:8888/griddb

And using the GridDB WebAPI, we can see the data we POSTed to FluentD:

$ curl -X POST --basic -u admin:admin -H "Content-type:application/json" -d '{"limit":1000}' http://localhost:8080/griddb/v2/defaultCluster/dbs/public/containers/container_1/rows

This is just the beginning of what you can do with Fluentd and its GridDB plugin. For more information, please check out the detailed documentation.

If you have any questions about the blog, please create a Stack Overflow post here https://stackoverflow.com/questions/ask?tags=griddb .
Make sure that you use the “griddb” tag so our engineers can quickly reply to your questions.