{"id":46719,"date":"2022-08-03T00:00:00","date_gmt":"2022-08-03T07:00:00","guid":{"rendered":"https:\/\/griddb-linux-hte8hndjf8cka8ht.westus-01.azurewebsites.net\/blog\/logstash-griddb\/"},"modified":"2025-11-13T12:56:12","modified_gmt":"2025-11-13T20:56:12","slug":"logstash-griddb","status":"publish","type":"post","link":"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/","title":{"rendered":"Sending Syslog Messages into GridDB Database Using the Logstash Output Plugin"},"content":{"rendered":"

In this blog, we’ll cover the usage of the GridDB Output plugin for Logstash which enables us to write syslog (and more) log records to a GridDB Cloud container. Logstash<\/a> is a server-side log file parser that allows you to store metrics in a database of your choosing. It is often used alongside Elasticsearch (to stash the data) and Kibana (to visualize the data) as part of the ELK-stack. Replacing Elasticsearch with GridDB leverages GridDB’s high performance to lower operational costs of your log monitoring infrastructure.<\/p>\n

To demonstrate the GridDB Logstash plugin, we’ll install and configure Logstash with some filters to push unsuccessful SSH login messages generated by the local server’s Syslog daemon that Logstash will then store in GridDB Cloud. Finally, using GridDB Cloud again, we’ll build a query for a simple SSH intrusion detection system.<\/p>\n

Installation<\/h2>\n

The GridDB Logstash plugin requires CentOS 7.9 (x64), Ruby v2.4+, Logstash 7.15+<\/p>\n

First we install RVM & Ruby:<\/p>\n

\n
$ curl -sSL https:\/\/rvm.io\/mpapis.asc | gpg --import -\n$ curl -sSL https:\/\/rvm.io\/pkuczynski.asc | gpg --import -\n$ curl -L get.rvm.io | bash -s stable\n$ source ~\/.rvm\/scripts\/rvm\n$ rvm reload\n$ rvm install 2.5.3<\/code><\/pre>\n<\/div>\n
Now we setup the Logstash YUM Repo and install Logstash:<\/p>\n
\n
$ cat << EOF | tee \/etc\/yum.repos.d\/logstash.repo\n[logstash-7.x] \nname=Elastic repository for 7.x packages \nbaseurl=https:\/\/artifacts.elastic.co\/packages\/7.x\/yum \ngpgcheck=1 \ngpgkey=https:\/\/artifacts.elastic.co\/GPG-KEY-elasticsearch \nenabled=1 \nautorefresh=1 \ntype=rpm-md\nEOF\n\n$ yum install logstash<\/code><\/pre>\n<\/div>\n
Now we’re ready to build and install the GridDB Logstash plugin:<\/p>\n
\n
$ cd \/path\/to\/logstash-output-griddb\n$ gem build logstash-output-griddb.gemspec\n$ \/usr\/share\/logstash\/bin\/logstash-plugin install logstash-output-griddb-1.0.0.gem<\/code><\/pre>\n<\/div>\n
Now we’re ready to start using Logstash with the GridDB output plugin.<\/p>\n
Basic Usage<\/h2>\n
First, we need to create a config file \/etc\/logstash\/mylogstash.conf with both input and output sections. In the input section, we configure Logstash to simply read \/var\/log\/secure and parse it as a syslog output. This will let us track all logins.<\/p>\n
input { file { path => “\/var\/log\/secure” type => “syslog” } }<\/p>\n
output { griddb { host => “https:\/\/cloud1.griddb.com\/trial1234” cluster => “gs_clustertrial1234” database => “public” container => “secure_syslog”
\nusername => “logstash” password => “” insert_mode => “append” } } In the output section, we configure the GridDB output plugin to write to our GridDB cloud account. Using append mode which will append to an existing container if it exists. There is also a Replace mode which deletes and creates the container every time Logstash restarts.<\/yourpassword><\/p>\n
Create a Logstash user in the GridDB Cloud portal:<\/p>\n
\"\"<\/a><\/p>\n
Now start Logstash with your config file: $ sudo \/usr\/share\/logstash\/bin\/logstash -f \/etc\/logstash\/mylogstash.conf –path.settings \/etc\/logstash<\/p>\n
Now when a user logins or uses sudo, the secure log messages will be written to GridDB.<\/p>\n
\"\"<\/a><\/p>\n
\"\"<\/a><\/p>\n
Filtering, Grok and Mutate<\/h2>\n
Secure log has more than just SSHD messages, we can filter out non-SSH messages by adding the following filter:<\/p>\n
\n
filter {\n    if ([message] !~ \"sshd\") {\n        drop { }\n    }\n} <\/code><\/pre>\n<\/div>\n
What if you only want to track to unsuccessful logins? When a login fails, a “Connection closed by $remotehost port $port [preauth]” entry is added to secure log. First we filter out all messages that do not have contain “Connection closed by” and “[preauth]”.<\/p>\n
\n
    if ([message] !~ \"[preauth]\") {\n        drop { }\n    }\n    if ([message] !~ \"Connection closed by\") {\n        drop { }\n    }\n <\/code><\/pre>\n<\/div>\n
Then we can grok for the log entry to get the remote host which is added to the remhost field. This block also gets to the filter block. Grok is extremely powerful, it is worth checking out its capabilities here<\/a>.<\/p>\n
\n
    grok { \n        match => { \"message\" => \"closed by %{IPORHOST:remhost} port \"  }\n    }\n<\/code><\/pre>\n<\/div>\n
Now we can use mutate to remove unnecessary fields. Mutate can alter fields in many ways documented here<\/a>.<\/p>\n
\n
    mutate {\n        remove_field => [\"path\", \"type\", \"version\"]\n    }\n<\/code><\/pre>\n<\/div>\n
Now we can use aggregation functions in the query to count the number of unsuccessful logins from different remote hosts.<\/p>\n
\"\"<\/a><\/p>\n
Taking the output of that query, we can see the remote hosts with the most number of unsuccessful logins. Rules can be added to the firewall to prevent these attackers from even connecting to your SSH server.<\/p>\n","protected":false},"excerpt":{"rendered":"
In this blog, we’ll cover the usage of the GridDB Output plugin for Logstash which enables us to write syslog (and more) log records to a GridDB Cloud container. Logstash is a server-side log file parser that allows you to store metrics in a database of your choosing. It is often used alongside Elasticsearch (to […]<\/p>\n","protected":false},"author":71,"featured_media":28678,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[121],"tags":[],"class_list":["post-46719","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog"],"acf":[],"yoast_head":"\nSending Syslog Messages into GridDB Database Using the Logstash Output Plugin | GridDB: Open Source Time Series Database for IoT<\/title>\n<meta name=\"description\" content=\"In this blog, we'll cover the usage of the GridDB Output plugin for Logstash which enables us to write syslog (and more) log records to a GridDB Cloud\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Sending Syslog Messages into GridDB Database Using the Logstash Output Plugin | GridDB: Open Source Time Series Database for IoT\" \/>\n<meta property=\"og:description\" content=\"In this blog, we'll cover the usage of the GridDB Output plugin for Logstash which enables us to write syslog (and more) log records to a GridDB Cloud\" \/>\n<meta property=\"og:url\" content=\"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/\" \/>\n<meta property=\"og:site_name\" content=\"GridDB: Open Source Time Series Database for IoT\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/griddbcommunity\/\" \/>\n<meta property=\"article:published_time\" content=\"2022-08-03T07:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-11-13T20:56:12+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/griddb.net\/wp-content\/uploads\/2022\/08\/logstash.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1160\" \/>\n\t<meta property=\"og:image:height\" content=\"653\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Owen\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@GridDBCommunity\" \/>\n<meta name=\"twitter:site\" content=\"@GridDBCommunity\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Owen\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/\"},\"author\":{\"name\":\"Owen\",\"@id\":\"https:\/\/griddb.net\/en\/#\/schema\/person\/0f2f6d4b593adde8c43cf3ea5c794c66\"},\"headline\":\"Sending Syslog Messages into GridDB Database Using the Logstash Output Plugin\",\"datePublished\":\"2022-08-03T07:00:00+00:00\",\"dateModified\":\"2025-11-13T20:56:12+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/\"},\"wordCount\":558,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/griddb.net\/en\/#organization\"},\"image\":{\"@id\":\"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/#primaryimage\"},\"thumbnailUrl\":\"\/wp-content\/uploads\/2022\/08\/logstash.png\",\"articleSection\":[\"Blog\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/\",\"url\":\"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/\",\"name\":\"Sending Syslog Messages into GridDB Database Using the Logstash Output Plugin | GridDB: Open Source Time Series Database for IoT\",\"isPartOf\":{\"@id\":\"https:\/\/griddb.net\/en\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/#primaryimage\"},\"thumbnailUrl\":\"\/wp-content\/uploads\/2022\/08\/logstash.png\",\"datePublished\":\"2022-08-03T07:00:00+00:00\",\"dateModified\":\"2025-11-13T20:56:12+00:00\",\"description\":\"In this blog, we'll cover the usage of the GridDB Output plugin for Logstash which enables us to write syslog (and more) log records to a GridDB Cloud\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/#primaryimage\",\"url\":\"\/wp-content\/uploads\/2022\/08\/logstash.png\",\"contentUrl\":\"\/wp-content\/uploads\/2022\/08\/logstash.png\",\"width\":1160,\"height\":653},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/griddb.net\/en\/#website\",\"url\":\"https:\/\/griddb.net\/en\/\",\"name\":\"GridDB: Open Source Time Series Database for IoT\",\"description\":\"GridDB is an open source time-series database with the performance of NoSQL and convenience of SQL\",\"publisher\":{\"@id\":\"https:\/\/griddb.net\/en\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/griddb.net\/en\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/griddb.net\/en\/#organization\",\"name\":\"Fixstars\",\"url\":\"https:\/\/griddb.net\/en\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/griddb.net\/en\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/griddb.net\/wp-content\/uploads\/2019\/04\/fixstars_logo_web_tagline.png\",\"contentUrl\":\"https:\/\/griddb.net\/wp-content\/uploads\/2019\/04\/fixstars_logo_web_tagline.png\",\"width\":200,\"height\":83,\"caption\":\"Fixstars\"},\"image\":{\"@id\":\"https:\/\/griddb.net\/en\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/griddbcommunity\/\",\"https:\/\/x.com\/GridDBCommunity\",\"https:\/\/www.linkedin.com\/company\/griddb-by-toshiba\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/griddb.net\/en\/#\/schema\/person\/0f2f6d4b593adde8c43cf3ea5c794c66\",\"name\":\"Owen\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/griddb.net\/en\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/47438a5c81215c7a9043be1b427e0bbd8dc0f77bd536f147f8495575149e4325?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/47438a5c81215c7a9043be1b427e0bbd8dc0f77bd536f147f8495575149e4325?s=96&d=mm&r=g\",\"caption\":\"Owen\"},\"url\":\"https:\/\/griddb.net\/en\/author\/owen\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Sending Syslog Messages into GridDB Database Using the Logstash Output Plugin | GridDB: Open Source Time Series Database for IoT","description":"In this blog, we'll cover the usage of the GridDB Output plugin for Logstash which enables us to write syslog (and more) log records to a GridDB Cloud","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/","og_locale":"en_US","og_type":"article","og_title":"Sending Syslog Messages into GridDB Database Using the Logstash Output Plugin | GridDB: Open Source Time Series Database for IoT","og_description":"In this blog, we'll cover the usage of the GridDB Output plugin for Logstash which enables us to write syslog (and more) log records to a GridDB Cloud","og_url":"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/","og_site_name":"GridDB: Open Source Time Series Database for IoT","article_publisher":"https:\/\/www.facebook.com\/griddbcommunity\/","article_published_time":"2022-08-03T07:00:00+00:00","article_modified_time":"2025-11-13T20:56:12+00:00","og_image":[{"width":1160,"height":653,"url":"https:\/\/griddb.net\/wp-content\/uploads\/2022\/08\/logstash.png","type":"image\/png"}],"author":"Owen","twitter_card":"summary_large_image","twitter_creator":"@GridDBCommunity","twitter_site":"@GridDBCommunity","twitter_misc":{"Written by":"Owen","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/#article","isPartOf":{"@id":"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/"},"author":{"name":"Owen","@id":"https:\/\/griddb.net\/en\/#\/schema\/person\/0f2f6d4b593adde8c43cf3ea5c794c66"},"headline":"Sending Syslog Messages into GridDB Database Using the Logstash Output Plugin","datePublished":"2022-08-03T07:00:00+00:00","dateModified":"2025-11-13T20:56:12+00:00","mainEntityOfPage":{"@id":"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/"},"wordCount":558,"commentCount":0,"publisher":{"@id":"https:\/\/griddb.net\/en\/#organization"},"image":{"@id":"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/#primaryimage"},"thumbnailUrl":"\/wp-content\/uploads\/2022\/08\/logstash.png","articleSection":["Blog"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/griddb.net\/en\/blog\/logstash-griddb\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/","url":"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/","name":"Sending Syslog Messages into GridDB Database Using the Logstash Output Plugin | GridDB: Open Source Time Series Database for IoT","isPartOf":{"@id":"https:\/\/griddb.net\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/#primaryimage"},"image":{"@id":"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/#primaryimage"},"thumbnailUrl":"\/wp-content\/uploads\/2022\/08\/logstash.png","datePublished":"2022-08-03T07:00:00+00:00","dateModified":"2025-11-13T20:56:12+00:00","description":"In this blog, we'll cover the usage of the GridDB Output plugin for Logstash which enables us to write syslog (and more) log records to a GridDB Cloud","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/griddb.net\/en\/blog\/logstash-griddb\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/griddb.net\/en\/blog\/logstash-griddb\/#primaryimage","url":"\/wp-content\/uploads\/2022\/08\/logstash.png","contentUrl":"\/wp-content\/uploads\/2022\/08\/logstash.png","width":1160,"height":653},{"@type":"WebSite","@id":"https:\/\/griddb.net\/en\/#website","url":"https:\/\/griddb.net\/en\/","name":"GridDB: Open Source Time Series Database for IoT","description":"GridDB is an open source time-series database with the performance of NoSQL and convenience of SQL","publisher":{"@id":"https:\/\/griddb.net\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/griddb.net\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/griddb.net\/en\/#organization","name":"Fixstars","url":"https:\/\/griddb.net\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/griddb.net\/en\/#\/schema\/logo\/image\/","url":"https:\/\/griddb.net\/wp-content\/uploads\/2019\/04\/fixstars_logo_web_tagline.png","contentUrl":"https:\/\/griddb.net\/wp-content\/uploads\/2019\/04\/fixstars_logo_web_tagline.png","width":200,"height":83,"caption":"Fixstars"},"image":{"@id":"https:\/\/griddb.net\/en\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/griddbcommunity\/","https:\/\/x.com\/GridDBCommunity","https:\/\/www.linkedin.com\/company\/griddb-by-toshiba"]},{"@type":"Person","@id":"https:\/\/griddb.net\/en\/#\/schema\/person\/0f2f6d4b593adde8c43cf3ea5c794c66","name":"Owen","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/griddb.net\/en\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/47438a5c81215c7a9043be1b427e0bbd8dc0f77bd536f147f8495575149e4325?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/47438a5c81215c7a9043be1b427e0bbd8dc0f77bd536f147f8495575149e4325?s=96&d=mm&r=g","caption":"Owen"},"url":"https:\/\/griddb.net\/en\/author\/owen\/"}]}},"_links":{"self":[{"href":"https:\/\/griddb.net\/en\/wp-json\/wp\/v2\/posts\/46719","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/griddb.net\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/griddb.net\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/griddb.net\/en\/wp-json\/wp\/v2\/users\/71"}],"replies":[{"embeddable":true,"href":"https:\/\/griddb.net\/en\/wp-json\/wp\/v2\/comments?post=46719"}],"version-history":[{"count":1,"href":"https:\/\/griddb.net\/en\/wp-json\/wp\/v2\/posts\/46719\/revisions"}],"predecessor-version":[{"id":51391,"href":"https:\/\/griddb.net\/en\/wp-json\/wp\/v2\/posts\/46719\/revisions\/51391"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/griddb.net\/en\/wp-json\/wp\/v2\/media\/28678"}],"wp:attachment":[{"href":"https:\/\/griddb.net\/en\/wp-json\/wp\/v2\/media?parent=46719"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/griddb.net\/en\/wp-json\/wp\/v2\/categories?post=46719"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/griddb.net\/en\/wp-json\/wp\/v2\/tags?post=46719"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}