Blog

Fluentd is one of the most widely deployed log collectors in the world, and log ingestion is one of the workloads GridDB was built for: high-volume, append-heavy, time-stamped data. In this article we wire the two together using the fluent-plugin-griddb output plugin and GridDB Cloud, and then keep increasing the ingestion rate until something gives. As a brief summary: one small VM went from 15,000 to 150,000 rows per second by changing a single buffer parameter, and when we then quadrupled the collector hardware to push past that, throughput barely moved. We found the ceiling, and the way it shows up is more interesting than the number itself: not errors, not rejections, but quiet backpressure. Along the way we’ll cover a realistic installation (the plugin targets an older Ruby, so there is some dependency archaeology involved), a small patch the plugin needs before its buffered mode works, the container schema gotchas that the Web API’s positional row format will happily punish you for, and a version-drift bug that dressed itself up as a TLS problem. Why Fluentd GridDB has steadily grown its connector ecosystem: Kafka, JDBC, Grafana, and a range of collection agents can all read from or write to GridDB, which means it usually slots into an existing data pipeline rather than demanding a new one. Fluentd is a natural addition to that list. It is the de facto standard for log collection in a lot of Kubernetes and traditional server environments, it has hundreds of input and parser plugins, and once events are flowing through it, pointing them at GridDB is a single block. Everything in this post talks to GridDB Cloud through its Web API: JSON rows over HTTPS; it works from anywhere without special client libraries, and, as the numbers below will show, with sensible batching it moves more data than most logging workloads will ever generate. Installation For installation, fluent-plugin-griddb targets Ruby 2.7 and Fluentd 1.12. Ruby 2.7 predates OpenSSL 3, which is what modern Ubuntu ships, so Ruby will not compile against the system OpenSSL. The fix is to build OpenSSL 1.1.1 from source into its own prefix, build Ruby 2.7.2 against it with rbenv, and then pin RubyGems, Bundler, and Fluentd to the last versions that still support Ruby 2.7. It sounds worse than it is; each step is mechanical. 1. System dependencies $ sudo apt update $ sudo apt install -y git curl build-essential libreadline-dev zlib1g-dev \ libyaml-dev libncurses5-dev libffi-dev libgdbm-dev 2. OpenSSL 1.1.1 from source $ cd /tmp $ wget https://www.openssl.org/source/openssl-1.1.1w.tar.gz $ tar xvfz openssl-1.1.1w.tar.gz $ cd openssl-1.1.1w $ ./config –prefix=/opt/openssl-1.1.1w –openssldir=/opt/openssl-1.1.1w shared zlib $ make -j$(nproc) $ sudo make install One step that is easy to miss: the freshly built OpenSSL has an empty certificate directory, so anything compiled against it will fail TLS verification, including HTTPS calls to GridDB Cloud. Point it at the system certificate store: $ sudo rm -rf /opt/openssl-1.1.1w/certs $ sudo ln -s /etc/ssl/certs /opt/openssl-1.1.1w 3. rbenv and Ruby 2.7.2 Install rbenv, but pull ruby-build directly from GitHub. The version definitions that come with the packaged rbenv are stale and may not build 2.7.2 cleanly: $ sudo apt install -y rbenv $ mkdir -p "$(rbenv root)"/plugins $ git clone https://github.com/rbenv/ruby-build.git "$(rbenv root)"/plugins/ruby-build $ RUBY_CONFIGURE_OPTS="–with-openssl-dir=/opt/openssl-1.1.1w" rbenv install 2.7.2 $ rbenv global 2.7.2 $ echo 'eval "$(rbenv init – bash)"' >> ~/.bashrc $ source ~/.bashrc $ ruby –version # should report 2.7.2 4. Fluentd and the plugin Ruby 2.7 also needs older RubyGems and Bundler; the current releases of both have dropped support. Pin them, then install Fluentd 1.12.0 and the plugin gem: $ gem update –system 3.4.22 $ gem install bundler -v 2.4.22 $ gem install fluentd –version "1.12.0" –no-doc $ rbenv rehash $ gem install –force –local fluent-plugin-griddb-1.0.2.gem $ rbenv rehash A quick fluentd –version confirms the pipeline pieces are in place. And make sure it is 1.0.2 of the plugin specifically; more on why in a moment. Patching the Plugin Version 1.0.2 of the plugin has a latent bug: it works in its default, unbuffered mode, but the moment you add a section to the match block (which you will want for any serious throughput), Fluentd switches the plugin to its buffered code path, and that path calls a method that does not exist: $ error_class=NoMethodError error="undefined method `handle_record' for #<Fluent::Plugin::GriddbOutput> Did you mean? handle_create_container" Inside out_griddb.rb, the unbuffered process path routes records through a method called handle_insert_type, which works. The buffered write(chunk) path calls handle_record, which was never implemented. Every chunk raises, gets flagged as a bad chunk, and is shunted to Fluentd’s backup directory. The fix is one line: make the buffered path call the same method the working path uses. $ sed -i 's/handle_record(record_array)/handle_insert_type(record_array)/' \ $ "$(gem which fluent/plugin/out_griddb)" Note the use of gem which to locate the file. If you have the plugin’s source repository checked out as well as the gem installed, it is easy to edit the wrong copy and wonder why nothing changed. gem which fluent/plugin/out_griddb tells you exactly which file Fluentd loads. Two related notes while we are in the plugin internals. First, the plugin’s insert_type parameter accepts single and multiple, but multiple is currently a stub that logs “Currently insert type only support for single type” and does nothing, so leave it on single. Second, and this matters for the results later: even in single mode, the buffered path sends an entire buffer chunk in one HTTP request. So batch size is controlled by the buffer’s chunk settings, not by the insert type. Creating the Target Container The Web API inserts rows positionally: each row arrives as an array of values, matched to columns strictly by order. That makes the container schema definition load-bearing in a way that name-based systems are not, and it produced two instructive failures before the schema below worked. The event we will generate has six fields, plus a seq counter that Fluentd’s sample input appends to the end of each record. Appends, as in: seq arrives as the last value in the row, so it must be the last column in the schema. Defining it first produces The specified data cannot be converted to LONG type as the host string lands in the seq slot. Defining extra columns the record does not send (say, copying the schema of an apache-parsed container with user and referer fields) produces Row data is invalid from the value-count mismatch. The schema that matches what the plugin actually sends: $ curl -X POST \ $ -u "$GRIDDB_USER:$GRIDDB_PASS" \ $ -H 'Content-Type: application/json' \ $ "$GRIDDB_WEBAPI_URL/dbs/$GRIDDB_DB/containers" \ $ -d '{ $ "container_name": "fluentdStress", $ "container_type": "COLLECTION", $ "rowkey": false, $ "columns": [ $ {"name": "host", "type": "STRING"}, $ {"name": "method", "type": "STRING"}, $ {"name": "path", "type": "STRING"}, $ {"name": "code", "type": "INTEGER"}, $ {"name": "size", "type": "INTEGER"}, $ {"name": "agent", "type": "STRING"}, $ {"name": "seq", "type": "LONG"} $ ] $ }' Or you can use the GridDB Cloud CLI Tool: griddb-cloud-cli create -i. We use a COLLECTION with rowkey false rather than a time-series container on purpose. A time-series container upserts on duplicate timestamps, and at tens of thousands of rows per second many events share a millisecond. A throughput test against a time-series container without a carefully unique key silently measures overwrites while reporting success. A rowkey-less collection accepts every insert as a distinct row, which is what we want to count. The Stress Configuration For a throughput test, tailing a log file written by a generator makes the generator and the disk the bottleneck. Fluentd’s built-in sample input creates events in memory instead, which lets us dial the input rate directly: $ <system> $ workers 4 # match your core count $ </system> $ <source> $ @type sample $ tag stress.griddb $ size 200 # events per emit $ rate 50000 # events per second $ auto_increment_key seq # unique trailing counter per event $ dummy {"host":"10.0.0.1","method":"GET","path":"/api/v1/users","code":200,"size":4096,"agent":"gridstress/1.0"} $ </source> $ <match stress.griddb> $ @type griddb $ host "$GRIDDB_CLOUD_HOST" $ cluster "$GRIDDB_CLUSTER" $ database "$GRIDDB_DB" $ container "fluentdStress" $ insert_mode "append" $ insert_type "single" $ username "$GRIDDB_USER" $ password "$GRIDDB_PASS" $ <buffer> $ @type memory $ chunk_limit_records 50000 # rows per HTTP request: the knob that matters $ chunk_limit_size 64MB $ flush_thread_count 8 $ flush_mode interval $ flush_interval 1s $ flush_thread_interval 0.1 $ total_limit_size 4GB $ retry_max_times 3 $ retry_type periodic $ retry_wait 1s $ overflow_action throw_exception $ </buffer> $ </match> The parameters worth understanding: chunk_limit_records sets how many rows travel in each PUT to the Web API, flush_thread_count sets how many of those requests run in parallel, and workers in the system block spreads the whole pipeline across CPU cores. Everything else is safety plumbing: total_limit_size bounds the memory buffer, and overflow_action throw_exception makes backpressure loud instead of silent, which is exactly what you want when the point of the exercise is to see what breaks. Measuring Fluentd’s own logs tell you what it sent; they do not tell you what the database kept. The only number that counts is the row count in the container, so we sample it and compute the delta over the actual elapsed time: $ A=$(griddb-cloud-cli sql query -s "select count(*) from fluentdStress" \ $ | grep -o '"Value":[0-9]*' | grep -o '[0-9]*'); T1=$(date +%s) $ sleep 60 $ B=$(griddb-cloud-cli sql query -s "select count(*) from fluentdStress" \ $ | grep -o '"Value":[0-9]*' | grep -o '[0-9]*'); T2=$(date +%s) $ echo "confirmed rows/sec: $(( (B – A) / (T2 – T1) ))" One measurement lesson learned the hard way: divide by real elapsed time, not the interval you asked for. Once the container holds tens of millions of rows, count(*) itself takes several seconds, and a sampler that assumes its nominal interval will overstate throughput badly. And confirmed rows per second, measured at the destination, is the honest metric. If Fluentd claims success while this number lags what was sent, rows are being dropped somewhere. In our runs that gap never appeared: everything sent was stored. Round One: Batch Size Is the Whole Game All single-node runs use the same event shape. The only variables changed between runs are the chunk size and the worker count. chunk_limit_records workers node(s) sustained rows/sec 5,000 1 1 x 2-core ~15,000 20,000 2 1 x 2-core ~116,000 50,000 2 1 x 2-core ~150,000 The leap from 15k to 150k comes almost entirely from batching. Per-request overhead, HTTP, TLS, JSON framing, is fixed cost; the more rows amortize it, the closer you get to the pipeline’s real capacity. And do the request math: at 150,000 rows per second in 50,000-row chunks, the client is making roughly three HTTP requests per second. This resolves a confusion that comes up constantly with cloud database quotas: platform limits are typically expressed against requests and connections, not rows, and a well-batched pipeline moves enormous row volume through a trickle of requests. Row throughput and request rate are independent axes. Round Two: More Hardware, Same Ceiling To push the ingestion even further, we figured would throw in more cores and more nodes. So: two VMs, 4 cores each, 4 Fluentd workers each, identical configs, both writing to the same container. Eight workers across eight cores, four times the collector CPU that produced the 150k number. Setting up the second node produced a bug worth recounting, because it disguised itself well. The second machine’s Fluentd immediately failed every request with: 400 The plain HTTP request was sent to HTTPS port Microsoft-Azure-Application-Gateway/v2 The configs were byte-identical. OpenSSL versions matched. Certificate symlinks matched. No proxy variables anywhere. The actual cause: gem install fluent-plugin-griddb without a version pin had installed 1.0.1 on the new box instead of 1.0.2, and 1.0.1 handles the URL scheme differently, quietly speaking plaintext HTTP to port 443. A minor plugin version drift between two “identical” nodes produced what looked exactly like a TLS or gateway misconfiguration. Pin your plugin versions across a fleet. And remember that reinstalling the gem also wipes the handle_record patch, which must be reapplied on every fresh install. With both nodes on patched 1.0.2, the combined result over a sustained run: configuration total collector cores sustained rows/sec (combined) 1 node, 2 cores 2 ~150,000 2 nodes, 4 cores each 8 ~168,000 Quadrupling collector capacity bought about 12%. That flatline is the signature of a shared ceiling downstream of the collectors: somewhere in GridDB Cloud’s ingest path, whether the database engine or the Azure Application Gateway fronting it, the pipeline saturates around 150-170k rows per second on this plan. What the Ceiling Looks Like Here is the part that matters for anyone running this in production: the ceiling does not announce itself with errors. There were no 429s, no 503s, no failed inserts, no dropped rows at any point. Every single request returned success. What changed was latency. Buffer flushes that normally complete in about a second began taking 20 to 29 seconds: $ [warn]: #1 buffer flush took longer time than slow_flush_log_threshold: $ elapsed_time=26.086 slow_flush_log_threshold=20.0 $ [warn]: #2 buffer flush took longer time than slow_flush_log_threshold: $ elapsed_time=27.094 slow_flush_log_threshold=20.0 $ [warn]: #3 buffer flush took longer time than slow_flush_log_threshold: $ elapsed_time=28.739 slow_flush_log_threshold=20.0 Under sustained saturation, throughput oscillates between roughly 120k and 190k rows per second as flush queues build and drain. GridDB Cloud applies backpressure by slowing acceptance, not by rejecting requests. That is graceful behavior, nothing is lost, but it has a monitoring consequence: an alerting setup that watches only error rates will report a perfectly healthy pipeline while ingest latency has grown by 25x. Watch flush duration, not just status codes. Conclusion & Takeaways Tune the buffer before anything else. The difference between a naive Fluentd-to-GridDB config and a tuned one is an order of magnitude, and it is almost entirely chunk_limit_records. If your pipeline feels slow, your batches are almost certainly too small. One well-tuned node covers most workloads. A single 2-core VM sustained 150,000 rows per second. Before scaling out collectors, check whether you are anywhere near that; most logging workloads are not. The ceiling is real, and it is polite. Around 150-170k rows per second on this plan, the ingest path saturates regardless of collector count. It degrades by latency, not by error, so monitor flush duration. Sent and stored never diverged: even at saturation, nothing was dropped. Pin plugin versions across nodes. A one-patch-version drift between two collectors produced a protocol-level failure that looked like a TLS misconfiguration. Fleet consistency is not optional. Measure at the destination, against real elapsed time. count(*) deltas are the ground truth, and they get slow as the container grows; divide by actual seconds or your numbers flatter you. The Web API, for its part, holds up incredibly well. JSON over HTTPS carries overhead, and the native transport would spend fewer cycles per row. But sustaining 150,000+ rows per second, error-free, through the least exotic interface imaginable is more ingest than the vast majority of logging workloads will ever need. We set out to break GridDB Cloud, and the closest we got was making it take a little longer to say

If you are choosing an open source time series database to run on your own infrastructure, GridDB and TimescaleDB are two of the strongest candidates — and they take fundamentally different approaches. GridDB is a purpose-built time series database designed from the ground up for IoT-scale ingestion. TimescaleDB is a PostgreSQL extension that adds time series capabilities to a general-purpose relational database. For most, GridDB is the better choice for high-volume IoT and sensor workloads. But TimescaleDB is a genuinely good database, and there are workloads where it can be the correct choice. A note on scope: this comparison is GridDB Community Edition vs the open source edition of TimescaleDB At a Glance GridDB Community Edition TimescaleDB Current version 5.9 2.27 (May 2026) Vendor Toshiba TigerData (formerly Timescale Inc.) Architecture Purpose-built time series database, in-memory-first with disk persistence PostgreSQL extension (hypertables on top of Postgres) Data model Key-Container (Collection + TimeSeries containers) Relational tables, partitioned into hypertables/chunks Query interfaces NoSQL API (TQL) + SQL (JDBC) Full PostgreSQL SQL Horizontal scale-out (free edition) No — Community Edition is single-node; multi-node clustering is Enterprise Edition (commercial) No — multi-node was removed in v2.14; single-node only (scale-out via managed Tiger Cloud) High availability (free edition) No — clustering and failover are Enterprise Edition features Via external PostgreSQL replication tooling (Patroni, streaming replication) Compression Built-in time series compression, in the open source edition Columnstore compression — Timescale License (TSL) only, not in the Apache 2 edition Continuous aggregates / rollups Aggregation via TQL/SQL Continuous aggregates — TSL only License Server: AGPL v3 · Client libraries: Apache 2.0 Apache 2.0 (core) · TSL (source-available) for most advanced features Ecosystem GridDB clients (Java, Python, C, Go, Node.js, PHP), Kafka Connector, Grafana plugin The entire PostgreSQL ecosystem Best for Purpose-built high-cardinality IoT ingestion with advanced features under genuine open source Teams already on PostgreSQL, mixed relational + time series workloads Architecture: Purpose-Built vs Extension This is the most important difference, and a lot else flows from it. GridDB was designed by Toshiba specifically for IoT telemetry. Its architecture is in-memory-first — memory is the primary store and disk is secondary — with an event-driven engine designed to minimize overhead per operation. Data is organized into containers, and in a clustered deployment (Enterprise Edition) distributed across nodes automatically. The database was built assuming millions of devices writing high-frequency data, and the design decisions reflect that. TimescaleDB inherits PostgreSQL’s architecture: a row-oriented, disk-first, general-purpose relational engine. The extension adds hypertables — tables transparently partitioned into time-based chunks — plus a columnar compression engine for older data. This is a clever design that gets impressive results out of Postgres, but it is fundamentally an adaptation of an OLTP database to a time series workload, not a ground-up time series engine. The practical consequence: GridDB’s hot path for sensor ingestion (append a timestamped row to a TimeSeries container, in memory) is shorter than TimescaleDB’s (full PostgreSQL write path — WAL, MVCC bookkeeping, buffer management). For write-heavy IoT workloads, that architectural difference shows up in throughput and resource efficiency. The flip side: PostgreSQL’s general-purpose engine means TimescaleDB handles things GridDB doesn’t try to — complex multi-table joins, foreign keys against relational business data, window functions across arbitrary tables, and the full breadth of SQL. Data Model: Key-Container vs Hypertables GridDB uses a Key-Container model. Each data source — typically each sensor or device — gets its own container, addressed by key. TimeSeries containers treat the timestamp as the row key and provide time-specific operations (sampling, interpolation, time-window aggregation) natively. This maps one-to-one onto how real IoT systems are structured: a fleet of devices, each with its own schema and stream. Locating one device’s data never requires scanning an index shared with a million other devices. TimescaleDB keeps the relational model: all sensors typically write into one wide hypertable with a device ID column, and chunks partition it by time. This is comfortable if you think in SQL, and it makes cross-device analytical queries natural. The trade-off is that per-device access patterns compete inside shared structures, and schema differences between device types get awkward (sparse columns, JSONB, or table sprawl). Which model is better depends on your access pattern. Per-device reads and writes at high cardinality favor GridDB’s model. Ad-hoc analytics across all devices favor the relational model. Scalability: Both Free Editions Are Single-Node TimescaleDB’s multi-node support was deprecated and removed in version 2.14 (early 2024). Self-hosted TimescaleDB today is a single-node database: you scale by buying a bigger machine, and TigerData’s path to horizontal scale is their managed Tiger Cloud service. GridDB is the same story on the open source side: Community Edition runs as a single node, and multi-node clustering — automatic data partitioning, replication, and failover — is reserved for the commercial GridDB Enterprise Edition (and GridDB Cloud). So basically, we recommend you choose your free edition on single-node merits, and treat clustering as a separate, paid decision for both. A single GridDB node is less limiting than it sounds — its hybrid memory-and-disk design supports very large datasets (on the order of tens of terabytes per node) — and the same vertical-scaling logic applies to a single large TimescaleDB node. The meaningful open source difference is what each gives you within that single node, which is where licensing comes in. Licensing: Where the Open Source Editions Really Differ Neither database is “just open source,” and the difference here is the crux of the open source comparison. TimescaleDB ships in two editions. The Apache 2 edition is genuinely permissive open source — but it excludes most of what makes TimescaleDB compelling. Columnstore compression, continuous aggregates, and many recent performance features are part of the Community Edition, licensed under the Timescale License (TSL). The TSL is source-available, not OSI open source: it’s free to use as long as you are not offering TimescaleDB as a hosted database service, but it is a proprietary license with usage restrictions. In practice, the TimescaleDB you read about in benchmarks — compressed, with continuous aggregates — means accepting the TSL. GridDB Community Edition licenses the server under AGPL v3 and the client libraries under Apache 2.0, and — importantly — its advanced features, including built-in time series compression, are part of that open source edition. AGPL is a true OSI-approved open source license, though a strong copyleft one: if you modify the GridDB server and offer it as a network service, you must publish your modifications. For the overwhelmingly common case — running GridDB unmodified as the database behind your application, talking to it through the Apache 2.0 client libraries — AGPL imposes no obligations on your application code. Organizations with blanket AGPL bans should note this; everyone else will find it a non-issue. To summarize: GridDB CE gives you a purpose-built time series engine with its advanced features under an OSI open source license. TimescaleDB’s truly-open Apache edition is feature-limited, and its full capability is source-available (TSL) rather than open source. That — not clustering — is where GridDB’s free edition pulls ahead. Performance Both databases are fast, and both vendors publish benchmarks that favor themselves — including us. Rather than quote numbers here, we’ll note the architectural expectations and point you to reproducible tests. GridDB’s in-memory-first design and per-container data layout favor high-frequency ingestion and per-device range scans — the core IoT pattern. TimescaleDB’s columnstore compression and vectorized execution (significantly expanded in the 2.26–2.27 releases) favor analytical scans over compressed historical data. The Time Series Benchmark Suite (TSBS) supports both databases, and we encourage you to run it against your own workload shape — benchmark results vary enormously with cardinality, batch size, and query mix. TSDB_Evaluation_of_GridDB_QuestDB_and_TimescaleDB Ecosystem and Operations TimescaleDB wins on ecosystem. It is PostgreSQL, so every Postgres driver, ORM, BI tool, backup utility, monitoring exporter, and DBA skill set applies directly. If your team already runs Postgres, the operational learning curve is nearly zero. GridDB’s ecosystem is smaller but covers the IoT stack well: official clients for Java, Python, C, Go, Node.js, and PHP; a Kafka Connector for streaming pipelines; a Grafana plugin for visualization; and integrations with Telegraf, Fluentd, and Logstash for telemetry collection. Operationally, GridDB requires learning its own tooling — there is no equivalent of the Postgres DBA talent pool. When to Choose Which Choose GridDB if: Your workload is high-frequency sensor/IoT ingestion at high device cardinality You want a purpose-built time series engine rather than an extension on a general-purpose database You want advanced features like compression under a genuine open source license, not a source-available one You value having both a fast NoSQL (TQL) path and SQL (JDBC) access to the same data Choose TimescaleDB if: Your team already runs PostgreSQL and values that ecosystem above all Your workload mixes relational business data with time series and needs rich joins between them You’re comfortable with the TSL for compression and continuous aggregates, or plan to use Tiger Cloud A note on scale for both: if you outgrow a single node, horizontal clustering is a paid step either way — GridDB Enterprise Edition on one side, Tiger Cloud on the other. Factor that into a long-term decision rather than assuming the free edition will scale out. FAQ Is TimescaleDB still open source? Partially. The Apache 2 edition is open source but excludes columnstore compression, continuous aggregates, and most advanced features. The full-featured Community Edition uses the Timescale License (TSL), which is source-available but not OSI open source. Timescale Inc. rebranded as TigerData in June 2025. Can free TimescaleDB run as a cluster? No. Multi-node support was removed in TimescaleDB 2.14. Self-hosted TimescaleDB is single-node; horizontal scale requires their managed Tiger Cloud service. Can free GridDB run as a cluster? No. GridDB Community Edition runs as a single node. Multi-node clustering, replication, and failover are features of the commercial GridDB Enterprise Edition (and GridDB Cloud). Is GridDB SQL-compatible? Yes — GridDB provides a SQL interface via JDBC (its NewSQL interface) alongside its NoSQL API (TQL), and both are available in the open source Community Edition. Its SQL dialect is not as comprehensive as PostgreSQL’s, which is the trade-off for its purpose-built engine. Which is faster? It depends on workload shape. GridDB’s architecture favors high-frequency ingestion and per-device queries; TimescaleDB’s compression favors analytical scans of historical data. Run TSBS against your own workload — both databases support it. GridDB Community Edition is free and open source. Download it from GitHub or get started with the quick start

If you are choosing a database for time series data, GridDB and MongoDB are two of the most established options. In 2026 they make a surprisingly different bargain with you. GridDB is a purpose-built time series database whose Community Edition can query and analyze your full data history. MongoDB is a general-purpose document database that added native time series collections in version 5.0, keeping the document model and the rest of the MongoDB feature set available alongside them. We took GridDB Community Edition and MongoDB Community Server, ran through the workloads that matter to ingest, retention, and query history, and this page lays out the differences honestly so you can decide for yourself. A note on editions before we start. This comparison is primarily between GridDB Community Edition and MongoDB Community Server, since those are the two free, self-hosted, open source editions. Where the managed services (GridDB Cloud and MongoDB Atlas) change the answer, we say so. MongoDB’s time series collections have been available since 5.0 and are in the Community edition, not held back for commercial tiers. At a Glance GridDB Community Edition MongoDB Community Server Vendor Toshiba MongoDB, Inc. Architecture Purpose-built time series database, in-memory-first with disk persistence General-purpose document database; time series collections stored in an internal bucketed format Data model Key-Container (Collection and TimeSeries containers), strongly typed BSON documents with a timeField and an optional metaField; flexible per-document schema Query interfaces NoSQL API (TQL) plus SQL-92 over JDBC/ODBC MongoDB Query API and aggregation pipeline; SQL is read-only and Atlas-only Time series functions Built-in time-weighted average, time-bucket aggregation, interpolated sampling $setWindowFields, $densify, $fill, $dateTrunc in the aggregation pipeline Updates to measurements Full. Standard row inserts, updates, and deletes Restricted. Updates can only match on and modify the metaField; no upserts Writes in transactions Supported No. Time series collections cannot be written to inside a transaction; reads are supported Horizontal scale-out (free edition) No. Community Edition is single-node; multi-node clustering is in the commercial Advanced Edition Yes. Replica sets and sharding are included in Community Server High availability (free edition) No. Clustering and failover are Advanced Edition features Yes. Replica sets provide HA in the free edition Expiry (TTL) Yes, via expiration settings on the container Yes, via expireAfterSeconds Secondary indexes Yes, on container columns Yes, on any field since 6.0; compound metaField and timeField index created automatically Geospatial Native GEOMETRY type with spatial index and WKT, via the NoSQL interface GeoJSON with 2dsphere indexing, well established across the product License Server: AGPL v3 · Clients: Apache 2.0 SSPL for the server; drivers under Apache 2.0 Ecosystem GridDB clients (Java, Python, C, Go, Node.js, Ruby, PHP, Perl), Kafka Connector, Grafana, Fluentd, Telegraf, MQTT, Node-RED, Apache Arrow Very large general-purpose ecosystem: official drivers for nearly every language, Atlas, Compass, BI Connector, Kafka Connector Best for SQL-based analytics on high-frequency IoT and sensor data, on-premises or at the edge Applications where time series is one workload among several and the schema varies per document Architecture: Purpose-Built vs General-Purpose Both databases are well-engineered. They optimize for different things. GridDB was designed by Toshiba specifically for IoT telemetry. Its storage engine, container model, and built-in functions all assume you are ingesting large volumes of timestamped data at high frequency and querying it by time. It runs an in-memory-first architecture, a hybrid of memory and disk, tuned for high write throughput on this kind of data. The design assumes high-frequency data from many devices, retained and queryable over time. MongoDB approaches time series as a specialized capability layered onto a flexible document store. When you create a time series collection, MongoDB organizes writes so that data from the same source is stored alongside other data points from a similar point in time, in an internally optimized bucketed format. Under the hood, MongoDB treats time series collections as writable non-materialized views backed by an internal collection. The practical consequence cuts both ways. MongoDB’s bucketing gives you real time series efficiency without giving up the document model, so one system can serve both your operational data and your telemetry. GridDB’s narrower focus means the whole engine, not a collection type within it, is built around the time series access pattern. Data Model: Key-Container vs Time Series Collections GridDB uses a Key-Container model. Each data source, typically a sensor or device, gets its own container, addressed by key. A TimeSeries container uses a TIMESTAMP column as its row key. Data is strongly typed, which means schema is fixed up front and enforced. Through the SQL interface, a container is simply a table. MongoDB stores each measurement as a BSON document inside a time series collection. Each document carries a required timeField, an optional metaField identifying the source, and any number of measurement fields. The schema is flexible, so documents in the same collection can carry different fields entirely. The difference is clearest at creation time. MongoDB db.createCollection("sensors", { timeseries: { timeField: "timestamp", metaField: "sensorId", granularity: "seconds" }, expireAfterSeconds: 2592000 // auto-remove data after 30 days }) GridDB (SQL interface) CREATE TABLE sensors ( ts TIMESTAMP PRIMARY KEY, value DOUBLE ) USING TIMESERIES WITH (expiration_type='ROW', expiration_time=30, expiration_time_unit='DAY'); Which model suits you depends on your data. If every device emits the same fixed set of readings, GridDB’s typed containers give you enforcement and compactness. If your payloads vary by device generation, firmware version, or vendor, MongoDB’s per-document flexibility saves you from schema migrations. Query Interfaces: SQL vs Aggregation Pipeline This is often the deciding factor, and it is worth being precise because both databases are widely misdescribed here. GridDB offers two interfaces. TQL is a lightweight query language for the NoSQL side, and the SQL interface is SQL-92 compliant over standard JDBC and ODBC. A team that already knows SQL writes SELECT, JOIN, GROUP BY, and window queries without learning anything proprietary. The often-repeated claim that GridDB cannot do JOINs applies to TQL, not to the SQL interface. MongoDB uses its own Query API and aggregation pipeline rather than SQL, and for time series work it is genuinely capable. $setWindowFields gives you window functions such as moving averages and rank. $densify fills in missing time points and $fill populates their values by interpolation or carry-forward. $dateTrunc handles time bucketing for downsampling. The tradeoff is portability of skills and tooling. These are MongoDB-specific constructs, and a team standardized on SQL and SQL-based BI tools will face a learning curve. Atlas offers a read-only SQL interface for BI use cases, but that is a managed-service feature and it is read-only. The Update Restriction This is the difference that matters most between the two for operational workloads, and the one most likely to surprise people coming from a general-purpose MongoDB collection. MongoDB time series collections are designed around the assumption that telemetry is append-only. That assumption is enforced. Update commands must match only on the metaField value and can modify only the metaField value. Your update document can contain only update operator expressions, updates must be multi-document (multi: true or updateMany()), and upsert: true is not allowed. Deletes are supported and follow similar matching rules. Separately, you cannot write to a time series collection inside a multi-document transaction, though reads within transactions are supported. For a pure telemetry pipeline, none of this matters. Sensors emit readings, you never go back and edit them, and TTL handles the aging out. But a large share of real deployments do need corrections: a miscalibrated sensor whose readings need rescaling after the fact, a backfill that overwrites provisional values with settled ones, an ETL job that needs to be idempotent and therefore wants upserts. On MongoDB time series collections those patterns require a workaround, typically deleting and reinserting, or staging corrections in a separate regular collection. GridDB places no equivalent restriction on updating measurement values. If your workload involves in-place correction of individual readings, or writes that need to participate in transactions, test that path early on both databases rather than discovering it late. Scalability: Where the Free Editions Differ It is worth being precise here, because MongoDB has the advantage in the free edition and we would rather say so than have you find out later. MongoDB Community Server includes both replica sets and sharding. You get high availability and horizontal scale-out without paying for a commercial license. For time series collections specifically, the recommended practice is to include the metaField in the shard key so that data from the same source stays co-located. A few constraints apply: shard keys containing the timeField are deprecated as of 8.0, zone sharding is not supported for time series collections, and resharding a time series collection became available only in 8.0.10. MongoDB 8.0 also brought substantial improvements to sharding speed and cost. GridDB Community Edition is single-node. Multi-node clustering, replication, and automatic failover are part of the commercial Advanced Edition and of GridDB Cloud. The underlying architecture is a distributed shared-nothing design built for large-scale ingest, but you reach it through the commercial editions, not the free one. So if clustering or high availability on a zero-cost license is a hard requirement, MongoDB Community meets it and GridDB Community does not. If you are comparing commercial tier to commercial tier, both scale horizontally and the comparison comes back to workload fit. For either database, benchmark against a representative slice of your own workload rather than relying on general throughput claims. Results depend heavily on cardinality, batch sizes, and query patterns. Licensing GridDB ships the server under AGPL v3 with clients under Apache 2.0. AGPL imposes obligations on your application code: the copyleft only triggers if you modify the GridDB server itself and offer that modified server as a network service, but organizations with blanket AGPL bans should note this. Commercial Standard, Advanced, and Vector editions are available, as is the managed GridDB Cloud. MongoDB uses the SSPL for the server, with drivers under Apache 2.0. The SSPL is not an OSI-approved open source license, and it is more aggressive than AGPL in one specific direction: it targets organizations offering the database itself as a service. For the overwhelming majority of users, who are building an application on top of MongoDB rather than reselling MongoDB, it imposes no practical burden. But it has been enough to get MongoDB removed from some Linux distribution repositories, and some enterprise legal teams treat SSPL as disqualifying on principle. Neither license is a problem for a typical internal application. Both are worth a five-minute conversation with legal before you commit. Performance GridDB’s in-memory-first design and per-container layout favor high-frequency ingestion and per-device range scans across the full dataset. MongoDB’s bucketed storage format substantially narrows the gap against general-purpose document storage, and its compound metaField and timeField index serves the common “one source over a time window” query well. Rather than repeat vendor claims, we would point you at the published benchmark work and, more importantly, at your own workload. Cardinality, batch size, payload shape, and query mix move these numbers more than any architectural difference does. Ecosystem and Operations MongoDB wins on ecosystem breadth, and it is not close. Official drivers for essentially every language, one of the largest developer communities in the industry, extensive learning material, mature tooling such as Compass and the BI Connector, and Atlas as a fully managed global service. If hiring pool and tool availability are priorities, this is a real and durable advantage. GridDB’s ecosystem is narrower but aimed squarely at the IoT stack: official clients for Java, Python, C and C++, Go, Node.js, Ruby, PHP, and Perl, a REST/HTTP Web API, a Kafka Connect connector, and integrations with Grafana, Fluentd, Telegraf, MQTT, and Node-RED, plus an Apache Arrow interface for analytics. These are the components most IoT and observability pipelines are actually assembled from. When to Choose Which Choose GridDB if: Time series and IoT telemetry are your core workload, not a side workload Your team works in SQL and wants JDBC/ODBC access from existing BI tooling You need to update or correct individual measurement values in place You want writes that can participate in transactions You are deploying on-premises or at the edge, where a compact resource footprint matters You want purpose-built time functions rather than a general-purpose framework adapted to time series Choose MongoDB if: Time series is one workload among several and you want a single system for all of it Your measurement payloads vary by device, firmware, or vendor and a flexible schema saves you migrations You need clustering and high availability on the free edition Your team is already fluent in the aggregation pipeline, or already runs MongoDB or Atlas Ecosystem breadth, tooling, and hiring pool are high priorities Your telemetry is genuinely append-only, so the update restrictions never bind FAQ Does MongoDB support SQL for time series data? Not for writes, and not in the self-hosted Community edition. MongoDB uses its Query API and aggregation pipeline. Atlas provides a read-only SQL interface aimed at BI tools. GridDB provides a full SQL-92 interface over JDBC and ODBC in the free edition. Can I update a measurement value in a MongoDB time series collection? Not directly. Updates can only match on and modify the metaField, upserts are not allowed, and writes cannot occur inside a transaction. Correcting a measurement value generally means deleting and reinserting the document. Can GridDB Community Edition run as a cluster? No. GridDB Community Edition is single-node. Multi-node clustering, replication, and failover are part of the commercial Advanced Edition and GridDB Cloud. Can MongoDB Community Server run as a cluster? Yes. Replica sets and sharding are both included in the free Community Server, which is a genuine advantage over GridDB Community Edition on this specific point. Is MongoDB open source? Partially. The server is under the SSPL, which is not OSI-approved, though the source is available. The official drivers are Apache 2.0. In practice the SSPL only creates obligations for organizations offering MongoDB itself as a service, but some legal teams treat it as disqualifying regardless. Do MongoDB time series collections support TTL and secondary indexes? Yes to both. Automatic expiry is configured with expireAfterSeconds, and since version 6.0 secondary indexes can be created on any field, in addition to the compound metaField and timeField index MongoDB creates automatically. Which is faster? It depends on the workload. GridDB’s purpose-built engine favors high-frequency ingestion and per-device range scans over the full dataset. MongoDB’s bucketed format performs well on the common source-plus-time-window query and lets you keep telemetry alongside your operational data. Cardinality, batch size, and query mix will move your results more than the architectural difference will, so benchmark with your own

If you are choosing an open source time series database to run on your own infrastructure, GridDB and InfluxDB are two of the most established options — and in 2026 they make a surprisingly different bargain with you. GridDB is a purpose-built time series database whose free Community Edition can query and analyze your full data history. InfluxDB’s free, open source edition — InfluxDB 3 Core — is a modern, fast, single-node engine that InfluxData itself describes as a “recent-data engine,” with the production-grade historical and scaling capabilities reserved for its commercial tiers. We build GridDB, so we’ll say it plainly: for self-hosted, open source time series workloads that need to retain and query history, we think GridDB Community Edition is the stronger choice. But InfluxDB 3 is an impressive piece of engineering, it has the larger ecosystem by far, and there are real workloads where Core is exactly right. This page lays out the differences honestly so you can decide for yourself. A note on versions before we start: InfluxDB 1.x and 2.x are now in maintenance mode, and InfluxDB 3 is the current line. This comparison is primarily GridDB Community Edition vs InfluxDB 3 Core, since those are the two free, self-hosted, open source editions — an apples-to-apples match for the scope most people are actually deciding on. At a Glance GridDB Community Edition InfluxDB 3 Core Current version 5.9 3.x (Core, GA April 2025) Vendor Toshiba InfluxData Architecture Purpose-built time series database, in-memory-first with disk persistence Rust rewrite on Apache Arrow + DataFusion + Parquet; diskless, object-storage-backed Data model Key-Container (Collection + TimeSeries containers) Measurements with tags/fields, stored columnar Query interfaces NoSQL API (TQL) + SQL (JDBC) SQL + InfluxQL (Flux de-emphasized in v3) Horizontal scale-out (free edition) No — Community Edition is single-node; multi-node clustering is the commercial Enterprise Edition No — Core is single-node only; clustering requires commercial Enterprise/Clustered High availability (free edition) No — clustering and failover are Enterprise Edition features No — high availability is an Enterprise feature Long-term historical queries (free edition) Yes — full historical range, single node Limited — Core has no compactor and is optimized for recent data (~days); long-range analysis is an Enterprise feature Compression Built-in time series compression (in the open source edition) Parquet columnar compression (in Core) Cardinality Designed for high device cardinality Unlimited cardinality (a major v3 improvement over the old TSM engine) Storage backend Local disk Object storage (S3, GCS, Azure Blob) — diskless License Server: AGPL v3 · Clients: Apache 2.0 MIT + Apache 2.0 (genuinely permissive OSS) Ecosystem GridDB clients (Java, Python, C, Go, Node.js, PHP), Kafka Connector, Grafana, Telegraf, Fluentd Telegraf + the large InfluxDB ecosystem, broad SQL tooling Best for Long-term retention and analysis of high-cardinality IoT data on the open source edition Single-node real-time monitoring of recent data, object-storage-native deployments, teams wanting a modern columnar SQL engine Architecture: Purpose-Built vs Object-Storage Engine Both databases are well-engineered; they optimize for different things. GridDB was designed by Toshiba specifically for IoT telemetry. It is in-memory-first — memory is the primary store, disk is secondary — with an event-driven engine built to minimize per-operation overhead. In a clustered deployment (Enterprise Edition) data is distributed across nodes automatically; the open source Community Edition runs that engine on a single node. The design assumes high-frequency data from many devices, retained and queryable over time. InfluxDB 3 is a ground-up rewrite (the engine formerly called InfluxDB IOx) built in Rust on a modern open-data foundation: Apache Arrow for in-memory columnar processing, Apache DataFusion for query execution, and Apache Parquet for columnar storage on object stores. Its headline architectural feature is being diskless — it persists to S3, GCS, or Azure Blob Storage, separating compute from storage. This is genuinely modern and has real advantages: cheap and effectively unlimited storage, stateless and portable nodes, and a hot-data cache that serves recent queries from memory without touching the object store. The practical consequence cuts both ways. InfluxDB 3’s columnar-on-Parquet design is excellent for analytical scans and gives it the unlimited cardinality that the old TSM engine struggled with. GridDB’s in-memory-first, per-container layout favors high-frequency ingestion and per-device range scans, and — as the next sections cover — keeps the full dataset queryable on local disk without the recent-data caveats that constrain InfluxDB 3 Core. Data Model: Key-Container vs Measurements GridDB uses a Key-Container model. Each data source — typically each sensor or device — gets its own container, addressed by key. TimeSeries containers treat the timestamp as the row key and provide time-specific operations (sampling, interpolation, time-window aggregation) natively. This maps directly onto how IoT fleets are structured: many devices, each its own stream, each locatable without scanning structures shared with every other device. InfluxDB organizes data into measurements containing tagged points (tags are indexed metadata; fields are the values). In v3 this is stored columnar, which is what enables fast analytical aggregation and unlimited tag cardinality. If you think in terms of metrics and dimensions, and you want SQL over them, this model is comfortable and powerful. Which is better depends on access pattern: per-device reads/writes at high cardinality favor GridDB’s containers; broad analytical aggregation across all series favors InfluxDB’s columnar measurements. Scalability: Both Free Editions Are Single-Node It’s worth being precise here, because neither free edition scales out horizontally — and that’s true on both sides. InfluxDB 3 Core is single-node, full stop. Clustering, multi-node deployment, high availability, and read replicas are all reserved for the commercial InfluxDB 3 Enterprise (or the Kubernetes-based InfluxDB Clustered) product. This is not new behavior: InfluxData removed clustering from the open source edition back in 2016 and has kept it commercial ever since. GridDB is the same on the open source side: Community Edition runs as a single node, and multi-node clustering, replication, and failover are part of the commercial GridDB Enterprise Edition (and GridDB Cloud). So on clustering, it’s a wash — both put horizontal scale-out and HA behind a paid tier. If multi-node clustering on a free, open source edition is your hard requirement, neither fits. But clustering is not where these two free editions actually diverge. The decisive difference is what a single node can do with your data over time — and that’s the next section. The Recent-Data Limitation This is the difference that matters most between the free editions, and the one most likely to surprise people coming from InfluxDB 1.x or 2.x. InfluxData describes InfluxDB 3 Core as a “recent-data engine,” and that phrasing is deliberate. Core does not include a compactor — the background process that rewrites the many small Parquet files produced by ingestion into larger, sorted, indexed blocks. Without compaction, small files accumulate and query performance on older data degrades over time. The compactor is an Enterprise feature. There is also a single-query time-span limit. Historically, Core capped both writes and queries to a 72-hour window. In early 2025 InfluxData relaxed this: you can now write data with any historical timestamp, and the restriction on what period you can query was lifted — but the span a single query can cover remains limited (rooted in a cap on how many Parquet files one query plan will scan), precisely because there is no compactor to consolidate them. The net effect, per InfluxData’s own positioning and third-party documentation, is that Core is suited to real-time monitoring of roughly the last few days of data and is not recommended for long-term storage and historical analysis. For that, you move to Enterprise. This is where GridDB’s free edition has a decisive edge. GridDB Community Edition — on its single node — queries your full historical range with no comparable caveat, and includes built-in time series compression in the open source edition. Long-term retention and analysis are simply part of CE. So while neither free edition clusters, GridDB CE’s single node is a genuine long-term time series store, whereas InfluxDB 3 Core’s single node is, by design, a recent-data engine. We want to be fair: if your actual use case is real-time dashboards and alerting over recent data on a single node — a very common pattern — Core handles it extremely well, with sub-10ms queries on hot data and a genuinely “run and go” setup. The limitation only bites when you need long-range history. Licensing This is where InfluxDB earns an honest point against us. InfluxDB 3 Core’s license is more permissive than GridDB’s. Core is released under the MIT and Apache 2.0 licenses — classic, unrestricted open source with no copyleft obligations. You can do essentially anything with it. GridDB Community Edition licenses the server under AGPL v3 (a strong copyleft license) and the client libraries under Apache 2.0. For the common case — running GridDB unmodified as the database behind your application, talking to it through the Apache 2.0 clients — AGPL imposes no obligations on your application code. The copyleft only triggers if you modify the GridDB server itself and offer that modified server as a network service. Organizations with blanket AGPL bans should note this; most others will find it a non-issue. But license permissiveness is only one axis, and it’s worth being clear about what each free edition actually delivers. Both reserve multi-node clustering for a paid tier. The difference is in the single node: InfluxDB 3 Core is freely licensed but deliberately capability-limited for anything beyond recent data — no compaction, recent-data focus — so long-term historical analysis means buying Enterprise. GridDB CE’s single node is fully capable for long-term workloads — compression, full-range queries, and a dual NoSQL/SQL interface, all under the AGPL — without Core’s recent-data ceiling. So InfluxDB has the more permissive license; GridDB has the more capable free edition for historical work. (For completeness: InfluxData offers a free Home license for InfluxDB 3 Enterprise for non-commercial at-home use, which unlocks the full feature set for hobbyists. It is not licensed for commercial self-hosting.) Performance GridDB’s in-memory-first design and per-container layout favor high-frequency ingestion and per-device range scans across the full dataset. InfluxDB 3’s columnar Parquet engine and DataFusion query layer favor analytical aggregation and high-cardinality scans of recent data, with object storage decoupling storage cost from compute. The Time Series Benchmark Suite (TSBS) supports both, and results vary enormously with cardinality, batch size, query mix, and — for InfluxDB Core specifically — how much historical data has accumulated without compaction. Run it against your own workload shape. White Paper: Cloud_TSDB_Evaluation_GridDB_Influx_Mongo White Paper: TimeSeries_Database_Benchmark_GridDB_InfluxDB Ecosystem and Operations InfluxDB wins on ecosystem. It is one of the most widely deployed time series databases in the world, with Telegraf as a best-in-class collection agent, broad SQL tooling compatibility, an enormous body of community content, integrations everywhere, and a built-in Python processing engine in v3 for in-database transformation and alerting. GridDB’s ecosystem is smaller but covers the IoT stack well: official clients for Java, Python, C, Go, Node.js, and PHP; a Kafka Connector for streaming pipelines; a Grafana plugin; and integrations with Telegraf, Fluentd, and Logstash. When to Choose Which Choose GridDB if: Your workload is high-frequency, high-cardinality sensor/IoT ingestion You need to store and query long-term historical data on the free edition (InfluxDB Core can’t, without compaction) You want built-in compression and a purpose-built time series engine in the open source edition You value having both a fast NoSQL (TQL) path and SQL (JDBC) access to the same data Choose InfluxDB 3 Core if: Your use case is real-time monitoring and alerting over recent data (roughly the last few days) A single node meets your needs and you don’t require built-in HA You want an object-storage-native, diskless deployment (S3/GCS/Azure Blob) You value the InfluxDB/Telegraf ecosystem and a modern columnar SQL engine, and the most permissive possible license A note on scale for both: if you need clustering or high availability, that’s a paid step either way — GridDB Enterprise Edition on one side, InfluxDB 3 Enterprise on the other. Neither free edition clusters, so factor that into a long-term decision. FAQ Is InfluxDB open source? Partially, and it depends on the edition. InfluxDB 3 Core is genuinely open source under MIT and Apache 2.0 — but it is single-node and optimized for recent data. The production capabilities (clustering, high availability, compaction for long-term data) are in InfluxDB 3 Enterprise and InfluxDB Clustered, which are commercial products. InfluxDB 1.x and 2.x are open source but now in maintenance mode. Can free InfluxDB run as a cluster? No. InfluxDB 3 Core is single-node only; clustering has been a commercial (Enterprise) feature since 2016. Can free GridDB run as a cluster? No. GridDB Community Edition is single-node. Multi-node clustering, replication, and failover are part of the commercial Enterprise Edition (and GridDB Cloud). What is the InfluxDB 3 Core “72-hour limit”? A design constraint stemming from Core’s lack of a compactor: a single query plan is limited in how many Parquet files (and therefore how wide a time range) it can scan. InfluxData lifted the original restrictions on writing and querying historical periods in early 2025, but the single-query-span limit remains, and query performance on aged data degrades without compaction. For long-range historical analysis, Enterprise (which includes the compactor) is the intended product. Can GridDB query long-term historical data on the free edition? Yes — GridDB Community Edition queries your full historical range on a single node, without the recent-data caveats that apply to InfluxDB 3 Core. I’m still on InfluxDB 1.x or 2.x — does this apply? Those versions are single-node open source on the older TSM engine (with the high-cardinality limitations v3 was built to fix), and they are now in maintenance. They could query full history, unlike Core’s single-query span limit. InfluxData’s recommended path is migration to v3. If you’re evaluating a move anyway, it’s a natural moment to compare against GridDB. Which is faster? It depends on workload shape. GridDB favors high-frequency ingestion and per-device queries across the full dataset; InfluxDB 3 favors analytical scans of recent, high-cardinality data on its columnar

IoT devices can be messy and unweildy. The same sensor that reports a clean temperature reading one second will spit out an ugly, text-based firmware error the next, with both arriving in the same stream. If you force a single tool to handle that mixed data input, you end up compromising somewhere: metrics agents are miserable at parsing free-form text, and log processors are wasteful for high-volume numeric data. In this article we will not compromise. We will build a pipeline that splits the stream at the front door and hands each half to the tool that is good at it: Telegraf, InfluxData’s metrics agent, consumes the clean JSON metrics. Logstash, Elastic’s log processor, uses its grok filters to chop the unstructured firmware errors into queryable fields. Azure IoT Hub sits in front of both, acting as the router that routes each message to the right lane. GridDB Cloud is the destination for everything, using the official Telegraf and Logstash output plugins from the GridDB Cloud v3.2 bundled third-party plugin pack. If you read our previous article on storing OpenTelemetry signals in GridDB Cloud with Kafka, this architecture will feel familiar: a producer at the edge, a routing/buffering layer in the middle, specialized processors, and GridDB Cloud as the vault at the end. The difference is that this time there is zero custom code. Where the Kafka pipeline needed a hand-written Go bridge to flatten OTLP payloads, every stage here is an off-the-shelf component wired together with configuration. Though this blog uses the GridDB Web API, let’s not forget about this recent release: GridDB Cloud v3.2’s non-WebAPI connection support. If you have not read about that release yet, start here: Connecting to GridDB Cloud v3.2 from Your Local Dev Environment. The Architecture: Four Layers The architecture for this is a bit strange; it’s been separated into four parts: Layer 1 — Full data stream of mixed data (edge). Devices emit both signal types into one stream: structured JSON metrics and unstructured text logs, interleaved. Each message carries a small label (msgType) declaring which kind it is. Layer 2 — The router (Azure IoT Hub). IoT Hub’s message routing reads the label and splits the stream: metrics go to one Azure Event Hub, logs go to another. After this point the data is perfectly separated into two independent queues. Layer 3 — The specialized processors (Telegraf and Logstash). Because the split already happened, neither tool has to do a job it is bad at. Telegraf pulls from the metrics queue and parses JSON. Logstash pulls from the logs queue (text) and runs grok. Layer 4 — Datastore (GridDB Cloud). Both tools have an official GridDB output plugin. Telegraf writes its numbers into an optimized TIME_SERIES container; Logstash writes its parsed log rows into its own container. One database, two purpose-built containers. ┌─(msgType='metric')─► Event Hub eh-metrics ─► Telegraf ─► iot_sensor_metrics ┐ device ──► Azure IoT Hub ─┤ ├─ GridDB Cloud └─(msgType='log')────► Event Hub eh-logs ────► Logstash ─► iot_firmware_logs ┘ The two Event Hubs in the middle are queues. The Azure IoT Hub pushes into them; Telegraf and Logstash pull from them at their own pace. That decoupling is what makes the pipeline resilient: if Logstash goes down for an hour, log messages simply pile up in eh-logs and get drained when it comes back. Nothing is lost and nothing blocks the devices. Setting Up the Azure Side We will keep this section brief since the Azure resources are standard. You need: A resource group to hold everything. An Event Hubs namespace (Basic tier is fine — both consumers speak AMQP natively, so you do not need the Kafka-compatible endpoint that Standard tier adds) containing two event hubs: eh-metrics and eh-logs. Two authorization rules per event hub: one with Send rights (used by IoT Hub’s routing endpoints) and one with Listen rights (used by Telegraf and Logstash). Keeping them separate is necessary because mixing them up produces a very specific error. An IoT Hub on the B1 tier or above. The free F1 tier allows only one custom routing endpoint, and this architecture needs two. IoT Hub names are also globally unique across Azure, so pick something distinctive. Two custom endpoints on the IoT Hub, each pointing at one event hub via its Send connection string. Two message routes, which are the heart of the whole design: $ az iot hub message-route create -g <resource-group> -n <iot-hub-name> \ $ –route-name metrics-route –endpoint-name ep-metrics –source devicemessages \ $ –condition "msgType = 'metric'" $ az iot hub message-route create -g <resource-group> -n <iot-hub-name> \ $ –route-name logs-route –endpoint-name ep-logs –source devicemessages \ $ –condition "msgType = 'log'" A device identity for the simulator to authenticate as. Let’s discuss a couple of issues up front: first, IoT Hub routing cannot inspect an arbitrary message body to decide “is this JSON or plain text” — body-based routing queries only work when the message declares contentType = application/json and contentEncoding = utf-8, and plain-text bodies are opaque to the routing engine entirely. The robust pattern is what we use here: the device stamps an application property (msgType) on every message and the routes filter on that property. Second, once you add custom routes, IoT Hub’s fallback route to the built-in endpoint is disabled by default — any message that matches no route is dropped, so a typo in the property name fails silently. Third, on recent Azure CLI versions, creating an event hub on Basic tier requires passing –cleanup-policy Delete –retention-time 24 together (the CLI’s defaults request 7-day retention, which Basic rejects, while the retention flag alone trips a serialization error). Layer 1: The Device Simulator For a data source we will use a small Python script that plays the role of an IoT device. It alternates between two data streams: most of the time it sends a clean JSON metrics payload, and roughly 30% of the time it also emits a raw, syslog-style firmware error line. Crucially, it labels every message with the msgType application property that the routes key on. Install the SDK with pip install azure-iot-device, then: import json, random, time from datetime import datetime, timezone from azure.iot.device import IoTHubDeviceClient, Message CONN = "<your device connection string>" client = IoTHubDeviceClient.create_from_connection_string(CONN) ERRORS = ["E042 sensor read timeout", "E107 fw checksum mismatch", "E019 wifi rssi below threshold"] while True: m = Message(json.dumps({ "deviceId": "sensor-001", "temperature": round(random.uniform(20, 35), 2), "humidity": round(random.uniform(30, 70), 2), })) m.content_type = "application/json" m.content_encoding = "utf-8" m.custom_properties["msgType"] = "metric" client.send_message(m) # occasionally: an ugly text log, labeled for the logs route if random.random() < 0.3: line = f'{datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")} sensor-001 ERROR {random.choice(ERRORS)} ip=192.168.1.{random.randint(2,254)}' l = Message(line) l.content_type = "text/plain" l.content_encoding = "utf-8" l.custom_properties["msgType"] = "log" client.send_message(l) time.sleep(3) Get the device connection string with: $ az iot hub device-identity connection-string show -g <resource-group> -n <iot-hub-name> -d sensor-001 -o tsv Before wiring up either consumer, it is worth verifying the split with a throwaway script that reads each event hub directly (the azure-eventhub pip package makes this a ten-liner). You want to see only JSON blobs landing in eh-metrics and only text lines in eh-logs. Debugging the routing and the consumers at the same time is a miserable experience — confirm the iot hub router is doing its job first. Layer 3a: Telegraf — The Metrics Lane Telegraf is InfluxData’s metrics-collection agent. It offers hundreds of input plugins and dozens of output plugins for shipping collected metrics to a time-series store. The GridDB Telegraf plugin is one of those outputs, and conveniently, Telegraf also ships an eventhub_consumer input plugin whose primary use case is exactly this: consuming from Azure Event Hubs and IoT Hub. Building Telegraf With the GridDB Plugin The GridDB output plugin is not distributed as a prebuilt binary, it must be compiled into Telegraf from source. The v3.2 bundled pack ships the plugin source code, which you place into a Telegraf source checkout before building. $ mkdir -p ~/go/src/github.com/influxdata $ cd ~/go/src/github.com/influxdata $ git clone https://github.com/influxdata/telegraf.git $ cd telegraf # Copy in the plugin source from the v3.2 bundle $ cp -r /path/to/telegraf-output-plugin/plugins ./ This places plugins/outputs/griddb/griddb.go into the Telegraf source tree. An important caveat: having the plugin source alone is not sufficient for Telegraf to recognize it. Modern Telegraf (v1.20+) uses a build-tag registration pattern in which each plugin requires a one-line import file under plugins/outputs/all/. Without this file, the plugin compiles into the binary as dead code and Telegraf will reject your configuration with undefined but requested output: griddb. Create plugins/outputs/all/griddb.go: //go:build !custom || outputs || outputs.griddb package all import _ "github.com/influxdata/telegraf/plugins/outputs/griddb" // register plugin Then build and verify: $ make telegraf $ ./telegraf –output-list | grep griddb If griddb appears in the output list, the plugin has been registered correctly. Configuring Telegraf as a Stream Consumer In our previous walkthrough Telegraf played its traditional role of collecting local system metrics. Here it plays a different one: a stream worker that pulls device metrics off an Azure queue. The input section changes; the GridDB output section is identical to before. Create a griddb-iot.conf: $ [[inputs.eventhub_consumer]] $ ## The Listen connection string for eh-metrics, including EntityPath $ connection_string = "Endpoint=sb://<namespace>.servicebus.windows.net/;SharedAccessKeyName=consumer-listen;SharedAccessKey=<key>;EntityPath=eh-metrics" $ data_format = "json" $ json_string_fields = ["deviceId"] $ ## Use the time IoT Hub received the message as the row timestamp $ iot_hub_enqueued_time_as_ts = true $ ## This becomes the GridDB container name $ name_override = "iot_sensor_metrics" $ [[outputs.griddb]] $ api_url = "https://cloud8737.griddb.com:443/griddb/v2/gs_clustermfcloud8737/dbs/nl7QftSt" $ database = "${GRIDDB_DATABASE}" $ cluster_name = "gs_clustermfcloud8737" $ username = "${GRIDDB_USERNAME}" $ password = "${GRIDDB_PASSWORD}" $ update_mode = "append" $ containers = [] $ is_timeseries = true $ timestamp_column = "timestamp" $ [agent] $ interval = "10s" $ flush_interval = "10s" $ metric_batch_size = 1000 $ metric_buffer_limit = 10000 $ omit_hostname = true $ debug = true Adjust the api_url, database, cluster_name, and username values to match your own GridDB Cloud instance, and set is_timeseries = true so the plugin creates a TIME_SERIES container — the appropriate choice for metrics data. Two details worth calling out. omit_hostname = true matters more here than it did before: without it, Telegraf stamps every row with the hostname of the machine running Telegraf, which in an IoT pipeline is misleading — the data came from sensor-001, not from your consumer box, and the deviceId field already carries the real source. And note that eventhub_consumer is a service input: unlike ordinary polling inputs, it listens for events rather than gathering on an interval, which means the –test and –once dry-run flags from our previous walkthrough may produce no output for it. Just run it for real: $ export GRIDDB_PASSWORD='your-password' $ ./telegraf –config griddb-iot.conf With the simulator running, the debug log shows Telegraf writing batches within seconds, and an iot_sensor_metrics container appears in the GridDB Cloud portal with temperature, humidity, and deviceId columns: Layer 3b: Logstash — The Logs Lane Where Telegraf handles metrics, Logstash handles logs. It ingests unstructured text events, parses them into structured fields with its grok filter, and ships them to a destination of your choice. Our destination is GridDB Cloud, and our source is the eh-logs queue full of raw firmware error lines. Installing Logstash The bundle’s README points to a yum-based CentOS install path. On macOS we will download the tarball directly from Elastic — Homebrew’s elastic/tap formula is currently broken on recent Homebrew versions. $ mkdir -p ~/logstash-demo && cd ~/logstash-demo $ curl -O https://artifacts.elastic.co/downloads/logstash/logstash-9.4.1-darwin-aarch64.tar.gz $ tar -xzf logstash-9.4.1-darwin-aarch64.tar.gz $ mv logstash-9.4.1 logstash $ ./logstash/bin/logstash –version Installing the GridDB Output Plugin The plugin ships as a prebuilt .gem, so no Ruby build step is required. Copy it alongside the Logstash install. Note that the leading ./ matters — without it, logstash-plugin treats the argument as a remote plugin name and constructs an invalid URL: $ cp /path/to/logstash-output-plugin/logstash-output-griddb-1.0.0.gem ~/logstash-demo/ $ cd ~/logstash-demo $ ./logstash/bin/logstash-plugin install ./logstash-output-griddb-1.0.0.gem $ ./logstash/bin/logstash-plugin list | grep griddb The input side needs no installation at all: the azure_event_hubs input plugin ships bundled with Logstash. The Logstash Config Save the following as ~/logstash-demo/iot-logs-to-griddb.conf: input { azure_event_hubs { event_hub_connections => ["Endpoint=sb://<namespace>.servicebus.windows.net/;SharedAccessKeyName=consumer-listen;SharedAccessKey=<key>;EntityPath=eh-logs"] initial_position => "beginning" } } filter { grok { match => { "message" => '%{TIMESTAMP_ISO8601:log_ts} %{NOTSPACE:device_id} %{LOGLEVEL:level} %{NOTSPACE:error_code} %{DATA:error_msg} ip=%{IP:device_ip}' } } date { match => [ "log_ts", "ISO8601" ] target => "@timestamp" } mutate { remove_field => [ "message", "log_ts", "event", "log", "@version", "host" ] } } output { stdout { codec => rubydebug } griddb { host => "https://cloud8737.griddb.com:443" cluster => "your-cluster" database => "your-database" container => "iot_firmware_logs" username => "your-username" password => "${GRIDDB_PASSWORD}" insert_mode => "append" } } The grok filter does the core work: it takes a raw line like 2026-07-16T23:28:33Z sensor-001 ERROR E107 fw checksum mismatch ip=192.168.1.99 and breaks it into device_id, level, error_code, error_msg, and device_ip fields. The date filter takes the timestamp embedded inside the log line — the moment the device actually recorded the error — and uses it as the event’s real timestamp, rather than the moment Logstash happened to process it. The connection string deserves special attention, because it bit us twice while building this. It must be the complete string — starting with Endpoint=sb:// and ending with EntityPath=eh-logs — or the plugin fails at startup with Error parsing event hub string name for connection. And it must use the Listen authorization rule, not the Send rule you gave to IoT Hub’s routing endpoints. If you paste the Send string, Logstash connects successfully, discovers the partitions, and then fails on every receive with Unauthorized access. ‘Listen’ claim(s) are required — an error that is easy to miss in the AMQP log spam. You will also see a startup warning that no storage_connection_string is configured. Logstash uses an Azure Storage account to checkpoint its position across restarts and to coordinate multiple Logstash instances; for a single-instance demo it is safe to ignore, at the cost of re-reading the queue from the beginning on each restart. Running It $ cd ~/logstash-demo $ export GRIDDB_PASSWORD='your-password' $ ./logstash/bin/logstash -f iot-logs-to-griddb.conf Startup takes 20–30 seconds. Because of initial_position => “beginning”, Logstash immediately drains everything sitting in eh-logs, so you should see a burst of parsed events scroll past in the rubydebug output, then a steady trickle as the simulator keeps emitting. Each one becomes a row in the iot_firmware_logs container: The resulting schema is clean: one self-documenting column per parsed field, and timestamps that reflect when each error actually occurred on the device. What Lands in GridDB Cloud At this point the full pipeline is live. Two containers, each shaped by the tool that filled it: iot_sensor_metrics — a TIME_SERIES container with one row per metrics message: timestamp, deviceId, temperature, humidity. Created and populated by Telegraf. iot_firmware_logs — one row per parsed firmware error: timestamp, device_id, level, error_code, error_msg, device_ip. Created and populated by Logstash. The same device produced both, the same IoT Hub received both, and the same database stores both — but each signal traveled its own lane and was processed by the tool built for it. A note on topology: in this walkthrough, one machine plays every role — it runs the simulator, Telegraf, and Logstash, so the data flows out to Azure and right back. That looks redundant locally, but it is actually the proof that the architecture is decoupled: no component knows where the others run, because they only ever talk to Azure endpoints. In production, the simulator becomes a fleet of real devices in the field (each with its own IoT Hub device identity), and Telegraf and Logstash become small VMs or containers colocated in the same Azure region as the Event Hubs namespace. Everything in between stays exactly the same. Extra: Grafana Dashboards Over GridDB Cloud The pipeline above is complete on its own, but if you want to visualize what you just ingested, GridDB also ships a Grafana data source plugin in the v3.2 bundle. The installation is somewhat involved, so we are including it here. Important: this plugin is built on AngularJS, which Grafana deprecated in v11 and fully removed in v12. There is no flag or workaround on modern Grafana, as the framework is no longer present. For now, you need Grafana 10.x. You can read more in Grafana’s removal announcement. Installing Grafana 10 Download the last 10.x release directly from Grafana’s download page: $ mkdir -p ~/grafana-demo && cd ~/grafana-demo $ curl -O https://dl.grafana.com/oss/release/grafana-10.4.15.darwin-arm64.tar.gz $ tar -xzf grafana-10.4.15.darwin-arm64.tar.gz $ mv grafana-v10.4.15 grafana Enabling the Plugin Two configuration changes are needed before the plugin will load. Create ~/grafana-demo/grafana/conf/custom.ini (Grafana automatically merges this with the defaults): [plugins] allow_loading_unsigned_plugins = griddb-datasource [security] angular_support_enabled = true The first setting whitelists the unsigned plugin; the second enables the AngularJS compatibility mode that Grafana 10 still provides. Copy the plugin into Grafana’s plugins directory: $ mkdir -p ~/grafana-demo/grafana/data/plugins/griddb-datasource $ cp -r /path/to/grafana-input-plugin/dist/* \ $ ~/grafana-demo/grafana/data/plugins/griddb-datasource/ Start Grafana: $ cd ~/grafana-demo/grafana $ ./bin/grafana server Open http://localhost:3000, log in (admin/admin by default; you will be prompted to set a new password), and add a new GridDB data source under Connections → Data sources. Fill in: Host: https://cloud8737.griddb.com:443 (no trailing path — the plugin appends /griddb/v2/… itself) Cluster: your cluster name Database: your database name User / Password: your GridDB Cloud credentials A Note on the Password Field Once the form is complete, click Save & Test. If the test fails with a TXN_AUTH_FAILED error even though your credentials are correct, the data source form may not have persisted the password to Grafana’s secure storage (this occurred consistently in our testing). You can confirm by inspecting Grafana’s sqlite store: $ sqlite3 ~/grafana-demo/grafana/data/grafana.db \ $ "SELECT name, basic_auth_user, length(secure_json_data) FROM data_source;" If length(secure_json_data) is 2, the password did not save ({} is two characters). The workaround is to set the password via Grafana’s HTTP API, which writes the credentials properly: $ curl -X PUT \ $ -u 'admin:YOUR_GRAFANA_ADMIN_PASSWORD' \ $ -H "Content-Type: application/json" \ $ http://localhost:3000/api/datasources/1 \ $ -d '{ $ "id": 1, $ "name": "griddb-datasource", $ "type": "griddb-datasource", $ "url": "https://cloud8737.griddb.com:443", $ "access": "proxy", $ "basicAuth": true, $ "basicAuthUser": "your-griddb-username", $ "secureJsonData": { "basicAuthPassword": "your-griddb-password" }, $ "jsonData": { $ "xgridcluster": "your-cluster-name", $ "xgriddatabase": "your-database-name", $ "minInterval": "1s" $ } $ }' (Single-quote the -u argument if your Grafana admin password contains shell metacharacters.) After that, secure_json_data will contain a long encrypted blob, and the data source will authenticate cleanly. Querying Your Data With the data source connected, build a panel using the plugin’s query syntax against the containers this pipeline created: $griddb_query_data(iot_sensor_metrics, temperature, select * order by timestamp) The three arguments are the container, the columns to select, and a TQL clause. You can also use $griddb_container_list to populate template variables for a container picker, or $griddb_column_list({container}) to drive column dropdowns. Wrapping Up Four layers, each doing one job: The edge produces a mixed stream of metrics and logs, each message labeled with what it is. Azure IoT Hub routes on that label, splitting the firehose into two clean queues. Telegraf and Logstash each consume the lane they are built for — JSON parsing on one side, grok on the other. GridDB Cloud stores both signals in purpose-built containers, side by side and queryable together. Compared to our Kafka-based OpenTelemetry pipeline, the striking thing about this one is what is missing: there is no bridge, no custom flattening code, no bespoke glue. IoT Hub’s routing rules replace the splitting logic, and the official eventhub_consumer and azure_event_hubs input plugins replace the consuming logic. The GridDB Cloud v3.2 bundled plugin pack supplies both output plugins — one download, and every stage of a cross-cloud IoT pipeline is configuration rather than

Introduction GridDB combines the horizontal scalability of a distributed key-value store with the queryability of a relational database, making it ideal for automation workloads. In this guide, we show how to pair GridDB with n8n to create a simple Pokémon API service with minimal code. We’ll set up a local n8n environment, connect it to GridDB, and build a reusable workflow so you can adapt it for another application. Whether you’re experimenting with AI-assisted operations or building production-ready automations, the steps below will help you get started quickly and confidently. What will we build? In this project, we will build a Pokémon API service using n8n automation. Why n8n? n8n is a workflow automation platform that provides technical teams with the flexibility of code and the speed of no-code. Its core is source-available under the Sustainable Use License, which allows you to view, use, and modify the code, but it also imposes business restrictions. However, for our use case, the community edition is more than enough for development. How to run the project You need n8n installed on your system. Please look into this section for the n8n installation. 1. Download the n8n workflow from the repository The n8n workflow file can be downloaded from here. 2. Import the workflow into the n8n dashboard Open the n8n workspace, create a new workflow, and then import the downloaded n8n workflow file. 3. Set up the GridDB credentials In n8n community edition, you can use the Credentials feature, but it’s still not supported base64 cred encoding, so in our project, we need to set the credentials manually for each HTTP Request. Here are the n8n nodes you need to set: GridDB Check Connection Check Containers Create Container Get All Data Insert/Update/Delete Data Get Data by ID Double-click the node and then replace the value after Basic with your base64 encoding of username and password. In Mac or Linux, you can use this command to encode username and password using base64 encoding: $ echo -n 'username:password' | base64 Other than credentials, you also need to change the GridDB Cloud URL in each HTTP Request nodes. To test the workflow, you can run the tests command from this section. Prerequisites Node.js The project sample in this article is using Node.js. Make sure to install the latest Node.js LTS version on your machine. GridDB Sign Up for GridDB Cloud Free Plan If you would like to sign up for a GridDB Cloud Free instance, you can do so at the following link: https://form.ict-toshiba.jp/download_form_griddb_cloud_freeplan_e. After successfully signing up, you will receive a free instance along with the necessary details to access the GridDB Cloud Management GUI, including the GridDB Cloud Portal URL, Contract ID, Login, and Password. GridDB WebAPI URL Go to the GridDB Cloud Portal and copy the WebAPI URL from the Clusters section. It should look like this: GridDB Username and Password Go to the GridDB Users section of the GridDB Cloud portal and create or copy the username. The password is set when the user is created for the first time. Use this as the password. For more details, to get started with GridDB Cloud, please follow this quick start guide. IP Whitelist When running this project, please ensure that the IP address where the project is running is whitelisted. Failure to do so will result in a 403 status code or forbidden access. You can use a website like What Is My IP Address to find your public IP address. To whitelist the IP, go to the GridDB Cloud Admin and navigate to the Network Access menu. Install n8n on Local Machine You need to run an n8n instance to try the automation workflow. The n8n can be hosted on the cloud or locally. For this article, we will install the n8n community edition locally. To install it, follow these steps: 1. Clone the source code Go to the n8n GitHub repository and then clone it. Currently, the latest release version of n8n is 1.113.3. $ git clone –branch 1.113.3 https://github.com/n8n-io/n8n.git 2. Install deps The n8n basically is a runnable npm package, and it needs pnpm to build and run. $ cd n8n $ pnpm install 3. Build and run It’s better to build the n8n first so you will run much faster if you want to run it again at other times. pnpm build pnpm run dev Once the n8n is running, by default, you can access it in the URL http://localhost:5678. If you want to deploy it to the cloud and expose it to the public, there are a few environment variable settings that will override the n8n default settings. Please read their official documentation for deployment for more information. Project Before we dig deeper into the app project, it’s valuable to know what main operations we will use in this n8n automation related to access to the GridDB database. Setup GridDB Connection This n8n workflow will check the connection to the GridDB database. It will respond with a 200 HTTP status code if the connection is successful. The Webhook node is the entry node for automation. In this node, we can set the HTTP operation and authentication. It also gives us a test and production address where the client can access it publicly. For our project, this will be local. For example: $ http://localhost:5678/webhook/35f92470-024d-49da-8c2f-63a2f212e639 The GridDB Check Connection node basically is a GET HTTP call to the GridDB Cloud, another form of this curl command: $ curl -i –location –request GET 'https://cloud5197.griddb.com:443/griddb/v2/cluster-name/dbs/db-name/checkConnection' –header 'Authorization: Basic base64(username:password)' For real production, replace the GridDB cloud address and make sure to encode the username and password into the base64 encoding. Check containers The basic curl command to check the existing containers in GridDB is as follows: $ curl -i –location –request GET 'https://cloud5197.griddb.com:443/griddb/v2/cluster-name/dbs/db-name/containers?limit=100' –header 'Authorization:Basic base64(username:password)' –header 'Content-Type: application/json The above command can be converted to n8n automation very easily by using the HTTP Request node (using the import cURL button). Create containers To create a new container, we can use curl directly as long as we have the right credentials. Ok, now let’s create a container named “pokemon”: $ curl -i –location –request POST \ $ 'https://cloud5197.griddb.com:443/griddb/v2/cluster-name/dbs/db-name/containers' \ $ –header 'Authorization: Basic base64(username:password)' \ $ –header 'Content-Type: application/json' \ $ –data '{ $ "container_name": "pokemon", $ "container_type": "COLLECTION", $ "rowkey": true, $ "columns": [ $ {"name": "id", "type": "INTEGER", "index": []}, $ {"name": "name", "type": "STRING", "index": []}, $ {"name": "skills", "type": "STRING", "index": []}, $ {"name": "level", "type": "STRING", "index": []} $ ] $ }' In the n8n automation workflow, we can put the node after checking the existing container, and if it doesn’t exist, then we can create it. Get all data To get data from a known container, we can use this curl command to get all data: $ curl -i –location –request POST 'https://cloud5197.griddb.com:443/griddb/v2/cluster-name/dbs/db-name/tql/' –header 'Authorization: Basic base64(username:password)' –header 'Content-Type: application/json' –data '[{"name":"pokemon","stmt":"select * limit 10","columns":null,"hasPartialExecution":true}]' The command uses TQL to select all data from GridDB pokemon container. As with the other operations, the read data can also be easily implemented in the n8n using the HTTP Request node. Get Specific Data To get specific data from the GridDB database, for example, by its ID, you can use this curl command: $ curl -i –location –request POST \ $ 'https://cloud5197.griddb.com:443/griddb/v2/cluster-name/dbs/db-name/tql/' \ $ –header 'Authorization: Basic base64(username:password)' \ $ –header 'Content-Type: application/json' \ $ –data '[{"name":"pokemon","stmt":"select * where id = 4167 limit 100","columns":null,"hasPartialExecution":true}]' In n8n, the curl can also be easily converted into the n8n node using the HTTP Request node. Insert, Update, and Delete data There are a few ways to insert data in GridDB using web API endpoints: /sql/dml/update: accepts any SQL DML (INSERT/UPDATE/DELETE). /tql/: processes TQL statements, including PUT (…) for row upserts and SELECT queries. /containers//rows: writes raw row arrays directly. Ideal for bulk inserts, and you must send values in column order. We will use the /sql/dml/update in this n8n automation because we will support 3 data operations, which are: INSERT, UPDATE, and DELETE. Insert $ curl -i –location –request POST 'https://cloud5197.griddb.com:443/griddb/v2/cluster-name/dbs/db-name/sql/dml/update' –header 'Authorization: Basic base64(username:password)' –header 'Content-Type: application/json' –data "[{\"stmt\":\"INSERT INTO pokemon(id, name, skills, level) VALUES (26, 'Charmander', 'Flamethrower,Dragon Breath', 'Starter')\"}]" Update $ curl -i –location –request POST 'https://cloud5197.griddb.com:443/griddb/v2/cluster-name/dbs/db-name/sql/dml/update' –header 'Authorization: Basic base64(username:password)' –header 'Content-Type: application/json' –data "[{\"stmt\":\"UPDATE pokemon SET level = 'Master' WHERE id = 25\"}]" Delete $ curl -i –location –request POST 'https://cloud5197.griddb.com:443/griddb/v2/cluster-name/dbs/db-name/sql/dml/update' –header 'Authorization: Basic base64(username:password)' –header 'Content-Type: application/json' –data "[{\"stmt\":\"DELETE FROM pokemon WHERE id = 26\"}]" From all the curl commands above, each can be easily converted into the n8n node using the HTTP Request node, or we can use just one n8n node and make the stmt a variable. Shouldn’t there be three nodes? It’s not necessary because the only changing part in the command is the SQL statement, and the other parts are pretty much the same. So, for the user to be able to insert, update, or delete data, the user or client needs to send a SQL statement in the payload data. Full n8n Workflow The full basic n8n workflow for our Pokémon app service can be accessed by using the webhook node. This webhook is basically an exposed or public URL that can be used by the client, and you also need to activate the workflow by clicking the Active toggle menu (top right). Data Payload Any user or client that uses n8n needs to send data with either this payload format: $ { $ "container": "container_name", $ "operation": "data_operation", $ "id": id_integer, $ "statement": "SQLStatement" $ } Fields: For the get_all_data operation, the required fields are container and operation. For the get_data_by_id operation, the required fields are container, operation, and id. For the insert, update, and delete operations, you need to add a TQL statement. To grasp how the workflow works with data, please look into the tests below. Workflow Tests To test the n8n workflow, you can also use the curl command or other tools such as Postman. In this blog post, we use curl for portability and this webhook URL: $ http://localhost:5678/webhook/35f92470-024d-49da-8c2f-63a2f212e639 Get All Data Test Use this command to get all data: $ curl -i –location –request POST 'http://localhost:5678/webhook/35f92470-024d-49da-8c2f-63a2f212e639' –header 'Content-Type: application/json' –data "{\"container\":\"pokemon\",\"operation\":\"get_all_data\"}" Get Data by Id Test To get specific data using its ID: $ curl -i –location –request POST 'http://localhost:5678/webhook/35f92470-024d-49da-8c2f-63a2f212e639' –header 'Content-Type: application/json' –data "{\"container\":\"pokemon\", \"operation\":\"get_data_by_id\",\"id\":25}" Insert Data Test To insert data: $ curl -i –location –request POST 'http://localhost:5678/webhook/35f92470-024d-49da-8c2f-63a2f212e639' –header 'Content-Type: application/json' –data "{\"container\":\"pokemon\",\"operation\":\"insert\",\"statement\":\"INSERT INTO pokemon(id, name, skills, level) VALUES (26, 'Charmander', 'Flamethrower,Dragon Breath', 'Starter')\"}" Update Data Test To update existing data, for example, a data with the id is 25: $ curl -i –location –request POST 'http://localhost:5678/webhook/35f92470-024d-49da-8c2f-63a2f212e639' –header 'Content-Type: application/json' –data "{\"container\":\"pokemon\",\"operation\":\"update\",\"id\":25,\"statement\":\"UPDATE pokemon SET level = 'Master' WHERE id = 25\"}" Delete Data Test To delete the data, use this curl command: $ curl -i –location –request POST 'http://localhost:5678/webhook/35f92470-024d-49da-8c2f-63a2f212e639' –header 'Content-Type: application/json' –data "{\"container\":\"pokemon\", \"operation\":\"delete\",\"id\":26,\"statement\":\"DELETE FROM pokemon WHERE id = 26\"} Logs n8n also provides Logs for every workflow execution. If something goes wrong, this is the best place to check which node is causing the

> Medical Disclaimer: All health metric estimations in this project (blood pressure, blood sugar) are simplified rule-based simulations for demonstration and educational purposes only. They are not clinically accurate and should never be used for real medical decision-making. This blog is the longer version of the hackathon entry from VitalWatch. You can see their original submission here: https://gallery.griddb-linux-griddb-linux-clone-gnbah8a6f2fwd5eb.westus-01.azurewebsites.net/projects/vitalwatch.md Introduction In healthcare, early warning often means the difference between a manageable situation and a critical emergency. Wearable devices like smartwatches and fitness bands now collect continuous streams of biometric data — heart rate, blood oxygen, temperature, and activity level — all day long. While there is no shortage of data, most systems fail by evaluating each vital sign in isolation. They flag a high heart rate or a low SpO2 reading independently, without asking the obvious follow-up question: are these two changes happening at the same time, and in a physiologically connected way? A standalone elevated heart rate reading might mean the wearer just walked upstairs. But if heart rate rises and SpO2 simultaneously drops and body temperature climbs — that is a physiological chain event, not a coincidence. Most dashboards miss that story. VitalWatch addresses exactly this gap. Rather than building another static alert panel, we built a Wearable Health Intelligence Layer on top of GridDB Cloud that stores continuous vital sign streams, detects physiological chain events across multiple vital signs, and estimates derived health metrics like blood pressure and blood sugar from the raw sensor data. Why Time-Series Databases Matter for Wearable Health IoT A standalone SpO2 reading of 91% doesn’t tell the whole story. If it dropped from 98% in twenty minutes, it could indicate respiratory distress; if it has safely fluctuated between 90-92% for hours, it might just be a resting artifact. Evaluating sequences of readings over time provides the necessary context to tell the difference. Wearable devices generate this time-series data continuously — potentially dozens of readings per minute across multiple sensors. Traditional relational databases, optimized for structured record storage and joins, were not designed for the high-frequency append-heavy workloads that wearable telemetry produces. They can struggle with the combination of write volume, timestamped queries, and time-window aggregations that health monitoring requires. Time-series databases are purpose-built for this pattern. They act like a medical chart that never runs out of paper, allowing applications to look back at recent vital sign history and understand whether a wearer is moving toward an unsafe physiological state. Why GridDB Cloud for Wearable Health Monitoring GridDB Cloud provides an environment where sensor telemetry storage and retrieval are handled efficiently, letting the monitoring application focus entirely on health analysis rather than database infrastructure management. GridDB is a highly scalable, memory-first NoSQL database designed for high-frequency time-series IoT workloads. For VitalWatch, we use GridDB Cloud to: Store continuous wearable sensor data in TIME_SERIES containers with timestamp-based row keys. Maintain a fixed per-wearer schema for predictable, fast reads and writes. Ingest vital sign data in memory first and persist it safely to disk. Query recent time windows efficiently for early-warning detection. Correlate multiple vital sign streams in real time for chain event detection. Replay sensor history before an alert for physiological incident analysis. Provide the historical dataset that trains the machine learning health risk classifier. Without GridDB: Real-time multi-vital correlation would require expensive joins across separate tables. High-frequency wearable ingestion would create write bottlenecks. ML training on historical health data would require a separate ETL pipeline. System Architecture VitalWatch simulates a small health monitoring scenario for three wearers — an Athlete, an Elderly Patient, and an Office Worker — enrolled in a group monitoring program. Each wearer carries a simulated smartwatch that generates Heart Rate, SpO2, Body Temperature, and Activity Level readings. Derived health metrics (estimated blood pressure and blood sugar) are computed from the raw readings before storage. The overall system consists of the following main components: Sensor Simulation Layer A Python-based simulator generates realistic vital sign readings for each wearer. It applies a three-phase simulation model: Normal (all vitals safe), Stress (athlete begins showing cardiovascular signs), and Deterioration (athlete reaches critical levels while other wearers show secondary stress signals). Health Estimation Layer A rule-based estimator computes derived metrics — systolic/diastolic blood pressure and blood glucose — from the raw sensor values. These estimates are included in every row stored in GridDB so that the dashboard and ML model have access to the full health picture. Data Ingestion Layer The backend application receives simulated vital sign data and writes it to GridDB Cloud using the multi_put batch API, which allows rows for multiple wearer containers to be written in a single network round-trip. GridDB Cloud Database GridDB stores the complete vital sign history for each wearer in dedicated TIME_SERIES containers. The monitoring application queries this data to evaluate health conditions and detect physiological chain events. Monitoring Dashboard A web-based dashboard displays real-time vital signs, physiological chain alerts, and the fleet-wide health risk score. It polls the Flask API every three seconds and visualizes live trends using Chart.js. ML Health Risk Classifier A scikit-learn Random Forest classifier is trained directly on the historical data stored in GridDB. It learns to predict one of four health risk levels (Normal, Stress, Distress, Critical) from the seven-dimensional feature vector of each reading. Setting Up GridDB Cloud To store wearable telemetry, a GridDB Cloud instance can be deployed through the Microsoft Azure Marketplace. After subscribing, you receive the cluster connection details including the notification provider address, cluster name, and authentication credentials. Using the GridDB Python client, applications connect to the cluster through the native API. The following example shows how VitalWatch establishes a per-thread GridDB connection to avoid concurrency errors when multiple Flask worker threads are active: import griddb_python as griddb import threading _local = threading.local() def get_store(): """Return a per-thread GridDB connection to avoid concurrent access errors.""" if not hasattr(_local, "store") or _local.store is None: factory = griddb.StoreFactory.get_instance() _local.store = factory.get_store( notification_member=NOTIFICATION_MEMBER, cluster_name=CLUSTER_NAME, username=USERNAME, password=PASSWORD, ) return _local.store Connection credentials are read from environment variables so that no secrets appear in source code. Project Overview VitalWatch builds a smart health monitoring system for three simulated wearers enrolled in a group monitoring program. The system tracks each wearer’s vitals continuously and identifies early signs of physiological stress before they become critical events. Key ideas behind the system: Multi-vital monitoring: Rather than evaluating each vital sign independently, the system looks for correlated changes that indicate a linked physiological event. Physiological chain detection: It recognizes known biological relationships — for example, that falling SpO2 typically triggers compensatory heart rate elevation — and fires chain alerts when both sides of the correlation appear simultaneously. Profile-aware thresholds: Each wearer has individually tuned normal ranges and alert thresholds. An athlete’s resting heart rate of 55 bpm is healthy; the same reading might warrant attention for a sedentary office worker. Continuous risk scoring: Instead of a binary safe/unsafe flag, each wearer receives a 0–100 risk score that moves smoothly as vitals drift toward danger zones. Simulating Wearable Sensor Data Since real wearable devices are not available, the system uses a Python sensor simulator to generate realistic vital sign streams. The simulator creates readings for three wearer profiles: Athlete: Heart rate, SpO2, body temperature, activity level (trained physiology, lower resting HR) Elderly Patient: Heart rate, SpO2, body temperature, activity level (lower critical thresholds, highest monitoring priority) Office Worker: Heart rate, SpO2, body temperature, activity level (standard adult thresholds, sedentary baseline) To model a realistic health event, the dataset is generated across three phases: Normal: All wearers operating within their safe vital ranges. Stress: The athlete begins showing cardiovascular stress — heart rate rises, SpO2 drops. Deterioration: The athlete reaches critical levels; the elderly patient begins showing secondary stress; the office worker shows early signs of physiological disturbance. This models how a shared environmental trigger (extreme heat, altitude) affects multiple wearers simultaneously. To make the simulation physically plausible rather than generating flat hardcoded values, the system uses proportional interpolation with environmental noise: def _interpolate(low: float, high: float, progress: float = None) -> float: """Smoothly interpolate between two boundary values with optional random progress.""" if progress is None: progress = random.uniform(0.0, 1.0) return low + progress * (high – low) def generate_reading(wearer_id, status="Normal", progress=0.0, timestamp=None): nrm = WEARERS[wearer_id]["normal"] thr = WEARERS[wearer_id]["thresholds"] if status == "Stress": # HR ramps upward; SpO2 begins to drop (inverted: lower = worse) hr_base = _interpolate(nrm["heart_rate"][1], thr["heart_rate"]["warning"], progress) spo2_base = _interpolate(nrm["spo2"][0], thr["spo2"]["warning"], progress) t_base = _interpolate(nrm["temperature"][1], thr["temperature"]["warning"], progress * 0.6) # Environmental noise added to every reading regardless of phase noise_hr = random.uniform(-2.0, 2.0) noise_spo2 = random.uniform(-0.3, 0.3) noise_temp = random.uniform(-0.1, 0.1) This approach ensures that simulated vitals drift naturally toward thresholds rather than jumping abruptly, making the dashboard transitions visually and physiologically meaningful. Health Metric Estimation One of VitalWatch’s key additions is a rule-based health estimator that derives secondary health metrics from the raw wearable sensor data. Since wearable devices primarily measure heart rate, SpO2, temperature, and motion, higher-order metrics like blood pressure and blood glucose must be inferred. The estimations use simplified physiological relationships that are deliberately documented as approximations: def estimate_systolic_bp(heart_rate: float, spo2: float, activity_level: float) -> float: """ Estimate systolic blood pressure (mmHg) from HR, SpO2, and activity. NOTE: Simplified demonstration formula — not clinically accurate. """ baseline = 120.0 hr_delta = (heart_rate – 72) * 0.50 # HR deviation from resting reference activity_adj = (activity_level / 10.0) * 4.0 spo2_stress = max(0.0, (96.0 – spo2) * 0.80) # hypoxic stress contribution noise = random.uniform(-3.0, 3.0) return round(baseline + hr_delta + activity_adj + spo2_stress + noise, 1) def estimate_blood_sugar(heart_rate: float, activity_level: float, spo2: float) -> float: """ Estimate blood glucose (mg/dL) from HR, activity, and SpO2. NOTE: Simplified demonstration formula — not clinically accurate. """ baseline = 90.0 hr_delta = (heart_rate – 70) * 0.30 activity_adj = activity_level * 0.40 spo2_stress = max(0.0, (96.0 – spo2) * 1.50) noise = random.uniform(-5.0, 5.0) return round(baseline + hr_delta + activity_adj + spo2_stress + noise, 1) A single derive_health_metrics() function acts as the entry point used by the sensor simulator so that every row stored in GridDB automatically includes the full estimated health picture: def derive_health_metrics(heart_rate, spo2, temperature, activity_level) -> dict: """Given raw vitals, return all derived health metric estimates.""" return { "systolic_bp": estimate_systolic_bp(heart_rate, spo2, activity_level), "diastolic_bp": estimate_diastolic_bp(heart_rate, activity_level), "blood_sugar": estimate_blood_sugar(heart_rate, activity_level, spo2), } Live Alert Simulation To demonstrate how a health deterioration event develops over time, the project includes a live alert simulation script. Rather than inserting a static pre-built dataset, the script gradually writes escalating vital sign readings into GridDB Cloud every second, mimicking how a real health event would evolve. The simulation follows three stages: Normal Baseline – All wearers within safe ranges for the first few seconds. Athlete Stress – The athlete’s heart rate climbs and SpO2 begins to drop, proportionally ramped using interpolation. Full Deterioration – The athlete reaches critical levels; the elderly patient shows secondary stress; the office worker starts showing early disturbance signals. A simplified view of the simulation loop is shown below: def trigger_alert(): """Simulates a live health event that persists until resolved on the dashboard.""" store = insert_data.get_gridstore() start_sim() # signals the Flask API that simulation is active i = 0 while is_sim_active(): ts = datetime.now(timezone.utc) batch = {} for wearer_id in WEARERS: # Determine target health state based on simulation phase target = determine_phase(i, wearer_id) row = make_row(ts, wearer_id, target) batch[WEARERS[wearer_id]["container"]] = [row] store.multi_put(batch) i += 1 time.sleep(1) When the dashboard operator clicks “Alert Addressed”, the Flask API clears the simulation flag. The script detects this, injects a final batch of normal-range readings into GridDB Cloud, and exits. Those healthy readings immediately restore the dashboard to a stable state without requiring a manual reset. Storing Health Data in GridDB Cloud After generating the simulated vital sign readings, the data is stored in GridDB Cloud so the monitoring system can access recent history. Each wearer is assigned their own TIME_SERIES container with a fixed schema that stores all raw and derived health metrics. The timestamp acts as the primary key, allowing efficient time-ordered storage and retrieval of each wearer’s health history. Creating a TIME_SERIES container per wearer: def setup_containers(store) -> dict: """Create one TIME_SERIES container per wearer profile.""" containers = {} for wearer_id, cfg in WEARERS.items(): con_info = griddb.ContainerInfo( cfg["container"], [ ["timestamp", griddb.Type.TIMESTAMP], ["heart_rate", griddb.Type.DOUBLE], ["spo2", griddb.Type.DOUBLE], ["temperature", griddb.Type.DOUBLE], ["activity_level", griddb.Type.DOUBLE], ["systolic_bp", griddb.Type.DOUBLE], # estimated ["diastolic_bp", griddb.Type.DOUBLE], # estimated ["blood_sugar", griddb.Type.DOUBLE], # estimated ], griddb.ContainerType.TIME_SERIES, ) containers[wearer_id] = store.put_container(con_info) return containers Inserting a full dataset using multi_put: def insert_dataset(store, dataset: dict) -> None: """Bulk-insert all wearer readings in a single multi_put call.""" batch = {} for wearer_id, readings in dataset.items(): container_name = WEARERS[wearer_id]["container"] rows = [] for r in readings: ts = datetime.fromisoformat(r["timestamp"].replace("Z", "+00:00")) rows.append([ts, r["heart_rate"], r["spo2"], r["temperature"], r["activity_level"], r["systolic_bp"], r["diastolic_bp"], r["blood_sugar"]]) batch[container_name] = rows store.multi_put(batch) The multi_put operation writes rows for all three wearer containers in a single request, significantly improving ingestion efficiency when handling continuous vital sign streams. Querying Health Data and Detecting Pre-Alert Conditions Once vital sign data is stored in GridDB Cloud, the monitoring system queries recent readings to evaluate each wearer’s health condition. The system retrieves the latest records from each wearer container using GridDB’s Time-Series Query Language (TQL). def query_recent(store, wearer_id: str, limit: int = 20) -> list: """Fetch the most recent readings for a wearer using GridDB TQL.""" container = store.get_container(WEARERS[wearer_id]["container"]) query = container.query(f"select * order by timestamp desc limit {limit}") rs = query.fetch() readings = [] while rs.has_next(): row = rs.next() readings.append({ "timestamp": row[0].isoformat(), "heart_rate": row[1], "spo2": row[2], "temperature": row[3], "activity_level": row[4], "systolic_bp": row[5], "diastolic_bp": row[6], "blood_sugar": row[7], }) return readings To avoid false alerts caused by momentary sensor spikes, the system evaluates health conditions using a rolling window of the three most recent readings. By averaging the latest values, the monitoring logic becomes stable and resistant to transient noise: window = readings[:3] avg_hr = sum(r["heart_rate"] for r in window) / len(window) avg_spo2 = sum(r["spo2"] for r in window) / len(window) avg_temp = sum(r["temperature"] for r in window) / len(window) avg_act = sum(r["activity_level"] for r in window) / len(window) A key implementation detail is the handling of SpO2 as an inverted vital: unlike heart rate or temperature where higher values indicate danger, a lower SpO2 value indicates physiological risk. The severity function handles both directions using an inverted flag in the threshold configuration: def vital_severity(value: float, vital_key: str, wearer_id: str) -> float: """Return a 0.0–1.5 severity score. Handles both normal and inverted vitals.""" thr = WEARERS[wearer_id]["thresholds"][vital_key] inverted = thr.get("inverted", False) if inverted: # SpO2: lower value = higher severity normal_safe = WEARERS[wearer_id]["normal"][vital_key][0] if value >= normal_safe: return 0.0 elif value >= thr["warning"]: return 0.30 * (normal_safe – value) / (normal_safe – thr["warning"]) elif value >= thr["critical"]: return 0.30 + 0.70 * (thr["warning"] – value) / (thr["warning"] – thr["critical"]) else: overshoot = (thr["critical"] – value) / max(thr["warning"] – thr["critical"], 0.1) return min(1.0 + overshoot * 0.5, 1.50) else: # HR, temperature, activity: higher value = higher severity … Risk Scoring Instead of a simple status label, each wearer receives a continuous risk score between 0 and 100 based on exactly how far their vital signs have drifted from their individual safe ranges. For example, an athlete with a heart rate of 160 bpm and an elderly patient both showing the same reading would receive very different risk scores, because their normal ranges and thresholds are configured independently. def wearer_risk_score(wearer_id, avg_hr, avg_spo2, avg_temp, avg_act) -> int: """Calculate a 0–100 health risk score using weighted vital severities.""" s_hr = vital_severity(avg_hr, "heart_rate", wearer_id) s_spo2 = vital_severity(avg_spo2, "spo2", wearer_id) s_temp = vital_severity(avg_temp, "temperature", wearer_id) s_act = vital_severity(avg_act, "activity_level", wearer_id) weighted = ( s_hr * VITAL_WEIGHTS["heart_rate"] + # 35% s_spo2 * VITAL_WEIGHTS["spo2"] + # 40% ← SpO2 carries most weight s_temp * VITAL_WEIGHTS["temperature"] + # 15% s_act * VITAL_WEIGHTS["activity_level"] # 10% ) return min(round(weighted * 100), 100) The fleet-wide risk score is a weighted average across all wearers — the elderly patient contributes 50% of the weight because they are the highest-priority monitoring subject. Active physiological chain events add extra penalty points on top. Physiological Chain Detection VitalWatch understands that vital signs are physiologically linked. A falling SpO2 causes the cardiovascular system to compensate by increasing heart rate. Sustained high heart rate generates excess metabolic heat, raising body temperature. These relationships are encoded as rules: VITAL_CHAIN_RULES = [ { "source": "spo2", "target": "heart_rate", "message": ( "Oxygen-Cardiac Chain: Falling SpO2 is driving compensatory heart rate elevation. " "The body is increasing cardiac output to offset reduced blood oxygen." ), }, { "source": "heart_rate", "target": "temperature", "message": ( "Cardiac-Thermal Chain: Elevated heart rate is correlating with rising body temperature. " "Increased metabolic activity is generating excess heat." ), }, { "source": "spo2", "target": "temperature", "message": ( "Full Physiological Chain: Simultaneous SpO2 desaturation and thermal elevation detected. " "This pattern may indicate acute physiological stress or systemic illness." ), }, ] The chain detection function evaluates each wearer’s per-vital statuses against these rules. A chain alert fires only when the source vital is in a danger state and the target vital also shows an at-risk reading — confirming a linked physiological event rather than an isolated measurement anomaly: def detect_health_chains(wearer_id: str, vital_statuses: dict) -> list: chains = [] danger = {"Distress", "Critical"} at_risk = {"Stress", "Distress", "Critical"} for rule in VITAL_CHAIN_RULES: src = vital_statuses.get(rule["source"], "Normal") tgt = vital_statuses.get(rule["target"], "Normal") if src in danger and tgt in at_risk: chains.append({ "wearer_id": wearer_id, "source": rule["source"], "target": rule["target"], "message": rule["message"], }) return chains Machine Learning Health Risk Classifier VitalWatch includes a complete machine learning pipeline that uses the historical data stored in GridDB to train a health risk classifier. The model learns to predict one of four health states — Normal, Stress, Distress, or Critical — from the seven-dimensional feature vector of a single wearable reading. Why Random Forest? Random Forest is an ideal fit for this task because it seamlessly handles mixed physiological features across different units without requiring normalization. It also remains highly robust even with the smaller datasets typical of wearable monitoring demos. Additionally, the model outputs clear feature importances, making its health risk predictions easily explainable. Loading data from GridDB The training pipeline loads all available readings directly from the GridDB containers: def load_training_data(store) -> tuple: """Query all readings from GridDB and build a feature matrix with auto-generated labels.""" X, y, wearer_labels = [], [], [] for wearer_id in WEARERS: readings = query_recent(store, wearer_id, limit=500) for r in readings: features = [r["heart_rate"], r["spo2"], r["temperature"], r["activity_level"], r["systolic_bp"], r["diastolic_bp"], r["blood_sugar"]] # Labels are generated using the same rule-based logic as the dashboard label = STATUS_TO_INT[analyze_wearer(wearer_id, [r])["status"]] X.append(features) y.append(label) return np.array(X), np.array(y), wearer_labels This approach makes labels consistent with the dashboard display: the ML model learns to replicate the same health risk judgments that the rule-based monitoring system makes, but as a learnable function of the raw feature values. Training and evaluation model = RandomForestClassifier( n_estimators=150, max_depth=10, min_samples_leaf=2, random_state=42, n_jobs=-1, ) model.fit(X_train, y_train) After training, the pipeline prints a classification report and confusion matrix, followed by ranked feature importances: Feature Importances: spo2 ████████████████████████████████████████ 0.3912 heart_rate ████████████████████████████████ 0.3104 systolic_bp ████████████ 0.1187 blood_sugar ████████ 0.0823 temperature █████ 0.0521 diastolic_bp ████ 0.0401 activity_level ██ 0.0052 SpO2 and heart rate emerge as the most predictive features — a result that aligns with the physiological chain rules encoded in the rule-based detection layer. Building the Monitoring Dashboard Visualization is essential for making vital sign data actionable. The VitalWatch dashboard was built using HTML, JavaScript, and Chart.js to display real-time wearer status, vital sign trend charts, physiological chain alerts, and the fleet-wide health risk score. The Flask backend exposes three main data API endpoints that the dashboard polls every three seconds: Endpoint What it returns GET /api/fleet (Main) Status of all wearers + chain alerts + risk score GET /api/wearer/ Vital sign history for a single wearer (used for charts) GET /api/timeline Log of health status transitions over time A simplified view of the dashboard polling function: async function pollFleet() { try { const res = await fetch('/api/fleet'); const data = await res.json(); // Update each wearer card for (const [wearerId, status] of Object.entries(data.wearers)) { setCard(wearerId, status.status, status.message, status.latest); } renderChains(data.chains || []); updateFleetScore(data.risk_score ?? 0); } catch (e) { console.error('Fleet poll failed:', e); } } The dashboard calls the fleet endpoint every three seconds to keep the interface synchronized with the latest health data in GridDB Cloud. The chart endpoint provides rolling vital sign history for each wearer, rendered as a dual-axis mini trend chart (HR on the left axis, SpO2 on the right). Running the Project Make sure GridDB Cloud is running and credentials are configured as environment variables. Then run the following in order: # Step 1: Create containers and seed historical data $ $ python src/insert_data.py # Step 2: Start the monitoring backend $ $ python src/app.py # Step 3 (optional): Keep the dashboard alive with a live heartbeat $ $ python src/insert_data.py –live # Step 4 (optional): Trigger a live alert simulation $ $ python src/simulate_alert.py # Step 5 (optional): Train the ML health risk classifier $ $ python src/train_model.py Open http://localhost:5000 to view the dashboard. The Background Heartbeat (insert_data.py –live): This script provides a stable, healthy baseline so the dashboard stays “green” by default. The Manual Alert (simulate_alert.py): This script temporarily introduces a deteriorating health event to show how vital signs cascade. Click “Alert Addressed” in the dashboard to resolve it. One of the advantages of using GridDB for this architecture is automatic recovery: once the simulation finishes, the background producer continues sending normal-range readings. These healthy readings naturally displace the temporary critical spikes in the rolling query window, and the system returns to a safe state without a manual database reset. Results and Dashboard Overview After running the system, the monitoring dashboard displays the real-time health status of all three wearers. The interface shows live vital signs for Heart Rate, SpO2, Body Temperature, and Activity Level for each wearer, along with trend charts, chain alert panels, and the fleet-wide risk score. Normal State In the normal state, all wearers show green status indicators and a fleet risk score near 0. The escalation timeline panel is empty, confirming no status transitions have occurred. Stress Detected As the athlete’s heart rate climbs and SpO2 begins to fall, individual vital signs transition to warning state. The athlete card updates to “Stress” and the overall risk score begins rising. Physiological Chains Active When the athlete’s SpO2 drops into the danger zone while heart rate is simultaneously elevated, the system fires a chain alert: “Oxygen-Cardiac Chain: Falling SpO2 is driving compensatory heart rate elevation.” A second chain may fire if body temperature also begins rising. Alert Resolved Once the operator clicks “Alert Addressed,” the simulation stops, recovery readings are injected, and the dashboard returns to normal. The risk score drops back toward zero and chain alerts clear. By visualizing vital sign trends, physiological chain events, and fleet risk levels together, the dashboard gives health monitoring operators a clear and actionable picture of wearer health states. Conclusion Wearable health monitoring generates high-frequency, time-ordered data that tells a much richer story than any single sensor reading. Detecting that story requires a database purpose-built for time-series workloads and an analysis layer that understands physiological correlations rather than evaluating each vital in isolation. In this project, we built VitalWatch — a complete wearable health monitoring prototype using GridDB Cloud to store and query vital sign telemetry. By combining simulated sensor readings with rule-based physiological chain detection and a machine learning health risk classifier, the system can detect early warning patterns before they escalate into critical events. The complete implementation can be found in the project repository. : https://github.com/DoneByManthan/GridDB-VitalWatch.git In the future, this approach could be extended with: Personalized ML models trained on each wearer’s individual baseline rather than group averages stored in GridDB. Anomaly detection using unsupervised methods (e.g., Isolation Forest) on the GridDB historical data to catch unusual patterns that fall outside rule-defined thresholds. Real device integration by replacing the Python simulator with a Bluetooth LE or MQTT data ingestion layer that receives readings directly from consumer wearables. Federated learning where each wearer’s device trains a local model on-device and only sends model weight updates to a central server, preserving health data privacy while improving the global

Large Language Models (LLMs) allow developers to combine advanced AI reasoning with powerful databases to analyze and visualize complex datasets. In this article, you will see how to build a tabular data visualization assistant using GridDB Cloud, LangGraph, and the OpenAI GPT-4o model. We will import the Titanic dataset into GridDB, query it programmatically, and then use a LangGraph ReAct agent to answer questions and generate plots automatically. GridDB’s flexible schema, high-performance design, and compatibility with structured data make it well-suited for storing both tabular and time series data. Prerequisites: You will need the following to run scripts in this article: A GridDB cloud account. Sign up for GridDB cloud and complete configuration settings. OpenAI API Key. You can use any other LLM provider, but you will need to update the scripts in this article slightly. Installing and Importing Required Libraries The following script installs and imports the required libraries for this application: !pip install langchain !pip install langchain-core !pip install langchain-community !pip install langgraph !pip install langchain_huggingface !pip install tabulate !pip uninstall -y pydantic !pip install –no-cache-dir "pydantic>=2.11,<3" import pandas as pd import json import datetime as dt import base64 import requests import numpy as np from pathlib import Path import matplotlib matplotlib.use("Agg") # safe, consistent backend import matplotlib.pyplot as plt from typing_extensions import Annotated from operator import add # used as list reducer from typing import TypedDict, List, Dict from pydantic import BaseModel, Field from IPython.display import Image, display from langchain_core.prompts import ChatPromptTemplate from langchain_core.tools import tool from langchain_openai import ChatOpenAI, OpenAI from langgraph.graph import START, END, StateGraph from langchain_core.messages import HumanMessage from langchain_experimental.agents import create_pandas_dataframe_agent from langgraph.prebuilt import create_react_agent from langchain.agents.agent_types import AgentType Importing the Dataset We will insert the Titanic dataset into GridDB and create visualizations using this data. The following script imports the data into a Pandas dataframe. dataset = pd.read_csv("https://raw.githubusercontent.com/datasciencedojo/datasets/refs/heads/master/titanic.csv", encoding = 'utf-8') dataset.head() Output: Establishing a Connection with GridDB Cloud To establish a connection with GridDB, replace your credentials in the following script and run it. username = "USER_NAME" password = "PASSWORD" base_url = "GRIDDB_CLOUD_URL" url = f"{base_url}/checkConnection" credentials = f"{username}:{password}" encoded_credentials = base64.b64encode(credentials.encode()).decode() headers = { 'Content-Type': 'application/json', # Added this header to specify JSON content 'Authorization': f'Basic {encoded_credentials}', 'User-Agent': 'PostmanRuntime/7.29.0' } response = requests.get(url, headers=headers) print(response.status_code) print(response.text) Output: 200 If you see the above message, you have successfully connected with GridDB cloud. Inserting Data in GridDB Cloud Dataset To insert data in GridDB, you first need to map your dataset types to GridDB dataset types and then create a GridDB container. Creating a Container for the Titanic Dataset in GridDB The following script maps your dataset column types to GridDB column types. dataset.insert(0, "SerialNo", dataset.index + 1) dataset.columns.name = None # Mapping pandas dtypes to GridDB types type_mapping = { "int64": "LONG", "float64": "DOUBLE", "bool": "BOOL", 'datetime64': "TIMESTAMP", "object": "STRING", "category": "STRING", } # Generate the columns part of the payload dynamically columns = [] for col, dtype in dataset.dtypes.items(): griddb_type = type_mapping.get(str(dtype), "STRING") # Default to STRING if unknown columns.append({ "name": col, "type": griddb_type }) print(columns) [{'name': 'SerialNo', 'type': 'LONG'}, {'name': 'PassengerId', 'type': 'LONG'}, {'name': 'Survived', 'type': 'LONG'}, {'name': 'Pclass', 'type': 'LONG'}, {'name': 'Name', 'type': 'STRING'}, {'name': 'Sex', 'type': 'STRING'}, {'name': 'Age', 'type': 'DOUBLE'}, {'name': 'SibSp', 'type': 'LONG'}, {'name': 'Parch', 'type': 'LONG'}, {'name': 'Ticket', 'type': 'STRING'}, {'name': 'Fare', 'type': 'DOUBLE'}, {'name': 'Cabin', 'type': 'STRING'}, {'name': 'Embarked', 'type': 'STRING'}] Next, we will create a collection type container titanic_db in our GridDB cloud database. url = f"{base_url}/containers" container_name = "titanic_db" # Create the payload for the POST request payload = json.dumps({ "container_name": container_name, "container_type": "COLLECTION", "rowkey": True, # Assuming the first column as rowkey "columns": columns }) # Make the POST request to create the container response = requests.post(url, headers=headers, data=payload) # Print the response print(f"Status Code: {response.status_code}") Output: Status Code: 201 Inserting Titanic Dataset in GridDB Next, we will iterate through the rows in our dataset, create a JSON payload containing the data, and will insert the data into the container we created in the previous section. url = f"{base_url}/containers/{container_name}/rows" # Convert dataset to list of lists (row-wise) with proper formatting def format_row(row): formatted = [] for item in row: if pd.isna(item): formatted.append(None) # Convert NaN to None elif isinstance(item, bool): formatted.append(str(item).lower()) # Convert True/False to true/false elif isinstance(item, (int, float)): formatted.append(item) # Keep integers and floats as they are else: formatted.append(str(item)) # Convert other types to string return formatted # Prepare rows with correct formatting rows = [format_row(row) for row in dataset.values.tolist()] # Create payload as a JSON string payload = json.dumps(rows) # Make the PUT request to add the rows to the container response = requests.put(url, headers=headers, data=payload) # Print the response print(f"Status Code: {response.status_code}") print(f"Response Text: {response.text}") Output: Status Code: 200 Response Text: {"count":891} The above output shows that the data has been successfully inserted into GridDB. Next, we will see how to retrieve data from GridDB and plot visualizations using it. Visualizing GridDB Results Using OpenAI and ReAct Agent The following script reads data from GridDB and inserts it in a Pandas dataframe. container_name = "titanic_db" url = f"{base_url}/containers/{container_name}/rows" # Define the payload for the query payload = json.dumps({ "offset": 0, # Start from the first row "limit": 10000, # Limit the number of rows returned "condition": "", # No filtering condition (you can customize it) "sort": "" # No sorting (you can customize it) }) # Make the POST request to read data from the container response = requests.post(url, headers=headers, data=payload) # Check response status and print output print(f"Status Code: {response.status_code}") if response.status_code == 200: try: data = response.json() print("Data retrieved successfully!") # Convert the response to a DataFrame rows = data.get("rows", []) titanic_dataset = pd.DataFrame(rows, columns=[col for col in dataset.columns]) except json.JSONDecodeError: print("Error: Failed to decode JSON response.") else: print(f"Error: Failed to query data from the container. Response: {response.text}") print(titanic_dataset.shape) titanic_dataset.head() Output: Let’s try to plot the average for the passengers who survived and those who didn’t. We will use these values to verify the result from our ReAct agent. avg = titanic_dataset.groupby('Survived')['Fare'].mean().round(2) print(avg) Output: Survived 0 22.12 1 48.40 Name: Fare, dtype: float64 Creating a LangGraph ReAct Agent for Data Visualization To plot visualizations, we will create a LangGraph ReAct agent with two tools: df_answer and save_plot. The df_answer tool will use the create_pandas_dataframe_agent to retrieve information from the database, including the plot, if any. The save_plot tool saves the plot to the local drive. The following script defines the agent’s state and the large language model (OpenAI GPT-4o in this case) we will use to answer the user’s question. class State(TypedDict): question: str answer: str plots: Annotated[List[Dict[str, str]], add] api_key = "YOUR_OPENAI_API_KEY" llm = ChatOpenAI(model="gpt-4o", api_key=api_key, temperature = 0) Next, we define the create_pandas_dataframe_agent function that returns information from the Pandas dataframe retrieved from GridDB. We also define the df_answer tool that calls the create_pandas_dataframe_agent. df_agent = create_pandas_dataframe_agent(llm, titanic_dataset, verbose=True, agent_type=AgentType.ZERO_SHOT_REACT_DESCRIPTION, allow_dangerous_code=True) _LAST = {"fig": None} # keep a handle to the last real figure @tool("df_answer") def df_answer(question: str) -> str: """ Use the pandas DataFrame agent to compute/plot. IMPORTANT: do NOT call plt.show() or plt.close() in the generated code. """ res = df_agent.invoke( question ) # CAPTURE whichever figure the agent actually created fig_nums = plt.get_fignums() # existing figures in this process if fig_nums: _LAST["fig"] = plt.figure(fig_nums[-1]) # latest real figure return res["output"] The following script defines the save_plot tool that saves the plot generated by the df_answer tool. @tool("save_plot") def save_plot(filename: str = "plot.png", dpi: int = 200, close: bool = True) -> str: """ Save the most recent existing Matplotlib figure (not an empty gcf()). Returns {"plot": {"name": …, "path": …}} or {"error": …}. """ Path("plots").mkdir(exist_ok=True) fig = _LAST.get("fig") # Fallback: grab last live figure if we didn't capture yet if fig is None: nums = plt.get_fignums() if not nums: return json.dumps({"error": "no_figure", "message": "No active figure to save."}) fig = plt.figure(nums[-1]) # Render + save fig.tight_layout() try: fig.canvas.draw() # ensure render except Exception: pass out = Path("plots") / filename fig.savefig(out, dpi=dpi, bbox_inches="tight") if close: plt.close(fig) # avoid accumulating figures return json.dumps({"plot": {"name": filename, "path": str(out.resolve())}}) Finally we define the ReAct agent using the LLM and the tool we just defined. SYSTEM = """ You work over a Titanic pandas DataFrame. – To compute answers or create charts, call `df_answer(question=…)`. – If a plot should be saved, call `save_plot(filename=…, dpi=200)`. – Keep text concise. If you saved a plot, you may echo the absolute path. """ react = create_react_agent( llm, tools=[df_answer, save_plot], prompt=SYSTEM, ) The following script creates our final graph object. def run_react(state: State) -> State: out = react.invoke({"messages": [("user", state["question"])]}) msgs = out["messages"] final_text = msgs[-1].content new_plots = [] for m in msgs: # Tool messages include the tool's return in `content` try: data = json.loads(getattr(m, "content", "") or "{}") except Exception: data = None if isinstance(data, dict) and "plot" in data: new_plots.append(data["plot"]) # {"name": "…", "path": "…"} return {"answer": final_text, "plots": new_plots} graph_builder = StateGraph(State) graph_builder.add_node("ask_question", run_react) graph_builder.add_edge(START, "ask_question") graph_builder.add_edge("ask_question", END) graph = graph_builder.compile() display(Image(graph.get_graph(xray=True).draw_mermaid_png())) Output: The above output shows the flow of the graph. The user’s question is passed to the ReAct agent, which decides which tools it requires to answer the user’s query. Testing the Agent & Generating Responses Let’s test the agent. We will first ask a simple question that doesn’t require saving or plotting a graph. # A) plain Q&A s = graph.invoke({"question": "What is the average Fare for passengers that survive and those who did not?"}) print(f"\nFinal Answer: {s['answer']}") Output: The output displays the average fares for passengers who survived and those who did not. Note that these values are identical to the ones we retrieved earlier by executing a direct operation on the Pandas dataframe. Next, we will request our agent to plot a chart using the results. # B) plot + save (the agent will call save_plot internally) s = graph.invoke({ "question": ("Plot a bar chart of average Fare for passengers that survive and those who did not?"), }) print(f"\nFinal Answer: {s['answer']}") print("plots so far:", s["plots"]) Output: The output shows that the agent saved the plot and also returned its location in the output. If you open the plot, you can see the average fare by survival rate plotted in the form of a bar chart. Conclusion This article demonstrates how to integrate GridDB Cloud with LangGraph and OpenAI to create a ReAct agent that can query tabular datasets and generate visualizations. By combining structured storage with the reasoning power of LLMs, we developed a system that seamlessly handles both textual answers and graphical plots. If you have questions or need support with GridDB Cloud, feel free to post them on Stack Overflow using the griddb tag. The GridDB team will be happy to help. For the complete code and additional examples, visit GridDB Blogs GitHub