Blog

Introduction The capacity to monitor geological events in real-time is a critical component of modern public safety and environmental management strategies. For events such as volcanic eruptions, where precursors like increased seismic activity can offer vital windows for action, the ability to ingest, process, and visualize data with minimal latency is paramount. An effective monitoring system is foundational for reliable early-warning protocols and efficient disaster response. This technical guide details the development of a high-performance volcanic activity monitoring system. The solution leverages GridDB Cloud, a database optimized for time-series data, in conjunction with the Spring Boot framework for a robust backend architecture. The system ingests live data from USGS APIs, persists it to GridDB, and presents the information on a dynamic, map-centric web interface, providing a powerful and practical monitoring tool. Database Selection: The Case for GridDB The choice of database is a primary architectural decision in any data-intensive application. For this monitoring system, which inherently deals with time-indexed event data, GridDB presents several key advantages. Optimized for Time-Series Workloads: GridDB is purpose-built for IoT and time-series data. Its in-memory-oriented architecture storage model are designed for high-throughput ingestion and querying, ensuring the application can handle a continuous stream of data without performance bottlenecks. A-Data-Model-for-Time-Series: The container-based data model is a natural fit. VolcanoData and SeismicData can be logically separated into their own containers, using the event timestamp as the row key. This schema facilitates efficient time-based querying and data lifecycle management. Flexible API Access: While offering native Java clients for maximum performance, GridDB also provides a comprehensive REST API. This grants developers language-agnostic access and allows a standard Spring Boot application to integrate seamlessly using conventional tools like RestTemplate, avoiding the need for additional complex dependencies. System Architecture and Data Flow The application employs a layered, unidirectional data flow architecture. This design promotes modularity, simplifies debugging, and enhances scalability. Scheduled Ingestion: The data pipeline is initiated by DataSchedulerService, a Spring-scheduled bean that executes at a defined interval. External Data Sourcing: This scheduler invokes VolcanicDataService, which acts as a client to the public USGS APIs, handling HTTP request execution and response reception. Data Parsing and Transformation: The raw JSON responses are parsed using the Jackson ObjectMapper into a JsonNode tree. Specific fields are then extracted and mapped into strongly-typed Java domain models (VolcanoData, SeismicData). This practice ensures data integrity and insulates the application from changes in the external API’s structure. Data Persistence: The curated Java objects are passed to GridDBService, which functions as an adapter to the GridDB REST API. It transforms the application data into the JSON structure required by GridDB’s bulk insertion endpoint and manages authentication. Presentation Layer: User interaction begins with a request to the DashboardController. This controller retrieves the latest data from GridDB via the GridDBService and forwards it to the Thymeleaf view layer. Dynamic Rendering: The Thymeleaf engine processes the dashboard.html template, rendering the data into HTML tables and embedding the geospatial data directly into client-side JavaScript variables. The browser then executes this script, using Leaflet.js to render the final interactive map visualization. Backend Implementation: The Data Pipeline The backend constitutes the core data processing engine of the application. Project Configuration (application.properties) The application.properties file provides the foundational configuration for the application, most importantly defining the connection to GridDB. # GridDB REST API Endpoint griddb.rest.url=https://your-griddb-cloud-instance.griddb.com:443/griddb/v2/your-cluster-name/dbs/your-database-name # GridDB Basic Authentication Credentials griddb.api.key=YOUR_GRIDDB_BASIC_AUTH_KEY # Spring Boot Actuator and Server Port management.endpoints.web.exposure.include=* server.port=8081 The griddb.api.key property facilitates HTTP Basic Authentication. This value must be a Base64 encoded string of the GridDB username:password. For instance, admin:admin is encoded as YWRtaW46YWRtaW4=. The GridDB Service Layer (GridDBService) This service is the most critical integration component, managing all communication with the GridDB REST API. Authentication: API access requires that every request be authenticated. The createHeaders method isolates this responsibility, programmatically constructing the necessary Authorization header for each outgoing request. private HttpHeaders createHeaders() { HttpHeaders headers = new HttpHeaders(); headers.set("Authorization", "Basic " + griddbApiKey); headers.set("Content-Type", "application/json"); return headers; } Data Transformation for Ingestion: To maximize efficiency, the GridDB REST API supports bulk row insertion via a JSON array of arrays. The putVolcanoData method performs this data transformation, converting a List into the required payload format before serialization and transmission. This approach is significantly more performant than sending individual requests for each row. // Inside GridDBService.java public void putVolcanoData(List<VolcanoData> volcanoData) { List<List<Object>> payload = new ArrayList<>(); for (VolcanoData data : volcanoData) { List<Object> row = new ArrayList<>(); row.add(ISO_INSTANT_FORMATTER.format(data.timestamp)); row.add(data.name); // … other fields are added in the correct order … payload.add(row); } // … payload is serialized and sent to the /rows endpoint … } Of particular note is the use of a custom DateTimeFormatter. Consistent timestamp formatting is critical for time-series integrity. This formatter ensures that Java Instant objects are always serialized to a standardized ISO-8601 string that GridDB can parse into its native TIMESTAMP data type. Frontend Implementation: The User Interface The utility of a powerful backend is contingent upon a clear and intuitive data presentation layer. The frontend is engineered to provide just that. A Strategic Selection of Technologies Thymeleaf: Chosen for its “natural templating” feature, which allows templates to be rendered as valid, static HTML during development. This improves the development workflow and decouples the view’s design from its server-side processing. Bootstrap: Provides a robust CSS framework for a responsive, mobile-first design, enabling the use of polished UI components like card and navbar with minimal custom styling. Leaflet.js: A lightweight and powerful open-source library for creating the interactive map, which is central to visualizing the geospatial data. Functional Interactivity The map style switcher is a key interactive feature, allowing users to select a base layer that provides the most relevant context for their analysis, whether it be satellite imagery or a street map. This is implemented using a clean JavaScript pattern that manages a collection of tile layer providers. var tileLayers = { street: L.tileLayer('…OpenStreetMap URL…', { … }), satellite: L.tileLayer('…Esri Imagery URL…', { … }), terrain: L.tileLayer('…OpenTopoMap URL…', { … }) }; function setMapStyle(style) { if (currentTileLayer) { map.removeLayer(currentTileLayer); } currentTileLayer = tileLayers[style]; currentTileLayer.addTo(map); } // Set the initial map style on page load setMapStyle('street'); This architecture, combining a well-engineered backend with a thoughtfully designed frontend, results in a highly functional and insightful application. Running the Application and Accessing the Dashboard To run the Spring Boot application and access the volcanic activity dashboard, follow these steps: Ensure Prerequisites are Met: Java Development Kit (JDK) 17 or higher: The application is built with Java. Maven: Used for building the project and managing dependencies. GridDB Cloud Instance: Ensure your GridDB Cloud instance is running and you have the necessary API key and endpoint URL configured in src/main/resources/application.properties. Build the Application: Navigate to the my-griddb-app directory in your terminal and build the project using Maven: $ cd my-griddb-app $ mvn clean install This command compiles the code, runs tests, and packages the application into a JAR file. Run the Application: After a successful build, you can run the Spring Boot application from the my-griddb-app directory: $ java -jar target/my-griddb-app-1.0-SNAPSHOT.jar Alternatively, you can run it directly using Maven Spring Boot plugin: $ mvn spring-boot:run The application will start, and you will see logs indicating that the embedded Tomcat server has started on the configured port (default is 8081, as set in application.properties). Access the Dashboard: Open your web browser and navigate to the application’s dashboard URL. By default, this will be: http://localhost:8081/dashboard You should see the interactive volcanic activity dashboard displaying the latest data fetched from GridDB. Interactive map view of volcanic activity. Detailed table view of volcanic data. Conclusion This project serves as a comprehensive technical blueprint for building high-performance, real-time monitoring systems. The combination of GridDB for data persistence and Spring Boot for backend logic proves to be a powerful stack for such applications. The final dashboard effectively ingests, stores, and visualizes critical geological data. The modular design of the application provides a solid foundation for future enhancements, such as: Automated Alerting: A service could be implemented to analyze incoming data for predefined critical patterns and trigger automated alerts. Historical Data Analysis: The user interface could be extended to support querying and visualizing historical data, enabling trend analysis. Expanded Data Integration: The system could be enhanced by integrating additional data sources, such as weather feeds or gas emission data, to create a more holistic monitoring

Embulk is an open-source bulk data loader that moves records between databases, storage services, and file formats using plugins. GridDB has an official Embulk output plugin, embulk-output-griddb, which lets you load data from any Embulk input source (CSV files, other databases, cloud storage) straight into GridDB containers. The plugin was written back in the Embulk 0.9 days, and getting it built and running in 2026 takes some work. The Maven repositories it depends on have shut down, it needs Java 8, and connecting to GridDB Cloud has some networking requirements that aren’t obvious at first. This post goes through the whole process, from a fresh download to 50,000 rows sitting in a GridDB Cloud container, including every problem we ran into along the way. Here’s the pipeline we’re building: sample.csv –> Embulk (csv parser) –> embulk-output-griddb –> GridDB Cloud Prerequisites Java 8. Embulk 0.9.x runs on JRuby and won’t start on Java 11 or newer. Don’t try to get around this by using a newer Embulk (0.11/1.0). The plugin loading model changed completely in 0.10+, and this plugin relies on embulk-core APIs that don’t exist there anymore. Embulk 0.9.17. Download the jar from the Embulk GitHub releases page. The embulk-output-griddb source, which comes with three jars in its lib/ directory: gridstore.jar, gridstore-jdbc.jar, and griddb-tools-common.jar. A GridDB Cloud instance, or a self-hosted cluster. We’ll get into the difference below. If you use SDKMAN, installing Java 8 is one command: $ sdk install java 8.0.402-tem SDKMAN also sets JAVA_HOME, which matters more than whatever java is on your PATH. Gradle reads JAVA_HOME directly, and the Gradle 4.1 wrapper this project uses can’t even parse the version string of a modern JDK. If you see this: Could not determine java version from '21.0.4'. then JAVA_HOME is pointing at the wrong JDK, even if java -version says 8. Gotcha #1: read-only directories from the zip The plugin comes as a zip, and depending on how it was packaged, the extracted directories can end up with mode 555 (read-only, even for you). Gradle then fails with a confusing error about its cache: Could not create service of type CrossBuildFileHashCache > Failed to create parent directory '.gradle' The fix: $ chmod -R u+w embulk-output-griddb If you still get the CrossBuildFileHashCache error after fixing permissions, there’s probably leftover state from an earlier failed build. Stop the Gradle daemon and clear the project cache: $ ./gradlew –stop $ rm -rf .gradle Gotcha #2: the build depends on dead repositories This one takes the most work. The original build.gradle depends on three things that don’t exist anymore: JCenter. JFrog shut down the jcenter() repository, and requests now redirect to Maven Central, which doesn’t have the same artifacts. The Bintray Gradle plugin, which needs http-builder:0.7.2. That artifact was only ever on JCenter. The jruby-gradle plugin’s rubygems proxies (rubygems-proxy.torquebox.org and rubygems.lasagna.io). One returns HTML error pages that Gradle tries to parse as POM files, and the other doesn’t resolve in DNS at all. On top of that, org.embulk:embulk-core:0.9.17 was also JCenter-only. Maven Central only has Embulk from 0.10.29 onward, so there’s no repository left that can resolve it. The good news is you don’t need any of this to build the plugin jar. The Bintray and JRuby plugins are only used for publishing to Bintray (gone) and RubyGems. And all of the embulk-core classes are already inside the Embulk runtime jar you downloaded, so you can compile against that instead. Put embulk-0.9.17.jar in the project’s lib/ directory: $ curl -L -o lib/embulk-0.9.17.jar \ https://github.com/embulk/embulk/releases/download/v0.9.17/embulk-0.9.17.jar Then replace build.gradle with this trimmed-down version: plugins { id "java" id "checkstyle" } repositories { mavenCentral() } version = "0.1.0" sourceCompatibility = 1.8 targetCompatibility = 1.8 configurations { provided } dependencies { compileOnly files("lib/embulk-0.9.17.jar") compile group: 'com.sun.jersey', name: 'jersey-core', version: '1.19.4' compile files('lib/griddb-tools-common.jar') compile files("lib/gridstore.jar") compile files("lib/gridstore-jdbc.jar") } task classpath(type: Copy, dependsOn: ["jar"]) { doFirst { file("classpath").deleteDir() } from (configurations.runtime – configurations.provided + files(jar.archivePath)) into "classpath" } clean { delete "classpath" } checkstyle { configFile = file("${project.rootDir}/config/checkstyle/checkstyle.xml") toolVersion = '8.23' ignoreFailures = true } task "package"(dependsOn: ["classpath"]) { doLast { println "> Build succeeded." println "> You can run embulk with '-L ${file(".").absolutePath}' argument." } } What changed: jcenter(), the Bintray plugin, and the JRuby plugin are removed, along with the gem and gemPush publishing tasks. embulk-core is now a compileOnly dependency on the local jar. compileOnly keeps it off the runtime classpath, so the classpath task doesn’t copy the 40 MB Embulk runtime into the plugin output. That’s what the old provided configuration was doing manually. ignoreFailures = true on checkstyle, so style warnings in the original source don’t fail the build. Now build it: $ ./gradlew package > Task :compileJava > Task :package > Build succeeded. > You can run embulk with '-L /home/user/embulk-output-griddb' argument. BUILD SUCCESSFUL You might see warnings that gridstore.jar is a “malformed jar.” The bundled jars have minimal manifests, and Gradle 4.x just warns about it. You can ignore them. The build produces a classpath/ directory with the plugin jar and its dependencies. You point Embulk at it with the -L flag, and that’s it. Nothing gets installed system-wide. Understanding the connection model Before configuring anything, it’s important to know that this plugin uses the GridDB native Java client (NoSQL interface), not the Web API. It connects over the binary protocol on the transaction port (10001 by default) and finds the cluster nodes in PROVIDER mode, which means it downloads a JSON file from a provider_url that lists the node addresses. I’m pointing this out because the other GridDB Cloud connectors use the Web API, and the networking requirements are completely different. Why not just whitelist your IP? Since GridDB Cloud v3.2, native clients can connect over the public internet. You whitelist your IP in the “GridDB Access” tab, use the HTTPS provider URL, and add connectionRoute=PUBLIC to the connection. That’s covered in Connecting to GridDB Cloud v3.2 from Your Local Dev Environment (No VPN, No VNet Peering), and it’s the first thing we tried. It doesn’t work with this plugin as shipped. HTTPS provider support comes from a separate library, gridstore-advanced.jar, and the gridstore.jar bundled with the plugin is an old build from before that existed. Pointing the plugin at the HTTPS provider URL fails right away: [145005:JC_ILLEGAL_PROPERTY_ENTRY] HTTPS Provider not available because of lack of extra library What works: VNet peering and the HTTP provider GridDB Cloud also serves the provider JSON from a plain HTTP URL, and the old client handles that fine. The catch is that the node addresses in that JSON are private IPs inside the GridDB Cloud VNet, and without the public route, your client has to reach them directly. So: Run Embulk from inside a VNet peered to the GridDB Cloud tenant VNet. You can set up peering from the Cloud management console. Whitelisting your IP won’t give you a route to the private node addresses. Use the HTTP provider URL, which is reachable from inside the peered network, instead of the HTTPS one. That’s the setup we use for the rest of this post. If peering isn’t an option for you, you could try replacing the plugin’s lib/gridstore.jar with a current client from the Cloud console’s library download, adding gridstore-advanced.jar next to it, referencing both in build.gradle, and using the HTTPS provider URL with connectionRoute=PUBLIC. We haven’t tested that with this plugin’s code, so your mileage may vary. One more Azure note if you’re setting up peering: peered VNets can’t have overlapping address spaces, and Azure checks this across every VNet already peered to the GridDB Cloud side. If the Cloud VNet is already peered to another VNet using 10.0.0.0/16, yours can’t use 10.0.0.0/16 either, even in a different subscription. Azure will reject it with VnetAddressSpaceOverlapsWithAlreadyPeeredVnet. Either pick an address space that doesn’t overlap, or run Embulk from a VM in a VNet that’s already peered. Configuring the pipeline Here’s a full config.yml that loads a four-column CSV with no header: in: type: file path_prefix: /home/user/sample parser: charset: UTF-8 newline: LF type: csv delimiter: ',' quote: '"' trim_if_not_quoted: false allow_extra_columns: false allow_optional_columns: false columns: – {name: id, type: long} – {name: name, type: string} – {name: quantity, type: long} – {name: price, type: double} out: type: griddb mode_cluster: PROVIDER provider_url: http://<storage-account>.blob.core.windows.net/<container>/<cluster>.json cluster: <your-cluster-name> database: <your-database> container: con_25 user: <your-user> password: <your-password> mode_insert: replace default_timezone: UTC batch_size: 1000 exec: max_threads: 1 min_output_tasks: 1 A few notes: path_prefix is a prefix, not a full path. /home/user/sample matches sample.csv, sample2.csv, and so on. Embulk lists files in the parent directory, so the user running it needs read access there. The plugin’s bundled instructions assume you run everything as root out of /root, but you don’t need to and shouldn’t. Don’t add a gzip decoder unless your file is actually gzipped. A decoders block with gzip on a plain CSV fails with Not in GZIP format, which shows up wrapped in a misleading DB schema is invalid error. The parser schema has to match the container. In append mode, the plugin checks your Embulk column types against the existing container and won’t run if they don’t match. In replace mode, it drops and recreates the container from your schema, which is easier for a first load. Don’t skip the exec block. The next section explains why. Gotcha #3: parallel output tasks race on container creation By default, Embulk runs two output tasks per input task. If you run without the exec block and watch the logs, you’ll see both writers connect, and then a few milliseconds apart: (embulk-output-executor-1): Create new container:con_25 (embulk-output-executor-0): Create new container:con_25 Each output task checks for the container and creates it on its own. In replace mode, that means two tasks can each drop and recreate the container without knowing about the other. In our 50,000-row test, the two tasks split the file (25,016 and 24,984 rows) and all 50,000 made it in, but that was luck. With a slower connection or a bigger file, one task’s replace could wipe out rows the other task already wrote. Until the plugin fixes this, force a single output task: exec: max_threads: 1 min_output_tasks: 1 You lose parallel writes, but the results are consistent from run to run. Running it $ java -jar lib/embulk-0.9.17.jar run config.yml -L . -L . points Embulk at the plugin project directory (the one with classpath/ in it). Before a real run, it’s worth doing a preview. It only runs the input and parser, so it catches CSV and schema problems without touching GridDB: $ java -jar lib/embulk-0.9.17.jar preview config.yml -L . A successful run ends like this: Finish import: Database: <your-database> Container: con_25 Model: replace Total rows: 50000 Committed. And checking from the GridDB side: $ griddb-cloud-cli sql query -s "SELECT COUNT(*) from con_25" [[{"Name":"","Type":"LONG","Value":50000}]] All 50,000 rows loaded from a CSV through the native client. Summary of the traps Symptom Cause Fix Could not determine java version Gradle 4.1 reading a modern JDK from JAVA_HOME Point JAVA_HOME at Java 8 Could not create service … CrossBuildFileHashCache Read-only directories from the zip, or a stale daemon cache chmod -R u+w, then ./gradlew –stop && rm -rf .gradle Could not find http-builder:0.7.2 Bintray plugin depends on a JCenter-only artifact Remove the Bintray plugin Could not parse POM … torquebox.org JRuby plugin’s rubygems proxies are dead Remove the JRuby plugin Could not find org.embulk:embulk-core:0.9.17 Artifact was JCenter-only, never on Maven Central compileOnly files(“lib/embulk-0.9.17.jar”) HTTPS Provider not available Bundled client is missing the HTTPS provider library Use the HTTP provider URL from inside a peered VNet JC_BAD_CONNECTION … connect timed out No route to the cluster’s private node addresses VNet peering, with the node ports open from your subnet VnetAddressSpaceOverlapsWithAlreadyPeeredVnet Your VNet’s address space overlaps one already peered to the Cloud VNet Use a non-overlapping range, or run from an already-peered VNet DB schema is invalid (with a gzip error in the logs) gzip decoder on a plain-text file, or the parser and container schemas don’t match Remove the decoder; match the schemas or use replace Two Create new container log lines Output tasks race on container creation min_output_tasks: 1 TXN_DENY_REQUEST … DENY(NoSQL) and a low row count Cloud rejects native writes under concurrent load, and the plugin doesn’t retry or resume Lower concurrency, and always check with COUNT(*) Scaling up: what happens under concurrent load 50,000 rows is a smoke test, so we wanted to push it harder. We generated a 10-million-row CSV with the same four-column schema and ran three loader processes at the same time, each writing to its own container (con_a, con_b, con_c). Each process had min_output_tasks: 1 set so the container-creation race from Gotcha #3 wasn’t a factor. The goal was to see how the native client and GridDB Cloud handle concurrent load. All three runs died within about 100 milliseconds of each other. Here are the row counts: Container Rows loaded Expected con_a 922,484 10,000,000 con_b 922,484 10,000,000 con_c 918,268 10,000,000 Each process dropped roughly 9 million rows. The logs show why: [10104:TXN_DENY_REQUEST] Request constraints occured, rule is DENY(NoSQL) (address=172.26.30.69:10001, partitionId=170) GridDB Cloud rejected the writes on the server side. DENY(NoSQL) is a Cloud constraint that refuses native (NoSQL interface) requests under concurrent load. All three processes hit it at the same moment, which is why they failed so close together and ended up with nearly identical row counts. The bigger problem is what the plugin does after that: nothing. There’s no retry and no resume. Any batches in flight are lost, and since the load runs in replace mode, you’re left with a container that only holds what was committed before the denial. The process exits, and unless you run a COUNT(*) afterward, it looks like the load worked. In our case it quietly lost 91% of the data. If you’re running this plugin against GridDB Cloud in production, keep this in mind: without a row-count check, a partial load looks the same as a successful one. Two smaller issues showed up in the same logs: PooledBufferAllocator$BufferDoubleReleasedException: Detected double release() call of a buffer: the plugin’s error handling releases a buffer twice when a write fails. It doesn’t matter much since the run is already dying, but it’s still a bug in the failure path. Cannot create container / DB schema is invalid: the three processes stepping on each other’s container operations after the denial. This is fallout from the main failure, not a separate bug. Takeaways Concurrent native-client writes to GridDB Cloud can trigger a TXN_DENY_REQUEST constraint. It’s worth checking your plan to see whether that’s a connection limit, a request-rate limit, or something tier-specific. The plugin doesn’t recover from a denial mid-load. No retry, no resume, and no clear signal that the load was incomplete. Always verify with a row count. replace mode makes this worse, because a failed load leaves you with a truncated container instead of an untouched one. Next we’ll narrow down the cause: one process loading all 10M rows (to rule out a hard volume cap), then two concurrent processes (to see if the denial still fires at lower concurrency). That should show where the concurrency limit actually sits, and we’ll cover it in a follow-up post. Embulk vs Fluentd: choosing the right tool None of this means Embulk is a bad tool. Embulk and Fluentd are built for different jobs, and what broke here was using a batch loader for something that behaves more like a streaming workload: a lot of data pushed at a rate-limited endpoint that pushes back. Fluentd is built for high-volume, continuous streaming ingestion. Data keeps arriving and has to be delivered reliably as it comes in. Fluentd keeps a buffer on the client side (in memory or on disk), so when GridDB pushes back, it holds onto the data and retries until GridDB accepts it. That’s why Fluentd handles the same rate and concurrency limits that broke the Embulk run. Embulk is built for bulk, batch transfers. Think one-time migrations or scheduled loads of large, fixed datasets between structured datastores. That’s where it has the edge: Batch migration is what it’s for. Moving a known dataset from A to B, like a database dump, a folder of CSV or Parquet files, or a table migration. If you need to load 50 million existing rows once, Embulk fits that job and Fluentd doesn’t. Lots of datastore connectors. Embulk has plugins for databases, files, and cloud storage (MySQL, PostgreSQL, S3, BigQuery, CSV, JSON, Parquet, and more). It’s designed to move data between systems, while Fluentd’s plugins lean toward log and event sources and observability backends. Explicit schema and type mapping. Embulk parses input with defined column types and per-column config (type conversion, timestamp formatting) and maps the source schema to the destination. Fluentd treats records as loosely typed events. You can check that it finished. A batch job has a start and an end, so you can confirm all N rows made it with a row count. Fluentd runs continuously, so “did everything arrive” doesn’t really apply. Repeatable transforms. The same input and config give you the same output, which is what you want for ETL and scheduled loads. In short: use Fluentd for continuous streaming, and Embulk for bulk or batch migration between structured systems. One caveat for this post specifically: the plugin’s lack of retry and resume makes Embulk a weaker fit for GridDB Cloud than it would otherwise be. It’s still the better tool for batch migration in general, but loading large volumes into GridDB Cloud with the current plugin means adding your own guardrails: run a single process, tune the batch size, and ideally wrap it in some retry logic. Against a self-hosted GridDB cluster, where those Cloud-side limits don’t apply, Embulk’s strengths come through a lot more

Introduction Vehicle emissions do not become dangerous in a single instant. The process is gradual. A diesel truck running a slightly rich fuel mixture will show elevated CO readings before its PM2.5 levels cross into dangerous territory. A CNG bus operating in stop-start city traffic will accumulate NOx gradually across a shift, not all at once. The problem with most fleet monitoring systems is not a lack of sensors. Modern vehicles carry several gas sensors, temperature probes, and air quality detectors. The real problem is that these sensors are treated as independent data points. A system that watches CO in isolation will miss the connection between rising CO and the PM2.5 spike that follows incomplete combustion. In this project, we did not just build a chart that shows numbers. We built a fleet emission monitoring system called EmissGuard that connects sensor readings across eight vehicles, tracks how emission patterns change over time, and identifies cross-sensor correlations that a single-reading alert would miss. GridDB stores every reading as a time-series record and makes it available for both real-time dashboard queries and machine learning training. Why Time-Series Databases Matter for Vehicle Emission Data Imagine a sensor reports an MQ135 reading of 520. Is that a problem? If the reading was 180 three minutes ago and has been climbing steadily, the vehicle is likely developing a combustion issue that needs immediate attention. If the reading has been between 490 and 530 for the past two hours, the vehicle may simply be operating under heavy load within its normal emission range for that condition. A single sensor reading is just a number. A sequence of readings over time tells the actual story. Fleet vehicles produce a continuous stream of timestamped sensor data. Traditional relational databases were designed for transactional records. They struggle when asked to store thousands of ordered sensor readings per minute and then retrieve the most recent window efficiently. Time-series databases are built for exactly this type of workload. GridDB stores each sensor reading with its UTC timestamp as the row key. Retrieving the last 30 readings for any vehicle is a single TQL query that returns results in microseconds. This makes it practical to run rolling-window analysis on every poll cycle without putting pressure on the database. Why GridDB for Fleet Emission Monitoring Fleet emission monitoring systems need a database that can handle high-frequency sensor ingestion from multiple vehicles simultaneously and return recent readings fast enough for real-time dashboard polling. GridDB is a memory-first NoSQL database built specifically for time-series workloads, which makes it a natural fit for this type of system. We use GridDB in EmissGuard to: Store continuous vehicle sensor data using TIME_SERIES containers with timestamp-based row keys. Maintain a fixed schema per vehicle for predictable and fast reads and writes. Ingest sensor batches from all eight vehicles in a single multi_put operation. Query recent time windows efficiently using TQL for rolling-window health analysis. Correlate multiple sensor streams (MQ2, CO, NOx, PM2.5) across time to detect emission patterns. Feed stored historical data directly into a scikit-learn machine learning training pipeline. Support explainable alerts by giving the dashboard fast access to recent sensor history per vehicle. Without GridDB: Rolling-window AQI analysis would require expensive full scans on every poll cycle. Batch ingestion from eight vehicles simultaneously would create write bottlenecks. The ML pipeline would have no reliable source of labelled time-series data to train on. System Architecture EmissGuard simulates a fleet of eight vehicles across different fuel types and vehicle categories. Each vehicle generates readings from eight sensors at 15-second intervals. The sensor data is generated by a Python simulator, stored in GridDB through a batch insert, and served to a monitoring dashboard through a Flask REST API. The system also detects cross-sensor emission correlations and produces a fleet-wide emission index in addition to per-vehicle AQI scores. The overall system consists of the following main components: Sensor Simulation Layer A Python-based simulator generates sensor readings for all eight vehicles. It produces MQ2, MQ135, CO2, CO, NOx, PM2.5, temperature, and humidity values at regular intervals using proportional interpolation and random noise to simulate how real vehicle sensors behave over time. Data Ingestion Layer The backend application receives the simulated readings and writes them into GridDB using the multi_put operation, which batches all eight vehicle containers into a single write request. GridDB Database GridDB stores the vehicle sensor data generated during the simulation. It acts as the central data layer from which the application retrieves recent readings when computing AQI scores and emission correlations. Monitoring Dashboard A web-based dashboard built with HTML, JavaScript, and Chart.js displays per-vehicle emission status, sensor history charts, fleet-wide emission index, and cross-sensor correlation alerts in real time. Setting Up GridDB (Local Installation) EmissGuard uses a locally installed GridDB instance on Ubuntu 24.04. After installing GridDB through the official Ubuntu package and starting the gridstore service, a Python application connects using the notification_member address. import griddb_python as griddb import os NOTIFICATION_MEMBER = os.environ.get("GRIDDB_NOTIFICATION_MEMBER", "127.0.0.1:10001") CLUSTER_NAME = os.environ.get("GRIDDB_CLUSTER_NAME", "myCluster") USERNAME = os.environ.get("GRIDDB_USERNAME", "admin") PASSWORD = os.environ.get("GRIDDB_PASSWORD", "admin") def get_gridstore(): factory = griddb.StoreFactory.get_instance() return factory.get_store( notification_member=NOTIFICATION_MEMBER, cluster_name=CLUSTER_NAME, username=USERNAME, password=PASSWORD, ) All connection parameters are read from environment variables so the project runs on any GridDB installation without code changes. The defaults match a standard local setup. Once connected, the application can create containers and begin storing vehicle sensor data. Project Overview EmissGuard monitors eight vehicles, each with its own sensor profile, warning thresholds, and GridDB container. The profiles are defined in a central configuration file called vehicles.py, which every other module imports from. This is the single source of truth for sensor ranges, critical thresholds, container names, and vehicle metadata. VEHICLES = { "VH001": { "display_name": "Truck Alpha", "vehicle_type": "Heavy Truck", "fuel_type": "Diesel", "container": "emissguard_vh001", "description": "Long-haul freight truck. Diesel engine, high NOx emitter.", "normal": { "mq2": (100, 300), "mq135": (100, 350), "co2_ppm": (400, 600), "co_ppm": (0, 9), "nox_ppb": (10, 80), "pm25_ugm3": (0, 15), "temperature": (55, 90), "humidity": (35, 65), }, "thresholds": { "mq2": {"warning": 450, "critical": 700}, "mq135": {"warning": 500, "critical": 750}, "co2_ppm": {"warning": 1000, "critical": 2000}, "co_ppm": {"warning": 15, "critical": 35}, "nox_ppb": {"warning": 150, "critical": 300}, "pm25_ugm3": {"warning": 35, "critical": 75}, "temperature": {"warning": 120, "critical": 160}, "humidity": {"warning": 80, "critical": 95}, }, }, # VH002 through VH008 follow the same structure … } Key ideas behind the system: Fleet-wide monitoring: Instead of analyzing each vehicle in isolation, the system observes all eight vehicles together and can detect when multiple vehicles show concurrent emission stress. Pattern detection: It identifies small but meaningful changes in sensor values over time that may signal a developing combustion issue, rather than reacting only to single-point threshold crossings. Cross-sensor correlation awareness: It models how one emission measurement can drive another. High CO often means incomplete combustion, which produces more PM2.5. Rising exhaust temperature promotes NOx formation. These relationships are checked on every analysis cycle. Simulating Vehicle Sensor Data Since real vehicle exhaust sensors were not available, the system uses a sensor simulator to generate realistic emission readings. The simulator creates data for eight vehicles across four fuel types: Diesel, Petrol, and CNG. The dataset is generated in four phases: Safe: All vehicles operate within normal emission limits. Moderate: VH001 (Truck Alpha) and VH007 (Lorry Eta) begin showing elevated readings. Poor: Both heavy vehicles enter the poor zone; VH004 and VH002 enter moderate. Dangerous: VH001 and VH007 cross critical thresholds; other vehicles show a cascade of elevated readings. This models how emission events in real fleets often start with the heaviest vehicles and spread across the fleet through shared operational conditions. To make the simulation physically realistic, the system avoids generating flat or hardcoded values. Instead it uses proportional interpolation and per-sensor random noise to calculate values dynamically: def _interpolate(low, high, progress=None): """Smoothly interpolate between two values with optional random progress.""" if progress is None: progress = random.uniform(0.0, 1.0) return low + progress * (high – low) def generate_reading(vehicle_id, status="Safe", progress=0.0, timestamp=None): cfg = VEHICLES[vehicle_id] nrm = cfg["normal"] thr = cfg["thresholds"] vals = {k: random.uniform(*nrm[k]) for k in nrm} noise = { "mq2": random.uniform(-12, 12), "mq135": random.uniform(-12, 12), "co2_ppm": random.uniform(-15, 15), "co_ppm": random.uniform(-0.5, 0.5), "nox_ppb": random.uniform(-4, 4), "pm25_ugm3": random.uniform(-1.5, 1.5), "temperature": random.uniform(-2, 2), "humidity": random.uniform(-2, 2), } if status == "Moderate": prog = random.uniform(0.2, 0.55) for key in ("mq2", "mq135", "co2_ppm", "co_ppm", "nox_ppb", "pm25_ugm3"): vals[key] = _interpolate(nrm[key][1], thr[key]["warning"], prog) elif status == "Poor": prog = random.uniform(0.45, 0.85) for key in ("mq2", "mq135", "co2_ppm", "co_ppm", "nox_ppb", "pm25_ugm3"): vals[key] = _interpolate(thr[key]["warning"], thr[key]["critical"], prog * 0.7) elif status == "Dangerous": prog = random.uniform(0.8, 1.25) for key in ("mq2", "mq135", "co2_ppm", "co_ppm", "nox_ppb", "pm25_ugm3"): overshoot = thr[key]["critical"] + ( thr[key]["critical"] – thr[key]["warning"] ) * (prog – 1.0) * 0.4 vals[key] = max(thr[key]["warning"], overshoot) # … This approach guarantees that simulated sensors drift naturally within a range and occasionally spike, rather than sitting at a fixed median value. Live Escalation Simulation To demonstrate how emission events develop over time, the project includes a Python script that simulates a live escalation across the fleet. Instead of inserting a full dataset all at once, the script gradually sends sensor readings to GridDB to show how emission problems evolve in real vehicle fleets. The simulation follows four phases: Safe Operation: All vehicles emit within normal limits. Moderate Escalation: VH001 and VH007 begin showing elevated MQ2 and CO2. Poor Condition: Both heavy vehicles enter the poor zone; medium vehicles pick up elevated readings. Dangerous Cascade: VH001 and VH007 cross critical thresholds; the fleet emission index climbs sharply. During the simulation, new sensor readings are continuously inserted into GridDB. This allows the dashboard to react in real time as the escalation develops across vehicles. A simplified example of the escalation loop is shown below: def trigger_emission(): store = insert_data.get_gridstore() start_sim() i = 0 while is_sim_active(): ts = datetime.now(timezone.utc) phase = min(i // 4, 3) batch = {} for vid, cfg in VEHICLES.items(): status = PHASE_PLAN[phase].get(vid, "Safe") r = generate_reading(vid, status, timestamp=ts) batch[cfg["container"]] = [[ ts, r["mq2"], r["mq135"], r["co2_ppm"], r["co_ppm"], r["nox_ppb"], r["pm25_ugm3"], r["temperature"], r["humidity"], r["aqi_status"], ]] store.multi_put(batch) i += 1 time.sleep(1) When the operator clicks the “Issue Resolved” button in the dashboard, the backend sets the simulation flag to False. The escalation loop stops and the script injects a final batch of Safe readings into GridDB. These clean readings immediately restore the fleet to a normal state on the dashboard without any manual intervention. Storing Vehicle Sensor Data in GridDB After generating the simulated sensor readings, the data is stored in GridDB so the monitoring system can access it. Each vehicle is assigned its own TIME_SERIES container, where readings for all eight sensors are stored with a UTC timestamp as the row key. This allows the system to retrieve vehicle sensor history in chronological order using a single TQL query. Creating a TIME_SERIES Container def setup_containers(store): for vid, cfg in VEHICLES.items(): con_info = griddb.ContainerInfo( cfg["container"], [ ["timestamp", griddb.Type.TIMESTAMP], ["mq2", griddb.Type.DOUBLE], ["mq135", griddb.Type.DOUBLE], ["co2_ppm", griddb.Type.DOUBLE], ["co_ppm", griddb.Type.DOUBLE], ["nox_ppb", griddb.Type.DOUBLE], ["pm25_ugm3", griddb.Type.DOUBLE], ["temperature", griddb.Type.DOUBLE], ["humidity", griddb.Type.DOUBLE], ["aqi_status", griddb.Type.STRING], ], griddb.ContainerType.TIME_SERIES, ) store.put_container(con_info) print(f" Container ready: {cfg[‘container’]}") The put_container call is idempotent. Calling it on an existing container returns a reference to it without losing any stored data. This makes it safe to run insert_data.py multiple times during development and testing. Inserting Data into GridDB def insert_dataset(store, dataset): batch = {} for vid, readings in dataset.items(): container_name = VEHICLES[vid]["container"] rows = [] for r in readings: ts = datetime.fromisoformat(r["timestamp"].replace("Z", "+00:00")) rows.append([ ts, r["mq2"], r["mq135"], r["co2_ppm"], r["co_ppm"], r["nox_ppb"], r["pm25_ugm3"], r["temperature"], r["humidity"], r["aqi_status"], ]) batch[container_name] = rows store.multi_put(batch) The multi_put operation writes rows to multiple containers in a single request. This significantly improves ingestion throughput when handling sensor data from eight vehicles simultaneously. Querying Sensor Data and Computing AQI Scores Once vehicle sensor data is stored in GridDB, the monitoring system queries recent readings to evaluate each vehicle’s emission status. The system retrieves the latest records for each vehicle and computes an AQI score based on how far the sensor values have drifted from normal ranges. GridDB’s Time-Series Query Language (TQL) allows the system to fetch the most recent readings from each vehicle container efficiently: def query_recent(store, vehicle_id, limit=30): container = store.get_container(VEHICLES[vehicle_id]["container"]) query = container.query( f"select * order by timestamp desc limit {limit}" ) rs = query.fetch() readings = [] while rs.has_next(): row = rs.next() readings.append({ "timestamp": row[0].isoformat(), "mq2": row[1], "mq135": row[2], "co2_ppm": row[3], "co_ppm": row[4], "nox_ppb": row[5], "pm25_ugm3": row[6], "temperature": row[7], "humidity": row[8], "aqi_status": row[9], }) return readings To prevent sudden status changes caused by a single noisy sensor reading, the system evaluates vehicle conditions using a rolling window of the five most recent readings. Averaging the latest five values makes the monitoring logic stable and resistant to temporary spikes: window = readings[:5] avgs = { k: sum(r[k] for r in window) / len(window) for k in ("mq2","mq135","co2_ppm","co_ppm","nox_ppb","pm25_ugm3","temperature","humidity") } AQI Risk Scoring Instead of a simple status label, each vehicle receives a continuous AQI score from 0 to 100 based on exactly how far its sensor values have drifted from safe ranges. A vehicle with CO at 18 ppm (just above the 15 ppm warning threshold) and a vehicle with CO at 40 ppm (past the 35 ppm critical threshold) both count as elevated in a simple threshold system. With continuous scoring, the second vehicle scores significantly higher because its readings are more serious. def sensor_severity(value, normal_max, warning, critical): if value <= normal_max: return 0.0 elif value <= warning: return 0.30 * (value – normal_max) / max(warning – normal_max, 1e-9) elif value <= critical: return 0.30 + 0.70 * (value – warning) / max(critical – warning, 1e-9) else: return min(1.0 + ((value – critical) / max(abs(critical – warning), 1e-9)) * 0.5, 1.5) def vehicle_aqi_score(vehicle_id, avgs): thr = VEHICLES[vehicle_id]["thresholds"] nrm = VEHICLES[vehicle_id]["normal"] scores = { "mq2": sensor_severity(avgs["mq2"], nrm["mq2"][1], thr["mq2"]["warning"], thr["mq2"]["critical"]), "mq135": sensor_severity(avgs["mq135"], nrm["mq135"][1], thr["mq135"]["warning"], thr["mq135"]["critical"]), "co2_ppm": sensor_severity(avgs["co2_ppm"], nrm["co2_ppm"][1], thr["co2_ppm"]["warning"], thr["co2_ppm"]["critical"]), "co_ppm": sensor_severity(avgs["co_ppm"], nrm["co_ppm"][1], thr["co_ppm"]["warning"], thr["co_ppm"]["critical"]), "nox_ppb": sensor_severity(avgs["nox_ppb"], nrm["nox_ppb"][1], thr["nox_ppb"]["warning"], thr["nox_ppb"]["critical"]), "pm25_ugm3": sensor_severity(avgs["pm25_ugm3"], nrm["pm25_ugm3"][1], thr["pm25_ugm3"]["warning"], thr["pm25_ugm3"]["critical"]), } weighted = sum(scores[k] * SENSOR_WEIGHTS[k] for k in SENSOR_WEIGHTS) return min(round(weighted * 100), 100) The fleet-wide emission index is a weighted average across all eight vehicles. Heavier diesel vehicles like VH001 and VH007 carry higher weights because their emissions have a greater impact on the overall fleet footprint. Active emission correlations add a penalty of 8 points each, up to a maximum of 25 additional points. Emission Correlation Detection The system understands that emission sensors are chemically connected. Elevated CO usually indicates incomplete combustion, which produces more particulate matter. High exhaust temperature drives NOx formation. These relationships are defined as rules in vehicles.py: EMISSION_RULES = [ { "source": "co_ppm", "target": "pm25_ugm3", "message": ( "Incomplete Combustion: Elevated CO is correlated with rising PM2.5. " "Rich fuel mixture may be causing unburnt particulate matter in exhaust." ), }, { "source": "temperature", "target": "nox_ppb", "message": ( "Thermal NOx Formation: High exhaust temperature is correlated with " "elevated NOx levels. High-temperature combustion promotes nitrogen oxide formation." ), }, { "source": "mq2", "target": "co2_ppm", "message": ( "Rich Mixture Alert: High MQ2 reading is correlated with elevated CO2. " "Fuel-rich combustion detected." ), }, ] When a vehicle’s source sensor is in a Poor or Dangerous state, a correlation alert is raised on the dashboard showing which secondary sensor is likely to be affected next. Continuous Telemetry Stream In addition to inserting the initial dataset, the system supports a background heartbeat producer that continuously sends Safe readings to GridDB every five seconds. This keeps the dashboard feed alive with fresh data between simulation runs. python insert_data.py –live When the emission simulation is triggered, the producer can be left running. Its Safe readings will be overwritten by the simulation’s escalating data during the active simulation window. Once the simulation ends and the Issue Resolved button is clicked, the system injects a recovery batch of Safe readings and the dashboard returns to a clean state immediately. Building the Monitoring Dashboard The EmissGuard dashboard is a single HTML file served directly by the Flask application. It uses Chart.js for time-series visualization and polls the backend API every four seconds. The interface displays per-vehicle emission cards, AQI score bars, sensor history charts across three tab panels, emission correlation alerts, and a fleet-wide escalation timeline. The backend exposes a set of simple API endpoints that the dashboard polls periodically: Endpoint What it returns GET /api/fleet Status of all vehicles + correlations + emission index GET /api/vehicle/<id> Sensor history for one vehicle (used for charts) GET /api/stats/<id> Min, max, and average per sensor over recent history GET /api/timeline Log of emission status-change events GET /api/latest Single most recent reading per vehicle Below is a simplified example of how the dashboard fetches fleet data and updates the interface: async function pollFleet() { try { const res = await fetch(‘/api/fleet’); const data = await res.json(); updateStrip(data); buildSidebar(data); updateCorr(data); if (!selectedVid) buildCards(data); document.getElementById(‘lastTs’).textContent = new Date().toLocaleTimeString(‘en-IN’, {hour12: false}); } catch(e) { console.warn(‘Poll error:’, e); } } The complete implementation can be found in the project repository. The dashboard calls the fleet endpoint every four seconds to stay synchronized with the latest vehicle data stored in GridDB. The vehicle detail view fetches sensor history from /api/vehicle/<id> and renders eight Chart.js line charts across three tabs: Primary Sensors (MQ2 and MQ135), Emission Gases (CO2, CO, NOx, PM2.5), and Temperature plus Humidity. Running the Project Make sure GridDB is running and credentials are configured as environment variables. Then run the following in order: # Step 1: Create containers and seed historical data (1,480 rows across 8 vehicles) python src/insert_data.py # Step 2: Verify that data was stored correctly python src/query_data.py # Step 3: Start the monitoring backend and dashboard python src/app.py # Step 4 (optional): Keep the dashboard alive with a live heartbeat python src/insert_data.py –live # Step 5 (optional): Trigger a live escalation demonstration python src/simulate_emission.py Open http://localhost:5050 to view the EmissGuard dashboard. The project uses two separate data-feeding mechanisms to make the demonstration repeatable: The Background Heartbeat (insert_data.py –live): This script provides a stable Safe baseline so the dashboard stays active and shows green status by default. The Manual Escalation (simulate_emission.py): This script temporarily takes over with a four-phase emission event to show how pollution builds across the fleet. One of the practical advantages of using GridDB here is the automatic recovery behavior. Once the escalation script finishes and the operator clicks Issue Resolved, the background producer continues sending Safe data. These new clean readings overwrite the elevated values in the rolling window and the dashboard returns to a normal state without requiring any manual reset. Results and Dashboard Overview After running the system, the EmissGuard dashboard displays the real-time emission status of all eight vehicles. The interface shows sensor readings for MQ2, MQ135, CO2, and PM2.5 on the fleet card view, along with the AQI score and current status badge for each vehicle. Safe Operation In the safe state, all eight vehicles operate within their normal emission ranges. The dashboard displays a low fleet emission index and all vehicle cards show a green Safe badge. The emission correlation panel shows no active alerts. Moderate and Poor Conditions As the simulation advances, VH001 and VH007 begin showing elevated readings. The dashboard updates their status badges to Moderate and then Poor. The fleet emission index rises as the weighted average reflects the degrading sensor values. When two or more sensors on a vehicle cross their warning thresholds simultaneously, the system detects an emission correlation and displays it in the correlation panel with a description of the likely chemical cause. Dangerous Cascade When VH001 and VH007 enter the Dangerous state, the fleet emission index climbs sharply. The correlation panel shows multiple active alerts. The escalation timeline at the bottom of the dashboard logs every status-change event with a timestamp, vehicle name, and fuel type. The operator can click Issue Resolved to inject recovery data and watch the system return to normal in real time. Vehicle Detail View Clicking any vehicle card opens a detail panel with four summary statistics (AQI score, average CO2, average CO, and maximum PM2.5) and eight Chart.js line charts organized across three tabs. The charts render up to 200 historical readings pulled from GridDB through the /api/vehicle/<id> endpoint, giving a clear picture of how each sensor has behaved over the monitored period. Conclusion Vehicle emissions do not usually become a problem at a single moment in time. The readings build gradually. A system that only checks whether a value is above a fixed threshold today will miss the trend that has been developing over the past ten minutes. In this project, we built EmissGuard, a fleet emission monitoring prototype using GridDB to store and query vehicle sensor data. By combining simulated multi-sensor readings with rolling-window analysis, weighted AQI scoring, and cross-sensor correlation rules, the system can detect early emission patterns before individual sensors cross their critical thresholds. GridDB TIME_SERIES containers, the multi_put batch ingestion operation, and TQL time-window queries made it straightforward to build a system that ingests data from eight vehicles simultaneously, analyzes it on every poll cycle, and feeds it into an ML training pipeline, all from a simple Python codebase with no external dependencies beyond the GridDB client and Flask. The complete implementation can be found in the project repository: github.com/DoneByManthan/GridDB-EmissGuard In the future, this approach could be extended with anomaly detection models trained on the historical GridDB data to identify unusual emission signatures that do not fit any predefined rule. Per-vehicle baseline learning, where the system builds a personal normal profile from the first week of data for each vehicle, would make the threshold system more accurate for mixed fleets where different vehicles have genuinely different operating profiles. Integration with real OBD-II sensor hardware would allow the same codebase to transition from simulation to production monitoring without changes to the GridDB storage or analysis

A household electricity meter, read once a minute, turns out to be a surprisingly candid sensor. Over four years it records when people wake up, when they cook, when they travel, and how hard the winter bites. The only real requirement is somewhere to keep two million timestamped readings and a quick way to ask questions of them. This post takes the UCI Individual Household Electric Power Consumption dataset, keeps it in GridDB as a time-series store, and then digs into what it says about how one household near Paris ran its home between December 2006 and November 2010. It is all Java: GridDB’s Web API for the data, and JFreeChart for the visuals. GridDB is the database behind it, holding the series and doing the time-based aggregation. Everything is reproducible, and the code is on GitHub (link at the end). The dataset The data is a public release from the UCI Machine Learning Repository: 2,075,259 one-minute measurements from a single home. Each row carries the whole-house active power draw in kilowatts, plus voltage, current, and three sub-meters: Sub-meter 1, kitchen: dishwasher, oven, microwave Sub-meter 2, laundry: washing machine, tumble dryer, fridge, a light Sub-meter 3, climate: an electric water heater and an air conditioner Two million minute-level rows is finer than we need to see daily and seasonal behaviour, so a short resampling step (in the repo) rolls them up to hourly: the mean for the instantaneous quantities, and the sum for the sub-meter energy counters. That leaves 34,168 clean hourly rows, which is what lands in GridDB. The data lives in GridDB Loading is two calls against the Web API: create a TIME_SERIES container with a schema, then PUT the rows in batches. On the free Cloud tier there is no driver to install; Java’s built-in HttpClient sends JSON over HTTPS. client.createTimeSeriesContainer("home_power_hourly", List.of( new Column("ts", "TIMESTAMP"), new Column("active_power_kw", "DOUBLE"), new Column("voltage_v", "DOUBLE"), // …current and the three sub-meters )); client.putRows("home_power_hourly", rows, 2000); // 34k rows, ~18 requests From there the analysis reads the series back out of the database. A single SQL SELECT streams every row into a list of typed records, and one TQL call asks GridDB for a time-weighted average without pulling anything client-side: public List<Reading> fetchAll(String container) throws Exception { JsonNode results = sqlSelect("SELECT * FROM " + container + " ORDER BY ts") .get(0).get("results"); List<Reading> readings = new ArrayList<>(results.size()); for (JsonNode r : results) { readings.add(new Reading(Instant.parse(r.get(0).asText()), r.get(1).asDouble(), r.get(2).asDouble(), r.get(3).asDouble(), r.get(4).asDouble(), r.get(5).asDouble(), r.get(6).asDouble())); } return readings; } client.tql("home_power_hourly", "select TIME_AVG(active_power_kw)"); // -> 1.092 (kW, time-weighted across the full four years) So over four years this home pulled about 1.1 kW on average. That single number hides everything interesting, though, and the hourly detail is where the story lives. Why a database, not a flat file You could read the CSV straight into memory, so it is fair to ask what GridDB buys you here. The answer shows up when the dataset stops being one home. A TIME_SERIES container keeps rows ordered by their timestamp, so a query for “every reading in March 2008” or “the hourly average for this meter” reads only the slice it needs instead of walking a file end to end, and an aggregate like the average above is computed inside the database rather than in the JVM. This one home is two million raw readings over four years; a thousand of them reporting every minute would add more than a million rows a day. That is the scale where flat files stop being an option, and the code you write against one home is the code you run against a thousand. Charting with JFreeChart Each chart is the same handful of steps: group the readings that GridDB handed back, drop the numbers into a JFreeChart dataset, and save a PNG. The daily load curve, for instance: double[] mean = meanByHour(readings, Reading::activePowerKw); // plain loop over the records XYSeries s = new XYSeries("Mean active power"); for (int h = 0; h < 24; h++) s.add(h, mean[h]); JFreeChart chart = ChartFactory.createXYLineChart( "Average electricity demand by hour of day", "Hour of day", "Mean active power (kW)", new XYSeriesCollection(s), PlotOrientation.VERTICAL, false, false, false); ChartUtils.saveChartAsPNG(new File("images/daily_load_curve.png"), chart, 800, 450); With that pattern in place, here is what four years of this home looks like. The daily rhythm Group every hour by its position in the day and you get the household’s average load curve, and it reads like a daily routine. Demand bottoms out at 0.44 kW around 4 AM, when little more than the fridge and standby loads are running. It climbs through a morning bump just after 7 AM (about 1.5 kW), eases off through the working day, then builds to a hard evening peak of 1.90 kW at 8 PM: dinner, appliances, lights, everyone home. That is a 4.3x swing between the quietest and busiest hours of an average day, and it is exactly the kind of peak that utilities build their pricing around. Weekdays versus weekends Splitting the week apart separates two routines. Weekdays show the sharper, later evening spike you would expect from people arriving home from work, at 1.85 kW around 9 PM. Weekends run higher overall and peak earlier, at 2.05 kW near 7 PM, and they never sink into the flat mid-day lull that weekdays have, because someone is home using the house all afternoon. The seasons Zoom out to a monthly average and the seasons take over. Winter months average 1.44 kW against 0.73 kW in summer, very nearly double. In a home whose largest metered load is an electric water heater, that fits: cold incoming water takes far more energy to bring up to temperature, and the darker months add lighting on top. The hour-by-month heatmap is my favourite view of the whole dataset. Each column is a calendar month, each row an hour of the day, and colour is average power. The bright evening band across hours 19 to 21 is present all year but blazes from November through March. Then look at month 8: August is dark at every single hour. This household effectively switched off for the month, the classic summer holiday written straight into the electricity. Where the power actually goes Stacking the three sub-meters by hour shows which circuits drive the day. The climate circuit (water heater and AC) dominates, with a big morning surge as the tank reheats after overnight use. The kitchen spikes at the obvious moments, a small bump at lunch and a larger one at dinner. Laundry stays spread across the day and peaks in the early afternoon, since the always-on fridge sits on that circuit too. The most revealing figure, though, is the one that is not on any chart. Add up all three sub-meters across four years and they account for only 48.8% of the home’s total energy. The other 51.2%, which is lighting, wall sockets, electronics, and everything not on a named circuit, is simply invisible to the sub-meters. If you are building anything that reasons about home energy, that number is worth remembering: the circuits you can see may be the minority of what you are trying to explain. Of the metered half, the climate circuit alone is roughly 13,200 kWh, against about 2,700 for laundry and 2,300 for the kitchen. Wrapping up None of this analysis is exotic; it is grouping and averaging a few arrays. It works cleanly only because the data sits somewhere that makes time-slicing cheap. GridDB held the four-year series as one TIME_SERIES container, handed the whole thing back in a single query, and computed the time-weighted average server-side, leaving Java to do the grouping and JFreeChart the drawing. From here the same container feeds the obvious next steps, none of which need the raw file again: a short-term load forecast, an anomaly detector that flags an hour drawing far more than its usual for that time of day and season, or a live dashboard that reads the most recent hours straight from GridDB. The complete project, from the resampled data to the five charts, is on GitHub. Point it at your own free GridDB Cloud instance, set three environment variables, and you will get the same charts. Full source and data: github.com/radiantmind125/griddb-energy-java Dataset: Individual Household Electric Power Consumption, UCI Machine Learning

Introduction Forget the old times when you had to manually create infographics. With Nano Banana Pro, you can now create stunning infographics in minutes. Nano Banana Pro is an advanced AI image generation and editing model developed by Google DeepMind as part of the Gemini AI family. It’s essentially a professional grade tool for creating high-quality visuals—like illustrations, infographics, mockups, and edited photos using natural language text prompts or reference images. In this blog post, we will build a web application that leverages Nano Banana Pro to generate stunning infographics from any accessible content such as articles, and store the generated infographics data using GridDB as the high-performance database backend. Running the Application To run this application, you need to: Clone this repository and run the following commands: $ git clone https://github.com/junwatu/infographic-nbp-griddb.git $ cd infographic-nbp-griddb/apps $ npm install Set up the environment variables. Copy the .env.example file to .env and fill in the values: $ WEB_URL=http://localhost:3000 $ # GridDB Web API Configuration (Primary) $ GRIDDB_WEBAPI_URL="" $ GRIDDB_USERNAME="" $ GRIDDB_PASSWORD="" $ # DeepSeek API key through Anthropic-compatible endpoint $ NEBIUS_API_KEY="" $ # Nano Banana Pro through Fal API $ FAL_KEY="" Start the application: $ npm run dev Open http://localhost:3000 in your browser and paste an article URL to generate an infographic. Please note that the article should be publicly accessible from the internet and not behind a paywall. System Architecture A user submits an input through the browser, which is sent to a Node.js backend that contains two core components: a Content Retriever and a Prompt AI Agent (DeepSeek-V3.2). The Content Retriever enriches or contextualizes the user request (e.g., fetching references, templates, or prior data), after which the Prompt AI Agent constructs a structured prompt. The prompt, along with image metadata, is persisted to a GridDB database for future retrieval. The finalized prompt is then forwarded to an external image generation AI model, Nano Banana Pro, which produces the image artifact and stores it in a file system. The generated image is subsequently served back to the browser, closing the loop. Technologies Used Node.js This project is built using React + Vite, which requires Node.js version 16 or higher. You can download and install Node.js from nodejs.org. Nano Banana Pro API Key You need to create a Google API key to use the Nano Banana Pro. In this project we use FAL inference provider to access the Nano Banana Pro API. Please note that using the Nano Banana Pro API requires a paid subscription plan. DeepSeek-V3.2 API Key The latest version of DeepSeek model is V3.2, which is a recently released large language model (LLM) developed by the Chinese AI company DeepSeek, specifically optimized for coding, agentic workflows, and tool-use tasks rather than just general text chat. To use the DeepSeek-V3.2 API, you must obtain an API key from the Nebius Platform. Simply create an account and generate your key from the dashboard. Please note that an active subscription plan is required to access the API features. > The application uses the OpenAI Agents SDK which connects to DeepSeek-V3.2 via an OpenAI-compatible base URL. This is why the environment variable is named NEBIUS_API_KEY. GridDB Sign Up for GridDB Cloud Free Plan If you would like to sign up for a GridDB Cloud Free instance, you can do so at the following link: https://form.ict-toshiba.jp/download_form_griddb_cloud_freeplan_e. After successfully signing up, you will receive a free instance along with the necessary details to access the GridDB Cloud Management GUI, including the GridDB Cloud Portal URL, Contract ID, Login, and Password. GridDB WebAPI URL Go to the GridDB Cloud Portal and copy the WebAPI URL from the Clusters section. It should look like this: GridDB Username and Password Go to the GridDB Users section of the GridDB Cloud portal and create or copy the username for GRIDDB_USERNAME. The password is set when the user is created for the first time. Use this as the GRIDDB_PASSWORD. For more details, to get started with GridDB Cloud, please follow this quick start guide. IP Whitelist When running this project, please ensure that the IP address where the project is running is whitelisted. Failure to do so will result in a 403 status code or forbidden access. You can use a website like What Is My IP Address to find your public IP address. To whitelist the IP, go to the GridDB Cloud Admin and navigate to the Network Access menu. AI Agents This application uses a multi-agent architecture powered by the OpenAI Agents SDK with DeepSeek-V3.2 as the underlying language model. The system consists of two specialized agents that work together in a coordinated pipeline. Agent Architecture Content Fetcher Agent The Content Fetcher is a specialized agent responsible for retrieving and cleaning article content from URLs. It’s extract the main content from web pages, removing ads, navigation, and other clutter. const createContentAgent = (model: any) => { return new Agent({ name: "Content Fetcher", instructions: "You are an expert at fetching and cleaning article content from URLs.", model, tools: [fetchArticleTool], }); }; The fetchArticleTool internally calls the fetchAndSanitize function which: Fetches the HTML from the provided URL Parses it using JSDOM Extracts the main content using Readability Returns a simplified object containing the title and textContent to the agent. Infographic Orchestrator Agent The Orchestrator is the manager agent that coordinates the entire infographic generation pipeline. It has access to two tools: fetch_article – The Content Fetcher agent exposed as a tool image_generator – Generates images using Nano Banana Pro via Fal const createOrchestratorAgent = (model: any, contentAgent: Agent) => { return new Agent({ name: "Infographic Orchestrator", instructions: ` You are the manager of an infographic generation pipeline. Your goal is to take a URL, fetch the content, create a prompt, and finally generate an image. Follow these steps strictly: 1. Use 'fetch_article' to get the article content. 2. Generate a detailed infographic prompt based on that content. 3. Call 'image_generator' with the prompt to create the final image. `, model, tools: [ contentAgent.asTool({ toolName: "fetch_article", toolDescription: "Fetches article content from a URL.", }), imageGeneratorTool, ], }); }; The Pipeline Flow When generateInfographicForUrl(url) is called, the orchestrator executes the following pipeline: const result = await run( orchestratorAgent, `Follow the infographic pipeline for this article: ${url} Step 1: Fetch the article content. Step 2: Generate a detailed prompt for an infographic. Step 3: CALL the 'image_generator' tool with that prompt. (MANDATORY) Finally, return ONLY a breakdown as JSON in this format: { "title": "Article Title", "image": "The generated image URL", "article_url": "${url}", "description": "Short summary", "prompt": "The full detailed prompt you used for 'image_generator'" }`, { maxTurns: 10 } ); The result is a structured JSON object containing: Field Description title Article title image Generated infographic URL article_url Original article URL description Short summary prompt The full prompt used for image generation Prompt Engineering The key to generating high-quality infographics lies in the prompt engineering. The orchestrator agent is instructed to create a detailed infographic prompt based on the article content. This prompt should include: Key statistics and data points from the article Visual layout suggestions (sections, hierarchy) Color scheme recommendations Typography and iconography hints The main message or takeaway The DeepSeek-V3.2 model excels at this task because it’s specifically optimized for agentic workflows and tool-use, making it ideal for following the multi-step pipeline instructions reliably. GridDB Data Layer The application uses a layered approach for database operations, with two key files: griddb-base.ts for database configuration and griddb.ts for infographic-specific operations. Database Schema The infographic container stores generated infographic metadata: Column Type Description id INTEGER Unique record identifier (rowkey) prompt STRING AI prompt used for image generation infographic STRING URL of the generated image article_url STRING Source article URL generation_timestamp TIMESTAMP When the record was created Base Configuration The base module (griddb-base.ts) handles database initialization with retry support: export const GRIDDB_CONFIG: GridDBConfig = { griddbWebApiUrl: process.env.GRIDDB_WEBAPI_URL || '', username: process.env.GRIDDB_USERNAME || 'admin', password: process.env.GRIDDB_PASSWORD || 'admin', timeout: 30000, retryAttempts: 3, retryDelay: 1000 }; The module also provides ensureContainer() which automatically creates the container with the defined schema if it doesn’t exist during server startup. CRUD Operations The main GridDB module (griddb.ts) exposes these functions: Function Description getInfographics(limit) Retrieve records ordered by timestamp (default: 100) getInfographicById(id) Get a single record by ID deleteInfographic(id) Remove a record by ID > Note: The saveInfographic() function is demonstrated in the Generate Infographic & Save to GridDB section below. Server Side API Routes Method Endpoint Description Request Body GET /health Server health check N/A POST /generate-infographic Generate and save infographic from URL { “url”: “string” } POST /api/infographics Manually save infographic record Infographic Object GET /api/infographics Retrieve all infographic records N/A GET /api/infographics/:id Retrieve a specific record by ID N/A DELETE /api/infographics/:id Delete a specific record by ID N/A Generate Infographic & Save to GridDB There are few endpoints available for the server and the most used is /generate-infographic route. This route is used to generate an infographic from a URL and also save the record to GridDB. This route will call the generateInfographicForUrl function from the orchestrator, which is an AI agent that orchestrates the infographic generation pipeline. More on this later. const result = await generateInfographicForUrl(url); Then if the image generation is successful, the result will be saved to GridDB. try { await saveInfographic({ id: 0, // ID is generated inside saveInfographic prompt: result.prompt, infographic: result.image, article_url: result.article_url || url, generation_timestamp: new Date().toISOString() }); } catch (dbError) { console.error('[Server] Failed to save to GridDB, but continuing:', dbError); } > Please note that the infographic record is an image URL instead of the image itself. The server also provide manual save endpoint /api/infographics to save infographic record to GridDB using POST request. This is useful for testing and debugging. Get All Infographics The GET /api/infographics endpoint retrieves all infographic records from GridDB by calling the getInfographics() wrapper function. Test with curl: $ curl -s http://localhost:3000/api/infographics Get Infographic by ID The GET /api/infographics/:id endpoint retrieves a specific infographic record. It includes validation for the ID parameter and proper error handling: const id = parseInt(req.params.id); if (isNaN(id)) { return res.status(400).json({ error: 'Invalid ID format' }); } const data = await getInfographicById(id); if (!data) { return res.status(404).json({ error: 'Infographic not found' }); } res.json(data); Test with curl: $ curl -s http://localhost:3000/api/infographics/368368397 Delete Infographic by ID The DELETE /api/infographics/:id endpoint removes a specific infographic record from GridDB. It follows the same ID validation pattern as the GET endpoint, then calls deleteInfographic(id) to remove the record. Test with curl: $ curl -X DELETE http://localhost:3000/api/infographics/368368397 > All endpoints follow a consistent pattern: validate input, call the appropriate GridDB wrapper function, and return JSON responses with proper HTTP status codes. Client Side The frontend is built with React and Vite, providing a fast and responsive user interface for generating infographics. UI Components The application uses a set of reusable UI components located in src/components/ui/: Component Description Button Primary action button with loading state support Card Container component with header, content, and footer sections Input Text input field with icon support Skeleton Loading placeholder for content Application Flow The main App.tsx component manages the entire user interaction flow: 1. URL Input Users enter an article URL into the input field. The form submission triggers the handleGenerate function: const handleGenerate = async (e: React.FormEvent) => { e.preventDefault() setLoading(true) const response = await fetch('/generate-infographic', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ url }), }) const data = await response.json() setResult(data.result) } 2. Loading State While the infographic is being generated, a skeleton loader is displayed with an animated message: {loading && ( <Skeleton className="h-[400px] w-full rounded-xl" /> <p className="animate-pulse">Analyzing content and generating visuals…</p> )} 3. Result Display Once generated, the infographic is displayed in a styled card with: Article title and description The generated infographic image Link to the original article Download button for the image {result && ( <Card> <CardHeader> <CardTitle>{result.title}</CardTitle> <CardDescription>{result.description}</CardDescription> </CardHeader> <CardContent> <img src={result.image} alt={result.title} /> </CardContent> <CardFooter> <a href={result.article_url}>Original Article</a> <Button>Download Image</Button> </CardFooter> </Card> )} 4. Error Handling If an error occurs during generation, a styled error card is displayed: {error && ( <Card className="border-destructive bg-destructive/10"> <CardTitle className="text-destructive">Error</CardTitle> <CardDescription>{error}</CardDescription> </Card> )} State Management The application uses React’s useState hook to manage four pieces of state: State Type Purpose url string The article URL entered by the user loading boolean Whether a generation is in progress result InfographicResult \ null The generated infographic data error string \ null Error message if generation fails Limitations This article is a proof of concept and may have limitations, such as: Does not handle paywall articles. Doesnt handle URL malformed articles, API limit exceeded, etc. References Nano Banana Pro Nebius

Modern AI systems increasingly rely on external tools, databases, and services, which makes interoperability a core requirement for building flexible

Modern factory floors continuously generate streams of sensor data, including temperature, humidity, vibration and how much they are producing. On their own, each reading might not seem important. When you collect this data from many machines over a long time it is the best way for a factory to find problems early and avoid shutting down for a long time. The main challenge is that this data is time-series: it is timestamped, arrives quickly, and is usually written once. Using a general-purpose relational table can work when there is not much data, but as you add more sensors and keep data longer, problems appear. Indexes get too large, range queries slow down, and deleting old data becomes difficult. This post walks through a complete solution: a Java application that simulates readings from three factory sensors, stores them in GridDB‘s native TimeSeries containers, and serves a live web dashboard that displays rolling averages and threshold alerts in real time. GridDB is an open-source distributed NoSQL database purpose-built for IoT and time-series workloads. Its TimeSeries containers are optimized for exactly the shape of problem a sensor fleet like this one produces: high write throughput paired with efficient temporal queries. — What you’ll learn By the end of this tutorial, you’ll be able to: Set up a complete GridDB + Java environment using Docker Compose, with no local installation required Ingest time-series sensor data using GridDB’s TimeSeries containers and the append() method Query and analyze data with built-in aggregations (AVG, MIN, MAX) and TQL, GridDB’s SQL-like query language Visualize live data with a dashboard featuring sparklines, range gauges, and an alert log Deploy a pipeline that works on Windows, Mac, and Linux without host-networking issues — Prerequisites Docker and Docker Compose (Docker Desktop, with the WSL2 backend enabled if you’re on Windows) Java 17 and Maven — only needed if you want to build or modify the Java app outside of Docker; the provided Dockerfile handles both automatically if you’re just running the demo Git, to clone the project repository No local GridDB installation is required — it runs entirely inside the Docker Compose stack — Project structure griddb-dashboard-demo/ ├── docker-compose.yml └── app/ ├── Dockerfile ├── pom.xml └── src/main/ ├── java/com/example/griddb/ │ ├── Main.java │ ├── GridDbConnection.java │ ├── SensorReading.java │ ├── SensorIngestor.java │ └── DashboardServer.java └── resources/ └── dashboard.html docker-compose.yml at the root wires the GridDB server and the Java app together on a shared Docker network. Everything under app/ is the Java side: the Dockerfile builds it, pom.xml declares the GridDB client dependency, and dashboard.html is the front end served directly by the Java app’s embedded HTTP server. — Why a time-series database here GridDB is built around two container types: Collection, for general keyed data, and TimeSeries, which uses the row’s timestamp as its key and is optimized for exactly this append-heavy, time-ordered access pattern. For a monitoring use case, that means two things fall out almost for free: Efficient aggregation over time windows. Asking “what was the average temperature for sensor 3 over the last 24 hours” is a single built-in aggregation call, not a hand-rolled GROUP BY over a timestamp column. Natural isolation per device. Each sensor gets its own TimeSeries container, so a burst of writes from one noisy machine doesn’t contend with reads against another. It is worth comparing this approach with a traditional relational database design. A common starting point for many IoT projects is to store all sensor readings in a single relational table containing columns such as device_id, timestamp, and the measured values, with indexes added to support time-range queries. While this approach works well for small-scale deployments, it becomes increasingly difficult to maintain as the number of devices, data ingestion rate, and retention period grow. As sensor data grows, traditional relational databases become increasingly difficult to manage. Every new reading requires index updates, time-range queries must search ever-larger tables, and removing historical data often requires costly delete operations or custom partitioning strategies. GridDB is designed to address these challenges. Its native TimeSeries containers optimize sequential writes and time-based queries, while built-in row expiration automatically removes outdated data based on configurable retention policies. This reduces database maintenance and simplifies application development, making GridDB well suited for large-scale Industrial IoT and real-time monitoring applications. GridDB Compared with PostgreSQL and InfluxDB Feature GridDB PostgreSQL InfluxDB Native Time-Series Support Yes Partial (via extensions such as TimescaleDB) Yes Horizontal Scaling Yes Primarily vertical (distributed extensions available) Yes Query Language TQL (SQL-like) SQL InfluxQL / Flux Designed for Time-Series & IoT Yes General-purpose database Yes ACID Transactions Yes Yes Limited* Official Java Support Official Java Client JDBC Official Java Client Docker Support Yes Yes Yes > \* InfluxDB provides durability and consistency for time-series workloads but does not implement full ACID transactional semantics across multiple records in the same way as traditional relational databases such as PostgreSQL. — Architecture The system has four main components: GridDB server — running in Docker, one container from the standard GridDB image. A Java ingestion process — seeds historical data, then keeps appending a new reading every few seconds per simulated device. An embedded HTTP API — built on the JDK’s built-in com.sun.net.httpserver, so there’s no extra web framework to configure. It exposes /api/stats, /api/alerts, and /api/history. A dashboard page — polls those endpoints and renders a fleet summary strip, per-device panels with live sparkline trend lines and range gauges, and an alarm log. Everything ships in one docker-compose.yml, so docker compose up –build gets a full GridDB cluster and the Java app talking to each other on a bridge network. The complete source — the Dockerfile, docker-compose.yml, and all Java classes — is on GitHub: dagmawit-sudo-cloud/griddb-dashboard-demo. Clone it now to follow along and run each step as you read. Data flow, end to end — Why Docker Docker packages both GridDB and the Java application into isolated containers, so the exact same setup runs the same way regardless of whether it’s on your laptop, a teammate’s machine, or a CI pipeline — no “works on my machine” gap between installing GridDB natively and getting the Java client to find it. It also sidesteps a real platform-specific headache: Docker Desktop on Windows and Mac doesn’t support host networking the way Linux does, so a Compose setup using a proper bridge network (as this one does) is the version that actually works cross-platform, rather than one that only works if you happen to be on Linux. — Modeling a sensor reading The Java side talks to GridDB through its official Java Client API — the same supported gridstore library GridDB documents for production use, not a community wrapper or a raw protocol implementation. Every device writes rows shaped like this: public class SensorReading { @RowKey Date timestamp; String deviceId; double temperature; double humidity; boolean alertTriggered; } The @RowKey annotation on timestamp is what tells GridDB this class belongs in a TimeSeries container rather than a plain Collection. alertTriggered is computed at write time rather than derived later — efficient to compute once, and it lets the alerts endpoint run a simple boolean filter instead of recomputing a threshold check on every poll. Connecting, with retry Docker Compose starts containers in parallel, so the Java app frequently comes up before GridDB is actually ready to accept connections. The connection helper handles that with a bounded retry loop instead of failing hard on the first attempt: Properties props = new Properties(); props.setProperty("notificationMember", notificationMember); props.setProperty("clusterName", clusterName); props.setProperty("user", user); props.setProperty("password", password); for (int attempt = 1; attempt <= maxAttempts; attempt++) { try { GridStore store = GridStoreFactory.getInstance().getGridStore(props); System.out.println("Connected to GridDB at " + notificationMember); return store; } catch (GSException e) { System.out.println("Attempt " + attempt + "/" + maxAttempts + ": GridDB not ready yet (" + e.getMessage() + "), retrying in 5s…"); Thread.sleep(5000); } } All four connection properties — notification member, cluster name, user, password — are read from environment variables set in docker-compose.yml, which keeps the same image usable across dev and staging without a rebuild. > Tip: GridDB’s cluster takes 30–60 seconds to form after container startup. The retry loop ensures your application connects reliably even if it starts before the database is ready. Packaging the Java app The Java side builds as a two-stage Docker image, so the final container doesn’t carry a Maven install around at runtime: FROM maven:3.9-eclipse-temurin-17 AS build WORKDIR /build COPY pom.xml . COPY src ./src RUN mvn -q clean package -DskipTests FROM eclipse-temurin:17-jre WORKDIR /app COPY –from=build /build/target/app.jar app.jar EXPOSE 8080 CMD ["java", "-jar", "app.jar"] The build stage compiles the fat jar with Maven; the runtime stage starts from a bare JRE image and copies only the finished artifact across. The net effect is a noticeably smaller image and a faster docker compose up on the second run, since Docker’s layer cache skips the Maven download step entirely as long as pom.xml hasn’t changed. docker-compose.yml wires this image up to the GridDB container on a dedicated bridge network and passes in the connection properties as environment variables, so the two containers can find each other by service name (griddb-server:10001) without any manual network configuration. — Generating and appending data The application supports two ingestion workflows: an initial historical data load to populate the dashboard and a continuous streaming process that appends new sensor readings as they are generated. public static void ingest(GridStore store, String deviceId, int readingCount, double baseTemp) throws GSException { TimeSeries<SensorReading> ts = store.putTimeSeries(deviceId, SensorReading.class); for (int i = 0; i < readingCount; i++) { ts.append(buildReading(deviceId, baseTemp, new Random())); } } append() is the right call for time-series writes specifically — it assumes rows arrive in increasing timestamp order and skips the row-key lookup a generic put() would do, which matters once you’re writing thousands of readings a minute across many devices. The continuous variant runs the same append() call on a daemon thread every three seconds per device. — Querying: aggregates and alerts The stats endpoint leans on GridDB’s built-in aggregation functions rather than pulling raw rows into Java and averaging them by hand: Date now = TimestampUtils.current(); Date oneDayAgo = TimestampUtils.add(now, -24, TimeUnit.HOUR); double avg = ts.aggregate(oneDayAgo, now, "temperature", Aggregation.AVERAGE).getDouble(); double min = ts.aggregate(oneDayAgo, now, "temperature", Aggregation.MINIMUM).getDouble(); double max = ts.aggregate(oneDayAgo, now, "temperature", Aggregation.MAXIMUM).getDouble(); The alerts endpoint uses GridDB’s TQL query language instead, since it’s a row filter rather than an aggregate: Query<SensorReading> query = ts.query("WHERE alertTriggered", SensorReading.class); RowSet<SensorReading> rows = query.fetch(); > TQL pitfall: For boolean fields, TQL allows conditions to be written directly as WHERE alertTriggered, which is the form used throughout this example. Both endpoints return hand-built JSON strings over the JDK’s embedded HttpServer. That’s a deliberate simplification for a demo of this size — swap in your JSON library and framework of choice once the endpoint list grows past two. API endpoints Method & route Purpose GET /api/health Basic liveness check GET /api/stats Per-device avg/min/max/reading count over the configured window GET /api/alerts The most recent threshold-breaching readings across all devices GET /api/history?device= The last 40 readings for one device, oldest first — powers the sparkline trend line GET / The dashboard HTML page itself — The dashboard The dashboard is implemented as a lightweight, static HTML page served directly by the Java application’s embedded HTTP server. The interface is designed to resemble an industrial control-room dashboard, providing operators with a clear, real-time view of factory conditions. Every five seconds, the dashboard requests the latest data from the REST API and updates the display with three key monitoring components: A fleet summary displays an overview of the monitored environment, including the number of active devices, the fleet-wide average temperature, and the current number of active alerts. Device Monitoring panels present a dedicated panel for each sensor, featuring a status indicator, the latest sensor reading, a sparkline showing recent temperature trends, and a min/max range gauge with a configurable alert threshold. An alarm log displays the most recent threshold breaches and presents a clear empty state when no alerts have been triggered. That five-second polling interval is a knob worth tuning in your own setup tighter for a control-room display, looser if you’re polling over a slow network link. A close-up of a single device panel, showing the sparkline trend line and the min/max range gauge with the alert threshold marker. — Running it docker compose up –build Then open http://localhost:8080. Within a few seconds the device panels populate with live readings and trend lines, and if the simulated readings cross the alert threshold (27.5°C in this example), rows start appearing in the alarm log. Terminal output of docker compose up –build completing successfully, showing GridDB’s cluster starting and the Java app connecting. docker ps showing both containers — griddb-server and griddb-java-app — up and running on the shared bridge network. Checking the API directly It’s worth hitting the endpoints on their own before trusting the dashboard, both to confirm the data looks right and as a sanity check while you’re adapting this to your own sensors: $ curl http://localhost:8080/api/stats # [{"deviceId":"factory_floor_sensor_01","avg":26.84,"min":24.02,"max":31.97,"count":118}, …] $ curl http://localhost:8080/api/alerts # [{"deviceId":"factory_floor_sensor_01","timestamp":"2026-07-01T14:22:09.441Z","temperature":29.13}, …] Because both routes set Access-Control-Allow-Origin: *, they’re also easy to point a separate front end or a tool like Grafana’s JSON API plugin at, if the bundled dashboard ends up being a placeholder for something more elaborate. — Conclusion GridDB’s native time-series support, high write throughput, and straightforward Docker integration make it a strong fit for factory monitoring: TimeSeries containers handle the append-heavy, time-ordered access pattern that sensor data naturally produces, without the index maintenance and partitioning work a general-purpose relational table would require at the same scale. The full source for this project — the five Java classes, Dockerfile, and docker-compose.yml — is available on GitHub: dagmawit-sudo-cloud/griddb-dashboard-demo. References GridDB Java API Reference – https://www.griddb.net/en/resources/griddb-java-api-reference/ GridDB GitHub Repository – https://github.com/griddb/griddb Docker Documentation – https://docs.docker.com/ Chart.js Documentation –

Fluentd is one of the most widely deployed log collectors in the world, and log ingestion is one of the workloads GridDB was built for: high-volume, append-heavy, time-stamped data. In this article we wire the two together using the fluent-plugin-griddb output plugin and GridDB Cloud, and then keep increasing the ingestion rate until something gives. As a brief summary: one small VM went from 15,000 to 150,000 rows per second by changing a single buffer parameter, and when we then quadrupled the collector hardware to push past that, throughput barely moved. We found the ceiling, and the way it shows up is more interesting than the number itself: not errors, not rejections, but quiet backpressure. Along the way we’ll cover a realistic installation (the plugin targets an older Ruby, so there is some dependency archaeology involved), a small patch the plugin needs before its buffered mode works, the container schema gotchas that the Web API’s positional row format will happily punish you for, and a version-drift bug that dressed itself up as a TLS problem. Why Fluentd GridDB has steadily grown its connector ecosystem: Kafka, JDBC, Grafana, and a range of collection agents can all read from or write to GridDB, which means it usually slots into an existing data pipeline rather than demanding a new one. Fluentd is a natural addition to that list. It is the de facto standard for log collection in a lot of Kubernetes and traditional server environments, it has hundreds of input and parser plugins, and once events are flowing through it, pointing them at GridDB is a single block. Everything in this post talks to GridDB Cloud through its Web API: JSON rows over HTTPS; it works from anywhere without special client libraries, and, as the numbers below will show, with sensible batching it moves more data than most logging workloads will ever generate. Installation For installation, fluent-plugin-griddb targets Ruby 2.7 and Fluentd 1.12. Ruby 2.7 predates OpenSSL 3, which is what modern Ubuntu ships, so Ruby will not compile against the system OpenSSL. The fix is to build OpenSSL 1.1.1 from source into its own prefix, build Ruby 2.7.2 against it with rbenv, and then pin RubyGems, Bundler, and Fluentd to the last versions that still support Ruby 2.7. It sounds worse than it is; each step is mechanical. 1. System dependencies $ sudo apt update $ sudo apt install -y git curl build-essential libreadline-dev zlib1g-dev \ libyaml-dev libncurses5-dev libffi-dev libgdbm-dev 2. OpenSSL 1.1.1 from source $ cd /tmp $ wget https://www.openssl.org/source/openssl-1.1.1w.tar.gz $ tar xvfz openssl-1.1.1w.tar.gz $ cd openssl-1.1.1w $ ./config –prefix=/opt/openssl-1.1.1w –openssldir=/opt/openssl-1.1.1w shared zlib $ make -j$(nproc) $ sudo make install One step that is easy to miss: the freshly built OpenSSL has an empty certificate directory, so anything compiled against it will fail TLS verification, including HTTPS calls to GridDB Cloud. Point it at the system certificate store: $ sudo rm -rf /opt/openssl-1.1.1w/certs $ sudo ln -s /etc/ssl/certs /opt/openssl-1.1.1w 3. rbenv and Ruby 2.7.2 Install rbenv, but pull ruby-build directly from GitHub. The version definitions that come with the packaged rbenv are stale and may not build 2.7.2 cleanly: $ sudo apt install -y rbenv $ mkdir -p "$(rbenv root)"/plugins $ git clone https://github.com/rbenv/ruby-build.git "$(rbenv root)"/plugins/ruby-build $ RUBY_CONFIGURE_OPTS="–with-openssl-dir=/opt/openssl-1.1.1w" rbenv install 2.7.2 $ rbenv global 2.7.2 $ echo 'eval "$(rbenv init – bash)"' >> ~/.bashrc $ source ~/.bashrc $ ruby –version # should report 2.7.2 4. Fluentd and the plugin Ruby 2.7 also needs older RubyGems and Bundler; the current releases of both have dropped support. Pin them, then install Fluentd 1.12.0 and the plugin gem: $ gem update –system 3.4.22 $ gem install bundler -v 2.4.22 $ gem install fluentd –version "1.12.0" –no-doc $ rbenv rehash $ gem install –force –local fluent-plugin-griddb-1.0.2.gem $ rbenv rehash A quick fluentd –version confirms the pipeline pieces are in place. And make sure it is 1.0.2 of the plugin specifically; more on why in a moment. Patching the Plugin Version 1.0.2 of the plugin has a latent bug: it works in its default, unbuffered mode, but the moment you add a section to the match block (which you will want for any serious throughput), Fluentd switches the plugin to its buffered code path, and that path calls a method that does not exist: $ error_class=NoMethodError error="undefined method `handle_record' for #<Fluent::Plugin::GriddbOutput> Did you mean? handle_create_container" Inside out_griddb.rb, the unbuffered process path routes records through a method called handle_insert_type, which works. The buffered write(chunk) path calls handle_record, which was never implemented. Every chunk raises, gets flagged as a bad chunk, and is shunted to Fluentd’s backup directory. The fix is one line: make the buffered path call the same method the working path uses. $ sed -i 's/handle_record(record_array)/handle_insert_type(record_array)/' \ $ "$(gem which fluent/plugin/out_griddb)" Note the use of gem which to locate the file. If you have the plugin’s source repository checked out as well as the gem installed, it is easy to edit the wrong copy and wonder why nothing changed. gem which fluent/plugin/out_griddb tells you exactly which file Fluentd loads. Two related notes while we are in the plugin internals. First, the plugin’s insert_type parameter accepts single and multiple, but multiple is currently a stub that logs “Currently insert type only support for single type” and does nothing, so leave it on single. Second, and this matters for the results later: even in single mode, the buffered path sends an entire buffer chunk in one HTTP request. So batch size is controlled by the buffer’s chunk settings, not by the insert type. Creating the Target Container The Web API inserts rows positionally: each row arrives as an array of values, matched to columns strictly by order. That makes the container schema definition load-bearing in a way that name-based systems are not, and it produced two instructive failures before the schema below worked. The event we will generate has six fields, plus a seq counter that Fluentd’s sample input appends to the end of each record. Appends, as in: seq arrives as the last value in the row, so it must be the last column in the schema. Defining it first produces The specified data cannot be converted to LONG type as the host string lands in the seq slot. Defining extra columns the record does not send (say, copying the schema of an apache-parsed container with user and referer fields) produces Row data is invalid from the value-count mismatch. The schema that matches what the plugin actually sends: $ curl -X POST \ $ -u "$GRIDDB_USER:$GRIDDB_PASS" \ $ -H 'Content-Type: application/json' \ $ "$GRIDDB_WEBAPI_URL/dbs/$GRIDDB_DB/containers" \ $ -d '{ $ "container_name": "fluentdStress", $ "container_type": "COLLECTION", $ "rowkey": false, $ "columns": [ $ {"name": "host", "type": "STRING"}, $ {"name": "method", "type": "STRING"}, $ {"name": "path", "type": "STRING"}, $ {"name": "code", "type": "INTEGER"}, $ {"name": "size", "type": "INTEGER"}, $ {"name": "agent", "type": "STRING"}, $ {"name": "seq", "type": "LONG"} $ ] $ }' Or you can use the GridDB Cloud CLI Tool: griddb-cloud-cli create -i. We use a COLLECTION with rowkey false rather than a time-series container on purpose. A time-series container upserts on duplicate timestamps, and at tens of thousands of rows per second many events share a millisecond. A throughput test against a time-series container without a carefully unique key silently measures overwrites while reporting success. A rowkey-less collection accepts every insert as a distinct row, which is what we want to count. The Stress Configuration For a throughput test, tailing a log file written by a generator makes the generator and the disk the bottleneck. Fluentd’s built-in sample input creates events in memory instead, which lets us dial the input rate directly: $ <system> $ workers 4 # match your core count $ </system> $ <source> $ @type sample $ tag stress.griddb $ size 200 # events per emit $ rate 50000 # events per second $ auto_increment_key seq # unique trailing counter per event $ dummy {"host":"10.0.0.1","method":"GET","path":"/api/v1/users","code":200,"size":4096,"agent":"gridstress/1.0"} $ </source> $ <match stress.griddb> $ @type griddb $ host "$GRIDDB_CLOUD_HOST" $ cluster "$GRIDDB_CLUSTER" $ database "$GRIDDB_DB" $ container "fluentdStress" $ insert_mode "append" $ insert_type "single" $ username "$GRIDDB_USER" $ password "$GRIDDB_PASS" $ <buffer> $ @type memory $ chunk_limit_records 50000 # rows per HTTP request: the knob that matters $ chunk_limit_size 64MB $ flush_thread_count 8 $ flush_mode interval $ flush_interval 1s $ flush_thread_interval 0.1 $ total_limit_size 4GB $ retry_max_times 3 $ retry_type periodic $ retry_wait 1s $ overflow_action throw_exception $ </buffer> $ </match> The parameters worth understanding: chunk_limit_records sets how many rows travel in each PUT to the Web API, flush_thread_count sets how many of those requests run in parallel, and workers in the system block spreads the whole pipeline across CPU cores. Everything else is safety plumbing: total_limit_size bounds the memory buffer, and overflow_action throw_exception makes backpressure loud instead of silent, which is exactly what you want when the point of the exercise is to see what breaks. Measuring Fluentd’s own logs tell you what it sent; they do not tell you what the database kept. The only number that counts is the row count in the container, so we sample it and compute the delta over the actual elapsed time: $ A=$(griddb-cloud-cli sql query -s "select count(*) from fluentdStress" \ $ | grep -o '"Value":[0-9]*' | grep -o '[0-9]*'); T1=$(date +%s) $ sleep 60 $ B=$(griddb-cloud-cli sql query -s "select count(*) from fluentdStress" \ $ | grep -o '"Value":[0-9]*' | grep -o '[0-9]*'); T2=$(date +%s) $ echo "confirmed rows/sec: $(( (B – A) / (T2 – T1) ))" One measurement lesson learned the hard way: divide by real elapsed time, not the interval you asked for. Once the container holds tens of millions of rows, count(*) itself takes several seconds, and a sampler that assumes its nominal interval will overstate throughput badly. And confirmed rows per second, measured at the destination, is the honest metric. If Fluentd claims success while this number lags what was sent, rows are being dropped somewhere. In our runs that gap never appeared: everything sent was stored. Round One: Batch Size Is the Whole Game All single-node runs use the same event shape. The only variables changed between runs are the chunk size and the worker count. chunk_limit_records workers node(s) sustained rows/sec 5,000 1 1 x 2-core ~15,000 20,000 2 1 x 2-core ~116,000 50,000 2 1 x 2-core ~150,000 The leap from 15k to 150k comes almost entirely from batching. Per-request overhead, HTTP, TLS, JSON framing, is fixed cost; the more rows amortize it, the closer you get to the pipeline’s real capacity. And do the request math: at 150,000 rows per second in 50,000-row chunks, the client is making roughly three HTTP requests per second. This resolves a confusion that comes up constantly with cloud database quotas: platform limits are typically expressed against requests and connections, not rows, and a well-batched pipeline moves enormous row volume through a trickle of requests. Row throughput and request rate are independent axes. Round Two: More Hardware, Same Ceiling To push the ingestion even further, we figured would throw in more cores and more nodes. So: two VMs, 4 cores each, 4 Fluentd workers each, identical configs, both writing to the same container. Eight workers across eight cores, four times the collector CPU that produced the 150k number. Setting up the second node produced a bug worth recounting, because it disguised itself well. The second machine’s Fluentd immediately failed every request with: 400 The plain HTTP request was sent to HTTPS port Microsoft-Azure-Application-Gateway/v2 The configs were byte-identical. OpenSSL versions matched. Certificate symlinks matched. No proxy variables anywhere. The actual cause: gem install fluent-plugin-griddb without a version pin had installed 1.0.1 on the new box instead of 1.0.2, and 1.0.1 handles the URL scheme differently, quietly speaking plaintext HTTP to port 443. A minor plugin version drift between two “identical” nodes produced what looked exactly like a TLS or gateway misconfiguration. Pin your plugin versions across a fleet. And remember that reinstalling the gem also wipes the handle_record patch, which must be reapplied on every fresh install. With both nodes on patched 1.0.2, the combined result over a sustained run: configuration total collector cores sustained rows/sec (combined) 1 node, 2 cores 2 ~150,000 2 nodes, 4 cores each 8 ~168,000 Quadrupling collector capacity bought about 12%. That flatline is the signature of a shared ceiling downstream of the collectors: somewhere in GridDB Cloud’s ingest path, whether the database engine or the Azure Application Gateway fronting it, the pipeline saturates around 150-170k rows per second on this plan. What the Ceiling Looks Like Here is the part that matters for anyone running this in production: the ceiling does not announce itself with errors. There were no 429s, no 503s, no failed inserts, no dropped rows at any point. Every single request returned success. What changed was latency. Buffer flushes that normally complete in about a second began taking 20 to 29 seconds: $ [warn]: #1 buffer flush took longer time than slow_flush_log_threshold: $ elapsed_time=26.086 slow_flush_log_threshold=20.0 $ [warn]: #2 buffer flush took longer time than slow_flush_log_threshold: $ elapsed_time=27.094 slow_flush_log_threshold=20.0 $ [warn]: #3 buffer flush took longer time than slow_flush_log_threshold: $ elapsed_time=28.739 slow_flush_log_threshold=20.0 Under sustained saturation, throughput oscillates between roughly 120k and 190k rows per second as flush queues build and drain. GridDB Cloud applies backpressure by slowing acceptance, not by rejecting requests. That is graceful behavior, nothing is lost, but it has a monitoring consequence: an alerting setup that watches only error rates will report a perfectly healthy pipeline while ingest latency has grown by 25x. Watch flush duration, not just status codes. Conclusion & Takeaways Tune the buffer before anything else. The difference between a naive Fluentd-to-GridDB config and a tuned one is an order of magnitude, and it is almost entirely chunk_limit_records. If your pipeline feels slow, your batches are almost certainly too small. One well-tuned node covers most workloads. A single 2-core VM sustained 150,000 rows per second. Before scaling out collectors, check whether you are anywhere near that; most logging workloads are not. The ceiling is real, and it is polite. Around 150-170k rows per second on this plan, the ingest path saturates regardless of collector count. It degrades by latency, not by error, so monitor flush duration. Sent and stored never diverged: even at saturation, nothing was dropped. Pin plugin versions across nodes. A one-patch-version drift between two collectors produced a protocol-level failure that looked like a TLS misconfiguration. Fleet consistency is not optional. Measure at the destination, against real elapsed time. count(*) deltas are the ground truth, and they get slow as the container grows; divide by actual seconds or your numbers flatter you. The Web API, for its part, holds up incredibly well. JSON over HTTPS carries overhead, and the native transport would spend fewer cycles per row. But sustaining 150,000+ rows per second, error-free, through the least exotic interface imaginable is more ingest than the vast majority of logging workloads will ever need. We set out to break GridDB Cloud, and the closest we got was making it take a little longer to say