*This is a summary of the Benchmarking a Sensor Billing Application using GridDB and MariaDB whitepaper.
Introduction {#intro} By utilizing a use-case common to the GridDB database, Fixstars was able to create an application to accurately benchmark the NoSQL database against MariaDB, a MySQL database. The application in question gathers info from sensors for billing applications — a very common IoT (internet of things) scenario. What makes this comparison particularly striking and apt is that these sorts of applications have traditionally been run on RDBM (Relational Database Management) systems. Though many different types of SQL databases can be used for these applications, Fixstars chose to focus on MariaDB because of its popularity and because it is free and open source.
Test Methodology {#Methodology} The application was separated into two test cases: load (ingestion) and extraction. This method allowed Fixstars to stay true to the real-world scenario which was being simulated, as well provide different metrics to fairly and accurately assess performance. The ingest test performs operations as quickly as possible where each thread is responsible for one device. The extraction test performs an aggregation on a subset of the data generated in the load phase.
Benchmark Results
Several metrics were tested when measuring the performance between these two databases. The above image shows GridDB’s superior performance in one of the metrics (ingestion), but it also won out when measuring resource use and time taken for extraction of data. The ability for NoSQL databases to properly scale outwards and add more nodes as necessary was easily showcased here. Benchmarks were performed using Microsoft’s Cloud-Based web service Azure which allowed an easy interface to add or remove nodes at will. The benchmark results show the stark difference in ability to scale between the two databases. The ingestion (or load) test measured the number of updates per second that each database can write. GridDB was able to process 31,916 OPS (operations per second) where MariaDB lagged behind a bit with 2,423. Extract measured the SUM-aggregation of month’s data for each device. Fixstars then compared how long it took to run the query against one device. The results showed GridDB’s query time remained consistent as more and more sensors/devices were added; MariaDB’s query times gradually worsened as query-able items were increased. 
As previously mentioned, resource use was also measured and recorded during these tests. GridDB was able to perform at much higher levels while using up less CPU and memory resources. 
Testing Process {#testing} The benchmarks were performed using Microsoft Azure virtual machines in the WestUS region. There is one database server, a B2ms instance and a mix of three B2ms and two A2 client instances. The B2ms instances have dual core Intel(R) Xeon(R) E5-2673 v3 running at 2.40GHz and 4GB of memory while A2 instances have a dual -core Intel(R) Xeon(R) E5-2660 CPU and 3.5GB of memory. Performance for compute power on cloud services tend to fluctuate a bit, so each test was repeated three times to maintain as much veracity as possible.
Reference
- To download the source code used for this whitepaper, please click here: rdbms-benchmark-master.zip
- The full report detailing the results, methodologies, configurations and practices used for the benchmark test can be found in the full whitepaper.
- GridDB Community Edition was used for benchmark testing. MariaDB version 5.5.56 was used as the MariaDB database.
- Fixstars also previously completed a GridDB vs. Cassandra Benchmark Comparison. The Summary and Full Report can be found here.
- GridDB vs. InfluxDB Benchmark Comparison: The Summary and Full Report can be found here.
If you have any questions about the blog, please create a Stack Overflow post here https://stackoverflow.com/questions/ask?tags=griddb .
Make sure that you use the “griddb” tag so our engineers can quickly reply to your questions.