GridDB, a high-performance NoSQL database, is particularly suited for handling the large-scale, real-time data generated by IoT systems due to its efficient in-memory processing and time series capabilities.<\/p>\n
Using GridDB\u00e2\u0080\u0099s powerful IoT data management features along with advanced machine learning and deep learning, we will build a predictive model that identifies potential threats to IoT devices.<\/p>\n
Note:<\/strong> You can find codes for the tutorial in my GridDB Blogs GitHub repository<\/a>.<\/p>\n You need to install the following libraries to run codes in this article.<\/p>\n To install these libraries, follow the installation instructions on GridDB Python Package Index (Pypi)<\/a>.<\/p>\n Since the code is executed in Google Colab<\/a>, you do not need to install any other libraries.<\/p>\n Run the following script to import required libraries into your Python application.<\/p>\n In this section, you will see how to download IoT data from kaggle, import it into your Python application and store it in the GridDB database. Along the way you, you will learn to connect your Python application to GridDB, create a GridDB container and insert data into the GridDB container you created.<\/p>\n We will insert the IoT Intrusion Detection dataset from Kaggle<\/a> into the GridDB.<\/p>\n The dataset consists of different types of IoT intrusions. the following script imports the dataset into a Pandas dataframe.<\/p>\n Output:<\/strong> The above output shows that the dataset consists of 1048575 rows and 47 columns. The Output:<\/strong> The above output shows all the 34 intrusion types in our dataset with For simplification’s sake we will group the 34 categories into 9 major categories using the following script.<\/p>\n Output:<\/strong> You can now see that Let’s plot a bar plot for the class distribution.<\/p>\n Output:<\/strong> The above output shows that our dataset is highly imbalanced.<\/p>\n In the next section, we will insert this data into GridDB.<\/p>\n You need to perform the following steps to connect your Python application to a GridDB instance.<\/p>\n Create an instance of the Create a GridDB store factory object by calling the To test the connection create a container by calling the The following script shows how to connect to GridDB.<\/p>\n Output:<\/strong><\/p>\n If you see the above output, you have successfully connected to GridB.<\/p>\n GridDB stores data in containers. Therefore, you need to create a container to store your IoT data.<\/p>\n You will need to perform the following steps to create a GridDb container capable of storing your IoT data.<\/p>\n First, check if a container with the name you want to use already exists. If it does, either delete the existing container or choose a different name for your new container.<\/p>\n<\/li>\n Convert your dataset into a format that the GridDB expects. For example, GridDB expects the dataset to have a unique ID, therefore you need to add unique add if it doesn’t already exist. Similarly GriDB doesn’t accept spaces in column names. You will need to preprocess the column names too.<\/p>\n<\/li>\n GridDB data types are different than the Pandas dataframe types. You need to Define mapping from Pandas dataframe column types to GridDB data types.<\/p>\n<\/li>\n You need to create a column info list that contains column names and their corresponding mapped data type.<\/p>\n<\/li>\n Finally, you need to call the To test if the container is successfully created, call the The following script creates the We are now ready to insert the dataframe into the GridDB container we just created. The following script inserts our IoT data from the Pandas dataframe into the GriDB Output:<\/strong><\/p>\n In the next sections we will retrieve the IoT data from GridDB and will train machine learning and deep learning models for predicting intrusion type.<\/p>\n We will try both machine learning and deep learning approaches for predicting intrusion type. But first we will see how to retrieve data from GridDB.<\/p>\n To retrieve GridDB data you have to retrieve the container, call The following script defines the Output:<\/strong> We will first use the Random Forest Classification<\/a> algorithm, a classification machine learning algorithm and predict the intrusion type.<\/p>\n To do so, we will divide our dataset into features and labels set and then train and test sets. We will also standardize our dataset.<\/p>\n Next, we will use the Finally, we can make predictions using the Output:<\/strong> The above output shows that our model achieves an accuracy of 99% on the test set. It is important to note that since Prerequisites<\/h2>\n
\n
import pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.metrics import accuracy_score, classification_report\nfrom sklearn.preprocessing import LabelEncoder\n\nimport tensorflow as tf\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Dropout, BatchNormalization\nfrom tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.models import load_model\n\nimport griddb_python as griddb<\/code><\/pre>\n<\/div>\nInserting IoT Data into GridDB<\/h2>\n
Downloading and Importing the IoT Dataset From Kaggle<\/h3>\n
# Dataset download link\n#https:\/\/www.kaggle.com\/datasets\/subhajournal\/iotintrusion\/data\n\ndataset = pd.read_csv(\"IoT_Intrusion.csv\")\nprint(f\"The dataset consists of {dataset.shape[0]} rows and {dataset.shape[1]} columns\")\ndataset.head()<\/code><\/pre>\n<\/div>\n
\n![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2025\/06\/img1-iot-dataset.png\")
<\/a><\/p>\n
\nFor the sake of simplicity, we will train our machine learning models on 200k records. The following script randomly selects 200k records from the original dataset.<\/p>\nlabel<\/code> column contains the intrusion types. We will also plot the count for each intrusion type.<\/p>\ndataset = dataset.sample(n=200000, random_state=42)\nprint(f\"The dataset consists of {dataset.shape[0]} rows and {dataset.shape[1]} columns\")\nprint(f\"The total number of output labels are {dataset['label'].nunique()}\")\ndataset['label'].value_counts()<\/code><\/pre>\n<\/div>\n
\n![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2025\/06\/img2-iot-label-distribution.png\")
<\/a><\/p>\nDDoS-ICMP_FLOOD<\/code> being the most frequently occurring intrusion while Uploading_Attack<\/code> is the least frequently occurring intrusion type.<\/p>\ncategory_map = {\n 'DDoS': [\n 'DDoS-ICMP_Flood', 'DDoS-UDP_Flood', 'DDoS-TCP_Flood', 'DDoS-PSHACK_Flood',\n 'DDoS-SYN_Flood', 'DDoS-RSTFINFlood', 'DDoS-SynonymousIP_Flood', 'DDoS-ICMP_Fragmentation',\n 'DDoS-ACK_Fragmentation', 'DDoS-UDP_Fragmentation', 'DDoS-HTTP_Flood', 'DDoS-SlowLoris'\n ],\n 'DoS': [\n 'DoS-UDP_Flood', 'DoS-TCP_Flood', 'DoS-SYN_Flood', 'DoS-HTTP_Flood'\n ],\n 'Brute Force': [\n 'DictionaryBruteForce'\n ],\n 'Spoofing': [\n 'MITM-ArpSpoofing', 'DNS_Spoofing'\n ],\n 'Recon': [\n 'Recon-HostDiscovery', 'Recon-OSScan', 'Recon-PortScan', 'Recon-PingSweep'\n ],\n 'Web-based': [\n 'SqlInjection', 'CommandInjection', 'XSS', 'BrowserHijacking', 'Uploading_Attack'\n ],\n 'Mirai': [\n 'Mirai-greeth_flood', 'Mirai-udpplain', 'Mirai-greip_flood'\n ],\n 'Other': [\n 'VulnerabilityScan', 'Backdoor_Malware'\n ],\n 'Benign-trafic': [\n 'BenignTraffic'\n ]\n}\n\n# Reverse the mapping to allow lookup by subcategory\nsubcategory_to_parent = {subcat: parent for parent, subcats in category_map.items() for subcat in subcats}\n\n# Add the 'class' column using the mapping\ndataset['class'] = dataset['label'].map(subcategory_to_parent)\n\ndataset['class'].value_counts()\n<\/code><\/pre>\n<\/div>\n
\n![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2025\/06\/img3-processed-iot-class-labels.png\")
<\/a><\/p>\nDDoS<\/code> intrusion is the most frequently occuring intrusion followed by DoS<\/code> and Mirai<\/code>.<\/p>\nclass_counts = dataset['class'].value_counts()\n\nsns.set(style=\"darkgrid\")\n\nplt.figure(figsize=(10, 6))\nsns.barplot(x=class_counts.index, y=class_counts.values)\nplt.title(\"Class Distribution\")\nplt.xlabel(\"Class\")\nplt.ylabel(\"Count\")\nplt.xticks(rotation=45)\nplt.show()<\/code><\/pre>\n<\/div>\n
\n![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2025\/06\/img4-processed-iot-class-labels-bar-plot.png\")
<\/a><\/p>\nConnect to GridDB<\/h3>\n
\n
griddb.StoreFactory<\/code> object using the get_instance()<\/code> method.<\/p>\n<\/li>\nget_store()<\/code> method on the StoreFactory<\/code> object. You need to pass the GridDB host and cluster name, and the user and password that you use to connect to the GridDB instance. This should establish a connection to your GridDB instance.<\/p>\n<\/li>\nget_container()<\/code> method and pass to it a dummy container name. This step is optional and only tests the connection.<\/p>\n<\/li>\n<\/ol>\n# GridDB connection details\nDB_HOST = \"127.0.0.1:10001\"\nDB_CLUSTER = \"myCluster\"\nDB_USER = \"admin\"\nDB_PASS = \"admin\"\n\n# creating a connection\n\nfactory = griddb.StoreFactory.get_instance()\n\ntry:\n gridstore = factory.get_store(\n notification_member = DB_HOST,\n cluster_name = DB_CLUSTER,\n username = DB_USER,\n password = DB_PASS\n )\n\n container1 = gridstore.get_container(\"container1\")\n if container1 == None:\n print(\"Container does not exist\")\n print(\"Successfully connected to GridDB\")\n\nexcept griddb.GSException as e:\n for i in range(e.get_error_stack_size()):\n print(\"[\", i, \"]\")\n print(e.get_error_code(i))\n print(e.get_location(i))\n print(e.get_message(i))<\/code><\/pre>\n<\/div>\nContainer does not exist\nSuccessfully connected to GridDB<\/code><\/pre>\n<\/div>\nCreate Container for IoT Data in GridDB<\/h3>\n
\n
put_container()<\/code> method on the store factory object and pass it the container name, the column_info<\/code> list and the container type (griddb.ContainerType.COLLECTION<\/code> in this case). The row_key<\/code> is set to true since each row has unique ID.<\/p>\n<\/li>\nget_container()<\/code> method to retrieve the container you created.<\/p>\n<\/li>\n<\/ol>\nIoT_Data<\/code> container in our GridB instance.<\/p>\n# drop container if already exists\ngridstore.drop_container(\"IoT_Data\")\n\n# Add an primary key column\ndataset.insert(0, 'ID', range(1, len(dataset) + 1))\n\n# Clean column names to remove spaces or forbidden characters in the GridDB container\ndataset.columns = [col.strip().replace(\" \", \"_\") for col in dataset.columns]\n\n# Mapping from pandas data types to GridDB data types\ntype_mapping = {\n 'float64': griddb.Type.DOUBLE,\n 'int64': griddb.Type.INTEGER,\n 'object': griddb.Type.STRING\n}\n\n\n# Generate column_info dynamically, adding ID as the first entry\ncolumn_info = [[\"ID\", griddb.Type.INTEGER]] + [\n [col, type_mapping[str(dtype)]] for col, dtype in dataset.dtypes.items() if col != \"ID\"\n]\n\n# Define the container info with ID as the primary key and as a collection container\ncontainer_name = \"IoT_Data\"\ncontainer_info = griddb.ContainerInfo(\n container_name, column_info, griddb.ContainerType.COLLECTION, row_key=True\n)\n\n# Connecting to GridDB and creating the container\ntry:\n gridstore.put_container(container_info)\n container = gridstore.get_container(container_name)\n if container is None:\n print(f\"Failed to create container: {container_name}\")\n else:\n print(f\"Successfully created container: {container_name}\")\n\nexcept griddb.GSException as e:\n print(f\"Error creating container {container_name}:\")\n for i in range(e.get_error_stack_size()):\n print(f\"[{i}] Error code: {e.get_error_code(i)}, Message: {e.get_message(i)}\")<\/code><\/pre>\n<\/div>\nInsert IoT Data into GridDB<\/h3>\n
\nTo do so, we iterate through all the rows in our dataset, fetch the column data and column type for each row and insert the data using the container.put()<\/code> method.<\/p>\nIoT_Data<\/code> container.<\/p>\n\n\ntry:\n for _, row in dataset.iterrows():\n # Prepare each row's data in the exact order as defined in `column_info`\n row_data = [\n int(row[col]) if dtype == griddb.Type.INTEGER else\n float(row[col]) if dtype == griddb.Type.DOUBLE else\n str(row[col])\n for col, dtype in column_info\n ]\n # Insert the row data into the container\n container.put(row_data)\n\n print(f\"Successfully inserted {len(dataset)} rows of data into {container_name}\")\n\nexcept griddb.GSException as e:\n print(f\"Error inserting data into container {container_name}:\")\n for i in range(e.get_error_stack_size()):\n print(f\"[{i}] Error code: {e.get_error_code(i)}, Message: {e.get_message(i)}\")<\/code><\/pre>\n<\/div>\nSuccessfully inserted 200000 rows of data into IoT_Data<\/code><\/pre>\n<\/div>\nForecasting IoT Intrusion Using Machine Learning and Deep Learning<\/h2>\n
Retrieving Data From GridDB<\/h3>\n
query()<\/code> method and pass the SELECT *<\/code> query to the method. This will retrieve all the records from the container. Next, call the fetch()<\/code> method to retrieve the data from the container. Finally, call the fetch_rows()<\/code> method to store data in a Pandas dataframe.<\/p>\nretrieve_data_from_griddb()<\/code> method that retrieves data from a GridDB container into a Pandas dataframe.<\/p>\ndef retrieve_data_from_griddb(container_name):\n\n try:\n data_container = gridstore.get_container(container_name)\n\n # Query all data from the container\n query = data_container.query(\"select *\")\n rs = query.fetch() \n\n data = rs.fetch_rows()\n data .set_index(\"ID\", inplace=True)\n return data\n\n except griddb.GSException as e:\n print(f\"Error retrieving data from GridDB: {e.get_message()}\")\n return None\n\n\niot_data = retrieve_data_from_griddb(\"IoT_Data\")\niot_data.head()<\/code><\/pre>\n<\/div>\n
\n![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2025\/06\/img5-iot-data-from-griddb.png\")
<\/a><\/p>\nIoT Data Classification Using Machine Learning<\/h3>\n
# Separate the features (X) and the output class (y)\n\nX = iot_data.drop(columns=['label', 'class']) # Dropping both `label` and `class` columns as `class` is the target\ny = iot_data['class'] # Output target\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\nscaler = StandardScaler()\nX_train = scaler.fit_transform(X_train)\nX_test = scaler.transform(X_test)\n<\/code><\/pre>\n<\/div>\nRandomForestClassifier()<\/code> class from the sklearn<\/a> module and call its fit()<\/code> method to train the algorithm on the training data.<\/p>\nrf_model = RandomForestClassifier(random_state=42)\nrf_model.fit(X_train, y_train)<\/code><\/pre>\n<\/div>\npredict()<\/code> method and compare the predictions with the actual labels in the test set to calculate the model accuracy.<\/p>\n\nrf_predictions = rf_model.predict(X_test)\nrf_accuracy = accuracy_score(y_test, rf_predictions)\n\nclassification_rep = classification_report(y_test, rf_predictions, zero_division=1)\nprint(\"Classification Report:n\", classification_rep)<\/code><\/pre>\n<\/div>\n
\n![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2025\/06\/img6-machine-learning-classification-report.png\")
<\/a><\/p>\nBrute Force<\/code> had only 47 instances in the training set, the model is not able to learn much about this category.<\/p>\nIoT Data Classification Using Deep Learning<\/h3>\n
![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2025\/06\/img7-deep-learning-training-results.png\")
<\/a><\/p>\n![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2025\/06\/img8-deep-learning-classification-report.png\")
<\/a><\/p>\n