Run the Project<\/h2>\n
To launch the project, visit the project’s repository<\/a>. Clone the repository, ensure that you have Node.js version 18, install all required dependencies, and start the project.<\/p>\n Go to http:\/\/localhost:5173 in the browser to open the web map.<\/p>\n The development flow for this project can be breakdown into a few steps:<\/p>\n First, we need to get real-time data from worldometers<\/a>. We can get the data by scrapping the website or using API and then getting the needed data.<\/p>\n The data will be extracted at regular intervals and then stored in the GridDB database. The data stored can be used for further analysis, prediction, reporting, etc.<\/p>\n We will use Node.js to code the server-side application server.<\/p>\n Why Node.js?<\/em><\/p>\n JavaScript is the lingua franca for full-stack application development, so it’s the obvious choice.<\/p>\n<\/blockquote>\n The application server we make is WebSocket-based, and Node.js will interact with GridDB for data storage, handle retrieval data from worldometers, and handle requests and responses from the client UI.<\/p>\n We use React.js to create the user interface and to display the world population data visually, we’ll use Mapbox GL JS, a powerful mapping library. This library enables the creation of interactive, customizable maps. We’ll integrate Mapbox GL JS with our React application to render a world map with labels representing the population data.<\/p>\n Before we code the application, we need to set up the software and tools for development. We use Ubuntu 20.04 on WSL 2 on Windows 11 OS.<\/p>\n WSL is only available on Windows 10 version 2004 and higher (Build 19041 and higher). Go ahead to this link<\/a> for a new WSL installation.<\/p>\n<\/blockquote>\n GridDB<\/a>\u2122 is a highly scalable, in-memory NoSQL time-series database optimized for IoT and Big Data. What’s important to note is it has two types of container categories:<\/p>\n This type of container is similar to a traditional relational database. The data is stored as key values, and the collection container support basic CRUD (Create, Read, Update, and Delete) operation.<\/p>\n This container is designed explicitly for managing time-series data, a sequence of data points indexed by time. Each record in a TimeSeries container has a timestamp, which serves as its unique key, and the data in this container is “append only” except upon deletion if requested.<\/p>\n To install GridDB on WSL there are a few things that we should know.<\/p>\n \u26a0\ufe0f GridDB deb package uses Inside your Ubuntu instance, add the following modification to Open the Windows terminal to restart wsl with the following command.<\/p>\n Then start wsl again with the command.<\/p>\n To install GridDB, follow the installation instruction on https:\/\/docs.griddb.net\/latest\/gettingstarted\/using-apt\/#install-with-apt-get<\/a>.<\/p>\n Run GridDB and check if the service is running. Use this command<\/p>\n And if everything ok you will get a message like this.<\/p>\n No need to start GridDB manually because on every OS reboot, this service will go up.<\/p>\n To install2<\/a><\/sup> Node.js LTS, follow the commands below<\/p>\n We will be using griddb node-api<\/a> to connect our application with GridDB, but before that, we should install the griddb c client<\/a>. The GridDB C Client provides a C interface for GridDB.<\/p>\n To connect to GridDB from Node.js, we should use 1. Compile from the source code. This way is the right way if you need to use griddb-node-api on a specific node.js version.<\/p>\n If there is an error message like this<\/p>\n That means you should install the 2. Install the npm package We will use the second way for our project because it’s simpler, and we use Node 18 LTS.<\/p>\n As stated earlier, we use OS Ubuntu 20.04 on WSL Windows 11. To code from Windows, we use Visual Studio Code because it has a remarkable remote WSL plugin<\/a><\/p>\n Ok. That’s a long setup before we code the application itself, but it’s necessary so our development environment is transparent.<\/p>\n We use To activate Why monorepo?<\/em><\/p>\n Because it’s good for future development, if you want to add another collaborator, everyone will be working on the same code base.<\/p>\n<\/blockquote>\n Start the project by creating a directory for the monorepo<\/p>\n Then create This tree structure might be typical of a Node.js project that has separate client and server codebases. Organized as packages within a monorepo and managed with the The first thing to do is initialize the server project and install the main npm packages.<\/p>\n This project uses data from Worldometers.<\/p>\n Worldometers is a website that provides real-time statistics on various topics, including world population, government and economics, society and media, the environment, food, water, energy, and health.<\/p>\n<\/blockquote>\n To get data from a website there are a few ways we can use:<\/p>\n Unfortunately, Worldometers does not provide API, so our last option is to scrap the website.<\/p>\n One thing to note is Worldometers have dynamic data, meaning they provide real-time data. You cannot use JavaScript libraries such as Cheerio for data extraction. The best choice is to use Puppeteer to get such dynamic content.<\/p>\n Puppeteer can be more resource-intensive as it launches a headless browser instance to render web pages. However, it offers more capabilities, such as handling dynamic content and user interactions.<\/p>\n<\/blockquote>\n Two endpoint URLs will provide us with the data. One is for the total world population, and the second is for the total world population by country.<\/p>\n The data returned by The code for data extraction of world population by country is in the repository project.<\/p>\n The output will be an array, with keys: We will use time-series containers for storing world population data. As with any other database, we need to connect to it first. On GridDB, we need to connect to the cluster.<\/p>\n You can see the cluster configuration at<\/p>\n And make sure that the port we put in the code is the value of the transaction key because there are a few ports there.<\/p>\n After that, we need to create a container to store our data. The Please note that the Store the data is pretty easy. First, get the container connection, then use The important thing to note for this project is the data should be stored periodically. The simplest way to do this is by using JavaScript native function, The default time to extract data is set by variable GridDB supports SQL, so you can use raw SQL to retrieve the data. By using For the projet purposes, it need to display the latest data with the latest time stamp. We can use SQL queries to achieve this.<\/p>\n We use WebSocket to deliver data to the project web client, built with React and Mapbox.<\/p>\n Why WebSocket?<\/em><\/p>\n WebSocket is the optimal choice for applications requiring intensive data usage because it enables real-time, bidirectional communication between the client and server. WebSocket allows for efficient data transfer and minimizes latency, which is essential in data-intensive applications where continuous updates and rapid responsiveness are crucial.<\/p>\n<\/blockquote>\n This snippet code shows us how the world population data is saved to and queried from GridDB and then sent it to the web client via WebSocket In this project, we use ws<\/a> for WebSocket and express.js<\/a> for HTTP server.<\/p>\n There are so many tools today for creating React-based applications. Vite<\/a> is a next-generation front-end tool. It offers enhanced speed, supports ESM, and React. Making it a standout choice.<\/p>\n To create a new frontend project using Vite.<\/p>\n The command will ask you a few questions and make sure to choose JavaScript and React. The source code for web client is live inside the folder The client side project structure.<\/p>\n There is not much to change in the files or directories created by Vite, except Mapbox GL JS<\/a> is a JavaScript library for vector maps on the web. Its performance, real-time styling, and interactivity features set the bar for anyone building fast and immersive web maps.<\/p>\n Add Mapbox GL JS to our client-side application.<\/p>\n Since modern browsers already support WebSocket natively, we don’t need an additional package to connect to our WebSocket server.<\/p>\n And to get the world population data, just listen to With React, we can build components easily. For this project, we use it for creating a sidebar.<\/p>\n This UI shows the total world population. Using React Conceptually, this project appears straightforward. However, the implementation of a web application to visualize global population data using Mapbox, Node.js, and GridDB with real-time live data integration and live UI updates presents a more complex challenge. The real-world nature of the data and the dynamic aspects of the application add multiple layers of complexity, making it a non-trivial task to execute effectively.<\/p>\n https:\/\/ubuntu.com\/blog\/ubuntu-wsl-enable-systemd ↩<\/a><\/p>\n<\/li>\ngit clone git@github.com:junwatu\/griddb-codes.git\n\ncd griddb-codes\n\ncore enable\n\ncorepack prepare pnpm@7.30.0 --activate\n\npnpm install\n\ncd packages\/data-server\/\n\npnpm install\n\ncd ..\/client\n\npnpm install\n\ncd ..\/..\/\n\nnpm start<\/code><\/pre>\n<\/div>\n![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2023\/03\/run-world-population-project.gif\")
<\/a><\/p>\nThe Development Flow<\/h2>\n
![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2023\/03\/project-diagram.svg\")
<\/a><\/p>\nData Acquisition<\/h2>\n
Back-end Development<\/h2>\n
\n
Front-end Development<\/h2>\n
Prerequisite<\/h2>\n
\n
GridDB<\/h2>\n
Collection Containers<\/h3>\n
TimeSeries Containers<\/h3>\n
Installation<\/h3>\n
systemd<\/code>, but Ubuntu 20.04 on WSL 2 Windows 11 uses SysVinit<\/code>, so you need to enable systemd<\/code> on Ubuntu WSL by editing the file \/etc\/wsl.conf<\/code> (create it if this file doesn’t exist)<\/p>\n\/etc\/wsl.conf<\/code>1<\/a><\/sup><\/p>\n[boot]\nsystemd=true<\/code><\/pre>\n<\/div>\nwsl --shutdown<\/code><\/pre>\n<\/div>\nwsl<\/code><\/pre>\n<\/div>\nsudo systemctl status gridstore<\/code><\/pre>\n<\/div>\n\u25cf gridstore.service - GridDB database server.\n Loaded: loaded (\/lib\/systemd\/system\/gridstore.service; enabled; vendor preset: enabled)\n Active: active (running) since Thu 2023-03-16 18:56:13 +07; 58min ago\n Process: 314 ExecStart=\/usr\/griddb\/bin\/gridstore start (code=exited, status=0\/SUCCESS)\n Main PID: 393 (gsserver)\n Tasks: 34 (limit: 4605)\n Memory: 138.5M\n CPU: 20.129s\n CGroup: \/system.slice\/gridstore.service\n \u2514\u2500393 \/usr\/bin\/gsserver --conf \/var\/lib\/gridstore\/conf\n\nMar 16 18:56:10 GenAI systemd[1]: Starting GridDB database server....\nMar 16 18:56:10 GenAI gridstore[314]: Starting gridstore service:\nMar 16 18:56:13 GenAI gridstore[392]: ..\nMar 16 18:56:13 GenAI gridstore[392]: Started node.\nMar 16 18:56:13 GenAI gridstore[314]: [ OK ]\nMar 16 18:56:13 GenAI systemd[1]: Started GridDB database server..<\/code><\/pre>\n<\/div>\nNode.js<\/h2>\n
curl -fsSL https:\/\/deb.nodesource.com\/setup_18.x | sudo -E bash - &&\nsudo apt-get install -y nodejs<\/code><\/pre>\n<\/div>\nwget https:\/\/github.com\/griddb\/c_client\/releases\/download\/v5.0.0\/griddb-c-client_5.0.0_amd64.deb\nsudo dpkg -i griddb-c-client_5.0.0_amd64.deb<\/code><\/pre>\n<\/div>\nGridDB node-api<\/h3>\n
griddb-node-api<\/code>. This package is built using node-addon-api, and there are two ways to use it:<\/p>\ngit clone git@github.com:griddb\/node-api.git\ncd node-api\nnpm install<\/code><\/pre>\n<\/div>\ngyp ERR! stack Error: not found: make<\/code><\/pre>\n<\/div>\nbuild-essentials<\/code> package in Ubuntu, and if everything success you will get a file, griddb.node<\/code>, and then you need to include it in NODE_PATH<\/code><\/p>\nexport NODE_PATH=$(pwd)<\/code><\/pre>\n<\/div>\ngriddb-node-api<\/code> directly into our project.<\/p>\nnpm install griddb-node-api<\/code><\/pre>\n<\/div>\nVSCode<\/h2>\n
![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2023\/03\/vscode-remote.png\")
<\/a><\/p>\nProject Structure Directory<\/h2>\n
pnpm<\/code> instead of npm because pnpm<\/code> is storage efficient and supports workspaces. We will create a monorepo that holds server and client codes.<\/p>\npnpm<\/code> use corepack<\/code> from Node.js<\/p>\ncorepack prepare pnpm@7.30.0 --activate<\/code><\/pre>\n<\/div>\n\n
mkdir world-population\ncd world-population\npnpm init<\/code><\/pre>\n<\/div>\npackages<\/code> directory that will hold our project codes. Server code in data-server<\/code> directory and client code in client<\/code> directory.<\/p>\nmkdir packages\nmkdir packages\/data-server\nmkdir packages\/client<\/code><\/pre>\n<\/div>\npnpm<\/code> handling project workspaces by reading the config from pnpm-workspaces.yaml<\/code>, so you need to create it and fill with the content below.<\/p>\n# pnpm-workspaces.yaml\npackages:\n # all packages in direct subdirs of packages\/\n - \"packages\/*<\/code><\/pre>\n<\/div>\npnpm<\/code> package manager.<\/p>\n.\n\u251c\u2500\u2500 .gitignore\n\u251c\u2500\u2500 package.json\n\u251c\u2500\u2500 packages\n\u2502 \u251c\u2500\u2500 data-server\n\u2502 \u2514\u2500\u2500 client\n\u2514\u2500\u2500 pnpm-workspaces.yaml<\/code><\/pre>\n<\/div>\nLet’s Code<\/h2>\n
cd packages\/data-server\npnpm init\ncd ..\/..\/\npnpm --filter data-server install griddb-node-api express ws puppeteer<\/code><\/pre>\n<\/div>\nData Extraction<\/h3>\n
\n
\n
\n
\n
const worldPopDataSource = \"https:\/\/www.worldometers.info\/world-population\/\";\n\/**\n * fetch world population data\n *\/\nconst fetchWorldPopulationData = async () => {\n const browser = await puppeteer.launch();\n const page = await browser.newPage();\n\n await page.goto(worldPopDataSource);\n\n const extractData = async () => {\n \/\/ Extract data from the page\n const data = await page.evaluate(() => {\n const populationElement = document.querySelector(\".rts-counter\");\n const population = populationElement.textContent.replace(\/[ns]+\/g, \"\");\n return { population };\n });\n\n return data;\n };\n\n const popData = await extractData();\n await browser.close();\n\n return popData;\n};<\/code><\/pre>\n<\/div>\nfetchWorldPopulationData<\/code> function is simply a JavaScript object.<\/p>\n{\n type: 'worldPopulationData',\n worldPopulation: {\n population: '8,022,758,957',\n timestamp: 1679222700112\n }\n}<\/code><\/pre>\n<\/div>\ncountry<\/code> name, country coordinate<\/code> and the population<\/code>.<\/p>\n![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2023\/03\/world-pop-by-country.png\")
<\/a><\/p>\nData Store<\/h2>\n
\/var\/lib\/gridstore\/conf\/gs_cluster.json<\/code><\/pre>\n<\/div>\nconst initStore = async () => {\n const factory = griddb.StoreFactory.getInstance();\n try {\n \/\/ Connect to GridDB Cluster\n const store = await factory.getStore({\n host: \"127.0.0.1\",\n port: 10001,\n clusterName: \"myCluster\",\n username: \"admin\",\n password: \"admin\",\n });\n return store;\n } catch (e) {\n throw e;\n }\n};<\/code><\/pre>\n<\/div>\ninitContainer()<\/code> function provides information about the container’s schema, including the names and data types of its columns.<\/p>\nfunction initContainer() {\n const conInfo = new griddb.ContainerInfo({\n name: containerName,\n columnInfoList: [\n [\"timestamp\", griddb.Type.TIMESTAMP],\n [\"value\", griddb.Type.DOUBLE],\n ],\n type: griddb.ContainerType.TIME_SERIES,\n });\n\n return conInfo;\n}<\/code><\/pre>\n<\/div>\ninitContainer()<\/code> function will not create a container. To create a container, GridDB API provide us with putContainer()<\/code> function.<\/p>\nasync function createContainer(store, conInfo) {\n try {\n const timeSeriesDB = await store.putContainer(conInfo);\n return timeSeriesDB;\n } catch (err) {\n console.error(err);\n throw err;\n }\n}<\/code><\/pre>\n<\/div>\nHow to Store the Data<\/h3>\n
put()<\/code> to save the data into the GridDB container.<\/p>\ndb.put(data);<\/code><\/pre>\n<\/div>\ndb<\/code> is the connection reference that is returned by createConnection()<\/code> function.<\/p>\nsetInterval()<\/code>.<\/p>\nfunction updateClientsWithWorldPopulationDataPeriodically(clients) {\n updateClientsWithWorldPopulationData(clients);\n setTimeout(\n () => updateClientsWithWorldPopulationDataPeriodically(clients),\n worldDataUpdateTime\n );\n}\n\n\/\/ Call the function to start the periodic updates\nupdateClientsWithWorldPopulationDataPeriodically(wss.clients);<\/code><\/pre>\n<\/div>\nworldDataUpdateTime<\/code> which is 5 second. Means that, every 5 second we update the data.<\/p>\nRead Data from GridDB<\/h2>\n
query()<\/code> and fetch()<\/code> we can easily get the data based on a SQL query.<\/p>\nasync function queryAll(db) {\n const query = db.query(\n `SELECT * FROM ${containerName} ORDER BY timestamp DESC LIMIT 1`\n );\n\n try {\n const rowset = await query.fetch();\n const results = [];\n\n while (rowset.hasNext()) {\n const row = rowset.next();\n const rowData = { timestamp: `${row[0]}`, population: row[1] };\n results.push(rowData);\n }\n\n return results;\n } catch (err) {\n console.log(err);\n throw err;\n }\n}<\/code><\/pre>\n<\/div>\nsql\nSELECT * FROM [containerName] ORDER BY timestamp DESC LIMIT 1<\/code><\/pre>\n<\/div>\nDeliver Data to the Web Client<\/h2>\n
\n
ws<\/code>.<\/p>\nconst worldPopData = [worldPopulation.timestamp, worldPopulation.population];\nawait GridDB.insert(worldPopData, timeSeriesDb);\n\nconst result = await GridDB.queryAll(timeSeriesDb);\n\nconst jsonArray = result.map((item) => {\n return {\n timestamp: item.timestamp.toString(),\n population: item.population,\n };\n});\n\n\/\/ client is WebSocket client\nclients.forEach((client) => {\n if (client.readyState === WebSocket.OPEN) {\n try {\n client.send(JSON.stringify(jsonArray));\n } catch (error) {\n console.error(\"Error sending data to client:\", error);\n }\n }\n});<\/code><\/pre>\n<\/div>\nWeb Client React + Mapbox GL JS<\/h2>\n
pnpm create vite\ncd client\npnpm install<\/code><\/pre>\n<\/div>\npackages\/client<\/code>.<\/p>\n.\n\u251c\u2500\u2500 index.html\n\u251c\u2500\u2500 node_modules\n\u251c\u2500\u2500 package.json\n\u251c\u2500\u2500 pnpm-lock.yaml\n\u251c\u2500\u2500 public\n\u2502 \u2514\u2500\u2500 vite.svg\n\u251c\u2500\u2500 src\n\u2502 \u251c\u2500\u2500 App.css\n\u2502 \u251c\u2500\u2500 App.jsx\n\u2502 \u251c\u2500\u2500 assets\n\u2502 \u2502 \u2514\u2500\u2500 react.svg\n\u2502 \u251c\u2500\u2500 index.css\n\u2502 \u2514\u2500\u2500 main.jsx\n\u2514\u2500\u2500 vite.config.js<\/code><\/pre>\n<\/div>\nApp.jsx<\/code> and index.css<\/code> files.<\/p>\nApp.jsx<\/code> is where we code our react component and UI for Mapbox. There are only two components. One is sidebar<\/code> to show the total world population, and the second is map-container<\/code> to display the world map and label each country’s total population.<\/p>\n<div className=\"App\">\n <div className=\"sidebar\">World Population: {worldPopulation}<\/div>\n <div ref=\"{mapContainer}\" className=\"map-container\" \/>\n<\/div><\/code><\/pre>\n<\/div>\nMapbox GL JS<\/h3>\n
pnpm install mapbox-gl<\/code><\/pre>\n<\/div>\nconst ws = new WebSocket(\"ws:\/\/localhost:3000\");\nsetSocket(ws);<\/code><\/pre>\n<\/div>\nmessage<\/code> event<\/p>\nws.addEventListener(\"message\", (event) => {\n const data = JSON.parse(event.data);\n\n if (data[0].country) {\n console.log(\"country\", data);\n updateLabels(data);\n } else {\n setWorldPopulation(data[0].population);\n }\n});<\/code><\/pre>\n<\/div>\nupdateLabels(data)<\/code> function will update every country’s label in the top 10 world population.<\/p>\n![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2023\/03\/country-map-label.png\")
<\/a><\/p>\nReact<\/h3>\n
![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2023\/03\/sidebar.png\")
<\/a><\/p>\nuseState<\/code>, every data update from WebSocket will trigger and render new data on the sidebar.<\/p>\nconst [worldPopulation, setWorldPopulation] = useState(0);\n\n\/\/...\/\/\n\nws.addEventListener(\"message\", (event) => {\n const data = JSON.parse(event.data);\n\n if (data[0].country) {\n console.log(\"country\", data);\n updateLabels(data);\n } else {\n setWorldPopulation(data[0].population);\n }\n});<\/code><\/pre>\n<\/div>\n
\n\n