DevOps has changed how many developers and system administrators develop, publish, and operate their software services. These days developers will be using software such as Docker, Kubernetes, Jenkins, and more. In this blog, we’re going to build on our previous Docker post and show a ready built Dockerfile and startup script that allows developers to deploy individual, easily customizable containers for their GridDB server, Python client, and Java client.
First off, a user-created Docker network is required to allow clients to easily connect to the GridDB server, which can be created with the following command…
$ docker network create griddb-netIf you prefer to learn via video:
The Server
The server container consists of two parts, the Dockerfile and the griddb_start.sh script. The Dockerfile is quite simple, it installs the GridDB RPM from Github, sets environment variables, exposes the required ports, and then executes the griddb_start.sh script which does most of the heavy lifting.
$ docker pull griddbnet/griddbFROM centos:7 RUN rpm -Uvh https://github.com/griddb/griddb_nosql/releases/download/v4.2.1/griddb_nosql-4.2.1-1.linux.x86_64.rpm ENV GS_HOME /var/lib/gridstore ENV GS_LOG $GS_HOME/log ENV HOME $GS_HOME WORKDIR $HOME ADD start_griddb.sh / USER gsadm CMD /start_griddb.sh EXPOSE 10001 10010 10020 10030 10040 10050 10080 20001
In griddb_start.sh, the container’s IP address is retrieved so that it can be used within the gs_cluster.json configuration file which is written via cat. gs_node.json is also written with cat allowing changes to be easily made.
#!/bin/bash
chown gsadm.gridstore /var/lib/gridstore/data
IP=`grep $HOSTNAME /etc/hosts | awk ' { print $1 }'`
cat << EOF > /var/lib/gridstore/conf/gs_cluster.json
{
"dataStore":{
"partitionNum":128,
"storeBlockSize":"64KB"
},
"cluster":{
"clusterName":"defaultCluster",
"replicationNum":2,
"notificationInterval":"5s",
"heartbeatInterval":"5s",
"loadbalanceCheckInterval":"180s",
"notificationMember": [
{
"cluster": {"address":"$IP", "port":10010},
"sync": {"address":"$IP", "port":10020},
"system": {"address":"$IP", "port":10080},
"transaction": {"address":"$IP", "port":10001},
"sql": {"address":"$IP", "port":20001}
}
]
},
"sync":{
"timeoutInterval":"30s"
}
}
EOF
cat << EOF > /var/lib/gridstore/conf/gs_node.json
{
"dataStore":{
"dbPath":"data",
"backupPath":"backup",
"syncTempPath":"sync",
"storeMemoryLimit":"1024MB",
"storeWarmStart":false,
"storeCompressionMode":"NO_COMPRESSION",
"concurrency":4,
"logWriteMode":1,
"persistencyMode":"NORMAL",
"affinityGroupSize":4,
"autoExpire":false
},
"checkpoint":{
"checkpointInterval":"60s",
"checkpointMemoryLimit":"1024MB",
"useParallelMode":false
},
"cluster":{
"servicePort":10010
},
"sync":{
"servicePort":10020
},
"system":{
"servicePort":10040,
"eventLogPath":"log"
},
"transaction":{
"servicePort":10001,
"connectionLimit":5000
},
"trace":{
"default":"LEVEL_ERROR",
"dataStore":"LEVEL_ERROR",
"collection":"LEVEL_ERROR",
"timeSeries":"LEVEL_ERROR",
"chunkManager":"LEVEL_ERROR",
"objectManager":"LEVEL_ERROR",
"checkpointFile":"LEVEL_ERROR",
"checkpointService":"LEVEL_INFO",
"logManager":"LEVEL_WARNING",
"clusterService":"LEVEL_ERROR",
"syncService":"LEVEL_ERROR",
"systemService":"LEVEL_INFO",
"transactionManager":"LEVEL_ERROR",
"transactionService":"LEVEL_ERROR",
"transactionTimeout":"LEVEL_WARNING",
"triggerService":"LEVEL_ERROR",
"sessionTimeout":"LEVEL_WARNING",
"replicationTimeout":"LEVEL_WARNING",
"recoveryManager":"LEVEL_INFO",
"eventEngine":"LEVEL_WARNING",
"clusterOperation":"LEVEL_INFO",
"ioMonitor":"LEVEL_WARNING"
}
}
EOF
A password is set and the node started, this happens before a while loop waits for the GridDB service to finish recovering which then joins the cluster. Finally, tailing the gridstore log file, GridDB logs can be accessed easily using the log command.
gs_passwd admin -p admin
gs_startnode
while gs_stat -u admin/admin | grep RECOV > /dev/null; do
echo Waiting for GridDB to be ready.
sleep 5
done
gs_joincluster -n 1 -u admin/admin
tail -f /var/lib/gridstore/log/gridstore*.log
The image is built with docker build:
$ docker build -t griddb-server .Now, if you don’t need or want persistent data storage (if you kill the container or it otherwise shuts down, all data in GridDB will be lost) you can start the GridDB server container with the following:
$ docker run --network griddb-net --name griddb-server -d -t griddbnet/griddbIf you want data to be stored, first you need to create a volume to store it in:
$ docker volume create griddb-dataThen, you can start GridDB to include the mount option:
$ docker run --network griddb-net --name griddb-server \
--mount source=griddb-data,target=/var/lib/gridstore/data -d -t griddbnet/griddbThat’s it, your GridDB server Docker container should now be running.
$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES bef6eecc959d griddb-server "/bin/sh -c /start_g…" 5 minutes ago Up 5 minutes 10001/tcp, 10010/tcp, 10020/tcp, 10030/tcp, 10040/tcp, 10050/tcp, 10080/tcp, 20001/tcp griddb-server $ docker logs bef6 Waiting for GridDB to be ready. Waiting for GridDB to be ready. Waiting for GridDB to be ready. Waiting for GridDB to be ready. Waiting for GridDB to be ready. Waiting for GridDB to be ready. 2019-07-31T19:23:41.270Z bef6eecc959d 13 INFO RECOVERY_MANAGER [160903:RM_REDO_LOG_STATUS] Redo finished (pId=125, lsn=0) 2019-07-31T19:23:41.270Z bef6eecc959d 13 INFO RECOVERY_MANAGER [160903:RM_REDO_LOG_STATUS] Redo finished (pId=126, lsn=0) 2019-07-31T19:23:41.270Z bef6eecc959d 13 INFO RECOVERY_MANAGER [160903:RM_REDO_LOG_STATUS] Redo finished (pId=127, lsn=0) 2019-07-31T19:23:41.270Z bef6eecc959d 13 INFO CHECKPOINT_SERVICE [30902:CP_STATUS] [RecoveryCheckpoint] 2019-07-31T19:23:42.366Z bef6eecc959d 36 INFO CHECKPOINT_SERVICE [30902:CP_STATUS] [CP_START] mode=RECOVERY_CHECKPOINT, backupPath= 2019-07-31T19:23:48.913Z bef6eecc959d 36 INFO CHECKPOINT_SERVICE [30902:CP_STATUS] [CP_END] mode=RECOVERY_CHECKPOINT, backupPath=, commandElapsedMillis=6547 2019-07-31T19:23:48.913Z bef6eecc959d 20 INFO CLUSTER_OPERATION [40906:CS_NORMAL_OPERATION] Recovery checkpoint is completed, start all services 2019-07-31T19:24:42.417Z bef6eecc959d 36 INFO CHECKPOINT_SERVICE [30902:CP_STATUS] [CP_END] mode=NORMAL_CHECKPOINT, backupPath=, commandElapsedMillis=0 2019-07-31T19:24:42.785Z bef6eecc959d 16 INFO SYSTEM_SERVICE [50903:SC_WEB_API_CALLED] Join cluster called (clusterName=defaultCluster, clusterNodeNum=1) 2019-07-31T19:24:42.785Z bef6eecc959d 20 INFO CLUSTER_OPERATION [40906:CS_NORMAL_OPERATION] Called Join cluster 2019-07-31T19:24:47.486Z bef6eecc959d 20 INFO CLUSTER_OPERATION [40906:CS_NORMAL_OPERATION] Detect cluster status change (active nodeCount=1, checked nodeNum=1) 2019-07-31T19:24:47.486Z bef6eecc959d 20 WARNING CLUSTER_OPERATION [180061:CLM_STATUS_TO_MASTER] Cluster status change to MASTER... snip ... 2019-07-31T19:28:43.734Z bef6eecc959d 36 INFO CHECKPOINT_SERVICE [30902:CP_STATUS] [CP_END] mode=NORMAL_CHECKPOINT, backupPath=, commandElapsedMillis=0
Java Container
The GridDB Java client Dockerfile is quite simple; the sequence of steps are: installing the OpenJDK Development package and then the GridDB RPM, adding the source, compiling, and executing it.
FROM centos:7 RUN yum install -y java-1.8.0-openjdk-devel RUN rpm -Uvh https://github.com/griddb/griddb_nosql/releases/download/v4.2.1/griddb_nosql-4.2.1-1.linux.x86_64.rpm ADD Sample1.java / ENV CLASSPATH /usr/share/java/gridstore.jar:$CLASSPATH RUN javac Sample1.java CMD java Sample1
The only particular source code change required is with the Gridstore Properties object which connects to “griddb-server”, the griddb-server container’s name.
Properties props = new Properties();
props.setProperty("notificationMember", "griddb-server:10001");
props.setProperty("clusterName", "defaultCluster");
props.setProperty("user", "admin");
props.setProperty("password", "admin");
If you don’t want to run your application in a Docker container, you can change the notificationMember to point to the IP address of the server container. The following command will output its IP address if you do not know how to look it up:
$ CONT=`docker ps | grep griddb-server | awk '{ print $1 }'`; docker exec $CONT cat /etc/hosts | grep $CONT | awk '{ print $1 }'
172.18.0.2
$ CONT=`docker ps | grep griddb-server | awk '{ print $1 }'`; docker exec $CONT cat /etc/hosts | grep $CONT | awk '{ print $1 }'172.18.0.2
Building and running are both quite simple:
$ docker build -t griddb-java . Sending build context to Docker daemon 5.12kB Step 1/7 : FROM centos:7 ---> 9f38484d220f Step 2/7 : RUN yum install -y java-1.8.0-openjdk-devel ---> Using cache ---> 34e283d179ac Step 3/7 : RUN rpm -Uvh https://github.com/griddb/griddb_nosql/releases/download/v4.2.1/griddb_nosql-4.2.1-1.linux.x86_64.rpm ---> Using cache ---> 83669ae6deda Step 4/7 : ADD Sample1.java / ---> Using cache ---> 042eea3c5645 Step 5/7 : ENV CLASSPATH /usr/share/java/gridstore.jar:$CLASSPATH ---> Using cache ---> c6e5f9a2760a Step 6/7 : RUN javac Sample1.java ---> Using cache ---> b0c5757cdb90 Step 7/7 : CMD java Sample1 ---> Using cache ---> eb70502014bd Successfully built eb70502014bd Successfully tagged griddb-java:latest
As is running the container:
$ docker run --network griddb-net -t griddb-javaPerson: name=name02 status=false count=2 lob=[65, 66, 67, 68, 69, 70, 71, 72, 73, 74]
Python Container
The Python client’s Dockerfile is a bit more complex as PCRE, SWIG, and the Python Client need to be compiled.
$ docker pull griddbnet/griddb-pythonFROM centos:7 RUN yum -y groupinstall "Development Tools" RUN yum -y install epel-release wget RUN yum -y install python36 python36-devel RUN rpm -Uvh https://github.com/griddb/c_client/releases/download/v4.2.0/griddb_c_client-4.2.0-1.linux.x86_64.rpm RUN ln -sf /usr/include/python3.6m /usr/include/python3.6 RUN wget https://sourceforge.net/projects/pcre/files/pcre/8.39/pcre-8.39.tar.gz RUN tar xvfz pcre-8.39.tar.gz RUN cd pcre-8.39 && ./configure && make && make install RUN cd .. RUN wget https://prdownloads.sourceforge.net/swig/swig-3.0.12.tar.gz RUN tar xvfz swig-3.0.12.tar.gz RUN cd swig-3.0.12 && ./configure && make && make install RUN cd .. RUN wget https://github.com/griddb/python_client/archive/0.8.1-rc0.tar.gz RUN tar xvfz 0.8.1-rc0.tar.gz RUN cd python_client-0.8.1-rc0 && make ENV PYTHONPATH /python_client-0.8.1-rc0 ADD sample1.py / CMD /sample1.py
The only changes required the python source code is changing Gridstore parameters to point to the server container:
gridstore = factory.get_store(
notification_member="griddb-server:10001",
cluster_name="defaultCluster",
username="admin",
password="admin"
)Like with Java, if you don’t want to run your application in a Docker container, you can change the notificationMember to point to the IP address of the server container. The following command will output its IP address. If you do not know how to look it up:
$ CONT=`docker ps | grep griddb-server | awk '{ print $1 }'`; docker exec $CONT cat /etc/hosts | grep $CONT | awk '{ print $1 }'172.18.0.2
Building is essentially the same as the Java client:
$ docker build -t griddb-python . Sending build context to Docker daemon 5.12kB Step 1/20 : FROM centos:7 ---> 9f38484d220f Step 2/20 : RUN yum -y groupinstall "Development Tools" ---> Using cache ---> 2eb55fc14acb Step 3/20 : RUN yum -y install epel-release wget ---> Using cache ---> 3f8a1bfb8999 Step 4/20 : RUN yum -y install python36 python36-devel ---> Using cache ---> 8ced1273c292 Step 5/20 : RUN rpm -Uvh https://github.com/griddb/c_client/releases/download/v4.2.0/griddb_c_client-4.2.0-1.linux.x86_64.rpm ---> Using cache ---> e5188e0704e0 Step 6/20 : RUN ln -sf /usr/include/python3.6m /usr/include/python3.6 ---> Using cache ---> 4828bdbb0711 Step 7/20 : RUN wget https://sourceforge.net/projects/pcre/files/pcre/8.39/pcre-8.39.tar.gz ---> Using cache ---> b57e7fa48df5 Step 8/20 : RUN tar xvfz pcre-8.39.tar.gz ---> Using cache ---> f06336db97c0 Step 9/20 : RUN cd pcre-8.39 && ./configure && make && make install ---> Using cache ---> c28540943d49 Step 10/20 : RUN cd .. ---> Using cache ---> 3b24d85b842c Step 11/20 : RUN wget https://prdownloads.sourceforge.net/swig/swig-3.0.12.tar.gz ---> Using cache ---> 71df54189d11 Step 12/20 : RUN tar xvfz swig-3.0.12.tar.gz ---> Using cache ---> 96cbb828bef0 Step 13/20 : RUN cd swig-3.0.12 && ./configure && make && make install ---> Using cache ---> 9af12ad99d1d Step 14/20 : RUN cd .. ---> Using cache ---> e1dd5e9259b7 Step 15/20 : RUN wget https://github.com/griddb/python_client/archive/0.8.1-rc0.tar.gz ---> Using cache ---> 746a4db157bf Step 16/20 : RUN tar xvfz 0.8.1-rc0.tar.gz ---> Using cache ---> 278cd4d8b37a Step 17/20 : RUN cd python_client-0.8.1-rc0 && make ---> Using cache ---> ead9d38a0424 Step 18/20 : ENV PYTHONPATH /python_client-0.8.1-rc0 ---> Using cache ---> f5326e3a4c7d Step 19/20 : ADD sample1.py / ---> c5426a34cc29 Step 20/20 : CMD /sample1.py ---> Running in b4b197e78139 Removing intermediate container b4b197e78139 ---> 95e6c3cc7859 Successfully built 95e6c3cc7859 Successfully tagged griddb-python:latest
Running the Python Client:
$ docker run --network griddb-net -t griddbnet/griddb-pythonPerson: name=name02 status=False count=2 lob=[65, 66, 67, 68, 69, 70, 71, 72, 73, 74]
We hope this blog post has helped demonstrate just how easy it is to start using GridDB in Docker containers. If you’d like to download the Dockerfiles, startup scripts, and samples they can be downloaded here: [download id=”26095″] (zip)
If you have any questions about the blog, please create a Stack Overflow post here https://stackoverflow.com/questions/ask?tags=griddb .
Make sure that you use the “griddb” tag so our engineers can quickly reply to your questions.