Blog

Project Overview A modern web application that transforms spoken descriptions into high-quality images using cutting-edge AI technologies. Users can record their voice describing an image they want to create, and the system will transcribe their speech and generate a corresponding image. What Problem We Solved Traditional image generation tools require users to type detailed prompts, which can be: Time-consuming for complex descriptions. Limiting for users with typing difficulties. Less natural than speaking. This project solution makes AI image generation more accessible through voice interaction. Architecture & Tech Stack This diagram shows a pipeline for converting speech into images and storing the result in the GridDB database: User speaks into a microphone. Speech recording captures the audio. Audio is sent to ElevenLabs (Scriber-1) for speech-to-text transcription. The transcribed text becomes a prompt for Imagen 4 API, which generates an image. The data saved into the database are: The audio reference and The image The prompt text Frontend Stack In this project, we will use Next.js as the frontend framework. Backend Stack There is no specific backend code because all services used are from APIs. There are three main APIs: 1. Speech-to-Text API This project utilizes the ElevenLabs API for speech-to-text transcription. The API is athttps://api.elevenlabs.io/v1/text-to-speech/. ElevenLabs also provides JavaScript SDK for easier API integration. You can see the SDK documentation for more details. 2. Image Generation API This project uses Imagen 4 API from fal. The API is hosted on https://fal.ai/models/fal-ai/imagen4/preview. Fal provides JavaScript SDK for easier API integration. You can see the SDK documentation for more details. 3. Database API We will use the GridDB Cloud version in this project. So there is no need for local installation. Please read the next section on how to set up GridDB Cloud. Prerequisites Node.js This project is built using Next.js, which requires Node.js version 16 or higher. You can download and install Node.js from https://nodejs.org/en. GridDB Sign Up for GridDB Cloud Free Plan If you would like to sign up for a GridDB Cloud Free instance, you can do so at the following link: https://form.ict-toshiba.jp/download_form_griddb_cloud_freeplan_e. After successfully signing up, you will receive a free instance along with the necessary details to access the GridDB Cloud Management GUI, including the GridDB Cloud Portal URL, Contract ID, Login, and Password. GridDB WebAPI URL Go to the GridDB Cloud Portal and copy the WebAPI URL from the Clusters section. It should look like this: GridDB Username and Password Go to the GridDB Users section of the GridDB Cloud portal and create or copy the username for GRIDDB_USERNAME. The password is set when the user is created for the first time, use this as the GRIDDB_PASSWORD. For more details, to get started with GridDB Cloud, please follow this quick start guide. IP Whitelist When running this project, please ensure that the IP address where the project is running is whitelisted. Failure to do so will result in a 403 status code or forbidden access. You can use a website like What Is My IP Address to find your public IP address. To whitelist the IP, go to the GridDB Cloud Admin and navigate to the Network Access menu. ElevenLabs You need an ElevenLabs account and API key to use this project. You can sign up for an account at https://elevenlabs.io/signup. After signing up, go to the Account section, and create and copy your API key. Imagen 4 API You need an Imagen 4 API key to use this project. You can sign up for an account at https://fal.ai. After signing up, go to the Account section, and create and copy your API key. How to Run 1. Clone the repository Clone the repository from https://github.com/junwatu/speech-image-gen to your local machine. git clone https://github.com/junwatu/speech-image-gen.git cd speech-image-gen cd apps 2. Install dependencies Install all project dependencies using npm. npm install 3. Set up environment variables Copy file .env.example to .env and fill in the values: # Copy this file to .env.local and add your actual API keys # Never commit .env.local to version control # Fal.ai API Key for Imagen 4 # Get your key from: https://fal.ai/dashboard FAL_KEY= # ElevenLabs API Key for Speech-to-Text # Get your key from: https://elevenlabs.io/app/speech-synthesis ELEVENLABS_API_KEY= GRIDDB_WEBAPI_URL= GRIDDB_PASSWORD= GRIDDB_USERNAME= Please look the section on Prerequisites before running the project. 4. Run the project Run the project using the following command: npm run dev 5. Open the application Open the application in your browser at http://localhost:3000. You also need to allow the browser to access your microphone. Implementation Details Speech Recording The user will speak into the microphone and the audio will be recorded. The audio will be sent to ElevenLabs API for speech-to-text transcription. Please, remember that the language supported is English. The code to save the recording file is in the main page.tsx. It uses a native media recorder HTML 5 API to record the audio. Below is the snippet code: const startRecording = useCallback(async () => { try { setError(null); const stream = await navigator.mediaDevices.getUserMedia({ audio: { echoCancellation: true, noiseSuppression: true, sampleRate: 44100 } }); // Try different MIME types based on browser support let mimeType = ‘audio/webm;codecs=opus’; if (!MediaRecorder.isTypeSupported(mimeType)) { mimeType = ‘audio/webm’; if (!MediaRecorder.isTypeSupported(mimeType)) { mimeType = ‘audio/mp4’; if (!MediaRecorder.isTypeSupported(mimeType)) { mimeType = ”; // Let browser choose } } } const mediaRecorder = new MediaRecorder(stream, { …(mimeType && { mimeType }) }); mediaRecorderRef.current = mediaRecorder; audioChunksRef.current = []; recordingStartTimeRef.current = Date.now(); mediaRecorder.ondataavailable = (event) => { if (event.data.size > 0) { audioChunksRef.current.push(event.data); } }; mediaRecorder.onstop = async () => { const duration = Date.now() – recordingStartTimeRef.current; const audioBlob = new Blob(audioChunksRef.current, { type: mimeType || ‘audio/webm’ }); const audioUrl = URL.createObjectURL(audioBlob); const recording: AudioRecording = { blob: audioBlob, url: audioUrl, duration, timestamp: new Date() }; setCurrentRecording(recording); await transcribeAudio(recording); stream.getTracks().forEach(track => track.stop()); }; mediaRecorder.start(1000); // Collect data every second setIsRecording(true); } catch (error) { setError(‘Failed to access microphone. Please check your permissions and try again.’); } }, []); The audio processing flow is as follows: User clicks record button → startRecording() is called. Requests microphone access via getUserMedia(). Creates MediaRecorder with optimal settings. Collects audio data in chunks. When stopped, create an audio blob and trigger transcription. The audio data will be saved in the public/uploads/audio folder. Below is the snippet code to save the audio file: export async function saveAudioToFile(audioBlob: Blob, extension: string = ‘webm’): Promise { // Create uploads directory if it doesn’t exist const uploadsDir = join(process.cwd(), ‘public’, ‘uploads’, ‘audio’); await mkdir(uploadsDir, { recursive: true }); // Generate unique filename const filename = `${generateRandomID()}.${extension}`; const filePath = join(uploadsDir, filename); // Convert blob to buffer and save file const arrayBuffer = await audioBlob.arrayBuffer(); const buffer = Buffer.from(arrayBuffer); await writeFile(filePath, buffer); // Return relative path for storage in database return `/uploads/audio/${filename}`; } The full code for the saveAudioToFile() function is in the app/lib/audio-storage.ts file. Speech to Text Transcription The transcribed text will be sent to ElevenLabs API for text-to-speech synthesis. The code to send the audio to ElevenLabs API is in the transcribeAudio() function. The full code is in the lib/elevenlabs-client.ts file. // Main transcription function export async function transcribeAudio( client: ElevenLabsClient, audioBuffer: Buffer, modelId: ElevenLabsModel = ELEVENLABS_MODELS.SCRIBE_V1 ) { try { const result = await client.speechToText.convert({ audio: audioBuffer, model_id: modelId, }) as TranscriptionResponse; return { success: true, text: result.text, language_code: result.language_code, language_probability: result.language_probability, words: result.words || [], additional_formats: result.additional_formats || [] }; } catch (error) { console.error(‘ElevenLabs transcription error:’, error); return { success: false, error: error instanceof Error ? error.message : ‘Unknown error’ }; } } Transcription Route The transcribeAudio() function is called when accessing the /api/transcribe route. This route only accepts the POST method and processes the audio file sent in the request body. The ELEVENLABS_API_KEY environment variable in the .env is used in the route to initialize the ElevenLabs client. export async function POST(request: NextRequest) { // Get audio file from form data const formData = await request.formData(); const audioFile = formData.get(‘audio’) as File; // Convert to buffer const arrayBuffer = await audioFile.arrayBuffer(); const audioBuffer = Buffer.from(arrayBuffer); // Initialize ElevenLabs client const elevenlabs = new ElevenLabsClient({ apiKey: apiKey }); // Convert audio to text const result = await elevenlabs.speechToText.convert({ file: audioBlob, modelId: “scribe_v1”, languageCode: “en”, tagAudioEvents: true, diarize: false, }); return NextResponse.json({ transcription: result.text, language_code: result.languageCode, language_probability: result.languageProbability, words: result.words }); } The route will return the following JSON object: { “transcription”: “Transcribed text from the audio”, “language_code”: “en”, “language_probability”: 0.99, “words”: [ { “start”: 0.0, “end”: 1.0, “word”: “Transcribed”, “probability”: 0.99 }, // … more words ] } The transcribed text serves as the input prompt for image generation using Imagen 4 from fal.ai, which creates high-quality images based on the provided text description. Image Generation The Fal API endpoint used is fal-ai/imagen4/preview. You must have a Fal API key to use this endpoint and set the FAL_KEY in the .env file. Please look into this section on how to get the API key. The Fal Imagen 4 image generation API is called directly in the /api/generate-image route. The route will create the image using the subscribe() method from the @fal-ai/client SDK package. export async function POST(request: NextRequest) { const { prompt, style = ‘photorealistic’ } = await request.json(); // Configure fal client fal.config({ credentials: process.env.FAL_KEY || ” }); // Generate image using fal.ai Imagen 4 const result = await fal.subscribe(“fal-ai/imagen4/preview”, { input: { prompt: prompt, // Add style to prompt if needed …(style !== ‘photorealistic’ && { prompt: `${prompt}, ${style} style` }) }, logs: true, onQueueUpdate: (update) => { if (update.status === “IN_PROGRESS”) { update.logs.map((log) => log.message).forEach(console.log); } }, }); // Extract image URLs from the result const images = result.data?.images || []; const imageUrls = images.map((img: any) => img.url || img); return NextResponse.json({ images: imageUrls, prompt: prompt, style: style, requestId: result.requestId }); } The route will return JSON with the following structure: { “images”: [ “https://v3.fal.media/files/panda/YCl2K_C4yG87sDH_riyJl_output.png” ], “prompt”: “Floating red jerry can on the blue sea, wide shot, side view”, “style”: “photorealistic”, “requestId”: “8a0e13db-5760-48d4-9acd-5c793b14e1ee” } The image data, along with the prompt and audio file path, will be saved into the GridDB database. Database Operation We use the GridDB Cloud version for saving the image generation, prompt, and audio file path. It’s easy to use and accessible using API. The container or database name for this project is genvoiceai. Save Data to GridDB We can save any data to the database therefore we need to define the data schema or structure. We will use the following data schema or structure for this project: export interface GridDBData { id: string | number; images: Blob; // Stored as base64 string prompts: string; // Text prompt audioFiles: string; // File path to audio file } In real-world applications, best practice is to separate binary files from their references. However, for simplicity in this example, we store the image directly in the database as a base64-encoded string. Before saving to the database, the image needs to be converted to base64 format: // Convert image blob to base64 string for GridDB storage const imageBuffer = await imageBlob.arrayBuffer(); const imageBase64 = Buffer.from(imageBuffer).toString(‘base64’); Please look into the lib/griddb.ts file for the implementation details. The insertData() function is the actual database insertion. async function insertData({ data, containerName = ‘genvoiceai’ }) { const row = [ parseInt(data.id.toString()), // ID as integer data.images, // Base64 image string data.prompts, // Text prompt data.audioFiles // Audio file path ]; const path = `/containers/${containerName}/rows`; return await makeRequest(path, [row], ‘PUT’); } Get Data from GridDB To get data from the database, you can use the GET request from the route /api/save-data. This route uses SQL query to get specific or all data from the database. // For specific ID query = { type: ‘sql-select’, stmt: `SELECT * FROM genvoiceai WHERE id = ${parseInt(id)}` }; // For recent entries query = { type: ‘sql-select’, stmt: `SELECT * FROM genvoiceai ORDER BY id DESC LIMIT ${parseInt(limit)}` }; For detailed code implementation, please look into the app/api/save-data/route.ts file. Server Routes This project uses Next.js serverless functions to handle API requests. This means there is no separate backend code to handle APIs, as they are integrated directly into the Next.js application. The routes used by the frontend are as follows: Route Method Description /api/generate-image POST Generate images using fal.ai Imagen 4 /api/transcribe POST Convert audio to text using ElevenLabs /api/save-data POST Save image, prompt, and audio data to GridDB /api/save-data GET Retrieve saved data from GridDB /api/audio/[filename] GET Serve audio files from uploads directory User Interface The main entry of the frontend is in the page.tsx file. The user interface is built with Next.js and its single-page applications with several key sections: Voice Recording Section: Large microphone button for audio recording. Transcribed Text Display: Shows the converted speech-to-text with language detection. You can also edit the prompt here before generating the image. Style Selection: A dropdown menu that allows users to choose different image generation styles, including photorealistic, artistic, anime, and abstract styles. Generated Images Grid: Displays created images with download/save options. Saved Data Viewer: Shows previously saved generations from the database. The saved data will be displayed in the Saved Data Viewer section, you can show and hide it by clicking the Show Saved button on the top right. Each saved entry will include the image, the prompt used to generate it, the audio reference, and the request ID. You can also play the audio and download the image. Future Enhancements This project is a basic demo and can be further enhanced with additional features, such as: User authentication and authorization for saved data. Image editing or customization options. Integration with other AI models for image generation. Speech recognition improvements for different languages. Currently, it supports only

This blog post provides a comprehensive, step-by-step guide to Developing a Personal Finance Tracker Web with Spring Boot, Thymeleaf, and GridDB Cloud database. If you’re looking to build a full-stack Java web application from scratch, this tutorial is the perfect place to start. We will walk you through the entire process, from initializing the project to creating a functional application that can track your income and expenses. You will learn how to leverage the power of the Spring Boot framework to rapidly build a robust backend and web application, including setting up Web controllers, RESTful controllers, service layers, and data repositories. For the database, we will use GridDB Cloud, a powerful and scalable time-series NoSQL database, which is an excellent choice for handling financial data. We’ll cover how to seamlessly connect a cloud instance to our Spring application. On the frontend, you’ll see how Thymeleaf, a modern server-side template engine, integrates with Spring Boot to create dynamic and interactive web pages. We will build user-friendly interfaces to display a list of all transactions, show a summary of your financial transaction with charts, and add new entries through a simple form. By the end of this article, you will have not only a practical personal finance application, but also a solid understanding of how these three powerful technologies work together. This hands-on project will equip you with valuable skills applicable to a wide range of web development scenarios. Core Features Before delving into the technical aspects of the web application development, let’s write down the core features. Add a new transaction (date, description, category, amount, type – income/expense). View a list of transactions. Edit existing transactions. Display a summary (total income, total expenses). Categorization of transactions (e.g., Salary, Groceries, Rent, Transport). Basic dashboard view. Prerequisites & Project Setup First, let’s make sure we have everything installed: – Java 17 or later, Maven 3.5+, and your favorite text editor (Intellij IDEA, or VS Code) A GridDB Cloud account. You can sign up for a GridDB Cloud Free instance at https://form.ict-toshiba.jp/download_form_griddb_cloud_freeplan_e. After completing the prerequisites, we’ll start create a new Spring Boot application, which is surprisingly quick thanks to Spring Initializr. You can use this pre-initialized project and click Generate to download a ZIP file. This will give us the basic structure, and then we’ll add the necessary dependencies. Note: To skip starting from scratch, you can clone the source repository here. Configuring the GridDB Cloud Connection To connect to the GridDB Web API via HTTP Requests, we have to get the base URL and include basic authentication in the HTTP Request’s headers. You can set up and get these by following the GridDB Cloud quick start guide. Add those values into src/main/resources/application.properties. Using plain text # GridDB Configuration griddbcloud.base-url=https://cloud5197.griddb.com:443/griddb/v2/gs_cluster griddbcloud.auth-token=TTAxxxxxxx auth-token: Base64 encoded string of the username and password, separated by a colon. Or using environment variables from OS or command-line arguments or IDE configuration. # GridDB Configuration griddbcloud.base-url=${GRIDDBCLOUD_BASE_URL} griddbcloud.auth-token=${GRIDDBCLOUD_AUTH_TOKEN} Building the Backend Logic First, let’s add the required dependencies as follows: <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-thymeleaf</artifactId> </dependency> <dependency> <groupId>nz.net.ultraq.thymeleaf</groupId> <artifactId>thymeleaf-layout-dialect</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>com.github.f4b6a3</groupId> <artifactId>tsid-creator</artifactId> <version>5.2.5</version> </dependency> The spring-boot-starter-thymeleaf will auto-configure everything needed to use the Thymeleaf template engine. The Thymeleaf templates are simple HTML static file (.html extension). The template files by default located at src/main/resources/templates folder. The spring-boot-starter-test provides Spring Test, JUnit, and Mockito dependencies for writing effective tests. tsid-creator: a Java library for generating Time-Sorted Unique Identifiers (TSID). A TSID is a number that is formed by the creation time along with random bits. We’ll use this library for generating IDs. Next, create the package structure based on the technical layers as follows: com.example.springbootpft ├── config ├── controller ├── model ├── service ├── util ├── webapi Before we build the View and Controller layer, we should have the Model layer, that represents our data structure, mirroring our database table. In this post, we will maintain 3 domain: User, Category, and Transaction as follow: //TransactionRecord.java public record TransactionRecord(String id, String categoryId, Double amount, LocalDateTime transactionDate, String transactionType, String description, String userId) {} //CategoryRecord.java public record CategoryRecord(String id, String name) {} //UserRecord.java public record UserRecord(String id, String email, String fullName) {} Next, we bind the config values from application.properties to a Java class by using ConfigurationProperties annotation. @Component @ConfigurationProperties(prefix = “griddbcloud”) public class GridDbCloudClientProperties { private String baseUrl; private String authToken; // Setter and Getter } Tip: You can generate Setters and Getters by using VS Code extension Code Generator For Java Next, we’ll create GridDbCloudClient class, that encapsulates the plumbing required to communicate with GridDB Cloud via HTTP. public class GridDbCloudClient { private final RestClient restClient; public GridDbCloudClient(String baseUrl, String authToken) { this.restClient = RestClient.builder()RestClient.builder() .baseUrl(baseUrl) .defaultHeader(“Authorization”, “Basic ” + authToken) .defaultHeader(“Content-Type”, “application/json”) .defaultHeader(“Accept”, “application/json”) .build(); } } We create an instance of RestClient, a new synchronous HTTP client. We use the builder() method to construct and configure an instance of RestClient. Method baseURL() for setting a default base URL for all requests, and defaultHeader() for specifying the default Authorization header with the encoded Basic authentication string. Other than that, we specify the outgoing payloads and response we expect are JSON. By centralising these values in a single constructor, we avoid duplicate configuration and ensure that every call to GridDB Cloud is issued with the correct parametes. Next, we expose GridDbCloudClient to make it available into services classes and managed by the Spring IoC container. We define gridDbCloudClient method within GridDbCloudClientConfig class annotated with @Configuration, and return the object by using a new keyword and passing GridDbCloudClientProperties as the construction parameters. @Configuration public class GridDbCloudClientConfig { @Bean public GridDbCloudClient gridDbCloudClient(GridDbCloudClientProperties properties) { return new GridDbCloudClient(properties.getBaseUrl(), properties.getAuthToken()); } } Check the connection to the specified database. //GridDbCloudClient.java private void checkConnection() { try { restClient.get().uri(“/checkConnection”).retrieve().toBodilessEntity(); } catch (Exception e) { throw new GridDbException(“Failed to connect to GridDBCloud”, HttpStatusCode.valueOf(500), e.getMessage(), e); } } The database name already specified in base URL. Creating a container (table) A container is a data structure that serves as an interface with the user. The container is a rough equivalent of the table in a relational database. Before registering data, we need to make sure that a container is created beforehand. //GridDbCloudClient.java public void createContainer(GridDbContainerDefinition containerDefinition) { try { restClient.post().uri(“/containers”) .body(containerDefinition) .retrieve().toBodilessEntity(); } catch (Exception e) { if (e.getStatusCode().value() == 409) { return; } throw new GridDbException(“Failed to create container”, HttpStatusCode.valueOf(500), e.getMessage(), e); } } The createContainer method in GridDbCloudClient is responsible for creating a new container (similar to a table) in GridDB Cloud using the REST API. It takes a GridDbContainerDefinition object as input, which defines the schema and properties of the container to be created. The method sends a POST request to the {baseURL}/containers endpoint with this definition as the request body. If the container already exists (indicated by an HTTP 409 Conflict error), the method silently returns without throwing an exception, making the operation idempotent. For other errors, it wraps and re-throws them as a custom GridDbException, ensuring consistent error handling. This approach simplifies container creation logic for the caller and gracefully handles the already exists case. The GridDbContainerDefinition is mapped from the following JSON: { “container_name” : “container1”, “container_type” : “COLLECTION”, “rowkey” : true, “columns” : [ {“name”: “col1 name”, “type”: “TIMESTAMP”, “index”: [“TREE”]}, {“name”: “col2 name”, “type”: “DOUBLE”, “index”: []}, {“name”: “col3 name”, “type”: “STRING”, “index”: []} ] } Now, let’s create a helper class per container that defined each table and inject the GridDbCloudClient. //TransactionContainer.java public class TransactionContainer { private final GridDbCloudClient gridDbCloudClient; private static final String TBL_NAME = “PFTTransaction”; public TransactionContainer(GridDbCloudClient gridDbCloudClient) { this.gridDbCloudClient = gridDbCloudClient; } public void createTable() { List columns = List.of( new GridDbColumn(“id”, “STRING”, Set.of(“TREE”)), new GridDbColumn(“categoryId”, “STRING”), new GridDbColumn(“amount”, “DOUBLE”), new GridDbColumn(“transactionDate”, “TIMESTAMP”), new GridDbColumn(“transactionType”, “STRING”), new GridDbColumn(“description”, “STRING”), new GridDbColumn(“userId”, “STRING”, Set.of(“TREE”))); GridDbContainerDefinition containerDefinition = GridDbContainerDefinition.build(TBL_NAME, columns); this.gridDbCloudClient.createContainer(containerDefinition); } } The createTable() method in the TransactionContainer class defines and creates a new container (table) named PFTTransaction in GridDB. It constructs a list of GridDbColumn objects representing the table schema, including columns like id, categoryId, amount, transactionDate, transactionType, description, and userId, with appropriate data types and indexes. It then builds a GridDbContainerDefinition using these columns and calls gridDbCloudClient.createContainer to create the container in GridDB Cloud. Insert or Update Rows //GridDbCloudClient.java public void registerRows(String containerName, Object body) { try { ResponseEntity result = restClient.put() .uri(“/containers/” + containerName + “/rows”) .body(body) .retrieve().toEntity(String.class); } catch (GridDbException e) { throw e; } catch (Exception e) { throw new GridDbException(“Failed to execute PUT request”, HttpStatusCode.valueOf(500), e.getMessage(), e); } } The registerRows method: insert or update multiple rows of data in a specific GridDB container using the REST API. It takes the container’s name and the data to be registered as parameters. Internally, it sends a PUT request to the {baseURL}/containers/{containerName}/rows endpoint, with the data as the request body. Any failure is wrapped in a single GridDbException, so callers only need to catch on an exception type instead of digging through HTTP or JSON errors. Let’s see how to use this client for inserting transactions. We’ll create a method that bridge domain objects and GridDB’s bulk-insert endpoint. //TransactionContainer.java public void saveRecords(List cRecords) { StringBuilder sb = new StringBuilder(); sb.append(“[“); for (int i = 0; i < cRecords.size(); i++) { TransactionRecord record = cRecords.get(i); sb.append("["); sb.append("\"").append(record.id()).append("\""); sb.append(", "); sb.append("\"").append(record.categoryId()).append("\""); sb.append(", "); sb.append("\"").append(record.amount()).append("\""); sb.append(", "); sb.append("\"").append(DateTimeUtil.formatToZoneDateTimeString(record.transactionDate())).append("\""); sb.append(", "); sb.append("\"").append(record.transactionType()).append("\""); sb.append(", "); if (record.description() == null) { sb.append("null"); } else { sb.append("\"").append(record.description()).append("\""); } sb.append(", "); sb.append("\"").append(record.userId()).append("\""); sb.append("]"); if (i < cRecords.size() - 1) { sb.append(", "); } } sb.append("]"); String result = sb.toString(); this.gridDbCloudClient.registerRows(TBL_NAME, result); } The method takes a list of TransactionRecord objects and turns them into a single JSON-style string that GridDB Cloud can ingest in one shot. It loops over every record, formats each field in the exact order the container expects, and wraps the whole thing in square brackets to create an array of arrays. After the string built, it hands the paylod to gridDbCloudClient.registerRows, which fires an HTTP Put for batch insertion. Retrieves Rows This method fetches rows from GridDB container through a REST endpoint. public AcquireRowsResponse acquireRows(String containerName, AcquireRowsRequest requestBody) { try { ResponseEntity responseEntity = restClient.post().uri("/containers/" + containerName + "/rows") .body(requestBody) .retrieve().toEntity(AcquireRowsResponse.class); return responseEntity.getBody(); } catch (GridDbException e) { throw e; } catch (Exception e) { throw new GridDbException("Failed to execute GET request", HttpStatusCode.valueOf(500), e.getMessage(), e); } } We specify the container name and an AcquireRowsRequest object and POST that request to {baseURL}/containers/{containerName}/rows endpoint. An AcquireRowsRequest defines offset, limit, sort, and condition (Conditional expression). Mapped from the following JSON: { "offset" : 10, "limit" : 100, "condition" : "id >= 50″, “sort” : “id desc” } GridDB return the matching rows packaged in an AcquireRowsResponse, a POJO that contains the resulting columns, rows and metadata. Example of the response body: { “columns” : [ {“name”: “col1 name”, “type”: “TIMESTAMP” }, {“name”: “col2 name”, “type”: “DOUBLE” }, {“name”: “col3 name”, “type”: “STRING” } ], “rows” : [ [“2016-01-16T10:25:00.253Z”, 100.5, “normal” ], [“2016-01-16T10:35:00.691Z”, 173.9, “normal” ], [“2016-01-16T10:45:00.032Z”, 173.9, null ] ], “total” : 100000, “offset” : 0, “limit” : 3 } Now let’s add a method for fetching all transactions. public List getAll() { AcquireRowsRequest requestBody = AcquireRowsRequest.builder().limit(50L).sort(“transactionDate DESC”).build(); AcquireRowsResponse response = this.gridDbCloudClient.acquireRows(TBL_NAME, requestBody); if (response == null || response.getRows() == null) { return List.of(); } List transactions = convertResponseToRecord(response); return transactions; } The getAll() method retrieves up to 50 transaction records from the PFTTransaction table in GridDB, sorted by transactionDate in descending order. It builds an AcquireRowsRequest with these parameters and calls gridDbCloudClient.acquireRows() to fetch the data. It converts the raw response rows into a list of TransactionRecord objects using the convertResponseToRecord() helper method, and returns the list. This method provides a simple way to get the most recent transactions from the database. Service Layer With the Model layer now fully defined, we can move up the stack. Enter the Service layer: the single place where business rules live and abstract the data access. //TransactionService.java @Service public class TransactionService { private final TransactionContainer transactionContainer; public TransactionService(TransactionContainer transactionContainer) { this.transactionContainer = transactionContainer; } public static String nextId() { return “trx_” + TsidCreator.getTsid().format(“%S”); } public List findAll() { final List transactions = transactionContainer.getAll(); return transactions.stream() .map(transaction -> mapToDTO(transaction, new TransactionDTO())) .collect(Collectors.toList()); } public String create(final TransactionDTO transactionDTO) { final String id = (transactionDTO.getId() != null) ? transactionDTO.getId() : nextId(); TransactionRecord newTransaction = new TransactionRecord( id, transactionDTO.getCategoryId(), transactionDTO.getAmount(), transactionDTO.getTransactionDate(), transactionDTO.getTransactionType(), transactionDTO.getDescription(), transactionDTO.getUserId()); transactionContainer.saveRecords(List.of(newTransaction)); return newTransaction.id(); } } The TransactionService depends on transactionContainer. On create(), we convert the TransactionDTO that we get from the controller into the Data model. On findAll(), after receiving the TransactionRecord we convert it into the DTO The Controller Layer The controller acts as an intermediary between Model and View. Controllers handle user requests, orchestrate the flow of data, and execute business logic. Let’s create the controller to handle web request to transaction data. @Controller @RequestMapping(“/transactions”) public class TransactionController { private final TransactionService transactionService; private final CategoryService categoryService; private final UserService userService; private final Map transactionTypes = Arrays.stream(TransactionType.values()) .collect(Collectors.toMap(Enum::name, TransactionType::getLabel)); public TransactionController( final TransactionService transactionService, final CategoryService categoryService, UserService userService) { this.transactionService = transactionService; this.categoryService = categoryService; this.userService = userService; } } The constructor uses dependency injection to receive instances of TransactionService, CategoryService, and UserService, which are assigned to final fields for use throughout the controller. This ensures the controller has access to all necessary services and supporting data for handling transaction-related web requests. @RequestMapping(“/transactions”) is a Spring MVC annotation that maps all HTTP requests with the /transactions URL path to the TransactionController class. This means any request whose path starts with /transactions will be handled by methods within this controller. By centralizing the base path at the class level, it keeps route definitions organized and avoids repetition, allowing individual handler methods to specify only the remaining part of the URL. Handle list page: @GetMapping public String list(final Model model) { List categories = categoryService.findAll().stream() .map(category -> new CategoryResponse(category.getId(), category.getName())).collect(Collectors.toList()); List users = userService.findAll().stream() .map(user -> new UserResponse(user.getId(), user.getFullName())).collect(Collectors.toList()); List transactions = transactionService.findAll().stream().map(transaction -> new TransactionResponse( transaction.getId(), findCategory(categories, transaction.getCategoryId()), transaction.getAmount(), transaction.getTransactionDate(), transaction.getTransactionType(), transaction.getDescription(), findUser(users, transaction.getUserId()))) .collect(Collectors.toList()); model.addAttribute(“transactions”, transactions); return “transaction/list”; } The list() method is what runs when someone hits /transactions in the browser. It grabs every category and user, turns them into tiny lookup objects, then pulls all the transactions. For each transactions, it builds a friendly response with the real category and usernames instead of just IDs. It drops that list into the model as transactions and tells Spring to show the transaction/list page. The page now has everything it needs to display the list. @GetMapping(“/add”) public String add(@ModelAttribute(“transaction”) final TransactionDTO transactionDTO) { return “transaction/add”; } @PostMapping(“/add”) public String add( @ModelAttribute(“transaction”) @Valid final TransactionDTO transactionDTO, final BindingResult bindingResult, final RedirectAttributes redirectAttributes) { if (bindingResult.hasErrors()) { return “transaction/add”; } transactionDTO.setUserId(userService.findAll().get(0).getId()); // Set default userId transactionService.createAll(List.of(transactionDTO)); redirectAttributes.addFlashAttribute(WebUtils.MSG_SUCCESS, WebUtils.getMessage(“transaction.create.success”)); return “redirect:/transactions”; } Both methods handle adding new transactions. When you click Add Transaction, the first method just shows the form. When you hit Save, the second method checks your input, fills in the default user ID, saves the new transaction, and then redirects you to the list page with a success message. Dashboard The DashboardController.java handles request to /dashboard, passes the transaction summary list to the view. @Controller @RequestMapping(“/dashboard”) public class DashboardController { private final TransactionService transactionService; private final UserService userService; public DashboardController( final TransactionService transactionService, UserService userService) { this.transactionService = transactionService; this.userService = userService; } @RequestMapping public String dashboard(final Model model) { UserDTO currentUser = userService.getByEmail(TableSeeder.DUMMY_USER1_EMAIL); List transactionSum = transactionService.getTransactionSummary(currentUser.getId()); model.addAttribute(“transactionSummary”, transactionSum); double totalIncome = transactionSum.stream().mapToDouble(TransactionSummary::incomeAmount).sum(); double totalExpenses = transactionSum.stream().mapToDouble(TransactionSummary::expenseAmount).sum(); double difference = totalIncome – totalExpenses; model.addAttribute(“totalIncome”, totalIncome); model.addAttribute(“totalExpenses”, totalExpenses); model.addAttribute(“totalDifference”, difference); return “dashboard/index.html”; } } Creating the Frontend with Thymeleaf After completing the controller, let’s continue to create the View using Thymeleaf, a modern server-side Java template engine for web. Transactions Page We start by creating the transaction directory under src/main/resources/templates. Create list.html for displaying all transactions. <!DOCTYPE HTML> <html xmlns:th=”http://www.thymeleaf.org” xmlns:layout=”http://www.ultraq.net.nz/thymeleaf/layout” layout:decorate=”~{layout}”> <head> <title>[[#{transaction.list.headline}]]</title> </head> <body> <div layout:fragment=”content”> <div class=”d-flex flex-wrap mb-4 align-items-center justify-content-between”> <div class=”card w-100″> <div class=”card-header d-flex align-items-center justify-content-between”> <h1 class=”fw-medium display-6 mb-0″>[[#{transaction.list.headline}]]</h1> <a th:href=”@{/transactions/add}” class=”btn btn-primary”>[[#{transaction.list.createNew}]]</a> <div th:if=”${transactions.empty}”> <form th:action=”@{/transactions/generate}” data-confirm-message=”Generate ?” method=”post” class=”js-submit-confirm d-inline”> <button type=”submit” class=”btn btn-secondary btn-sm” style=”color: #b8d935; background-color: #f4fbdb;”>Generate Demo data</button> </form> </div> </div> <div class=”card-body”> <div th:if=”${transactions.empty}” class=”text-muted”>[[#{transaction.list.empty}]]</div> <div th:if=”${!transactions.empty}” class=”table-responsive”> <table class=”table align-middle”> <thead> <tr> <th scope=”col” class=”text-start”>[[#{transaction.id.label}]]</th> <th scope=”col” class=”text-start”>Category</th> <th scope=”col” class=”text-start”>[[#{transaction.amount.label}]]</th> <th scope=”col” class=”text-start”>[[#{transaction.transactionDate.label}]]</th> <th scope=”col” class=”text-start”>[[#{transaction.transactionType.label}]]</th> <th scope=”col” class=”text-start”>User</th> <th></th> </tr> </thead> <tbody> <tr th:each=”transaction : ${transactions}”> <td>[[${transaction.id}]]</td> <td>[[${transaction.category.name}]]</td> <td>[[${transaction.amount}]]</td> <td>[[${transaction.transactionDate}]]</td> <td>[[${transaction.transactionType}]]</td> <td>[[${transaction.user.fullName}]]</td> <td> <div class=”d-flex justify-content-end gap-2″> <a th:href=”@{/transactions/edit/{id}(id=${transaction.id})}” class=”btn btn-secondary btn-sm”>[[#{transaction.list.edit}]]</a> <form th:action=”@{/transactions/delete/{id}(id=${transaction.id})}” th:data-confirm-message=”#{delete.confirm}” method=”post” class=”js-submit-confirm d-inline”> <button type=”submit” class=”btn btn-secondary btn-sm”>[[#{transaction.list.delete}]]</button> </form> </div> </td> </tr> </tbody> </table> </div> </div> </div> </div> </div> </body> </html> Using the th:each attribute to iterate over the transactions parameter that was set in the controller. With Thymeleaf Layout Dialect, we have a reusable template to improve code reuse. Having a common layout.html file, all pages will have the same assets in <head> and use the same sidebar layout. Here is a preview of the transaction list page. < p> < p> Now, create add.html for adding a new transaction. <!DOCTYPE HTML> <html xmlns:th=”http://www.thymeleaf.org” xmlns:layout=”http://www.ultraq.net.nz/thymeleaf/layout” layout:decorate=”~{layout}”> <head> <title>[[#{transaction.add.headline}]]</title> </head> <body> <div layout:fragment=”content”> <div class=”d-flex flex-wrap mb-4 align-items-center justify-content-between”> <h1 class=”fw-medium display-6 mb-0″>[[#{transaction.add.headline}]]</h1> <div> <a th:href=”@{/transactions}” class=”btn btn-secondary”>[[#{transaction.add.back}]]</a> </div> </div> <div th:replace=”~{fragments/forms::globalErrors(‘transaction’)}” /> <form th:action=”${requestUri}” method=”post” novalidate=””> <div th:replace=”~{fragments/forms::inputRow(object=’transaction’, field=’categoryId’, required=true, type=’select’)}” /> <div th:replace=”~{fragments/forms::inputRow(object=’transaction’, field=’amount’, required=true)}” /> <div th:replace=”~{fragments/forms::inputRow(object=’transaction’, field=’transactionDate’, inputClass=’js-datetimepicker’, autocomplete=’off’)}” /> <div th:replace=”~{fragments/forms::inputRow(object=’transaction’, field=’transactionType’, required=true, type=’select’)}” /> <div th:replace=”~{fragments/forms::inputRow(object=’transaction’, field=’description’, type=’textarea’)}” /> <input type=”submit” th:value=”#{transaction.add.headline}” class=”btn btn-primary mt-4″ /> </form> <script type=”text/javascript”> document.addEventListener(‘DOMContentLoaded’, function () { const transactionDateInput = document.getElementById(‘transactionDate’); transactionDateInput.setAttribute(‘autocomplete’, ‘off’); }); </script> </div> </body> </html> The ~{fragments/forms::inputRow(object=’transaction’, field=’amount’, required=true)} will create a Thymeleaf expression th:object=”${transaction}” and th:field=”*{__${field}__}” for collecting the form data of transaction object. Income and Expenses Chart Create a dashboard directory under src/main/resources/templates. For serving URL /dashboard we create index.html. <body> <div layout:fragment=”content”> <div class=”d-flex flex-wrap mb-4 align-items-center justify-content-between”> <div class=”card w-100″> <div class=”card-header d-flex align-items-center justify-content-between”> <h1 class=”fw-medium display-6 mb-0″>Income and Expenses</h1> </div> <div class=”card-body”> <div class=”chart-container” style=”position: relative; height:50vh; width:50vw”> <canvas id=”barchart”></canvas> </div> </div> </div> <div class=”row g-3 mt-4″> <div class=”col”> <div class=”card”> <div class=”card-header”> <h2 class=”fw-medium display-6 mb-0 text-nowrap”>Total Income</h2> </div> <div class=”card-body fw-bold”> <span class=”fs-4 fw-semibold”>$</span><span th:text=”${totalIncome}” class=”fs-2 fw-bold”></span> </div> </div> </div> <div class=”col”> <div class=”card”> <div class=”card-header”> <h2 class=”fw-medium display-6 mb-0 text-nowrap”>Total Expenses</h2> </div> <div class=”card-body fw-bold”> <span class=”fs-4 fw-semibold”>$</span><span th:text=”${totalExpenses}” class=”fs-2 fw-bold”></span> </div> </div> </div> <div class=”col”> <div class=”card”> <div class=”card-header”> <h2 class=”fw-medium display-6 mb-0 text-nowrap”>Difference</h2> </div> <div class=”card-body fw-bold”> <span class=”fs-4 fw-semibold”>$</span><span th:text=”${totalDifference}” class=”fs-2 fw-bold”></span> </div> </div> </div> </div> </div> <!– https://stackoverflow.com/questions/27176640/how-to-put-code-thymeleaf-in-an-external-javascript-file –> <script th:replace=”dashboard/barchartjs :: js_chart”></script> </div> </body> This page shows the total of expenses and incomes. This page had a canvas element to show the chart. The chart created using Chart.js and externalized in dashboard/barchartjs.html. We use a bar chart to show transaction values as vertical bars. Here is the JavaScript code to initialize the bar chart: Chart.register(ChartDataLabels); const data = /*[[${transactionSummary}]]*/[ { timeLabel: “2025-07-01”, expenseAmount: 10 }, { timeLabel: “2025-07-02″, expenseAmount: 15 } ]; const ctx2 = document.getElementById(‘barchart’); new Chart(ctx2, { type: ‘bar’, data: { labels: data.map(row => row.timeLabel), datasets: [{ label: ‘Expense’, data: data.map(row => row.expenseAmount), borderWidth: 2, borderColor: ‘#ff6384’, backgroundColor: ‘#ff6384’, borderRadius: 10, }, { label: ‘Income’, data: data.map(row => row.incomeAmount), // Assuming incomeAmount is available borderWidth: 2, borderColor: ‘#4fc9da’, backgroundColor: ‘#4fc9da’, borderRadius: 10, }] }, options: { scales: { y: { stacked: true, beginAtZero: false, ticks: { // Include a dollar sign in the ticks callback: function (value, index, ticks) { return ‘$’ + value; } } }, x: { stacked: false, title: { color: ‘#b8d935’, display: true, text: ‘Date’ } } }, plugins: { legend: { display: true, position: ‘top’, labels: { color: ‘#4fc9da’, font: { size: 20 } } }, datalabels: { color: ‘black’, font: { weight: ‘bold’ }, align: ‘end’, anchor: ‘end’, formatter: function (value, index, context) { if (value > 0) { return value; } else { return ”; } } } } } }); Register chartjs-plugin-datalabels to customize the appearance and position of each label. Using JavaScript inlining in Thymeleaf, enabling the dynamic injection of server-side data into client-side scripts. The ${transactionSummary} expression will evaluate to an Array, and Thymeleaf will convert it to JavaScript syntax. The real value will be provided by the DashboardController. Conclusion In this tutorial, you’ve gained hands-on experience in full-stack web development using the powerful Java Spring Boot ecosystem. You learned how to interact with a cloud database API by creating helper classes (SDK). While this application provides a solid foundation, there are numerous features we can consider adding: Secure session management to protect users financial data. Auto update the chart from the backend without manually refreshing the page. Filtering transactions by date range, and category. Intelligent transaction

Extracting meaningful insights from a large corpus of documents is a challenging task. With advancements in Large Language Models (LLMS), it is now possible to automate the process of structured data extraction from text documents. In this article, you will learn how to extract structured data from PDF documents using LLMs in LangChain and store it in GridDB. GridDB is a high-performance NoSQL database suited for managing complex and dynamic datasets. Its high-throughput NOSQL capabilities make it ideal for storing large structured datasets containing text insights. We will begin by downloading a PDF document dataset from Kaggle and extracting structured information from the documents using LangChain. We will then store the structured data in a GridB container. Finally, we will retrieve the data from the GridDB container and analyze the structured metadata for the PDF documents. Note: See the GridDB Blogs GitHub repository for codes. Prerequisites You need to install the following libraries to run the codes in this article. GridDB C Client GridDB Python client You can install these libraries following the instructions on the GridDB Python Package Index (Pypi) In addition, you need to install the langchain, openai, pydantic, pandas, pypdf, openai, tiktoken, and tqdm libraries to run codes in this article. The following script installs these libraries. !pip install –upgrade -q langchain !pip install –upgrade -q pydantic !pip install –upgrade -q langchain-community !pip install –upgrade -q langchain-core !pip install –upgrade -q langchain-openai !pip install –upgrade -q pydantic pandas pypdf openai tiktoken tqdm Finally, run the script below to import the required libraries and modules into your Python application. from pathlib import Path import re import pandas as pd from tqdm import tqdm from itertools import islice from typing import Literal, Optional import matplotlib.pyplot as plt from langchain_core.pydantic_v1 import BaseModel, Field, validator from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder from langchain_openai import ChatOpenAI from langchain.agents import create_openai_functions_agent, AgentExecutor from langchain_experimental.tools import PythonREPLTool from langchain_community.document_loaders import PyPDFDirectoryLoader import griddb_python as griddb Extracting Structured Data from PDF Documents We will extract structured information from the PDF files dataset from Kaggle. Download the dataset into your local directory and run the following script. The dataset contains over a thousand files; however, for the sake of testing, we will extract structured information from 100 documents. The following script extracts data from the first 100 documents and stores it in a Python list. # https://www.kaggle.com/datasets/manisha717/dataset-of-pdf-files pdf_dir = Path(“/home/mani/Datasets/Pdf”) loader = PyPDFDirectoryLoader( pdf_dir, recursive=True, silent_errors=True )# raises warning if a PDF document doesnt contain valid text # first 100 that load cleanly docs_iter = loader.lazy_load() # generator → 1 Document per good PDF docs = list(islice(docs_iter, 100)) docs[0] Output: The above output shows the contents of the first document. In this article, we will use a large language model (LLM) with a structured response in LangChain to extract the title, summary, document type, topic category, and sentiment from a PDF document. To retrieve structured data, we have to define the scheme of the data we want to retrieve. For example, we will predefine some categories for document type, topic category, and sentiment, as shown in the following script. DOC_TYPES = ( “report”, “article”, “manual”, “white_paper”, “thesis”, “presentation”, “policy_brief”, “email”, “letter”, “other”, ) TOPIC_CATS = ( “science”, “technology”, “history”, “business”, “literature”, “health”, “education”, “art”, “politics”, “other”, ) Sentiment = Literal[“positive”, “neutral”, “negative”] Next, we will define a Pydantic BaseModel class object, which contains fields for the structured information we want to extract from the PDF documents. The descriptions of the fields tell LLMs what information to store in them. class PDFRecord(BaseModel): “””Validated metadata for a single PDF.””” title: str = Field( …, description=”Document title. If the text contains no clear title, ” “generate a concise 6–12-word title that reflects the content.” ) summary: str = Field( …, description=”Two- to three-sentence synopsis of the document.” ) doc_type: Literal[DOC_TYPES] = Field( default=”other”, description=”Document genre; choose one from: ” + “, “.join(DOC_TYPES) ) topic_category: Literal[TOPIC_CATS] = Field( default=”other”, description=”Primary subject domain; choose one from: ” + “, “.join(TOPIC_CATS) ) sentiment: Sentiment = Field( default=”neutral”, description=”Overall tone of the document: positive, neutral, or negative.” ) # — fallback helpers so bad labels never crash validation — @validator(“doc_type”, pre=True, always=True) def _doc_fallback(cls, v): return v if v in DOC_TYPES else “other” @validator(“topic_category”, pre=True, always=True) def _topic_fallback(cls, v): return v if v in TOPIC_CATS else “other” The next step is to define a prompt for an LLM that guides the LLM in extracting structured data from PDF documents and converting it to JSON format. The BaseModel class we defined before can extract JSON data from a structured LLM response. Notice that the prompt contains the pdf_text placeholder. This placeholder will store the text of the PDF document. prompt = ChatPromptTemplate.from_messages([ (“system”, “You are a meticulous analyst. ” “Extract only what is explicitly present in the text, ” “but you MAY generate a succinct title if none exists.”), (“human”, f””” **Task** Fill the JSON schema fields shown below. **Fields** • title – exact title if present; otherwise invent a 6-12-word title • summary – 2–3 sentence synopsis • doc_type – one of: {“, “.join(DOC_TYPES)} • topic_category – one of: {“, “.join(TOPIC_CATS)} • sentiment – positive, neutral, or negative overall tone **Rules** – If a category is uncertain, use “other”. – Respond ONLY in the JSON format supplied automatically. **Document begins** {{pdf_text}} “””) ]) The next step is to define an LLM. We will use the OpenAI gpt-4o-mini model and create the ChatOpenAI object that supports chat-like interaction with the LLM. You can use any other supported by the LangChain framework. To extract structured data, we call the with_structured_output() function using the ChatOpenAI object and pass it the PDFRecord base model class we defined earlier. Finally, we combine the prompt and LLM to create a LangChain runnable object. llm = ChatOpenAI(model_name=”gpt-4o-mini”, openai_api_key = “YOUR_OPENAI_API_KEY”, temperature=0) structured_llm = llm.with_structured_output(PDFRecord) chain = prompt | structured_llm We will extract the text of each document from the list of PDF documents and invoke the chain we defined. Notice that we are passing the PDF text (doc.page_content) as a value for the pdf_text key since the prompt contains a placeholder with the same name. The response from the LLM chain is appended to the rows list. rows = [] for doc in tqdm(docs, desc=”Processing PDFs”): record = chain.invoke({“pdf_text”: doc.page_content}) # → PDFRecord row = record.dict() # plain dict row[“path”] = doc.metadata[“source”] rows.append(row) The rows list now contains Python dictionaries containing structured information extracted from the PDF documents. We convert this list into a Pandas DataFrame and store it as a CSV file for later use. dataset = pd.DataFrame(rows) dataset.to_csv(“pdf_catalog.csv”, index=False) print(“✓ Saved pdf_catalog.csv with”, len(rows), “rows”) dataset.head(10) Output: The above output shows the data extracted from PDF documents. Each row corresponds to a single PDF document. Next, we will insert this data in GridDB. Inserting Structured Data from PDF into GridDB Inserting data into GridDB is a three-step process. You establish a connection with a GridDB host, create a container, and insert data into it. Creating a Connection with GridDB To create a GridDB connection, call the griddb.StoreFactory.get_instance() function to get a factory object. Next, call the get_store() function on the factory object and pass it the database host, cluster name, and user name and password. The following script creates a connection to the locally hosted GridDB server and tests the connection by retrieving a dummy container. factory = griddb.StoreFactory.get_instance() DB_HOST = “127.0.0.1:10001” DB_CLUSTER = “myCluster” DB_USER = “admin” DB_PASS = “admin” try: gridstore = factory.get_store( notification_member = DB_HOST, cluster_name = DB_CLUSTER, username = DB_USER, password = DB_PASS ) container1 = gridstore.get_container(“container1”) if container1 == None: print(“Container does not exist”) print(“Successfully connected to GridDB”) except griddb.GSException as e: for i in range(e.get_error_stack_size()): print(“[“, i, “]”) print(e.get_error_code(i)) print(e.get_location(i)) print(e.get_message(i)) Output: Container does not exist Successfully connected to GridDB If you see the above output, you successfully established a connection with the GridDB server. Inserting Data into GridDB Next, we will insert the data from our Pandas DataFrame into the GridDB container. To do so, we define the map_pandas_dtype_to_griddb() function, which maps the Pandas column types to GridDB data types. We iterate through all the column names and types and create a list of lists, each nested list containing a column name and GridDB data type for the column. Next, we create a ContainerInfo object using the container name, the container columns, and the types lists. Since we are storing tabular data, we set the container type to COLLECTION. Next, we store the container in GridDB using the gridstore.put_container() function. Finally, we iterate through all the rows in our pdf document dataset and store it in the container we created using the put() function. # see all GridDB data types: https://docs.griddb.net/architecture/data-model/#data-type def map_pandas_dtype_to_griddb(dtype): if dtype == ‘int64’: return griddb.Type.LONG elif dtype == ‘float64’: return griddb.Type.FLOAT elif dtype == ‘object’: return griddb.Type.STRING # Add more column types if you want else: raise ValueError(f’Unsupported pandas type: {dtype}’) container_columns = [] for column_name, dtype in dataset.dtypes.items(): griddb_dtype = map_pandas_dtype_to_griddb(str(dtype)) container_columns.append([column_name, griddb_dtype]) container_name = “PDFData” container_info = griddb.ContainerInfo(container_name, container_columns, griddb.ContainerType.COLLECTION, True) try: cont = gridstore.put_container(container_info) for index, row in dataset.iterrows(): cont.put(row.tolist()) print(“All rows have been successfully stored in the GridDB container.”) except griddb.GSException as e: for i in range(e.get_error_stack_size()): print(“[“, i, “]”) print(e.get_error_code(i)) print(e.get_location(i)) print(e.get_message(i)) Finally, we will retrieve data from GridDB and analyze the dataset. Retrieving Data from GridDB and Performing Analysis To retrieve data from a GridDB container, you must first retrieve the container using the get_container() function and then execute an SQL query on the container object using the query() function, as shown in the script below. To execute the select query, you need to call the fetch() function, and to retrieve data as a Pandas dataframe, call the fetch_rows() function. The following script retrieves structured data from our GridDB container and stores it in the pdf_dataset dataframe. def retrieve_data_from_griddb(container_name): try: data_container = gridstore.get_container(container_name) # Query all data from the container query = data_container.query(“select *”) rs = query.fetch() data = rs.fetch_rows() return data except griddb.GSException as e: print(f”Error retrieving data from GridDB: {e.get_message()}”) return None pdf_dataset = retrieve_data_from_griddb(container_name) pdf_dataset.head() Output: The above output shows the data retrieved from our GridDB container. Once we store data from a GridDB container in a Pandas DataFrame, we can perform various analyses on it. Using a Pie chart, Let’s see the topic category distribution in all PDF documents. pdf_dataset[“topic_category”].value_counts().plot.pie(autopct=”%1.1f%%”) plt.title(“Distribution of topic categories”) plt.ylabel(“”) plt.show() Output: The output shows that the majority of documents are related to science, followed by business. Next, we can plot the distribution of document types using a donut chart. df[“doc_type”].value_counts().plot.pie( autopct=”%1.1f%%”, wedgeprops=dict(width=0.50) # makes the “donut” hole ) plt.title(“Document type”) plt.ylabel(“”) plt.gca().set_aspect(“equal”) plt.show() Output: The output shows that the majority of documents are reports. Finally, we can plot the sentiments expressed in documents as a bar plot. pdf_dataset[“sentiment”].value_counts().plot.bar() plt.title(“Distribution of sentiment values”) plt.xlabel(“sentiment”) plt.ylabel(“count”) plt.tight_layout() plt.show() Output: The above output shows that most of the documents have neutral sentiments. Conclusion This article explained how to build a complete pipeline for extracting metadata from unstructured PDF documents using LLMs and storing the result in GridDB. You explored using LangChain with OpenAI’s GPT-4 model to extract key information such as document title, summary, type, category, and sentiment and how to save this structured output into a GridDB container. The combination of LLM-driven data extraction and GridDB’s performance-oriented architecture makes this approach suitable for intelligent document processing in real-time applications. If you have questions or need assistance with GridDB please ask on Stack Overflow using the griddb tag. Our team is always happy to help. For the complete code, visit my GridDB Blogs GitHub

In a previous article, we talked about some new features in the GridDB Cloud CLI Tool which showcased the ability to make tables based on schemas in the form of JSON files. Included in that was also the ability to migrate from GridDB Community Edition (on prem) to the Cloud in just a few commands; you can read that article here: New Features of the GridDB Cloud CLI Tool. In this article, we will again feature the GridDB Cloud CLI Tool, but this time we want to showcase the ability to migrate from your on-prem PostgreSQL database directly to your GridDB Cloud instance. We will get into the details later in the article, but here is a basic rundown of the steps required to make this work: Export PostgreSQL tables as CSV row data Export schemas of all PostgreSQL tables as JSON file Use GridDB Cloud CLI Tool to transform the data, create the corresponding tables, and then load the data into GridDB Cloud Also please note: this tool is intended for developers and is not of production level quality — it is not made by the official GridDB Development team, so please if you face any issues, leave a comment in this article or within the GridDB Cloud CLI Tool’s repo and we will look into it. Downloading the Tool and Examples The source code for this tool and the binaries to get the latest version are found on GitHub: https://github.com/Imisrael/griddb-cloud-cli. The example directories with CSV data that you can test with are found in the migrate-psql branch: https://github.com/Imisrael/griddb-cloud-cli/tree/migrate-psql/migration_dirs. Scripts to Export PostgreSQL Before we showcase running an example of this process, let’s first take a look at the scripts needed to export the data from your PostgreSQL instance into a more neutral and workable data format (CSV). We will use two different scripts in this phase, one for extract row data, and the other to grab ‘meta’/schema data; both of these scripts rely on the psql command to directly communicate with the database to grab the data we need. Script for Row Data Here is the script to extract your row data; it will generate a unique CSV file for each table in your database, with the filename corresponding directly to the name of the table — handy! #!/bin/bash # Check if a database name was provided. if [ -z “$1” ]; then echo “Usage: $0 ” exit 1 fi DB_NAME=$1 EXPORT_DIR=”.” # Export to the current directory. # Get a list of all tables in the ‘public’ schema. TABLES=$(psql -d $DB_NAME -t -c “SELECT table_name FROM information_schema.tables WHERE table_schema = ‘public’ AND table_type = ‘BASE TABLE’;”) # Loop through the tables and export each one to a CSV file. for TBL in $TABLES; do echo “Exporting table: $TBL” psql -d $DB_NAME -c “\copy (SELECT * FROM $TBL) TO ‘$EXPORT_DIR/$TBL.csv’ WITH (FORMAT CSV, HEADER);” done echo “Export complete.” The script itself is self-explanatory: it exports all tables into the current working directory with the header as the top row and every subsequent row being the data from the table. Script for Schema Data This script is a bit more clever and more of a unique feature for PostgreSQL. It utilizes its json_agg() function which “is an aggregate function that collects values from multiple rows and returns them as a single JSON array.” source. Here is what the script looks like: #!/bin/bash psql -d postgres -X -A -t -c ” WITH column_details AS ( — First, get all the column info for each table SELECT table_name, json_agg( json_build_object( ‘column_name’, column_name, ‘data_type’, udt_name, ‘is_nullable’, is_nullable, ‘column_default’, column_default ) ORDER BY ordinal_position ) AS columns FROM information_schema.columns WHERE table_schema = ‘public’ GROUP BY table_name ), primary_key_details AS ( — Next, find the primary key columns for each table SELECT kcu.table_name, json_agg(kcu.column_name) AS pk_columns FROM information_schema.table_constraints AS tc JOIN information_schema.key_column_usage AS kcu ON tc.constraint_name = kcu.constraint_name AND tc.table_schema = kcu.table_schema WHERE tc.constraint_type = ‘PRIMARY KEY’ AND tc.table_schema = ‘public’ GROUP BY kcu.table_name ) — Finally, join them together SELECT json_object_agg( cd.table_name, json_build_object( ‘primary_key’, COALESCE(pkd.pk_columns, ‘[]’::json), ‘columns’, cd.columns ) ) FROM column_details cd LEFT JOIN primary_key_details pkd ON cd.table_name = pkd.table_name; ” > schema.json If you were just grabbing the schema, I think this script would be about half of this length, but we actually are grabbing the primary key as well which requires a JOIN operation. We grab this data in order to ensure we migrate time series tables into TIME_SERIES containers on the GridDB side (more on that later). Changes to the GridDB Cloud CLI Tool To get this process to work, we needed to first separate out the migrate tool. Prior to this release, the migrate tool only expected to work with GridDB CE, and so, the command worked like this: griddb-cloud-cli migrate [griddb-out-directory]. Now obviously this no longer works as we need to indicate to the tool what sort of database we are importing to GridDB Cloud. So what we did was separate out the commands as sub commands, and this meant keeping the more generic functions that work for both in the root of the command — things like reading a CSV file, or parsing a JSON file — and keeping the DB-specific functions inside of the respective subcommand files. So now our migrate/ directory has three files instead of one: migrateCmd.go, migrateGridDB.go, and migratePSQL.go Here’s a brief example of a function which is unique to the PostgreSQL import process func typeSwitcher(s string) string { switch s { case “bool”: return “BOOL” case “char”, “varchar”, “text”: return “STRING” case “int”, “int2”, “int4”: return “INTEGER” case “int8”: return “LONG” case “decimal”, “real”, “numeric”: return “FLOAT” case “float”, “float8”: return “DOUBLE” case “timetz”, “timestamptz”: return “TIMESTAMP” default: return strings.ToUpper(s) } } Migrating from PostgreSQL Now that we have our data ready for use, let’s run the migration tool and see the results. First, place the contents of your exporting efforts into their own directory $ mkdir psql_exported $ mv *.csv psql_exported/ $ mv schema.json psql_exprted/ And now we can run the tool: $ griddb-cloud-cli migrate psql psql_exported/ {“container_name”:”customers”,”container_type”:”COLLECTION”,”rowkey”:false,”columns”:[{“name”:”customer_id”,”type”:”INTEGER”,”index”:null},{“name”:”first_name”,”type”:”STRING”,”index”:null},{“name”:”last_name”,”type”:”STRING”,”index”:null},{“name”:”email”,”type”:”STRING”,”index”:null},{“name”:”phone_number”,”type”:”STRING”,”index”:null},{“name”:”address”,”type”:”STRING”,”index”:null},{“name”:”created_at”,”type”:”TIMESTAMP”,”index”:null}]} ✔ Make Container? { “container_name”: “customers”, “container_type”: “COLLECTION”, “rowkey”: false, “columns”: [ { “name”: “customer_id”, “type”: “INTEGER”, “index”: null }, { “name”: “first_name”, “type”: “STRING”, “index”: null }, { “name”: “last_name”, “type”: “STRING”, “index”: null }, { “name”: “email”, “type”: “STRING”, “index”: null }, { “name”: “phone_number”, “type”: “STRING”, “index”: null }, { “name”: “address”, “type”: “STRING”, “index”: null }, { “name”: “created_at”, “type”: “TIMESTAMP”, “index”: null } ] } … YES 201 Created inserting into (customers). csv: psql_exported/customers.csv 200 OK {“container_name”:”products”,”container_type”:”COLLECTION”,”rowkey”:false,”columns”:[{“name”:”id”,”type”:”INTEGER”,”index”:null},{“name”:”name”,”type”:”STRING”,”index”:null},{“name”:”category”,”type”:”STRING”,”index”:null},{“name”:”price”,”type”:”FLOAT”,”index”:null},{“name”:”stock_quantity”,”type”:”INTEGER”,”index”:null}]} ✔ Make Container? { “container_name”: “products”, “container_type”: “COLLECTION”, “rowkey”: false, “columns”: [ { “name”: “id”, “type”: “INTEGER”, “index”: null }, { “name”: “name”, “type”: “STRING”, “index”: null }, { “name”: “category”, “type”: “STRING”, “index”: null }, { “name”: “price”, “type”: “FLOAT”, “index”: null }, { “name”: “stock_quantity”, “type”: “INTEGER”, “index”: null } ] } … YES 201 Created inserting into (products). csv: psql_exported/products.csv 200 OK And now, of course, is the last step: checking to make sure our data was successfully migrated. $ griddb-cloud-cli show customers { “container_name”: “customers”, “container_type”: “COLLECTION”, “rowkey”: false, “columns”: [ { “name”: “customer_id”, “type”: “INTEGER”, “index”: [] }, { “name”: “first_name”, “type”: “STRING”, “index”: [] }, { “name”: “last_name”, “type”: “STRING”, “index”: [] }, { “name”: “email”, “type”: “STRING”, “index”: [] }, { “name”: “phone_number”, “type”: “STRING”, “index”: [] }, { “name”: “address”, “type”: “STRING”, “index”: [] }, { “name”: “created_at”, “type”: “TIMESTAMP”, “timePrecision”: “MILLISECOND”, “index”: [] } ] } And a quick read: $ griddb-cloud-cli read customers -r [ { “name”: “customers”, “stmt”: “select * limit 50”, “columns”: null, “hasPartialExecution”: true }] customer_id,first_name,last_name,email,phone_number,address,created_at, [1 Alice Johnson alice.j@email.com 555-0101 123 Maple St, Springfield 2025-09-04T06:32:05.700Z] [2 Bob Smith bob.smith@email.com 555-0102 456 Oak Ave, Shelbyville 2025-09-04T06:32:05.700Z] [3 Charlie Brown charlie@email.com 555-0103 789 Pine Ln, Capital City 2025-09-04T06:32:05.700Z] [4 Diana Prince diana.p@email.com 901 Birch Rd, Themyscira 2025-09-04T06:32:05.700Z] [5 Ethan Hunt ethan.hunt@email.com 555-0105 1122 Mission St, Los Angeles 2025-09-04T06:32:05.700Z] And for fun: griddb-cloud-cli read graph products –columns ‘stock_quantity’ [“stock_quantity”] [ { “name”: “products”, “stmt”: “select * limit 50”, “columns”: [“stock_quantity”], “hasPartialExecution”: true }] 936 ┤ ╭╮ 906 ┤ ││ 876 ┤ ││ 845 ┤ │╰╮ ╭╮ 815 ┤ │ │ ││ 785 ┤ │ │ ││ 755 ┤ │ │ ╭─╮ ││ 724 ┤ │ │ │ │ ╭╯│ 694 ┤ │ │ │ │ │ ╰╮ ╭╮ 664 ┤ ╭╯ │ │ ╰╮│ │ ││ 633 ┤ │ │ │ ││ │ ││ 603 ┤ │ ╰╮╭╯ ╰╯ │ ││ 573 ┤ │ ││ │ ╭╯│ 543 ┤ │ ││ │ │ │ 512 ┤ │ ││ │ │ ╰╮ ╭╮ 482 ┤ │ ╰╯ │ ╭╮ │ │ │╰╮ 452 ┤ │ │ ││ ╭╯ │ │ │ 421 ┤ ╭─╮ │ │ │╰╮ │ │ │ │ 391 ┤ │ │ ╭╮ ╭╯ │ ╭╯ │ ╭╯ │ │ │ ╭╮ 361 ┤ │ │ ││ │ ╰╮ │ ╰─╯ │ │ │ │╰╮ 331 ┤ │ │ │╰╮ │ │ │ │ │ │ ╭╯ ╰ 300 ┤ ╭╯ │ ╭╯ │ ╭╮ │ │ │ ╰╮ ╭╮ ╭╯ │ │ 270 ┤ │ │ │ ╰╮ ││ │ │ │ │ ╭╯╰╮ │ ╰╮ │ 240 ┤ │ ╰╮ │ │ ╭╯│ ╭╯ │ ╭╯ │ ╭╯ ╰╮ │ │ │ 209 ┤ ╭╯ │ │ ╰╮ │ ╰╮ ╭─╮ │ │ │ │ │ ╰╮ │ │╭╯ 179 ┤ ╭╯ │ ╭╯ │ ╭╯ │ ╭──╯ │ ╭╯ │ │ │╭─╯ │ │ ││ 149 ┤ ╭╯ │ │ ╰╮ │ ╰╮ ╭╯ ╰╮│ │ │ ╰╯ ╰╮│ ││ 119 ┤ ╭╯ │ ╭╮ │ │╭─╯ │ ╭╯ ╰╯ │ │ ╰╯ ││ 88 ┤╭╯ │ ╭╯╰─╯ ╰╯ ╰─╮╭╯ │ │ ╰╯ 58 ┼╯ ╰─╯ ╰╯ │ ╭─╯ 28 ┤ ╰───╯ Col names from container products ■ stock_quantity Conclusion Never has it been easier to migrate from an on-prem SQL database to GridDB Cloud, and we hope these tools make any of those looking to try out GridDB Cloud a much simpler