{"id":52247,"date":"2025-10-17T00:00:00","date_gmt":"2025-10-17T07:00:00","guid":{"rendered":"https:\/\/griddb-linux-hte8hndjf8cka8ht.westus-01.azurewebsites.net\/blog\/building-a-hybrid-rag-using-griddb-llms-and-faiss-for-efficient-text-analysis\/"},"modified":"2025-10-17T00:00:00","modified_gmt":"2025-10-17T07:00:00","slug":"building-a-hybrid-rag-using-griddb-llms-and-faiss-for-efficient-text-analysis","status":"publish","type":"post","link":"https:\/\/griddb.net\/en\/blog\/building-a-hybrid-rag-using-griddb-llms-and-faiss-for-efficient-text-analysis\/","title":{"rendered":"Building a Hybrid RAG using GridDB, LLMs and FAISS for Efficient Text Analysis"},"content":{"rendered":"
\n
\n
\n
\n<\/div>\n
\n
In\u00a0[279]:<\/div>\n
\n
\n
<\/span>import<\/span> <\/span>os<\/span>\nimport<\/span> <\/span>pandas<\/span> <\/span>as<\/span> <\/span>pd<\/span>\nimport<\/span> <\/span>numpy<\/span> <\/span>as<\/span> <\/span>np<\/span>\nimport<\/span> <\/span>requests<\/span>\nimport<\/span> <\/span>subprocess<\/span>\nfrom<\/span> <\/span>pathlib<\/span> <\/span>import<\/span> Path<\/span>\nimport<\/span> <\/span>zipfile<\/span>\nimport<\/span> <\/span>kagglehub<\/span>\nimport<\/span> <\/span>shutil<\/span>\nimport<\/span> <\/span>json<\/span>\nimport<\/span> <\/span>base64<\/span>\nimport<\/span> <\/span>http<\/span>\nhttp<\/span>.<\/span>client<\/span>.<\/span>HTTPConnection<\/span>.<\/span>debuglevel<\/span> =<\/span> 0<\/span>\nfrom<\/span> <\/span>IPython.display<\/span> <\/span>import<\/span> Image<\/span>,<\/span> display<\/span>,<\/span> HTML<\/span>\nfrom<\/span> <\/span>transformers<\/span> <\/span>import<\/span> AutoTokenizer<\/span>,<\/span> AutoModelForSeq2SeqLM<\/span>,<\/span> pipeline<\/span> # Generative Model<\/span>\nimport<\/span> <\/span>sentencepiece<\/span> # Generative Model<\/span>\nimport<\/span> <\/span>faiss<\/span> # Search & Indexing<\/span>\nfrom<\/span> <\/span>sentence_transformers<\/span> <\/span>import<\/span> SentenceTransformer<\/span> #Retrieval Model<\/span>\nfrom<\/span> <\/span>collections<\/span> <\/span>import<\/span> OrderedDict<\/span> #Standardising Results<\/span>\n<\/pre><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
\n<\/div>\n
\n<\/div>
\n
Introduction\u00b6<\/a><\/h2>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
\n<\/div>\n
\n<\/div>
\n
 In this article, we demonstrate an innovative use case involving GridDB and FAISS to implement a hybrid RAG (Retrieval-Augmented Generation) system. While open-source libraries like FAISS can index and search semantic similarities in text effortlessly, high-performance databases like GridDB can help narrow down to the desired subset of data using structured queries across large volumes of unstructured data. This combination of structured retrieval followed by semantic retrieval using FAISS reduces operational overhead and minimizes computational costs associated with mining large-scale unstructured data. <\/p>\n
\nThe dataset we use for this purpose is the Kaggle News Category Dataset. This dataset comprises news headlines and metadata sourced between 2012 and 2022, and has news snippets in various categories, including politics, wellness, and others. The dataset can be downloaded using the Python library 'kagglehub'. Refer to https:\/\/www.kaggle.com\/datasets\/rmisra\/news-category-dataset\/data to learn more ab out downloading the dataset using the 'Download' option. Below is a snapshot of the download option. <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>
\n
\n
\n<\/div>\n
\n
In\u00a0[275]:<\/div>\n
\n
\n
<\/span>Image<\/span>(<\/span>filename<\/span>=<\/span>'Kagglehub_Dataset.png'<\/span>,<\/span>width<\/span>=<\/span>800<\/span>,<\/span> height<\/span>=<\/span>500<\/span>)<\/span>\n<\/pre><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n<\/div>\n
\n
\n
Out[275]:<\/div>\n
\n\"No\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
\n<\/div>\n
\n<\/div>
\n
Downloading the Dataset\u00b6<\/a><\/h2>\n<\/div>\n<\/div>\n<\/div>\n<\/div>
\n
\n
\n<\/div>\n
\n
In\u00a0[\u00a0]:<\/div>\n
\n
\n
<\/span># Download and get dataset folder path<\/span>\ndataset_dir<\/span> =<\/span> kagglehub<\/span>.<\/span>dataset_download<\/span>(<\/span>\"rmisra\/news-category-dataset\"<\/span>)<\/span>\n\n# Destination: your current working directory<\/span>\ndest_dir<\/span> =<\/span> os<\/span>.<\/span>getcwd<\/span>()<\/span>\n\n# Copy all files from kagglehub's dataset dir to the current working dir<\/span>\nfor<\/span> filename<\/span> in<\/span> os<\/span>.<\/span>listdir<\/span>(<\/span>dataset_dir<\/span>):<\/span>\n    src_file<\/span> =<\/span> os<\/span>.<\/span>path<\/span>.<\/span>join<\/span>(<\/span>dataset_dir<\/span>,<\/span> filename<\/span>)<\/span>\n    dst_file<\/span> =<\/span> os<\/span>.<\/span>path<\/span>.<\/span>join<\/span>(<\/span>dest_dir<\/span>,<\/span> filename<\/span>)<\/span>\n\n    # Only copy if it's a file<\/span>\n    if<\/span> os<\/span>.<\/span>path<\/span>.<\/span>isfile<\/span>(<\/span>src_file<\/span>):<\/span>\n        shutil<\/span>.<\/span>copy<\/span>(<\/span>src_file<\/span>,<\/span> dst_file<\/span>)<\/span>\n        print<\/span>(<\/span>f<\/span>\"Copied <\/span>{<\/span>filename<\/span>}<\/span> to <\/span>{<\/span>dst_file<\/span>}<\/span>\"<\/span>)<\/span>\n\nprint<\/span>(<\/span>\"All files copied.\"<\/span>)<\/span>\n<\/pre><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>
\n
\n
\n<\/div>\n
\n
In\u00a0[3]:<\/div>\n
\n
\n
<\/span># Path to the downloaded JSON file<\/span>\njson_path<\/span> =<\/span> \"News_Category_Dataset_v3.json\"<\/span>  # or whatever the filename is<\/span>\n\n# Load as a list of JSON objects (one per line)<\/span>\nwith<\/span> open<\/span>(<\/span>json_path<\/span>,<\/span> 'r'<\/span>,<\/span> encoding<\/span>=<\/span>'utf-8'<\/span>)<\/span> as<\/span> f<\/span>:<\/span>\n    data<\/span> =<\/span> [<\/span>json<\/span>.<\/span>loads<\/span>(<\/span>line<\/span>)<\/span> for<\/span> line<\/span> in<\/span> f<\/span>]<\/span>\n\n# Convert to DataFrame<\/span>\nnews_data<\/span> =<\/span> pd<\/span>.<\/span>DataFrame<\/span>(<\/span>data<\/span>)<\/span>\n<\/pre><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
\n<\/div>\n
\n<\/div>
\n
Standardizing and Subsetting Data for Ingestion (GridDB)\u00b6<\/a><\/h2>\n<\/div>\n<\/div>\n<\/div>\n<\/div>
\n
\n
\n<\/div>\n
\n
In\u00a0[4]:<\/div>\n
\n
\n
<\/span>trending_categories<\/span> =<\/span> [<\/span>\n    \"WELLNESS\"<\/span>,<\/span> \"TECH\"<\/span>,<\/span> \"SCIENCE\"<\/span>,<\/span>\n    \"TRAVEL\"<\/span>,<\/span> \"STYLE & BEAUTY\"<\/span>,<\/span> \"ENTERTAINMENT\"<\/span>,<\/span>\"POLITICS\"<\/span>]<\/span>\nnews_trending<\/span> =<\/span> news_data<\/span>[<\/span>news_data<\/span>[<\/span>'category'<\/span>]<\/span>.<\/span>isin<\/span>(<\/span>trending_categories<\/span>)]<\/span>.<\/span>copy<\/span>()<\/span>\nnews_trending<\/span>[<\/span>'category'<\/span>]<\/span>.<\/span>value_counts<\/span>()<\/span>\n<\/pre><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n<\/div>\n
\n
\n
Out[4]:<\/div>\n
\n
category\nPOLITICS          35602\nWELLNESS          17945\nENTERTAINMENT     17362\nTRAVEL             9900\nSTYLE & BEAUTY     9814\nSCIENCE            2206\nTECH               2104\nName: count, dtype: int64<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
\n<\/div>\n
\n<\/div>
\n
Adding new columns - id and text to the dataframe ('id' - rowkey)\u00b6<\/a><\/h3>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
\n<\/div>\n
\n<\/div>
\n
    \n
  1. Add a unique ID column (We create this for GridDB's row key requirement for collection containers)<\/li>\n
  2. Create a combined text column (recommended for RAG embedding input)<\/li>\n
  3. Then chunk and upload<\/li>\n<\/ol>\n<\/div>\n<\/div>\n<\/div>\n<\/div>
    \n
    \n
    \n<\/div>\n
    \n
    In\u00a0[24]:<\/div>\n
    \n
    \n
    <\/span>news_trending<\/span> =<\/span> news_trending<\/span>.<\/span>reset_index<\/span>(<\/span>drop<\/span>=<\/span>True<\/span>)<\/span>\nnews_trending<\/span>[<\/span>'id'<\/span>]<\/span> =<\/span> news_trending<\/span>.<\/span>index<\/span> +<\/span> 1<\/span>\nnews_trending<\/span>[<\/span>'text'<\/span>]<\/span> =<\/span> news_trending<\/span>[<\/span>'headline'<\/span>]<\/span> +<\/span> \" \u2014 \"<\/span> +<\/span> news_trending<\/span>[<\/span>'short_description'<\/span>]<\/span>\n<\/pre><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
    \n
    \n
    \n<\/div>\n
    \n<\/div>
    \n
    Connecting to GridDB\u00b6<\/a><\/h2>\n<\/div>\n<\/div>\n<\/div>\n<\/div>
    \n
    \n
    \n<\/div>\n
    \n
    In\u00a0[\u00a0]:<\/div>\n
    \n
    \n
    <\/span>username<\/span> =<\/span> XX<\/span>  #Place your username here<\/span>\npassword<\/span> =<\/span>   XX<\/span> #Place your password here<\/span>\ncredentials<\/span> =<\/span> f<\/span>\"<\/span>{<\/span>username<\/span>}<\/span>:<\/span>{<\/span>password<\/span>}<\/span>\"<\/span>\nencoded_credentials<\/span> =<\/span> base64<\/span>.<\/span>b64encode<\/span>(<\/span>credentials<\/span>.<\/span>encode<\/span>())<\/span>.<\/span>decode<\/span>()<\/span>\nprint<\/span>(<\/span>f<\/span>\"Encoded credentials: Basic <\/span>{<\/span>encoded_credentials<\/span>}<\/span>\"<\/span>)<\/span>\n<\/pre><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>
    \n
    \n
    \n<\/div>\n
    \n
    In\u00a0[7]:<\/div>\n
    \n
    \n
    <\/span>#Construct an object to hold the request headers (ensure that you replace the XXX placeholder with the correct value that matches the credentials for your GridDB instance)<\/span>\n#header_obj = {\"Authorization\":\"Basic dXNlck0wMWt3VzE0Y3Q6Q1BYZ2liNzg4\",\"Content-Type\":\"application\/json; charset=UTF-8\",\"User-Agent\":\"PostmanRuntime\/7.29.0\"}<\/span>\n\nheader_obj<\/span> =<\/span> {<\/span>\n    \"Authorization\"<\/span>:<\/span> f<\/span>\"Basic <\/span>{<\/span>encoded_credentials<\/span>}<\/span>\"<\/span>,<\/span>  # Add encoded credentials here<\/span>\n    \"Content-Type\"<\/span>:<\/span> \"application\/json\"<\/span>,<\/span>  # Optional; depends on API requirements<\/span>\n    \"charset\"<\/span>:<\/span> \"UTF-8\"<\/span>,<\/span>\n    \"User-Agent\"<\/span>:<\/span>\"PostmanRuntime\/7.29.0\"<\/span>\n}<\/span>\n\n#Construct the base URL based on your GridDB cluster you'd like to connect to (ensure that you replace the placeholders in the URL below with the correct values <\/span>\n#that correspond to your GridDB instance)<\/span>\nbase_url<\/span> =<\/span> 'https:\/\/[host]:[port]\/griddb\/v2\/[clustername]\/dbs\/[database_name]\/'<\/span>\n<\/pre><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
    \n
    \n
    \n<\/div>\n
    \n<\/div>
    \n
    Generating Container Structures for use in GridDB\u00b6<\/a><\/h2>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
    \n
    \n
    \n<\/div>\n
    \n<\/div>
    \n
    The following functions help in -<\/p>\n
      \n
    1.  mapping the datatypes of the columns in the dataframe to the corresponding datatype in GridDB. <\/li>\n
    2.  creating the container structure that needs to be passed to the Web API. <\/li>\n<\/ol>\n<\/div>\n<\/div>\n<\/div>\n<\/div>
      \n
      \n
      \n<\/div>\n
      \n
      In\u00a0[8]:<\/div>\n
      \n
      \n
      <\/span>def<\/span> <\/span>map_dtype_to_griddb<\/span>(<\/span>dtype<\/span>):<\/span>\n    <\/span>\"\"\"<\/span>\n    Maps Pandas data types to GridDB data types.<\/span>\n    \"\"\"<\/span>\n    if<\/span> pd<\/span>.<\/span>api<\/span>.<\/span>types<\/span>.<\/span>is_integer_dtype<\/span>(<\/span>dtype<\/span>):<\/span>\n        return<\/span> \"INTEGER\"<\/span>\n    elif<\/span> pd<\/span>.<\/span>api<\/span>.<\/span>types<\/span>.<\/span>is_float_dtype<\/span>(<\/span>dtype<\/span>):<\/span>\n        return<\/span> \"DOUBLE\"<\/span>\n    elif<\/span> pd<\/span>.<\/span>api<\/span>.<\/span>types<\/span>.<\/span>is_bool_dtype<\/span>(<\/span>dtype<\/span>):<\/span>\n        return<\/span> \"BOOL\"<\/span>\n    elif<\/span> pd<\/span>.<\/span>api<\/span>.<\/span>types<\/span>.<\/span>is_datetime64_any_dtype<\/span>(<\/span>dtype<\/span>):<\/span>\n        return<\/span> \"LONG\"<\/span>  # GridDB stores timestamps as LONG<\/span>\n    else<\/span>:<\/span>\n        return<\/span> \"STRING\"<\/span>\n\ndef<\/span> <\/span>generate_griddb_data_obj<\/span>(<\/span>df<\/span>,<\/span> container_name<\/span>=<\/span>\"MyContainer\"<\/span>,<\/span> container_type<\/span>=<\/span>\"COLLECTION\"<\/span>,<\/span> rowkey<\/span>=<\/span>False<\/span>):<\/span>\n    <\/span>\"\"\"<\/span>\n    Generates a GridDB container data object for API request.<\/span>\n    \"\"\"<\/span>\n    columns<\/span> =<\/span> []<\/span>\n    for<\/span> col<\/span> in<\/span> df<\/span>.<\/span>columns<\/span>:<\/span>\n        griddb_type<\/span> =<\/span> map_dtype_to_griddb<\/span>(<\/span>df<\/span>[<\/span>col<\/span>]<\/span>.<\/span>dtype<\/span>)<\/span>\n        columns<\/span>.<\/span>append<\/span>({<\/span>\"name\"<\/span>:<\/span> col<\/span>,<\/span> \"type\"<\/span>:<\/span> griddb_type<\/span>})<\/span>\n    \n    data_obj<\/span> =<\/span> {<\/span>\n        \"container_name\"<\/span>:<\/span> container_name<\/span>,<\/span>\n        \"container_type\"<\/span>:<\/span> container_type<\/span>,<\/span>\n        \"rowkey\"<\/span>:<\/span> rowkey<\/span>,<\/span>\n        \"columns\"<\/span>:<\/span> columns<\/span>\n    }<\/span>\n    \n    return<\/span> data_obj<\/span>\n<\/pre><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
      \n
      \n
      \n<\/div>\n
      \n<\/div>
      \n
      Creating Containers in GridDB\u00b6<\/a><\/h2>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
      \n
      \n
      \n<\/div>\n
      \n<\/div>
      \n
      GridDB supports two types of containers namely Collections and Timeseries. While collections are ideal for storing key-value data similar to relational data tables, timeseries containers are optimized for time-stamped data like IoT and sensor data. In this case, we store the unstructured article text and related metadata in collection containers. Refer to  this resource <\/a> to learn more.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>
      \n
      \n
      \n<\/div>\n
      \n
      In\u00a0[81]:<\/div>\n
      \n
      \n
      <\/span>data_obj<\/span> =<\/span> generate_griddb_data_obj<\/span>(<\/span>news_trending<\/span>,<\/span> container_name<\/span>=<\/span>\"news_category_data\"<\/span>)<\/span>\n\n#Set up the GridDB WebAPI URL<\/span>\nurl<\/span> =<\/span> base_url<\/span> +<\/span> 'containers'<\/span>\n\n#Invoke the GridDB WebAPI with the headers and the request body<\/span>\nx<\/span> =<\/span> requests<\/span>.<\/span>post<\/span>(<\/span>url<\/span>,<\/span> json<\/span> =<\/span> data_obj<\/span>,<\/span> headers<\/span> =<\/span> header_obj<\/span>)<\/span>\n<\/pre><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
      \n
      \n
      \n<\/div>\n
      \n<\/div>
      \n
      Row-Registration & the Role of a rowkey in Collection Containers\u00b6<\/a><\/h2>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
      \n
      \n
      \n<\/div>\n
      \n<\/div>
      \n
      Row Registration in GridDB is analogous to inserting data in relational SQL databases. Refer to the  GridDB Data Model <\/a> to learn more about row registration and to know how GridDB processes transactions.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>
      \n
      \n
      \n<\/div>\n
      \n
      In\u00a0[9]:<\/div>\n
      \n
      \n
      <\/span># Parse the JSON string into a Python list of dictionaries<\/span>\nrequest_body_news_data<\/span> =<\/span> news_trending<\/span>.<\/span>to_json<\/span>(<\/span>orient<\/span>=<\/span>'values'<\/span>)<\/span>\n<\/pre><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>
      \n
      \n
      \n<\/div>\n
      \n
      In\u00a0[10]:<\/div>\n
      \n
      \n
      <\/span>#Setup the URL to be used to invoke the GridDB WebAPI to register rows in the container created previously<\/span>\nurl<\/span> =<\/span> base_url<\/span> +<\/span> 'containers\/news_category_data\/rows'<\/span>\n\n#Invoke the GridDB WebAPI using the request constructed<\/span>\n#x = requests.put(url, data=request_body_news_data, headers=header_obj)<\/span>\n<\/pre><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
      \n
      \n
      \n<\/div>\n
      \n<\/div>
      \n
      In GridDB, the rowkey is optimal but not essential. The rowkey acts as a unique identifier (similar to a primary key in relational data tables) in collections, but they are not strictly necessary.<\/p>\n
      However, having rowkey=true for a collection enables -<\/p>\n