{"id":46701,"date":"2022-04-28T00:00:00","date_gmt":"2022-04-28T07:00:00","guid":{"rendered":"https:\/\/griddb-linux-hte8hndjf8cka8ht.westus-01.azurewebsites.net\/blog\/topic-modeling-with-lda-using-python-and-griddb\/"},"modified":"2025-11-13T12:56:01","modified_gmt":"2025-11-13T20:56:01","slug":"topic-modeling-with-lda-using-python-and-griddb","status":"publish","type":"post","link":"https:\/\/griddb.net\/en\/blog\/topic-modeling-with-lda-using-python-and-griddb\/","title":{"rendered":"Topic Modeling with LDA Using Python and GridDB"},"content":{"rendered":"

In natural language processing, topic modeling assigns a topic to a given corpus based on the words in it. Due to the fact that text data is unlabeled, it is an unsupervised technique. It is increasingly important to categorize documents according to topics in this world filled with data. As an example, if a company receives hundreds of reviews, the company will need to know what categories of reviews are the most important and vice versa.<\/p>\n

As keywords, topics can be used to describe a document, for example, when we think of a topic related to economics, we think of stock market, USD, inflation, GPD, etc. Topic models are models that can automatically detect topics based on words appearing in a document. The problem we will tackle here is topic modeling.<\/p>\n

LDA - (Latent Dirichlet Allocation)<\/code><\/p>\n

The word latent means hidden, something that has yet to be discovered. As indicated by Dirichlet, the Dirichlet distribution is assumed to govern the distribution of topics and word patterns in documents. “Allocation” here refers to the process of giving something, in this case, topics.<\/p>\n

In this tutorial, we\u2019ll use the reviews in the following dataset to generate topics from the reviews. In this way, we can know about what users are talking about, what they are focusing on, and perhaps where app developers should make progress at.<\/p>\n

The outline of the tutorial is as follows:<\/p>\n

    \n
  1. Prerequisites and Environment setup<\/li>\n
  2. Dataset overview<\/li>\n
  3. Importing required libraries<\/li>\n
  4. Loading the dataset<\/li>\n
  5. Data Cleaning and Preprocessing<\/li>\n
  6. Building and Training a Machine Learning Model<\/li>\n
  7. Conclusion<\/li>\n<\/ol>\n

    1. Prerequisites and Environment setup<\/h2>\n

    This tutorial is carried out in Anaconda Navigator (Python version \u2013 3.8.3) on Windows Operating System. The following packages need to be installed before you continue with the tutorial \u2013<\/p>\n

      \n
    1. \n

      Pandas<\/p>\n<\/li>\n

    2. \n

      NumPy<\/p>\n<\/li>\n

    3. \n

      Sklearn<\/p>\n<\/li>\n

    4. \n

      nltk<\/p>\n<\/li>\n

    5. \n

      re<\/p>\n<\/li>\n

    6. \n

      griddb_python<\/p>\n<\/li>\n

    7. \n

      spacy<\/p>\n<\/li>\n

    8. \n

      gensim<\/p>\n<\/li>\n<\/ol>\n

      You can install these packages in Conda\u2019s virtual environment using conda install package-name<\/code>. In case you are using Python directly via terminal\/command prompt, pip install package-name<\/code> will do the work.<\/p>\n

      GridDB installation<\/h3>\n

      While loading the dataset, this tutorial will cover two methods \u2013 Using GridDB as well as Using Pandas. To access GridDB using Python, the following packages also need to be installed beforehand:<\/p>\n

        \n
      1. GridDB C-client<\/a><\/li>\n
      2. SWIG (Simplified Wrapper and Interface Generator)<\/li>\n
      3. GridDB Python Client<\/a><\/li>\n<\/ol>\n

        2. Dataset Overview<\/h2>\n

        Google Play Store Apps Dataset : Web scraped data of 10,000 Play Store apps for analyzing the Android market.<\/p>\n

        It can be downloaded from here (https:\/\/www.kaggle.com\/datasets\/lava18\/google-play-store-apps\/version\/5<\/code>).<\/p>\n

        3. Importing Required Libraries<\/h2>\n
        \n
        import griddb_python as griddb\nimport numpy as np\nimport pandas as pd\nimport re, nltk, spacy, gensim\nfrom sklearn.decomposition import LatentDirichletAllocation, TruncatedSVD\nfrom sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\nfrom sklearn.model_selection import GridSearchCV\nfrom pprint import pprint<\/code><\/pre>\n<\/div>\n
        4. Loading the Dataset<\/h2>\n
        Let\u2019s proceed and load the dataset into our notebook.<\/p>\n
        4.a Using GridDB<\/h3>\n
        Toshiba GridDB\u2122 is a highly scalable NoSQL database best suited for IoT and Big Data. The foundation of GridDB\u2019s principles is based upon offering a versatile data store that is optimized for IoT, provides high scalability, tuned for high performance, and ensures high reliability.<\/p>\n
        To store large amounts of data, a CSV file can be cumbersome. GridDB serves as a perfect alternative as it in open-source and a highly scalable database. GridDB is a scalable, in-memory, No SQL database which makes it easier for you to store large amounts of data. If you are new to GridDB, a tutorial on reading and writing to GridDB<\/a> can be useful.<\/p>\n
        Assuming that you have already set up your database, we will now write the SQL query in python to load our dataset.<\/p>\n
        \n
        sql_statement = ('SELECT * FROM googleplaystore_user_reviews')\ndataset = pd.read_sql_query(sql_statement, cont)<\/code><\/pre>\n<\/div>\n
        Note that the cont<\/code> variable has the container information where our data is stored. Replace the bbc-text<\/code> with the name of your container. More info can be found in this tutorial reading and writing to GridDB<\/a>.<\/p>\n
        When it comes to IoT and Big Data use cases, GridDB clearly stands out among other databases in the Relational and NoSQL space. Overall, GridDB offers multiple reliability features for mission-critical applications that require high availability and data retention.<\/p>\n
        4.b Using pandas read_csv<\/h3>\n
        In Python you need to give access to a file by opening it. You can do it by using the open() function. Open returns a file object, which has methods and attributes for getting information about and manipulating the opened file. Both of the above methods will lead to the same output as the data is loaded in the form of a pandas dataframe using either of the methods.<\/p>\n
        \n
        df = pd.read_csv(\"googleplaystore_user_reviews.csv\")\ndf = df.dropna(subset=[\"Translated_Review\"])<\/code><\/pre>\n<\/div>\n
        \n
        df.head()<\/code><\/pre>\n<\/div>\n
        \n