We also recommend installing Anaconda Navigator<\/a>, if not already installed. Anaconda provides a large range of tools for data scientists to experiment with. Also, a virtual environment can help you meet the specific version requirements while running an application without interfering with the actual system paths.<\/p>\n For this tutorial, we will be using Anaconda\u00e2\u20ac\u2122s Command Line Interface and Jupyter Notebooks. Both of these tools can be found in the Anaconda Dashboard.<\/p>\n Creating a new project with scrapy is simple. Just type the following command within the directory you wish to create a new project folder in:<\/p>\n A new folder with the name Scrapy uses a class called The following lines of code collect the information about the text, author, and tags associated with a quote.<\/p>\n We will now save this python file in the Coming back to the home directory. Let\u00e2\u20ac\u2122s run this spider now and see what we get –<\/p>\n It takes some time to extract the data. Once the execution is completed, our output looks like –<\/p>\n To store the previously crawled data, we modify the above command by simply passing an additional parameter –<\/p>\n This will create a new file named The content of the If you\u00e2\u20ac\u2122re collecting continuous data over time, it is only a wise decision to store it in a Database. GridDB<\/a> allows you to store time-series data and is specially optimized for IoT and Big Data. Its highly scalable nature gives room to both SQL and NoSQL interfaces. Follow their tutorial<\/a> to get started.<\/p>\n We have our data collected from scrapy in a JSON file. Moreover, most websites allow you to export data in JSON format. Therefore, we will be writing a python script to load the data in our environment.<\/p>\n We will get an output similar to<\/p>\n To extract all the key-value pairs –<\/p>\n Now that we have extracted each key-value pair, let us initialize a GridDB instance.<\/p>\n Fill in your custom details in the above code. Note that in our case, the data type is essentially a Our JSON file is a list of dictionaries. Each dictionary contains 3 attributes: Text, Author, and Tags. We will have to run two loops to get items under each category.<\/p>\n The final insertion script looks like this –<\/p>\n Check out default cluster values on the official Github page of GridDB\u00e2\u20ac\u2122s python-client<\/a>.<\/p>\n In this tutorial, we saw how to create a spider in order to crawl data from a website. We stored the collected data in a JSON format so that it is easy to share among different platforms. Later on, we developed an insertion script for storing this data into GridDB.<\/p>\n Storing data in a database is crucial if you\u00e2\u20ac\u2122re working on continuous data. It can be hard to store multiple JSON files in such a case. GridDB<\/a> makes it easier to store every bit of information in one place. This saves time and helps team integrate without any hassle. Get started with GridDB<\/a> today!<\/p>\n","protected":false},"excerpt":{"rendered":" Introduction Today, we will cover how to scrape data from any website using Python\u00e2\u20ac\u2122s library Scrapy. We will then save the data in a JSON and HTML file. Finally, we will see how we can also store this data in GridDB for long-term and efficient use. Pre-requisites This post requires the prior installation of the […]<\/p>\n","protected":false},"author":41,"featured_media":27517,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[121],"tags":[],"class_list":["post-46650","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog"],"acf":[],"yoast_head":"\nCreating a new project using Scrapy<\/h2>\n
![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2021\/06\/unnamed.png\")
<\/a><\/p>\nscrapy startproject griddb_tutorial\n<\/code><\/pre>\n<\/div>\ngriddb_tutorial<\/code> is now created in the current directory. Let us look at the contents of this folder:<\/p>\ntree directory_path \/F\n<\/code><\/pre>\n<\/div>\nExtracting data from a URL<\/h2>\n
Spider<\/code> to crawl websites and extract information. We can write our custom code and mention initial requests inside this Spider<\/code> class. For this tutorial, we will be scraping funny quotes from the website quotes.toscrape.com<\/a> and storing the information in JSON format.<\/p>\nimport scrapy\n\n\nclass QuotesSpider(scrapy.Spider):\n name = \"quotes_funny\"\n start_urls = [\n 'http:\/\/quotes.toscrape.com\/tag\/humor',\n ]\n\n def parse(self, response):\n for quote in response.css('div.quote'):\n yield {\n 'text': quote.css('span.text::text').get(),\n 'author': quote.css('small.author::text').get(),\n 'tags': quote.css('div.tags a.tag::text').getall(),\n }<\/code><\/pre>\n<\/div>\n\/griddb_tutorial\/spider<\/code> directory. We execute a spider by providing its name to scrapy through the command line. Therefore, it is important to use a unique name for each spider.<\/p>\nscrapy crawl quotes_funny<\/code><\/pre>\n<\/div>\nDEBUG: Scraped from <200 http:\/\/quotes.toscrape.com\/tag\/humor\/>\n{'text': '\"The reason I talk to myself is because I'm the only one whose answers I accept.\"', 'author': 'George Carlin', 'tags': ['humor', 'insanity', 'lies', 'lying', 'self-indulgence', 'truth']}\n2021-05-29 21:29:44 [scrapy.core.engine] INFO: Closing spider (finished)<\/code><\/pre>\n<\/div>\nStoring data into JSON<\/h2>\n
scrapy crawl quotes_funny -O quotes_funny.json<\/code><\/pre>\n<\/div>\nquotes_funny.json<\/code> in the home directory. Note that the -O<\/code> command overwrites any existing file with the same name. In case, you want to append new content to an existing JSON<\/code> file, use -o<\/code> instead.<\/p>\nJSON<\/code> file will look like this:<\/p>\n[\n{\"text\": \"u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.u201d\", \"author\": \"Jane Austen\", \"tags\": [\"aliteracy\", \"books\", \"classic\", \"humor\"]},\n{\"text\": \"u201cAll you need is love. But a little chocolate now and then doesn't hurt.u201d\", \"author\": \"Charles M. Schulz\", \"tags\": [\"chocolate\", \"food\", \"humor\"]},\n{\"text\": \"u201cThe reason I talk to myself is because Iu2019m the only one whose answers I accept.u201d\", \"author\": \"George Carlin\", \"tags\": [\"humor\", \"insanity\", \"lies\", \"lying\", \"self-indulgence\", \"truth\"]}\n]<\/code><\/pre>\n<\/div>\nStoring data into GridDB<\/h2>\n
Reading data from a JSON file<\/h3>\n
import json\nf = open('quotes_funny.json',)\ndata = json.load(f)\nprint(data[0])<\/code><\/pre>\n<\/div>\n{'text': '\"The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\"',\n 'author': 'Albert Einstein',\n 'tags': ['change', 'deep-thoughts', 'thinking', 'world']}<\/code><\/pre>\n<\/div>\nfor d in data:\n for key, value in d.items():\n print(key, value)<\/code><\/pre>\n<\/div>\nGridDB container initialization<\/h3>\n
import griddb_python as griddb\n\nfactory = griddb.StoreFactory.get_instance()\n\n# Container Initialization\ntry:\n gridstore = factory.get_store(host=your_host, port=your_port, \n cluster_name=your_cluster_name, username=your_username, \n password=your_password)\n\n conInfo = griddb.ContainerInfo(\"Dataset_Name\",\n [[\"attribute1\", griddb.Type.STRING],[\"attribute2\",griddb.Type.FLOAT],\n ....],\n griddb.ContainerType.COLLECTION, True)\n \n cont = gridstore.put_container(conInfo) \n cont.create_index(\"id\", griddb.IndexType.DEFAULT)\n\nexcept griddb.GSException as e:\n for i in range(e.get_error_stack_size()):\n print(\"[\", i, \"]\")\n print(e.get_error_code(i))\n print(e.get_location(i))\n print(e.get_message(i))<\/code><\/pre>\n<\/div>\nSTRING<\/code>. More information on data types supported by GridDB can be found here<\/a>.<\/p>\nInsert data into the GridDB container<\/h3>\n
for d in data:\n for key in d:\n ret = cont.put(d[key])<\/code><\/pre>\n<\/div>\nimport griddb_python as griddb\n\nfactory = griddb.StoreFactory.get_instance()\n\n# Container Initialization\ntry:\n gridstore = factory.get_store(host=your_host, port=your_port, \n cluster_name=your_cluster_name, username=your_username, \n password=your_password)\n\n conInfo = griddb.ContainerInfo(\"Dataset_Name\",\n [[\"attribute1\", griddb.Type.INTEGER],[\"attribute2\",griddb.Type.FLOAT],\n ....],\n griddb.ContainerType.COLLECTION, True)\n \n cont = gridstore.put_container(conInfo) \n cont.create_index(\"id\", griddb.IndexType.DEFAULT)\n \n #Adding data to container\n for d in data:\n for key in d:\n ret = cont.put(d[key])\n\nexcept griddb.GSException as e:\n for i in range(e.get_error_stack_size()):\n print(\"[\", i, \"]\")\n print(e.get_error_code(i))\n print(e.get_location(i))\n print(e.get_message(i))<\/code><\/pre>\n<\/div>\nConclusion<\/h2>\n