Analyzing text about Data Science
Contents
42.9. Analyzing text about Data Science#
In this example, let’s do a simple exercise that covers all steps of a traditional data science process. You do not have to write any code, you can just click on the cells below to execute them and observe the result. As a challenge, you are encouraged to try this code out with different data.
42.9.1. Goal#
In this section, we have been discussing different concepts related to Data Science. Let’s try to discover more related concepts by doing some text mining. We will start with a text about Data Science, extract keywords from it, and then try to visualize the result.
As a text, I will use the page on Data Science from Wikipedia:
import ipytest
import unittest
import pytest
ipytest.autoconfig()
url = 'https://en.wikipedia.org/wiki/Data_science'
42.9.2. Step 1: Getting the data#
First step in every data science process is getting the data. We will use requests library to do that as below.
import requests
def http_get(url):
return requests.____.____
text = http_get(url)
print(text[:1000])
Check result by executing below... đź“ť
%%ipytest -qq
test_url = "https://bing.com"
def test_http_get_happy_case():
# act
actual_result = http_get(test_url)
# assert
assert actual_result.startswith("<!doctype html>")
def test_http_get_with_none_url():
# act & assert
with pytest.raises(Exception):
http_get(None)
def test_http_get_with_invalid_url():
# act & assert
with pytest.raises(Exception):
actual_result = http_get("https://notexisting")
👩‍💻 Hint
You can consider to use the get method of requests library to do that.
42.9.3. Step 2: Transforming the data#
The next step is to convert the data into the form suitable for processing. In our case, we have downloaded HTML source code from the page, and we need to convert it into plain text.
There are many ways this can be done. We will use the simplest built-in HTMLParser object from Python. We need to subclass the HTMLParser class and define the code that will collect all text inside HTML tags, except <script> and <style> tags.
from html.parser import HTMLParser
# The HTML tags not included in the result. such as <script /> and <style />.
TAGS_TO_SKIP = ____
class MyHTMLParser(HTMLParser):
# The flag to track if or not to skip the current tag.
_skip = False
res = ""
def handle_starttag(self, tag, attrs):
if tag.lower() ____: # tag is in the TAGS_TO_SKIP list
____ # will skip the tag
def handle_endtag(self, tag):
if tag.lower() ____: # tag is in the TAGS_TO_SKIP list
____ # finish to parse the content of the skipped tag and end the skipping
def handle_data(self, data):
if ____ or ____: # tag is skipped or the content is empty
return
# Apply the specific parsing logic if appliable.
# E.g. remove the Wikipedia's [ edit ] lable from the content.
parsed_data = ____
# Append the result
self.res = ____
parser = MyHTMLParser()
parser.feed(text)
text = parser.res
print(text[:1000])
Check result by executing below... đź“ť
%%ipytest -qq
class TestMyHTMLParser(unittest.TestCase):
def test_init(self):
# act
parser = MyHTMLParser()
# assert
assert parser
def test_feed_happy_case(self):
# assign
parser = MyHTMLParser()
test_content = "test_content"
# act
parser.feed(f"""
<div>
<p>{test_content}</p>
<a>{test_content}</a>
</div>
""")
actual_result = parser.res
# assert
assert actual_result.strip() == f"{test_content} {test_content}"
def test_feed_with_empty_tag(self):
# assign
parser = MyHTMLParser()
test_content = "test_content"
# act
parser.feed(f"""
<div>
<p></p>
<a>{test_content}</a>
</div>
""")
actual_result = parser.res
# assert
assert actual_result.strip() == test_content
def test_feed_with_skipped_tag(self):
# assign
parser = MyHTMLParser()
test_content = "test_content"
# act
parser.feed(f"""
<html>
<style>ANY_STYLE_CONTENT</style>
<script>ANY_SCRIPT_CONTENT</script>
<div>
<a>{test_content}</a>
</div>
</html>
""")
actual_result = parser.res
# assert
assert actual_result.strip() == test_content
def test_feed_with_empty_content(self):
# assign
parser = MyHTMLParser()
# act
parser.feed("")
actual_result = parser.res
# assert
assert actual_result == ""
def test_feed_with_none(self):
# assign
parser = MyHTMLParser()
# act & assert
with pytest.raises(Exception):
parser.feed(None)
👩‍💻 Hint
You can refer to the HTMLParser documentation to learn more.
42.9.4. Step 3: Getting insights#
The most important step is to turn our data into some form from which we can draw insights. In our case, we want to extract keywords from the text, and see which keywords are more meaningful.
We will use Python library called RAKE for keyword extraction. First, let’s install this library in case it is not present:
import sys
!{sys.executable} -m pip install nlp_rake
The main functionality is available from Rake object, which we can customize using some parameters. In our case, we will set the minimum length of a keyword to 5 characters, minimum frequency of a keyword in the document to 3, and maximum number of words in a keyword - to 2. Feel free to play around with other values and observe the result.
import nlp_rake
# Extract the keywords with these conditions: max words as 2, min frequence as 3, and min characters as 5.
extractor = nlp_rake.Rake(____)
extracted_text = extractor.apply(text)
extracted_text
👩‍💻 Hint
You can consider to use the nlp_rake library to do that.
We obtained a list terms together with associated degree of importance. As you can see, the most relevant disciplines, such as machine learning and big data, are present in the list at top positions.
42.9.5. Step 4: Visualizing the result#
People can interpret the data best in the visual form. Thus it often makes sense to visualize the data in order to draw some insights. We can use matplotlib library in Python to plot simple distribution of the keywords with their relevance:
import matplotlib.pyplot as plt
def plot(pair_list):
k, v = ____ # get the keys and values from pair_list
plt.bar(____, ____) # initialize a bar graph with the k and v
plt.xticks(____, ____, rotation='vertical') # set the ticks of the axies
plt.show() # show the graph
plot(extracted_text)
There is, however, even better way to visualize word frequencies - using Word Cloud. We will need to install another library to plot the word cloud from our keyword list.
!{sys.executable} -m pip install --quiet wordcloud
WordCloud object is responsible for taking in either original text, or pre-computed list of words with their frequencies, and returns and image, which can then be displayed using matplotlib:
from wordcloud import WordCloud
import matplotlib.pyplot as plt
wc = WordCloud(background_color='white', width=800, height=600)
plt.figure(figsize=(15, 7))
plt.imshow(wc.generate_from_frequencies(____)) # unpack the extracted_text as a dict
We can also pass in the original text to WordCloud - let’s see if we are able to get similar result:
plt.figure(figsize=(15, 7))
plt.imshow(wc.generate(____))
wc.generate(text).to_file('images/ds_wordcloud.png')
You can see that word cloud now looks more impressive, but it also contains a lot of noise (eg. unrelated words such as Retrieved on). Also, we get fewer keywords that consist of two words, such as data scientist, or computer science. This is because RAKE algorithm does much better job at selecting good keywords from text. This example illustrates the importance of data pre-processing and cleaning, because clear picture at the end will allow us to make better decisions.
In this exercise we have gone through a simple process of extracting some meaning from Wikipedia text, in the form of keywords and word cloud. This example is quite simple, but it demonstrates well all typical steps a data scientist will take when working with data, starting from data acquisition, up to visualization.
42.9.6. Acknowledgments#
Thanks to Microsoft for creating the open source course Data Science for Beginners. It inspires the majority of the content in this chapter.