Sentiment Analysis of Tweets Using Python

What is Sentiment Analysis?

Sentiment Analysis is the process of computationally determining whether a piece of content is positive, negative or neutral. It is also known as Opinion Mining.

Why Sentiment Analysis?

Sentiment Analysis is mainly used to gauge the views of public regarding any action, event, person, policy or product. It has become a very potent weapon even for politicians to assess the public reaction over their statements. These days Opinion Mining has reached an advanced stage where several outcomes can be predicted using large datasets and machine learning etc.

However, in this post, we will restrict ourselves to extracting 1000 tweets about Narendra Modi, Prime Minister of India and do a sentiment analysis and calculate the percentages of Positive, Negative and Neutral Views.

Tools & Libraries Required

Tweepy:

Tweepy is an easy to use Python library for accessing the Twitter API

We will be using Tweepy to extract tweets from Twitter Stream.

You can install tweepy using the command

pip install tweepy

TextBlob:

TextBlob is a Python (2 and 3) library for processing textual data. It provides a simple API for diving into common natural language processing (NLP) tasks such as part-of-speech tagging, noun phrase extraction, sentiment analysis, classification, translation, and more.

You can install textblob using the command

pip install textblob

You will also need to the Natural Language ToolKit.

python -m textblob.download_corpora

Extraction of Tweets

Registering App with Twitter

To extract tweets from Twitter Stream using API, we first need to register an App with Twitter. Go to TwitterApps and click on New App after signing up.

You can leave the Callback URL empty. Agree to the Developer Conditions and select Create App.

We need the Secret Keys and Access Tokens for the API to work. Please Click on “Keys and Access Tokens” Tab. You will find Consumer Key and Consumer Secret. Note them down.

Now, we need to create Access Tokens for our Account. Click on “Create my access token”

And then note down the “Access Token” and “Access Token Secret”

Now we are ready to retrieve tweets from Twitter Stream. Let us extract a few tweets using the above details

Retrieve Tweets

Let us write a class TwitterClient for extracting Tweets. We will initialize our connection in the constructor using the code

Run the program at the terminal – you will be finding 10 tweets that contain “Narendra Modi”

• We have defined a TwitterClient class that handles all interaction with Twitter API
• We use a __init__ function to handle the authentication.
• We are retrieving the tweets using the get_tweets function

Prepare for Natural Language Parsing

However, if you observe, the tweets contain external links, a lot of white spaces etc. For our Opinion Mining, we do not need all of it. Let us remove all external links, special characters using a regular expression. You can read more about regular expressions here

Let us add one function trim_text which handles the cleaning up of the tweet.

def trim_tweet(self, tweet):

Now instead of directly assigning into parsed_tweet, we will assign

parsed_tweet['text'] = self.trim_tweet(tweet.text)

Run the program again, you will find that Tweets are trimmed and shortened.

Now let us proceed to the most important part of the exercise – Opinion Mining using TextBlob

Opinion Mining Using TextBlob

When we create a TextBlob object, the following processing happens:

• Tokenize the tweet – The tweet will be split into constituent words.
• Remove stop words from the tweet, which are not relevant for the mining. For example, words like am, are etc.
• Parts of Speech Tagging is the most important activity done by TextBlob object. Here, adjectives, adverbs etc. which are relevant are tagged to the keyword.
• Passing the processed tokens to Sentiment Classifier which will return a value between -1.0 and 1.0

Let me explain a bit more about how the Sentiment Classifier works:

• TextBlob uses a large Movie Review Dataset which is pre-classified as positive and negative.
• TextBlob trains using the Naive Bayes classifier to determine positive and negative reviews.
• For the training, we can change the data set – but that is for another project😊
• Now, the sentiment classifier essentially calculates the polarity of tokens between -1.0 and 1.0
• -1.0 is negative, 0.0 is neutral and 1.0 is positive

Now that we understand the modus operandi of Opinion Mining, let us write a function get_tweet_sentiment

def get_tweet_sentiment(self, tweet):
   # create TextBlob object of passed tweet text
   analysis = TextBlob(self.trim_tweet(tweet))
   # set sentiment
   if analysis.sentiment.polarity > 0:
      return 'positive'
   elif analysis.sentiment.polarity == 0:
      return 'neutral'
   else:
      return 'negative'

Complete Program

You should get output something similar to this.

Want to learn Python? Get started now!