Introduction

Natural Language Processing (NLP): Natural Language Processing is an interdisciplinary field that combines computer science, artificial intelligence, and linguistics. It is dedicated to the development of computational models for processing and comprehending natural language. These models are utilized for various tasks such as semantic word grouping, text-to-speech synthesis, and language translation.

Sentiment Analysis: Sentiment Analysis, also known as opinion mining, is a valuable practice in text analysis. It involves the interpretation and classification of emotions expressed in textual data. Organizations use Sentiment Analysis to gain insights into public sentiment regarding specific words or topics, which can inform their decision-making processes and strategies.

In the following sections, we will embark on the creation of a robust Sentiment Analysis model. This model will be designed to categorize tweets into either a Positive or Negative sentiment category, facilitating sentiment analysis for a range of applications.

Table of Contents

1. Importing dependencies
2. Importing dataset
3. Preprocessing Text
4. Analyzing data
5. Splitting data
6. TF-IDF Vectorizer
7. Transforming Dataset
8. Creating and Evaluating Models
   • BernoulliNB Model
   • LinearSVC Model
   • Logistic Regression Model
9. Saving the Models
10. Using the Model

Importing Dependencies

Before diving into the analysis or modeling, the first crucial step is to import the necessary dependencies. These dependencies are external Python libraries that provide additional functionalities not available in the standard library. Importing them at the beginning of our script ensures that all the required tools are available and prevents runtime errors due to missing modules.

Importing Dataset

The dataset utilized in this project is the renowned Sentiment140 dataset, comprising 1,600,000 tweets extracted via the Twitter API. Each tweet is annotated with a sentiment label (0 for Negative, 4 for Positive), offering a foundation for sentiment detection.

The categorization in the training data is derived from the emojis in the text, which might not always accurately represent the sentiment. Hence, models trained on this dataset could exhibit lower accuracy, as the categorization is not infallible.

The dataset includes the following six fields:

• sentiment: the polarity of the tweet (0 for negative, 4 for positive).
• ids: The ID of the tweet (e.g., 2087).
• date: The date of the tweet (e.g., Sat May 16 23:58:44 UTC 2009).
• flag: The query (lyx). If no query, this value is NO_QUERY.
• user: The user who tweeted (e.g., robotickilldozr).
• text: The content of the tweet (e.g., Lyx is cool).

For our purposes, only the sentiment and text fields are required, hence the other fields will be disregarded. Additionally, the sentiment field will be re-encoded to reflect sentiment more intuitively (0 for Negative, 1 for Positive).

Preprocess Text

Text Preprocessing is traditionally an important step for Natural Language Processing (NLP) tasks. It transforms text into a more digestible form so that machine learning algorithms can perform better.

The Preprocessing steps taken are:

1. Lower Casing: Each text is converted to lowercase.
2. Replacing URLs: Links starting with "http", "https", or "www" are replaced by "URL".
3. Replacing Emojis: Replace emojis by using a pre-defined dictionary containing emojis along with their meaning. (e.g., ":)" to "EMOJIsmile")
4. Replacing Usernames: Replace @Usernames with word "USER". (e.g., "@Kaggle" to "USER")
5. Removing Non-Alphabets: Replacing characters except Digits and Alphabets with a space.
6. Removing Consecutive letters: 3 or more consecutive letters are replaced by 2 letters. (e.g., "Heyyyy" to "Heyy")
7. Removing Short Words: Words with length less than 2 are removed.
8. Removing Stopwords: Stopwords are the English words which do not add much meaning to a sentence. They can safely be ignored without sacrificing the meaning of the sentence. (e.g., "the", "he", "have")
9. Lemmatizing: Lemmatization is the process of converting a word to its base form. (e.g., "running" to "run")

Analysing the Data

Now we're going to analyse the preprocessed data to gain insights into it. We'll visualize Word Clouds for Positive and Negative tweets from our dataset to discern the most frequent words.

Word-Cloud for Negative tweets

Word-Cloud for Positive tweets

Splitting the Data

The Preprocessed Data is divided into two sets:

• Training Data: The dataset upon which the model would be trained on. Contains 95% of the data.
• Test Data: The dataset against which the model would be tested. Contains 5% of the data.

TF-IDF Vectoriser

TF-IDF (Term Frequency-Inverse Document Frequency) is a statistical measure used to evaluate the importance of a word in the context of a corpus of documents. It reflects how crucial a word is in understanding the essence of the documents.

Consider an example where a collection of students' essays discusses 'Environmental Conservation.' In these essays, common words like 'the' might appear frequently, but they hold little unique information about the topic. In contrast, less frequent but topic-specific words like 'biodiversity,' 'ecosystem,' or 'conservation' offer much more insight into the theme of the essays. TF-IDF weighs these words more heavily, thus enhancing the thematic significance within the dataset.

The TF-IDF Vectoriser transforms a collection of text documents into a matrix of TF-IDF features. This is crucial for algorithms that operate on numerical input rather than raw text. Typically, the vectoriser is trained on the training data (X_train) to capture the vocabulary and IDF of the corpus.

The ngram_range parameter defines the range of word sequences considered. For instance, with an ngram_range of (1,2), the phrase 'climate change' would be treated as a single feature in addition to the individual words 'climate' and 'change'. This allows the model to capture context that single words may not convey.

The max_features parameter limits the number of features to the most frequently occurring words, thus reducing dimensionality and potentially improving model performance. For example, if set to 1000, only the top 1000 words by frequency would be used as features.

Transforming the Dataset

Transforming the X_train and X_test datasets into matrices of TF-IDF Features using the TF-IDF Vectoriser. These datasets will be utilized to train the model and test its performance.

Creating and Evaluating Models

We are crafting three distinct models for our sentiment analysis challenge:

• Bernoulli Naive Bayes (BernoulliNB)
• Linear Support Vector Classification (LinearSVC)
• Logistic Regression (LR)

Given that our dataset is balanced with equal numbers of Positive and Negative predictions, we have selected Accuracy as our evaluation metric. Additionally, we are visualizing the Confusion Matrix to comprehend our model's performance across both classification categories.

Evaluate Model Function

BernoulliNB

LinearSVC

Logistic regresion model

The Logistic Regression Model stands out as the most proficient, attaining nearly 82% accuracy in sentiment classification of tweets. However, it's noteworthy that the BernoulliNB Model is the swiftest in terms of training and prediction, also achieving a commendable 80% accuracy.

Saving the Models

We're employing PICKLE to preserve the Vectoriser and BernoulliNB, Logistic Regression Model for subsequent utilization.

Using the Model

To deploy the model for Sentiment Prediction, we need to import the Vectoriser and LR Model using Pickle.

The vectoriser can be harnessed to transform data into a matrix of TF-IDF Features, while the model can be applied to predict the sentiment of the transformed data. Nevertheless, the text intended for sentiment prediction must undergo preprocessing.