Lexicon-Based Sentiment Analysis: What It Is and How to Run One

You have 14,000 customer comments about six US airlines and a meeting this afternoon where someone will ask how customers feel. Reading them all isn't an option. And if the customer experience team and the marketing team each read a sample and come back with different answers, the meeting turns into a debate about whose reading is right.

Sentiment analysis sorts text into positive, negative, or neutral automatically. Teams run it on reviews, support tickets, survey comments, and social posts. Lexicon-based sentiment analysis is the simplest way to do it: it scores each text by the words it contains, using a dictionary of words with a known sentiment. Every score traces back to specific words, so anyone can see why a comment was labeled negative.

Other approaches include traditional machine learning, deep learning, and large language models (LLMs). This guide explains how the lexicon-based approach works, where it falls short, and how we built a sentiment predictor in Knime that scores more than 14,000 airline tweets.

Lexicon-based sentiment analysis is a natural language processing technique that detects the sentiment of a text using a lexicon: a list of words and phrases labeled by the sentiment they carry. Lexicons are also called sentiment dictionaries, sentiment word lists, or valence dictionaries.

Each word can carry an emotional valence. “Great” has a positive valence and “terrible” has a negative one. In the sentence “Good airlines sometimes have bad days,” a valence dictionary labels “good” as positive, “bad” as negative, and the other words as neutral.

Once every word is labeled, you count the positive and negative words and combine them into a sentiment score. A common formula for the sentiment score (StSc) divides the difference between positive and negative words by their sum:

No model training is involved. The dictionary does the work, so you can score new text as soon as it arrives.

Sentiment lexicons are language-specific. Widely used English lexicons include the MPQA Subjectivity Lexicon, VADER, which was built for social media text, and the NRC Emotion Lexicon, which also offers automatic translations into more than 100 languages.

We built the example as a visual workflow in Knime. Every step, from the raw tweets to the final label, sits on the canvas, where anyone on the team can open it and check how a result came about.

• Download the example workflow.
• The data comes from a Kaggle dataset of more than 14,000 tweets about six US airlines.
• Human contributors labeled each tweet as positive, negative, or neutral. We compare our predictions with these labels to see how well the approach works.

Start by cleaning the data. Duplicate tweets would count the same opinion twice, so remove them with the Duplicate Row Filter node.

Next, convert each tweet and its contributor label into a document with the Strings To Document node. Most text processing nodes in Knime work on the Document type, and you can inspect documents with the Document Viewer node.

The documents now hold everything the analysis needs, so remove the other columns with the Column Filter node.

Now label the words in each document with a sentiment lexicon. We used the English MPQA Subjectivity Lexicon, split into two lists: one of positive words and one of negative words.

Tag each word by comparing it against both lists with two Dictionary Tagger nodes, one per list. Then keep only the tagged words with the Tag Filter node.

Because this example doesn't tag neutral words, the Tag Filter removes every word that isn't positive or negative. The total number of words per tweet in the formula is then the sum of the positive and negative words.

Some formulas also count neutral words, either from a dedicated neutral list or by treating every untagged word as neutral.

Create a bag of words for each tweet with the Bag Of Words Creator node. It lists every word in each document, one word per row.

Count how often each tagged word appears in each tweet with the TF node. It can return absolute counts or counts relative to the length of each document. Tweets are short, so relative counts add little, and we use absolute counts.

Then extract each word's tag with the Tags to String node, and sum the positive and negative counts per tweet with the Pivot node. If a tweet has no positive or negative words, set the missing count to zero with the Missing Value node. To keep the workflow easy to read, we wrapped these steps in a metanode.

Calculate the sentiment score for each tweet with the Math Formula node, using the StSc formula above. Then classify each tweet as positive, negative, or neutral based on its score with the Rule Engine node.

Contributors labeled every tweet as positive, negative, or neutral, so you can compare the predictions with human judgment. The three labels work as classes, which turns the comparison into a classification problem.

Set the column with the contributor labels as the target class with the Category to Class node, then compare it with the predictions in the Scorer node.

The results are modest. The predictor labeled 43% of the tweets correctly. It did worse on neutral tweets, likely because neither the formula nor the lexicon handles neutral words directly. Contributors labeled most tweets as negative or positive, and the predictor did better on negative tweets (F1-score of 53%) than on positive ones (42%).

Large language models can classify sentiment too, and they read context that a lexicon misses, such as sarcasm (“Great, another two-hour delay”) or negation (“not bad”). They come with tradeoffs. An LLM's label doesn't come with a word-by-word explanation, the same prompt can return a different label on a different run, and scoring thousands of comments with a hosted model costs money and sends your text to an external provider.

A lexicon-based predictor works the other way around. Every score traces back to the words that produced it, and the same text always gets the same score. That makes it a useful baseline. Score everything with the lexicon first, then compare its labels with an LLM on a sample. Where they agree, you have a quick result you can explain. Where they disagree, you know which comments need a closer look.

In Knime, both approaches can sit in the same workflow, and you choose which AI provider the prompts go to, including a model that runs locally.

Sentiment scores end up in decisions: which product issue to fix first, which region needs more support staff, whether a campaign landed. When two teams score the same comments and get different numbers, the conversation turns to whose number is right instead of what to do about it.

A lexicon-based predictor built as a visual workflow in Knime shows every step from the raw text to the final label, so every team works from the same logic. Adjust the formula, swap in a different lexicon, or add neutral words, and everyone can see what changed and how the results moved.

Explore the workflows and see how changing the sentiment score formula changes the results.

• Machine Learning for Sentiment Analysis: A Tutorial
• Automate Content Tagging in 4 Steps with AI
• 8 Best Practices for Effective Prompt Engineering
• A Beginner's Guide to Web Scraping