Limitations of Sentiment Analysis for Reputation management

A look at how sentiment analysis is inadequate for media reputation management.

This is the third post in a series of blog posts about AI techniques for media reputation management. See the previous post in the series here. In the last post we discussed sentiment analysis and how it can assist you with media reputation management. In this post we will look at the limitations of sentiment analysis in the context of reputation management.

Because of the abundance of review data available online, most machine learning systems for sentiment analysis are trained on language data from this domain. Consequently, most software tools for media monitoring and reputation management provide automatic sentiment analysis of brand mentions using machine models trained on review data.

These sentiment analysis systems have a number of huge shortcomings as a tool for reputation management however. Firstly, subjective opinions and feelings expressed towards your brand represent only a fraction of the information your care about when managing the reputation of your organization. Very often, online mentions of your brand will have an impact on your reputation regardless of whether any sentiment is expressed in the text.

Sentiment Neutral Statements with Negative or Positive Reputational Impact

Consider for example the following sentence taken from a news article about the Deepwater Horizon oil spill in 2010:

Lingering oil residues have altered the basic building blocks of life in the ocean by reducing biodiversity in sites closest to the spill, which occurred when a BP drilling rig exploded in April 2010, killing 11 workers and spewing about 4m barrels of oil into the Gulf.

The sentiment of the above phrase is close to neutral: no positive or negative feelings or opinions are directly expressed by the author or any other sources, it is simply a statement of facts. However, the reported facts clearly have a negative impact on the reputation of British Petroleum (BP), and obviously it would be extremely useful if their reputation management software platform could alert them to such mentions. Traditional sentiment analysis is fundamentally useless in this situation.

Incorrectly Targeted Sentiment

As an illustration, consider the following phrase taken from a news article about the incident in which a United Airlines passenger was forcibly removed from an overbooked flight in 2017:

Since the target of reviewer sentiment is so easily identified in a review, most machine learning systems trained on data from this domain make no special effort to distinguish between different targets of sentiment. For this reason, the analysis provided by these models when applied to data outside of the review domain, for example news or social media data, is not granular enough to be useful for reputation management.

After a further uproar, (even President Trump described United’s treatment of [the passenger] as “horrible”), United issued what was perceived as a more sincere apology, with [United’s CEO] promising such an incident would never happen again on United.

Both United Airlines and Donald Trump are mentioned in this sentence. A sentiment analysis system that does not distinguish between sentiment expressed towards different targets would most likely classify the sentence as expressing negative sentiment because of the presence of words such as horrible and uproar. This is correct if the target is considered to be United Airlines, but not if the target is considered to be Donald Trump. As a consequence, a reputation management system that assigns sentiment irrespective of which target is considered would give misleading information about Donald Trump in this case.

Sentiment analysis systems trained on review data are often much less accurate when applied to data from other domains such as news or social media because of the differences in how people express themselves in these domains: journalists generally do not express sentiment in the same way a reviewer does, which are both different from how a poster on social media expresses sentiment. As a consequence, the patterns a machine learning system trained on review data has learnt to recognize as evidence for predicting sentiment in this domain will generally not be useful for predicting sentiment in other domains.

Review Language is Dissimilar to Social Media and News Language

Even more problematically, most online review data is in English. For global organizations, successful reputation management requires monitoring media sources in many languages. In order to use sentiment analysis systems trained on English data exclusively, special steps must be taken that either involve costly translation of all relevant news articles and social media posts, or complex, state-of-the-art methods that allow the trained system to transfer what it has learned from one language to another. As a consequence, most reputation management systems support sentiment analysis of sources only in a small number of languages.

Conclusion

In this post we have looked at the limits of sentiment analysis for media reputation management. We have seen that sentiment analysis produces only a subset of the information you need to effectively manage your brands’ media reputation. We also discussed how many sentiment systems do not differentiate between different targets of sentiment within the same text, and that sentiment analysis systems that are based on english review data often produces low quality analysis when applied to news or social media data.

In the next post we will discuss how Hypefactors is working to develop solutions that overcomes these limitations.