What is this?

Once per day, the headlines from the frontpage of multiple online media outlets are scraped, and sent to machine learning models. The results are displayed in this website, so that anyone can explore them.

This website is an attempt to automate classifying biased and sensationalist reporting. Similar websites such as AllSides and Media Bias/Fact Check are powerful resources, but their classifications are done by humans, and as such don't scale well or are slow to react to editorial changes. Furthermore, they might be subject to human bias (although both the examples above have taken extensive measures to avoid that!).

Why is this important?

Comparing toxicity in headlines between different media outlets is important for several reasons. Firstly, toxic headlines can perpetuate negative stereotypes and fuel prejudice against certain groups of people. By comparing the level of toxicity in headlines across different newspapers, we can identify patterns and trends in the media's portrayal of certain groups and hold them accountable for perpetuating harmful narratives.

Secondly, toxic headlines can have a significant impact on people's mental health and well-being. Constant exposure to negative news and toxic headlines can lead to increased stress, anxiety, and even depression. To make matters worse, media outlets are encouraged to provide negative news due to humans' negativity bias. By comparing the level of toxicity in headlines across different newspapers, we can identify which newspapers are potentially causing harm to their readers and push for more responsible reporting.

Finally, comparing toxicity in headlines can also shed light on the biases and agendas of different newspapers. Some newspapers may have a political or ideological leaning that affects their reporting (see the work of AllSides), while others may prioritize sensationalism and clicks over accuracy and ethics. By comparing toxicity in headlines, we can gain a better understanding of the motives behind different newspapers' reporting and make more informed decisions about which sources to trust.

Methodology

Each website in our list is fetched once per day (at around 12pm UTC), and all the headlines in the frontpage are saved. The headlines then go through our machine learning models: one for toxicity, and one for sentiment analysis.

The toxicity model we're using is Unitary's Detoxify model, which was trained on online comments, and output a score between 0 and 1 for various types of toxicity (see the dataset for more details). In the daily page, the average of these scores are displayed for all the headlines in a particular days. Those numbers are then averaged over a time period and displayed in the main page.

The sentiment analysis model we're using is CardiffNLP's twitter-XLM-roBERTa-base for Sentiment Analysis , which was trained on almost 200 million tweets. The model outputs for each headline one of "positive", "neutral", or "negative". In the daily page, you can find the percentage of "positive"/"neutral"/"negative" headlines for that particular day. Those percentages are then averaged over a time period and displayed in the main page.

To collect the list of media outlets, we started by looking at the most popular US online media. Some popular examples were skipped for now due to being harder to parse the website, but are planned to be included in the future. Other countries and languages are also planned for the future. Please see the contributing section if you want to help!

Limitations

The biggest limitation is that, while this website is meant to automatically rank online newspapers based on toxicity/negativity, those attributes are not objective. The Machine Learning models used here are likely to have similar biases as humans, but due to the subjectivity of the topic, most likely will never agree 100% with you. Therefore, these numbers should never be used as the definite truth, but only as one possible indicator.

The other main issue is that the models used were not trained on headlines. The toxicity model was trained on comments from social media, and the sentiment analysis model was trained on tweets. The text in social media posts is very different to that found in headlines, and as such the models are likely less accurate than when used for social media comments. But again, the models seem to be good enough in the headlines domain to be useful, even if not perfect.

Contributing

This project is completely open source and is hosted on GitHub: https://github.com/dcferreira/toxic-news

If you want to help, these are the most pressing topics at the moment:

Data

If you're interested in the data we've collected for this project (namely headlines from many newspapers, with scores from our ML models), please reach out to Daniel. We're happy to share the raw data if it's useful for anyone else!