Methodology

Objective

The primary aim of our methodology is to systematically evaluate the ideological leanings and factual accuracy of media and information outlets. This is achieved through a multi-faceted approach that incorporates both quantitative metrics and qualitative assessments in accordance with our rigorously defined criteria.

Understanding Bias and Factual Reporting

While the concept of bias is inherently subjective and lacks a universally accepted scientific formula, our methodology employs a series of objective indicators to approximate it. We utilize a visual representation—a yellow dot on a scale—to signify the extent of bias for each evaluated source. This scale is accompanied by a “Detailed Report” section which elaborates on the source’s characteristics and the basis for its bias rating.

Our bias assessment encompasses various dimensions, including political orientation, factual integrity, and the utilization of credible, verifiable sources. It’s crucial to note that our bias scale is calibrated to the political spectrum of the United States, which may not align with the political landscapes of other nations.

Here is a look at some example scales from different media sources:

For example, CNN looks like this:

 

Fox News looks like this:

Reuters looks like this:

Scoring Mechanism

The placement of the yellow dot is determined through a composite score derived from four distinct categories: Biased Wording/Headlines, Factual/Sourcing, Story Choices, and Political Affiliation. Each category is rated on a scale of 0 to 10, with 0 indicating a lack of bias and 10 representing extreme bias. The average of these four scores is then plotted on the scale to indicate the source’s overall Left-Right bias. Scoring is as follows:

0 – 2 = Least Biased
2 – 5 = Left/Right Center Bias
5 – 8 = Left/Right Bias
8 – 10 = Extreme Bias

The categories are as follows:

1. Biased Wording/Headlines- Does the source use loaded words to convey emotion to sway the reader. Do headlines match the story?
2. Factual/Sourcing- Does the source report factually and back up claims with well-sourced evidence.
3. Story Choices: Does the source report news from both sides, or do they only publish one side.
4. Political Affiliation: How strongly does the source endorse a particular political ideology? Who do the owners support or donate to?

Here is an example of how CNN scored and why they were placed in the middle of Left Bias:

Biased Wording = 4 (CNN uses moderately biased words that favor the left and headlines typically match the story)
Factual/Sourcing = 4 (CNN has failed fact checks and sometimes omits critical information from stories to favor their perspective.)
Story Choices/Editorial = 9 (CNN almost always favors pro-liberal stories and publishes negative conservative stories)
Political Affiliation = 5 (CNN’s ownership owns other left-leaning outlets and favors Democratic Candidates)

Total = 22
Average 22/4 = 5.50
5.50 = Moderate Left Bias

Please see Left vs. Right Bias: How we rate the bias of media sources

Comprehensive Evaluation

For a thorough evaluation, we review a minimum of 10 headlines and 5 news stories from each source. Our methodology employs a variety of search techniques to ensure a comprehensive understanding of the source’s political affiliation and ideological leanings. This process can be time-consuming or very simple, depending on the source.

Factual Reporting Ratings

We also assign a “Factual Reporting” rating based on the Factual/Sourcing score. This rating ranges from “Very High” to “Very Low” and serves as an indicator of the source’s reliability and commitment to factual accuracy.

VERY HIGH (Score: 0)

A source with a “Very High” rating is consistently factual, relies on credible information, promptly corrects errors, and has never failed any fact checks in news reporting or opinion pieces.

HIGH (Score: 1-2)

A “High” rating indicates the source is mostly factual and uses mostly credible, low-biased or high-factual sources. They correct errors quickly and have failed only one news fact check and up to two op-ed fact checks. A source can still earn this rating if they produce a high volume of content or if some failures are based on opinion pieces. If a source has never been third-party fact-checked but is transparent and unbiased, it can also earn a “High” rating.

MOSTLY FACTUAL (Score: 3-4)

A “Mostly Factual” source is generally accurate but may have a few uncorrected fact-check failures. They can fail up to three op-ed fact checks, especially if they are a high-volume site. While they may use biased sources occasionally, they mostly link to factual content. These sources are usually pro-science but may sometimes use misleading wording or offer alternative viewpoints. They are reasonably transparent and are trustworthy most of the time, but caution is advised. Finally, a source that is entirely one-sided may be deemed mostly factual despite a clean fact check record, proper sourcing, and transparency as they omit too much context that may be misleading.

MIXED (Score: 5-6)

A “Mixed” factual rating indicates that a source may rely on improper sourcing or link to other biased or mixed-factual sources. Such sources often have multiple failed fact checks and do not correct false information. Sources that have not failed fact checks may also be labeled as mixed due to a lack of transparency (see more below), including the absence of a disclosed mission statement or ownership details. Furthermore, sources that reject established scientific consensus (or promote pseudoscience) on issues such as climate change or vaccines will receive a “Mixed” rating or lower. Additionally, sources identified as overt propaganda or designated as hate groups by reputable third-party evaluators cannot score higher than a “Mixed” factual rating due to their inherent bias and potential dissemination of misleading information. A highly biased, one-sided source that exhibits any one of the above will be deemed mixed based on omitting important information that provides context.

LOW (Score: 7-9)

A “Low” rating indicates the source is often unreliable and should be fact-checked for fake news, conspiracy theories, and propaganda.

VERY LOW (Score: 10)

A “Very Low” rating means the source is almost always unreliable and should always be fact-checked for intentional misinformation.

Fact-Check Verification

Our methodology incorporates findings from credible fact-checkers who are affiliated with the International Fact-Checking Network (IFCN). Only fact checks from the last five years are considered, and any corrected fact checks do not negatively impact the source’s rating.

Transparency and Accountability

We adhere to the highest standards of transparency and accountability, providing detailed explanations of our funding, organizational structure, and methodology. We are committed to making prompt and transparent corrections in accordance with our corrections policy.

Lack of Transparency

A source is considered to lack transparency if it fails to provide an ‘About’ page or a clear description of its mission. Transparency is further compromised if the ownership of the source is not openly disclosed, including the identification of the parent company and key individuals involved. Additionally, the absence of information about major donors, funding sources, or general revenue generation methods contributes to this lack of transparency. It is essential for the source to at least disclose the country, state, or city of operation and the name of the person responsible (such as the editor). While providing a physical address is not mandatory, meeting some of these transparency criteria is important. Inadequate transparency typically results in the source’s factual reporting rating being reduced by one or two levels, depending on the extent of the shortfall.

Questionable Sources

Questionable sources display extreme bias, propaganda, unreliable sourcing, or a lack of transparency. They may also engage in disseminating fake news for profit or influence. Such sources are generally unreliable and require fact-checking on an article-by-article basis. A source lacking transparency in mission, ownership, or authorship is automatically categorized as questionable. Additionally, sources from countries with significant government censorship are also deemed questionable.

Note: The term “fake news” is specifically used for entirely hoax-based websites. Hate groups are also included in this category. Sources with low or very low factual reporting are typically listed as questionable, as are factually mixed sources that engage in conspiracy theories, pseudoscience, or hate speech.

Conspiracy/Pseudoscience

This category is for sources that disseminate unverified information related to known conspiracies or pseudoscientific claims. For instance, sources denying human-influenced climate change or promoting anti-vaccination stances are labeled as pseudoscience. According to the Stanford Encyclopedia of Philosophy on Science and Pseudoscience, there’s a consensus among knowledge disciplines that certain topics, like creationism and climate change denial, are pseudosciences. To be included in this category, a source’s primary focus must be on conspiracies or pseudoscience.

See our Pseudoscience Dictionary for more.

Traffic Estimates

Media Bias Fact Check pulls page views data from Similar Web to determine the amount of traffic a source receives. We also factor in subscribers for print media and market size for TV/Radio. If data is not available, we use the best estimate. Our categories are as follows: 

Minimal Traffic: Under 250 K page views/print/media market viewers per month
Medium Traffic: 250K to 3 million page views/print/media market viewers per month
High Traffic: Over 3 million page views/print/media market viewers per month

MBFC Credibility Rating

The credibility of a media source is assessed using a 10-point scale. The formula prioritizes Factual Reporting, followed by Bias, and then Traffic/Longevity. Here’s how the scoring works:

Scoring Categories:

• Factual Reporting:
  • Very High: 4 points
  • High: 3 points
  • Mostly Factual: 2 points
  • Mixed: 1 point
  • Low: 0 points
• Bias:
  • Least Biased/Pro-Science: 3 points
  • Right-Center or Left-Center: 2 points
  • Left or Right: 1 point
  • Questionable/Conspiracy/Pseudoscience: 0 points
• Traffic/Longevity:
  • High Traffic: 2 points
  • Medium Traffic: 1 point
  • Minimal Traffic: 0 points
  • Bonus: 1 point for sources existing for 10 years or more
• Press Freedom (applicable for sources from countries with significant censorship or government control):
  • Limited Freedom: -1 point
  • Total Oppression: -2 points

Credibility Levels:

• High Credibility: A score of 6 or above.
• Medium Credibility: A score between 3-5 points. Sources lacking an ‘About’ page or ownership information are automatically rated as Medium Credibility.
• Low Credibility: A score of 0-2 points. Sources rated as Questionable, Conspiracy, or Pseudoscience are automatically classified as Low Credibility.

Example Scores:

Reuters 4+3+2+1=10 High
Bangkok Post: 1+2+2+1 – 1=5 Medium
The Daily Defender 1+1+0+0=2 Low

Pollster Rating Methodology:

Bias Assessment:

• • Derived from FiveThirtyEight’s Mean-Reverted Bias.
  • Bias is categorized as follows:
    • 0 to +0.9: Least Biased
    • 1.0 to 2.4: Left or Right-Center Bias
    • 2.5 or higher: Left or Right Bias

Factual Reporting:

• Measured using the “538 overall star ratings” metric.
• Factual reporting is rated as:
  • 3.0: Very High Factual
  • 2.0 – 2.9: High Factual
  • 1.5 – 1.9: Mostly Factual
  • 0.5 – 1.4: Mixed Factual
  • 0.4 or less: Low Factual

News Reporting Consideration:

• • • For pollsters that also report news on their website, their news content is factored into the overall rating.
    • This includes assessing the quality, bias, and accuracy of their news reporting.

Country Freedom Rating

We assess a country’s level of freedom by averaging scores from Reporters Without Borders (RSF) and Freedom House’s Freedom in the World Index. Both use a 0-100 scale, where 100 represents maximum freedom, and 0 indicates no freedom. RSF focuses on press freedom, while Freedom House evaluates democratic governance and various personal freedoms. The combined scores offer a comprehensive view of a country’s freedom status.

Scoring Breakdown:

• Excellent Freedom: 100-90 points
• Mostly Free: 89-70 points
• Moderate Freedom: 69-50 points
• Limited Freedom: 49-25 points
• Total Oppression: 24-0 points

If a country hasn’t been rated by either RSF or Freedom House, we turn to alternative credible sources like Unesco.org, Statista.org, or BBC Country Profiles to estimate its freedom level.

Country Government Bias Rating

Each country profile has a bias rating arrow like our source pages. Country government bias is determined by examining the head of state (President, Prime Minister, etc) and their political party positions. We also factor in the economic systems of each country, such as capitalism, socialism, communism, and democratic socialism. The right-left designation is mostly determined by that country’s economic policy and system with less consideration for social agenda.

Fact-Checking

Media Bias/Fact Check rarely conducts original fact checks as many other sources are faster and do a better job. We primarily rely on fact-checkers affiliated with the International Fact-Checking Network ( IFCN). Below is their code of principles.

Media Bias/Fact Check (MBFC News) adheres to the International Fact-Checking Network Fact-checkers’ Code of Principles.  The Poynter Institute developed these principles to promote excellence and standardization in Fact-Checking.

MBFC News strictly adheres to the following principles for all fact checks:

1. A COMMITMENT TO NONPARTISANSHIP AND FAIRNESS
   We fact-check claims using the same standard for every fact check. We do not concentrate our fact-checking on any one side. We follow the same process for every fact check and let the evidence dictate our conclusions. We do not advocate or take policy positions on the issues we fact-check.
2. A COMMITMENT TO TRANSPARENCY OF SOURCES
   We want our readers to be able to verify our findings themselves. We provide all sources in enough detail that readers can replicate our work, except in cases where a source’s personal security could be compromised. In such cases, we provide as much detail as possible.
3. A COMMITMENT TO TRANSPARENCY OF FUNDING & ORGANIZATION
   We are transparent about our funding sources. If we accept funding from other organizations, we ensure that funders do not influence the conclusions we reach in our reports. We detail all key figures’ professional backgrounds in our organization and explain our organizational structure and legal status. We clearly indicate a way for readers to communicate with us.
4. A COMMITMENT TO TRANSPARENCY OF METHODOLOGY
   We explain the methodology we use to select, research, write, edit, publish and correct our fact checks. We encourage readers to send us claims to fact-check and are transparent on why and how we fact-check.
5. A COMMITMENT TO OPEN AND HONEST CORRECTIONS
   We publish our corrections policy and follow it scrupulously. We correct clearly and transparently in line with our corrections policy, seeking so far as possible to ensure that readers see the corrected version.

Source: Poynter

References

Alexander Dyck & Natalya Volchkova & Luigi Zingales, 2008. “The Corporate Governance Role of the Media: Evidence from Russia,” Journal of Finance, American Finance Association, vol. 63(3), pages 1093-1135, 06.

Baron, David P. “Persistent media bias.” Journal of Public Economics 90.1 (2006): 1-36.

Bolinger, Dwight. Language-the loaded weapon: the use and abuse of language today. Routledge, 2014.

Chun-Fang Chiang & Brian Knight, 2011. “Media Bias and Influence: Evidence from Newspaper Endorsements,” Review of Economic Studies, Oxford University Press, vol. 78(3), pages 795-820.

DellaVigna, Stefano and Ethan Kaplan. “The Fox News Effect: Media Bias and Voting.” Quarterly Journal of Economics 122 (August 2007): 1187-1234.

Entman, R. M. (2007), Framing Bias: Media in the Distribution of Power. Journal of Communication, 57: 163–173. doi:10.1111/j.1460-2466.2006.00336.x

Eveland, W. P. and Shah, D. V. (2003), The Impact of Individual and Interpersonal Factors on Perceived News Media Bias. Political Psychology, 24: 101–117. doi:10.1111/0162-895X.00318

Farhi, Paul. How Biased is the Media Really. 27 April 2012. 20 November 2012. <http://www.washingtonpost.com/lifestyle/style/how-biased-is-the-media-really/2012/04/27/gIQA9jYLmT_story.html&gt;.

Gentzkow, Matthew and Jesse M. Shapiro. “Media Bias And Reputation,” Journal of Political Economy, 2006, v114(2,Apr), 280-316.

“How To Detect Bias In News Media.” FAIR. N.p., 2012. Web. 15 Feb. 2016.

Matthews, Jack. “The effect of loaded language on audience comprehension of speeches.” Communications Monographs 14.1-2 (1947): 176-186.

Morris, J. S. (2007), Slanted Objectivity? Perceived Media Bias, Cable News Exposure, and Political Attitudes^*. Social Science Quarterly, 88: 707–728. doi:10.1111/j.1540-6237.2007.00479.x

Nie, N. H., Miller, III, D. W., Golde, S., Butler, D. M. and Winneg, K. (2010), The World Wide Web and the U.S. Political News Market. American Journal of Political Science, 54: 428–439. doi:10.1111/j.1540-5907.2010.00439.x

Puglisi R, Snyder JM. Newspaper Coverage of Political Scandals. Journal of Politics. 2011;73(3):931-950.

Russell, Nick. “Morals and the Media Ethics in Canadien Journalism: Second Edition.” Vancouver: UBC Press, 2006.

S.I. Hayakawa, Alan Hayakawa. “Language in Thought and Action: Fifth Edition.” New York: Harcourt Brace Jovanovich,

Tewksbury, D., Jensen, J. and Coe, K. (2011), Video News Releases and the Public: The Impact of Source Labeling on the Perceived Credibility of Television News. Journal of Communication, 61: 328–348. doi:10.1111/j.1460-2466.2011.01542.x

The Quarterly Journal of Economics (2005) 120 (4): 1191-1237.doi: 10.1162/003355305775097542

Disclaimer: The methodology used by Media Bias Fact Check is our own.  It is not a tested scientific method.  It is meant as a simple guide for people to get an idea of a source’s bias.  Media Bias Fact Check will always review and change any factual errors when brought to our attention.  We make every effort to be as factual as possible.  Our goal is to have MBFC rated as least biased by our own criteria.

Last Updated on February 9, 2024 by Media Bias Fact Check