Objective: To determine the bias of media and information sources and the level of overall factual reporting through a combination of objective measures and subjective analysis through the use of our stated methodology.
When determining bias, there isn’t any true scientific formula that is 100% objective. There are objective measures that can be calculated, but ultimately there will be some subjective judgment to determine these. We have put up a scale with a yellow dot on each page that shows the degree of bias for each source. Each page also has a “detailed report” section that gives some details about the source and explains their bias.
When calculating bias, we are looking at political bias, how factual the information is, and links to credible, verifiable sources. Therefore, the yellow dot may indicate political bias or how factual a source is, or in many cases, both. It is important to note that our bias scale is based on the USA political scale, which may differ from other countries. For example, the Democratic Party of the USA is considered centrist or even right-center in many countries worldwide; however, in the USA, they are considered Left-Center. Please keep this in mind if our ratings seem off in your native country. (08/08/2022)
Here is a look at some example scales from different media sources:
For example, CNN looks like this:
Fox News looks like this:
Reuters looks like this:
The method for placement of the yellow dot is determined by ranking bias in four different categories. The source is rated on a 0-10 scale in each category, with 0 meaning without bias and 10 being the maximum bias(worst). These four numbers are then added up and divided by 4. This 0-10 number is then placed on the line according to their Left or Right bias. Scoring is as follows:
0 – 2 = Least Biased
2 – 5 = Left/Right Center Bias
5 – 8 = Left/Right Bias
8 – 10 = Extreme Bias
The categories are as follows:
- Biased Wording/Headlines- Does the source use loaded words to convey emotion to sway the reader. Do headlines match the story?
- Factual/Sourcing- Does the source report factually and back up claims with well-sourced evidence.
- Story Choices: Does the source report news from both sides, or do they only publish one side.
- Political Affiliation: How strongly does the source endorse a particular political ideology? Who do the owners support or donate to?
Here is an example of how CNN scored and why they were placed in the middle of Left Bias:
Biased Wording = 4 (CNN uses moderate biased words that favor liberals, and headlines typically match the story)
Factual/Sourcing = 5 (CNN has failed fact checks and sometimes omits critical information from stories to favor their perspective)
Story Choices/Editorial = 9 (CNN almost always favors pro-liberal stories and publishes negative conservative stories)
Political Affiliation = 5 (CNN’s ownership owns other left-leaning outlets and favors Democratic Candidates)
Total = 23
Average 23/4 = 5.75
5.75 = Moderate Left Bias
Please see: Left vs. Right Bias: How we rate the bias of media sources
Each source has a “Factual Reporting” rating based on their Factual/Sourcing score above. The ratings are as follows.
Factual Reporting: VERY HIGH = a score of 0, which means the source is always factual, sources to credible information, and makes immediate corrections to incorrect information, and has never failed a fact check in either news reporting or op-eds. (9/30/2020)
Factual Reporting: HIGH = a score of 1 – 2, which means the source is almost always factual, sources from mostly credible low biased or high factual information and makes immediate corrections to incorrect information, has failed only 1 news reporting fact check, 2 or less op-ed fact checks and uses reasonable language that retains context. A high-rated source may fail more than one fact check if they produce a high volume of content or if some are based on op-eds. Finally, if a source has never been fact-checked by a third party but uses credible sources, is transparent, not overtly biased, and makes corrections, it will earn a high factual rating. (08/09/2022)
Factual Reporting: MOSTLY FACTUAL = a score of 3 – 4, which means the source is usually factual but may have failed a fact check or two that was not properly corrected promptly. Further, they may fail up to 3 op-ed fact checks (less if a low-volume site). They may occasionally use biased sources but still link to factual content the majority of the time. These sources are generally pro-science but may sometimes be misleading in wording and may infrequently offer alternative viewpoints to the consensus. These sources are reasonably transparent and offer a mission statement and ownership. In general, these sources are trustworthy most of the time, but some caution is needed. A source that outright rejects scientific consensus in any area cannot be considered mostly factual despite clean fact-check records. (9/30/2020)
Factual Reporting: MIXED = a score of 5 – 6, which means the source does not always use proper sourcing or sources to other biased/mixed factual sources. Mixed sources will have failed one or many fact checks and do not correct false or misleading information. Further, any source that does not disclose either a mission statement or ownership information will automatically be deemed Mixed, as will a source that utilizes extremely loaded language that alters the context of facts, even if properly sourced and has not failed a fact check. Lastly, any source that does not support science’s consensus on such topics as Climate Change, GMO, Vaccinations, Evolution, or any other will automatically be rated Mixed for factual reporting if one or more Mixed criteria are met above. Further, a source that has failed more than 5 fact checks can remain on the Mixed list so long as they are a high-volume source. (9/30/2020)
Factual Reporting: LOW = a score of 7 – 9, which means the source rarely uses credible sources and is not trustworthy for reliable information. These are the sources that need to be fact-checked for intentional fake news, conspiracy, and propaganda. (12/10/2019)
Factual Reporting: VERY LOW = a score of 10, which means the source rarely uses credible sources and is not trustworthy for reliable information at all. These sources always need to be fact-checked for intentional fake news, conspiracy, and propaganda. (12/10/2019)
For each source, a minimum of 10 headlines are reviewed and a minimum of 5 news stories reviewed. We first review news reporting and follow that up by looking at editorial and opinion pieces. If there is still no clear evidence, we will use searching methods for articles on the site such as “Republican,” “Democrat,” “Liberal,” “Conservative,” “Trump,” “Clinton,” etc., until we are sure of political affiliation. This process can be time-consuming or very simple, depending on the source.
Failed Fact Checks: All failed fact checks must come from credible fact-checkers and indicate with evidence that claims are false or mostly false. Unproven or unsupported claims may be listed but do not count in factual grading. Furthermore, only failed fact checks in the last five years are counted; therefore, old failed fact checks may drop off during the re-review process, and a source’s rating may improve. Finally, any failed fact check that is promptly corrected will not count against the source’s rating.
To understand the ratings, it is important to understand the terminology. We look at all of these when researching a source:
Bias by Omission: leaving one side out of an article or a series of articles over a period of time; ignoring facts that tend to disprove liberal or conservative claims or that support liberal or conservative beliefs.
Bias by Labeling: Bias by labeling comes in two forms. The first is the tagging of conservative politicians and groups with extreme labels while leaving liberal politicians and groups unlabeled or with more mild labels, or vice versa. The second kind of bias by labeling occurs when a reporter not only fails to identify a liberal as a liberal or a conservative as a conservative but describes the person or group with positive labels, such as “an expert” or “independent consumer group.”
Bias by Placement: Whether on a website (or newspaper) or in an article, a story or event is printed; a pattern of placing news stories to downplay information supportive of either conservative views or liberal views.
Bias by Selection of Sources: including more sources that support one view over another.
Bias by Spin: is a reporter’s subjective comments about objective facts; makes one side’s ideological perspective look better than another.
Bias by Story Selection: a pattern of highlighting news stories that coincide with the agenda of either the Left or the Right while ignoring stories that coincide with the opposing view.
Confirmation Bias: also called confirmatory bias or my side bias, is the tendency to search for, interpret, favor, and recall information in a way that confirms one’s beliefs or hypotheses while giving disproportionately less consideration to alternative possibilities.
Connotation: the emotional and imaginative association surrounding a word that can be either positive or negative.
Denotation: the strict dictionary meaning of the word.
Loaded Language (Words): (also known as loaded terms or emotive language) is wording that attempts to influence an audience by appealing to emotion or stereotypes. Such wording is also known as high-inference language or language persuasive techniques.
Purr Words: words used to describe something that is favored or loved.
Snarl Words: words used when describing something that a person is against or hates.
Other factors to look for:
Do the headlines and stories match?
Are important stories featured prominently?
Does the story offer an alternative point of view?
Consider the source!
Questionable Sources
A questionable source exhibits any of the following: extreme bias, overt propaganda, poor or no sourcing of credible information, a complete lack of transparency, and/or is fake news. Fake News is the deliberate attempt to publish hoaxes and/or disinformation for profit or influence. Sources listed in the Questionable Category may be very untrustworthy and should be fact-checked on a per-article basis. Typically, the number one criterion for this category is how factual their reporting is—the less factual, the more likely to be placed in this category. Further, if a source does not disclose at least one of the following: Mission, Ownership, and Authorship, they will automatically be deemed Questionable due to a lack of transparency, despite how factual their content may be. Finally, sources originating from countries that exert extreme government censorship will automatically be placed on the Questionable list. (This is a work in progress). (12/10/2019)
We may occasionally label a source “fake news,” but this is designated specifically for 100% hoax websites. We also classify “hate groups” in this category. Finally, most Questionable sources will be listed as Low or Very Low in factual reporting; however, factually Mixed sources may also be classified as questionable if they publish conspiracy theories, pseudoscience or hate despite being reasonably factual. (11/29/2022)
Conspiracy/Pseudoscience
The Conspiracy/Pseudoscience designation is reserved for sources that publish unverifiable information related to known conspiracies such as the New World Order, Illuminati, False Flags, Aliens, anti-vaccination propaganda, etc. Pseudoscience is determined by publishing unverified health and scientific claims. For example, sources that promote human-influenced climate change denial or take anti-vaccination positions will be classified as pseudoscience. According to the Stanford Encyclopedia of Philosophy on Science and Pseudoscience, “Philosophers and other theoreticians of science differ widely in their views on what science is. Nevertheless, there is virtual unanimity in the community of knowledge disciplines on most particular issues of demarcation. There is a widespread agreement, for instance, that creationism, astrology, homeopathy, Kirlian photography, dowsing, ufology, ancient astronaut theory, Holocaust denialism, Velikovskian catastrophism, and climate change denialism are pseudosciences.“ To be classified in this group, the source’s central theme must revolve around conspiracies or pseudoscience. See out Pseudoscience Dictionary for more. (9/30/2020)
Traffic Estimates
Media Bias Fact Check pulls page views data from Similar Web to determine the amount of traffic a source receives. We also factor in subscribers for print media and market size for TV/Radio. If data is not available, we use the best estimate. Our categories are as follows: (4/06/2022)
Minimal Traffic: Under 150 K page views/print/media market viewers per month
Medium Traffic: 150K to 2.5 million page views/print/media market viewers per month
High Traffic: Over 2.5 million page views/print/media market viewers per month
MBFC Credibility Rating
This rating is calculated mathematically on a 10 point scale. The formula weighs Factual Reporting the highest followed by Bias and then Traffic/Longevity. It is calculated as follows:
Factual Reporting
Very High: 4 points
High: 3 points
Mostly Factual 2 points
Mixed: 1 Point
Low: 0 points
Bias
Least Biased/Pro-Science: 3 points
Right-Center or Left-Center: 2 points
Left or Right: 1 Point
Questionable/Conspiracy Pseudoscience: 0 points
Traffic/Longevity
High Traffic: 2 points
Medium Traffic: 1 point
Minimal Traffic: 0 points
10 years or more in existence: 1 bonus point
Press Freedom (From a Country with significant censorship or government control)
Limited Freedom: -1 point
Total Oppression: -2 points
High Credibility: A score of 6 or above.
Medium Credibility: A score of 3-5 points. Any source that does not provide an about page or list ownership will be listed as Medium Credibility regardless of how factual their content is.
Low Credibility: A score of 0-2 points or any source rated Questionable, Conspiracy or Pseudoscience will be rated Low Credibility by default. (12/26/2022)
Example Scores:
Associated Press 4+3+2+1=10 High
CNN: 1+1+2+1=5 Medium
The Daily Defender 1+1+0+0=2 Low
*We are in the process of updating credibility scores, so it is possible some may not currently match our scoring system. This is a work in progress. (03/03/2023)
Country Freedom Rating
We calculate the overall freedom levels of countries by averaging the scores from Reporter Without Borders (RSF) and Freedom House’s yearly Freedom in the World Index. Both sources rate the level of freedom on a 0 to 100-point scale, with 100 points being the maximum freedom and 0 being none. By combining these ratings, we come up with a unique picture of each country’s freedom as RSF focuses on Press Freedom, and in contrast, FreedomHouse focuses on the strengths of democracy, free elections, and social, and personal freedoms. This gives a more complete picture than either source on its own. Our ratings are based on the following scores:
100 – 90 = Excellent Freedom
89 – 70 = Mostly Free
69 – 50 = Moderate Freedom
49 – 25= Limited Freedom
24 – 0 = Total Oppression
If a country has not been rated by either source, we use other sources such as Unesco.org, Statista.org, BBC Country Profiles, or other credible sources to estimate the level of freedom.
Country Government Bias Rating
Each country profile has a bias rating arrow like our source pages. Country government bias is determined by examining the head of state (President, Prime Minister, etc) and their political party positions. We also factor in the economic systems of each country such as capitalism, socialism, communism, and democratic socialism. The right-left designation is mostly determined by that country’s economic policy and system with less consideration for social agenda. (04/24/2022)
Fact-Checking
Media Bias/Fact Check rarely conducts original fact checks as many other sources are faster and do a better job. We primarily rely on fact-checkers affiliated with the International Fact-Checking Network ( IFCN). Below is their code of principles. (9/30/2020)
Media Bias/Fact Check (MBFC News) adheres to the International Fact-Checking Network Fact-checkers’ Code of Principles. The Poynter Institute developed these principles to promote excellence and standardization in Fact-Checking.
MBFC News strictly adheres to the following principles for all fact checks:
- A COMMITMENT TO NONPARTISANSHIP AND FAIRNESS
We fact-check claims using the same standard for every fact check. We do not concentrate our fact-checking on any one side. We follow the same process for every fact check and let the evidence dictate our conclusions. We do not advocate or take policy positions on the issues we fact-check. - A COMMITMENT TO TRANSPARENCY OF SOURCES
We want our readers to be able to verify our findings themselves. We provide all sources in enough detail that readers can replicate our work, except in cases where a source’s personal security could be compromised. In such cases, we provide as much detail as possible. - A COMMITMENT TO TRANSPARENCY OF FUNDING & ORGANIZATION
We are transparent about our funding sources. If we accept funding from other organizations, we ensure that funders do not influence the conclusions we reach in our reports. We detail all key figures’ professional backgrounds in our organization and explain our organizational structure and legal status. We clearly indicate a way for readers to communicate with us. - A COMMITMENT TO TRANSPARENCY OF METHODOLOGY
We explain the methodology we use to select, research, write, edit, publish and correct our fact checks. We encourage readers to send us claims to fact-check and are transparent on why and how we fact-check. - A COMMITMENT TO OPEN AND HONEST CORRECTIONS
We publish our corrections policy and follow it scrupulously. We correct clearly and transparently in line with our corrections policy, seeking so far as possible to ensure that readers see the corrected version.
Source: Poynter
References
Alexander Dyck & Natalya Volchkova & Luigi Zingales, 2008. “The Corporate Governance Role of the Media: Evidence from Russia,” Journal of Finance, American Finance Association, vol. 63(3), pages 1093-1135, 06.
Baron, David P. “Persistent media bias.” Journal of Public Economics 90.1 (2006): 1-36.
Bolinger, Dwight. Language-the loaded weapon: the use and abuse of language today. Routledge, 2014.
Chun-Fang Chiang & Brian Knight, 2011. “Media Bias and Influence: Evidence from Newspaper Endorsements,” Review of Economic Studies, Oxford University Press, vol. 78(3), pages 795-820.
DellaVigna, Stefano and Ethan Kaplan. “The Fox News Effect: Media Bias and Voting.” Quarterly Journal of Economics 122 (August 2007): 1187-1234.
Entman, R. M. (2007), Framing Bias: Media in the Distribution of Power. Journal of Communication, 57: 163–173. doi:10.1111/j.1460-2466.2006.00336.x
Eveland, W. P. and Shah, D. V. (2003), The Impact of Individual and Interpersonal Factors on Perceived News Media Bias. Political Psychology, 24: 101–117. doi:10.1111/0162-895X.00318
Farhi, Paul. How Biased is the Media Really. 27 April 2012. 20 November 2012. <http://www.washingtonpost.com/lifestyle/style/how-biased-is-the-media-really/2012/04/27/gIQA9jYLmT_story.html>.
Gentzkow, Matthew and Jesse M. Shapiro. “Media Bias And Reputation,” Journal of Political Economy, 2006, v114(2,Apr), 280-316.
“How To Detect Bias In News Media.” FAIR. N.p., 2012. Web. 15 Feb. 2016.
Matthews, Jack. “The effect of loaded language on audience comprehension of speeches.” Communications Monographs 14.1-2 (1947): 176-186.
Morris, J. S. (2007), Slanted Objectivity? Perceived Media Bias, Cable News Exposure, and Political Attitudes*. Social Science Quarterly, 88: 707–728. doi:10.1111/j.1540-6237.2007.00479.x
Nie, N. H., Miller, III, D. W., Golde, S., Butler, D. M. and Winneg, K. (2010), The World Wide Web and the U.S. Political News Market. American Journal of Political Science, 54: 428–439. doi:10.1111/j.1540-5907.2010.00439.x
Puglisi R, Snyder JM. Newspaper Coverage of Political Scandals. Journal of Politics. 2011;73(3):931-950.
Russell, Nick. “Morals and the Media Ethics in Canadien Journalism: Second Edition.” Vancouver: UBC Press, 2006.
S.I. Hayakawa, Alan Hayakawa. “Language in Thought and Action: Fifth Edition.” New York: Harcourt Brace Jovanovich,
Tewksbury, D., Jensen, J. and Coe, K. (2011), Video News Releases and the Public: The Impact of Source Labeling on the Perceived Credibility of Television News. Journal of Communication, 61: 328–348. doi:10.1111/j.1460-2466.2011.01542.x
The Quarterly Journal of Economics (2005) 120 (4): 1191-1237.doi: 10.1162/003355305775097542
Disclaimer: The methodology used by Media Bias Fact Check is our own. It is not a tested scientific method. It is meant as a simple guide for people to get an idea of a source’s bias. Media Bias Fact Check will always review and change any factual errors when brought to our attention. We make every effort to be as factual as possible. Our goal is to have MBFC rated as least biased by our own criteria.
Last Updated on May 11, 2023 by Media Bias Fact Check
Left vs. Right Bias: How we rate the bias of media sources
