Fake News Detector AI Review

On 1/17/17 the International Business Times reported on a new website and Chrome Extension that will track fake news sources. According the International Business Times the new website will use artificial intelligence (AI) to track fake news sites.  The AI was created by Karan Singhal, a 19 year old computer science major from Stanford University.

According to Singhal, he is employing neural networks, that act as an artificial brain that tracks many factors at a time, then weighs each element and produces a verdict on the validity of a particular website.

Singhal told IBTimes his “algorithmic method sorts over site layout, popularity, writing style, the frequency of telling keywords like “liberal” and “conservative,” among other aspects of a given site. It checks its analysis to a list of known fake and real news websites, and if a website shares the telltale signs of a fake, it’s marked as such.”

When I read about this I was very excited to try it out to see how it compares with our own ratings. Unfortunately, I was terribly disappointed and quite frankly angry.  This AI simply does not work, or perhaps it needs some serious bugs worked out. It certainly does not compare to our methodology. I decided to test it with known fake websites to see the results.  The results are as follows:

Infowars – REAL (Wrong)
The Onion- REAL (Wrong)
Daily Squib – FAKE  (Correct)
Illuminati News – FAKE (Correct)
National Report – FAKE (Correct)

Three out of five correct, obviously not a large sample, but there are clear issues if the Onion is listed as real news. I then decided to type in my own site and that of other sites that do similar work. This is where my jaw hit the floor and I became furious. Here are the results:

Media Bias/Fact Check – FAKE (Wrong)
BS Detector – FAKE (Wrong)
Fake News Checker – FAKE (Wrong)

I stopped at this point and composed an E-mail to Mr. Singhal to request that he fix these problems. I am not a programmer, but it is pretty clear that because we mention “fake news” often in our reports and also have a whole category dedicated to fake news, that his AI is picking that up. In other words, his AI is not working as it should. It is very alarming that legit sites like ours, BS Detector and Fake News Checker are listed as fake. This is hurting legitimate sites who are actually trying to inform and fight against the very things Mr. Singhal’s AI is supposed to do.  I also find it very misleading to have Infowars and the Onion listed as real news.

We support all efforts to fight fake news and rampant bias, but this product simply is not accurate at this time.  Hopefully, Mr. Singhal will correct his AI so that legit sites do not get the fake label, which can hurt them severely.

Update: Mr. Singhal wrote back and informed me that the AI is only accurate with exclusive news sites.  Therefore, sites like MBFC that is 90% a media bias resource and only about 10% a news sources would not be read properly by the AI. Mr. Singhal states that he will put a disclaimer on the website to reflect this.  He is also working on correcting the issues where it gets satire and conspiracy sites wrong. I greatly appreciate Mr. Singhal writing back and clarifying.

Update 2: A correction was made to the AI and it is now listing Infowars and the Onion correctly. Further testing reveals that the AI is accurate with large news sources. We are hopeful, Mr. Singhal will address the issue with all websites that are not 100% news as being listed “fake.”

By Dave Van Zandt

Left vs. Right Bias: How we rate the bias of media sources