Google Down-Ranks Real News

Photo by enriscapes

As alluded to in yesterday’s post, the 2016 shock to what we might politely call political orthodoxy provided a boost to mainstream news subscriptions. “The [New York Times] added 276,000 net digital-only subscriptions in the final three months of 2016, the best showing since it implemented its paywall in 2011. In the weeks immediately following Mr. Trump’s election in November, subscriptions increased tenfold compared with the previous year,” wrote Shannon Bond for Financial Times in February.  Similar spikes occurred at The Washington Post and other traditional news sources. So, if nothing else, the bizarre theater of obfuscation and Twitter rants coming out of the new administration seemed at least to rekindle millions of Americans’ desire for credible reportage.

But get this…

Gerry Smith for Bloomberg reports that when The Wall Street Journal blocked Google users from reading its articles for free, its subscription business “soared” only to see this gain countered by a 44% drop in traffic from Google search.  It turns out, according to Smith, that Google’s algorithm prioritizes free content over paid content.  Assuming this is true, there’s a whole lot wrong with it, beginning with the fact that this belies Google’s boastful raison d’etre to “organize the world’s information” and deliver search results based on quality and relevance.

If the algorithm looks for free content first, this suggests that fake news and other junk content will be consistently prioritized over the WSJ, The New York Times, The Washington Post, The Daily Beast, and so on. Not only does Google’s policy in this case stifle these organizations’ flexibility to choose their own strategies for financial survival, but for the general public, it exacerbates the already toxic brew of bad information that is, at this point, literally threatening democracy itself.  And for what?

Money of course.  Google makes money by serving ads to content that users can more readily access without going through paywalls.  Consequently, makes it into the top results instead of, y’know, news—at least according to what Bloomberg is reporting. “The Journal’s experience could have implications across the news industry, where publishers are relying more on convincing readers to pay for their articles because tech giants like Google and Facebook are vacuuming up the lion’s share of online advertising,” writes Smith.

I’ve gone so far as to assert that we’ve actually lost the “information revolution.” The promise of a more enlightened society through digital technology has hardly been fulfilled, but we do have some very funny memes to stick on the fridge of history.  Given the extent to which the current narrative has been hijacked by a strange confluence of bored trolls and professional data manipulators, a sane person can be forgiven for deciding that it’s about time to unplug.  A recent report by the Data & Society Research Institute on the influence of—I guess we can call it “troll culture”—on even the mainstream media says the following:

“Mass media has greatly profited off the appeal of conspiracy theories despite their potential for harm. Network news channels feature ‘documentaries’ investigating theories without fully refuting them. In 2011, when Donald Trump began promoting the “Birther” conspiracy theory, claiming President Obama was born outside of the United States, mainstream news outlets like CNN and Fox News covered these claims extensively. Out of this environment, an entire industry of conspiracy and fringe theory has emerged.”

The report delves into the intricate network of internet subcultures described as “an amalgam of conspiracy theorists, techno-libertarians, white nationalists, Men’s Rights advocates, trolls, anti-feminists, anti-immigration activists, and bored young people,” who are directly influencing the narrative that many citizens around the world think of as the truth.  And this is bad enough.  “Google says its ‘first click free’ policy is good for both consumers and publishers. People want to get the news quickly and don’t want to immediately encounter a paywall,” writes Smith.

Sound familiar?  What’s good for Google is invariably “good for consumers.” And consumers invariably buy the pitch for a while.  Free?  Yeah, free sounds good.  Until it turns out that free actually a cost. Sometimes a very dire cost—like millions of voters who would sooner believe in alien abduction than climate science. And the point of the above quote about television news creating entertainment out of nonsense is that sensationalism will be the only thing left, if business models no longer support investigation, travel, research, fact checking, and other expensive human labor required to deliver quality journalism.  Add to all this that Google search will apparently down-rank legitimate news because it isn’t free?  Damn.

© 2017, David Newhoff. All rights reserved.

Follow IOM on social media:


  • Robert W Hill

    One of your best and most important pieces, David. Thanks.–Bob

  • This sentiment finally made me recently bite on a subscription to the economist. Well worth the $45 per quarter, much better use of my time over browsing hivemind sites like Reddit. You get what you pay for with news as much as anything else!

  • While your sentiment is spot on, David, I feel I must point your attention to your penultimate paragraph and the quote contained therein, because there I see the root of the rot, as it were.

    Notice how the DSRI report authors have – perhaps unwittingly, which is all the more worrying – managed to conflate “political positions we don’t like” with “things that are not true”. They are far from the first to do so, to the point where I’m hesitant to say that FOX News invented fake news.

    The problem is that the prioritisation of policy (a normative position) over the facts at hand (a positive position) is hardly restricted to one political party, one side of the political spectrum or even one business model of presenting the news. Everyone‘s doing it – including ourselves, if we’re not careful. I’m finding it increasingly difficult to engage with any sort of reporting – regardless of who’s doing it – because of the thick fog of political bias that hangs over any matters of fact that may, or may not, be contained therein.

    The reason I point this out, is because the retreat towards paywalls may be less a matter of looking for factual, unbiased reporting, but rather a search for a safe space, where opposing political views will not invade our bubble of epistemic security (one would guess that only very dedicated trolls would pay to view content that goes against their political bias). If this is the case – and I regret to say that I’m seeing many signs of this – it doesn’t really matter what Google does. The battle lines have been drawn even before their, famously lightweight, start page has loaded.

    • David Newhoff

      Faza, always good to hear from you, and as always, you’re observations add much to the conversation. In a couple of places, including this post, I’ve reported that data manipulation skews right for now but that there’s no reason to assume this will remain exclusively the handiwork of Trumpist et al. And of course, I’ve called out left-leaning organizations like EFF and FFTF for either engaging in, or turning a blind eye to, such practices.

      I’m not sure the DSRI focus is unwitting so much as it is very hard to look at the amalgam of players and the dynamics at work in this moment without criticizing Trump as a byproduct of the conclusions. This is especially true given that his particular detachment from reality is unprecedented in living memory for a President of the United States. For me, he’s a category separate from conservative v. liberal, but that’s a whole other matter.

      To your larger point, while I agree that the fog of politics makes the prospect of engaging with journalism somewhat disheartening, I still believe it very much depends on who is doing the reporting. If a 30-year veteran is working for The Daily Beast, even his opinions are going to be grounded in research and the tenets of his profession in stark contrast to some bogus website full of grabby headlines designed either to a) make cash for its owner or b) skew facts through volume or c) both.

      Although the meaning of “fake news” has now been distorted, it technically refers to a site for which someone literally invents material in order to trick people into clicking–basically wardrobe malfunctions plus politics. These sites are just cash cows, and their nature should be obvious to viewer, but I’ve seen plenty of friends share them because we’re all guilty of spreading headlines without reading the stories. A layer beyond that are politically-skewed aggregators like OccupyDemocrats that take the thread of a real story and tack on a headline that is both divisive and false. Beyond that is a more complex layer of coordinated manipulation that is designed to cause rumor to go viral to extent that it infects mainstream reporting a la the Brietbart network. (This is what I believed was happening with the anti-SOPA campaign and was the reason I started writing this blog.)

      While there is certainly discussion to be had about the internal standards at various traditional news sources, I would certainly defend the journalists I know as distinct from literal fake news and the financed manipulation network (whatever we want to call that). If we think of all that content holistically as a bad actor in contrast to what we call the news, I think Google search does matter quite a bit.

      It’s certainly true that Google did not draw the battle lines. My own view is that, in the US, everything started going to hell when CNN turned news into entertainment, which led to the inevitable customization of news catering to audiences according to political bias. I said in a post somewhere that TV news was better when it was mandated by law and unprofitable–best combination for journalistic integrity because there was no motivation to do anything else. The internet didn’t create the problem, but I do think it has exacerbated it by orders of magnitude.

      • Don’t get me wrong, David, for all I’ve said, I believe that there are still some tried-and-true muckrakers out there and – yes – we’re most likely to find them in the big news outlets.

        However, the deeper point – that I am, frankly, hesitant to spell out more than I have already – is that our very notion of what constitutes “truth” has become so intertwined with policy that even the best of us increasingly fall into the trap of filtering information through our normative filters.

        The sad truth is that Trump didn’t win on the strength of the fake news generated by Breitbart, Fox et consortes. On the contrary, it is the fast-and-loose approach coming from the other side that is ultimately to blame.

        In a way, it doesn’t matter how terrible Trump is. He won because:

        1. He is not part of the establishment,

        2. He was the only one willing to listen to a major portion of the population (other than Sanders, possibly, but Bernie was too nice a guy for the game, I guess).

        Rather than eat crow and reconsider their errors, the establishment circled the wagons. The Trump supporters are perfectly fine with this. If – and that’s a big if – the Trump presidency doesn’t turn out to be a disaster of biblical proportions, he might have a solid shot at a second term and his opponents still won’t realize what they’re doing wrong.

        It’s interesting to view this from a European perspective – especially, from a country that didn’t have as much luck as you did. France narrowly dodged a bullet – with my guess being that people got spooked by Trump and Brexit. However, once the boogeyman is manifest, it had better be hella scary, or it will soon cease to impress and we’re getting there pretty darn soon.

        My worry is that by that time there will be no sensible alternative. Or rather: there won’t be a sensible alternative, yet.

      • David Newhoff

        Suffice to say, Faza, I share your concerns. And I believe that Trump and Sanders represent a populism that is generally (and understandably) frustrated with government and the status quo, but that neither doctrine is particularly well-grounded in reality. Ultimately, what’s most dangerous about Trump, in my opinion, is that he’s fired up latent meanness and intolerance as a path to “greatness.” More than that, though, and this thread gets long and way beyond the editorial scope of this blog. Thanks as always.

  • It’s not clear how you think Google is supposed to index a site that blocks it from access.

    • David Newhoff

      If the publisher blocks Google search users from reading articles for free, that does not prevent Google from ranking links to the articles in top search results. That is the antithesis of the promise of search, and specifically Google’s promise, to link to the most relevant results.

      • So what is Google supposed to be going on, if they can’t see the content?

      • David Newhoff

        If you’re asking if I know precisely how the Google algorithm works, I don’t, and neither does anybody else past a certain point. Still SEO is not predicated on any of the search algorithms reviewing the full content of anything. It searches titles, urls, keyword terms, etc. and each publisher manually manages SEO as part of their process. Regardless, it stands to reason that if Google prioritizes free–as Bloomberg reports–this is not going to produce the most relevant, quality results when the credible news sources are increasingly migrating to some form of subscription model. It literally has the potential to put Fred’sCrazyNewsTheories ahead of anything written for legit news organizations. If anything, the free/paid attribute should be prioritized the other way around.

      • > Still SEO is not predicated on any of the search algorithms reviewing the full content of anything

        yes, but search directly is. e.g. I frequently search Google News for all manner of obscure stuff, and I am indeed interested in passing mentions. Google indexes the full text of pages, not just trivially-gameable keywords, and that is of utility to searchers.

        And by the way, speaking as a GNews-searching end user, hitting a paywall is frustrating and I feel it’s doing less for me, not more.

        I’m looking at this article. WSJ’s done okay in keywords, which is good, they’re coming to the party.

        I do see what you’re saying, but I’m entirely unconvinced it’s implementable in a fashion that serves users, and not just media outlets, just by declaring “make it so”.

      • David Newhoff

        So, as far as SEO, I have no idea whether or not the publishers make the full text visible to search engines, even if the articles are behind paywalls for users–that’s beyond my tech knowledge, though there are probably people I can ask. In regard to implementation, I fail to see why choosing not to prioritize free will not produce better results. Bottom line, as a user, if I search “Trump Russia Investigations” and the top results are not straightforward news articles telling me the latest factual developments, then search is basically useless to me. There’s nothing wrong with secondary or tertiary reporting and editorials (I write a blog), but I don’t need barriers between me and the first-tier journalists covering a story. Paywalls may be frustrating, but journalism costs money, and free doesn’t sustain the work. Amazon owns WaPo, which its own employees will now describe as a “tech company” due to the amount of adaptation–and they still adopted a paywall model. And there’s a reason why subscriptions jumped after at least some readers caught on to the fake news problem.

        We have a similar problem with business listings that can be undermined by paid priority for businesses that are either not legit or middle-men trying to divert users from transacting with the suppliers they really need. I once accidentally called an “Apple help line” that was a scam operation because the scam number appeared at the top of results and I wasn’t paying attention. There is only one legit Apple help line. Why should there be any other other result at the top? That’s not serving users, it’s serving Google’s bank account by exposing users to a criminal enterprise.

      • I mean, I can envisage WSJ telling Google “trust us this content is yuge”, but searches are in response to search terms, and if Google doesn’t get the content it’s got nothing to index as being a good response to those terms.

      • David Gerard is the poster child for an idiot wikipedian freetard.

        If the site hides everything behind a paywall then google shouldn’t index it. However, most paywall sites don’t hide everything they give the first couple of paragraphs, and there is nothing to say that a the implementation of the paywall has to blocks the crawler bot. It may well allow googlebot access along with the other major crawlers.

        Not all website programmers are are crap as the ones wikipedia employs.

  • Doug Mortenson

    The dumb get dumber, eh, David? Of course, al this would be wonderful if you knew the difference between “real” news and “fake” news. To call the Wall Street Journal, New York Times or The Washington Post “real” news is the very definition of clueless. One of your commenters refers to the Economist, which, like the other has-beens mentioned, once deserved to be called “real” news. Not anymore. The New York Times and Washington Post have both been caught kiting stories with no basis in fact, in a concerted effort to damage Donald Trump or members of his administration. You and your toadies may be stupid enough to believe the three CNN fabulists were nothing more than rogue miscreants, but they are “real” examples of the mindset raging at the Times, Journal and Post. Anyone who believes that the Russian/Trump fishing expedition has any basis in reality or is kept alive for any other reason than to net SOMETHING to impeach Donald Trump with should be made to watch the Project Veritas video of various CNN figmentalists, a la Alex in “Clock Work Orange”. Not because it would do any good, it wouldn’t. But, for progressives, who’ve never had an original thought in their squandered lives, truth is like kryptonite to Superman. It would definitely be cruel. The Daily Beast – how gullible do you have to be to consider anything the Daily Beast posts as critical or discerning? Seriously!

    • David Newhoff

      Doug, I was on the fence about whether or not this comment runs afoul of my own policy. On the one hand, you seem to want to generically defend Trump, and this blog is not the forum for that. Conversely, your angry comment sort of proves the point of the post. You have decided that the entire industry of professional journalists is guilty of cultural capture, which is an unsupportable position by any reasonable measure. The big brands, as it were, have at times been guilty of error, carelessness, and of glitzing news into entertainment. That’s an unfortunate market reality. But these organizations are also staffed by individuals who have been journalists for decades, people who have shed buckets of blood and sweat to report stories of great importance and with tremendous care. If what these individuals reveal doesn’t jibe with your politics, that doesn’t make the reporting inaccurate. We are living in a dangerously post-factual world, one that is technically apolitical. Self-described conservatives and liberals (and everyone in between) seems to be clinging to narratives that dismiss the very idea of expertise itself. Your view of these journalists seems to be just such a narrative. On the other hand, if you just feel like labeling as “clueless” everyone who doesn’t support Donald Trump, there are lots of other platforms for that. I hear Twitter’s a thing.

Join the discussion.

This site uses Akismet to reduce spam. Learn how your comment data is processed.