The dangers of lazy science reporting

Next time you see the words ‘in a study published today…’, do yourself a favour and ignore what follows. Unless the article you’re reading provides the proper context and background to the study it is reporting, it may very well be worse than useless.

Often, these stories will be about a cause of (or cure for) cancer, or the latest evidence on climate change. But such stories appear on just about every imaginable subject. For example, in the last few days you could have found stories about the ineffectiveness of annual flu jabs for the elderly (This is London); the failure of company boards to understand the value of their IT equipment and software (Financial Times); the suggestion that women are better at cooperation than men (the UK Daily Telegraph); women who suffer from allergies may be increasing the risk of asthma and eczema in their breastfed children (the Australian)… the list goes on and on, and the production of such stories is apparently endless.

This is not to dismiss science as a way of understanding the world around us. On the contrary, I think that science can tell us much of importance. And there may be interesting insights provided by these studies. But the genre of ‘study published today’ stories holds back understanding rather than enhancing it.

The stories that now populate the headlines are a symptom of a problem that runs deeper than media superficiality. If it were just a case of the media treating science in the same way as a celebrity wedding it would be regrettable but complaining about it would be much like complaining about the weather. The more substantive problem is that our culture – and especially decision makers such as politicians and business leaders – have come to see science as a sort of oracle that delivers ready-to-use truths.

‘Study published today’ stories closely fit this approach to science, treating results as free-floating nuggets of truth. Contextualisation, qualification and integration of a result into a larger picture becomes impossible. It is usually hard to tell from such a news story whether the study it refers to is groundbreaking or shoddy – or even just a prettied-up piece of public relations.

The starting point of such a story is usually the publication of a paper in a peer-reviewed journal, although papers presented at conferences – which may not even have the benefit of peer review – are often presented in the same way. Last weekend, for example, there were numerous reports about how women could substantially reduce their risk of cancer through losing weight, exercising and other lifestyle changes. But this story was not based on a study that had tried such an intervention but was simply a paper from a cancer research conference.

New York Times journalist Andrew Revkin claims that the ‘study published today’ model is driven by the ‘tyranny of the news peg’: reporters need a peg on which to hang their story and the publication or presentation of a paper plays this role. Researchers, universities and journals that issue press releases have been happy to collaborate with this model.

This may make sense from the point-of-view of journalistic convenience. It makes no sense as a way of assessing whether a piece of research really has made an important breakthrough. At the time of publication no one – not the authors nor the best experts – can do more than guess whether or not a paper will eventually prove to be groundbreaking. It may prove to be prophetic – or it may be overturned. Only work carried out over time will tell. It is the accumulation of results, the constant cross-checking, replication, testing, development and debate of ideas that gradually builds up the reliability of scientific knowledge.

Even worse, the more ‘revolutionary’ a paper, the more likely it is to make headlines. Yet it is precisely here that context is most needed. Some of these revolutions will survive, but most will not. Even when researchers work carefully, it is impossible to avoid all blind alleys, and there are many more wrong paths than correct ones. Peer-reviewed journals will rightly publish work that has been competently done but is later shown to be wrong.

While reporting paper-by-paper makes no sense from the point-of-view of developing real knowledge, superficial treatment of science goes beyond the media. The reduction of scientific results to a simple ‘message’, shorn of connection to the path by which it was derived, is one that sits well with downgrading political argument today in favour of technocratic ‘evidence-based policy’ that demands of science clear answers to the question of what is to be done.

A good example is a much cited essay by science historian Naomi Oreskes, seeking to assess the scientific consensus on climate change (1). Oreskes defined the consensus by quoting the Intergovernmenal Panel on Climate Change’s (IPCC) 2001 assessment: ‘Human activities … are modifying the concentration of atmospheric constituents… that absorb or scatter radiant energy. … [M]ost of the observed warming over the last 50 years is likely to have been due to the increase in greenhouse gas concentrations.’

To assess the strength of this consensus she typed key words into a database and retrieved 928 abstracts that she categorised in relation to the consensus. She found none that disagreed with the consensus.

It is hard to imagine a more superficial approach. To assess the contribution of a paper requires more than simply reading an abstract. Each paper needs to be read and appropriately weighed with expert judgement. It is possible that, when viewed in light of all the evidence taken as a whole, a single study should be taken as key. In other cases, even when many studies point in a particular direction there will be unresolved problems that prevent a decisive conclusion. For an expert, the background knowledge will be much wider than that retrieved by a general search for keywords on ‘climate change’. (The IPCC, for all its problems, at least approximates the process of expert review more closely.)

Perhaps Oreskes never intended her study to be definitive, and she makes the point that ‘details about climate interactions are not well understood… The question of what to do about climate change is also still open.’ But the fact that it is reached for so readily in discussions over climate shows how trivialised discussion of science has become.

Oreskes attracted a flock of critics from amongst the climate skeptics who challenged details such as her choice of keywords or the classification of particular abstracts. They carried out their own, rival versions of the study with the same methodology but claiming to reach different results (2). The fact that few, if any, of Oreskes’ critics seem to have pointed out the absurd superficiality of the study confirms that for all sides in public debate, scientific studies have become little more than symbolic tokens that can be used to bludgeon opponents.

As it happens, anyone who spends substantial time reading the scientific literature will see that today it is now possible to have even more confidence than the IPCC had in 2001 that there is a detectable human influence on the climate. But to reach that conclusion, and to sustain the argument, you would have to read way beyond the abstracts. If you do so you will also come away with a rich sense of the debates and uncertainties over the pace, consequences and mechanisms of both natural and anthropogenic change. These do not fit easily into a ‘consensus’ v ‘skepticism’ framework and do not come with pre-attached policy proposals.

There are many excellent science journalists working today, but they work best in the feature or book length format where they can give an overview of a field, and they tend to appear in scientific publications rather than the newspapers. The popular scientific essay, often a reflection on a more developed branch of science, has also proved its worth, as in the work of the late Stephen Jay Gould. There are many good popularisations by working scientists, and now interesting science blogs too. There is also one important exception to my injunction against ‘in a study published today…’ news stories. Journals such as Nature commission news pieces to accompany important papers. But this exception proves the rule, as these pieces are written by experts in the field who are given time to absorb the results pre-publication and their purpose is precisely to explain how a new result fits into existing work and assessing it in relation to known difficulties.

If a scientific insight emerges that truly matters to the wider world, then we will learn of it in good time. But as for science stories in the news section of the daily paper, let’s forget it.

Joe Kaplinsky is a science writer. He is speaking at the sessions The science and politics of climate change and Particle physics is sexy at the Battle of Ideas festival in London on 27-28 October.

(1) The scientific consensus on climate change, Naomi Oreskes, Science, 3 December 2004: Vol. 306. no. 5702, p. 1686

(2) Re: The scientific consensus on climate change, Benny Peiser, 4 January 2005