Intuition taster: How to judge the quality of evidence
“It’s all evidence-based” is an increasingly common mantra and, at face value, a welcome trend. But do the claims always stand up?
It is fashionable now for TV news, radio debates and government initiatives to lace their pronouncements with this reassuring phrase. The problem is that interests remain: commercial, political and lobbyist. In a competitive media world, companies, parties and organisations now use evidence claims to attract attention.
There are countless ways to dress up research results to add unjustifiable weight to a weak claim. Single studies can be cherry-picked, even though they diverge from the general trend of results. Conclusions from some studies might fail to disclose their limitations. Graphical information can be manipulated to emphasise favourable trends.
Such distortions can have serious effects in fields such as medicine, environmental science and civil engineering. In education they can also prove damaging, as the consequences of selective education, for example, show.
Fortunately a new tool from the Coalition for Evidence-Based Education (CEBE) aims to help practitioners, leaders and the general public combat this tendency. A website, ‘Assessing Claims in Education’, offers advice on how to judge a claim, using some 40 nuggets of information. Is correlation masquerading as causation? Are comparisons fair? Are results transferable?
What do confidence intervals and statistical significance tell us? The website is part of a broad international initiative, launched in healthcare and now spanning fields as diverse as agriculture, environment and social welfare. Many aspects of bad evidence claims are common to all fields, so experience elsewhere has enabled the CEBE team to adapt a large body of evidence about evidence for the benefit of education.
The tool has been designed for use in professional development sessions as well as for the individual. It is useful for curriculum leaders as well as research champions. The team is keen to refine the tool in response to user feedback, so please let us have your views.
References
Andrew Morris is an honorary senior lecturer at the UCL Institute of Education and a member of the national planning group of the Learning and Skills Research Network. Andrew is president of the education section of the British Science Association.
