Ben Goldacre’s Ladybird book of RCTs — and a free Geek Manifesto extract about RCTs and education

The randomised controlled trial (RCT)  is one of the greatest inventions of modern science — a tool that allows you, more reliably than any other, to compare two or more interventions and determine which is more effective for a given purpose. It’s the standard by which we rightly demand that most medicines are tested before they’re approved. And one of the core arguments of The Geek Manifesto is that RCTs could be profitably applied to public policy in education, criminal justice and social issues much more frequently than they are.

There are, fantastically, voices in the Cabinet Office who agree. Last month, it published the results of a terrific collaboration between Ben Goldacre, the author of Bad Science, Professor David Torgerson, Director of the York Trials Unit, and Laura Haynes and Owain Service, of the Behavioural Insights Team at the Cabinet Office.

Their paper, Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials, runs through several great examples where RCTs have already revealed powerful policy-relevant evidence — such as the effectiveness of sending personal text messages to remind people to pay court fines, and the ineffectiveness of paying people to attend adult literacy classes. It also offers a great introduction to the purpose and structure of well-run randomised trials. Goldacre describes it as the “Ladybird book of RCTs”.

Everyone should read this paper, especially if you work in Whitehall or Westminster. And everyone should email their MP to tell him or her to read it as well.

It’s a good moment to post another free extract from The Geek Manifesto — this time, about the role that RCTs could play in education policy. It begins here:

Teaching to the test

When people are asked what they consider to be the most important issue on the political agenda, education generally ranks second only to health. It is understood and accepted by British politicians of all parties that providing universal access to first-class medical care and providing excellent state schools that offer opportunity to all are among the core functions of government. Yet when it comes to evidence, these two central policy concerns are held to entirely different standards.

In healthcare, we expect drugs and medical procedures to be assiduously tested by the most rigorous appropriate methods before they are licensed for general use and before the state agrees to fund them. For the most part, this means assessment by randomized controlled trials  (RCTs) – the most reliable method yet devised for determining whether or not a particular intervention really works.

The RCT is commonly described as a kind of ‘gold standard’ for medical research because it seeks systematically to minimize potential bias through a series of simple safeguards. It must have a control group, so an intervention is tested against another intervention – an inert placebo, or an existing drug. This ensures that the influence of the treatment itself and not the act of treatment is measured. Patients must also be allocated at random to each group to  ensure that a researcher can’t prejudice the results consciously or unconsciously by, for example, picking sicker patients as controls.

RCTs are not without drawbacks. Critics have argued that it is unethical to experiment on human subjects this way, or to give patients a treatment that may be less than optimal. This type of research can be expensive and time-consuming, and there is often debate over the applicability of conclusions when the tight parameters of the trial are varied in the real world. But in medicine at least, RCTs are accepted as essential because they generate more dependable knowledge than any other approach. The  alternative to experimenting this way is not to avoid experiments. It is to conduct uncontrolled, non-random experiments which raise much greater ethical problems because they do not generate useful data.

RCTs have become commonplace in medicine because we accept that the evidence they generate is of paramount importance when life and health are at stake. Life chances and health are also profoundly affected by the educational opportunities that children receive, but we demand a different order of evidence for the teaching techniques that our schools employ.

There is no good reason why RCTs should not be used routinely to evaluate different approaches to teaching children to read, or to educate children with special needs. Yet they are not a standard part of the educational landscape.By the same token, too many new initiatives pass without formal evaluation, and plenty are unencumbered even by the need to collect useful data. Education, which could easily be a bastion of evidence-based policy, is replete with policy-based evidence.

The question of how best to teach children to read is among the most controversial issues in education. For many years the favoured approach was to teach children to recognize whole words. Then in the 1990s and 2000s, more and more teachers began to be won over by an alternative strategy, synthetic phonics, by which children learn to match sounds to letters and groups of letters. A seven-year study in Clackmannanshire, Scotland, suggested that children taught using phonics were three and a half years ahead of their peers in reading by the time they finished primary school. More and more parents began to demand it.

In 2006, an expert inquiry led by Sir Jim Rose, a former director of inspection at the Office for Standards in  Education, found an ‘overwhelming case’ for phonics. Ruth Kelly, the Education Secretary, duly mandated its use in primary schools the following year. Michael Gove, the current Education Secretary, is presently extending provision.

[For a detailed critique of the Rose review and the Clackmannanshire evidence, see Synthetic phonics and the teaching of reading: the debate surrounding England’s ‘Rose Report’, Dominic Wyse and Morag Styles, Literacy, Volume 41,Number 1, April 2007.]

In parallel with the Rose review, the government asked a team led by Carole Torgerson, then of the University of York, to examine the evidence for phonics. Her report noted that while there were indications of promise, these came principally from the US school system and that effect sizes were small. The highly cited Clackmannanshire study was the only substantial evidence from a UK context, and it had not been randomized. A second Clackmannanshire study had used random allocation, but had other weaknesses: the sample size was very small, and children in the intervention and control groups had been taught by the same teacher, creating clear potential for bias.

In short, robust evidence was lacking. If phonics was to be more widely adopted, Torgerson advised, the government should roll it out gradually, with the first areas to benefit being chosen at random. That way, it would be possible to determine whether or not it really worked.

‘That was never done,’ Torgerson says. ‘It just became policy. We had a real opportunity to do a randomized study that might have settled this issue, and it was missed. As a result, we still don’t know whether or not phonics works as the main teaching strategy for all children. Some of the recent evaluation work has demonstrated synthetic phonics may not be having the impact that was hoped for. If we’d done the randomized trial we would have known before the policy went national.’

This isn’t the only case in which opportunities to conduct RCTs have been spurned. In 2005, the government began what was supposed to be a three-year pilot study of the Every Child a Reader programme, which offered a one-to-one intervention called Reading Recovery to the lowest-achieving 5 per cent of children in the first two years of compulsory primary education.

As with synthetic phonics, however, the pilot wasn’t a randomized trial. Worse, it was  then deemed such a success after a single year that the programme was implemented nationally, before it had even finished. When the Commons Science and Technology Committee challenged Carole Willis, chief scientific adviser at the Department for Children, Schools and Families, about this, she said a randomized trial would have been too expensive and that it was too difficult to persuade schools to take part. Yet neither problem prevented the department from running an RCT of a parallel mathematics intervention, Every Child Counts. Her arguments, the committee found, ‘do not stand up to scrutiny . . . we conclude that a randomized controlled trial of Reading Recovery was both feasible and necessary.’

As this book was being written, a new government was missing another opportunity. In June 2011, Michael Gove announced a radical plan to convert the 200 weakest primary schools in England into academies, free from local authority control, to improve standards. Whether it will work is anybody’s guess, and it will probably remain so. It would have been simple, Torgerson says, to turn this initiative into a proper trial, by selecting 100 of the schools at random to get academy status immediately, with the others following the next year. That first year would have allowed researchers to compare outcomes, to establish whether or not there was a positive effect.