Analysis of Wakefield MMR study asks why flaws weren't spotted by Lancet editors

by Trisha Greenhalgh

from: http://briandeer.com/mmr/lancet-greenhalgh.htm

This page is research from an investigation by Brian Deer for The Sunday Times of London and the UK's Channel 4 Television into a campaign linking the MMR children's vaccine with autism. | Go to part I: The Lancet scandal | Go to part II: The Wakefield factor

On this page is an analysis of the Wakefield paper by Professor Trisha Greenhalgh of University College London. Greenhalgh (pictured below) is a regular reviewer for the British Medical Journal and the Lancet, and author of "How to Read a Paper - the basics of evidence-based medicine", BMJ Publications, 2002. This analysis is dated April 2004. Greenhalgh reviews the paper at face value, but Brian Deer has subsequently established that Wakefield's published report doesn't honestly represent the underlying data

A critical appraisal of the Wakefield et al paper The term ‘critical appraisal’ means a systematic analysis of a study that has been published in a scientific journal (or a study submitted to a scientific journal whose authors seek to publish it), using a checklist of key questions. When a scientist submits a paper to the Lancet, it is sent out to two or more reviewers with expertise in the field, who are required to send back a formal report based on a critical appraisal checklist.

1. Is the topic area important and relevant to the Lancet’s readership?

Yes. In 1998, MMR vaccine had recently been introduced in the UK. Autism was rising in incidence. Questions were being asked about a possible link. A paper describing a study that explored this link would certainly have been appropriate to the Lancet and highly relevant to a general medical readership, so long as it was scientifically robust.

2. Was the study original?

Yes. At the time, no previous study had explored in this way the link between MMR vaccine, bowel problems and autism in children.

3. Was the research hypothesis clearly stated?

No. The paper does not state a research hypothesis at all. Implicitly, the research hypothesis might be stated, “The administration of MMR vaccine to infants increases their risk of developing (a) a particular pattern of inflammatory damage in the gastro-intestinal tract and (b) autism or an autism-like syndrome.”

4. What was the study design?

The study design was a descriptive report on 12 children who had been referred to a paediatric gastroenterology clinic with both bowel symptoms (diarrhoea, abdominal pain, bloating, and food intolerance) and pervasive developmental disorder characterised by loss of skills that had been previously acquired. Various blood tests, gastrointestinal biopsies, and a sample of cerebrospinal fluid were taken from the children. The samples were examined to explore the extent of inflammatory reaction in the bowel and to exclude other diseases (such as thyroid disease, inherited metabolic syndromes and so on). Of dozens of tests done on each child, a number were abnormal, though no test was consistently abnormal in all the children. Eleven of the 12 children had microscopic evidence of inflammatory reaction in their bowel. The parents were asked to remember back and identify if and when MMR vaccine was given. In 8 of the 12 children, the onset of developmental delay was said to have occurred within 2 weeks of having the MMR vaccine, and in 3 it was said to have occurred within 48 hours.

5. Was this design an appropriate way to test the research hypothesis?

No. If the hypothesis was that there is a causal link between MMR and autism-bowel syndrome, this study design was incapable of proving that link one way or the other. There are six main reasons for this:

The sample was highly selected – that is, the authors deliberately picked out the tiny number of children who had been referred to a major specialist centre because they had both bowel symptoms and an autism-like syndrome. So the fact that these rare conditions occurred together proves nothing at all. The fact that children with diarrhoea or other chronic gastro-intestinal symptoms have microscopic evidence of inflamed bowels is also, in itself, unsurprising.

The sample was extremely small. I would expect a scientific study claiming a causal association between two events (in this case, giving MMR vaccine and developing autism-bowel syndrome) to have a formal statistical calculation of the number of individuals that ought to be looked at. This is known as a power calculation. The reason why the Lancet does not normally publish studies on just 12 individuals (the usual number of research participants is several hundred, and not uncommonly, several thousand) is that the smaller the study, the more likely it is that an apparent causal link will turn out to be due to chance association.

The study had no control group. When studying the possible harmful impact of a vaccine or environmental agent, it is standard scientific practice to include a control group of individuals who have not been exposed to the putative harmful agent. Indeed, epidemiologists can occasionally perform robust studies to explore the causes of rare but devastating events when only tiny numbers of individuals are affected (this was done, for example, to explore the possible link between overhead power lines and leukaemia) by carefully matching these individuals with others who are alike in many ways (age, sex, social class, smoking status and so on) but who have not been exposed to the putative causal agent. This sort of design is known as a case control study. No matching was not done by Wakefield and colleagues, which means we have no way of knowing that the unusual combination of bowel disease and autism-like syndrome might be equally frequent in children who had not received the MMR vaccine.

The alleged link with MMR vaccine was made on the basis of retrospective parental recall – in other words, parents (who had just signed a consent form to take part in a study of whether there is a link between MMR and autism) were asked to consider how closely in time the vaccine was with the onset of autism-like behaviour pattern in their child. Whilst there is no suggestion that parents deliberately fabricated the closeness of the link, the authors of the paper took no steps to guard against what is known as ‘recall bias’ (that is, remembering a closer association between two events than actually occurred). The notion that a previously healthy child was ‘normal’ one day and showed clear signs of autism the next day is at odds with the clinical course of pervasive developmental disorder. Such syndromes tend to have a period of weeks or months during which the child’s behaviour is causing some concern but is not clearly abnormal. Hence the firm statement of a “48 hour” or “2 week” interval between the administration of the vaccine and the diagnosis of autism is scientifically implausible and requires further explanation.

The follow-up period during which the children were studied was short – days or weeks rather than months or years.

The investigators were not ‘blinded’ – that is, the people who examined the children and analysed the specimens all knew that the children had received MMR vaccine and that a question had been raised about the its link with autism-bowel syndrome. They would also have been aware that ‘positive’ findings would be highly likely to lead to a prestigious publication whereas ‘negative’ findings would not. In a scientifically robust study, the people who do the tests should be unaware of the status of the samples.

6. Were the study’s conclusions supported by the data?

No. Whilst Wakefield and colleagues stated at one stage in their paper that their findings did not prove a causal link between MMR vaccine and autism-bowel syndrome, the overall tone of the paper strongly suggests that they believed that they had demonstrated such a link. This conclusion is unjustified for the reasons given above.

7. If the answer to (5) is no, would a more robust design have been practically possible to test the study’s main hypothesis?

Most certainly, yes. It is worth noting the key principles, set out by Sir Austin Bradford Hill in 1965, of any scientific study that seeks to prove causation rather than merely association:

Is there evidence from true experiments in humans?

Is the association strong?

Is the association consistent from study to study?

Is the temporal relationship appropriate (i.e. did the postulated cause precede the postulated effect)?

Is there a dose-response gradient (i.e. does more of the postulated effect follow more of the postulated cause)?

Does the association make epidemiological sense?

Does the association make biological sense?

Is the association specific?

Is the association analogous to a previously proven causal association?

A study from Finland, designed to be large enough pick up even very rare events, followed 1.8 million children prospectively from the day they received the MMR vaccine for a full 14 years. The total number of reported vaccine-associated events was 437, and they included allergic reactions and convulsions, but none of the children had autism linked with the vaccine.

Several subsequent studies, all much larger and better designed than Wakefield’s, have confirmed an absence of excess cases of autism or bowel disease in children who have received MMR vaccine. In 1999, for example, Dr Brent Taylor looked at 500 cases of autism in the Royal Free Hospital and found no excess in immunised children. A very large and well designed study in the USA – with a proper case control design and with all vaccination dates confirmed by medical records – looked at the incidence of inflammatory bowel disease in people who had received MMR vaccine, other measles-containing vaccine, and no vaccine. The risk of inflammatory bowel disease was the same for vaccinated or unvaccinated people. The average time between vaccination and the development of bowel disease was 12 years. Only 1% of cases developed inflammatory bowel disease within a year of vaccination – and 1% of controls developed inflammatory bowel disease during the same time period. This study was cited in the medical journal ‘Bandolier’ and its results are available free online on http://www.jr2.ox.ac.uk/bandolier/index.html

In conclusion, the Wakefield study was scientifically flawed on numerous counts. I am surprised that neither the editor nor the reviewers spotted these flaws when the paper was submitted. Had they done so, the public would have been saved the confusion and anxiety caused by false credibility conveyed by publication of the study in this prestigious journal.

 

 

Professor Trisha Greenhalgh OBE MD FRCP FRCGP

 

 

 

 

 

Professor of Primary Health Care and

 

 

Director, Unit for Evidence-Based Practice and Policy

 

 

University College London

 

[1] This combination has been given the informal name ‘autism-bowel syndrome’ though no such syndrome is officially accepted in the mainstream medical literature

[2] Bradford Hill, A, 1965.  The environment and disease: association or causation?  Proceedings of the Royal Society of Medicine. Volume 58, 295-300, cited with permission in Greenhalgh T, How to Read a Paper: the basics of evidence-based medicine; London: BMJ Publications, 2^nd edition 2002

[3] A Patja et al. Serious adverse events after measles-mumps-rubella vaccination during a fourteen-year prospective follow up. Pediatric Infectious Diseases Journal 2000 19: 1127-1134.

[4] B Taylor et al. Autism and measles, mumps, and rubella vaccine: no epidemiological evidence for a causal association. Lancet 1999 353: 2026-2029.

[5] RL Davis et al. Measles-mumps-rubella and other measles-containing vaccines do not increase the risk of inflammatory bowel disease. Arch Pediatr Adolesc Med 2001 155: 354-359