JRSM >
Volume 100, Number 4 >
Pp. 187-190
1 UK Cochrane Centre, NHS R&D Programme, Summertown Pavilion, Middle Way,
Oxford OX2 7LG, UK
2 James Lind Initiative, Summertown Pavilion, Middle Way, Oxford OX2 7LG, United
Kingdom
Correspondence to: Professor Mike Clarke E-mail: mclarke{at}cochrane.co.uk
 | SUMMARY |
|---|
 TOP SUMMARY INTRODUCTIONMETHODSRESULTSDISCUSSIONREFERENCESAPPENDIX 1
>References to... |
|---|
Design Assessment of the Introduction and Discussion sections in all reports of randomized trials published during May 2005 in five general medical journals.
Setting Reports of randomized trials in five general medical journals.
Participants Annals of Internal Medicine, BMJ, JAMA, Lancet and New England Journal of Medicine.
Interventions None.
Main outcome measures The inclusion or mention of one or more systematic reviews in the Introduction or Discussion section of each report assessed.
Results We found 18 reports of randomized trials. The Introduction sections referred to systematic reviews in five (27%) of these reports. None of the Discussion sections of the 15 reports of trials that were not the first published trials to address the question studied placed the results of the new trial in the context of an updated systematic review of other research. Although reference was made to relevant systematic reviews in five of these 15 reports, there was no integration—quantitative or qualitative—of the results of the new trials in an update of these reviews. In the remaining ten reports there was no evidence that any systematic attempt had been made to set the new results in the context of previous trials.
Conclusions There is no evidence of progress between 1997 and 2005 in the proportion of reports of trials published in general medical journals which discussed new results within the context of up-to-date systematic reviews of relevant evidence from other controlled trials. Although the proportion of trials referring to systematic reviews in Discussion sections has increased, the majority of reports continued to fail even to do this. Similarly, most researchers appear not to have considered a systematic review when designing their trial. Researchers and journal editors do a disservice to the interests of the public and others involved in healthcare decision-making by acquiescing in this situation.
| INTRODUCTION |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODSRESULTSDISCUSSIONREFERENCESAPPENDIX 1
>References to... |
|---|
In May 19973 and again in May 20014 we found and assessed 26 and 33 reports, respectively, of randomized trials in Annals of Internal Medicine, BMJ, JAMA, Lancet and the New England Journal of Medicine. We identified reports of apparently similar trials for 55 of these index trials, yet in only two were new results placed in the context of up-to-date systematic reviews of other relevant studies. In seven reports, systematic reviews were referred to in the Discussion sections of the reports, but in the other 46 reports no apparently systematic attempt had been made to set the results of the new trials in context. Thus, in both of our previous samples of trials, only a small proportion of the reports provided sufficient information to assess what contribution the new results had made to the totality of information, and thus permit reliable interpretation of their significance.
We repeated our studies in May 2005 to assess whether there had been any detectable improvement. In this study, as in the previous two, it was not our aim to assess the overall quality of the Discussion sections of trial reports, but simply to assess how well they had placed the results of the new trials in the context of other relevant research. Furthermore, we now also investigated the extent to which reports began by referring to systematic reviews to make clear the scientific and ethical justification for the design and conduct of the new research.
| METHODS |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODSRESULTSDISCUSSIONREFERENCESAPPENDIX 1
>References to... |
|---|
Identification of eligible reports and assessment of the Introduction and Discussion sections of the reports
We each searched the relevant issues of the journals in different random
orders. Any report judged to be eligible by at least one of us was considered
for inclusion.
We independently assessed the Introduction section of each eligible report to decide whether it referred to a systematic review (done by the authors or other people) that could have been used in planning the trial. We did not attempt to identify whether relevant systematic reviews were available at the time of planning those trials that did not refer to such a review in their Introduction. To do so would have required more knowledge than was available about when each trial was planned. Furthermore, the nonavailability of a review by others should not preclude the researchers from performing their own systematic review and, if they had done so, we might expect that they would have described this in their Introduction section.
We also independently assessed the Discussion section of each eligible report to decide whether an attempt had been made to integrate the results of the new trial within a systematic review, either qualitatively or quantitatively.
If a trial claimed to be the first or only trial of a topic, we searched for similar trials in the Cochrane Central Register of Controlled Trials (CENTRAL). We did not systematically search for all such trials or judge whether there was sufficient similarity between the new trial and other trials to combine them in a formal meta-analysis. We simply tried to find studies that might be considered for inclusion in a systematic review of the topic.
We resolved disagreements by discussion.
| RESULTS |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODSRESULTSDISCUSSIONREFERENCESAPPENDIX 1
>References to... |
|---|
Systematic reviews were referred to in the Introduction sections of five reports, and these could have been used in planning the newly reported trials. Reports of the remaining 13 trials did not provide any information on whether they had used systematic reviews done by themselves or other people to justify or inform the design of the index trial, or whether they had searched for systematic reviews.
In five reports, the authors claimed that their study was the only one to have addressed the question concerned, but we found apparently similar trials for two of these. None of the 15 reports for which there appear to be similar trials contained a discussion of the trial's results in the context of an up-to-date systematic review of earlier trials. Reference was made to relevant systematic reviews in the Discussion sections of five reports, but without integrating the results of the new trial—either qualitatively or quantitatively—into an update of the review. Four of these five had referred to a systematic review in their Introduction sections, while one had not done so. The fifth report in which a systematic review was referred to in the Introduction was categorized as a first and only trial when we assessed its Discussion section.5 It was a report of the 30-year follow-up of the first trial of a treatment, and none of the subsequent trials are old enough to evaluate this outcome. In ten reports, there was no evidence that any systematic attempt had been made to set the results of the new trials in context (Table 1).
|
| DISCUSSION |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODSRESULTSDISCUSSIONREFERENCESAPPENDIX 1
>References to... |
|---|
It is now a decade since the first CONSORT statement required that data from a new trial should be interpreted ‘in the light of the totality [emphasis added] of the available evidence’,6 and more than 40 years since Austin Bradford Hill suggested that the structure of a scientific paper could usefully be conceptualized in terms of four questions: ‘Why did you start? What did you do? What answer did you get? And what does it mean anyway?’.7 This is reflected in a common structure for scientific reports (sometimes referred to by the acronym ‘IMRaD’) comprising an ‘Introduction’; a description of the ‘Materials and Methods’; the ‘Results’; and a ‘Discussion’ of the findings. The Introduction should make clear why the study was worth starting and the Discussion should indicate the contribution of the new findings to the evidence available at the time of reporting.
The expectation that the results of a new randomized trial will be reported in the context of an up-to-date systematic review of earlier trials does not imply that the Discussion section of every report of a randomized trial should contain a full account of the material, methods and findings of such a review. The technology already exists to enable a brief review to be included in the Discussion section, with links to a relevant, up-to-date systematic review published elsewhere. And, now, with 3000 Cochrane reviews published in The Cochrane Library and protocols published for 1600 more, the availability and ready accessibility of systematic reviews has never been greater.
Randomized trials should be a key piece of the evidence for the public and others making decisions about health care, but to serve that purpose they need to be designed and reported in the light of other similar research. Systematic reviews provide the means to do this: they reduce unwanted duplication of research, help ensure that new research builds on lessons from earlier research, and place the findings of the new research in proper context.
In our three studies over the last eight years, we have repeatedly shown that the results of new randomized trials published in five general medical journals have not been presented within the context of updated systematic reviews of other relevant studies (Table 1). We have searched for but have not found any other reports of empirical research assessing the extent to which this issue has been addressed. We would welcome the conduct and reporting of such research.
In 2002, when we reported the second of our three studies, we wrote the following:
‘Because our expectations imply radical changes in the way that research is done and reported, we expect that not all researchers, journal editors, or publishers will agree with them. However, science is cumulative, and everyone, including the public, has a right to expect that this principle will be reflected more effectively in the way that science is conducted and reported. We feel that this imposes a duty on researchers to present their results in proper context and on journal editors to require them to do so.’4
A few years on, the findings we present here show that editors and authors—in these five high impact journals, at least—continue to fail to serve the needs of those who wish to use the results of randomized trials to make decisions about health care. Others have also shown how researchers designing new trials have not made proper use of systematic reviews, even where these are known to exist.8
Some years ago, the BMJ took the ground-breaking step of including a summary with each report of new research to show what is already known on a topic and what the new study adds. Unfortunately, this has not been followed by many other journals, nor extended to provide links to the evidence, such as systematic reviews, upon which these summaries are based. On a positive note, though, we are heartened to see that things might be changing in at least one of the journals we studied, with the announcement by the Lancet in mid 2005 that it will require all reports of clinical trials to include a clear summary of previous research findings and to explain how the new trial's findings affect this summary.9 Examples of this policy's implementation have appeared.10
| Footnotes |
|---|
Funding There was no dedicated funding for this project.
Ethical approval This project did not require ethical approval. It did not involve the care of any people or animals.
Contributorship All authors were involved in the design, conduct and interpretation of this project and the preparation and approval of this manuscript. The manuscript was prepared within the context of the authors' employment by the NHS and the MRC, and is subject to Crown Copyright. The views expressed are not necessarily those of these organizations or The Cochrane Collaboration.
| REFERENCES |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODSRESULTSDISCUSSIONREFERENCESAPPENDIX 1
>References to... |
|---|
Chalmers I, Altman DG. How can medical journals help prevent poor medical research? Some opportunities presented by electronic publishing. Lancet1999; 353:490 -3[CrossRef][Medline]
Clarke M, Chalmers I. Discussion sections in reports of controlled
trials published in general medical journals: islands in search of continents?
JAMA1998; 280:280
-2
Clarke M, Alderson P, Chalmers I. Discussion sections in reports of
controlled trials published in general medical journals.
JAMA2002; 287:2799
-801
Dalziel SR, Walker NK, Parag V, et al. Cardiovascular risk factors after antenatal exposure to betamethasone: 30-year follow-up of a randomised controlled trial. Lancet2005; 365:1856 -62[CrossRef][Medline]
Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA1996; 276:637 -9[CrossRef][Medline]
Hill AB. The reasons for writing. BMJ1965; 4:870
Cooper NJ, Jones DR, Sutton AJ. The use of systematic reviews when designing studies. Clinical Trials2005; 2:260 -4[CrossRef][Medline]
Young C, Horton R. Putting clinical trials into context. Lancet2005; 366:107 -8[CrossRef][Medline]
ESPRIT Study Group. Aspirin plus dipyridamole versus aspirin alone after cerebral ischaemia of arterial origin (ESPRIT): randomised controlled trial. Lancet2006; 367:1665 -73[CrossRef][Medline]
|
APPENDIX 1 >References to the 18 identified trials. |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODSRESULTSDISCUSSIONREFERENCESAPPENDIX 1
>References to... |
|---|
Brain EG, Bachelot T, Serin D, et al. RAPP-01 Trial
Investigators. Life-threatening sepsis associated with adjuvant doxorubicin
plus docetaxel for intermediate-risk breast cancer.
JAMA2005; 293:2367
-71
Ciliberto H, Ciliberto M, Briend A, Ashorn P, Bier D, Manary M.
Antioxidant supplementation for the prevention of kwashiorkor in Malawian
children: randomised, double blind, placebo controlled trial.
BMJ2005; 330:1109
-11
Dalziel SR, Walker NK, Parag V, et al. Cardiovascular risk factors after antenatal exposure to betamethasone:30-year follow-up of a randomised controlled trial. Lancet2005; 365:1856 -62[CrossRef][Medline]
DuPont HL, Jiang ZD, Okhuysen PC, et al. A randomized,
double-blind, placebo-controlled trial of rifaximin to prevent travelers'
diarrhea. Ann Intern Med2005; 142:805
-12
Fairbank J, Frost H, Wilson-MacDonald J, Yu LM, Barker K, Collins
R. Spine Stabilisation Trial Group. Randomised controlled trial to compare
surgical stabilisation of the lumbar spine with an intensive rehabilitation
programme for patients with chronic low back pain: the MRC spine stabilisation
trial. BMJ2005; 330:1233
-8
Gardner CD, Coulston A, Chatterjee L, Rigby A, Spiller G, Farquhar
JW. The effect of a plant-based diet on plasma lipids in hypercholesterolemic
adults:a randomized trial. Ann Intern Med2005; 142:725
-33
Grant AM, Avenell A, Campbell MK, et al. for the RECORD Trial Group. Oral vitamin D3 and calcium for secondary prevention of low-trauma fractures in elderly people (Randomised Evaluation of Calcium or vitamin D, RECORD): a randomised placebo-controlled trial. Lancet2005; 365:1621 -8[CrossRef][Medline]
Guillou PJ, Quirke P, Thorpe H, et al. for the MRC CLASICC trial group. Short-term endpoints of conventional versus laparoscopic-assisted surgery in patients with colorectal cancer (MRC CLASICC trial): multicentre, randomised controlled trial. Lancet2005; 365:1718 -26[CrossRef][Medline]
Hakonarson H, Thorvaldsson S, Helgadottir A, et al.
Effects of a 5-lipoxygenase-activating protein inhibitor on biomarkers
associated with risk of myocardial infarction:a randomized trial.
JAMA2005; 293:2245
-56
Jokhio AH, Winter HR, Cheng KK. An intervention involving
traditional birth attendants and perinatal and maternal mortality in Pakistan.
N Engl J Med2005; 352:2091
-9
Korzenik JR, Dieckgraefe BK, Valentine JF, Hausman DF, Gilbert MJ.
Sargramostim in Crohn's Disease Study Group. Sargramostim for active Crohn's
disease. N Engl J Med2005; 352:2193
-201
Linde K, Streng A, Jurgens S, et al. Acupuncture for
patients with migraine: a randomized controlled trial.
JAMA2005; 293:2118
-25
MacMillan HL, Thomas BH, Jamieson E, et al. Effectiveness of home visitation by public-health nurses in prevention of the recurrence of child physical abuse and neglect: a randomised controlled trial. Lancet2005; 365:1786 -93[CrossRef][Medline]
Penny ME, Creed-Kanashiro HM, Robert RC, Narro MR, Caulfield LE, Black RE. Effectiveness of an educational intervention delivered through the health services to improve nutrition in young children: a cluster-randomised controlled trial. Lancet2005; 365:1863 -72[CrossRef][Medline]
Singh BN, Singh SN, Reda DJ, et al. for the Sotalol
Amiodarone Atrial Fibrillation Efficacy Trial (SAFE-T) Investigators.
Amiodarone versus sotalol for atrial fibrillation. N Engl J
Med 2005;352:1861
-72
Smith AD, Cowan JO, Brassett KP, Herbison GP, Taylor DR. Use of
exhaled nitric oxide measurements to guide treatment in chronic asthma.
N Engl J Med2005; 352:2163
-73
Valgimigli M, Percoco G, Malagutti P, et al. for the
STRATEGY Investigators. Tirofiban and sirolimus-eluting stent versus abciximab
and bare-metal stent for acute myocardial infarction: a randomized trial.
JAMA2005; 293:2109
-17
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
This article has been cited by other articles:
|
|
 |
|
  P. Wilson, M. Petticrew, and on behalf of the Medical Research Council's Popula Why promote the findings of single research studies? BMJ, March 29, 2008; 336(7646): 722 - 722. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
