JRSM >
Volume 99, Number 9 >
Pp. 470-472
1 Medical Student, Department of Surgical Oncology & Technology, Imperial
College London, UK
2 Consultant Cardiac Surgeon, Department of Surgical Oncology & Technology,
Imperial College London, UK
3 Professor of Primary Care Research & Development, Division of Community
Health Sciences: GP Section, University of Edinburgh, Scotland, UK
Correspondence to: Aziz Sheikh E-mail: aziz.sheikh{at}ed.ac.uk
 | SUMMARY |
|---|
 TOP SUMMARY INTRODUCTIONMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
To investigate the assertion that the methodological quality of studies conducted in surgery is in general poor and to assess the possible impact of new policy developments in the US, we sought to compare the number and proportion of published randomized controlled trials and systematic reviews in the leading two US and UK general surgical journals. Two reviewers systematically and independently hand searched all issues of these journals over a 12-month period to identify randomised controlled trials and systematic reviews.
Design Systematic searching and independent abstraction of data from all volumes of the top two general surgical journals published in the USA and the UK in 2004.
Setting 519 original reports in UK journals and 616 original reports in USA journals.
Main outcome measures Number and proportion of randomized controlled trials and systematic reviews.
Results Overall, the proportion of randomized controlled trials in all four journals was 5.6% (95% confidence interval [CI] 4.4-7.0) and 5.2% (95% CI 4.1-6.7) for systematic reviews. For journals published in the UK 29/519 (5.6%) of the publications were reports of randomized controlled trials, and for the USA journals this figure was 34/616 (5.5%); odds ratio [OR]=0.99, 95%CI 0.6-1.6; P=0.96. Systematic reviews were significantly more commonly reported in the UK journals: UK 37/519 (7.1%) versus USA 22/616 (3.6%); OR=0.48, 95%CI 0.3-0.8; P<0.01.
Conclusions The concerns expressed almost a decade ago remain valid: there are still very few reports of randomized controlled trials and systematic reviews published in leading USA and UK surgical journals, with relatively little difference between these countries in the proportion of reported studies employing these designs. The American College of Surgeons initiative has yet to make an impact.
| INTRODUCTION |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
Most studies of operations have historically been retrospective case series, with randomized controlled trials accounting for less than 10% of the total.6-8 In cases where trials have been performed, these have often been small and poorly designed, this also leading to concerns about interpreting findings as their design affords them unwarranted credibility.9 Based on these and other data it is estimated that treatments in general surgery are half as likely to be based on rigorous evidence as treatments used in internal medicine.8,10
The American College of Surgeons has since 2000 been undergoing major restructuring, this includes the establishment of an Office of Evidence Based Surgery (now renamed to Continuous Quality Improvement) designed to facilitate, by providing the administrative and infrastructure support needed, the conducting of rigorous studies including randomized controlled trials and systematic reviews.3 Given this major boost, we hypothesized that there would exist greater momentum to undertake and publish studies employing these designs in the USA when compared to the UK. To investigate this, we compared the proportions of randomized controlled trials and systematic reviews published in leading general surgical journals in the UK and the USA.
| METHODS |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
Data were independently extracted and appraised the data onto a pre-piloted sheet. Our aim was to describe the designs employed and each original research study was categorized as employing one of the following approaches:
Any disagreements were resolved through discussion, with referral to a third member of the team (AS) to arbitrate if necessary.
We excluded all other publications including editorials, non-systematic reviews, panel discussions, short communication letters with no original hypothesis/data, book reviews, errata, commentaries, critique, obituaries, presidential addresses, abstracts and Minerva.
Descriptive statistics were used to determine the proportion of the various
types of studies reported and the 
2 test was used to compare
difference in the proportions of these studies between USA and UK
journals.
Previous studies have shown that randomized controlled trials in UK journals comprise approximately 5% of the general surgical research literature.7,8 In the USA, we estimated this figure might be about 20%.11 In order to have 80% power, at the 5% significance level, of detecting a difference of this magnitude, we calculated that we would need to identify approximately 55 reports of randomized controlled trials in each country. A calendar year's analysis of two journals from each country was deemed sufficient to yield these numbers of trials.
| RESULTS |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
|
In the UK journals, 29/519 (5.6%) of publications were randomized
controlled trials, this being very comparable to the picture in USA journals:
34/616 (5.5%); OR=0.99, 95% CI 0.59-1.64; 2=1.01,
P=0.96.
Turning to systematic reviews/meta-analysis, 37/519 (7.1%) of publications
in UK journals employed this design, which is significantly higher than that
in the USA publications: 22/616 (3.6%); OR=0.48, 95% CI 0.2-0.8;
2=15.4, P50.01.
In the UK journals, 178/519 (34.3%) of studies were classified as
analytical, compared with 294/616 (47.7%) of the same in the USA journals
(OR=1.75, 95% CI 1.4-2.2, 2=22.8, P50.001). In
contrast, descriptive/qualitative studies were more common in UK journals: UK
116/519 (22.3%) versus USA 43/616 (12.8%); OR=0.51, 95% CI 0.4-0.7;
2=25.4, P50.001.
| DISCUSSION |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
Strengths and limitations of the work
The main strength of this work are the explicit criteria for selecting
journals for inclusion, the comprehensiveness of the searches of journals over
the study period and the independent assessment of studies to ensure validity
and reliability.
The limitations need to be appreciated. Our study has provided a 1-year generalized snapshot view of the state of surgical research by analysing original research reports in only four surgical journals published in the UK and USA. It provides some insight into trends over time;7,8 but we did not assess the trends in study size or the quality of the studies by using measurement scales such as those employed by the Cochrane Collaboration, as these were not reported for the original studies thus making direct comparison difficult.12 The different impact factors of the various journals incorporated into the study may also influence the outcome as different journals have different criteria on the particular types of studies that they will accept for publication. Surgical studies are also published in a range of other surgical journals and also in some generalist medical journals. We have in this study been unable to assess whether the picture uncovered in the present analysis is representative of the broader state of the general surgical literature. Furthermore, there are instances in which UKled studies are published in USA journals and vice versa. However, given that the impact factors for the USA journals were higher than those from the UK, any systematic misclassification error would have been likely to bias against the null hypothesis in question. It may also reasonably be argued that given the time it takes to secure funding and obtain ethics approval, execute, analyse and publish trial data, it was always unlikely that at this relatively early stage of its existence the Office of Evidence Based Surgery would have had the time to make much of an impact: that said, our work does provide a baseline from which to assess future progress of this key new initiative.
Considering the findings in relation to previous work
Several reasons have been cited for the poor proportion of randomized
controlled trials in surgical journals. Unlike in medicine, where clinical
trials are relatively easy to carry out using pharmacological interventions,
surgical trials are difficult to conduct. Most surgeons would rather avoid the
uncertainties, paperwork and hassle. Patients too may be reluctant to
participate in these studies as they may have a preference for a particular
technique, e.g. laparoscopic surgery. In addition, there are financial
challenges—pharmaceutical companies do not provide funding. Also, in the
USA the government imposes less stringent regulations on new operations and
technologies than it does on new drugs. One can, therefore, appreciate
surgeons' reluctance to undertake the large and administratively complex
trials that are often needed in order to provide a secure evidence-base for
procedures.2
Implications for research
Although the medical community accepts conventional surgical randomized
controlled trials as ethical, some surgeons may have ethical problems with
enrolling patients in a trial when they know they may have to do a procedure
with which they feel inexperienced. This problem does not arise to the same
extent in expertise-based randomized controlled trials because surgeons
perform only the procedures in which they have established
expertise.13 This
may, therefore, represent a more acceptable strategy than employed hitherto to
weaning academic surgeons off their unhealthy addiction to comic
operas.5
| CONCLUSIONS |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
| Footnotes |
|---|
Acknowledgments AS conceived this study, formulated the study protocol, and led the writing of the manuscript. SSP and RT extracted data. SSP was responsible for data analysis and write-up. TA helped to interpret the results. AS is guarantor.
| REFERENCES |
|---|
|
TOPSUMMARYINTRODUCTIONMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
