JRSM >
Volume 99, Number 12 >
Pp. 625-627
Series |
Cochrane Vaccines Field and Cochrane Acute Respiratory Infections Group, 00061 Anguillara Sabazia, Rome, Italy
E-mail: Jefferson.tom{at}gmail.com
AN INTRODUCTION TO BUFALA SPOTTING
During the year, busy healthcare workers have little time for reading or reflecting. This fact conflicts with the need to keep up-to-date with what is going on in medicine and to track the avalanche of paper, audio and electronic inputs we receive. The obvious starting point is a topic of interest, but we also need to select something which has some kind of credibility, either scientific or ethical or both (although some tell me there is no difference between the two). So we now have two linked problems: the quantity and the quality of the scientific works that we read.
|
Alas, not everything is so easy in life, research or medical practice. Often what seems is not what is. Let's start from quality control mechanisms as they apply to the publication of research papers and then move on to the issue of quantity. Quality (essentially editorial peer review, publishing's main quality control mechanism) is the key to understanding the issue of quantity.
EDITORIAL PEER REVIEW
The custom of sharing scientific production with one's peers or friends has a long and honourable tradition.1 In modern science it probably goes back to the age of reason, to Boyle's physics experiments in Oxford just after the end of the War of the Three Kingdoms, the English Civil War. The design and results of the experiments were shared with Boyle's peers and friends. In medical practice the use of one's ‘peers’ is firmly rooted in the time-honoured custom of the bedside consultation, in which the treating doctor, usually a generalist, calls one or more eminent specialist colleagues to elicit opinions on a difficult case. The masterful Collodi gives us a precious insight into the dynamics of the consultation in his masterpiece ‘The Adventures of Pinocchio’ (Figure 1).
|
Now, this very gentle representation of what goes on is still valid today, as peer review has evolved little in the last 200 years.1 Electronic submissions and evidence-based medicine have not changed the essence of peer review and the Fairy, Crow, Owl and Talking Cricket are still working their way through thousands of Pinocchios. But the world around them has changed and in the fiercely competitive research environment of today they are struggling and failing to do an effective job. As each one of us has a nefarious tale to tell about peer review—and the number of retracted fabricated papers mounts—we all suspect that something is not quite right. When the results of our Cochrane review on the evidence of the effects of peer review were published in JAMA in 2002, there was a deafening silence. The review showed that there was flimsy proof of the effectiveness of peer review and in a linked piece of work the authors also reported that its objectives were unclear.2,3 Updates of the reviews have failed to find evidence to change the conclusions, so has a similar review into grant application peer review.4 The BMJ Christmas issue of 2003 published a board game called ‘Get Peered!’ (Figure 2), which parodied the situation, showing the pitfalls of modern publishing: from plagiarism to salami slicing (or redundant publication, as it is more correctly known, one of the many ways of padding one's impact factor).5 The aim of the game is to become a Peer (or Lord) as soon as possible, by any means. Although a parody, the game had a serious message: publishing is a lottery.6
The main problem is probably the mediocrity of what is published today, and the extreme cases of plagiarism or data fabrication appear still, thankfully, relatively few.7-12
We are left with a more complex problem of summarizing a lot of research of variable quality. From the practical point of view of a busy clinician, the problem becomes: whom do I trust?
QUANTITY VERSUS QUALITY
How can we reconcile the contrasting requirements of selecting up-to-date, good, reliable knowledge from the huge range on offer, and very limited reading time?
Twelve months ago a book called ‘Attenti alle Bufale’ (‘Beware of red herrings’, in Italian) was published (www.attentiallebufale.it).13 In Italian slang, bufale are red herrings but in real life bufale are female water buffaloes. Their milk is the basic ingredient of the famous mozzarella cheese (mozzarella di bufala). The book's rationale was to publicize a series of basic quick instruments based on evidence-based medicine (EBM) and its practical applications. The instruments were meant to help folk who realize that a very high percentage of what they read or hear is a bufala, to spot the bufale and chuck them where they belong, thereby limiting the intellectual damage. Second, the book tried to make EBM relevant to busy health-care workers who have little time to waste.
The chapter presenting the instruments is called ‘Bufala spotting’ and it is structured in the following manner. For each type of communication (lecture, editorial, research paper, etc.) there is a quick version of the instrument, designed to be applied in two minutes. Next to it are longer and more detailed versions for those who have more time (a lot more time in some cases), with the rationale for the approach for each instrument. Next to each section there are a number of bufala heads to signify the danger of being taken for a bufala ride: four bufala heads mean extreme danger, one bufala head means low danger.
The instruments should not be taken as an absolute gospel but the following should be borne in mind:
Everything and anything we read or hear in biomedical sciences has to be approached critically. This is the fastest and most reliable universal instrument. Trust but verify, as Ronald Reagan used to say. A critical mentality is developed and nurtured, it does not grow overnight.
All quick instruments work, but they help you to distinguish what needs to be binned from what probably requires more careful consideration. The quick instruments are based on experience, but some of the elaborate ones have been validated and are beginning to make an impact on the quality of published science.
If you do not want to bother typing in all the URLs of the full instruments, visit www.attentiallebufale.it, where you have quick drill-through links to the various instruments.
The JRSM will present the instruments of bufala spotting, beginning with how to assess research papers. If you do not like the instruments, develop your own. If you do like them, try them. Either way, let us know what you think. We hope that you will enjoy using the instruments and they will perform a useful function and save you time. We also hope that will be interpreted for what they are: an evolution of the EBM paradigm as they try to use EBM principles, as daily tools.
Footnotes
This is the first in a series of articles on making evidence-based medicine work for you. The series is based on the book ‘Attenti Alle Bufale’ by Tom Jefferson.
REFERENCES
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
This article has been cited by other articles:
|
|
 |
|
  C. R Pearson Curious consent J R Soc Med, December 1, 2008; 101(12): 578 - 578. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
