To the Editor:

In “Doubling the Impact: Publication of Systematic Review Articles in Orthopaedic Journals” ,Bhandari et al. (1) reveal the unsatisfactory nature of the majority of reviews published in fifteen orthopaedic journals. They refer to their previous study of the methodological rigor of orthopaedic meta-analyses[1], where the findings of generally poor methodological quality applied to reviews published in journals but not to Cochrane reviews[2,3].

Obviously, Cochrane reviews could not be included in Bhandari et al., despite the frequency with which they are cited, as these are not currently indexed in the citation indexes. However the authors did allude to the fact that “The Cochrane Collaboration… has published in excess of 1000 systematic reviews since its inception"[4].

The number of rigorously conducted and maintained systematic reviews in the Cochrane Database of Systematic Reviews now approximates 2000[5]. At least 155 of these systematic reviews generated by members of the Back, Musculoskeletal, and Musculoskeletal Injuries Collaborative Review Groups are of direct relevance to orthopaedics. This contrasts to the seventeen systematic reviews identified by Bhandari et al. in the fifteen journals with high Science Citation Index impact factors for orthopaedics. It should be noted that four out of the eight systematic reviews identified in Spine are print versions of Cochrane reviews.

As acknowledged by Bhandari et al. impact factors are an imperfect measure[6]. While we endorse Bhandari et al's call for improvements in the methodological rigor in the conduct and reporting of reviews in journals, we caution against the use of impact factors as a major determinant of quality of individual reviews/papers[6]. Finally, we would like to encourage authors and journal editors alike to improve not only the quality of the design and reporting of review articles[7] but also of the primary research underpinning these reviews.

Helen H. Handoll, DPhil

Lesley D. Gillespie, MMedSci

William J. Gillespie, FRACS

Rajan Madhok, FRCS

Martyn J. Parker, MD

We are all members of the Cochrane Musculoskeletal Injuries Collaborative Review Group

Teesside Centre for Rehabilitation Sciences, The James Cook University Hospital, Marton Road, Middlesbrough, TS4 3BW, UK. E-mail address: h.handoll@ed.ac.uk

1. Bhandari M, Morrow F, Kulkarni AV, Tornetta P 3rd. Meta-analyses in orthopaedic surgery. A systematic review of their methodologies. J Bone Joint Surg Am. 2001; 83:15-24.

2. Parker M, Gillespie L, Gillespie W, Handoll H, Madhok R, Morton L et al. Systematic reviews, meta-analyses, and methodology. J Bone Joint Surg Am. 2001; 83:1433-4.

3. Bhandari M, Morrow F, Kulkarni AV, Tornetta P 3rd. Systematic reviews, meta-analyses, and methodology. J Bone Joint Surg Am. 2001; 83:1434-5.

4. Chalmers I. The Cochrane Collaboration: preparing, maintaining, and disseminating systematic reviews on the effects of health care. Ann NY Acad Sci. 1993;703:156-65.

5. The Cochrane Library, Issue 2, 2004. Chichester, UK: John Wiley & Sons, Ltd.

6. Kurmis AP. Understanding the limitations of the journal impact factor. J Bone Joint Surg Am. 2003; 85:2449-54.

7. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Lancet 1999; 354:1896-1900.