We apologize to Dr. Leahy and the readers of THE Surgeon, and to the editors and readers of the other journals mentioned for our unfortunate choice of words. We did not mean to offend.

To The Editor:

Your authors, in their excellent article(1), refer to trials which were reported in relatively "obscure" journals. On behalf of the journals mentioned, may I, as the Editor of THE Surgeon: Journal of the Royal College of Surgeons of Edinburgh and Ireland, respond. The use of the adjective "obscure" is an interesting one. Presumably the authors did not mean to imply that our journals were not prominent or famous, were relatively unknown, or not clearly seen or easily distinguished.

THE Surgeon circulates to over twenty thousand surgeons internationally and is a fully indexed journal with an impact factor 0.99. Obscure, I rather think not!

The author did not receive any outside funding or grants in support of his research for or preparation of this work. Neither he nor a member of his immediate family received payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity. No commercial entity paid or directed, or agreed to pay or direct, any benefits to any research fund, foundation, division, center, clinical practice, or other charitable or nonprofit organization with which the author, or a member of his immediate family, is affiliated or associated .

Reference:

1. Cowan J, Lozano-Calderon S, Ring D. Quality of prospective controlled randomized trials. Analysis of trials of treatment for lateral epicondylitis as an example. J Bone Joint Surg Am. 2007;89:1693-1699.

The welcome comments of Drs. Poolman and Bhandari further illustrate the passion, enthusiasm, and detail that is increasingly placed not only in performing good science, but also in evaluating it. If science can be characterized as "organized skepticism", then the time when seemingly scientific pronouncements could go without challenge is clearly fading.

To The Editor:

With interest we read the manuscript by Cowan et al.(1). This manuscript is another example of the increased awareness among orthopaedic surgeons of the quality of reporting in trials. We further applaud the authors’ effort to highlight importance of individual validity criteria in assessing the quality of a randomized trial. We wish to provide two suggestions: 1) Use of overall summary scores may not be as helpful in interpreting the data and 2) newer checklists that focus on non- pharmacological therapies may be of interest in future research in this area.

Summarizing scores of checklists has limitations. Juni et al. have elucidated this issue(2). When a mixture of pharmaceutical and non- pharmaceutical trials are evaluated, as Cowan and co-workers(1) did, a highest possible quality non-pharmaceutical RCT will score lower than a highest possible quality pharmaceutical RCT. Therefore, we and others recommend reporting individual checklist items rather than overall summary scores(3). The use of thresholds may also skew the direction of results and may lead to false conclusions in a meta-analyses(2). Furthermore, Juni et al. discouraged the use of individual scales as absolute and objective measures of trial quality and noted "relevant methodological aspects should be identified, ideally a priori, and assessed individually" (2). The authors did provide some data on the proportion of trials that met individual criteria and we suspect space constraints for journal publication may have limited complete tabular breakdowns of the proportion of RCTs that met each criterion of the Coleman and CONSORT checklists. This would be very helpful information to further interpret their findings. Perhaps the Journal could publish these findings online?

Ideally, scales that are used to measure the quality of reporting of surgical trials should be tailored to the maximal possible quality, rather than to a unique gold-standard quality(2,3). Therefore, the Cochrane Collaboration's handbook advises to describe aspects of critical appraisal separately and to avoid summarizing results(4). Our previous study confirmed the variability of scores across each item of the Cochrane reporting quality assessment tool reviewing RCTs published in the Journal (3).

We would also like to suggest checklists that have been developed to evaluate non-pharmaceutical studies(5,6). Although, frequently used, these checklists themselves lack a validation process. The Coleman Methodology Score is one example, especially if this score was modified. Utilization of modified scores without revalidation can skew results(7). Using a standardized checklist tailored to non-pharmaceutical trials, the CLEAR NPT(5) may facilitate comparison of different studies clarifying the quality of reporting in surgical trials including orthopaedics. This checklist was developed to overcome the limitations of the CONSORT checklist(5). Currently this checklist is under ongoing evaluation.

The authors did not receive any outside funding or grants in support of their research for or preparation of this work. Neither they nor a member of their immediate families received payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity. No commercial entity paid or directed, or agreed to pay or direct, any benefits to any research fund, foundation, division, center, clinical practice, or other charitable or nonprofit organization with which the authors, or a member of their immediate families, are affiliated or associated.

References:

1. Cowan, J., Lozano-Calderon, S., and Ring, D.: Quality of prospective controlled randomized trials. Analysis of trials of treatment for lateral epicondylitis as an example. J Bone Joint Surg Am. 89:1693- 1699, 2007.

2. Juni, P., Witschi, A., Bloch, R., and Egger, M.: The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 282:1054- 1060, 1999.

3. Poolman, R. W., Struijs, P. A., Krips, R., Sierevelt, I. N., Lutz, K. H., and Bhandari, M.: Does a "Level I Evidence" rating imply high quality of reporting in orthopaedic randomised controlled trials? BMC Medical Research Methodology. 6:44, 2006.

4. Higgins, J. P. T., Green, S., and Editors: Cochrane Handbook for Systematic Reviews of Interventions 4.2.5 [updated May 2005]. 2006.

5. Boutron, I., Moher, D., Tugwell, P., Giraudeau, B., Poiraudeau, S., Nizard, R., and Ravaud, P.: A checklist to evaluate a report of a nonpharmacological trial (CLEAR NPT) was developed using consensus. J Clin. Epidemiol. 58:1233-1240, 2005.

6. Jacquier, I., Boutron, I., Moher, D., Roy, C., and Ravaud, P.: The Reporting of Randomized Clinical Trials Using a Surgical Intervention Is in Need of Immediate Improvement: A Systematic Review. Ann Surg. 244:677-683, 2006.

7. Poolman, R. W., Struijs, P. A., Krips, R., Sierevelt, I. N., Marti, R. K., Farrokhyar, F., and Bhandari, M.: Reporting of outcomes in orthopaedic randomized trials: does blinding of outcome assessors matter? J Bone Joint Surg Am. 89:550-558, 2007.