To The Editor:
The editorial "The Millennium Enigma: More Is Less" (81-A: 1509-1510,
Nov. 1999), by Dr. Cowell, is critical of the lack of validity of
scientific papers published by the electronic media as well as journals
that do not have rigid peer review. While I agree with this point
of view, The Journal of Bone and Joint Surgery with
its peer-review system has not lived up to the standards that it
supports.
For over two decades, much has been made of the fact that bias
plays a major role in creating worthless science. In 1979, Sackett8 described thirty-five different types
of bias that can adversely impact scientific observations.
In 1981, in a series of articles, the Canadian Medical
Association Journal extensively addressed the problem of
qualitatively evaluating information from the medical literature4. The authors of these articles advised
physicians on the most efficacious way to keep up with the ever-enlarging
body of scientific literature. They described why we read scientific
articles, identified factors creating bias, defined the potential
value of different types of studies, and showed how flawed studies
confound conclusions. They rated research designs as to probable validity,
with the first in order being the randomized clinical trial, followed
by the cohort study, the case-control study, and, finally, the case
series.
The prospective, randomized, blinded study was thought to be
the best protocol upon which to base valid research1. If, for practical or ethical reasons,
such a protocol could not be carried out, The Journal of
Bone and Joint Surgery described and supported an alternative
method4,7.
Now, in the very issue in which The Journal criticizes others
and defends its own peer-review system (81-A: Nov. 1999), only one
of the nine original clinical studies meets the highest level of
validity, and that paperis from
a foreign country6. This is not
to say that some of the other articles are valueless. Yet, while
they are not simple case studies, none of the eight rises to the
validity level of a case-control study other than by using selected
historical and/or literature controls. Shouldn't a journal that
prides itself on its excellence demand more from authors wishing
to be heard in its pages?
To its credit, The Journal of Bone and Joint Surgery has
encouraged better science by publishing articles and editorials
on the scientific method3,5. Furthermore, The
Journal has taken actions such as requiring (1) specific methods
of dealing with statistics, (2) a two-year minimum follow-up, and
(3) statements regarding the source of research funding. However,
the quality of the papers typified in the November 1999 issue suggests
that these measures have not been enough.
To disparage without suggesting ways of improvement may be seen
as hypocritical. The following remedies might be considered:
1. Determine steps that other journals have taken to improve
the quality of papers submitted for publication.
2. Establish minimum standards requiring clinical papers to be
prospective and randomized or to have an appropriate comparison
cohort, which should be blinded whenever possible. Exceptions would
need to be justified. The implementation of such a policy could
be in the future, giving researchers a few years in which to comply.
3. Require prospective submission of the research design for
all papers intended for publication. This would include a description
of the statistical tests that would be used to validate the findings.
This protocol would be kept on file and would be used only if the
resultant paper were submitted. It would then be one of the bases
for review of the final product.
4. Fully acquaint The Journal's reviewers with
the requirements established by the editors, including specific
protections against bias and a detailed description of the statistical
criteria needed for validating the paper that is being reviewed.
5. Hire a full-time statistician to provide assistance throughout
the editorial process.
In conclusion, the past and present editors and boards have for
the most part served orthopaedists and their patients well. However,
in their editorial of 1985, Dr. Cowell and Dr. Curtiss3 stated: " it is not too much to hope
that in the future all journals will receive articles for publication
that, when applicable, meet the standards of the randomized therapeutic
trial." Further, in 1997, Dr. Clark2 stated:
"I believe that investigators should be encouraged to consider performing
such prospective studies if it is at all feasible."While the former wish is unfulfilled,
the latter admonishment is generally ignored.
Isn't it now time for The Journal of Bone and Joint Surgery to
require scientific research of the highest quality and to set the
standard of excellence for all print and electronic media?
Lawrence H. Gordon, M.D.
Division of Orthopaedics
John A. Burns School of Medicine
University of Hawaii-Manoa
1356 Lusitana Street, #614
Honolulu, Hawaii 96813-2478
H. R. Cowell replies:
I thank Dr. Gordon for his interest in my editorial and am pleased
that he agrees that the peer-review process is an important one.
However, the major focus of Dr. Gordon's letter is his displeasure
with the review process of The Journal, not the
lack of peer review on the Internet. Dr. Gordon does not think that
the peer-review process of The Journal is adequate,
despite the fact that fewer than 15 to 20 percent of the manuscripts
submitted to The Journal are accepted for publication.
The Journal is, of course, pleased to publish
prospective, randomized studies, but few manuscripts submitted to
any journal meet these criteria. Moreover, while such manuscripts
may be considered ideal, other types of manuscripts do provide useful
information to the reader.
Dr. Gordon suggests a number of ways to improve the peer-review
process, and his suggestions are appreciated. First, it is always appropriate
to learn from other journals, and I regularly discuss such issues
with editors of other journals. Second, The Journal does
have specific standards, as Dr. Gordon notes; to set the bar higher
would result in an even higher rejection rate. Third, The
Journal is not the appropriate agent to decide what studies
should be done, nor should it be. Authors should be fully aware
of the need for approval of a project by an appropriate institutional
review board and the importance of having a statistician evaluate
a project before it is begun. Fourth, reviewers and editors are
aware of the statistical criteria required to validate a study.
Fifth, The Journal has a deputy editor who has
a background in statistics.
Finally, The Journal does require that scientific
research be of the highest quality. I am sorry that the material
published in The Journal does not meet Dr. Gordon's
standards.
Henry R. Cowell, M.D., Ph.D.
Editor Emeritus
The Journal of Bone and Joint Surgery
Needham, Massachusetts