Commentary & Perspective

Commentary & Perspective on
"The Validity of Claims Made in Orthopaedic Print Advertisements"
by Timothy Bhattacharyya, MD, et al.

Commentary & Perspective by
Marc F. Swiontkowski, MD*,
Department of Orthopaedic Surgery, University of Minnesota, Minneapolis, Minnesota

Bhattacharyya et al. have made an important contribution to the orthopaedic community in studying the issue of advertisement claims made in orthopaedic scientific journals. While claims made in advertisements by the pharmaceutical industry have been studied, claims made in advertisements primarily for implants used in orthopaedic surgery have not been studied as far as I know. The orthopaedic community at large will be interested in the findings of this work because we all see (and read) these advertisements (often in passing) as we attempt to keep current with our journals.

The authors randomly selected fifty statements from fifty advertisements chosen from six peer-reviewed orthopaedic journals in a systematic and well-thought-out fashion. Statements within these advertisements were selected with use of a similarly well-thought-out randomization scheme. The researchers then approached the manufacturer with a request for supporting data. I assume that when the industry contact was made, the manufacturer was not told that the request was part of a study on the topic of advertisement claims, as that obviously would have potentially influenced the response rate and quality. The supporting data were reviewed and edited to remove identifiers, and then three senior orthopaedic surgeons weighted the data. Fifty percent of the advertisements had supporting data from clinical studies and 34%, from laboratory studies. Only 36% had supporting data from a peer-reviewed published source. Sixteen percent had no scientific data to back up the statement in the advertisement. Thankfully, the industry's nonresponse rate was relatively low (four of fifty). Although the rating system has not been validated, the interobserver agreement was good—that is, 0.78 for quality and 0.72 for supporting data. There was a strong correlation between the quality of data and the strength of support, as would be anticipated. Fifty-two percent of the claims were thought by all three reviewers to be not strongly supported, while only 14% were thought to be well supported.

We in the orthopaedic community assume that a print advertisement provides a credible source of information on which to base treatment decisions. I would point out that this is an assumption which, to my knowledge, has not been documented by appropriate survey research. It would be interesting to know whether the orthopaedic industry has undertaken marketing studies on the impact of print advertisements, or whether they have data on the influence of print advertisements on treatment decisions.

In terms of the methodology of this project, the six peer-reviewed journals had different advertising profiles with different target audiences. The sequence in which the advertisements were chosen, according to the randomization scheme, therefore impacted the subspecialty orientation of the advertisement. Because of this impact, I do not think that we should place a great deal of weight on the percentage of advertisements that relate to different subspecialty areas within orthopaedics, rather we should focus on the generalizability of the results.

A methodological concern is the background of the three senior orthopaedic surgeons who reviewed and rated the supporting data. All three are established, well-recognized academic orthopaedists. This type of background, in general, is not the target audience for industry, as 90% of the orthopaedic care in this country is delivered in nonacademic health centers. The more appropriate group to include would have been general orthopaedic surgeons who likely have a different orientation in reviewing claims made in print advertisements. Additionally, these three orthopaedic surgeons were co-investigators who were aware of the hypothesis of the study and therefore may have graded the evidence more strictly than they would have if they had not known the hypothesis. The study would have been better if the raters had been unaware of the purpose of the study as it would have removed the source of detection bias.

As the authors point out, the rating system is not validated. The second issue that they had to judge was whether the statement in the advertisement was supported by enough data to be used in clinical practice. These three individuals are subspecialists, and one would question whether an arthroplasty surgeon can make a relevant judgment regarding the use of a trauma implant or a suture anchor technique for ligament reconstruction.

Perhaps more importantly, the criteria used to determine whether data are suitable for publication in the peer-reviewed literature is subject to scrutiny. This reviewer would be willing to bet that all three of the academic senior orthopaedic surgeons have experienced rejection of their "best" academic work by a peer-reviewed publication for reasons that are not exactly clear. There is a substantial amount of literature on publication bias, which is influenced by many issues other than study quality. Therefore, this assumption, I think, is subject to criticism.

The fact that the manufacturers that sponsored all fourteen advertisements that claimed to have data on file actually produced the data when it was requested speaks to the professionalism of our industry partners.

The fact that, for 52% of the claims, all three reviewers agreed that the data were not strong enough to be used in clinical practice is of grave concern to the orthopaedic community. As the authors note, in the Discussion section, this finding may be simply a reflection of the poor quality of the literature in orthopaedic surgery. Lack of high-quality evidence is common in surgical subspecialties because of the difficulties of performing high-quality research designs in procedural-based endeavors. In the last fifteen years, the orthopaedic community has made major strides in improving our clinical research and in increasing the percentage of controlled trials that are published in our peer-reviewed literature each year.

This issue has important ramifications for the field of orthopaedic surgery, and the authors are to be congratulated for quantifying the quality of the claims made in advertisements in orthopaedic peer-reviewed journals. The orthopaedic community and industry need to work together on this issue to improve patient care. As orthopaedic surgeons, we must exert our professional influence to focus on issues of greatest relevance to patient care and the outcomes of that care. Our profession is in great need of multicenter cooperative trials that focus on the most important issues of patient care. If we collectively approach industry in a unified manner, there is ample evidence that it will support these initiatives. At the same time, we need to engage industry as a profession to provide what it needs, which is a professional opinion regarding issues of innovation and bioengineering design improvement.

Finally, the issue of the quality of our supporting literature is being addressed by The Journal of Bone and Joint Surgery. Beginning in 2003, we instituted a policy to report to the readers, in the abstract of each study, the level of evidence. Our approach is designed to educate the orthopaedic community with regard to the concepts of the strength of experimental design as it should relate to the impact on clinical practice. We are in the process of validating this scaling process and will inform the readership of our findings after the system has been in place for one year.

*The author did not receive grants or outside funding in support of his research or preparation of this manuscript. He did not receive payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity. No commercial entity paid or directed, or agreed to pay or direct, any benefits to any research fund, foundation, educational institution, or other charitable or nonprofit organization with which the author is affiliated or associated.