Dr. R. Buckley responds RE: A Bold Step in the Right Direction for Orthopaedic Clinical Research

Thank you for your kind words relating to our paper. We agree with a number of your comments and will try to answer. Our statistician has been most helpful in supporting us with our answers. The use of the pre-randomization strategy(1) is in our minds, an underused type of strategy. It works nicely in trials where there is operative versus nonoperative strategy. Where there is an operative vs. operative strategy (Widget 1 vs. Widget 2) it is really not difficult to obtain consent for most prospective randomized trials. In trials however, that are operative vs. nonoperative, this strategy is often helpful. We do not agree that this particular pre-randomization strategy can be viewed as unethical. Certainly as much information can be given in the education of a patient with this strategy as in any other strategy. There simply has to be the recognition that if surgery was an option, the patient would be willing to proceed in that direction. The calcaneal fractures certainly are a perfect example of this where both operative and nonoperative strategies can be used. When in 1991 the study started, it was obvious that nonoperative care or operative care were equal in there efficacy.

The most oft asked question in clinical trials recently, is whether or not an intention to treat analysis was performed. We analyzed all the data we had, on all the patients for whom any follow up material was available. So, of the eligible patients, we had follow up data on 309 patients. We are not sure how we could have obtained any data on those without follow up information. We would have included those who were withdrawn in the analysis (as per an intention to treat) except we did not have any information in the data base on them. We did however, come up with a very strong study, which can be found in the Canadian Journal of Surgery (2). This particular study dealt with those patients who are lost to follow up in our paper. There are not too many prospective randomized trials that go to the extent of actually writing up a “lost to follow up paper” which deals with all of their patients who did not come back for the conclusion of the period in the randomized trial.

We disagree completely with your statement that we did not define a primary outcome. This was very clear that we were looking to see which treatment group would have the best clinical outcome. This seems very clear to us. It was interesting in this study, because there were no validated outcomes when we started the study, we actually validated our own using all techniques that are standard for validating outcomes. You have picked up an interesting point that prospective trials often go on for a long period of time and that they grow as they mature. We were able to validate an outcome with the use of patients and proceed to use this along with the SF-36 which was just being defined as we started our study. We certainly knew that we would require a clinically relevant difference. Because the literature was so poor beforehand (there is a lovely paper on the history of calcaneal fractures by Crosby(3) there were no outcome measures before 1991 used in identifying calcaneal outcomes. In relation to our assigning of "clinically relevant difference" we took our interim analysis and noted that we would not identify a clinically significant difference unless we increased our numbers by four-fold and we proceeded to do such. We do not think that we changed the question to fit the data but only included enough patients to demonstrate just how difficult this particular question is. We do not agree with your statement that "if one cannot demonstrate a meaningful difference between two therapies, one should not start looking for clinically meaningless difference that can be proven numerically". Our study in fact, is a negative study. It shows no difference between the two groups.

As this was one of the first studies in this particular area that would be truly prospective and randomized, we decided to evaluate more than one dependant variable and wanted to firstly describe the covariates of surgical success as related to patient demographics, WCB status, etc. This approach allowed us to generate hypothesis that could be evaluated by others and also to describe to our peers the factors that influenced outcome.

In relation to your comments regarding independent and dependant variables, we bring to mind that statistics significance is truly clinical significance. When odds ratio exceeds two and the confidence interval excludes the null value, there is some evidence that findings are clinically relevant. All significant odds ratios in the logistic regression models are in excess of two, suggesting clinically relevant findings. Much of the raw data is presented in the tables so that the reader can determine for themselves what a clinical meaningful difference is for them, example, odds ration of 8.09 for WCB - those on WCB were 8X more likely to have lower values on the SF-36 than those who were not WCB - is this clinically meaningful? This certainly allows the public, including yourself, to have thoughts on the subject. We feel that mortality is a very hard outcome, which many epidemiologists would argue is not really relevant in our area of medicine; and morbidity, although the concept is "harder to measure" is really key.

We agree completely that the data needs to be interpreted with regard to clinical judgment and clinical meaning and not by epidemiology alone. That is why team work is so important and we have included multi-centre sites and many surgeons with experience and a PhD statistician. In relation to your comments on the categorization of SF-36, etc, at the mean - this is a great point, one that is often discussed by researchers (and depends generally on your thoughts to "lump" or "split" the generalist vs. the strategist approach). We thank you for highlighting this and we feel that this was important and the idea was to describe in broad terms what influences outcome. The decision to split that data at the mean, was based on the absence of prior data that indicated a different meaningful end point. We would acknowledge this limitation and recognize it as arbitrary and a place for further study or analysis. Interestingly, we are publishing(4) on the SF-36 and it's comparisons to other populations of patients and the calcaneal patient tends to be as bad as any chronic disease that affects a patient medically.

Thanks again for your kind words and we agree with you that it was an enormous task. In fact we are happy that it is over. We have however, made great strides and now in some ways, treat our displaced intra- articular calcaneal fractures differently with a better and more positive result. This particular paper, we feel, will help clarify many problems and will lead to better care of these patients in the future.

Sincerely,

Richard E. Buckley, M.D., FRCS(C) Suzanne Tough, PhD Robert McCormack, MD, FRCSC Graham Pate, MD, FRCSC Ross Leighton, MD, FRCSC Dave Petrie, MD, FRCSC Robert Galpin, MD, FRCSC

REB/mk

1. Zelen, M, A New Design for Randomized Clinical Trials, New England Journal of Medicine, 1979, 300 (22), page 1242-1245 2. Murnaghan, M.L., R.E..Buckley; Lost but not Forgotten: Patients Lost to Follow-Up in a Trauma Data Base, Canadian Journal of Surgery, Vol 45, No.3, pp 191-195, June 2002 3. Crosby, L, Camins, P., The History of the Calcaneal Fracture, Orthopaedic Review, Vol 20, no 6, June 1991, page 501-509 4. Unpublished data

There is no question that surgical intervention for bone and joint disorders profoundly alters their natural history. What is less apparent is whether or not the altered results always represent meaningful improvements over the natural history. As with most surgical specialties, the field of orthopaedics has lagged behind other fields of medicine in the procurement of rigorous evidence, in the form of randomized controlled trials, to guide decisions. The problems of risk, consent, randomization, cost, necessary length of follow-up, and lack of specific outcome measures represent more weighty challenges to the study of orthopaedic surgical interventions than to investigation of many pharmacologic therapies. However, the importance of obtaining rigorously generated evidence to guide clinicians in making rational treatment decisions is also heightened by these same challenges.

The recent article, “Operative compared with nonoperative treatment of displaced intra-articular calcaneal fractures,” by Buckley et al. [1], published in the October issue of the Journal exemplifies the challenges of prospective randomized trials of orthopaedic interventions. The authors must be commended both for the tremendous effort expended and for their willingness to question the efficacy of a surgical intervention that many believe to be the standard of care [2]. While so much of our literature is generated by the drive to instruct others in one’s own methods or report on one’s own results, the scientific willingness to question and to study should never go unnoticed or unappreciated.

One strength of the study is its sheer magnitude. The ability to consent and randomize 424 patients to treatments with vastly different psychological implications is truly astounding. This was, no doubt, aided by the use of a pre-randomization strategy. This strategy, originally described by Zelen in 1979 [3] randomizes eligible subjects prior to obtaining consent. The patient is then consented for the treatment arm to which they were randomized. This aims at increasing recruitment by removing the anxiety of randomization for patients and improving the physician-patient relationship by removing discussion of uncertainty in treatment choice. In his original formulation of the plan, only subjects randomized to experimental therapies would be consented, but the strategy has been modified by others [4] subsequently to include post-randomization consent for each treatment arm of the study. Patients are informed that they were randomized, but the discussion focuses on the treatment arm to which they were assigned. The pre-randomization strategy has never been very widely used and has recently been questioned due to ethical concerns about the possibility of incompletely informed consent. Nonetheless, it represents a means for bridging many of the obstacles to randomizing patients to surgical treatments. While many treatments that are routinely offered to patients have not yet been rigorously substantiated with evidence, surgeons do not usually lay the entire burden of uncertainty upon the patients’ shoulders during the consent process. The shouldering of some of that uncertainty is considered to be part of the responsibility of patient care. Perhaps pre-randomization strategies and the ethics involved need to be revisited and revised by orthopaedic surgeons as necessary for public and institutional approval.

Despite the recruitment power of their pre-randomization strategy, Buckley et al. failed to include an intention to treat analysis. It is not erroneous to include analyses limited to patients who received only their randomized protocol, but these cannot be done to the exclusion of an intention to treat analysis. Such an analysis is fundamental to the validity of a pre-randomization scheme. The drive to question is most usefully linked to carefully constructed, testable hypotheses. One fault of the methods described for this study is the lack of an unequivocally defined primary outcome. The investigators utilized two validated outcomes instruments in patient follow-up, but the failure to define a priori a focused, dependent variable that would be used to evaluate and compare the treatment arms, ultimately hampered analysis of the results. To their defense, the authors recognized ahead of time that their selection of a clinically relevant difference for the power analysis was necessarily arbitrary. That the investigators changed the numerical value assigned to a clinically relevant difference after interim analysis brings into question whether their statistically significant difference was truly clinically significant. It is poor science to change the question only to fit the data. What difference is considered meaningful—or, worthy of a change in clinical practice--should not be affected by the observed differences between two therapies. If one cannot demonstrate a meaningful difference between two therapies, one should not start looking for a clinically meaningless difference that can proven numerically. The defined clinically meaningful difference should be arbitrary only numerically. It should be a difference in the primary outcome that is considered a sufficient benefit to the patient to balance the costs and risks involved in the intervention. When the number involves an outcome such as mortality by any cause, the investigator must determine how many deaths are enough to warrant the use of an alternative therapy. When the primary outcome is a score on a total quality of life outcome instrument, the investigator must determine what difference in quality of life is worth a change in practice. The clinical judgment required for such decisions is why orthopaedic surgeons, rather than epidemiologists alone, study orthopaedic conditions and treatments.

The reported analysis of the results demonstrates another missed chance to apply clinical judgment to study design. Trends and distributions, as part of the information provided by continuous variables, can add significantly to understanding, but the process of decision-making depends on lumping and splitting variables into categories. It is therefore a rather common practice to categorize independent or dependent continuous variables according to arbitrarily limited groups. Two principles are wisely followed in this process of categorization. First, the arbitrary limits should be determined prior to data collection. Second, the limits should represent clinically relevant groupings. In the case of an outcomes instrument, a clinician investigator may choose to define a clinical failure as any score below a certain number. While the limit may be numerically arbitrary, in that the few scores falling just above and just below it may not be meaningfully different, success and failure defined as the entire groups above and below the limit might give a clearer clinical perspective on the data overall. In contradistinction to these principles, the secondary analyses of this study set the limit for dichotomizing continuous variables as the mean value from all groups. This decision turned the clinically helpful effort of categorization into a clinically arbitrary manipulation of numbers.

More randomized trials of orthopaedic interventions are needed. The time taken to plan them carefully from the details of randomization and consent, through the planned statistical analyses must be invested to secure the quality of the evidence. Buckley et al. undertook an enormous task and should be commended for their overall success. Hopefully, we can learn from both their example and their few mishaps along the way.

REFERENCES CITED

1. Buckley, R., et al., Operative compared with nonoperative treatment of displaced intra-articular calcaneal fractures: a prospective, randomized, controlled multicenter trial. J Bone Joint Surg Am, 2002. 84- A(10): p. 1733-44. 2. Thermann, H., et al., Management of calcaneal fractures in adults. Conservative versus operative treatment. Clin Orthop, 1998(353): p. 107- 24. 3. Zelen, M., A new design for randomized clinical trials. N Engl J Med, 1979. 300(22): p. 1242-5. 4. Chang, R.W., et al., Prerandomization: an alternative to classic randomization. The effects on recruitment in a controlled trial of arthroscopy for osteoarthrosis of the knee. J Bone Joint Surg Am, 1990. 72(10): p. 1451-5.