| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Reviews |
Clinical Operational Research Unit, Department of Mathematics, UCL, London WC1H 0BT, United Kingdom
Received 19 June 2008; accepted 26 August 2008.
* Corresponding author. Tel.: +44 7957 168754. (Email: tom.treasure{at}gmail.com).
 |
Abstract |
|---|
Key Words: Evidence based medicine • Clinical trials • Innovation • Emerging technology
|
1. Introduction |
|---|
The process of evaluation of a new drug is well characterised and the features of phase 1, phase 2 and phase 3 studies are quite clearly defined. In contrast, the process by which we evaluate surgical innovation is far from formalised and in many instances the introduction of new techniques and devices is haphazard. Regulation in the evaluation of interventions is very variable from one jurisdiction to another and is generally poorly defined. Many treatments have come into practice without ever being formally evaluated. It seems self evident that unavailing or inferior treatments should be replaced with better ones, or abandoned altogether, but getting evidence that a widely used treatment is relatively ineffective is even more difficult than evaluating a new one.
Amongst the agencies involved in the provision and funding of contemporary health care, there is keen interest in evaluation. I am increasingly being asked to give lectures, engage in seminars and colloquia, or to join working groups addressing the evaluation of surgical innovation. The debates and discussions are of necessity exploratory and speculative but the meeting of minds is steadily advancing thinking on this difficult subject.
I will list three examples.
The matter has been brought into sharper focus because of pressures to show that an operation is not only effective (that it ‘works’) but is also cost effective. And this does not simply mean cheaper. The new treatment may cost more but if it gains more life years and/or delivers better quality of life, it should be preferred and be introduced and displace less cost effective procedures. The first step is to prove effectiveness and quantify any net benefit. Does this always require a randomised controlled trial?
In this review I will set out the nature of the debate. A typical evaluation sequence for an intervention is set out in the left hand side of Table 1 while on the right side is the well established evidence based medicine (EBM) hierarchy of the quality of evidence. Note that the sequence followed in interventions appears to be exactly contrary to the hierarchy espoused by EBM. Whereas an RCT is the highest quality of evidence in EBM, in surgical evaluation it comes at the bottom of the list. This is not as contradictory as it might at first appear. It is a consequence of different starting points and different circumstances [4].
|
|
2. The observation of cause and effect |
|---|
Not all we do is that clear cut. Sometimes it takes time and healing for the benefits of surgery to accrue. Sometimes several treatments are going on at once. Was it the surgery or the chemotherapy that controlled a cancer? In empyema, how much credit to do we apportion to the drain and how much to the antibiotic? Many diseases naturally remit and a well-timed operation may get the credit. For example facial nerve palsy is known to recover after acupuncture; but we know that a similar proportion of patients recover if we do nothing. More confusing are diseases that run an intermittent or cyclical course (asthma, migraine, angina) and since we more often intervene when they are in a bad phase, subsequent improvement may not be related to the intervention but be the effect of regression to the mean (also known as convergence towards the norm).
An example where the parachute analogy was invoked and rejected was in the case of lung volume reduction surgery [6,7]. Given the potential harm of the surgical intervention itself, the heterogeneity of the cases, and their unpredictable rate of natural progression, the benefit for this group of patients was not as clear cut as that of a parachute saving the skydiver from certain death. The signal was not sufficiently discernable from the noise for outside scrutineers to accept that analogy and they forced a randomised trial by refusing funding for surgery outside the trial [7].
Seeing ‘the signal from the noise’ in terms of treatment effects has been more carefully considered since. Workers in evidence based medicine in Oxford University used the example of ‘the mother's kiss’ rather than the parachute. A bead or a dried pea which an infant has pushed up in its nose can be promptly and safely delivered by the mother blowing sharply into the child's mouth while occluding the other nostril with a finger or thumb [8]. The beneficial effect is immediate, total, and mechanistically related in a very obvious way to the intervention. The authors analysed the criteria by which a clear cut causal relationship might be established without the need for more rigorous evaluation. The effect has to be large, the problem being treated unlikely to spontaneously remit, the relief closely related in time and the whole thing should make mechanistic sense. The ‘signal to noise ratio’ must be high for a single instance to set rules for practice.
|
3. Clinical experience |
|---|
There are nevertheless several limitations to using clinical experience as a form of evidence:
In summary if clinical experience produced evidence based practice why are there so many and varied practices and we still need to know which are the more cost effective.
|
4. Follow-up studies |
|---|
We take a different view [14]. It should be remembered that a series is no more than a collection of cases. The limitations of discerning signal from noise (see Section 1 above) are still there as are the many biases at every stage of building and reporting the clinical experience (see Section 2 above) of which a follow-up study is a summation.
Even the claim that they prove safety is often poorly considered. Studies of relatively few patients undergoing an innovative procedure often conclude that the procedure had been shown to be ‘safe’. Even if there have been no adverse incidents in the first 20 patients (often the sort of number used for these claims) it is well within statistical confidence limits that a zero numerator with a small denominator is only an illusion of safety. The exact 95% confidence limits of 0/20 are 0–16.84%. Five deaths in the next 80, giving a 5% death rate, would not be improbable [15].
Inclusion of various forms of multivariable analysis help distinguish between patients who might be predicted to have better or worse results but leaves us without evidence of what would have happened to patients just like these if they had another or no intervention.
I will explore two limitations of surgeon's follow-up studies in the next two sections. These are:
|
5. Prospective studies |
|---|
These elements are largely or completely missing in a follow-up study in which only patients who have undergone the treatment are included. At first sight this seems fair enough since that is the treatment under evaluation. However the flaw becomes obvious if we consider treatments where there are several elements and sequential selection. This is exemplified in follow-up studies of extrapleural pneumonectomy for mesothelioma [16]. Patients included had to have completed the designated two or three components of multimodality treatment. Having the final treatment was conditional on being well enough after the prior treatment to enter the next phase. Who knows how many had embarked on this journey and had fallen by the way side? Where both intention to treat and completed protocol analyses are performed the difference can be seen. For example those who after EPP went on to complete the planned intensity modulated radiotherapy did better than those who did not complete treatment. A possible explanation is that the drop outs were not doing well and the completed therapy patients were a self selected group of better survivors [17].
Registries and research databases are forms of prospective study. There is an important distinction to be drawn between them. Registries aim to be comprehensive in collecting all patients but cannot capture all elements of importance. Research databases aim to collect much more comprehensive information but perforce limit inclusion to prespecified groups such as high volume centres. They both can produce useful information if carefully analysed [18] but of the vast quantities of information collected by database enthusiasts very little is used and much is of poor quality. In the study of risk avoidance already alluded to the data were from databases set up by accountants for re-imbursement for consultation and for prescribing costs rather than clinical research purposes [12].
|
6. Comparative studies |
|---|
A Chinese village acupuncturist whom I had the privilege to learn from, told me that following her treatment of lower motor neurone facial palsy many patients recovered. Amongst several patients with Bell's palsy there was one with a stroke. She did not of course use these terms but had her own (translated to me as ‘wind stroke’) but she divided the patients into prognostic groups on the basis of the physical signs familiar to us: the ability to furrow the brow. She took me aside and explained to me through our interpreter that established upper neuron palsy caused by stroke did not respond to acupuncture. Nevertheless she treated them both so that those whom she believed would not benefit would still have hope and not feel neglected. There is no question that she was a very good person and a kind doctor.
In the case of pulmonary metastasectomy the implicit belief of the authors of many reports is that all would be dead within the follow-up period. It is not true as shown by Aberg: 3/12 patients in a group that matched those who were being offered metastasectomy were alive five years later [19].
The very least we should do is have a reasonable knowledge of the natural history of a disease. Very soon after the first series of operations for congenital heart disease were performed, Brock's colleague Maurice Campbell stressed the importance of documenting the natural history of the patient. He wrote:
‘...but now that we have to take decisions about when to advise operation for these patients we realise the urgent need of accurate knowledge of the natural history and prognosis’.
He made this recommendation so that those who had poor prognosis could be selected for what was high-risk surgery and those who naturally did well could be left alone [20]. It is worth noting that in coronary and valvular heart disease the cases with the worse natural history are selected for surgery because that is where we get the biggest health difference from our intervention. In cancer the emphasis is on selecting the patients who do well after treatment with the consequence of excluding patients with the worst natural history. This immediately casts doubt on how effective our surgery is in cancer. Patients who ‘do well’ with surgery may have done well without it. This was Aberg's point [19,21] and indeed the selection of cases for metastasectomy from patients with metastatic colorectal cancer at the time was extreme [22].
There are many ways of finding and analysing data for comparative purposes. They are probably underused and not the subject of this review.
|
7. Randomised controlled trial (RCT) |
|---|
 TopAbstract1. Introduction2. The observation of...3. Clinical experience4. Follow-up studies5. Prospective studies6. Comparative studies 7. Randomised controlled trial... 8. ConclusionsReferences |
|---|
There is a convenient acronym called the PICO used to summarise the research question and the design. For a surgical study it is as follows:
In the context of surgery, blinding (masking) and other elements of pharmaceutical trials are difficult. The following indicates some considerations as they apply to surgeons.
|
8. Conclusions |
|---|
There is an often an overlooked aspect which is the need to re-evaluate interventions in use in order to stop less effective existing treatments. Many have slipped into practice before or outside any formal process of evaluation. While there is a critical need to get effective treatments to patients as soon as possible but to spare them harm, there is as much need to stop doing things that do not work as to start things that do [24].
The subject deserves critical but thoughtful analysis but too often views are polarised and expressed in inflexible terms.
Practitioners of practices loosely and variously called alternative or complementary medicine are often in this latter camp arguing that randomisation leaves out the very essence of their practice. Indeed they are right. When studied in a carefully performed randomised trial patients with irritable bowel syndrome did not do nearly as well if they underwent the sham acupuncture procedure without comment compared with those given exactly the same physical intervention within an empathetic consultation [25]. So it is not the procedure alone but a placebo effect attributable to the accompanying care that makes the difference. It matters not if you regard the process of delivering care by the alternative practitioner as ‘mumbo jumbo’. What is clear is that empathy and support help people feel better and can have a major effect, which may be wrongly attributed to the procedure itself.
Surgeons are usually convinced that it is the mechanistic effect of their operations that have the largest beneficial effect or they would not continue to do them. Those who object to subjecting their work to critical evaluation run the risk of being seen in the same light as homeopaths, attracting scorn such as that of the editor of The Lancet in his now famous editorial ‘Surgical research or comic opera’ [26].
|
Footnotes |
|---|
Presented at the Postgraduate Course of the 16th European Conference on General Thoracic Surgery, Bologna, Italy, June 8–11, 2008. |
References |
|---|
This article has been cited by other articles:
|
|
 |
|
  K. Moghissi Editorial comment: Are randomised trials needed in the era of rapidly evolving technologies? Eur. J. Cardiothorac. Surg., March 1, 2009; 35(3): 478 - 479. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| ANN THORAC SURG | ASIAN CARDIOVASC THORAC ANN | EUR J CARDIOTHORAC SURG |
| J THORAC CARDIOVASC SURG | ICVTS | ALL CTSNet JOURNALS |
