A Simple, Fast, Easy Method To Identify The Evidence Base In Pain Relief Research: Validation Of A Computer Search Strategy Used Alone To Identify Quality Randomized Controlled Trials.




Authors: Tony K. F. Chow FANZCAa*, Elean Tob MB BS, Colin S. Goodchilda PhD, John J. McNeilc PhD


a Department of Anaesthesia, Monash University, Monash Medical Centre, Clayton, Victoria, 3168, Australia.

b Templestowe District Medical Centre, Templestowe, Victoria, 3109, Australia.

c Department of Epidemiology and Preventive Medicine, Monash University, Alfred Hospital, Prahran, Victoria, 3181, Australia.

* Corresponding author and reprint requests.

Tel.: (61) (3) 9594 5290 Fax: (61) (3) 9594 6611.

E-mail address: tony.chow@med.monash.edu.au







Running title computer search strategy in pain medicine

Abstract


Clinicians need a simple, fast, reliable and economic way of identifying the evidence base relevant to their clinical practice. It is commonly believed that the only way to identify all relevant evidence is to perform hand searches of the literature to supplement computer searches; this is complex and labor intensive. However the majority of quality randomized controlled trials cited in systematic reviews in pain medicine are listed in computer databases. Two studies were performed to investigate the efficiency, in terms of sensitivity, specificity and precision of three computer search strategies: OSSS which is used by the Cochrane Collaboration; RCT.pt, a standard MEDLINE strategy; and DBRCT.af which is a new single-line computer algorithm based on the assumption that double blinded, randomized controlled trials would be indexed with ‘double blind’, ‘random’ or variations of these terms in MEDLINE and EMBASE. DBRCT.af was found to be highly sensitive (97%) in identifying quality randomized controlled trials in pain medicine. The precision (ratio of randomized controlled trials compared with number of non-randomized trials identified) was 82% and the specificity in excluding the non-randomized controlled trials was 98%. We conclude that clinicians can now use DBRCT.af to update and conduct de-novo systematic reviews in pain relief research.


Key words:

Double blinded, randomized controlled trials

Computer search strategy

Sensitivity

Specificity

Precision

Implication Statement


Quality evidence about what is good clinical practice in pain treatment is buried in the medical literature amongst large quantities of other information. This paper describes how any clinician with access to the Internet can identify those quality studies reliably, quickly and cheaply.

Introduction


“It is surely a great criticism of our profession that we have not organized a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomized controlled trials.” (1)


Good medicine is based upon scientific evidence. A comprehensive database of randomized controlled trials (RCTs) in medicine would provide the essential data for evaluation of current therapy and pinpoint areas in need of further research. The Oxford University Pain Relief Unit has created a database of RCTs in pain relief medicine. This database was created by a refined MEDLINE search strategy from 1966 to 1990 and by an exhaustive hand search of more than 1,000,000 pages from 40 biomedical journals, encompassing all the major anesthetic / analgesic journals, published from 1950 to 1990. (2) This database appears on the Cochrane Library and is maintained by the Cochrane Pain, Palliative and Supportive Care Collaborative Group. (3) The database is updated with electronic searches in MEDLINE, EMBASE and the Cochrane Library using a combination of free text words and subject heading terms. Additional reports are identified from reference lists of retrieved reports, review articles and specialist textbooks. This database contains a large number of potential articles that constituted the starting point from which many systematic reviews have resulted. However hand searching is inefficient and is not feasible on an individual basis. The Oxford Unit concedes that more than 97% of the time spent by hand searchers were in looking at irrelevant material. (2) The large effort, time and cost of this method restrict the method to large organizations and even these find it difficult to find the resources to update reviews that have already been performed. This labor intensive searching method has not changed significantly over the past two decades, in which, advances in information technology have allowed unprecedented access to medical information. This, along with the increase in awareness of proper indexing of RCTs, suggests that it is time to explore the possibilities of computer search strategies, as the sole method of identification of RCTs in the medical literature.


However, reports on the sensitivities of computer strategies vary between 50 to 80%,(4) compared with the methods involving hand searching. It has also been found that the more sensitive the computer strategy is made, the less precise it becomes; estimated precisions range from 2 to 82%. (4) The latter leads to large numbers of irrelevant articles concealing a smaller number of papers that might contribute information to a review. These observations make it difficult to justify preparing systematic reviews using computer strategy alone.


These reports, however, do not take into account the quality of the RCTs. It has been well demonstrated that compared with double blinded, randomized controlled trials (DBRCTs), the treatment effects are exaggerated if the RCTs are unblinded or if the concealment of randomization is unclear or inadequate. (5) It is generally accepted that systematic reviews should be conducted with high quality RCTs. Based upon this premise, it is not so important to identify all RCTs, but rather to identify all high quality RCTs.


Evidence exists that the majority of quality RCTs are listed in computer databases. Knipschild conducted an exhaustive search for vitamin C related RCTs and found a total of 61 eligible trials, from which only 22 were computer-listed. (6) Interestingly, after grading the trials according to methodological soundness, Knipschild concluded that there were 15 good quality trials and all but one were computer-listed. There is a similar pattern in systematic reviews pertaining to methods of pain relief. This has led to the investigations reported here on the efficiency of computer strategies in identifying these computer-listed RCTs. Two studies were performed to define the sensitivity, precision and specificity of three computer search strategies: Optimally Sensitive Search Strategy (OSSS) which is used by the Cochrane Collaboration; randomized controlled trials publication type (RCT.pt) which is a standard MEDLINE strategy; and double blinded, randomized controlled trials in all fields (DBRCT.af) which is a new single-line computer algorithm, based on the assumption that double blinded, randomized controlled trials would be indexed with ‘double blind’, ‘random’ or variations of these terms in MEDLINE and EMBASE.

Methods


Three computer strategies were used to identify RCTs in pain relief research.


The Optimally Sensitive Search Strategy

The Optimally Sensitive Search Strategy (OSSS) is a comprehensive, 29-line computer strategy (Table1A). It is contained in the Cochrane Collaboration Reviewers’ Handbook. (4;7) It was developed with the intention to cast a wide net in order to identify as many eligible RCTs as possible.


Randomized controlled trial publication type

The United States National Library of Medicine introduced the MEDLINE field of randomized controlled trial publication type (RCT.pt) in 1991 (Table 1B). (8) The RCTs are identified by trained searchers and coded appropriately into MEDLINE. To the best of our knowledge, RCT.pt is the only computer search strategy that has been studied and described in print. RCT.pt is contained within the OSSS and DBRCT.af (detailed below) and contributes to the sensitivity of these two computer strategies when used in MEDLINE. The RCT.pt field does not exist in EMBASE, an important electronic resource that is commonly used to conduct systematic reviews.


Double blinded, randomized controlled trials in all fields

The advanced OVID search labels (Ovid Technologies© 2000-2001) within MEDLINE and EMBASE were used to develop a single-line computer algorithm: (9-11)


(double blind$ or random$).af.


This strategy assumes that DBRCTs will be indexed with ‘double blind’, ‘random’ or variations of these terms. This algorithm (abbreviated to DBRCT.af) identifies specific terms with unlimited truncation (e.g. double blinded, double blinding, randomly, random allocation, randomized, randomised, randomization, randomisation and randomized controlled trial etc.). Conversely, DBRCT.af excludes study designs other than DBRCT, RCTs and double blinded but non-RCTs, and hence should not be indexed with the abovementioned terms. Additional strategies will need to be developed to identify these other studies, which may be better to inform about rare events, diagnosis or prognosis. DBRCT.af identifies these terms in all fields (.af). The fields of most importance are the title, abstract, subject and publication type. The simple program shown in Table 1C creates a database of eligible RCTs in MEDLINE and EMBASE. Lines 2 and 3 limit the database to human articles; the same limitation phrases were used in OSSS. These three strategies were evaluated in two studies to determine their relative specificity, precision and sensitivity in identifying RCTs in the field of pain medicine.

Study 1: Morphine.af Study


This study was performed to determine the specificity and precision of the OSSS and DBRCT.af. All articles published in the medical scientific literature between 1980 and 1996 relating to the use of morphine as an analgesic were extracted from EMBASE and MEDLINE by the general search algorithm morphine.af (Table 2, lines 1-4). One thousand (1000) of these were chosen using the random number generation analysis tool in Microsoft Excel 2000. The list of random numbers and references can be found on the following web address: www.med.monash.edu.au/anaesthesia/.


Hard copies of the 1000 articles were obtained. Two authors (Tony Chow and Elean To) read the methods section to classify the articles into RCTs or non-RCTs. Once this was completed the OSSS and DBRCT.af were applied to the database containing these articles. The specificity and precision of each of the methods was then calculated as shown in the example for DBRCT.af in Table 2, lines 5-6.


Precision was defined as:

Number of RCTs identified by the strategy x 100

Number of all articles identified by the strategy


Specificity was defined as:

Number of non-RCTs excluded by the computer strategy x 100

Number of non-RCTs identified by morphine.af

Study 2: Comparison with published reviews


The sensitivity of the three computer strategies in pain medicine was assessed by comparing the RCTs identified by the three computer strategies used alone, with actual work published in the field by the Oxford University Pain Relief Unit, in a textbook that contains 23 quality systematic reviews. (12) The Oxford Strategy used to identify articles suitable for inclusion in a systematic review consisted of computer searches through MEDLINE, EMBASE and the Cochrane Library, supplemented by large-scale hand searches of reference lists, reviews, textbooks and 40 biomedical journals. All 23 systematic reviews published in the Oxford Unit’s textbook were used for this study. Reviews were included for comparison having fulfilled the following criteria:

They were published in a peer-reviewed journal

They contained only published clinical RCTs. Hence unpublished and non-clinical material were excluded

The randomization and concealment used in the RCTs were adequate


Hard copies of the original systematic reviews and the referenced RCTs were obtained. The referenced articles were firstly categorised into those listed in MEDLINE and EMBASE, and those that were not. The computer-listed RCTs were then checked to see if they were contained in the databases generated by the respective computer search strategies: that is, in line-29 of Table 1A and in line 3 of Tables 1B and 1C. These three databases were stored as citations in Reference Manager and Microsoft Excel files.


The sensitivity of each of the computer search strategies was calculated in two ways:

Number of RCTs identified by the computer strategy x 100

Number of all RCTs identified by McQuay and Moore


Number of RCTs identified by the computer strategy x 100

Number of computer-listed RCTs identified by McQuay and Moore


Finally, analyses that contained more than three RCTs were extracted from each review. The methods of analysis used by the Oxford Unit to draw their conclusions were repeated using the RCTs identified by the three computer strategies to see if the same conclusions could be drawn. Regarding the quantitative analyses, the following data were extracted: the clinical setting, treatment groups, patient numbers, duration, mode and the dose of treatment, mean and derived pain relief outcomes (TOPAR, SPID, VASTOTPAR, VASSPID) and dichotomous pain relief outcomes. Applying verified equations developed by McQuay and Moore, derived outcomes were calculated (50%maxTOTPAR) to facilitate calculations of relative risk, numbers-needed-to-treat and corresponding 95% confidence intervals using a random effect model. Statistical significance was assumed in the relative risk when the 95% confidence intervals did not include one and in the numbers-needed-to-treat when the 95% confidence intervals did not over lap. Regarding the qualitative analyses, the following data were extracted: the clinical setting, treatment groups, patient numbers, duration, mode and the dose of treatment and pain outcomes. Trials were considered as positive or negative according to the original authors’ statistical estimate, with p<0.05 considered as statistically significant. The treatment effect was considered to be present when positive trials out number the negative trials. No distinctions were made between listed and non-listed RCTs in this evaluation. Analyses containing less than four RCTs were deemed inconclusive.

Results

Study 1: Morphine.af Study


A total of 36,780 morphine-related, human articles published between 1980-1996 were identified by the morphine.af search of MEDLINE and EMBASE (Line 4, Table 2). The period of 1980-1996 was chosen to correspond with the Oxford morphine-related RCTs and this set of morphine-related articles contained all of the 32 computer-listed RCTs, analyzed in the two Oxford reviews that studied morphine RCTs. (13;14) The majority of the morphine-related articles (4:1) were listed in EMBASE compared with MEDLINE. This proportion was preserved in the experimental sample with 800 articles selected randomly from EMBASE and 200 from MEDLINE. The random numbers were evenly distributed to 2.7% (standard error +/- 0.1%) of articles published per year.


From the 1000 morphine-related articles, there were 97 RCTs and 903 non-RCTs (approximately one RCT per ten articles identified; Table 3). The OSSS improved the precision to 32% (three RCTs per ten articles) and DBRCT.af improved the precision to 83% (eight RCTs per ten articles). Both the OSSS and DBRCT.af were highly specific in excluding the non-RCTs. The OSSS excluded 705 (specificity 78%), whilst DBRCT.af excluded 886 (specificity 98%). Conversely, the OSSS failed to exclude 198 non-RCTs compared with DBRCT.af, which failed to exclude 17 non-RCTs. Thus, this translates into 91% less work if one used DBRCT.af to identify RCTs for a systematic review instead of OSSS because it is better able to exclude non-RCTs. The precision and specificity of RCT.pt could not be compared with the OSSS or DBRCT.af because RCT.pt cannot be used in EMBASE and hence has a denominator of only 200 articles. Of the 17 non-RCTs that were incorrectly identified by DBRCT.af, six were reviews which discussed RCTs, five were double blinded but non-randomized trials, four were surveys which included a random sample, one was a letter that commented on a RCT and one was a case study which was indexed with ‘Double Blind Procedure’ in the keywords section.

Study 2: Comparison with published reviews


Fifteen reviews, nine pertaining to acute pain management, (14-22) five to chronic pain management (19;23-26) and one pertaining to both, (27) fulfilled the inclusion criteria (Table 4). Eight reviews were excluded: one non-peer-reviewed publication on the effectiveness of TENS in chronic pain and two reviews that included non-randomized and unpublished data were excluded (28;29) along with five unpublished reviews relating to epidural corticosteroids for sciatica, spinal cord stimulators for back pain, steroid injections for shoulder disorder, the use of dihydrocodeine in post-operative pain and the use of topical capsaicin in chronic pain. A total of 288 clinical RCTs were referenced, of which 284 (99%) were listed in MEDLINE or EMBASE and four were unlisted. (30-33) The sensitivity in identifying all RCTs was 98%, 95% and 65% for the OSSS, DBRCT.af and RCT.pt respectively (Table 4). The sensitivity in identifying MEDLINE and EMBASE listed RCTs was 99.6%, 96% and 65% for the OSSS, DBRCT.af and RCT.pt respectively (Table 4). The distribution of publication dates of those RCTs in consecutive 5-yr epochs is presented in Figure 1. This analysis shows that most of the RCTs utilized by the McQuay and Moore were contained within the more recent strata. (12) The sensitivities of the three computer strategies in identifying computer-listed RCTs over the same 5-yr epochs are plotted in Figure 2; all three strategies improved with later epochs. It can be seen that OSSS was 100% sensitive for publications after 1970 and DBRCT.af was >95% sensitive for publications after 1980 whereas RCT.pt remained less than 90% sensitive for publications in all epochs.


From the fifteen systematic reviews, there were 30 analyses containing more than three RCTs (Table 5): 17 quantitative and 13 qualitative analyses. DBRCT.af was 97% sensitive in identifying 236 of the 244 MEDLINE / EMBASE listed RCTs and 96% (236/246) sensitive in identifying all the RCTs used by McQuay and Moore in their reviews (range of 75 to 100%; mode 100% (21/30)). When the same analyses performed by McQuay and Moore were applied to the RCTs identified by DBRCT.af, the conclusions drawn were not altered significantly. Twenty-one analyses (14 quantitative and 7 qualitative analyses) drew precisely the same conclusions because DBRCT.af identified all the RCTs analyzed by the McQuay and Moore. The remaining three quantitative analyses drew the same conclusion with minor, insignificant differences in the calculated relative risks and numbers-needed-to-treat. (17;17;18) All but one of the six qualitative analyses drew the same conclusion with 85-95% of the original patient numbers. (2;15;22;27;27) In the remaining qualitative analysis, DBRCT.af identified three of the four original RCTs and hence the analysis was deemed inconclusive because of the small number of trials identified. (20) The data of the three quantitative and six qualitative analyses were extensive and can be viewed on our web address: www.med.monash.edu.au/anaesthesia/.


The OSSS identified all of the computer-listed RCTs included in the 30 analyses. As a result, 28 of the 30 analyses drew precisely the same conclusion. The remaining two analyses drew the same conclusion with 85% of the original patient numbers (Table 4). (23;27) There was insufficient data to reproduce the analyses with the RCTs identified by RCT.pt due to its poor sensitivity.

Discussion

Study 1: Morphine.af Study


The first step in organizing a critical summary and periodic update of pain relief research is to identify the relevant RCTs. Ideally this should be done quickly, with high sensitivity and specificity. Morphine-related articles were used to test the precision and specificity of computer search strategies because morphine is one of the oldest and most commonly used analgesics worldwide. As such it is the gold standard with which all analgesics are compared against and there are many scientific publications pertaining to morphine. This study showed that 90% of the published morphine-related articles are non-RCTs. This would lead to a large amount of effort, time and cost to exclude these by hand prior to a systematic review. In order to identify all eligible RCTs it is necessary to read all of the extracted articles to exclude the non-RCTs. Computers are far more efficient in performing such monotonous tasks if the compilation of the databases can be relied upon for correct classification.


Efficiency of search strategies can be measured by calculating specificity and precision. Specificity (the ability to exclude non-RCTs) is a better measure than precision (the ratio of RCTs and non-RCTs identified). Although both measures depend on the amount of non-RCTs identified, the number of RCTs identified regardless of its quality, confounds the calculated precision. For example, if one calculates precision with all eligible RCTs as the numerator (as is the usual practice) then a strategy that identifies more poor quality RCTs will appear more precise. Conversely, if one calculates precision with the high quality RCTs that are normally included in the systematic reviews as the numerator then all strategies are imprecise. To the best of our knowledge, we are not aware of the specificity of a search strategy ever being reported because the number of articles excluded has not been reported. However it is the low specificity that prevents clinicians with limited resources from performing systematic reviews. Even in institutions with large resources, most are reluctant to update reviews already published because so much labor, time and cost is required in separating the quality trials that can go into the analysis, from those that need to be excluded.


The specificity of DBRCT.af (98%) translates into substantial savings. It costed our department approximately $A10,000 to obtain the 903 non-RCTs identified in this study in order to read them and exclude them by hand. Using the OSSS reduces this to around $A2,000 to obtain the 198 non-RCTs identified by that strategy and this reduces further to around $A200 for the 17 non-RCTs caught by using DBRCT.af. The effort and time spent to obtain, read and exclude these articles are in the same proportions. Remembering that these estimates pertain to a random sample consisting only 2.7% of the 36,780 morphine-related articles, the actual savings are considerable. Having made these savings, it is important to know whether the results of the search identify enough quality RCTs, such that the subsequent systematic review can come to a meaningful conclusion. This was addressed by comparing the results of the computer searches with established high quality systematic reviews in pain medicine.

Study 2: Comparison with published reviews


From the published works of the Oxford Pain Relief Unit, one may see that the majority of RCTs (99%) used in their systematic reviews are listed in the major computer databases, EMBASE and MEDLINE. The Oxford Strategy used to identify RCTs is exhaustive, recognized as a gold standard, and is well regarded by the Cochrane Collaboration and the International Association for the Study of Pain. The large number of systematic reviews published by the McQuay and Moore provided an opportunity to evaluate the efficiency of computer search strategies used alone to identify relevant RCTs.


The results of this study showed that the majority of computer-listed RCTs were identified solely by computer strategies such as the OSSS (99.6%) and DBRCT.af (96%), but not with RCT.pt (65%). The OSSS and DBRCT.af were sufficiently sensitive, such that when the RCTs identified by them were used to reproduce the Oxford analyses, the conclusions drawn were representative of the conclusions published by Oxford Unit. This is a very important result and leads one to ask whether it is necessary to include all RCTs to conduct a systematic review. It is true to say that when one conducts a clinical trial, only a sample of the entire patient population is examined. Scientific principles of randomization and blinding reduce bias sufficiently so that the studied sample is representative of the entire population. In conducting systematic reviews, it may be sufficient to accept a representative sample of RCTs rather than to strive to capture and include all RCTs. The results from this study provide support for this suggestion.


An important trend in the indexing of RCTs over the past 30 years was revealed in Figure 2, when the sensitivities of the three computer search strategies were compared. The sensitivity of RCT.pt, from 1966 to 1997, showed the efforts by the United States Library of Medicine and the Cochrane Collaboration in retrospectively indexing MEDLINE articles. In essence, the sensitivity of RCT.pt was static from 1971 to 1985, and improved thereafter. In comparison, the sensitivity of DBRCT.af progressively increased from 1966 to 1980, and remained above 95% thereafter. (Figure 2) Although RCT.pt is contained in DBRCT.af (and OSSS) and contributes to its sensitivity, it is clear that RCT.pt did not contribute to the progressive increase in sensitivity over the 1974 to 1985 period. This period can loosely be coined the pre-Cochrane era, that is, before Cochrane criticized the medical profession for not utilizing RCTs effectively. The progressive increase was more likely because authors who conducted RCTs recognized the need to index their publications with the appropriate terms. Strategy DBRCT.af was developed with the assumption that double blinded, randomized controlled trials would be indexed with these terms. Despite the efforts in retrospective indexing the sensitivity of RCT.pt did not reach 90%. In part, this may be because RCT.pt is limited to searching MEDLINE. It is also likely that retrospective indexing does not identify all RCTs. Other publication type strategies such as controlled clinical trial.pt and clinical trial.pt were not investigated. (8) These two strategies are similar to RCT.pt in that they rely on trained searchers to code the articles appropriately. Unlike RCT.pt however, they are not contained in DBRCT.af. Since DBRCT.af was highly sensitive and specific, it did not seem relevant to investigate these two strategies.


In conclusion, the two studies reported in this paper demonstrate that the OSSS and DBRCT.af are highly efficient compared with the Oxford Strategy. Their routine use can lead to considerable savings in effort, time and cost. In light of these results, we recommend that these computer strategies should be used to periodically update the 15 Oxford reviews. The consistent efficiency demonstrated in these quality reviews, of wide-ranging topics, suggest that this may apply throughout the specialty of pain relief research. We also believe that sufficient number of quality RCTs can be identified by the OSSS or DBRCT.af to conduct de novo systematic reviews to give reasonably robust answers in the efficacy of pain relief treatments. In considering poorly reported data such as adverse events, one needs to be cautious in drawing conclusions with limited information. A reasonable approach may be to search with DBRCT.af and progress to more extensive strategies if more information is required. Further work is required to confirm that the conclusions drawn by using computer strategies alone are representative of the true clinical picture.


The application of DBRCT.af in medical specialties other than pain relief would require validation. In Ophthalmology for example, the term ‘masked’ is preferred over blinded. In this regard, DBRCT.af would rely upon identifying the term random or variations there of, to identify DBRCTs. In contrast to the OSSS, DBRCT.af was designed not to include all possible terms. It is intended that in maintaining the simplicity of DBRCT.af that the message to improve the indexing of articles filter through as researchers read and conduct reviews using DBRCT.af.

Acknowledgements


We would like to acknowledge the Oxford University Pain Relief Unit for conducting the systematic reviews, without which this article would not be possible. We thank OVID Technology for their technical assistance. We would also like to thank Professor Chris Silagy (since deceased) and Mr Steve McDonald of the Institute of Public Health & Human Services Research, Monash University, Monash Medical Centre for their invaluable support and advice.

Table 1 Three Computer Strategies

A) Optimally Sensitive Search Strategy

  1. randomized controlled trial.pt.

  2. controlled clinical trial.pt.

  3. randomized controlled trials.sh.

  4. random allocation.sh.

  5. double blind method.sh.

  6. single blind method.sh.

  7. 1 or 2 or 3 or 4 or 5 or 6

  8. (animal not human).sh.

  9. 7 not 8

  10. clinical trial.pt.

  11. exp clinical trials/

  12. (clin$ adj25 trial$).ti,ab.

  13. ((singl$ or doubl$ or trebl$ or tripl$) adj25 (blind$ or mask$)).ti,ab.

  14. placebo$.sh.

  15. placebo$.ti,ab.

  16. random$.ti,ab.

  17. research design.sh.

  18. 10 or 11 or 12 or 13 or 14 or 15 or 16 or 17

  19. 18 not 8

  20. 19 not 9

  21. comparative study.sh.

  22. exp evaluation studies/

  23. (control$ or prospective$ or volunteer$).ti,ab.

  24. follow up studies.sh.

  25. prospective studies.sh.

  26. 21 or 22 or 23 or 24 or 25

  27. 26 not 8

  28. 27 not (9 or 20)

  29. 9 or 20 or 28


B) Randomized controlled trial publication type

  1. randomized controlled trial.pt

  2. (animal not human).sh.

  3. 1 not 2


C) Double blinded, randomized controlled trials in all fields

  1. (double blind$ or random$).af.

  2. (animal not human).sh.

  3. 1 not 2

Table 2 Morphine-related articles in EMBASE and MEDLINE (1980 to 1996)

Set Search Results

  1. morphine.af. 65839

  2. limit 1 to yr=1980-1996 46878

  3. (animal not human).sh. 2432523

  4. 2 not 3 36780

  5. (double blind$ or random$).af. 502363

  6. 4 and 5 3090


Results indicate the number of articles identified.

Table 3 The specificity and precision of the three computer strategies

Morphine.af OSSS DBRCT.af RCT.pt

RCTs identified 97 92 85 37

Non-RCTs identified 903 198 17 1

Non-RCTs excluded 705 886 156

Specificity 78% 98% 99%

Precision 10% 32% 83% 97%

Table 4 Comparison of RCTs identified by the Oxford Strategy and the 3 computer strategies

Reference Topic Oxford OSSS DBRCT.af RCT.pt

Jadad 95 IV regional sympathetic blocks 6 (1) 6 6 4

McQuay 95 Pre-emptive analgesia 15 (1) 15 14 12

McQuay 95 Anticonvulsants 20 20 18 7

Carroll 96 TENS in post-operative pain 17 17 16 12

McQuay 96 Antidepressants 20 19 19 14

Carroll 97 TENS in labor 8 8 8 6

Kalso 97 Intraarticular morphine 15 15 15 14

McQuay 97 Radiotherapy 14 14 14 12

Moore 97 Paracetamol with/without codeine 39 39 39 19

Picard 97 Peripheral opioids (not intra-articular) 22 22 20 20

Collins 98 Dextropropoxyphene 11 11 10 3

Collins 98 Oral NSAIDs 38 38 37 20

Kalso 98 Systemic local anaesthetic 17 17 17 13

Tramer 98 NSAIDs by differing routes 25 (1) 25 23 21

McQuay 99 Parental opioids 17 (1) 17 17 9

Total RCTs identified 284 (4) 283 273 86

Sensitivity in identifying all RCTs 98% 95% 65%

Sensitivity in identifying computer-listed RCTs 99.6% 96% 65%

Legend to Table 4


RCT.pt = Randomized controlled trial.pt

DBRCT.af = (double blind$ or random$).af

OSSS = Optimally Sensitive Search Strategy

N (n) = Number of MEDLINE / EMBASE listed RCTs (number of non-listed RCTs)

Table 5 Comparison of analyses with >3 RCTs between Oxford and DBRCT.af

Reference Topic Oxford DBRCT.af

Jadad 95 IV sympathetic blocks in reflex sympathetic dystrophy 6 (1) 6

McQuay 95 Pre-emptive NSAIDs or paracetamol in POP 4 3

Pre-emptive epidural or caudal in POP 4 4

Carroll 96 TENS vs sham TENS in POP 14 13

TENS & opioids vs opioids in POP 7 7

McQuay 96 Antidepressants vs placebo in diabetic neuropathy 13 13

Antidepressants vs control in diabetic neuropathy 7 7

Carroll 97 TENS vs sham TENS in labor pain 5 5

Kalso 97 IA morphine vs IA placebo with active control in POP 6 6

IA morphine & bupivacaine vs IA placebo in POP 4 4

McQuay 97 Multiple fractions in metastatic bone pain 5 5

Strontium & rhenium in metastatic bone pain 4 4

Moore 97 Paracetamol 600mg vs placebo in POP 10 10

Paracetamol 1000mg vs placebo in POP 7 7

Paracetamol 600mg vs placebo in POP 7 7

Paracetamol 300mg & codeine 30mg vs placebo in POP 5 5

Paracetamol 600mg & codeine 60mg vs placebo in POP 13 13

Paracetamol 600mg vs 600mg & codeine 60mg in POP 11 11

Picard 97 Peripheral opioids via Bier’s block in POP 4 4

Peripheral opioids via brachial plexus block in POP 10 9

Peripheral opioids via other sites in POP 5 5

Collins 98 Dextropropoxyphene vs placebo in POP 6 5

Dextropropoxyphene & paracetamol vs placebo in POP 5 5

Collins 98 Oral ibuprofen 200mg vs placebo in POP 8 7

Oral ibuprofen 400mg vs placebo in POP 30 29

Oral diclofenac 50mg vs placebo in POP 6 6

Tramer 98 NSAIDs in rheumatoid arthritis 6 5

NSAIDs in POP 13 (1) 12

NSAIDs in renal colic 4 4

McQuay 99 10mg intramuscular morphine vs placebo in POP 15 15

Total 244 (2) 236

Legend to Table 5


There were 30 analyses with four or more RCTs. DBRCT.af identified 97% (236/244) of the listed RCTs and 96% overall. The three remaining quantitative analyses drew the same conclusion with minor, insignificant differences in the calculated relative risks and numbers-needed-to-treat. Five of the six qualitative analyses drew the same conclusion with 85-95% of the original patient numbers. In the remaining qualitative analysis, DBRCT.af identified three of the four original RCTs and hence the analysis was inconclusive. The RCTs identified by DBRCT.af were comprised of in 97% of the Oxford analyses (29/30).

N (n) = Number of MEDLINE / EMBASE listed studies (number of non-listed studies).

POP = post-operative pain


Figure 1 Randomized controlled trials identified by Oxford with respect to publication year


Figure 2 The sensitivity of the 3 computer strategies with respect to publication year


Sensitivity (%)

Reference List


(1) Cochrane AL. 1931-1971: a critical review, with particular reference to the medical profession. In Medicines for the year 2000. London: Office of Health Economics; 1979: 1-11.

(2) Jadad AR, Carroll D, Moore A, McQuay H. Developing a database of published reports of randomised clinical trials in pain research. Pain. 1996;66:239-46.

(3) The Cochrane Collaboration. Cochrane Hand Search Manual. 2000.

(4) Dickersin K, Scherer R, and , Lefebvre C. Identifying relevant studies for systematic reviews. BMJ 309(6964), 1286-1291. 1994.

(5) Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995;273:408-12.

(6) Knipschild P. Systematic reviews. Some examples. BMJ 309(6986), 719-721. 1994.

(7) Clarke M, Oxman AD, editors.In: The Cochrane Library Issue 2, 2002. Cochrane Reviewers' Handbook 4.1.5. The Cochrane Library, Oxford; 2002.

(8) McDonald SJ, Lefebvre C, and Clarke MJ. Identifying reports of controlled trials in the BMJ and the Lancet. BMJ 313(7065), 1116-1117. 1996.

(9) Chow T, To E, Goodchild C, and McNeil J. Improved indexing of randomised controlled trials has enabled computer search strategies to identify them with high sensitivity and specificity, in pain relief research. Conference Proceedings of the 8th International Cochrane Colloquium, 41-42. 2000. Cape Town, South Africa.

(10) Chow TKF, To E, Goodchild CS, and McNeil JJ. Computer search strategies are efficient in identifying the quality randomised controlled trials in pain relief research. Anaesthesia & Intensive Care 29(6), 658. 2001.

(11) Chow TKF, Goodchild CS, and To E. A simple, sensitive and precise strategy (Strategy-DR) to identify randomised controlled trials in medicine. Anaesthesia and Intensive Care 27(1 Abstract), 77. 1999.

(12) McQuay H, Moore A. An evidence-based resource for pain relief. Oxford University Press; 1998.

(13) Kalso E, Tramer MR, Carroll D, McQuay HJ, Moore RA. Pain relief from intra-articular morphine after knee surgery: a qualitative systematic review. Pain. 1997;71:127-34.

(14) McQuay HJ, Carroll D, Moore RA. Injected morphine in postoperative pain: a quantitative systematic review. Journal of Pain & Symptom Management. 1999;17:164-74.

(15) Carroll D, Tramer M, McQuay H, Nye B, Moore A. Randomization is important in studies with pain outcomes: systematic review of transcutaneous electrical nerve stimulation in acute postoperative pain. British Journal of Anaesthesia. 1996;77:798-803.

(16) Carroll D, Tramer M, McQuay H, Nye B, Moore A. Transcutaneous electrical nerve stimulation in labour pain: a systematic review. British Journal of Obstetrics & Gynaecology. 1997;104:169-75.

(17) Collins SL, Moore RA, McQuay HJ, Wiffen PJ. Oral ibuprofen and diclofenac in post-operative pain: A quantitative systematic review. European Journal of Pain. 1998;2:285-91.

(18) Collins SL, Edwards JE, Moore RA, McQuay HJ. Single-dose dextropropoxyphene in post-operative pain: a quantitative systematic review. European Journal of Clinical Pharmacology. 1998;54:107-12.

(19) Kalso E, Tramer MR, McQuay HJ, Moore RA. Systemic local-anaesthetic-type drugs in chronic pain: A systematic review. European Journal of Pain. 1998;2:3-14.

(20) McQuay HJ. Pre-emptive analgesia: a systematic review of clinical studies. Annals of Medicine. 1995;27:249-56.

(21) Moore A, Collins S, Carroll D, McQuay H. Paracetamol with and without codeine in acute pain: a quantitative systematic review. Pain. 1997;70:193-201.

(22) Picard PR, Tramer MR, McQuay HJ, Moore RA. Analgesic efficacy of peripheral opioids (all except intra-articular): a qualitative systematic review of randomised controlled trials. Pain. 1997;72:309-18.

(23) Jadad AR, Carroll D, Glynn CJ, McQuay HJ. Intravenous regional sympathetic blockade for pain relief in reflex sympathetic dystrophy: A systematic review and a randomized, double-blind crossover study. Journal of Pain & Symptom Management. 1995;10:13-20.

(24) McQuay H, Carroll D, Jadad AR, Wiffen P, Moore A. Anticonvulsant drugs for management of pain: a systematic review. BMJ. 1995;311:1047-52.

(25) McQuay HJ, Tramer M, Nye BA, Carroll D, Wiffen PJ, Moore RA. A systematic review of antidepressants in neuropathic pain. Pain. 1996;68:217-27.

(26) McQuay HJ, Collins SL, Carroll D, Moore RA. Radiotherapy for the palliation of painful bone metastases. Cochrane Database of Systematic Reviews. 2000;CD001793.

(27) Tramer MR, Williams JE, Carroll D, Wiffen PJ, Moore RA, McQuay HJ. Comparing analgesic efficacy of non-steroidal anti-inflammatory drugs given by different routes in acute and chronic pain: a qualitative systematic review. Acta Anaesthesiologica Scandinavica. 1998;42:71-79.

(28) Moore RA, McQuay HJ. Single-patient data meta-analysis of 3453 postoperative patients: oral tramadol versus placebo, codeine and combination analgesics. Pain. 1997;69:287-94.

(29) Moore RA, Tramer MR, Carroll D, Wiffen PJ, McQuay HJ. Quantitative systematic review of topically applied non-steroidal anti-inflammatory drugs. BMJ. 1998;316:333-38.

(30) Amanor-Boadu SD, Jadad AR, Glynn CJ, Jack TM, McQuay HJ. Influence of pre and postoperative morphine administration on pain after body surface surgery: a double blind randomised controlled study. Proceedings of 7th World Congress on Pain. 1993;538-39.

(31) Beaver WT, Feise G. Comparison of the analgesic effects of morphine, hydroxyzine, and their combination in patients with postoperative pain. Advances in Pain Research and Therapy. 1976;1:553-57.

(32) Dhar S, McGlone F, Dean J, Klenerman L. The temporal significance of the placebo response in refiex sympathetic dystrophy. Proceedings of the British Orthopaedic Research Society Meeting. 1992;16:16-17.

(33) Moore AP, Thorpe JA, Mulley BA, Cruise M. Which diclofenac form is best? Post-thoracotomy pain: intramuscular v. rectal administration of diclofenac. Hospital Pharmacy Practice. 1993;431-35.


23