The systematic review

The systematic review

What is it?

A systematic review¹ ( Chalmers, I. &Altman, D.G. (eds) (1995) Systematic Reviews. British Medical Journal Publishing Group, London) is a formalized and stringent process of combining the information from all relevant studies (both published and unpublished) of the same health condition; these studies are usually clinical trials (Topic 14) of the same or similar treatments but may be observational studies (Topics 15 and 16). Clearly, a systematic review is an integral part of evidence based medicine (EBM; Topic 37), which applies the results of the best available evidence, together with clinical expertise, to the care of patients. So important is its role in EBM, that it has become the focus of an international network of clinicians, methodologists and consumers who have formed the Cochrane Collaboration. They have created the Cochrane ControlledTrials Register, and publish continually updated systematic reviews in various forms (e.g. on CD ROM).

What does it achieve?

• Refinement and reduction - large quantities of information are refined and reduced to a manageable size.
• Efficiency -the systematic review is usually quicker and less costly to perform than a new study. It may prevent others embarking on unnecessary studies, and can shorten the time lag between medical developments and their implementation.
• Generalizability and consistency-results can often be generalized to a wider patient population in a broader setting than would be possible from a single study. Consistencies in the results from different studies can be assessed, and any inconsistencies determined.
• Reliability - the systematic review aims to reduce errors,
• and so tends to improve the reliability and accuracy of recommendations when compared with haphazard reviews or single studies.
• Power and precision- the quantitative systematic review (see meta-analysis) has greater power (Topic 18) to detect effects of interest and provides more precise estimates of them than a single study.

Meta-analysis

What is it?

A meta-analysis or overview is a particular type of systematic review that focuses on the numerical results. The main aim of a meta-analysis is to combine the results from individual studies to produce, if ppropriate, an estimate ofthe overall or average effect of interest (e.g. the relative risk, RR; Topic 15). The direction and magnitude of this average effect, together with a consideration of the associated confidence interval and hypothesis test result, can be used to make decisions about the therapy under investigation and the management of patients.

Statistical approach

• We decide on the effect of interest and, if the raw data are available, evaluate it for each study. However, in practice, we may have to extract these effects from published results. If the outcome in a clinical trial comparing two treatments is:
• numerical-the effect may be the difference in treatment means. A zero difference implies no treatment effect;
• binary (e.g. died1survived)-we consider the risks of the outcome (e.g. death) in the treatment groups. The effect may be the difference in risks or their ratio, the RR. If the difference in risks equals zero or RR = 1 then there is no treatment effect.
• Obtain an estimate of statistical heterogeneity and check for statistical homogeneity -we have statistical heterogeneity when there is considerable variation between the estimates of the effect of interest from the different studies. We can measure it, and perform a hypothesis test to investigate whether the individual estimates are compatible (i.e. homogeneous). If there is significant statistical heterogeneity, we should proceed cautiously, investigate the reasons for its presence and modify our approach accordingly.
• Estimate the average effect of interest (with a confidence interval), and perform the appropriate hypothesis test on the effect (e.g. that the true RR = 1)-you may come across the terms 'fixed-effects' and 'random-effects' models in this context. Although the underlying concepts are beyond the scope of this book, note that we generally use a fixed-effects model if there is no evidence of statistical heterogeneity, and a random-effects model otherwise.
• Interpret the results and present the findings-it is helpful to summarize the results from each trial (e.g. the sample size, baseline characteristics, effect of interest such as the RR, and related confidence intervals, CI) in a table (see Example). The most common graphical display is a forest plot (Fig. 38.1) in which the estimated effect (with CI) for each trial, and their average, are marked along the length of a vertical line which represents 'no treatment effect' (e.g. this line corresponds to the value 'one' if the effect is a RR). Initially, we examine whether the estimated effects from the different studies are on the same side of the
• line. Then we can use the CIS to judge whether the results are compatible (if the CIS overlap), to determine whether incompatible results can be explained by small sample sizes (if CIS are wide) and to assess the significance of the individual and overall effects (by observing whether the vertical line crosses some or all of the CIS).

Advantages and disadvantages

• As a meta-analysis is a particular form of systematic review, it offers all the advantages of the latter (see 'what does it achieve?'). In particular, a meta-analysis, because of its inflated sample size, is able to detect treatment effects with greater power and estimate these effects with greater precision than any single study. Its advantages, together with the introduction of meta-analysis software, have led meta-analyses to proliferate. However, improper use can lead to erroneous conclusions regarding treatment efficacy. The following principal problems should be thoroughly investigated and resolved before a meta-analysis is
• performed.
• Publication bias-the tendency to include in the analysis only the results from published papers; these favour statistically significant findings.
• Clinical heterogeneity-in which differences in the patient population, outcome measures, definition of variables, and/or duration of follow-up of the studies included in the analysis create problems of non-compatibility.
• Quality differences-the design and conduct of the studies may vary in their quality. Although giving more weight to the better studies is one solution to this dilemma, any weighting system can be criticized on the grounds that it is arbitrary.
• Dependence-the results from studies included in the analysis may not be independent, e.g. when results from a study are published on more than one occasion.