Undertaking, publishing and maintaining Cochrane Reviews that compare multiple interventions
The Comparing Multiple Interventions Methods Group (CMIMG) has received funding from the Cochrane Methods Innovation Fund to set up three working groups to address specific issues related to reviews that compare multiple interventions. The project is bringing together investigators, methodologists, authors, consumers, managing editors and other interested parties to help CMIMG develop consensus guidance.
Group 1 has been tasked with addressing fundamental issues associated with the initiation and logistics of undertaking, publishing and maintaining reviews of multiple interventions. In particular, the group is addressing the decision of whether to use the Cochrane Overview or Cochrane Intervention Review format for this task.
Authors who wish to compare multiple interventions in a single review may choose to do so using either a Cochrane Overview or a Cochrane Intervention Review.
The Comparing Multiple Interventions Methods Group (CMIMG) has received funding from the Cochrane Methods Innovation Fund to explore the issues involved in this decision. We have conducted a broad consultation with Collaboration contributors including investigators, methodologists, authors, consumers, managing editors and other interested parties and have developed an editorial decision tree (Click here to download a copy) to help guide prospective authors in this decision.
The decision tree is based on a new set of recommendations which will be incorporated into future editions of The
Overview or Intervention Review? - Group 1 Recommendations
Click here to download a background paper explaining the rationale for these recommendations.
1 - Define Overviews as reviews of reviews
The Cochrane Handbook, indicates that Overviews should search for reviews and provide their synthesis at the review level. Previous suggestions (including some from the CMIMG) that Overviews might also search for individual trials and include individual trial data in their synthesis have proven to be confusing for both authors and readers. The CMIMG therfore recommends that authors and Review Groups return to the Handbook definition of Overviews as Reviews that use other systematic reviews rather than individual trials as their basic unit of seach and analysis.
2 - Cochrane authors who wish to include indirect comparisons or network meta-analysis in their reviews should examine individual trial reports and use individual trial data in their analyses
The primary reason for this recommendation is that it may be very difficult to be certain that the assumptions for indirect comparison and network meta-analysis have been satisfied using only information provided in reviews.
3 - The Overview format should only be used to do indirect comparisons when the Overview authors have evaluated the assumption of transitivity for all of the trials included in those comparisons
This follows as a logical conclusion from #1 and #2. This may be possible in some cases. For example, when the overview authors are also authors of all of the included reviews they will have already examined all of the trials and may feel that they are able to address the transitivity issues using the Overview format. If the included reviews are all up to date and have taken a consistent approach to trial inclusion criteria, selection of outcomes, and approach to potential sources of heterogeneity, then it may be possible to use summary statistics from the included reviews in indirect comparisons and network meta-analyses. However the situations in which all of these conditions apply are likely to be rare.
4 - Overviews that facilitate “informal” indirect comparison by readers must address transitivity issues
As noted above, the transitivity assumption is not only an issue for reviews that include a network meta-analysis or other formal statistical approaches to indirect comparisons. Overview authors who choose to juxtapose data from different systematic reviews in a single table or figure are inviting readers to do their own informal indirect comparisons. A table or figure of this sort should only be used if the Overview authors are confident that the trials included in these reviews could have been “jointly randomizable”. If there is a good possibility that the various included reviews contain data from trials that were not sufficiently similar to one another, Overview authors should present results of the included reviews on a review-by-review basis.
5 - The Collaboration should re-examine the issue of “overlapping” reviews, and find an approach that allows both reviews that include a network meta-analysis and reviews that examine a more restricted subset of the available interventions to be published in The Cochrane Library.
[NB – This recommendation was discussed by the Coordinating Editors Board, the Methods Executive and by MARS at the Oxford 2013 mid-year meeting. There was strong agreement with the principle that “overlapping” reviews should be allowed in The Cochrane Library when one of the reviews was a review that compared multiple interventions with the intent of finding the interventions likely to have the highest efficacy or fewest adverse effects. We are currently exploring mechanisms for clearly flagging these comparative reviews in a way that minimizes reader confusion.]
One logical conclusion from recommendations 1-3 above would be that NMAs should be performed in Cochrane Intervention Reviews rather than in Overviews. There is a potential problem with this, however because of the desire within the Collaboration to minimize overlap between reviews. If an intervention including a NMA is proposed when CDSR already includes a series of Intervention reviews with a more narrow scope, many of the trials to be included in this network meta-analysis will be included in the ‘simple’ intervention reviews. There could be some overlap in the analyses as well, since specific direct comparisons would be included in the NMA review as well as in the reviews with a narrower scope.
We would suggest an agreement across the Collaboration that overlapping Intervention Reviews would be allowed in this specific case – i.e. when one Intervention Review containing an network meta-analysis overlaps with one or more other Intervention Reviews.
6 - The Collaboration should develop mechanisms to facilitate the production of Reviews that include network meta-analyses
Because of recent methodological developments in this area, the number of reviews that include a statistical approach to indirect comparisons has been growing (see our bibliography). Unfortunately, relatively few of these reviews have been published in the Cochrane Database of Systematic Reviews. In theory, Cochrane authors should have a head start in producing a review with indirect comparisons, because much of the trial data needed for these comparisons will have already been extracted for direct comparisons in other Cochrane Reviews. Unfortunately, there is currently no way to pull already extracted study characteristics or data from one Cochrane Review for use in another. Developing methods to do this would facilitate the production of Cochrane Reviews that provide rigorous comparisons of multiple interventions and should lead to the inclusion of more such reviews in CDSR.
How were these recommendations produced?
We invited a cross section of Cochrane contributors to a meeting held in conjunction with the Cochrane mid-year meeting in Paris on April 19, 2012. Minutes and slides from that meeting are available here.
A background paper explaining the rationale for these recommendations was discussed in follow-up meetings with the Coordinating Editors Board, the Methods Executive and by MARS during the Oxford 2013 mid-year meeting.
The ideas underlying our recommendations were presented for reaction in a workshop at the Auckland Colloquium (Comparing multiple treatments 1: Overview or intervention review? - Slides available here) and at the 2013 Cochrane Methods Training Event (Editorial considerations for reviews that compare multiple interventions).