Up-to-date composition and critical appraisal of meta-analyses of comparative studies

Introduction

As of June 22^nd, 2024, in the PubMed database, of the 57,524 meta-analyses related to “surgery”, 2211 (3.8%) concerned abdominal surgery. The annual number of meta-analyses published in abdominal surgery has risen steadily, from 118 in 2013 to 247 in 2023.

Despite this increase in terms of quantity, there are many common errors committed either in the conception, or the analysis, whether for an original meta-analysis article or in the discussion of any manuscript to support the results published in meta-analyses today (1).

This article aimed to (I) describe the main features of a meta-analysis comparing two procedures, (II) assess the methodology of meta-analyses published between July and December 2023 in 10 leading surgical journals, and (III) evaluate the impact of the most common errors encountered in the literature.

Main characteristics of a meta-analysis comparing two interventions

According to the Cochrane website (2), meta-analysis is the statistical combination of results from two or more separate studies, ideally with the same endpoint(s). The goal is to improve the precision, the ability to answer questions that are not posed by individual studies or to settle controversies arising from conflicting claims. Variations across studies (either clinical or statistical heterogeneity) must be considered and analyzed to ascertain its causes. Prediction intervals (PIs) derived from random-effects meta-analyses are useful to present the extent of between-study variation. Lastly, publication bias should be evaluated using the Funnel plot followed by Duval and Tweedie’s Trim & Fill methods.

This allows us to compare subgroups, which may be intervention variants, different types of patients, with different follow-up periods.

The seven essential steps necessary to compose, publish and critically appraise a meta-analysis of comparative studies are summarized in Table 1.

Step 1

The aim should be formulated according to the acronym of Population, Intervention, Comparison, Outcome Measures and Study Design (PICOS): the PICOS acronym can also be used to identify key words.

Step 2

The search must be as broad as possible, it should meet the PICOS acronym. Search sources may include: (I) MEDLINE, (II) Embase, (III) Scopus, (IV) Cochrane Central Register of Controlled Trials, (V) other databases such as Sciences Direct, Web Sciences, Ovid, Google Scholar, and (VI) if possible additional hand search.

Step 3

A rigorous and objective selection of relevant studies should be performed according to PICOS. At least two independent reviewers should work in parallel, according to clearly defined inclusion criteria. Any discrepancy should be resolved by discussion, or a third person (another co-author or senior author).

Step 4

Critical appraisal of selected studies to assess quality of methodology, based on three questions: (I) What is the internal validity of the study (methodology), (II) What is the external validity (exportable results), (III) Are the results applicable to my patient? Assessment of quality of randomized controlled trial (RCT) relies on the CONSORT guidelines (3) or Jadad score (4), while for observational studies the MINORS (5) or Newcastle-Ottawa Scale (NOS) (6) are appropriate.

Step 5

Extraction of relevant data from each study should be done by two authors independently and disparities settled after discussion with a third author.

Step 6: data synthesis (quantitative analysis)

Assessment of the mean effect size. This depends on the type of variables

• For dichotomous variables, either the risk ratio (RR) [i.e., the risk of event (e.g., death) in the experimental treated group/risk of event (death) in the control group] or the odds ratio (OR) [i.e., the odds of the event occurring (e.g., death) (ratio of dying to living) in the experimental (treated) group/odds of the event occurring (ratio of dying to living) in the control group]; risk difference (RD) can be used. Each estimate should be accompanied by their 95% confidence interval (CI) (measure of certainty in the sampling method) an index for precision (7) (the range of values that likely would contain an unknown population parameter when a random sample is drawn several times). Of note, the RR and OR are insensitive to differences in baseline events, while the RD is an absolute measure, very sensitive to the baseline risk (7).
• For continuous variables: the effect can be calculated as the difference in means, or standard difference in means (7). The difference in means is used when all studies in the analysis use the same scale. The standardized difference in means (Cohen’s d or Hedges’g) transforms all effect sizes to a common metric, helping researchers to use different outcome measures in the same meta-analysis (7,8).

Fixed-effect vs. random effects models

Borenstein et al. (9) mentioned that “The fact that these two models use similar formulas to compute statistics, sometimes they provide similar estimates, many people believe that the models are equivalent”. These two models correspond to different hypotheses (9). The selection of the appropriate model is important to ensure that results are estimated correctly (9).

• If patients, interventions and endpoints are sufficiently similar, and the aim is to compute the common effect size, the “fixed-effect model” is appropriate. However, the mean effect size, (calculated with the fixed effect) cannot be generalized for other populations not included in the analysis (9).
• If patients, interventions and endpoints are not similar, the “random-effect model” (Der Simonian & Laird’s model), the random-effects model is more appropriate. The mean effect size, (calculated with the random-effect model) can be generalized to other populations not included in the analysis (9).
• For both models, the Z statistic test should be applied (Z-value tests the null hypothesis) and a forest plot should summarize the synthesis providing the mean effect size with its CI.

Heterogeneity

Heterogeneity refers to the dispersion of true effects across studies included in the meta-analysis. The result of each study is based on a unique population, which varies from one population to another. All variations that lead to the result, or the way they are assessed may vary from study to study, among others, and may have an impact on the effect size; this variation is the heterogeneity. Heterogeneity should be assessed by the Cochrane Chi² test (Q test), the variance Tau² which is used to calculate the 95% PI, an index of dispersion (10,11). The goal of the Q-test is to address the question “Is there evidence that the true effect size varies from one study to another?” The PI provides a range of values that would contain the value of a single new observation, based on previous data. PI aims to answer the question “how much the effect size varies?” When Tau² is equal to zero, there is no heterogeneity, and the PI is reduced to a point which is the mean effect size, considered as the true effect size and all populations will have this true effect size. The presence of heterogeneity corresponds to Tau² >0. Of note, the estimation of heterogeneity needs at least ten studies (10,11). PI intervene in the clinical interpretation of heterogeneity in that they estimate the true treatment effect that can be expected in future investigations. The greater the heterogeneity, the larger the PI (and the difference in the limits of the range of CI). This is why the PI can lead to different conclusions.

How to explain heterogeneity: is there publication bias and what are the reasons of heterogeneity?

• Publication bias is evaluated using the funnel plot followed by the Duval and Tweedie’s Trim & Fill methods. Egger’s regression is a third option to assess the potential impact of publication bias (12).
• Reasons for heterogeneity should be investigated by testing interactions between relevant factors termed moderators and the effect size using subgroup analysis and/or meta-regression.

Step 7: conclusions and recommendations

These seven essential steps, necessary to compose, publish and critically appraise a meta-analysis of comparative studies, mentioned above and summarized in Table 1, are in accordance with the content of the Cochrane Handbook for Systematic Reviews of Interventions (13) regarding the approach with GRADE evaluation.

Survey: errors identified in meta-analyses published in surgery in ten journals between July 1, 2023 and December 31, 2023

Methodology

We selected the top ten visceral surgery journals in which we searched for meta-analyses published from July 1 to December 31, 2023. The variables sought were (Table 2): (I) design of included studies [RCT, non randomized studies of interventions (NRSI), RCT + NRSI], (II) effect size categorical [RR, OR, RD, hazard ratio (HR) or combination], (III) continuous effect size (mean difference, Cohen d index), (IV) models (not mentioned, fixed model, random model, fixed & random), (V) heterogeneity evaluation (not done, I² index, Tau² variance + I²), (VI) heterogeneity explanation (not done, subgroup analysis, meta-regression, subgroup analysis + meta-regression), (VII) publication bias (not done, funnel plot, Egger test, funnel plot + Egger test). A descriptive analysis was performed and each variable was represented by its percentage.

Results

We collected 50 meta-analyses (33 full texts and 17 abstracts) and focused on full-text meta-analyses (Table 2).

Of the 33 full-text meta-analyses: 36% were meta-analyses of randomized trials while 64% included observational studies associated or not with randomized trials. Four out of 13 meta-analyses of randomized trials used RR. The random model was applied in 20 (61%) of cases, the fixed model in one (3%) and both models at the same time in 12 (36%) of cases. Heterogeneity was assessed by the I² index in 30 (91%) while the PI was never calculated. Publication bias was not investigated in 16 (49%) of cases.

Some incorrect expressions were mentioned, such as “when heterogeneity was suitably low, a fixed-effects model was used to cluster the data” or “a random-effects model was selected, otherwise a fixed-effects model was applied”. These assertions corresponded to the use of both fixed and random models for I² full text (meta-analyses 36%) (Table 2). Moreover, statements such as—heterogeneity was assessed using the I² metric were classified as “might not be significant (0–40%)”, “might represent moderate heterogeneity (30–60%)”, “might represent substantial heterogeneity (50–90%)”, and “considerable heterogeneity (75–100%)”—often accompanied the using of I² in 91% (Table 2).

Comments regarding the above survey

The four steps in section 6 (Table 1) were not respected. These mistakes can lead to misinterpretation of data results. The effect size for meta-analysis of RCT should be calculated with the RR. For meta-analyses of observational studies associated or not with RCT, the OR is the most appropriate.

Only the random model should be used in clinical research studies to be able to make a universal conclusion.

Heterogeneity should be assessed with the PI. The I² index is no longer appropriate. However, one of the problems today is that not all statisticians agree on what is essential, and formulas and metrics such as the I² are still widely used. All we want to highlight is the additional value of predictive intervals. All these concepts should be mentioned in articles reporting a meta-analysis.

We recommend that all readers be aware that these errors can reduce the quality of the meta-analysis results.

If you want to use a meta-analysis to support (or refute) your findings in the discussion of your paper, these steps and critical analysis of the meta-analysis are highly relevant.

Consequences of methodological errors, how to avoid the main methodological errors?

In this section, we explain the consequences of these errors when interpreting data. Two scenarios are proposed to illustrate the impact of these errors:

Scenario 1

Larkins et al. (14) reported a systematic review with meta-analysis in 2022 comparing robotic to laparoscopic surgery. The conclusion was that robotic surgery gives lower conversion (in all cases). The authors used the I² to evaluate heterogeneity (15). I² was equal to zero, therefore they claimed there was no heterogeneity.

Comments of scenario 1

• The authors used the OR which is correct but the RD would have been more clinically meaningful. The RD is more sensitive to baseline risks. (7)
• A letter to the editor (16) recalculated the effect size using the RD and applied the PI statistics (10,11). The PI included zero. This shows that some populations benefited from robotic surgery [lower conversion (RD <0)], while other similar populations would benefit from laparoscopy [lower conversion (RD >0) or perhaps, robotic surgery could be ‘harmful’ for these populations] (15). This conclusion differs from that reported in the article of Larkins: who concluded “that robotic surgery is associated with lower conversion rates” (14).

Scenario 2

Stoop et al.’s meta-analysis (17) entitled Systematic Review and Meta-analysis of the Role of Total Pancreatectomy as an Alternative to Pancreaticoduodenectomy in Patients at High Risk for Postoperative Pancreatic Fistula: Is it a Justifiable Indication? was published in the Annals of Surgery 2023. The authors mentioned that “The Mantel-Haenszel random-effect model was used for the outcome measures major morbidity and mortality to calculate pooled RR with their corresponding 95% CIs” (17). They concluded that the 90-day mortality did not differ, however “Major morbidity rate was lower after TP compared to PD [26.7% vs. 38.3%; RR =0.65 (95% CI: 0.48–0.89)]” (17).

Comments of scenario 2

In this paper, the authors mixed one (underpowered) randomized trial with two single center observational matched studies, and two single-center observational non-matched studies. Although meta-analyses were originally designed to sum up effect sizes from only randomized trials, there are increasing numbers of meta-analyses that include non-randomized studies today. However, mixing randomized studies with non-randomized studies raises several dilemmas. Methodology exists to allow this sort of mix (18) but as far as we can determine, the authors did not use it.

The conclusion “Major morbidity rate was lower after TP compared to PD [26.7% vs. 38.3%; RR =0.65 (95% CI: 0.48–0.89)]” (17) assumed that this meta-analysis concerned only randomized trials. However, the mixed design of this meta-analysis does not allow any inference of causality to be made. The authors reported RR in their analysis. RRs are inappropriate for retrospective studies, because the total number of exposed participants is theoretically unavailable. This is another problem when the meta-analyses mix the two types of studies, and the solution is unsatisfactory.

Last, the authors assessed heterogeneity with the I² statistic. It is now widely admitted, and even by the inventor of the I² statistic, himself, JP Higgins (15), that this concept is wrong (1,15,19). Heterogeneity should be measured by the Tau² statistic and PIs, not the I² statistic (15,20). PIs correspond to how much the true effect size varies. The authors indicated the Tau² in their forest plots. Concerning the outcome mortality, they indicated that the Tau² was = 0, which corresponds to no heterogeneity (17). However, concerning the outcome of morbidity, the Tau² varied from 0.0497 for major morbidity in studies without high risk of bias to 0.1204 for morbidity in studies with only TP performed because of a high risk for postoperative pancreatic fistula (POPF).

Concerning the outcome of major morbidity for overall patients (17), the heterogeneity assessed with the I² was 35%, which would correspond to “might not be important” according to their methodology (17). We redid the calculations with CMA version 4 for the major morbidity (Figure 1), including five studies (21-25) as referenced in Stoop et al.’s study (17). Like the authors, we found that the diamond was to the left of the null effect line, in favor of TP, but, to the contrary of the authors’ forest plot, the PI crosses the line of 1 (no effect). In Figure 1, we found an OR of 0.501 with a 95% CI ranging from 0.307 to 0.817 and a 95% PI ranging from 0.140 to 1.787. While CIs indicate the upper and lower limits within which the true value lies with a certain degree (usually 95%) of probability (or the variance due to sampling, rather than considering the entire population), PIs indicate the upper and lower limits within which the value of a single new observation, based on previous data, may lie. In other words, similar populations could have a reduction of major morbidity following TP (morbidity is 0.14 times less likely to occur after TP) while other similar populations could also experience an increase in the risk of major morbidity (morbidity is 1.787 times more likely to occur after TP); this notion is completely in contradiction with the authors’ conclusion. In practical terms, the CI used in Stoop et al.’s paper (17) is an index of precision of what was found in the studies, while the predictive interval is an index of dispersion, which can be expected for future estimations (1,15,19).

Figure 1 Forest plot showing major morbidity for overall patients with effect size represented by odds ratio, confidence interval and prediction interval. The included studies in this figure (21-25) were mentioned in the Stoop et al.’s meta-analysis (17). The reference of Stoop et al. (22) published in 2022 contained three stratums I, II and III which were also reported in the figure. TP, total pancreatectomy; PD, pancreatoduodenectomy.

In conclusion, we voice a word of caution in the interpretation of this paper: firstly, because not all the studies were randomized, no causality can be inferred. The authors should add a comment to their paper stating this clearly so that the readership does not go away with the message that there is (always) less morbidity after TP. Secondly, heterogeneity can taint the overall results of meta-analyses and must be accounted for; the Tau² statistic and associated PI should replace the currently commonly reported I² statistic. CI and PI transmit different but complementary information (26), and thirdly, there is indeed a need for greater rigor in coming to conclusions when dealing with non-randomized or underpowered trials.

Conclusions

The main methodological errors in the composition and appraisal of studies for meta-analyses to avoid are:

• Choice of the appropriate effect size: the difference in means is used when all studies in the analysis used the same scale. The standardized difference in means (Cohen’s d or Hedges’g) can be used to transform all effect sizes to a common metric, helping researchers to use different outcome measures in the same meta-analysis. The RR and OR can be used but one has to remember that they are insensitive to differences in baseline events, while the RD is an absolute measure, very sensitive to the baseline risk.
• The random-effects model is recommended for meta-analysis of published studies in literature. It is incorrect to start with the fixed-effect model, and then switch to the random-effects model. This corresponds to the fact that heterogeneity across studies is statistically significant (9).
• Neither the I-squared statistic (I²) nor the classification of heterogeneity as being “low”, “moderate”, or “high” based on the I² (15,26) are correct and should no longer be used alone. The PI (based on the Tau² statistic) “how much the true effect size varies” is the correct metric to evaluate heterogeneity.
• The PI is the only statistic which evaluates heterogeneity “how much the true effect size varies?”
• When heterogeneity is present, this does not decrease the utility of meta-analysis (27). It is the authors’ responsibility to explain the heterogeneity and how it impacts the analysis.
• Reasons for heterogeneity should be investigated by testing interactions between relevant factors termed moderators and the effect size using subgroup analysis and/or meta-regression.
• Publication bias should be evaluated using the funnel plot followed by Duval and Tweedie’s Trim & Fill methods. Egger’s regression is a third option to assess the potential impact of publication bias.

Acknowledgments

Funding: None.

Footnote

Peer Review File: Available at https://ales.amegroups.com/article/view/10.21037/ales-24-26/prf

Conflicts of Interest: Both authors have completed the ICMJE uniform disclosure form (available at https://ales.amegroups.com/article/view/10.21037/ales-24-26/coif). A.F. serves as the co-Editor-in-Chief of Annals of Laparoscopic and Endoscopic Surgery from April 2016 to April 2026. C.D. reports that he is an Editorial Board Member of the American Journal of Surgery. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Dziri C, Slim K. What the surgeons need to know about meta-analyses in 2023, beyond the simple Odds ratio. J Visc Surg 2022;159:480-5. [Crossref] [PubMed]
2. Deeks JJ, Higgins JPT, Altman DG, et al. Chapter 10: Analysing data and undertaking meta-analyses. In: Higgins JPT, Thomas J, Chandler J, et al. editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023). Cochrane; 2023.
3. Schulz KF, Altman DG, Moher D, et al. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Obstet Gynecol 2010;115:1063-70. [Crossref] [PubMed]
4. Clark HD, Wells GA, Huët C, et al. Assessing the quality of randomized trials: reliability of the Jadad scale. Control Clin Trials 1999;20:448-52. [Crossref] [PubMed]
5. Slim K, Nini E, Forestier D, et al. Methodological index for non-randomized studies (minors): development and validation of a new instrument. ANZ J Surg 2003;73:712-6. [Crossref] [PubMed]
6. Wells GA, Shea B, O’Connell D, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality if nonrandomised studies in meta-analyses. Available online: http://www.ohri.ca/programs/clinical_epidemiology/oxford.htm
7. Borenstein M, Hedges LV, Higgins JPT, et al. Choosing an effect size index. In: Introduction to meta-analysis. 2nd ed. Oxford: Wiley; 2021:38.
8. Borenstein M, Hedges LV, Higgins JPT, et al. Introduction to meta-analysis, effect size based on means: standardized mean difference, d and g. John Wiley & Sons; 2009:25-30.
9. Borenstein M, Hedges LV, Higgins JP, et al. A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Methods 2010;1:97-111. [Crossref] [PubMed]
10. Borenstein M, Hedges LV, Higgins JPT, et al. Introduction to Meta-Analysis, Second Edition, Prediction Intervals (Chapter 17) - Oxford. John Wiley & Sons; 2021:119-25.
11. Borenstein M. Common Mistakes in Meta-Analysis and How to Avoid Them, the Prediction Interval. New Jersey (USA): Biostat; 2019:85–93.
12. Borenstein M, Hedges LV, Higgins JPT, et al. Introduction to Meta-Analysis, Second Edition, Publication bias (Chapter 35). Oxford: John Wiley & Sons; 2021:313-31.
13. Schünemann HJ, Higgins JPT, Vist GE, et al. Chapter 14: Completing ‘Summary of findings’ tables and grading the certainty of the evidence. In: Higgins JPT, Thomas J, Chandler J, et al. editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023). Cochrane, 2023.
14. Larkins K, Mohan H, Apte SS, et al. A systematic review and meta-analysis of robotic resections for diverticular disease. Colorectal Dis 2022;24:1105-16. [Crossref] [PubMed]
15. Borenstein M, Higgins JP, Hedges LV, et al. Basics of meta-analysis: I2 is not an absolute measure of heterogeneity. Res Synth Methods 2017;8:5-18. [Crossref] [PubMed]
16. Dziri C. Evaluation of heterogeneity in meta-analysis. Colorectal Dis 2023;25:1037-8. [Crossref] [PubMed]
17. Stoop TF, Bergquist E, Theijse RT, et al. Systematic Review and Meta-analysis of the Role of Total Pancreatectomy as an Alternative to Pancreatoduodenectomy in Patients at High Risk for Postoperative Pancreatic Fistula: Is it a Justifiable Indication? Ann Surg 2023;278:e702-11. [Crossref] [PubMed]
18. Efthimiou O, Mavridis D, Debray TP, et al. Combining randomized and non-randomized evidence in network meta-analysis. Stat Med 2017;36:1210-26. [Crossref] [PubMed]
19. Borenstein M. Avoiding common mistakes in meta-analysis: Understanding the distinct roles of Q, I-squared, tau-squared, and the prediction interval in reporting heterogeneity. Res Synth Methods 2024;15:354-68. [Crossref] [PubMed]
20. Borenstein M. Research Note: In a meta-analysis, the I2 index does not tell us how much the effect size varies across studies. J Physiother 2020;66:135-39. [Crossref] [PubMed]
21. Hempel S, Oehme F, Tahirukaj E, et al. More is More? Total Pancreatectomy for Periampullary Cancer as an Alternative in Patients with High-Risk Pancreatic Anastomosis: A Propensity Score-Matched Analysis. Ann Surg Oncol 2021;28:8309-17. [Crossref] [PubMed]
22. Stoop TF, Ghorbani P, Scholten L, et al. Total pancreatectomy as an alternative to high-risk pancreatojejunostomy after pancreatoduodenectomy: a propensity score analysis on surgical outcome and quality of life. HPB (Oxford) 2022;24:1261-70. [Crossref] [PubMed]
23. Balzano G, Zerbi A, Aleotti F, et al. Total Pancreatectomy With Islet Autotransplantation as an Alternative to High-risk Pancreatojejunostomy After Pancreaticoduodenectomy: A Prospective Randomized Trial. Ann Surg 2023;277:894-903. [Crossref] [PubMed]
24. Marchegiani G, Perri G, Burelli A, et al. High-risk Pancreatic Anastomosis Versus Total Pancreatectomy After Pancreatoduodenectomy: Postoperative Outcomes and Quality of Life Analysis. Ann Surg 2022;276:e905-13. [Crossref] [PubMed]
25. Luu AM, Olchanetski B, Herzog T, et al. Is primary total pancreatectomy in patients with high-risk pancreatic remnant justified and preferable to pancreaticoduodenectomy? -a matched-pairs analysis of 200 patients. Gland Surg 2021;10:618-28. [Crossref] [PubMed]
26. Chiolero A, Santschi V, Burnand B, et al. Meta-analyses: with confidence or prediction intervals? Eur J Epidemiol 2012;27:823-5. [Crossref] [PubMed]
27. Borenstein M. Systematic Reviews and Meta-Analyses in Regional Anesthesia and Pain Medicine. Anesth Analg 2024;138:376-8. [Crossref] [PubMed]