Modern random-effects methods change the significance judgement of Cochrane pairwise meta-analyses in 8.2% of cases

Abstract

Does modernising RevMan's random-effects defaults from DerSimonian–Laird to REML+HKSJ with a prediction interval change the significance judgement of Cochrane pairwise meta-analyses? We re-analysed 6,386 loadable pairwise meta-analyses from 582 Cochrane reviews in the Pairwise70 corpus via the MetaAudit loader. For each meta-analysis we compared DL against REML+HKSJ+PI with a Q*/(k−1) floor and classified three flip tiers (significance, direction, clinically-important). Across the full corpus 8.2% of comparable meta-analyses flipped significance, 0.6% flipped direction, and 3.9% of the MID-available subset exceeded the clinically-important threshold. The effect concentrated in small-k and non-reproducible reviews; non-reproducible meta-analyses were approximately twice as method-sensitive as reproducible ones at every stratum examined. Method choice alone changes the significance judgement in roughly one in twelve Cochrane pairwise reviews, with the largest effect in continuous small-k analyses. Findings apply to aggregate pairwise data only; network meta-analyses, IPD reanalysis, and publication-bias methods are out of scope for v0.1.