One corpus, six lenses: how much of a Cochrane conclusion survives analyst choices, bias methods, and computational re-execution

Abstract

Background: Whether a meta-analytic conclusion is robust depends on choices that are usually examined one review and one method at a time: which specification an analyst picks, which publication-bias method is trusted, how much evidence has accumulated, what was selectively reported, and whether the numbers can be re-derived at all. These properties are rarely measured together, and almost never on a single standardised corpus large enough to make the answer a property of the evidence base rather than of a chosen example. Methods: We assembled one validated corpus -- the Pairwise70 extract of roughly 500 Cochrane pairwise reviews -- and first established its near-independence so it could serve as a single clean benchmark (the keystone), then interrogated it with five complementary forensic engines: multiverse specification search, publication-bias method discordance, cumulative analytic stability, outcome-reporting-bias excess significance, and computational re-execution from open-access sources. Denominators differ by each engine's eligibility filter (307-591 reviews; 11,899-14,340 studies) and are reconciled explicitly. Every headline rate was reused from an independently verified component package or recomputed here from the row-level artifacts. Results: The keystone holds: the corrected covered area was 0.0001, with only 5.0% of 11,899 studies appearing in more than one of 591 reviews -- the corpus is near-independent. The forensic lenses then converge. Two robustness lenses put fragility at 40.0% of 403 reviews (Fragile or Unstable across 648 specifications) and 53.4% of 365 reviews never reaching analytic stability. Two bias lenses put the flagged share near one in six: 13.7% of 307 reviews were discordant across eight publication-bias methods, and 15.2% of 473 carried high outcome-reporting-bias risk. The re-execution lens found that only 11.8% of 14,340 studies (1,688) had the open data needed to re-verify them. Across the multiverse, bias correction was the single dominant source of specification variance (eta-squared = 0.929 in 99.0% of reviews). Conclusion: One near-independent corpus, read through five complementary lenses, returns a convergent verdict: a large share of Cochrane conclusions do not survive contact with defensible analyst choices, about one in six carry a bias or selective-reporting signature, and only about one in eight can even be re-executed. Robustness is measurable at corpus scale, reproducibly and honestly -- and the most informative output is the convergence, not any single rate.