PredModelMA: browser-based meta-analysis of prediction-model external validation with logit-c and log-O:E pooling and PROBAST risk of bias
Abstract
Can prediction-model external-validation studies be synthesised in a browser with the rigour of R packages such as metamisc? PredModelMA reanalyses twelve published external validations of the original Framingham Risk Score (Wilson 1998) in men, taken from Damen and colleagues' open meta-analysis dataset. It pools the c-statistic on the logit scale and the observed-to-expected ratio on the log scale (delta-method standard errors), under REML or DerSimonian–Laird random effects. The pooled c-statistic was 0.68 (95% CI 0.66–0.70, I²=67%) and the pooled O:E ratio 0.54 (0.40–0.74, I²=98%), indicating moderate discrimination but substantial over-prediction of coronary risk. Estimates reproduced metafor and an independent re-implementation to four decimal places, and the fourteen-validation calibration set matched the source review's published pooled O:E of 0.58. PROBAST charts flagged high risk of bias in most validations, while wide prediction intervals (O:E 0.16–1.87) confirm the heterogeneity is genuine, not a forced-to-zero artefact. The tool pools neither calibration-in-the-large by meta-regression nor net-benefit and decision-curve measures.
References
Damen JAAG, Pajouheshnia R, Heus P, et al. Performance of the Framingham risk models and pooled cohort equations for predicting 10-year risk of cardiovascular disease: a systematic review and meta-analysis. BMC Med. 2019;17:109. doi:10.1186/s12916-019-1340-7
Wilson PWF, D'Agostino RB, Levy D, Belanger AM, Silbershatz H, Kannel WB. Prediction of coronary heart disease using risk factor categories. Circulation. 1998;97:1837–1847. doi:10.1161/01.cir.97.18.1837
Debray TPA, Damen JAAG, Snell KIE, et al. A guide to systematic review and meta-analysis of prediction model performance. BMJ. 2017;356:i6460. doi:10.1136/bmj.i6460
Debray TPA, Damen JAAG, Riley RD, et al. A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes. Stat Methods Med Res. 2019;28:2768–2786. doi:10.1177/0962280218785504
Snell KIE, Ensor J, Debray TPA, Moons KGM, Riley RD. Meta-analysis of prediction model performance across multiple studies: which scale helps ensure between-study normality for the C-statistic and calibration measures? Stat Methods Med Res. 2018;27:3505–3522. doi:10.1177/0962280217705678
Moons KGM, Wolff RF, Riley RD, et al. PROBAST: a tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration. Ann Intern Med. 2019;170:W1–W33. doi:10.7326/M18-1377
Wolff RF, Moons KGM, Riley RD, et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med. 2019;170:51–58. doi:10.7326/M18-1376
Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010;36:1–48. doi:10.18637/jss.v036.i03
D'Agostino RB, Grundy S, Sullivan LM, Wilson P. Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA. 2001;286:180–187. doi:10.1001/jama.286.2.180
Riley RD, Higgins JPT, Deeks JJ. Interpretation of random effects meta-analyses. BMJ. 2011;342:d549. doi:10.1136/bmj.d549
Published
Issue
Section
License
Copyright (c) 2026 Mahmood Ahmad
This work is licensed under a Creative Commons Attribution 4.0 International License.
Articles in Sapience are published under the CC BY 4.0 licence. Authors retain copyright.
