Hannun, A. Y. et al. Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nat. Med. 25, 65–69 (2019).
Esteva, A. et al. A guide to deep learning in healthcare. Nat. Med. 25, 24–29 (2019).
Pirracchio, R. et al. Big data and targeted machine learning in action to assist medical decision in the ICU. Anaesth. Crit Care Pain Med. 38, 377–384 (2019).
Liu, S. et al. Reinforcement learning for clinical decision support in critical care: comprehensive review. J. Med. Internet Res. 22, e18477 (2020).
Adegboro, C. O., Choudhury, A., Asan, O. & Kelly, M. M. Artificial intelligence to improve health outcomes in the NICU and PICU: a systematic review. Hosp Pediatr 12, 93–110 (2022).
Choudhury, A. & Asan, O. Role of artificial intelligence in patient safety outcomes: systematic literature review. JMIR Med Inform. 8, e18599 (2020).
Benjamens, S., Dhunnoo, P. & Meskó, B. The state of artificial intelligence-based (fda-approved) medical devices and algorithms: an online database. NPJ Digit Med 3, 118 (2020).
Sculley, D. et al. Machine Learning: The High Interest Credit Card of Technical Debt. In Advances In Neural Information Processing Systems, vol. 28 (eds. Cortes, C., Lawrence, N., Lee, D., Sugiyama, M. & Garnett, R.) (Curran Associates, Inc., 2015).
Davis, S. E., Lasko, T. A., Chen, G., Siew, E. D. & Matheny, M. E. Calibration drift in regression and machine learning models for acute kidney injury. J. Am. Med. Inform. Assoc. 24, 1052–1061 (2017).
Chen, J. H., Alagappan, M., Goldstein, M. K., Asch, S. M. & Altman, R. B. Decaying relevance of clinical data towards future decisions in data-driven inpatient clinical order sets. Int. J. Med. Inform. 102, 71–79 (2017).
Nestor, B. et al. Feature robustness in non-stationary health records: caveats to deployable model performance in common clinical machine learning tasks. Machine Learning for Healthcare 106, 381–405 (2019).
Yoshida, E., Fei, S., Bavuso, K., Lagor, C. & Maviglia, S. The value of monitoring clinical decision support interventions. Appl. Clin. Inform. 9, 163–173 (2018).
Lee, C. S. & Lee, A. Y. Clinical applications of continual learning machine learning. Lancet Digital Health 2, e279–e281 (2020).
Vokinger, K. N., Feuerriegel, S. & Kesselheim, A. S. Continual learning in medical devices: FDA’s action plan and beyond. Lancet Digital Health 3, e337–e338 (2021).
U.S. Food and Drug Administration. Proposed regulatory framework for modifications to artificial intelligence/machine learning (AI/ML)-based software as a medical device (SaMD): discussion paper and request for feedback. Tech. Rep. (2019).
Liu, Y., Chen, P.-H. C., Krause, J. & Peng, L. How to read articles that use machine learning: Users’ guides to the medical literature. JAMA 322, 1806–1816 (2019).
Finlayson, S. G. et al. The clinician and dataset shift in artificial intelligence. N. Engl. J. Med. 385, 283–286 (2021).
Breck, E., Cai, S., Nielsen, E., Salib, M. & Sculley, D. The ML test score: A rubric for ML production readiness and technical debt reduction. In: 2017 IEEE International Conference on Big Data (Big Data), 1123–1132 (ieeexplore.ieee.org, 2017).
Amershi, S. et al. Software engineering for machine learning: a case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 291–300 (2019).
Benneyan, J. C., Lloyd, R. C. & Plsek, P. E. Statistical process control as a tool for research and healthcare improvement. Qual. Saf. Health Care 12, 458–464 (2003).
Thor, J. et al. Application of statistical process control in healthcare improvement: systematic review. Qual. Saf. Health Care 16, 387–399 (2007).
Backhouse, A. & Ogunlayi, F. Quality improvement into practice. BMJ 368, m865 (2020).
Hatib, F. et al. Machine-learning algorithm to predict hypotension based on high-fidelity arterial pressure waveform analysis. Anesthesiology 129, 663–674 (2018).
Duckworth, C. et al. Using explainable machine learning to characterise data drift and detect emergent health risks for emergency department admissions during COVID-19. Sci. Rep. 11, 23017 (2021).
Rubin, D. L. Artificial intelligence in imaging: The radiologist’s role. J. Am. Coll. Radiol. 16, 1309–1317 (2019).
Gossmann, A., Cha, K. H. & Sun, X. Performance deterioration of deep neural networks for lesion classification in mammography due to distribution shift: an analysis based on artificially created distribution shift. In: Medical Imaging 2020: Computer-Aided Diagnosis, Vol. 11314, (eds. Hahn, H. K. & Mazurowski, M. A.)1131404 (International Society for Optics and Photonics, 2020).
Cabitza, F. et al. The importance of being external. methodological insights for the external validation of machine learning models in medicine. Comput. Methods Programs Biomed. 208, 106288 (2021).
Subbaswamy, A., Schulam, P. & Saria, S. Preventing failures due to dataset shift: Learning predictive models that transport. In: Proc. Machine Learning Research Vol. 89 (eds. Chaudhuri, K. & Sugiyama, M.) 3118–3127 (PMLR, 2019).
Schölkopf, B. et al. On causal and anticausal learning. In: Proc. 29th International Coference on International Conference on Machine Learning, ICML’12 459–466 (Omnipress, 2012).
Quionero-Candela, J., Sugiyama, M., Schwaighofer, A. & Lawrence, N. D. Dataset Shift in Machine Learning (The MIT Press, 2009).
Montgomery, D. Introduction to Statistical Quality Control (Wiley, 2020).
Aggarwal, C. C. An introduction to outlier analysis. In: Outlier analysis 1–34 (Springer, 2017).
Greenland, S., Pearl, J. & Robins, J. M. Causal diagrams for epidemiologic research. Epidemiology 10, 37–48 (1999).
Castro, D. C., Walker, I. & Glocker, B. Causality matters in medical imaging. Nat. Commun. 11, 3673 (2020).
Page, E. S. Continuous inspection schemes. Biometrika 41, 100–115 (1954).
Bersimis, S., Psarakis, S. & Panaretos, J. Multivariate statistical process control charts: an overview. Qual. Reliab. Eng. Int. 23, 517–543 (2007).
Zou, C. & Qiu, P. Multivariate statistical process control using LASSO. J. Am. Stat. Assoc. 104, 1586–1596 (2009).
Qahtan, A. A., Alharbi, B., Wang, S. & Zhang, X. A PCA-Based change detection framework for multidimensional data streams: change detection in multidimensional data streams. In: Proc. 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 935–944 (Association for Computing Machinery, 2015).
Boracchi, G., Carrera, D., Cervellera, C. & Macciò, D. QuantTree: Histograms for change detection in multivariate data streams. In: Proc. 35th International Conference on Machine Learning Vol. 80 (eds. Dy, J. & Krause, A.) 639–648 (PMLR, 2018).
Rabanser, S., Günnemann, S. & Lipton, Z. Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. In: Advances in Neural Information Processing Systems Vol. 32 (eds. Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E. & Garnett, R.) 1396–1408 https://proceedings.neurips.cc/paper/2019/file/846c260d715e5b854ffad5f70a516c88-Paper.pdf (Curran Associates, Inc., 2019).
Qiu, P. Big data? statistical process control can help! Am. Stat. 74, 329–344 (2020).
Ditzler, G. & Polikar, R. Hellinger distance based drift detection for nonstationary environments. In: 2011 IEEE Symposium on Computational Intelligence in Dynamic and Uncertain Environments (CIDUE) 41-48 (2011).
Gretton, A., Borgwardt, K., Rasch, M., Schölkopf, B. & Smola, A. A kernel method for the Two-Sample-Problem. In: Advances in Neural Information Processing Systems Vol. 19 (eds. Schölkopf, B., Platt, J. & Hoffman, T.) (MIT Press, 2007).
Harchaoui, Z., Moulines, E. & Bach, F. Kernel change-point analysis. In Advances in Neural Information Processing Systems Vol. 21 (eds. Koller, D., Schuurmans, D., Bengio, Y. & Bottou, L.) (Curran Associates, Inc., 2009).
Williamson, B. D. & Feng, J. Efficient nonparametric statistical inference on population feature importance using shapley values. In: Proc. of the 37th International Conference on Machine Learning Vol. 119 (eds. Daumé. H. III & Singh, A.) 10282–10291 (PMLR, 2020).
Nishida, K. & Yamauchi, K. Detecting Concept Drift Using Statistical Testing. In: Discovery Science 264–269 https://doi.org/10.1007/978-3-540-75488-6_27 (Springer Berlin Heidelberg, 2007).
Shiryaev, A. N. On optimum methods in quickest detection problems. Theory Probab. Appl. 8, 22–46 (1963).
Roberts, S. W. A comparison of some control chart procedures. Technometrics 8, 411–430 (1966).
Siegmund, D. & Venkatraman, E. S. Using the generalized likelihood ratio statistic for sequential detection of a Change-Point. Ann. Statistics 23, 255–271 (1995).
Lai, T. L. & Xing, H. Sequential change-point detection when the pre- and post-change parameters are unknown. Seq. Anal. 29, 162–175 (2010).
Zeileis, A. & Hornik, K. Generalized m-fluctuation tests for parameter instability. Stat. Neerl. 61, 488–508 (2007).
Davis, S. E., Greevy, R. A. Jr., Lasko, T. A., Walsh, C. G. & Matheny, M. E. Detection of calibration drift in clinical prediction models to inform model updating. J. Biomed. Inform. 112, 103611 (2020).
Zou, C. & Tsung, F. Likelihood ratio-based distribution-free EWMA control charts. J. Commod. Sci. Technol. Qual. 42, 174–196 (2010).
Shin, J., Ramdas, A. & Rinaldo, A. Nonparametric Iterated-Logarithm extensions of the sequential generalized likelihood ratio test. IEEE J. Sel. Areas in Inform. Theory 2, 691–704 (2021).
Leonardi, F. & Bühlmann, P. Computationally efficient change point detection for high-dimensional regression Preprint at https://doi.org/10.48550/ARXIV.1601.03704 (arXiv, 2016).
Enikeeva, F. & Harchaoui, Z. High-dimensional change-point detection under sparse alternatives. Ann. Stat. 47, 2051–2079 (2019).
Liu, L., Salmon, J. & Harchaoui, Z. Score-Based change detection for Gradient-Based learning machines. In: ICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 4990–4994 (2021).
Woodall, W. H. The use of control charts in health-care and public-health surveillance. J. Qual. Technol. 38, 89–104 (2006).
Huang, Y. & Gilbert, P. B. Comparing biomarkers as principal surrogate endpoints. Biometrics 67, 1442–1451 (2011).
Price, B. L., Gilbert, P. B. & van der Laan, M. J. Estimation of the optimal surrogate based on a randomized trial. Biometrics 74, 1271–1281 (2018).
Asan, O. & Choudhury, A. Research trends in artificial intelligence applications in human factors health care: mapping review. JMIR Hum. Factors 8, e28236 (2021).
Paxton, C., Niculescu-Mizil, A. & Saria, S. Developing predictive models using electronic medical records: challenges and pitfalls. AMIA Annu. Symp. Proc. 2013, 1109–1115 (2013).
Dyagilev, K. & Saria, S. Learning (predictive) risk scores in the presence of censoring due to interventions. Mach. Learn. 102, 323–348 (2016).
Lenert, M. C., Matheny, M. E. & Walsh, C. G. Prognostic models will be victims of their own success, unless. J. Am. Med. Inform. Assoc. 26, 1645–1650 (2019).
Perdomo, J., Zrnic, T., Mendler-Dünner, C. & Hardt, M. Performative prediction. In Proc. of the 37th International Conference on Machine Learning Vol. 119 (eds. Daumé. H. III & Singh, A.) 7599–7609 http://proceedings.mlr.press/v119/perdomo20a/perdomo20a.pdf (PMLR, 2020).
Liley, J. et al. Model updating after interventions paradoxically introduces bias. Int. Conf. Artif. Intell. Statistics 130, 3916–3924 (2021).
Imbens, G. W. & Rubin, D. B. Causal Inference in Statistics, Social, and Biomedical Sciences (Cambridge University Press, 2015).
Zeileis, A., Leisch, F., Hornik, K. & Kleiber, C. strucchange: an r package for testing for structural change in linear regression models. J. Statistical Softw. 7, 1–38 (2002).
Harrison, D. A., Brady, A. R., Parry, G. J., Carpenter, J. R. & Rowan, K. Recalibration of risk prediction models in a large multicenter cohort of admissions to adult, general critical care units in the united kingdom. Crit. Care Med. 34, 1378–1388 (2006).
van den Boogaard, M. et al. Recalibration of the delirium prediction model for ICU patients (PRE-DELIRIC): a multinational observational study. Intensive Care Med. 40, 361–369 (2014).
Babic, B., Gerke, S., Evgeniou, T. & Cohen, I. G. Algorithms on regulatory lockdown in medicine. Science 366, 1202–1204 (2019).
European Medicines Agency. Regulation (EU) 2017/745 of the european parliament and of the council. Tech. Rep. (2020).
Ensign, D., Friedler, S. A., Neville, S., Scheidegger, C. & Venkatasubramanian, S. Runaway feedback loops in predictive policing. In: Accountability and Transparency Vol. 81 (eds. Friedler, S. A. & Wilson, C.) 160–171 (PMLR, 2018).
Hashimoto, T., Srivastava, M., Namkoong, H. & Liang, P. Fairness without demographics in repeated loss minimization. In Proc. 35th International Conference on Machine Learning Vol. 80 (eds. Dy, J. & Krause, A.) 1929–1938 (PMLR, 2018).
Liu, L. T., Dean, S., Rolf, E., Simchowitz, M. & Hardt, M. Delayed Impact of Fair Machine Learning Vol. 80, 3150-3158 (PMLR, 2018).
Chouldechova, A. & Roth, A. The frontiers of fairness in machine learning Preprint at https://doi.org/10.48550/ARXIV.1810.08810 (arXiv, 2018).
Hastie, T., Tibshirani, R. & Friedman, J. The Elements of Statistical Learning (Springer, 2009) .
James, G., Witten, D., Hastie, T. & Tibshirani, R. An Introduction to Statistical Learning (Springer, 2021).
Platt, J. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Adv. Large Margin Classifiers 10, 61–74 (1999).
Niculescu-Mizil, A. & Caruana, R. Predicting good probabilities with supervised learning. In: Proc. 22nd international conference on Machine learning, ICML’05 625–632 (Association for Computing Machinery, 2005).
Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. On calibration of modern neural networks. Int. Conf. Mach. Learning 70, 1321–1330 (2017).
Chen, W., Sahiner, B., Samuelson, F., Pezeshk, A. & Petrick, N. Calibration of medical diagnostic classifier scores to the probability of disease. Stat. Methods Med. Res. 27, 1394–1409 (2018).
Steyerberg, E. W. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating (Springer, 2009). .
Steyerberg, E. W., Borsboom, G. J. J. M., van Houwelingen, H. C., Eijkemans, M. J. C. & Habbema, J. D. F. Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat. Med. 23, 2567–2586 (2004).
Benkeser, D., Ju, C., Lendle, S. & van der Laan, M. Online cross-validation-based ensemble learning. Statistics Med. 37, 249–260 (2018).
McCormick, T. H. Dynamic logistic regression and dynamic model averaging for binary classification. Biometrics 68, 23–30 (2012).
Strobl, A. N. et al. Improving patient prostate cancer risk assessment: Moving from static, globally-applied to dynamic, practice-specific risk calculators. J. Biomed. Inform. 56, 87–93 (2015).
Futoma, J., Simons, M., Panch, T., Doshi-Velez, F. & Celi, L. A. The myth of generalisability in clinical research and machine learning in health care. Lancet Digit Health 2, e489–e492 (2020).
Vokinger, K. N., Feuerriegel, S. & Kesselheim, A. S. Continual learning in medical devices: FDA’s action plan and beyond. Lancet Digit Health 3, e337–e338 (2021).
Viering, T. J., Mey, A. & Loog, M. Making learners (more) monotone. In: Advances in Intelligent Data Analysis XVIII (eds. Berthold, M. R., Feelders, Ad & Krempl, G.) 535–547 https://doi.org/10.1007/978-3-030-44584-3_42 (Springer International Publishing, 2020).
Feng, J., Emerson, S. & Simon, N. Approval policies for modifications to machine learning-based software as a medical device: a study of bio-creep. Biometrics (2020).
Feng, J., Gossmann, A., Sahiner, B. & Pirracchio, R. Bayesian logistic regression for online recalibration and revision of risk prediction models with performance guarantees. J. Am. Med. Inform. Assoc. (2022).
Feng, J. Learning to safely approve updates to machine learning algorithms. In: Proc. Conference on Health, Inference, and Learning, CHIL’21 164–173 (Association for Computing Machinery, 2021).
Kohane, I. S. et al. What every reader should know about studies using electronic health record data but may be afraid to ask. J. Med. Internet Res. 23, e22219 (2021).
Beesley, L. J. et al. The emerging landscape of health research based on biobanks linked to electronic health records: existing resources, statistical challenges, and potential opportunities. Stat. Med. 39, 773–800 (2020).
Cosgriff, C. V., Stone, D. J., Weissman, G., Pirracchio, R. & Celi, L. A. The clinical artificial intelligence department: a prerequisite for success. BMJ Health Care Inform. 27, e100183 (2020).
Sheller, M. J. et al. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Sci. Rep. 10, 12598 (2020).
Warnat-Herresthal, S. et al. Swarm Learning for decentralized and confidential clinical machine learning. Nature 594, 265–270 (2021).
U.S. Food and Drug Administration. Sentinel system: 5-year strategy 2019-2023. Tech. Rep. (2019).
Harvey, H. & Cabitza, F. Algorithms are the new drugs? Reflections for a culture of impact assessment and vigilance. In: IADIS International Conference ICT, Society and Human Beings 2018 (eds. Macedo, M. & Kommers, P.) (part of MCCSIS 2018) (2018).
Cabitza, F. & Zeitoun, J.-D. The proof of the pudding: in praise of a culture of real-world validation for medical artificial intelligence. Ann Transl Med 7, 161 (2019).
Johnson, A. E. et al. MIMIC-III, a freely accessible critical care database. Sci Data 3, 160035 (2016).
Zeileis, A., Leisch, F., Hornik, K. & Kleiber, C. strucchange: an r package for testing for structural change in linear regression models. J. Statistical Softw. Articles 7, 1–38 (2002).
Bifet, A., Holmes, G., Kirkby, R. & Pfahringer, B. MOA: massive online analysis. J. Mach. Learn. Res. 11, 1601–1604 (2010).
Montiel, J., Read, J., Bifet, A. & Abdessalem, T. Scikit-multiflow: a multi-output streaming framework. J. Mach. Learn. Res. 19, 1–5 (2018).