Artificial intelligence (AI) has immense promise for revolutionising medical practice. Generative AI is a form of AI in which algorithms are trained on datasets that can be used to generate new content, such as text, images or video based on user prompts [1]. Examples include DALL-E (OpenAI), Midjourney, and Stable Diffusion for generating images from text prompts; and Bard (Google), ChatGPT (OpenAI), and LLaMA (Meta AI) for generating text from text prompts.
Large multi-modal models (LMMs), also known as general-purpose foundation models, can accept one or more types of data input and generate diverse outputs that are not limited to the type of data used to build the algorithm. According to the World Health Organization (WHO), LMMs have been adopted faster than any consumer application in history and are predicted to have wide application in healthcare, scientific research, public health and drug development (Table 1) [1]. Figure 1 outlines WHO consensus ethical principles for use of generative AI for health [1].
Figure 1: WHO consensus ethical principles for use of generative AI for health. Adapted from: World Health Organization. Ethics and governance of artificial intelligence for health: Guidance on large multi-modal models (2024). World Health Organization. https://www.who.int/publications/i/item/9789240084759
It is predicted that generative AI could lead to job losses and require health workers to retrain and adjust to the use of AI-based models [1]. According to a World Economic Forum (WEF) report, up to 85-million jobs may be lost by 2025 due to the shifting division of labour between people and machines [2]. College-educated individuals and women are more exposed to automation but also better poised to reap AI benefits, and older workers are potentially less able to adapt to the new technology, according to researchers from the International Monetary Fund [3]. The three occupations at the lowest risk of automation are medical practitioners, higher education teaching professionals, and senior educational professionals of educational establishments, according to an analysis by the Office for National Statistics [4].
Is AI ready to replace clinicians?
The role of medical AI in ophthalmology was explored in a debate during the 2024 Controversies in Ophthalmology (COPHy) Congress in Athens. Professor Giuseppe Querques (University Vita-Salute, IRCCS San Raffaele Scientific Institute, Milan, Italy) emphasised that AI in ophthalmology is already available for real-world applications, although the advent and further refinements of AI technology will require specific governance regulations for data protection and ethical governance [5,6]. Prof Querques highlighted current notable clinical applications of medical AI in the early detection and management of retinal diseases.
Several AI-based systems are available with high sensitivity and specificity for screening and grading diabetic retinopathy (DR) as well as for detection of prognostic indicators, providing clinicians with reliable and prompt information about progression [7,8]. A study using a deep learning (DL) algorithm to assist grading for DR found that a machine learning (ML) model for predicting DR severity improved physician grading performance (Table 2) [8].
Imaging-based AI systems can reliably and accurately grade age-related macular degeneration (AMD) and provide information about potential progression, such as early identification of treatment naive non-exudative macular neovascularisations at risk for exudation within the first two years of follow-up [9]. Deep learning technologies can effectively diagnose AMD, predict short-term risk of exudation, and need for intravitreal injections within the next two years [10].
The iPredict AI Eye Screening System (iHealthScreen), developed and tested using images from the National Eye Institute-funded Age-related Eye Disease Study (AREDS), can provide accurate identification of referrable AMD and predict risk of development of advanced AMD within one and two years [11]. Borrelli, et al. showed that a DL-based model applied to neovascular AMD can segment critical optical coherence tomography (OCT) features with performance similar to manual grading [12]. Artificial intelligence applied to pathologic myopia can help differentiate between subretinal haemorrhage without choroidal neovascularisation (CNV) (i.e. simple bleeding) and subretinal haemorrhage due to CNV onset, improving human ability to perform differential diagnosis on unprocessed baseline OCT B-scan images [13]. Further, creation of a DL classifier showed good sensitivity and specificity performances in predicting one-year visual outcomes in patients undergoing vitrectomy for epiretinal membrane [14].
Querques noted that colour fundus photo-based AI models can effectively support primary care in detection of high prevalence diseases. More advanced applications of AI in screening and managing retinal diseases may be applied in tertiary care centres to support clinical management and improve understanding of the pathophysiology of retinal conditions, he added. “AI is ready to replace lack of medical resources,” remarked Querques.
Professor Paolo Lanzetta (University of Udine, Udine, Italy) discussed current and evolving medical AI applications in ophthalmology, sounding a note of caution, emphasising external validation, clinical safety, and the need to ensure AI systems are reliably beneficial for patients [15].
The fields of drug discovery and development, personalised medicine, automated surgical assistance, and medical research including clinical trials are being transformed by generative AI applications. Artificial intelligence can help analyse vast amounts of data from genomics, proteomics, and other sources to identify new drug targets and facilitate drug design processes. Machine learning and AI could be employed to help improve and accelerate clinical trial design / simulations, optimise dose selection, identify the right patient population or enhance safety evaluations / monitoring. Also, AI could help tailor medical treatment to individual patient profiles, considering genetic makeup, lifestyle, and even social determinants of health to optimise treatment effectiveness, as well as support predictive analytics for personalised treatment.
However, there are sound reasons why AI and neural networks are not ready to replace physicians, noted Lanzetta. Machine learning systems are fallible and will inevitably make mistakes, as ML behaviour is dependent on the data on which it has been trained. The accuracy of the ML system can be compromised when patients are not adequately represented in the data used to train the ML system, e.g. different patient demographics, temporal changes, differences in the clinical stage of disease, inconsistencies defining a gold standard diagnosis, and differences in the algorithms used to scan a patient.
Explainable AI algorithms should be mandatory in healthcare and predictions generated by AI systems should be interpretable, observed Lanzetta. Empathy, compassion, and communication skills are closely related to physician characteristics shown to positively impact patient-centred outcomes, which AI cannot replicate. The subtleties of patient interaction and emotional support are beyond the capabilities of current AI technologies.
AI holds great promise in ophthalmology and medical practice generally, but there are performance limitations as well as ethical and legal concerns, continued Lanzetta. Artificial intelligence models can still underperform, make mistakes, and struggle with cases or situations different from their training data. The opaque ‘black box’ nature of some AI algorithms can make it difficult to trust and interpret their decisions fully. Further, issues of accountability, liability, data privacy, and potential biases in AI systems raise significant ethical and legal challenges that need to be addressed before AI can replace human physicians.
There will always be a need for human oversight and collaboration; rather than replacing physicians, AI should be viewed as a powerful tool to augment and complement human expertise, said Lanzetta.
Clinical decision support software
Aslam and Hoyle argued that AI platforms will increasingly be adopted to direct clinical care, but as an adjunct rather than replacement for clinicians [16]. Human clinicians will still have to apply their skill and nous to make sense of AI-enabled outputs and their relevance to individual clinical scenarios. For predictive AI models, algorithms are limited by the training data; clinical contexts different to that of the algorithm training and testing process may limit reliability and accuracy of related predictions. Clinicians should therefore demand systems that feature explainable AI.
Explainable AI teaches AI functionalities to end users [17], providing the ability of AI systems to provide understandable explanations for specific model predictions or decisions. A transparent AI system enables accountability by allowing stakeholders to validate and audit its decision-making processes, detect biases or unfairness, and ensure that the system is operating in alignment with ethical standards and legal requirements [18]. Interpretable AI models allow humans to estimate what a ‘black box’ model will predict given an input and understand when the model has made a mistake [18].
Large language models demonstrate potential as clinical decision support tools
A recently published study by Thirunavukarasu and colleagues tested the clinical potential of state-of-the-art LLMs in ophthalmology, using challenging questions used to assess the aptitude of eye doctors before they can be deemed fully qualified [19]. Based on comparisons with expert ophthalmologists and trainee doctors, the authors found that new LLMs, and most notably GPT-4, are approaching expert-level ophthalmological clinical knowledge and reasoning and significantly exceed the ability of non-specialist clinicians (i.e. unspecialised junior doctors). Of note, ophthalmological knowledge and reasoning capability of LLMs was judged general rather than limited to certain subspecialties or tasks.
The authors suggested that new-generation LLMs could be useful for providing eye-related advice and assistance to non-specialists or patients where access to eyecare professionals is limited, with the caveat that close monitoring is essential to avoid mistakes caused by inaccuracy or fact fabrication.
Study senior investigator Darren Shu Jeng Ting (Consultant Ophthalmologist, Birmingham and Midland Eye Centre, and Birmingham Health Partners Fellow, University of Birmingham, UK) commented in a telephone interview: “To evaluate whether LLMs could potentially be deployed in medical practice, we decided to conduct this multi-centre, cross-sectional study to undertake a direct benchmark comparison with working doctors at various levels, including junior doctors, ophthalmology trainees and consultant ophthalmologists. Study results are remarkable: LLMs outperformed junior doctors with respect to ophthalmological clinical knowledge and reasoning. Our study suggests that these models could potentially be used as valuable assistance or support tools. Based on the median scores achieved, expert ophthalmologists performed better than LLMs.”
Deep learning AI systems are already being implemented in some countries, for example as part of the Singapore Integrated Diabetic Retinopathy Programme. Wider applications in other countries are likely to follow.
“LLMs offer potential for diverse applications in medicine, including ophthalmology”, added Ting. “It will likely take time before LLMs can be implemented and deployed as support tools in front-line ophthalmic clinical practice. Their use will be subject to appropriate ethical approval and regulatory oversight being established as well as performance validation in clinical trial settings. Further model refinement may also be required for application in ophthalmology. The use of LLMs for generation of patient discharge summaries or for medical education and training for example may be more attainable in the near future.”
Biswas and colleagues recommended a conservative approach when using AI-based LLMs in ophthalmic practice, emphasising the need for human judgement for clinical decision-making and monitoring the accuracy of information [20]. The authors argued that AI-based LLMs such as ChatGPT are constrained by their nonspecific and outdated training, no access to current knowledge, generation of plausible-sounding ‘fake’ responses or hallucinations, inability to process images, lack of critical literature analysis, and ethical and copyright issues.
Regulating medical AI: an evolving ecosystem
The US Food and Drug Administration (FDA), Health Canada, and the UK Medicines and Healthcare products Regulatory Agency (MHRA) have jointly identified 10 guiding principles that can inform the development of Good Machine Learning Practice and promote safe, effective, and high-quality medical devices that use AI and ML (Table 3).
Predetermined change control plans (PCCPs) provide a new method for managing rapid product changes often seen with software and AI products. Five guiding principles for the use of PCCPs have also been jointly identified, to help ensure alignment between jurisdictions and products utilising them. Certain changes to ML-enabled medical devices, such as changes to a model or algorithm, may be substantive or significant and, for this reason, they can require regulatory oversight, such as additional premarket review. As the US FDA noted: “International harmonisation and stakeholder consensus on the core concepts of PCCPs will help support the advancement of responsible innovations in the digital health space.”
In March 2024 the European Parliament adopted the Artificial Intelligence Act (AI Act), providing a comprehensive horizontal (i.e. cross sectoral) legal framework for AI and EU-wide rules on data quality, transparency, human oversight and accountability. The regulation establishes obligations for AI based on its potential risks and level of impact.
A recent virtual web event on regulating medical AI, hosted by NEJM AI and sponsored by Elevance Health, Lyric, Microsoft and Viz.ai, addressed emerging multifaceted challenges and opportunities using medial AI. Speakers discussed how regulation, capturing international approaches and alignment, can optimise the application of AI in clinical practice.
Panel participant Professor Alastair Denniston (University of Birmingham) commented: “Regardless of whether we are in a period of regulatory evolution or revolution, or both, the underlying principles remain the same: we need to remember that however exciting the technology, it is first about people and only second about products. Our approach to regulation needs to remain robust, patient centred and evidence driven. This is a community endeavour as we work together to ensure that patients can benefit from products that are safe, effective, sustainable and equitable.”
Mind the algorithm
Robust positive results have been seen with the application of AI technologies in ophthalmology [21,22]. Ethical concerns with the use of medical AI models include transparency, responsibility, and scalability of use and infrastructure [23]. With careful oversight (including model monitoring of outputs, inputs, performance, efficacy and effectiveness) and sector-specific regulation, explainable and accessible AI-based systems may be used effectively in ophthalmology to supplement clinician decision-making, boost capacity, and potentially improve patient care and outcomes.
References
1. Ethics and governance of artificial intelligence for health: Guidance on large multi-modal models (2024). World Health Organization.
https://www.who.int/publications/
i/item/9789240084759
2. Boosting healthcare capacity with AI (2024). The World Economic Forum.
https://www.weforum.org/agenda/2024/
01/ai-in-healthcare-could-bridge-a-significant
-capacity-gap/#:~:text=Artificial%20intelligence%20(AI)
%20can%20help,spending%20due%
20to%20diminishing%20capacity
3. Cazzaniga M, Jaumotte F, Li L, et al. Gen-AI: Artificial Intelligence and the Future of Work. Staff Discussion Note SDN2024/001 (2024). International Monetary Fund.
4. Which occupations are at highest risk of being automated? (2019). Office for National Statistics.
https://www.ons.gov.uk/employmentandlabourmarket/
peopleinwork/employmentandemployeetypes/articles/which
occuppationsareathighestriskofbeingautomated/2019-03-25
5. Danese C, Kale AU, Aslam T, et al. The impact of artificial intelligence on retinal disease management: Vision Academy retinal expert consensus. Curr Opin Ophthalmol 2023;34(5):396–402.
6. Chou YB, Kale AU, Lanzetta P, et al. Current status and practical considerations of artificial intelligence use in screening and diagnosing retinal diseases: Vision Academy retinal expert consensus. Curr Opin Ophthalmol 2023;34(5):403–13.
7. Crincoli E, Sacconi R, Querques L, Querques G. OCT angiography 2023 update: focus on diabetic retinopathy. Acta Diabetol 2024;61(5):533–41.
8. Sayres R, Taly A, Rahimy E, et al. Using a deep learning algorithm and integrated gradients explanation to assist grading for diabetic retinopathy. Ophthalmology 2019;126(4):552–64.
9. Crincoli E, Catania F, Sacconi R, et al. Deep learning for automatic prediction of early activation of treatment naïve non-exudative MNVs in AMD. Retina 2024. Epub ahead of print.
10. Crincoli E, Sacconi R, Querques L, Querques G. Artificial intelligence in age-related macular degeneration: state of the art and recent updates. BMC Ophthalmol 2024;24(1):121.
11. Bhuiyan A, Wong TY, Ting DSW, et al. Artificial intelligence to stratify severity of age-related macular degeneration (AMD) and predict risk of progression to late AMD. Transl Vis Sci Technol 2020;9(2):25.
12. Borrelli E, Oakley JD, Iaccarino G, et al. Deep-learning based automated quantification of critical optical coherence tomography features in neovascular age-related macular degeneration. Eye (Lond) 2024;38(3):537–44.
13. Crincoli E, Servillo A, Catania F, et al. Artificial intelligence’s role in differentiating the origin for subretinal bleeding in pathologic myopia. Retina 2023;43(11):1881–9.
14. Crincoli E, Savastano MC, Savastano A, et al. New artificial intelligence analysis for prediction of long-term visual improvement after epiretinal membrane surgery. Retina 2023;43(2):173–81.
15. Veritti D, Rubinato L, Sarao V, et al. Behind the mask: a critical perspective on the ethical, moral, and legal implications of AI in ophthalmology. Graefes Arch Clin Exp Ophthalmol 2024;262(3):975–82.
16. Aslam TM, Hoyle DC. Translating the machine: skills that human clinicians must develop in the era of artificial intelligence. Ophthalmol Ther 2022;11(1):69–80.
17. Okada Y, Ning Y, Ong MEH. Explainable artificial intelligence in emergency medicine: an overview. Clin Exp Emerg Med 2023;10(4):354–62.
18. EDPS TechDispatch on Explainable Artificial Intelligence (2023). European Data Protection Supervisor.
https://www.edps.europa.eu/data-protection/
our-work/publications/techdispatch/2023-11-16
-techdispatch-22023-explainable
-artificial-intelligence_en
19. Thirunavukarasu AJ, Mahmood S, Malem A, et al. Large language models approach expert-level clinical knowledge and reasoning in ophthalmology: A head-to-head cross-sectional study. PLOS Digit Health 2024;3(4):e0000341.
20. Biswas S, Davies LN, Sheppard AL, et al. Utility of artificial intelligence-based large language models in ophthalmic care. Ophthalmic Physiol Opt 2024;44(3):641–71.
21. Ferro Desideri L, Rutigliani C, Corazza P, et al. The upcoming role of Artificial Intelligence (AI) for retinal and glaucomatous diseases. J Optom 2022;15(Supp1):S50–7.
22. Zhou Y, Chia MA, Wagner SK, et al. A foundation model for generalizable disease detection from retinal images. Nature 2023;622(7981):156–63.
23. Evans NG, Wenner DM, Cohen IG, et al. Emerging ethical considerations for the use of artificial intelligence in ophthalmology. Ophthalmol Sci 2022;2(2):100141.
[All links last accessed May 2024]
Declaration of competing interests: The author has provided consultancy services to Bayer AG, DORC International B.V., Johnson & Johnson Vision Care, Inc., Roche Products Ltd, and Thea Pharmaceuticals Ltd.
COMMENTS ARE WELCOME
