Scaling AI safely: a hypothetical vignette
| Isabel is the clinical director of her local primary care network (PCN), and is also a GP partner at a large surgery in Buckinghamshire. She had been using SkinScanner for the previous 2 years, an AI-enabled tool that can help analyse skin lesions, determine the change in appearance over time and suggest appropriate next steps. Isabel's practice has used SkinScanner to good effect, helping to reduce the number of unnecessary referrals and inappropriate investigations, and Isabel is keen to support the rollout of the software elsewhere in the PCN. However, she recently reviewed a patient, Eric, whom she'd last seen 6 months previously with a mole on his left forearm, which she'd advised him to monitor. Eric hadn't noticed any change in appearance over those 6 months, and Isabel also measured the mole with her dermatoscope, and was reassured to see that it hadn't increased in size since Eric's last appointment. However, she was surprised to find that SkinScanner judged the lesion to have increased in size, and recommended urgent specialist review. She arranged a priority virtual consultation with a consultant dermatologist, Dr Sharma, who agreed with Isabel and Eric's assessment that the lesion was not worrisome and did not warrant further investigation. |
| Isabel contacted the SkinScanner team to investigate the discrepancy in advice, who promptly paused the software until they could find out more. They conducted a thorough retrospective analysis of all skin lesions assessed using SkinScanner, and found a problem in <1% cases with the measuring ability of their tool. These cases were reviewed by a panel of expert dermatologists and, thankfully, it was found that patient care hadn't been compromised as a result (though the patients were notified of the issue). The developer team at SkinScanner implemented a patch to fix the error and upgraded their software, and continued to enforce vigilant post-market surveillance (PMS). |
| Reassured that no harm had come to patients as a result, and once again confident in the newly-upgraded software, Isabel was eager to try to scale SkinScanner in a manner that is effective and safe. |
GP = general practitioner.