Evolution of the clinical simulation approach to assess digital health technologies

Clinical simulation as an innovative methodology to generate evidence for DHTs

Clinical simulation refers to situating actual end-users in realistic clinical scenarios to perform tasks close to real-life environments. This requires high-fidelity synthetic patient cases, as well as good knowledge of local clinical workflows to ensure the right end-user is assessing a DHT at the right time in the right context. It differs from other types of simulation-based research, such as computational simulation, which uses computer models to conduct internal validation and bench testing, and system simulation, which uses engineering models to study the effect of an intervention in a system (Table 1). Research is broadly categorised into two types: clinical simulation as either a training methodology or as an investigative methodology, the latter being the focus of this article. To better visualise how studies using clinical simulation as an evaluative methodology are being conducted, these can be categorised into two types: using synthetic clinical cases presented to clinicians to assess a digital solution, and using scripted scenarios performed by actors to imitate real-life patient–provider interactions.

Here, we describe our programme of research to evaluate and develop clinical simulation methodology at the Institute of Global Health Innovation (IGHI) at Imperial College London. Researchers used clinical simulation with synthetic patient cases to evaluate several DHTs for use in oncology (Table 2). The solutions were evaluated based on parameters of feasibility, user experience and effectiveness compared with existing practice.

In the first study, the team conducted a series of simulated lung cancer multidisciplinary team (MDT) meetings to test the NAVIFY Tumor Board solution, a digital solution for the preparation and conduct of cancer tumour boards. In total, 56 healthcare professionals based in the UK were recruited to participate in 10 simulation sessions, where groups of five or six clinicians discussed mock patient cases at simulated tumour boards. During the session, participants discussed up to 10 synthetic patient cases across two simulated MDTs, first using standard tools commonly used to conduct MDTs and then using the NAVIFY Tumor Board solution, with a member of the study team navigating the software in view of the participants. Participants then completed a post-simulation survey asking their thoughts about the software for the conduct of MDTs, perceived benefits, areas for improvement and whether they would recommend it to colleagues. This was followed by a focus group to provide feedback on their perceptions of the solution compared with the standard methods. User-perceived benefits of the digital solution included the capturing of clinical information, radiology and pathology images in one platform; the interface looked clean and easy to navigate; the timeline feature could be useful as a record of previous discussions; the platform could be accessed remotely; and it could be relatively easy to add patient cases to the tumour board meeting.

For the second study, the research team assessed NAVIFY Clinical Trial Match, a digital solution that uses data on a patient's condition, genomic alterations and the post code of the healthcare institution to find suitable clinical trials. In this study, 25 clinicians and research staff were recruited to match 10 synthetic patient cases to clinical trials using both the clinical decision support (CDS) tool and publicly available online trial databases. Participants were asked to assume the role of a healthcare professional preparing to see a patient at an outpatient clinic and to find a suitable clinical trial that they could partake in. To match each synthetic patient case to a potential clinical trial, participants used either the NAVIFY Clinical Trial Match solution or existing online trial databases. They completed a post-simulation survey focused on usability, applications of the tool compared with the current practice and their experience in the simulation. Other data collected included the time taken to match the patient to each trial, the quality of the matches and cognitive burden on participants. The results of this study suggested that clinical simulation is better for assessing a single user's clinical decision because more data, both qualitative and quantitative, could be collected. This study demonstrated several strengths of the clinical simulation approach, including comparison with a legacy system, speed, agility and improved sensitivity, where it was possible to use the same method at different time intervals to evaluate the intervention as it is iteratively improved. Moreover, it provided a higher resolution in assessment because time, treatment decision and mental fatigue could be measured in the decision making of participants, which would be challenging in a real-world clinical setting.

Whereas these two studies were delivered in-person in the UK, for the third study, the research team set up remote sessions for US-based clinicians to evaluate a CDS tool called NAVIFY Guidelines, which supports adherence to the National Comprehensive Cancer Network's (NCCN) clinical guidelines, which are in widespread use in the USA. Ten oncologists were recruited to participate in simulation sessions, during which they were asked to review clinical information for 10 synthetic breast cancer cases. The clinicians were asked to choose the most appropriate guideline-recommended decision for each case, using the CDS tool for five cases and a PDF copy of the NCCN guidelines for five cases. A post-simulation survey was also conducted that included the System Usability Scale (SUS) to assess tool usability, along with a short interview. The score of the CDS tool on the SUS questionnaire was in the 70th percentile, with two items scoring in the top decile: ease of use and the ability to learn how to use the application quickly.

For all studies, high-fidelity synthetic patient cases were developed by the study team and clinical experts. Participants were not chosen based on previous platform knowledge, but instead were ‘platform naïve’ and recruited to have the appropriate clinical knowledge that is required for the specific scenario. Initial findings suggest that clinical simulation is better suited for the individual rather than the group setting. While detailed methodologies and findings of the studies are published elsewhere,^4–7 consideration of the combined learnings helps to shed light on how clinical simulation can be harnessed as a methodology to generate evidence for DHTs.

Benefits and challenges of clinical simulation

Our experience demonstrated the use of clinical simulation as an expedient and relatively inexpensive approach to generate robust evidence for DHTs that can be done remotely across multiple sites. Compared with conventional study designs, clinical simulation research is highly flexible, adaptable and scalable (eg different clinical scenarios or different types of users). Results can be generated pragmatically to inform continuous improvements and iterations, particularly on usability and feasibility. The approach can also be used across the developmental life cycle of DHTs in different evaluation activities, from product design to assessment in real clinical environments. This was our experience when evaluating the CDS tool for matching patients to cancer clinical trials, which we assessed at different time points in its development. Rather than as an alternative to clinical studies, we believe that clinical simulation is better understood as a complementary approach. It could guide the design of further clinical studies and offer indication as to when the DHT is ready for a real-world clinical study, as well as collect metrics less amenable to measurement in conventional studies.

The use of synthetic patient cases in clinical simulation confers three strengths as an innovative methodology in evaluation research. First, because no real patient data are involved, risks to personal data and privacy are minimised. Second, equity can be embedded across the product development cycle, from user design to evidence generation. Clinical vignettes can be developed to represent high-risk or vulnerable patients who are excluded from clinical studies because of potential harm and ethical considerations. Clinical simulation provides a controlled environment to test DHTs in specific patient groups without exposing real patients to risk. Simulated synthetic patient cases can also be adapted to low-resource settings to evaluate effectiveness across diverse contexts. In low–middle income countries where clinical simulation is widely used for healthcare training, it would also likely be easily transferable to research purposes. Third, the use of synthetic data reduces research timelines by removing the need to go through lengthy information governance approvals and recruitment processes, as well as by lowering the bar for ethics approval because of the removal of patient harm possibilities.

However, clinical simulation research has limitations. The high fidelity underpinning clinical simulation requires specialised clinical expertise and technical input. This relates to not only the quality of synthetic patient cases, but also the environment, equipment and task. Other risks of clinical simulation research include the possibility of introducing racial or ethnic bias through unrepresentative synthetic patient cases and the chance of ‘gaming the system’, in which developers might tailor the synthetic patient cases to best fit with the specifications of their products. These risks could be mitigated by enhancing the transparency of the research process. Although there are still challenges in utilising clinical simulation as a formal methodological tool, these do not appear to be insurmountable. Building on existing guidance on clinical simulation research, further work will be needed to develop consensus and guidance on best practices.⁸

The future of clinical simulation

The primary opportunity for clinical simulation is its application in registrational studies for regulated digital health solutions. A crucial purpose of developing robust innovative methods to evaluate DHTs is to ensure the safety, efficacy and cost-effectiveness of health interventions at a population level. With more clarity and consensus around the standards of conducting clinical simulation, it is our hope that this approach could offer a robust method to assess DHTs. However, further research is needed to ascertain the criteria required for the use of clinical simulation in the regulation of DHTs.

Funding statement

This paper received funding support from Roche Diagnostics.