UCL MotionInput: Touchless computing interactions in clinical training, radiology and operating theatres

The problem

In recent years the physical landscape of hospital spaces has changed, with monitors, workstations and mobile computers on wheels now forming critical IT design requirements for areas of care. This has led to new human computer interactions as clinicians are increasingly interacting with computer user interfaces in hospital wards, during clinical consultations, during team meetings, to assist with diagnostic processes and to facilitate virtual collaboration. However, there are also unintended consequences to this; for example, infection control is paramount in hospital environments, and this has been brought into even sharper focus by the COVID-19 pandemic. In a systematic review concerned with contamination of common ward-based computing equipment, 96.7% of keyboards were described as contaminated.¹ A recommendation was to find better methods of decontamination.

Reducing contamination with touchless computing

Another approach is to reduce direct contact with equipment. The COVID-19 pandemic period has renewed interest in touchless computing in healthcare because it makes it possible for individuals to control a computer without any form of physical contact. This includes recognition-based technologies that decipher human-speech, gesture, gaze, motion and other behavioural cues that allow for interactions that are free from being touched; for example, recognition-based interfaces (known as somatosensory interactions) enable interactions through using hand gestures without the need to touch input devices. They do this by tracking human movement.

Previous attempts at touchless interactions using depth cameras

With Project Natal in 2010, evolving into the MS Kinect platform, prior endeavours have achieved this using depth cameras that use specialised software for motion and skeletal tracking. These technologies have already begun to be experimented with in hospital settings, and most of the studies to date have been conducted in operating theatres and within interventional radiology.^2–4 In this example, they applied a three-dimensional sensor developed for gaming to healthcare interactions for skeletal body tracking and detection of the user's hand movements to enable interaction with patient data.3 While this impressed, and led to interesting research experiments, it did not achieve mainstream use in healthcare. Limited software development and access to hardware restricted the adoption into consumer patient and health technology spaces, and the original Kinect platform is no longer viable. The technology has now moved into the cloud processing arena, which presents a data protection issue for sensitive locations.

Further problems were that the technology of its day was expensive, and required installation of new hardware and additional software requirements. Therefore, over time, while this technology has advanced, this has not always been matched by improvements to usability, utility or user experience. As a result, while there is much research looking at how newer technologies can improve healthcare, in practice, however, little is in widespread use. This is salient for complex innovations in healthcare that require systemic or organisational adoption of technologies, and so we see rapid advancements in technology currently outpacing levels of uptake.

Using the technology that we already have

By taking a different approach, the MotionInput software developed in the department of computer science at University College London (UCL) is designed to navigate a computer interface touch free in a hospital or clinical working environment using a standard webcam. It works by taking the video output from a webcam and processing it through a library of on-device and open-source machine learning models for gesture-based input interactions. No network communications occur for the processing and all the response time latency has been finely tuned for viable navigation of user interfaces. This enables the use of a computer interface through hand gestures, body poses, head movement and eye tracking to facilitate touchless computing interactions (Fig 1).⁵

• Download figure
• Open in new tab
• Download powerpoint

Fig 1.

Early involvement of clinicians to trial out University College London MotionInput over clinically simulated theatre bed spaces.

The solution is only possible because of innovations in AI and machine learning from large multi-national corporations. The latest in state-of-the-art recognition software using video captured from standard webcams, the kind that you would see built into most laptops, would not have been possible several years ago.

A spectrum of motion recognition inputs

UCL MotionInput v2.0 is software designed to work across a broad range of contexts, healthcare workflows and users. The aim being to augment mouse, keyboard and touchpad input with touchless interactions. Of current primary interest is a spectrum of motion recognition inputs can be used for a broader set of applications. Therefore, MotionInput is developed as a series of modules comprising hand gesturing, head motion, eye tracking and body extremities, and repetitive triggers for exercise. This holds the potential to transform how users can navigate computer screens, play computer games, create art and exercise at home, as well as keep a sterile environment in hospitals.

Involving clinical simulation and clinical teams early in new human computer interactions in operating theatres and radiology

While there is much interest in the technical requirements for the next generation of touchless computing, it is important to consider the interactions that underpin these new behaviours.

Clinical workflows are made up by a series of complicated team-based and individual tasks. Therefore, in advance of the development of the MotionInput software, we took a human-centred interaction design approach embedded as part of the software engineering development process. This was achieved through observing-by-proxy simulated clinical spaces and clinical teams working together. This engaged clinicians early on, in an agile and nimble way, revealing optimal conditions for touchless computing by space, lighting and obstructions (Fig 2).⁶ Continued engagement with clinicians was important for acclimatisation and testing within a high-fidelity simulation suite as part of early build cycles. This enabled accuracy, learnability and error correction to be explored as part of the software development process interpreted through an interdisciplinary lens of healthcare, computer science and human computer interaction.

• Download figure
• Open in new tab
• Download powerpoint

Fig 2.

Early involvement of clinicians to trial out University College London MotionInput over clinically simulated theatre bed spaces.

Interesting interactions meets standards in e-Health for future works of UCL MotionInput

Next objectives include investigating the role of further in-air touchless selection with voice tagging; for example, gesturing selections on radiography images and adding voice notes. A further aim is to explore the potential for 5G connected gesture controls, specifically the application's capability and limitations for in-air two-handed gesture analogue joysticks to be tested in the context of simulated endoscopy training (Fig 3).

• Download figure
• Open in new tab
• Download powerpoint

Fig 3.

In-air analogue joysticks used to fly a plane in MS Flight Simulator. This allows a use case to lead to testing the limitations for touchless endoscopic control in a remote clinical 5G-enabled scenario.

The in-air keyboards solution (typing in the air) currently in development allows for custom keyboard key combinations per application for clinical data entry. Specific shortcut keys in applications can now be brought up on demand. Similarly, improvements to the current full body mode will allow exercise and rehabilitation programmes clinics to use MotionInput for measuring angles of joints of users’ performances, periods of motion and time taken for movements. The team is keen to build collaborative partnerships for development into useful areas of clinical research, and the solution has been designed with interoperability with Fast Healthcare Interoperability Resources (FHIR) in mind to better integrate future methods for patient assessments and we look forward to speaking to interested parties.⁷

UCL's department of computer science has released the current software as freeware. Clinicians, and indeed the public, are invited to trial and give feedback on touchless computing. The software currently runs on Windows 10/11 desktop and laptop machines with webcams and turns hand, full body, eyes and head movements into usable mouse and keyboard input replacements for Windows software and games (for more information, please see supplementary material S1).

The freeware software demonstrator can be found here: http://software.cs.ucl.ac.uk/MotionInput.html

Supplementary material

Additional supplementary material may be found in the online version of this article at www.rcpjournals.org/fhj:

S1 – Additional information and background for MotionInput.

Acknowledgements

We thank Emma Broughton, Linda Chigaru, Alasdair Ross and Martin Kostolny from Great Ormond Street Hospital for Children for securing the partnership with clinical simulation and the facilities of the suite to test MotionInput with clinicians as part of software engineering development cycles.

Funding

Sheena Visram acknowledges doctoral funding from Great Ormond Street Hospital for Children charities.