ABSTRACT
Background A qualitative fit test using bitter-tasting aerosols is the commonest way to determine filtering face-piece (FFP) mask leakage. This taste test is subjective and biased by placebo. We propose a cheap, quantitative modification of the taste test by measuring the amount of fluorescein stained filter paper behind the mask using image analysis.
Methods A bitter-tasting fluorescein solution was aerosolised during mask fit tests, with filter paper placed on masks' inner surfaces. Participants reported whether they could taste bitterness to determine taste test ‘pass’ or ‘fail’ results. Filter paper photographs were digitally analysed to quantify total fluorescence (TF).
Results Fifty-six healthcare professionals were fit tested; 32 (57%) ‘passed’ the taste test. TF between the taste test ‘pass’ and ‘fail’ groups was significantly different (p<0.001). A cut-off (TF = 5.0 × 106 units) was determined at precision (78%) and recall (84%), resulting in 5/56 participants (9%) reclassified from ‘pass’ to ‘fail’ by the fluorescein test. Seven out of 56 (12%) reclassified from ‘fail’ to ‘pass’.
Conclusion Fluorescein is detectable and sensitive at identifying FFP mask leaks. These low-cost adaptations can enhance exiting fit testing to determine ‘pass’ and ‘fail’ groups, protecting those who ‘passed’ the taste test but have high fluorescein leak, and reassuring those who ‘failed’ the taste test despite having little fluorescein leak.
Background
Filtering face-piece (FFP) masks are a key item of personal protective equipment (PPE) to reduce transmission of infectious pathogens via aerosols and small droplets.1–3 In light of the SARS-CoV-2 global pandemic, extensive guidelines have been published recommending the use of FFP masks by healthcare workers.4–7 Many public health bodies mandate healthcare workers to wear an FFP mask (that they have been fit tested to use) when undertaking aerosol generating procedures (AGP) or working in areas where these are occurring.5,8,9
FFP3 masks (equivalent to N99 in the USA) offer the highest grade of filtration and filter at least 99% of airborne particles, with an inward leak of approximately 2%.10,11 For healthcare professionals to benefit fully from this degree of filtration, any leaks in the mask's seal around the wearer's face must be avoided. Leaks could risk transmission of virus particles carried in aerosols and droplets by bypassing the mask's intrinsic filtration system. Before a mask can be deemed to be safe for use in clinical practice by an individual, a fit test is undertaken.8,9 Two types of fit test are available.12
Qualitative fit test (taste test). A strong-tasting bitter or sweet solution is aerosolised using handheld pumps near the face of the user while they are wearing the mask. The user undertakes a protocolised set of movements. If they perceive the taste, the test is terminated and said mask is deemed unsuitable for their use (‘fail’). If they complete the protocol without perceiving an abnormal taste, the mask is deemed to be safe for their use (‘pass’).
Quantitative fit test. Specialist equipment is used by trained practitioners to determine the effectiveness of the faceseal using particle counting.
The taste test is cheap (under $250), does not require much training to conduct and is used in almost all centres in the UK.13 However, it is a subjective test which relies on participants to accurately identify and report an abnormal taste. It can be challenging for participants to be sure that their perceptions are not spurious. Taste can be significantly altered by confounders such as recent food and drink ingestion and environmental smells. Furthermore, natural variations in individual sensitivities to smell and taste can contribute to false positives and false negatives. Data from another study at our centre (under review) suggest that a significant placebo effect exists, with 23% of participants reporting ‘bitter’ or ‘sweet’ taste during fit tests when only aerosolised water was used.14
Quantitative fit tests offer a measurable alternative to the qualitative tests. Obviating the reliance on taste reporting, they use particle counting to determine the effectiveness of the faceseal.12 They must be conducted by trained testers, and are much more expensive than the taste test, costing in excess of $5,000 (excluding training).15 They are believed to be more accurate and have a low ‘pass’ rate compared to the taste test.16,17 The quantitative test makes a hole in the mask to enable particle counting on both sides. The item must be discarded after the test, which greatly reduces this test's applicability in resource-strained situations as seen during the COVID-19 pandemic.4,18
Medical fluorescein is used extensively in clinical practice, most commonly in ophthalmic angiography.19,20 It has been shown to be safe for human inhalation in diagnostic tests.21 Fluorescent sprays and aerosols have previously been used to demonstrate which areas of face masks are likely to contain maximum virus particles and determine the site of faceseal leaks.22,23 Measuring fluorescent staining of filter paper has been used to evaluate aerosol contamination distance during dental procedures.24 In this study, we propose a novel fit test method by applying simple and well-established image analysis techniques to detect the amount of fluorescent staining on filter paper behind FFP masks after a testing protocol using aerosolised fluorescein. The main objectives of this study are to determine whether fluorescein is detectable in this context and whether this method is sufficiently discriminating to be used for fit testing of healthcare workers.
Methods
This study was approved by Imperial College Healthcare NHS Trust clinical audit and quality improvement committee. Written informed consent was obtained from all participants.
Healthcare professionals treating COVID-19 patients were invited to participate in an FFP mask fit test.25 An industry-standard qualitative mask fit test protocol (taste test) commonly used in the UK NHS was modified by the addition of 2 mL of 20% fluorescein added to 3 mL of 0.2% denatonium benzoate (F-JAS, Flint, UK) in a bespoke fit testing solution.26 The resulting concentration of fluorescein was lower than the concentration that has previously been shown to be safe for inhalation by humans.21
Two FFP3 masks (3M 1863, 3M, Bracknell, UK; Medline NON24510V, Medline Industries, Warrington, UK) were tested. The choice of mask was determined by stock availability in the clinical areas in which the tests were being conducted.
The fit tests were conducted in a neutral environment, away from clinical areas, and free of distracting odours or sounds to enable participants to fully concentrate on their taste perception. All testers had received training in the use of the testing equipment and had extensive experience in conducting mask fit tests prior to data collection for this study. Data collection was supervised by the most experienced mask fit tester in our department.
Fluorescein fit test
Participant positioning: Each participant was seated in a clean clinical area. A testing hood was placed over their head, initially without them wearing the FFP mask.
Establish taste sensitivity: A low concentration (0.02%) of denatonium benzoate (to which fluorescein was not added) was sprayed (using handheld atomisers) through a small hole in the hood, near the participant's mouth and nose until they could taste it. This determined the number of testing solution sprays they receive during the test.
Neutralise mouth: The participant rinsed out their mouth with drinking water.
Prepare filter paper and mask: The participant opened the mask and turned the inside surface to face upwards. A single sheet of (70 mm diameter) general purpose grade laboratory filter paper (King Scientific, Huddersfield, UK) was placed inside the FFP mask using tongs by the tester, near the participant's mouth and nose (Fig 1a).
Mask donning: The participant put on the mask without touching or changing the orientation of the filter paper. The filter paper was not visible from the outside. The hood was replaced over the participant's head (Fig 1b).
Fluorescein spraying: The solution of fluorescein and 0.2% denatonium benzoate was sprayed through a hole in the hood, near the participant's face and nose. The number of sprays was determined by their sensitivity. A standard protocol was followed (normal breathing, deep breathing, head turning, head nodding, talking, bending and normal breathing), with each stage lasting 1 minute.26
Mask doffing: At the end of the test, the hood was removed and the participant removes the mask without touching the filter paper. The filter paper was collected by the tester using sterile tongs and placed in a labelled specimen bag (Fig 1c).
Study protocol and stages of the fluorescein fit test, n = 56. TF = total fluorescence.
If the participant completed the test without tasting the solution, they were deemed to ‘pass’ the taste test. If they detected the taste at any stage, they were deemed to ‘fail’ the taste test, but the full protocol was completed to standardise fluorescein exposure between participants. All participants were exposed to 7 minutes of aerosolised fluorescein and denatonium benzoate.
Filter paper photography and fluorescence analysis
Filter paper preparation: Two sprays with a pure water atomiser were used to re-humidify the filter papers (Fig 1c).
Photography environment: To create fluorescein excitation, filter papers were photographed under a LED bulb lamp. The emitted light was white with a 475 nm short pass filter (Edmund Optics, York, UK). Light was received using a long pass filter at approximately 500 nm (Hoya F3 yellow lens filter, Hoya Vision Care Company, Bangkok, Thailand; Fig 1d).
Camera settings: A Canon EOS 700D camera with 18–55 mm lens (Canon Inc, Tokyo, Japan) was used for photography. The camera settings were ISO 1,600; shutter speed 0.25 seconds; aperture F=8.0; and white balance ‘halogen’.
Fluorescence quantification: The photograph was split into the red/green/blue (RGB) stack and the green channel selected using the Fiji distribution of ImageJ.27 The technique described by Hammond was adapted to calculate total fluorescence (TF; Fig 1d).28
Statistical analyses
Median TF was calculated for the ‘pass’ and ‘fail’ groups based on participants' taste test result (Fig 1e). The difference between median TF between the two groups was evaluated using the Mann Whitney U test due to their distributions being non-parametric. The precision and recall of the fluorescein test in comparison to the taste test was calculated for a range of possible TF thresholds. Maximising both (with a preference for recall) was used to determine a TF cut-off for the fluorescein test.
Results
Fifty-six healthcare professionals underwent the fluorescein fit test at our hospital in London, UK. Of these, 32 (57%) ‘passed’ the taste test (ie they completed the protocol without tasting the solution) and 24 (43%) ‘failed’ the taste test (ie they experienced an abnormal perception of taste at some point during the taste protocol). There were comparable baseline characteristics between the taste ‘pass’ and ‘fail’ groups (Table 1).
Baseline characteristics of participants
In the ungrouped analysis, the median TF of the entire cohort was 3.7 × 106 fluorescence units (interquartile range (IQR) 8.0 × 106). When participants were stratified by the outcome of their taste test, there was a significant difference in TF between the ‘pass’ and ‘fail’ groups (median TF ‘pass’ group 2.6 × 106 fluorescence units (IQR 2.9) vs median TF in ‘fail’ group 9.0 × 106 fluorescence units (IQR 12.6); p<0.001; Fig 2)
Total fluorescence.
Analysis of different TF thresholds on the precision and recall of the fluorescein test in comparison to the taste test determined a TF of 5.0 × 106 fluorescence units as the optimal cut-off value to separate the ‘pass’ and ‘fail’ groups for this dataset (precision = 78%; recall = 84%; Fig 3). Applying this value as the result threshold of the fluorescein test resulted in 12 (21%) participants' taste test results being overturned (5/56 (9%) ‘pass’ changed to ‘fail’; 7/56 (12%) ‘fail’ changed to ‘pass’).
Precision and recall of the fluorescein test in comparison with the taste test for different total fluorescence threshold (cut-off) values.
Discussion
In this study, we introduced a novel, quantitative fluorescein mask fit test to determine whether FFP respirators adequately fit healthcare workers before their use in clinical practice. This method is an adaptation of the qualitative taste test using cheap and readily available materials such as medical fluorescein and scientific filter paper. This study achieved its two main objectives by demonstrating that fluorescein is detectable in this context and that the test is sufficiently discriminating to classify users into distinct groups.
Compared with the taste test, the fluorescein test required no additional time to conduct, and no side effects were experienced by any participants. The fluorescein test is conducted at approximately one-tenth of the cost of the alternative quantitative test, and without the need for specialist trained staff or destruction of the mask during testing – a key resource-management benefit in the context of a global viral pandemic.
We have demonstrated that this test can separate users into ‘pass’ and ‘fail’ categories similarly to the existing taste test (p<0.001), people who could readily taste the bitter aerosols had a high TF and vice versa. Analysing these results from another point of view, they also validate the existing taste test, which performs reasonably well at classifying users (19/24 participants who were classified as ‘pass’ by the qualitative test truly did have low TF (less than 5.0 × 106 fluorescence units)).
The main benefit of the fluorescein test is that it provides a numerical TF output. This mitigates the greatest source of bias in the taste test: reliance on the user's subjective perception of taste. It cannot be biased by placebo, which has been observed in almost a quarter of the taste tests in our experience.14 TF has many positive implications including the comparison of different masks' performance for an individual, quantifying the severity of the leak caused by ill-fitting masks and measuring the effect of finer mask adjustments to overcome leaks. These attributes demonstrate the test's valuable ability to flex with unexpected changes in the PPE supply chain and clinical staff redeployment, as seen in the primary surge of the COVID-19 pandemic.4,18,29
A key utility of a quantifiable metric of mask-leakage (such as TF) is the ability to use cut-offs as a threshold to the test result. This can enable the reclassification of users who may have been misclassified by the taste test. In this study, the precision and recall of the fluorescein test in comparison to the taste test was calculated for different TF threshold values. A TF cut-off of 5.0 × 106 fluorescence units is suggested for this dataset because it achieves good precision (78%) and recall (84%). Due to the clinical implications of a higher false positive rate, recall was prioritised over precision when considering the cut-off value. Although further validation with a large sample size is required to determine a robust cut-off, adopting this as a threshold for the fluorescein test would result in 9% of participants who had previously been classed as ‘pass’ by the taste test now being classed as ‘fail’. This is a group of people who were sure that they did not taste the bitter aerosols during the full 7 minutes of the protocol, yet the filter papers behind their mask were substantially stained with fluorescein. Given that the background environmental fluorescence is negligible, the most likely explanation is that the fluorescein entered through a leak in their masks' faceseals. Ordinarily, this group of healthcare workers would have commenced or continued clinical practice wearing that mask, including participation in risky aerosol generating procedures without the benefit of this information, falsely reassured by their results in the taste test.
Healthcare professionals are disproportionately affected by COVID-19.30 The implications of reclassifying this group may have considerable impact on their safety and wellbeing by enabling their exclusion from high-risk areas, and the identification of more suitable FFP respirators.31 Unlike the taste test, the fluorescein test provides this crucial information that has as yet been unavailable to healthcare workers and their managers.
The TF distribution of participants who ‘failed’ the taste test (ie they reported tasting the bitter aerosols) has a very wide range. On one end of the spectrum, 38% recorded very high TF levels in excess of 15 × 106 fluorescence units. This group of people not only tasted the solution, but also truly had substantial leaks in the faceseal resulting in filter paper staining. This demonstrates that both the taste test and fluorescein test were effective in identifying this group. On the other end of the spectrum, 29% had low levels of TF (<5.0 × 106 fluorescence units) despite being certain that they tasted the solution. This can only be explained by either a placebo effect or a heightened sensitivity to tasting denatonium benzoate. Data from our experiments indicate that a placebo effect occurred in 23% of taste tests. This group (12% of our entire cohort) could be presented with the results of their fluorescein fit test and reassured that they actually had low levels of faceseal leak. This could enable a substantial proportion of the healthcare workforce to return to patient-facing roles if they were shielded or reduce the need for wearing uncomfortable and expensive respirators. In both cases, there would be a positive impact on public health, workforce productivity and human resource management.
Limitations
This study addressed quantification of fluid ingress past a mask using two methods that are easily implementable with minimal unit cost in routine practice. It does not use specialised laboratory techniques for viruses. It only assesses the feasibility of automated quantification of ingress that does not rely on the human perception of taste, which is known to be vulnerable. For example, in the other study from our centre (under review), 23% of subjects reported a definite taste sensation when given control fluid.14
We tested two commonly used FFP3 masks by two manufacturers. This was based on stock levels and availability in clinical areas when the tests were being conducted. The types of FFP3 face mask available in the UK has changed considerably over the preceding 9 months in response to the coronavirus pandemic. At the time this study was conducted, the 3M 1863 was one of the most common FFP3 masks in use in the UK. Participants of the study were undergoing fit testing for their regular clinical roles and so we were duty bound to test them with the mask they were most likely to encounter in clinical practice. More recently, the UK government has secured a contract with Honeywell UK (Bracknell, UK) to manufacture to produce some 65 million FFP2 and FFP3 masks. The structure of the Honeywell masks is not dissimilar to that of the masks used in this study. Furthermore, this study presents a comparison between fit testing methods rather than the masks themselves. The testing method is unaffected by the make, model and type of FFP3 mask used. There is no reason why our results are not broadly applicable to masks by other manufacturers.
In this study, the photography and image analysis of the filter papers were done spatially and temporally removed from the fit testing encounter. This was to identify the ideal photography environment through iterative experimentation. Now that the photographic parameters have been identified in this study, the image analysis could also be rapidly conducted on a portable computer or smart device. We aim to develop this environment as a point-of-care product and invite further validation.
Conclusion
This study shows that aerosolised fluorescein is detectable on filter paper placed behind FFP masks when added to the existing qualitative fit test protocol (the taste test). Fluorescent staining is sensitive in this context and can be quantified to produce a metric of leakage: TF. The fluorescein test has a similar discriminatory ability to the existing taste test, and is much cheaper than the quantitative test.
We advocate using fluorescein in mask fit testing to provide increased safety for ‘false passes’ in the taste test (9% in our cohort). After further validation, ‘false fails’ in the taste test may be reassured that this is most likely due to a placebo rather than substantial fluorescein leakage (12% in our cohort). This could enable them to safely return to clinical practice.
Summary
What is known?
Filtering face-piece (FFP) masks must have an adequate faceseal to protect healthcare workers. A qualitative taste test is currently the most common way to check the FFP seal. This is subjective and biased by placebo, falsely reassuring some, and wrongly excluding others.
What is the question?
Can medical fluorescein and digital image analysis be used to adapt the existing fit test with an objective measure of faceseal leak? Can the amount of fluorescein leak be used to divide mask wearers into ‘pass’ and ‘fail’ groups?
What was found?
The amount of fluorescein on filter paper behind FFP masks after fit testing was sensitive at detecting leaks. The fluorescein test discriminated between ‘pass’ and ‘fail’ groups. A cut-off of 5.0 × 106 fluorescence units balanced high precision (78%) and recall (84%), resulting in 9% reclassified from ‘pass’ to ‘fail’ and 12% vice versa.
What is the implication for practice now?
We advocate the use of this low-cost and reproducible adaptation to the existing mask fit test healthcare settings. This would protect healthcare workers who had fluorescein leak despite ‘passing’ the taste test. People who ‘failed’ the taste test but actually have very little fluorescein leak could be reassured to return to clinical practice safely.
Funding
Funding for this study was provided by a grant from the Imperial College COVID Research Fund (grant WHCP G26284). We are grateful for the infrastructural support from the National Institute of Health Research (NIHR) Biomedical Research Centre based at Imperial College Healthcare NHS Trust and Imperial College London.
- © Royal College of Physicians 2021. All rights reserved.
