A quality assessment of respiratory auscultation material on YouTube
Abstract
YouTube contains a large volume of medical educational material. This study assessed the quality of respiratory auscultation videos contained in YouTube. Videos were searched for using the terms ‘breath sounds’, ‘respiratory sounds’, ‘respiratory auscultation’ and/or ‘lung sounds’. In total, 6,022 videos were located, 36 of which were considered suitable for scoring for video accuracy, comprehensiveness and quality. The average score was 3.32/6 (55.3% ± 1.30). Video score correlated with time-adjusted YouTube metadata: hits per day (0.496, p=0.002) and likes per day (0.534, p=0.001). Video score also correlated with the first search page on which the video was located in the ‘breath sounds’ and ‘lung sounds’ searches (–0.571, p=0.001; –0.445, p=0.014, respectively). The quality of videos was variable. Correlation between video score and some metadata values suggests that there is value for their use in judging video quality. However, the large number of videos found and inability to filter these results quickly makes locating educational content difficult.
Introduction
The importance of the Internet as a resource in medical education continues to increase,1–4 and its use as a way of gathering information on personal health issues has been well examined previously.1,4–6 YouTube acts as an online video file repository, with the ease of access and free content being key factors in making the site popular. The use of YouTube to aid in the education of health professionals has been previously suggested.7,8
Respiratory auscultation is key to the examination of any individual with potential pulmonary or cardiac pathology.9,10 The importance of discriminating between sounds generated by the respiratory system, combined with an appreciation for the mechanism of noise generation, is clear.11–13 Many medical educators utilise simulated sounds to complement the physical examination to provide the necessary educational experience.14 Auditory information lends itself well to pre-recorded educational material utilising both audio and visual components. The popularity of the YouTube platform, combined with the ease with which auditory information can be converted into pre-recorded educational material, suggests that YouTube could be used extensively by both medical students and trainee doctors.
YouTube, and the Internet in general, provide an excellent tool for educational use.7,8 This role for the Internet was hypothesised early in its inception, a decade before the creation of YouTube.13,15 Previous studies have assessed the quality of medical procedure videos held on YouTube.2 However, there is a paucity of research relating to the quality of educational material in the field of respiratory medicine. The lack of regulatory mechanisms to vet or secure information contained within YouTube is potentially dangerous to those seeking online information.5 Previous work in the field of cardiology demonstrated multiple flawed educational videos.16 As such, it was hypothesised that YouTube currently hosts respiratory medicine videos of inadequate educational value. Therefore, in this study, we assessed the respiratory auscultation educational material contained in YouTube.
Method
A list of eight respiratory sounds was compiled by examining current medical textbooks.9,10 From these, four were identified as ‘core’ information for medical practice and medical education, with six further sounds being categorised as ‘extra’ (Table 1).
The YouTube website was queried between 18 and 20 May 2013 using four search terms: ‘breath sounds’, ‘respiratory sounds’, ‘respiratory auscultation’ and ‘lung sounds’. Quotation marks were used in place of a logic operator to specify that both terms must be present.
Two authors flagged videos appearing to contain respiratory auscultation for further analysis. For any videos where the title was ambiguous, the videos were examined to ascertain whether they met the inclusion criteria. Videos were excluded if they were non-English language, non-educational in nature, or where the topic of the video was not related to respiratory sounds or auscultation. An initial list of videos and the search page on which they were found was compiled. The uploaders of the videos found using the initial search terms were logged. All additional videos created by these uploaders, but not found by our search, were reviewed for additional relevant material and added to the search results. All videos were fully viewed to confirm that they contained auditory representations of respiratory sounds and were intended as educational tools. The initial search, subsequent screening, analysis and scoring of videos were performed independently by two of the authors (KG and AW) and any discrepancies were referred to authors NS and CFC to be resolved by consensus.
YouTube publishes additional information on all videos. These inbuilt metrics were catalogued for each selected video on 3 August 2013. The metrics catalogued from the website were: hits (the number of times a video had been viewed), and likes and dislikes (a crude scoring system that viewers can assign to videos). The date on which the video was uploaded to YouTube was collected to calculate the number of hits, likes and dislikes per day. The methodology for collating videos considered to be repeats or part of a larger series has been described previously.16
The videos were assessed for audiovisual quality, teaching quality, comprehensiveness and file metrics. Although previous studies have developed scoring systems for use in assessing online material,3,5 none were appropriate for the current study. Therefore, in an attempt to compare the videos accurately, we used our own purpose-built scoring system.16 The elements of this scoring system are outlined in Table 2. Given the importance of a comprehensive set of respiratory sounds, this factor was weighted more highly, with half marks being available for only a ‘core’ set of respiratory sounds. A set of anchor statements was outlined to secure the maximum level of objectivity between the two scorers. Video accuracy was assessed by comparing the YouTube video with the auditory files provided on the Littmann® Lung Sounds Online Tutorial.17 Given that the latter does not contain files for ‘pleural (friction) rub’ or ‘whispered pectoriloquy’, these files were obtained from the sound bank of Easy Auscultation (www.easyauscultation.com).18 All files were verified as correct and appropriate for teaching by the senior author, respiratory consultant GW, in conjunction with the Computerized Respiratory Sound Analysis (CORSA) guidelines for reproducing respiratory sounds.19
Statistical analysis
The median and interquartile range (IQR) were used to express the continuous variables. Given the non-parametric nature of the data collected, correlation between variables was assessed using Spearman's rho. Interobserver analysis was assessed by calculating Cohen's kappa score. SPSS Statistics 17.0 (IBM, Chicago, IL, USA) was used as the software to calculate statistical values. p values <0.05 were considered significant.
Results
A total of 6,022 videos was found on initially searching YouTube; 3,580 using ‘breath sounds’, 19 using ‘respiratory sounds’, three using ‘respiratory auscultation’ and 2,420 using ‘lung sounds’. After removing playlists, the number of videos in each category was reduced to 419 (7.5%), 19 (100%), three (100%) and 429 (17.7%), respectively. Following visual examination, 157, 12, three and 189 videos, respectively, met inclusion criteria. Concordance between observers in the initial search was 0.81 (95% confidence interval [CI] 0.77–0.86).
Combining search results and removal of duplicates resulted in 90 unique files. Those considered to be part of a series or copies of the same file were combined to create 36 unique videos. Of those videos selected for final analysis, 30 (83%) were found when searching ‘breath sounds’, 30 (83%) on searching ‘lung sounds’, four (11%) on searching ‘respiratory sounds’ and one (3%) on searching ‘respiratory auscultation’. A flow chart detailing the search process is shown in Fig 1.
The median score for all videos assessed was 3.5/6 (58.3%, 2.5–4.5) (Fig 2). No videos scored full marks; the highest-scoring video had a score of 5.5 (91.7%). The criteria most likely to be met were video accuracy (88.9%) and video quality (88.9%). The criteria least likely to be met were video comprehensiveness and video title (22.2% and 25.0%, respectively). A breakdown of item compliance is detailed in Table 3. The total concordance between authors scoring subjective items was 0.70 (95% CI 0.62–0.78). Of the score items that were subjective in nature, all had a kappa score greater than 0.60 (strong agreement), with the exception of video accuracy with a score of 0.53 (moderate agreement).
Correlation between the video score and number of hits per day attributed to that video was statistically significant (0.496, p=0.002), as was the correlation between video score and number of video likes per day (0.534, p=0.001). Significant correlations were also found between the video score and the first search page that the video was found on in the ‘breath sounds’ and ‘lung sounds’ searches (−0.571, p=0.001; −0.445, p=0.014, respectively). No correlation was seen between dislikes per day and video score (0.196, p=0.251). Table 4 details five highly recommended videos found in this study.
Discussion
YouTube is the third most-visited website on the Internet,20 giving it enormous potential for the dissemination of high-quality health-related information. Not solely confined to patient education, YouTube also has substantial potential as a medium to educate doctors and medical students. However, previous work carried out by this group on the quality of medical education material on YouTube revealed a significant body of poor-quality and, in some cases, factually incorrect resources.16 Other groups looking at both medical education material21,22 and patient information23 have also expressed concerns regarding misleading information on the website. The benefits of high-quality information on YouTube are several-fold because it can be used for private study or integrated into classroom teaching, while being engaging and appealing to multiple learning styles simultaneously.
The scoring system used in this study has a structured framework that takes account of several video aspects that are important in producing high-quality educational material. This system shows a high level of interobserver correlation while covering a range of potentially subjective elements. Despite the initial scorer agreement for the field of ‘video accuracy’ being lower (kappa 0.53) than the other subjective analyses, this prompted a full rescoring of this field by senior authors NS and CFC, which resulted in complete agreement.
Over 6,000 videos containing information relating to respiratory auscultation were obtained in this study, but only 36 represented appropriate teaching material. This alone represents a key problem with the search strategy of YouTube. Any normal website user is unlikely to be prepared to browse all 6,000 videos to find the few videos with high-quality content.
The quality of the videos located was variable. Not a single video or series of videos obtained the full score. This indicates a serious lack of quality content relating to respiratory auscultation on YouTube. Previous work on the quality of cardiac auscultation videos16 also discovered widely divergent final video scores; given similar findings in other fields of medical education,21,22 it is likely that this is generalisable to educational videos on many topics.
A standard moderating service for educational videos would potentially improve video quality, although this would be logistically challenging and costly to implement, given the volume of new content added to the system daily.20 Domain-based ranking, a system that returns videos coming from trusted sources (hospitals and health organisations) higher up in the search results, might be one method of improving the identification of trustworthy health-related videos on YouTube,23 although in our study only one video was found to have come from a recognised medical educational institution. Additionally, this group has previously suggested that organisations with a role in the education of medical students and other healthcare professionals should make an active effort to identify good-quality online teaching resources, perhaps through use of some form of ‘kitemark’ that need not be restricted to the YouTube website.
The ability to filter videos for educational content used in our previous work on cardiac sounds has since been disabled by YouTube. The current education section of YouTube has limited content and is mostly aimed at school-age users. We found that, without the inbuilt educational filter, searches returned a greater number of videos. Although in our previous work we concluded that its effectiveness was limited, the alternative of not having this filter means that an even greater number of non-educational videos are found, further limiting the use of YouTube as a source for high-quality educational material. Therefore, the administrators of YouTube should consider reinstating the filter with stricter criteria regarding the labelling of a video as educational.
Contrary to our previous findings,16 the significant correlation between video score and page location of the video when searched is a useful observation. It confirms that the YouTube search algorithm is, at least crudely, calibrated for searching for higher-quality educational content; a change in the search algorithm used by the YouTube website, to which we do not have access, could well underlie this observation. We also have demonstrated a significant correlation between video score and number of video hits per day, suggesting that users are inherently more likely to select a video from earlier pages, are more likely to return multiple times to high-scoring videos, or are more likely to pass these videos on to colleagues. Similarly, we found a significant correlation between video score and likes per day, whereas previously we found no correlation with likes;16 this would suggest that our score-defined definition of a good respiratory auscultation teaching video is grossly similar to that of the users accessing the educational content on YouTube. In the present study, metrics were adjusted for the time that the video had been available (ie on a per day basis), because we believe this to be an improvement on assessing the true popularity of a video. Significant correlations between YouTube metadata and video score in the present study might be a product of the search algorithm of YouTube. We do not believe it to be a product of our altered analysis because the same correlations are seen with the absolute data. Our top-scoring videos (score range 4.5–5.5) could be found on YouTube search pages that ranged from the 1st to the 13th; this suggests that most viewers will be unlikely to find all of the highest-quality teaching material.
The search strategy aimed to reach a high level of sensitivity. Unlike databases used for peer-reviewed publications, the search engine utilised by YouTube is simplistic and allows only a limited degree of search control. As such, this could explain the large number of video files found, especially when searching for ‘breath sounds’. Checking the other video files of any uploader, located using the initial search, would have identified any additional videos relating to the original searches. Although it is not certain that all videos were located using this search method, most users are likely to enter popular or common terms to locate videos.
The small number of search terms and lack of controlled keywords (ie MeSH terms) in our search strategy not only is a limitation, but also means that comprehensiveness is difficult to assess. We used common terms to obtain as large a -cross-section of relevant videos as was feasible and think this probably reflects real-life use of the website by most users. Not collecting YouTube video metrics on the date of originally accessing the videos is a limitation because extra videos might have been uploaded in the interim. However, we requeried YouTube for videos uploaded during this time period and found no new original material. However, there were eight reposts of material that we had already found; when added to our original dataset, these additional files did not alter our statistics. Restricting the video content language to English limits the content reviewed; however, given that most videos stored on YouTube are English-language videos, this effect might be small.20 By only searching YouTube and not including other video-storing websites, some videos might have been missed. The number of videos stored on other websites is difficult to assess, but the popularity of YouTube makes it the likely first port of call for those seeking educational material.20
Conclusions
We have demonstrated that our scoring system can easily be adapted to assess videos on other topics on YouTube. With the rapid advances in file-sharing utilisation for education, continued assessment of content is required. Others are encouraged to assess and improve upon the system described in this and our previous paper.16
- © 2014 Royal College of Physicians
References
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- Mason RJ,
- Broaddus VC,
- King TE
- ↵
- Forgacs P.
- ↵
- Murphy RL
- ↵
- ↵
- ↵
- ↵
- ↵
- 3MTM Littmann® Stethoscopes. Heart lung sounds tutorial
- ↵
- Easy Auscultation. Lung Sounds
- ↵
- Sovijärvi ARA,
- Dalmasso F,
- Vanderschoot J,
- et al.
- ↵
- ↵
- ↵
- ↵
Article Tools
Citation Manager Formats
Jump to section
Related Articles
- No related articles found.
Cited By...
- No citing articles found.