Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Perspective
  • Published:

A guide to deep learning in healthcare

Nature Medicine volume 25, pages 24–29 (2019)Cite this article

Subjects

Abstract

Here we present deep-learning techniques for healthcare, centering our discussion on deep learning in computer vision, natural language processing, reinforcement learning, and generalized methods. We describe how these computational techniques can impact a few key areas of medicine and explore how to build end-to-end systems. Our discussion of computer vision focuses largely on medical imaging, and we describe the application of natural language processing to domains such as electronic health record data. Similarly, reinforcement learning is discussed in the context of robotic-assisted surgery, and generalized deep-learning methods for genomics are reviewed.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Deep learning.
Fig. 2: Medical Imaging.
Fig. 3: Making predictions using EHRs.
Fig. 4: ML in genomics.

Similar content being viewed by others

References

  1. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

  2. Russakovsky, O.et al. Imagenet large scale visual recognition challenge. Int. J. Compute. Vis. 115, 211–252 (2015).

  3. Hirschberg, J. & Manning, C. D. Advances in natural language processing. Science 349, 261–266 2015).

    Article  CAS  Google Scholar 

  4. Geoffrey Hinton, et al. Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29, 82–97 (2012).

  5. Stanford Health. Harnessing the power of data in health. Stanford Medicine 2017 Health Trends Report (2017).

  6. Silver, D. et al. Mastering the game of go with deep neural networks and tree search. Nature 529, 484–489 (2016).

    Article  CAS  Google Scholar 

  7. Yohannes Kassahun, et al. Surgical robotics beyond enhanced dexterity instrumentation: a survey of machine learning techniques and their role in intelligent and autonomous surgical actions. Int. J. Comput. Assist. Radio. Surg. 11, 553–568 (2016).

    Article  Google Scholar 

  8. Litjens, G. et al. A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017).

    Article  Google Scholar 

  9. Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).

  10. Haenssle, H. A. et al. Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Ann. Oncol. 29, 1836–1842 (2018).

  11. Cheng, J.-Z. et al. Computer aided diagnosis with deep learning architecture: applications to breast lesions in us images and pulmonary nodules in CT scans. Sci. Rep. 6, 24454 (2016).

  12. Cicero, M. et al. Training and validating a deep convolutional neural network for computer-aided detection and classification of abnormalities on frontal chest radiographs. Invest. Radiol. 52, 281–287 (2017).

    Article  Google Scholar 

  13. Kooi, T. et al. Large scale deep learning for computer aided detection of mammographic lesions. Med. Image Anal. 35, 303–312 (2017).

    Article  Google Scholar 

  14. Barreira, C. M. et al. Abstract WP61: Automated large artery occlusion detection in st roke imaging-paladin study. Stroke 49, AWP61 (2018).

  15. Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316, 2402–2410 (2016).

    Article  Google Scholar 

  16. Poplin, R. et al. Webster. Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning. Nat. Biomed. Eng. 2, 158–164 (2018).

  17. Fauw, J. et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat. Med. 24, 1342 (2018).

    Article  Google Scholar 

  18. Cireşan, D. C., Giusti, A., Gambardella, L. M. & Schmidhuber, J. Mitosis detection in breast cancer histology images with deep neural networks. In International Conference on Medical Image Computing and Computer-assisted Intervention 411–418 (Springer, 2013).

  19. Liu, Y. et al. Detecting cancer metastases on gigapixel pathology images. Preprint at https://arxiv.org/abs/1703.02442 (2017).

  20. Charoentong, P. et al. Pan-cancer immunogenomic analyses reveal genotype–immunophenotype relationships and predictors of response to checkpoint blockade. Cell Rep. 18, 248–262 (2017).

  21. Beck, A. H. et al. Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Sci. Transl. Med. 3, 108ra113 (2011).

    Article  Google Scholar 

  22. Yosinski, J., Clune, J., Bengio, Y. and Lipson, L. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems 3320–3328 (2014).

  23. Jamaludin, A., Kadir, T. and Zisserman, A. Spinenet: automatically pinpointing classification evidence in spinal mris. In International Conference on Medical Image Computing and Computer-Assisted Intervention 166–175 (Springer, 2016).

  24. Kermany, D. S. et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172, 1122–1131 (2018).

    Article  CAS  Google Scholar 

  25. Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (2015).

  26. Goodfellow, I. et al. Generative adversarial nets. In Advances in Neural Information Processing Systems 2672–2680 (2014).

  27. Sutskever, I., Vinyals, O. & Le, Q. V. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 3104–3112 (2014).

  28. Wu, Y. et al. Google’s neural machine translation system: bridging the gap between human and machine translation. Preprint at https://arxiv.org/abs/1609.08144 (2016).

  29. Kannan, A. et al. Smart reply: automated response suggestion for email. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (ACM, 2016).

  30. Vinyals, O., Toshev, A., Bengio, S. & Erhan, D. Show and tell: a neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 3156–3164 (2015).

  31. The Office of the National Coordinator for Health Information Technology. Quick stats: health IT dashboard. https://dashboard.healthit.gov/quickstats/quickstats.php (2017).

  32. Shickel, B., Tighe, P. J., Bihorac, A. & Rashidi, P. Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE J. Biomed. Health Inform. 22, 1589–1604 (2017).

  33. Rajkomar, A. et al. Scalable and accurate deep learning with electronic health records. NPJ Digit. Med. 1, 18 (2018).

    Article  Google Scholar 

  34. Miotto, R. et al. Deep patient: an unsupervised representation to predict the future of patients from the electronic health records. Sci. Rep. 6, 26094 (2016).

  35. Liu, V., Kipnis, P., Gould, M. K. & Escobar, G. J. Length of stay predictions: improvements through the use of automated laboratory and comorbidity variables. Med. Care 48, 739–744 (2010).

    Article  Google Scholar 

  36. Choi, E. et al. Doctor AI: predicting clinical events via recurrent neural networks. In Machine Learning for Healthcare 301–318 (2016).

  37. Che, Z. et al. Recurrent neural networks for multivariate time series with missing values. Rep. 8, 1–12 (2018).

    Google Scholar 

  38. Suresh, H. et al. Clinical intervention prediction and understanding with deep neural networks. PMLR 68, 322–377 (2017).

  39. Johnson, A. E. W.et al. Mimic-iii, a freely accessible critical care database. Sci. Data 3, 160035 (2016).

    Article  CAS  Google Scholar 

  40. Mao, Q.et al. Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and icu. BMJ Open 8, e017833 (2018).

    Article  Google Scholar 

  41. Abbeel, P. & Ng, A. Y. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the Twenty-First International Conference on Machine Learning 1 (ACM, 2004).

  42. Ratliff, N. D., Silver, D. & Bagnell, J. A. Learning to search: functional gradient techniques for imitation learning. Autonomous Robots 27, 25–53 (2009).

    Article  Google Scholar 

  43. Schulman, J. et al. A case study of trajectory transfer through non-rigid registration for a simplified suturing scenario. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 4111–4117 (IEEE, 2013).

  44. Mayer, H.et al. A system for robotic heart surgery that learns to tie knots using recurrent neural networks. Adv. Robot 22, 1521–1537 (2008).

    Article  Google Scholar 

  45. Shvets, A., Rakhlin, A., Kalinin, A. A. and Iglovikov, V. Automatic instrument segmentation in robot-assisted surgery using deep learning. Preprint at https://arxiv.org/abs/1803.01207 (2018).

  46. Jin, A. et al. Tool detection and operative skill assessment in surgical videos using region-based convolutional neural networks. Preprint at https://arxiv.org/abs/1802.08774 (2018).

  47. Loh, P.-R. et al. Efficient bayesian mixed-model analysis increases association power in large cohorts. Nat. Genet. 47, 284 (2015).

    Article  CAS  Google Scholar 

  48. Abadi, M. et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. http://download.tensorflow.org/paper/whitepaper2015.pdf (2015).

  49. Tran, D. and Blei, D. M. Implicit causal models for genome-wide association studies. In International Conference on Learning Representations (2018).

  50. Lee, S.-I. et al. Learning a prior on regulatory potential from eqtl data. PLoS Genet. 5, e1000358 (2009).

    Article  Google Scholar 

  51. Kircher, M. et al. A general framework for estimating the relative pathogenicity of human genetic variants. Nat. Genet. 46, 310–315 (2014).

    Article  CAS  Google Scholar 

  52. Quang, D., Chen, Y. & Xie, X. Dann: a deep learning approach for annotating the pathogenicity of genetic variants. Bioinformatics 31, 761–3 (2015).

    Article  CAS  Google Scholar 

  53. Dudley, J. T. et al. Personalized medicine: from genotypes, molecular phenotypes and the quantified self, towards improved medicine. In Pacific Symposium on Biocomputing 342–346 (2014).

  54. Leung, M. K. K., Delong, A., Alipanahi, B. & Frey, B. J. Machine learning in genomic medicine: a review of computational problems and data sets. Proc. IEEE 104, 176–197 (2016).

  55. Xiong, H. Y. et al. The human splicing code reveals new insights into the genetic determinants of disease. Science 347, 1254806 (2015).

    Article  Google Scholar 

  56. Alipanahi, B. et al. Predicting the sequence specificities of dna-and rna-binding proteins by deep learning. Nature Biotechnol. 33, 831–838 (2015).

    Article  CAS  Google Scholar 

  57. Snyder, T. M., Khush, K. K., Valantine, H. A. & Quake, S. R. Universal noninvasive detection of solid organ transplant rejection. Proc. Natl Acad. Sci. USA 108, 6229–6234 (2011).

    Article  CAS  Google Scholar 

  58. Abril, M. K. et al. Diagnosis of capnocytophaga canimorsus sepsis by whole-genome next-generation sequencing. In Open Forum Infectious Diseases Vol. 3, ofw144 (Oxford University Press, 2016).

  59. Forshew, T. et al. Noninvasive identification and monitoring of cancer mutations by targeted deep sequencing of plasma dna. Sci. Transl. Med. 4, 136ra68–136ra68 (2012).

  60. Fan, H. C. et al. Non-invasive prenatal measurement of the fetal genome. Nature 487, 320–324 (2012).

    Article  CAS  Google Scholar 

  61. Ching, T. et al. Opportunities and obstacles for deep learning in biology and medicine. Preprint at https://doi.org/10.1101/142760 (2017).

  62. Angermueller, C., Lee, H. J., Reik, W. & Stegle, O. Deepcpg: accurate prediction of single-cell dna methylation states using deep learning. Genome Biol. 18, 67 (2017).

    Article  Google Scholar 

  63. Chen, Y. et al. Gene expression inference with deep learning. Bioinformatics 32, 1832–1839 (2016).

    Article  CAS  Google Scholar 

  64. Koh, P. W., Pierson, E. & Kundaje, A. Denoising genome-wide histone chip-seq with convolutional neural networks. Bioinformatics 33, i225–i233 (2017).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

The authors would like to thank D. Wang, E. Dorfman, and A. Rajkomar for the visual design of the figures in this paper and P. Nejad for insightful conversation and ideas.

Author information

Author notes

  1. These authors contributed equally: Andre Esteva, Alexandre Robicquet.

Authors and Affiliations

  1. Stanford University, Stanford, CA, USA

    Andre Esteva, Alexandre Robicquet, Bharath Ramsundar, Volodymyr Kuleshov & Sebastian Thrun

  2. Google Research, San Jose, CA, USA

    Mark DePristo, Katherine Chou, Claire Cui, Greg Corrado & Jeff Dean

Authors
  1. Andre Esteva
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alexandre Robicquet
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bharath Ramsundar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Volodymyr Kuleshov
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mark DePristo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Katherine Chou
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Claire Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Greg Corrado
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Sebastian Thrun
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Jeff Dean
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

B.R., V.K., M.D., and K.C. share second authorship. C.C., G.C., and S.T. share third authorship. J.D. is the principal investigator. A.E. and A.R. conceptualized the structure of the review and contributed to the computer vision and reinforcement learning sections. V.K., B.R., and M.D. contributed to the generalized deep learning section. K.C. and J.D. contributed to the natural language processing section. C.C., G.C., S.T., and J.D. oversaw the work. All authors contributed to multiple parts of the review, as well as the style and overall contents.

Corresponding author

Correspondence to Andre Esteva.

Ethics declarations

Competing interests

M.D., C.C., K.C., G.C. and J.D. are employees of Google Inc. This work was internally funded by Google Inc. G.C. is a board member at the Partnership on AI to Benefit People and Society. S.T. is an employee of Udacity, Inc. and the Kitty Hawk Corporation. He is on the faculty of Stanford University and Georgia Institute of Technology. B.R. is a partner of Computable LLC.​

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Esteva, A., Robicquet, A., Ramsundar, B. et al. A guide to deep learning in healthcare. Nat Med 25, 24–29 (2019). https://doi.org/10.1038/s41591-018-0316-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41591-018-0316-z

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing