An evaluation framework for Health Information Systems: human, organization and technology-fit factors (HOT-fit)

https://doi.org/10.1016/j.ijmedinf.2007.08.011Get rights and content

Abstract

Background and purpose

The realization of Health Information Systems (HIS) requires rigorous evaluation that addresses technology, human and organization issues. Our review indicates that current evaluation methods evaluate different aspects of HIS and they can be improved upon. A new evaluation framework, human, organization and technology-fit (HOT-fit) was developed after having conducted a critical appraisal of the findings of existing HIS evaluation studies. HOT-fit builds on previous models of IS evaluation—in particular, the IS Success Model and the IT-Organization Fit Model. This paper introduces the new framework for HIS evaluation that incorporates comprehensive dimensions and measures of HIS and provides a technological, human and organizational fit.

Methods

Literature review on HIS and IS evaluation studies and pilot testing of developed framework. The framework was used to evaluate a Fundus Imaging System (FIS) of a primary care organization in the UK. The case study was conducted through observation, interview and document analysis.

Results

The main findings show that having the right user attitude and skills base together with good leadership, IT-friendly environment and good communication can have positive influence on the system adoption.

Conclusions

Comprehensive, specific evaluation factors, dimensions and measures in the new framework (HOT-fit) are applicable in HIS evaluation. The use of such a framework is argued to be useful not only for comprehensive evaluation of the particular FIS system under investigation, but potentially also for any Health Information System in general.

Introduction

The benefits derived from a Health Information Systems (HIS) require rigorous evaluation. It is claimed that organizational and social issues are the main components of such a system [1]. The more technology, human and organization fit with each other, the greater the potential of HIS. Most existing evaluation studies of HIS focus on technical issues or clinical processes, which do not explain why HIS works well or poorly with a specific user in a specific setting [2], [3], [4], [5], [6], [7].

The central purpose of this paper is to show how we derived a framework for HIS evaluation that incorporates comprehensive dimensions and measures of HIS and provides a technological, human and organizational fit. A case study demonstrates the application of the proposed framework for describing the critical adoption factors of a particular HIS with a particular user in a particular setting. This proposed framework, human, organization and technology-fit (HOT-fit) is constructed so that it is capable of being useful in conducting a thorough evaluation study. It is also possible that it will assist researchers and practitioners to unfold and understand the perceived complexity of HIS evaluation. The new framework builds on previous work on the review of HIS evaluation [8], [9]. It also makes use of two models of IS evaluation, namely the IS Success Model [10], [11] and the IT-Organization Fit Model [12].

This paper is organized as follows. Section 2 discusses the theoretical background of the proposed framework. The aforementioned models are presented to explore their applicability in improving on those used in Health Informatics. This then forms the basis for our first proposed evaluation framework for HIS presented in Section 3. Section 4 discusses an existing Fundus Imaging System in a primary care organization that is used as a research case study. Section 5 presents the research methodology for applying our framework to this case study. The case study findings are presented in Section 6. Finally, discussion and conclusions are given in the last section.

Section snippets

Theoretical background

The proposed human, organization and technology-fit evaluation framework was developed after a critical investigation of the existing findings of HIS and IS evaluation studies (discussed in the preceding paper in this issue of the journal, [13] and from feedback gained from two conference presentations of earlier stages in the research [8], [9]). A review of success determinants of Inpatient Clinical IS indicates that the categories for success in the IS Success Model can be used to assess HIS

Proposed evaluation framework

Building on previous studies on the evaluation approach [8], [9], the proposed evaluation framework was developed after a critical appraisal of the existing findings of HIS and IS evaluation studies. It makes use of the IS Success Model in categorizing its evaluation factors, dimensions and measures. The IS Success Model is adopted based on its comprehensive, specific evaluation categories, extensive validation and its applicability to HIS evaluation. In addition, the IT-Organization Fit Model

The case of Fundus Imaging System (FIS)

Our research design consists of a case study strategy. The case study serves dual purposes: (1) to evaluate the adoption factors of HIS in the context of the phenomena under study; (2) to validate the proposed HOT-fit evaluation framework. The case study is also undertaken to obtain a comprehensive view and understanding of the development process of a HIS described in this section. The case study facilitated the conceptualisation of HIS adoption tending towards success and failure, in

Methodology

The approach used in this study was that of a subjectivist, case study strategy employing qualitative methods. A formative evaluation was undertaken of the adoption of FIS to identify system problems as they emerged and to improve the system as it was developed [40]. A subjectivist approach was employed in order to gain an extensive understanding of the healthcare context surrounding the FIS through detailed, insightful explanation of the study [34]. Further, qualitative methods were employed

Discussion of the findings from the Fundus Imaging System case

Based on observations that took place during the immersion and pilot study, a number of emerging themes from the framework, namely human, organizational and technological factors and net benefits have been identified. These issues are presented in the next sections, which are organized in terms of the three themes.

Further discussion and conclusions

The case of FIS has demonstrated the importance of having the right user attitude and knowledge in order to be able to use the system effectively and efficiently. Arguably, choosing the right people to fill the user role is more important than possessing the required skills, as skills can be acquired later. This can be seen from the use of FIS, which was primarily driven by the willingness of the GP to learn and use it after being persuaded by the senior partner and despite her limited

Acknowledgements

We gratefully acknowledge the funding received from both the Public Service Department of Malaysia and from the Universiti Kebangsaan Malaysia (National University of Malaysia) that helped sponsor this study.

References (43)

  • D.F. Sittig et al.

    A clinical information system research agenda for Kaiser Permanente

    Permanente J.

    (2002)
  • B. Kaplan et al.

    People, organizational and social issues: evaluation as an exemplar

    IMIA 2002 Yearbook

    (2002)
  • B. Kaplan et al.

    Future directions in evaluation research: people, organizational, and social issues

    Methods Inf. Med.

    (2004)
  • E. Coiera

    Guide to Health Informatics

    (2003)
  • M.M. Yusof et al.
  • M.M. Yusof et al.
  • W.H. DeLone et al.

    Information systems success: the quest for the dependent variable

    Inf. Syst. Res.

    (1992)
  • W.H. DeLone et al.

    Measuring e-commerce success: applying the DeLone & McLean Information Systems Success Model

    Int. J. Electron. Commerce

    (2004)
  • M.S. Scott Morton

    The Corporation of the 1990s

    (1991)
  • M.M. Yusof et al.

    Investigating evaluation frameworks for health information systems

    Int. J. Med. Inf.

    (2007)
  • M.J. Van der Meijden et al.

    Determinants of success of inpatient clinical information systems: a literature review

    J. Am. Med. Inform. Assoc.

    (2003)
  • Cited by (457)

    View all citing articles on Scopus
    View full text