An evaluation framework for Health Information Systems: human, organization and technology-fit factors (HOT-fit)
Introduction
The benefits derived from a Health Information Systems (HIS) require rigorous evaluation. It is claimed that organizational and social issues are the main components of such a system [1]. The more technology, human and organization fit with each other, the greater the potential of HIS. Most existing evaluation studies of HIS focus on technical issues or clinical processes, which do not explain why HIS works well or poorly with a specific user in a specific setting [2], [3], [4], [5], [6], [7].
The central purpose of this paper is to show how we derived a framework for HIS evaluation that incorporates comprehensive dimensions and measures of HIS and provides a technological, human and organizational fit. A case study demonstrates the application of the proposed framework for describing the critical adoption factors of a particular HIS with a particular user in a particular setting. This proposed framework, human, organization and technology-fit (HOT-fit) is constructed so that it is capable of being useful in conducting a thorough evaluation study. It is also possible that it will assist researchers and practitioners to unfold and understand the perceived complexity of HIS evaluation. The new framework builds on previous work on the review of HIS evaluation [8], [9]. It also makes use of two models of IS evaluation, namely the IS Success Model [10], [11] and the IT-Organization Fit Model [12].
This paper is organized as follows. Section 2 discusses the theoretical background of the proposed framework. The aforementioned models are presented to explore their applicability in improving on those used in Health Informatics. This then forms the basis for our first proposed evaluation framework for HIS presented in Section 3. Section 4 discusses an existing Fundus Imaging System in a primary care organization that is used as a research case study. Section 5 presents the research methodology for applying our framework to this case study. The case study findings are presented in Section 6. Finally, discussion and conclusions are given in the last section.
Section snippets
Theoretical background
The proposed human, organization and technology-fit evaluation framework was developed after a critical investigation of the existing findings of HIS and IS evaluation studies (discussed in the preceding paper in this issue of the journal, [13] and from feedback gained from two conference presentations of earlier stages in the research [8], [9]). A review of success determinants of Inpatient Clinical IS indicates that the categories for success in the IS Success Model can be used to assess HIS
Proposed evaluation framework
Building on previous studies on the evaluation approach [8], [9], the proposed evaluation framework was developed after a critical appraisal of the existing findings of HIS and IS evaluation studies. It makes use of the IS Success Model in categorizing its evaluation factors, dimensions and measures. The IS Success Model is adopted based on its comprehensive, specific evaluation categories, extensive validation and its applicability to HIS evaluation. In addition, the IT-Organization Fit Model
The case of Fundus Imaging System (FIS)
Our research design consists of a case study strategy. The case study serves dual purposes: (1) to evaluate the adoption factors of HIS in the context of the phenomena under study; (2) to validate the proposed HOT-fit evaluation framework. The case study is also undertaken to obtain a comprehensive view and understanding of the development process of a HIS described in this section. The case study facilitated the conceptualisation of HIS adoption tending towards success and failure, in
Methodology
The approach used in this study was that of a subjectivist, case study strategy employing qualitative methods. A formative evaluation was undertaken of the adoption of FIS to identify system problems as they emerged and to improve the system as it was developed [40]. A subjectivist approach was employed in order to gain an extensive understanding of the healthcare context surrounding the FIS through detailed, insightful explanation of the study [34]. Further, qualitative methods were employed
Discussion of the findings from the Fundus Imaging System case
Based on observations that took place during the immersion and pilot study, a number of emerging themes from the framework, namely human, organizational and technological factors and net benefits have been identified. These issues are presented in the next sections, which are organized in terms of the three themes.
Further discussion and conclusions
The case of FIS has demonstrated the importance of having the right user attitude and knowledge in order to be able to use the system effectively and efficiently. Arguably, choosing the right people to fill the user role is more important than possessing the required skills, as skills can be acquired later. This can be seen from the use of FIS, which was primarily driven by the willingness of the GP to learn and use it after being persuaded by the senior partner and despite her limited
Acknowledgements
We gratefully acknowledge the funding received from both the Public Service Department of Malaysia and from the Universiti Kebangsaan Malaysia (National University of Malaysia) that helped sponsor this study.
References (43)
Evaluating informatics applications—clinical decision support systems literature review
Int. J. Med. Inf.
(2001)Evaluating informatics applications-some alternative approaches: theory, social interactionism, and call for methodological pluralism
Int. J. Med. Inf.
(2001)- et al.
Research needs and priorities in health informatics
Int. J. Med. Inf.
(2000) - et al.
The Heidelberg Conference: setting an agenda for the IMIA working Group on Health Information Systems
Int. J. Med. Inf.
(2003) - et al.
Organizational ISSUES = change
Int. J. Med. Inf.
(2003) - et al.
Organizational issues in health informatics: a model approach
Int. J. Med. Inf.
(1998) Patient care information systems and health care work: a sociotechnical approach
Int. J. Med. Inf.
(1999)- et al.
User resistance and strategies for promoting acceptance across system types
Inf. Manage.
(2000) - et al.
Meeting patients’ needs with patient information systems: potential benefits of qualitative research methods
Int. J. Med. Inf.
(2001) ‘CHEATS’: a generic information communication technology (ICT) evaluation framework
Comput. Biol. Med.
(2002)
A clinical information system research agenda for Kaiser Permanente
Permanente J.
People, organizational and social issues: evaluation as an exemplar
IMIA 2002 Yearbook
Future directions in evaluation research: people, organizational, and social issues
Methods Inf. Med.
Guide to Health Informatics
Information systems success: the quest for the dependent variable
Inf. Syst. Res.
Measuring e-commerce success: applying the DeLone & McLean Information Systems Success Model
Int. J. Electron. Commerce
The Corporation of the 1990s
Investigating evaluation frameworks for health information systems
Int. J. Med. Inf.
Determinants of success of inpatient clinical information systems: a literature review
J. Am. Med. Inform. Assoc.
Cited by (457)
Perceptions of infection control professionals toward electronic surveillance software supporting inpatient infections: A mixed methods study
2024, International Journal of Medical InformaticsPrioritization of barriers to social media usage by non-governmental organizations (NGOs) in disaster preparedness phase: An integrated methodology
2023, International Journal of Disaster Risk ReductionAdoption of smart farming technologies in field operations: A systematic review and future research agenda
2023, Technology in SocietyA novel framework for investigating organizational adoption of AI-integrated CRM systems in the healthcare sector; using a hybrid fuzzy decision-making approach
2023, Telematics and Informatics Reports