|
 |
| ORIGINAL ARTICLE |
|
| Year : 2018 | Volume
: 9
| Issue : 4 | Page : 153-157 |
|
|
Usability testing of bed information management system: A think-aloud method
Mohammad Reza Mazaheri Habibi1, Reza Khajouei2, Saeid Eslami3, Majid Jangi1, Azadeh Kamel Ghalibaf1, Sara Zangouei4
1 Student Research Committee; Department of Medical Informatics, Faculty of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran
2 Medical Informatics Research Center, Institute for Futures Studies in Health; Department of Health Information Management and Technology, Faculty of Management and Medical Information Sciences, Kerman University of Medical Sciences, Kerman, Iran
3 Department of Medical Informatics, Faculty of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran; Department of Medical Informatics, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands
4 Department of Computer, Mashhad Branch, Islamic Azad University, Mashhad, Iran
| Date of Web Publication | 10-Dec-2018 |
Correspondence Address:
Dr. Reza Khajouei
Medical Informatics Research Center, Institute for Futures Studies in Health, Kerman University of Medical Sciences, Haft Bagh Highway, PO Box 7616911320, Kerman
Iran
 Source of Support: None, Conflict of Interest: None  | Check |
DOI: 10.4103/japtr.JAPTR_320_18
Not considering the usability in designing clinical information systems causes problems in human–computer interaction and patient dissatisfaction. Therefore, in this study, the usability of the bed information management system (BIMS) was examined by think-aloud method. This cross-sectional study was conducted on the BIMS in 50 noneducational hospitals. Participants consisted of three groups including users, facilitators, and technical support. To carry out the study, a scenario consisting of four tasks was designed. Three researchers analyzed the recorded files to identify the usability problems and their severity. The mean time of the evaluation process was 20:33 ± 4:47 s. The total number of the problems identifies by users was 80 cases. Data entry and layout problems with 38 (48%) and 33 (41%) cases were the most frequently found problems, respectively. About 61% and 55% of the data entry and layout problems had a minor severity (Severity 2), respectively. Furthermore, 43 (54%) cases of the problems were resolved by the users and 32 (40%) cases by the facilitator assistance. This study showed that a large number of the problems were due to the system poor design. Furthermore, by increasing the users' level of knowledge about the system, it is possible to enhance user-system interaction. It is recommended that before designing and implementing a system, the system should be evaluated for usability, and the users should be educated in clinical information systems.
Keywords: Bed information management system, think aloud, usability testing
How to cite this article:
Mazaheri Habibi MR, Khajouei R, Eslami S, Jangi M, Ghalibaf AK, Zangouei S. Usability testing of bed information management system: A think-aloud method. J Adv Pharm Technol Res 2018;9:153-7 |
How to cite this URL:
Mazaheri Habibi MR, Khajouei R, Eslami S, Jangi M, Ghalibaf AK, Zangouei S. Usability testing of bed information management system: A think-aloud method. J Adv Pharm Technol Res [serial online] 2018 [cited 2023 Mar 24];9:153-7. Available from: https://www.japtr.org/text.asp?2018/9/4/153/247176 |
| Introduction | |  |
In health-care domain, there are still significant barriers to successfully implement some health information systems.[1],[2] One of the most important barriers is the usability problems in these systems.[1],[3],[4] Applying usability ensures the quality and promotion of software products with the aim of reducing errors, accelerating the workflow, and increasing the efficiency and effectiveness of the health information technology systems.[5],[6],[7]
Previous studies have shown that not considering the usability in designing clinical information systems has caused problems in human–computer interaction, negative effects on clinical outcomes, and patient dissatisfaction.[1],[7],[8],[9] Therefore, the need to evaluate the usability of the health information systems is felt. There are two general methods for evaluating this indicator, including the usability inspection and usability-testing methods.
The think-aloud method is one of the user-based methods which are widely used.[10],[11] Studies have shown that this method provides more accessibility to the information about the user's thoughts, interactions, and strategies in complex working conditions,[1],[11] indicating the validity and reliability of this method.[11],[12]
In this study, the usability of the bed information management system (BIMS) was evaluated. This system is a part of the admission system and is one of the subsystems of the hospital information system (HIS), consisting of three processes including patient admission, placement, and discharge.[13] It is very important to appropriately use BIMS in this section suffering from visitors' heavy pressure along with limited capacity because inefficient bed management in hospitalization process leads to an imbalance between demand and capacity and negatively affects other therapeutic trends. The aim of this study was to assess the usability of this system using the think-aloud method.
| Materials and Methods | | |
In this descriptive cross-sectional study, the usability of BIMS was evaluated in 50 noneducational hospitals in Iran in 2016. The project was implemented with the license of hospitals' IT management.
Participants
Participants in this study consisted of three groups, including users, facilitators, and technical support. Users included 16 students of the computer engineering senior year. These people had knowledge about designing and analyzing computer systems as well as the principles and concepts of programming languages but no knowledge and experience about HIS. These people may be used as users of this system in the future. Before starting the evaluation, all the users were verbally familiar with the study objectives, and their informed consent was obtained. To carry out the evaluation process, one person was employed as facilitator alongside each user. Facilitators did not intervene in the evaluation process. Only if users halted in the usability testing, did they remind users to express their thoughts loudly.[14] Facilitators were medical informatics specialists who had knowledge and experience in the field of usability assessment. Due to the possibility of software and technical problems during the evaluation, a software specialist was also employed as a technical support.
Evaluation tool
The evaluation was performed in rooms similar to the users' real working environment. These rooms had enough light, a desk, two chairs, and a computer system. A client version of BIMS was installed on the computer system. To carry out the evaluation process, the Camtasia Studio software version 8.4.3 was used to capture user interaction with the system. The microphone and video camera were used to record user's voice and capture the faces and modes of the user, respectively.
To conduct the evaluation process, a scenario was designed based on the BIMS tasks. The scenario included four tasks: outpatient admission, completion of the inpatient admission form, selection of the inpatient's bed, and entry of inpatient's attendant information. A special form was designed for the registration of the report on the user's performance by the facilitator during the usability testing.
Evaluation process
Think aloud is an empirical method based on observing the function of the system at the time of use.[15],[16],[17] In fact, the purpose of this method is to collect the information about the users' cognitive interaction with the system. In this method, the users are asked to express what they see, think, feel, and what decisions they make.[14]
Before starting the assessment, a training session was held for 2 h by the main researchers for familiarization with the HIS and its processes. The think-aloud method was also taught to the users and facilitators for 15 min in detail in order to know how to express thoughts, feelings, and decisions. After the end of the evaluation, users were asked to submit their suggestions to improve system performance. These proposals were registered in the report registration form by the facilitators.
Analysis of results
After completing the assessments, three main researchers analyzed the files recorded by Camtasia software, audio files, user-recorded images, and report registration forms. The researchers independently developed a list of user-described usability problems and their severity. The identified usability problems and their severity were merged into a final list, and the disagreement between the researchers was resolved by reviewing the audio and video data. To classify problems, the method of Van den Haak et al. was used.[18] According to this method, the problems were divided into four main categories:
- Layout problems
- Terminology problems
- Data entry problems
- Comprehensiveness problems.
Regardless of these four categories of problems, users occasionally encountered technology problems such as trouble with the network connection. These problems were excluded from the analyses. To rate the usability problems' severity, Nielsen's method was used.[19] The severity rate was measured by three criteria including the problem frequency, the problem effect on the user, and the problem continuity.[20] Based on the Nielsen's method, the problems' severity was divided into five main categories:
- 0 = no usability problem
- 1 = cosmetic problem
- 2 = minor usability problem
- 3 = major usability problem
- 4 = usability catastrophe.
However, the severity rated as “0” was omitted from the problems list by researchers' consensus. Data were analyzed by SPSS software version 20 (IBM, USA). All data were reported as mean ± standard deviation.
| Results | | |
This study was conducted to investigate the usability problems of BIMS in 50 hospitals by think-aloud method. Participants included 13 (81%) males and 3 (19%) females with the mean age of 22.38 ± 1.36 years. The mean time of the evaluation process in this study was 20:33 ± 4:47 s. The duration of the evaluation process for each user is shown in [Chart 1].
The total number of the problems identified by the users was reported as 80 cases with an average of 5 problems per user. The classification and severity of the problems to which users encountered in the assessment process are shown in [Table 1]. According to this table, data entry problems were the most frequent problems with 38 (48%) cases. About 61% of the problems in this category had a minor severity (Severity 2). In this category, data entry problem was the most frequent in the field of birth date in outpatient admission ( first task) with 10 (13%) cases, followed by the field of therapeutic physician in completing the inpatient admission form (second task) with 6 (8%) cases. As shown in [Table 1], layout problems identified by the users were 33 (41%) cases. In this category, 18 problems were rated as Severity 2, and the problems of not finding the empty bed in each room (third task) and not finding the save key were 6 (8%) and 4 (5%) cases, respectively. | Table 1: Usability problems identified by users in terms of category and severity
Click here to view |
[Chart 2] shows how the usability problems were resolved in the evaluation process by users. According to this chart, 43 (54%) problems were resolved by the users without the facilitator intervention, and 32 (40%) cases were resolved with facilitator assistance without stopping the evaluation process. Furthermore, 5 (6%) problems remained as unresolved after the user's review, and the evaluation process continued without that task.
During the 16 evaluations using the think-aloud method, despite the presence of many problems, none of the users used the system guides. In addition, 13 (81%) users offered 35 suggestions to improve system performance. The most suggestions were related to the better designing of date field (17%) and designing fields as slider (11%).
| Discussion | | |
In this study, a usability testing was done for BIMS by means of a think-aloud method. The result of this study showed that during the evaluation process, 80 problems were identified by the users. As all the users had knowledge about the field of computer systems analysis and design, they were able to identify a large number of usability problems. Studies have shown that the more the people are aware about the information systems design, the greater the number of identified usability problems is.[21]
In this study, the most usability problems to which the users encountered were in data entry category (48%). Consistent with this study result, two studies conducted by Van den Haak et al. showed that the most common problems were data entry.[18],[22] The problem of data entry in the field of birth date gave the largest share in the data entry category. This finding can be attributed to the difference between the system designer's thinking model (based on the estimation of the user's needs) and the user's mental model (based on the interaction with similar systems).[23] Although the users had knowledge about the computer systems design, this problem was over-repeated due to the improper design of this field and the inconsistency in the manner of entering date in different languages.
In the present study, the layout problems (41%) were the second most commonly found problems after data entry problems. In this category, the most identified problem was in finding the empty bed in each room. This can be due to the lack of users' familiarity with HIS and the process of hospitalization.
In this study, the usability problems' severity was also investigated. Despite the high frequency of layout problems, they had the lowest severity average (2.24 ± 0.71). This result can be attributed to its less impact on the user's performance so that user was able to solve the problem with the trial and error method; therefore, these problems showed less severity. While the data entry problems had a great impact on the user's performance so that the user stopped during the evaluation process; thus, the severity of these problems was more. The study by Van den Haak et al. also showed that the layout problems had less severity than the other problems.[18]
During the evaluation process, 54% of the problems were resolved by the users and without the facilitators' intervention. Furthermore, none of the users employed the system help option to fix the usability problems encountered during the evaluation. It seems that the users' knowledge about the field of programming languages made them prefer to use the trial and error method to solve the problems rather than using the help system. Similar to this study result, in Yen et al. study, this finding was also observed.[11]
The purpose of the usability-testing studies is to identify system usability problems and provide solutions to solve these problems.[24] In this regard, the users made suggestions for solving these problems and improving system performance. Most of these suggestions were about the better designing of some fields, for example, designing fields as slider.
This study had several limitations. First, this study evaluated the usability of BIMS in noneducational hospitals where the conditions may be different from educational hospitals. Therefore, it is recommended that this review should be carried out in the educational hospitals. Second, due to the lack of licensing by hospitals due to security reasons, this assessment was performed in a separate environment from the hospital (laboratory conditions). However, it was tried to simulate all the actual environment conditions completely. On the other hand, one of the most important strengths of this study was the usability testing of BIMS by accurate think-aloud method. In addition, this study was one of the few studies in this field in Iran.
| Conclusions | | |
This study showed that a large number of the problems identified during the evaluation process were related to data entry. This was due to the system poor design in accordance with the needs of users. Therefore, it is recommended that before designing and implementing a system in health care, the system should be evaluated for usability to avoid possible errors in the treatment process.
Furthermore, the increase in users' level of knowledge and familiarity with the system leads to a better and easier interaction between them. Thus, it is recommended that users be educated in various health-care departments before using systems. Participants in this study had previous knowledge about the computer systems design. As a future work, a study would be conducted on users with no knowledge about the computer systems design, and the results of these two studies would be compared.
Acknowledgments
This study was supported by Mashhad University of Medical Sciences. We thank all students of Asrar University for the collaboration in this study.
Financial support and sponsorship
Nil.
Conflicts of interest
There are no conflicts of interest.
| References | | |
| 1. | Khajouei R, Hasman A, Jaspers MW. Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system. Int J Med Inform 2011;80:341-50.  |
| 2. | Ghorbani N, Ahmadi M, Sadoughi F, Ghanei M. Developing data elements for research information system in health; a starting point for systems integration. Iran J Public Health 2012;41:30-9. |
| 3. | Campbell EM, Guappone KP, Sittig DF, Dykstra RH, Ash JS. Computerized provider order entry adoption: Implications for clinical workflow. J Gen Intern Med 2009;24:21-6. |
| 4. | Kim MS, Shapiro JS, Genes N, Aguilar MV, Mohrer D, Baumlin K, et al. A pilot study on usability analysis of emergency department information system by nurses. Appl Clin Inform 2012;3:135-53. |
| 5. | Kushniruk AW, Borycki EM. Low-cost rapid usability engineering: Designing and customizing usable healthcare information systems. Healthc Q 2006;9:98-100,102. |
| 6. | Karahoca A, Bayraktar E, Tatoglu E, Karahoca D. Information system design for a hospital emergency department: A usability analysis of software prototypes. J Biomed Inform 2010;43:224-32. |
| 7. | Kushniruk AW, Triola MM, Borycki EM, Stein B, Kannry JL. Technology induced error and usability: The relationship between usability problems and prescription errors when using a handheld application. Int J Med Inform 2005;74:519-26. |
| 8. | Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005;293:1197-203. |
| 9. | Kim MS, Mohrer D, Trusko B, Landrigan P, Elkin P. World Trade Center Medical Monitoring and Treatment Program: A Clinical Workflow Analysis. AMIA Clinical Research Informatics Summit 2010. San Francisco, CA; 2010. |
| 10. | Thyvalikakath TP, Monaco V, Thambuganipalle H, Schleyer T. Comparative study of heuristic evaluation and usability testing methods. Stud Health Technol Inform 2009;143:322-7. |
| 11. | Yen PY, Bakken S, editors. A comparison of usability evaluation methods: Heuristic evaluation versus end-user think-aloud protocol – An example from a web-based communication tool for nurse scheduling. AMIA Annual Symposium Proceedings. American Medical Informatics Association; 2009. |
| 12. | Guan Z, Lee S, Cuddihy E, Ramey J, editors. The validity of the stimulated retrospective think-aloud method as measured by eye tracking. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM; 2006. |
| 13. | Proudlove N, Boaden R, Jorgensen J. Developing bed managers: The why and the how. J Nurs Manag 2007;15:34-42. |
| 14. | Van Someren M, Barnard Y, Sandberg J. The Think Aloud Method: A Practical Approach to Modelling Cognitive. Academic Press, London; 1994. |
| 15. | Lewis C. Using the ‘Thinking-Aloud’ Method in Cognitive Interface Design. Research Report RC9265. IBM TJ Watson Research Center; Yorktown Heights, New York; 1982. |
| 16. | Ericsson KA, Simon HA. Verbal reports as data. Psychological review. 1980;87:215. |
| 17. | Hix D, Hartson HR. Formative Evaluation: Ensuring Usability in user Interfaces. Virginia; 1992. |
| 18. | Van den Haak MJ, de Jong MD, Schellens PJ. Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: A methodological comparison. Interact Comput 2004;16:1153-70. |
| 19. | Nielsen J. Usability Engineering. Boston: Academic Press; 1993. |
| 20. | |
| 21. | Joshi A, Arora M, Dai L, Price K, Vizer L, Sears A, et al. Usability of a patient education and motivation tool using heuristic evaluation. J Med Internet Res 2009;11:e47. |
| 22. | Van Den Haak M, De Jong M, Jan Schellens P. Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behav Inf Technol 2003;22:339-51. |
| 23. | Donald N. The Design of Everyday Things. Doubled Currency. New York; 1988. |
| 24. | Beuscart-Zéphir MC, Elkin P, Pelayo S, Beuscart R. The human factors engineering approach to biomedical informatics projects: state of the art, results, benefits and challenges. Yearb Med Inform. 2007;16:109-27. |
[Table 1]
| This article has been cited by | | 1 |
The Think-Aloud Method for Evaluating the Usability of a Regional Atlas |
|
| Tomas Vanicek, Stanislav Popelka | | ISPRS International Journal of Geo-Information. 2023; 12(3): 95 | | [Pubmed] | [DOI] | | | 2 |
Integrating user experience assessment in Re-CRUD console framework development |
|
| Asyraf Wahi Anuar, Azri Azmi, Nazri Kama, Hazlifah Mohd Rusli, Nur Azaliah Abu Bakar, Norliza Mohamed | | Wireless Networks. 2022; | | [Pubmed] | [DOI] | | | 3 |
Participatory Design of an Electronic Medical Record for Paediatric Palliative Care: A Think-Aloud Study with Nurses and Physicians |
|
| Sven Kernebeck,Theresa Sophie Busse,Chantal Jux,Dorothee Meyer,Larissa Alice Dreier,Daniel Zenz,Boris Zernikow,Jan Peter Ehlers | | Children. 2021; 8(8): 695 | | [Pubmed] | [DOI] | | | 4 |
Development and usability testing of Riyadh Mother and Baby Multi-center cohort study registry |
|
| Nasriah Zakaria,Hayfaa Wahabi,Mohammed Al Qahtani | | Journal of Infection and Public Health. 2020; | | [Pubmed] | [DOI] | |
|
 |
|
|
|
Search Pubmed for