Contemporary EHF 2023
User centred approach to developing a Concept of Operations
| Document
|
Author Kate Shield & Elaine Thompson |
| Abstract The paper describes a user centred approach to the development of a Concept of Operations and its application to the Synthetic Environment, which is a digital signalling design tool that will be used to develop seamlessly integrated scheme designs for Network Rail. |
A process to assess the use of human augmentation technologies in defence
| Document
|
Author Alison Clerici |
| Abstract This research evaluated and adapted the use of the Early Human Factors Analysis (EHFA) process to create a methodology for assessing the use of human augmentation technologies in defence. |
Assessing pilots’ situation awareness using touchscreen inceptor
| Document
|
Author Kyle Hu, Wojciech Tomasz Korek & Wen-Chin Li |
| Abstract This research aims to investigate the innovative technology of touchscreen inceptor impact on pilots' situation awareness compared to a traditional sidestick. The Pupil Lab eye tracker collected pilots' fixation counts in the Future Systems Simulator (FSS), and subjective measure was the situation awareness rating technique (SART). A significant difference was spotted in the attention demand, understanding, and total score from SART analysis. Furthermore, the visual parameter of fixation counts indicated that pilots spent less time on OTW and more time on PFD when interacting with the touchscreen inceptor compared to the sidestick. The findings show the potential to implement a touchscreen in future flight deck designs. |
Identifying resilience: A system safety review of trauma and orthopaedic theatres
| Document
|
Author Victoria E. Wills, Andrew Seaton |
| Abstract A system safety review to assess the resilience in Trauma and Orthopaedic (T&O) theatres was conducted in response to a number of Never Events. The imminent publication of the Patient Safety Incident Response Framework (PSIRF) paved the way for an alternative to traditional serious incident investigation, proposing a systems-based approach and enabling subsequent improvements to be based on ‘work as done’, rather than ‘work as imagined’. Analysis identified opportunities for interventions that built system resilience, which were developed and tested by front line staff as part of a Quality Improvement (QI) collaborative. The approach demonstrated a practical application of the integration of systems theory, patient safety, resilience engineering and quality improvement approaches. |
Recreational Boating Safety: A Systems Analysis of the Causal Factors Contributing to Accidents
| Document
|
Author Helen Wordsworth, Patrick Waterson & Will Tutton |
| Abstract Recreational boating has become an extremely popular past-time in the UK, particularly since COVID-19, with boat equipment sales up by 25%, compared with pre-pandemic levels. Wilson (2022) predicts a further 11% growth in sales in 2021-2022 and the market is currently exceeding pre-pandemic trading levels. We used a set of 12 Accimaps to analyse of contributory factors leading to recreational boating accidents documented by the MAIB. The data from the Accimaps was then used to scope a set of questions which formed the basis of two surveys. One survey was for members of the public involved in recreational boating and the other for professional individuals involved in the recreational boating community. Key findings were that lack of training, knowledge and preparation were seen as key reasons leading to unsafe boating situations, in addition to recklessness. Most participants wore lifejackets whilst boating and approximately 50% thought alcohol consumption was acceptable at some point during a boating trip. Boat/sailing clubs were seen to reinforce good safety culture but outside of clubs was less positive. Some recommendations were developed following the study. It would be beneficial to introduce a mandatory qualification prior to the purchase of any motorised vessel, such as that employed in Australia or similar to the International Certificate for Operators of Pleasure Craft (ICC).It may be worthwhile developing more interactive, nationally consistent signage at popular launch locations. A final recommendation would be to place some responsibility on manufacturers. |
Critical Assessment of the usability of a New Modular Ward
| Document
|
Author Jonathan M McCloud & Lauren Morgan |
| Abstract An acute District General Trust commissioned a new modular building to house a surgical ward. The process of commissioning and design of the building was in accordance with standard trust practice and established building regulations for hospital buildings. The purpose of this project was to assess the ward layout in terms of safety and efficiency and offer mitigations for any problems found. Observation of the ward layout demonstrated significant problems with sight lines in to bays, layout and design of toileting facilities, and layout and positioning of key ward areas including kitchen facilities, office space, drug and fluids stores, and nursing stations. To further examine the ward layout, Hierarchical Task Analysis and link analysis were performed to review two tasks: one an emergency task (treatment of a deteriorating patient) and a routine common task (serving meals). We also considered the Systems Engineering Initiative in Patient Safety (SEIPS) framework to ensure that our analysis was as thorough as possible. This work demonstrated significant problems with the ward layout, in terms of sight lines, workspace, communication, and contributed to significant extra time doing both emergency and routine duties. This led to a series of recommendations to mitigate on the ward, as well as a recommendation that appropriate staff involvement is included at appropriate times in planning future new build and refurbishment projects. |
Lesson Learned – the similarities and differences of human factors in Aircraft Maintenance between JL123 and CI611
| Document
|
Author Punthit Kulsomboon, Edem Yao Tsei, Gayatri Rebbapragada & Wen-Chin Li |
| Abstract This paper makes use of the human factors analysis and classification system (HFACS) to analyse maintenance-related causal factors of two accidents - Japan Air Lines (JAL) flight 123 (JL123) and China Airlines (CAL) flight 611 (CI611). Furthermore, the pathways that could have resulted in the two accidents were identified by applying the HFACS framework. The study also compares the similarities and differences between these accidents. The findings of this paper lend support to past research on HFACS where higher levels at an organisation have been shown to have directly affected the lower levels. Lessons from these accidents have also been identified in order to prevent recurrences. |
Voluntariness and extent of telework – association with heart rate variability
| Document
|
Author Leticia Bergamin Januario, Marina Heiden, Svend Erik Mathiassen, Tea Korkeakunnas, Gunnar Bergström & David M Hallman |
| Abstract We evaluated the impact that (1) voluntariness and (2) extent of telework had on 24h heart rate variability (HRV) measured objectively for three days in white-collar workers during the COVID- 19 pandemic. Workers with high voluntariness had higher root mean square of successive differences between R-R intervals (RMSSD), however high extent of telework did not affect HRV metrics, after adjusting for suitable covariates. These results may indicate higher parasympathetic activity, an indicator of good health. |
Performance and workload using an audible intelligent assistant during pilot training
| Document
|
Author David A. Hudson & Michael A. Bromfield |
| Abstract Boeing forecast that globally an additional 602,000 pilots will be needed by 2041 to meet year on year growth of 3.6% in passenger traffic. All pilots need to be trained in accordance with accepted regulatory standards. New technology is continuously being developed to enhance training and reduce training time. Research into the effectiveness of training technologies and how these impact pilot performance and workload is key to future growth. One such technology under review at the University of Birmingham is the ‘audible intelligent assistant’. This artificial voice that provides real-time feedback to the pilot during flight training. The system provides warnings, cautions and instruction to the human pilot to enhance pilot training. Preliminary results of the research showed an improvement in pilot performance against a specified set of target parameters of airspeed and altitude and a corresponding decrease in workload for 80% of pilots when using the audible intelligent assistant. Without AIA, pilot performance improved by 35% due to learning effects alone, with AIA pilot performance showed an improvement of 65%. |
Driving at night and how it’s influenced by perceived driver skills
| Document
|
Author İbrahim Öztürk & Natasha Merat |
| Abstract Night-time driving is associated with higher crash rates, partly due to reduced visibility of the driving environment. Crash data show that young drivers are over-represented in night-time incidents, world-wide. Self-reported driver skills provide an important indicator of driving abilities, which can also interact with night-time driving performance. This study investigated the relationship between drivers’ self-reported driver skills and their perceived night-time driving challenges, comparing responses between young and older drivers. Results found that young drivers with lower self-reported perceptual-motor skills and higher safety skills experienced more difficulties associated with night-time driving. |
Assessing pilots’ mental workload using touchscreen inceptor for future flight deck design
| Document
|
Author Joao Paulo Macedo, Kyle Hu, Rani Quiram & Samarth Vilas Burande |
| Abstract Touchscreen displays are one of the pillars of future flight deck design and it is foreseen that at some point traditional flight control inceptors will be modified to a touchscreen version. However, this transition can only be safe and successful with due regard for human performance implications. This study addresses it by comparing pilots’ mental workload for a traditional sidestick and an innovative touchscreen control inceptor. The results indicate that the new technology increases pilot workload, suggesting that further development is required to use it in future flight decks. |
Envisaging regenerative futures through Good Work Design
| Document
|
Author Elise Crawford, Sara Pazell & Nektarios Karanikas |
| Abstract As we move towards a fifth industrial revolution, concerns about the future of work are heightened. To answer the call for work that we all want, this paper extends the concept of Good Work Design (GWD) introduced by the Human Factors and Ergonomics Society in Australia in 2020. Following an overview of GWD, we present a list of respective features with the purpose to advance a humancentred design-led approach to workplace strategy that reconciles business success with worker health. Moreover, we argue that effective design practice should be regenerative, expanding capacity and capability for design throughout the organisation, while supporting sustainable futures. The goal of this paper is to stimulate ongoing debate, research, and practice in good work design. |
Task Switching – Managing Workload within Digital AFV Systems
| Document
|
Author Trevor Dobbins, Ryan Meeks, Dan Evans, Stuart Howe & Stephen Barrett |
| Abstract The Armoured Fighting Vehicle Commander’s role is characterised by having multiple mission critical tasks. They are required to rapidly redirect their attention at short notice as events change. This paper describes how this task-switching is modelled and analysed, within the system model, to manage workload and develop/deliver a useable system |
Putting Ostomates at the Heart of Pouch Design
| Document
|
Author Anna McLister, Chloe Roberts & A C B Medeiros |
| Abstract Ostomy pouches are used daily by over 13,000 people in the UK each year, to collect effluent from their stomas. Although this Class I medical device has undergone a design revolution since the 1940s, ostomates’ needs are still not being fully realised. Building upon knowledge and insights gained from interviewing and surveying ostomates, this paper will explore how applying key Human Factors considerations could help inform the future design of ostomy pouches and ultimately, improve the quality of life of ostomates. |
Human Factors Guidance for Robotic and Autonomous Systems (RAS)
Document  |
Author Claire Hillyer, Hannah State-Davey, Nicole Hooker, Richard Farry, Russell Bond, James Campbell, Phillip Morgan, Dylan Jones, Juan D. Hernández Vega & Philip Butler |
| Abstract This paper outlines recent (2021/2022) work to produce Human Factors (HF) guidance to support the design, development, evaluation, and acquisition of Robotic and Autonomous Systems. |
Responding to rudeness: does instigator status and directness matter?
| Document
|
Author Amy Irwin, Helen Silver-MacMahon, Liz Mossop, Kendyl Macconnell & Luiz Santos |
| Abstract The current paper applies a biobehavioural model of workplace incivility to explore responses to rudeness within the veterinary context. Veterinarians and veterinary nurses (n=132) were asked to evaluate six fictional scenarios depicting two types of rudeness (direct versus indirect) across three instigators (clients, co-workers and senior colleagues). The findings indicated that direct rudeness (demeaning comments) was appraised more negatively than indirect rudeness (ignoring person). Responses varied across scenarios; direct rudeness was associated with reciprocation, exit, avoidance, discussion with manager and support seeking, whereas indirect rudeness was associated with affiliative and ignoring responses. There was a significant positive association between appraisal and confrontation, exit, avoidance, support seeking and reporting responses. The findings confirm the utility of the biobehavioural model of incivility response and build on this model in terms of variation in response selection according to directness and status effects. From a practical perspective the findings suggest that interventions to manage rudeness in veterinary practice should accommodate variation in rude behaviours and include tailored responses based on instigator. |
Assumption-Based Leading Indicators to Monitor Healthcare System Drift Towards Failures
| Document
|
Author Elizabeth Wilson-Lewis & Gyuchan Thomas Jun |
| Abstract This study explores how the assumption-based leading indicator approach can be applied to monitor organisational drift of healthcare systems towards failures based on the Mid Staffordshire NHS hospital failure case. |
Gender Equitable Human Factors and E-micromobility
| Document
|
Author Katie J. Parnell, Siobhan E. Merriman & Katherine L. Plant |
| Abstract Human Factors methodologies and principles can help to close the ‘gender data gap’ through equitable, user-centered research approaches and sociotechnical systems analysis. This paper presents research conducted into the use and uptake of electric micromobility (e-micromobility) through a gendered lens. Qualitative research is combined with systems methodologies to provide gender-equitable recommendations that highlight how this mode of travel can be more gender-equitable. |
Assessing the Benefits of Virtual Reality Training for Manual Assembly
| Document
|
Author Charlotte Temmink & Yee Mey Goh |
| Abstract The aim of this study is to assess, through experimentation, the effectiveness of Virtual Reality training when compared to a traditional written instructions in a manufacturing assembly process. The effectiveness was evaluated in terms of perceived workload and task performance, with participants asked to self-report on the experience during both the written and VR training. Findings indicate that VR training reduced the user’s workload perception and errors made during the task, however there was no significant impact on time taken to complete the task. |
Psychophysiological coherence training reduces pilots’ perceived stress in flight operations
| Document
|
Author Laurie Marsman, Jingyi Zhang & Wen-Chin Li |
| Abstract The present paper reports the results of a four-week study assessing the relationship between psychophysiological coherence training, which aims to improve the synchronisation between one’s physiological rhythms leading to a positive emotional state, and perceived stress levels in commercial air transport pilots. Next to three self-report stress questionnaire measurements, qualitative data were gathered as well to gain more insights into the training effects. Results show significant reductions in the flight crew’s perceived stress levels during the four-week psychophysiological coherence practice period. Finally, the results are discussed. |
Insights into Human Behaviour Hold the Key to the Energy Transition
| Document
|
Author Kirsty Novis & Thomas A. Norton |
| Abstract The paper describes how understanding user behaviour and implementing behaviour change initiatives may be the key to changing attitudes to energy consumption and energy transition. The objective of this paper is to explore how four key factors; behaviour centred-design, preventing rebound effects, providing meaningful feedback and, understanding how behaviour scales can make an impact. |
Defining Roles of the Remote Operator in Autonomous Vehicles
| Document
|
Author Hannah Parr, Catherine Harvey & Gary Burnett |
| Abstract This paper proposes four distinct roles of a remote operator in supporting future SAE Level 4 and 5 autonomous vehicles (AVs). These are Remote Monitoring, Remote Assistance, Remote Management and Remote Driving. A set of scenarios were created based on academic literature and industry reports to represent an extensive range of interactions which might occur between Remote Operators, AVs and other human agents. Operator sequence diagrams were created to represent the task in each scenario and analysed to identify the involvement of the different Remote Operator roles. This is used to draw conclusions on aspects of work for future Remote Operators and as a starting point for further investigation into information requirements and workstation design. |
Anthropometric and ergonomic assessments of braiding activity among female hairdressers in Lesotho using the ART method
| Document
|
Author Tebello Pusetso, Samuel Mekonnen & Hui Lyu |
| Abstract Hairdressers in their careers are at a high risk of work-related musculoskeletal disorders (MSD). The braiding process requires hairdressers to endure prolonged repetitive hand movements and awkward body postures. This study conducted hand anthropometric measurements of female hairdressers in Lesotho and assessed their risk of muscle injury during hair-braiding activities using the Assessment of Repetitive Task (ART) method. Results found that braiding activity is at high risk of muscular injury. |
Safety culture in nuclear power plant construction
| Document
|
Author Teemu Reiman, Kaupo Viitanen, Jesse Hakala & Karolina Wrona |
| Abstract Safety culture related lessons learned from nuclear power plant construction projects in Finland are presented. A set of questions are proposed for organizations to discuss. The implications of these questions and constraining questions are considered. |
Facilitators and barriers to a safe opioid prescribing process in general practice
| Document
|
Author Gill Gookey & Mike Fray |
| Abstract Opioids e.g., morphine are high-risk medications that are frequently prescribed using a complex process in general practice. The current opioid prescribing process within six general practices was mapped using template analysis which highlighted high levels of variation. The Systems Engineering Initiative for Patient Safety v2.0 framework was used to identify overall aims for a safe opioid prescribing process and associated facilitators and barriers. |
Augmented Design with Voice Recognition and Auditory Alerts in the Flight Deck
| Document
|
Author Niall Miranda, Jean-Baptiste Bonotaux & Wen-Chin Li |
| Abstract Devices using voice recognition and verbal auditory alerts have advanced rapidly in the modern world. In an era that drives the world using voice commands, aviation has lagged to implement this technology in cockpits. Contrary to the rapid pace of advancements in cockpit instruments, auditory alerts have also been largely primitive. With such technologies gaining rapid acceptance in many industries, it is about time to consider adopting the same in cockpits, especially to support pilots during enhanced operational workload. This paper conducts a use case study on the prevailing aural technologies in the cockpit whilst exploring the augmentation of contemporary technologies in voice recognition and verbal auditory alerts to aid pilots’ cognition and reduce mental workload. |
A Human Factors review of “the Blue Puffer” asthma reliever inhaler
| Document
|
Author Deborah Stratford & Susan Whalley-Lloyd |
| Abstract The literature reports that Asthma Inhaler technique has not improved during the last forty years, despite improvement strategies focused on educating users to improve their technique and compliance. This is particularly critical for reliever inhaler users when ‘use error’ may result in a full asthma attack and possible death. This paper presents a pilot study Human Factors design review of the standard UK reliever inhaler, commonly referred to as ‘the blue puffer’. The results indicate a mismatch between ‘work as done’ and ‘work as imagined’ and that this mismatch appears to be influenced by the design of the inhaler. Conceptually it appears possible to improve the design of technical components of the inhaler system to reduce use errors and hence improve patient safety. This would require appropriate scenario and user testing, with any changes being integrated into the system as a whole. |
Ghost Busting: A Novel On-Road Exploration of External HMIs for Autonomous Vehicles
| Document
|
Author David R. Large, Madeline Hallewell, Xuekun Li, Catherine Harvey & Gary Burnett |
| Abstract The absence of a human driver in future autonomous vehicles means that explicit pedestrian-driver communication is not possible. Building on the novel ‘Ghost Driver’ methodology to emulate an autonomous vehicle, we developed prototype external human-machine interfaces to replace existing cues, and report preliminary, qualitative findings captured from a sample of pedestrians (n=64) who encountered the vehicle when crossing the road, as well as reflecting on the method. |
Exploratory study of virtual reality flight training device for upset prevention and recovery training
| Document
Insert here |
Author Filip Florek |
| Abstract This exploratory study aimed to establish whether a virtual reality (VR) enabled flight training device will provide effective Upset Prevention and Recovery Training (UPRT) to ensure equivalent safety with the Flight Simulation Training Device (FSTD) considering pilot situational awareness competency. It was achieved by determining the effects of the pilot’s presence, task-related stress and cybersickness on situational awareness during upset prevention, and, if necessary, recovery and by assessing pilot acceptance of VR-enabled flight training device in UPRT. No evidence has been found that situational awareness was negatively affected by exposure to VR, with certain reactions to stimuli degradation, i.e., flight upset in Instrument Meteorological Conditions (IMC) resulting in a predictable outcome of increased attentional demand. |
Ergonomic mismatch between university student anthropometry and classroom furniture in Tanzania
| Document
|
Author Jecha Suleiman Jecha, Samia Rafique & Hui Lyu |
| Abstract The standard and guidelines of school furniture dimensions have been developed in many countries, but it’s never been explored for university students in Tanzania. This study evaluated the potential mismatch between classroom furniture dimensions and anthropometric characteristics of 289 Zanzibar university students (167 females, 122 males) aged 17- 27 years. The results indicated high rates of mismatches between the body dimensions of the students and the existing classroom furniture, with seat height (100%), desktop height (93.08%), and seat width (81.40%) being the furniture dimensions with higher level of mismatch and backrest height with a lower level of mismatch (66.26%). The findings suggest that the least developed countries should improve school furniture design based on anthropometric results to avoid or minimize student discomfort and MSD problems. |
Can Intersectionality Increase Active Travel in Marginalised Groups? A Literature Review
| Document
|
Author Joy McKay, Katie Parnell & Katie Plant |
| Abstract Active travel, such as walking, running and cycling, are cheap, sustainable, and healthy ways to transit within in the urban environment. Many marginalised groups are either underrepresented in active travel modes or find they are limited using them in certain neighbourhoods or at certain times. These limitations lessen the accessibility of a range opportunities including those of employment, social activities and cultural experiences to a wide range of citizens. This review endeavours to recognise those barriers to active travel which affect a diverse selection of society and understand affordances which encourage use of these modes. It aims to identify solutions which may encourage active travel across a diverse community leading to an urban environment which is more equitably accessible for all. |
Situation Awareness in Midwifery Practice
| Document
|
Author Rachael L. Martin & Paul Bowie |
| Abstract Situation Awareness (SA) is commonly defined as “the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future” (Endsley, 1995:36). From this cognitive perspective, SA is synonymous with perception or attention, and involves a continual monitoring of the status quo for changes that might require action by frontline operators (Flin et al 2008). Within the midwifery literature, “loss of situation awareness” has been cited as a contributory factor to adverse events and unwanted clinical outcomes (HSIB 2020; Knight et al 2014; RCOG 2017). This operationalisation of SA is problematic for multiple reasons which are explored in this discussion paper. The paper begins by exploring the transferability of human factors lessons between safety critical industries such as aviation and healthcare. Different theoretical perspectives on SA are evaluated, highlighting that the theoretical concept has been misapplied in midwifery, with distinct differences from Endsley’s original model in how it is defined and measured. The paper provides an overview of the difficulties in measuring SA, which limit the prospective utility of the construct. Furthermore, retrospective identification of loss of SA is value laden and subject to hindsight bias. This stands in opposition to the Human Factors systems approach where “human error” should be viewed as a symptom of systemic problems within an organisation, rather than a causal factor (Amer-Wahlin and Dekker, 2008; Shorrock and Williams 2016). This paper proposes that a more holistic perspective is required which considers the individual clinician within the context of the wider sociotechnical system, rather than focus solely on the performance of individuals. It is vital to identify the system factors which may lead to loss of situation awareness, in order to redesign the work environment to minimise patient harm and maximise safety (Singh et al 2006). Opportunity also exists for further research to investigate whether an alternative model of SA may be more appropriate for use in the healthcare context generally, and maternity care specifically, better reflecting the complex system in which clinicians work. |
Harnessing A Human Factors Approach to Improve Patient Safety
| Document
|
Author Jenny Sutcliffe, Suzi Lomax & Jennifer Macallan |
| Abstract The interest in employing Human Factors (HF) in healthcare is increasing. The SCReaM HF and Team Resource Management (TRM) programme is aimed at raising the awareness, understanding and application of the science of HF within healthcare to help staff improve their safety and wellbeing and that of their patients. The programme is divided into three strands: rolling training, HF Projects and HF Engineering. The programme has been successfully embedded into an NHS Trust and provides a good model for how HF can be introduced and utilised within healthcare. |
How the Accuracy of Interactive Voice Assistants Affect Perceived Trust
| Document
|
Author Wenhu Zhang & Chris Baber |
| Abstract We asked 30 participants to ask questions of an Interactive Voice Assistant (IVA) which we had modified to provide different levels of accuracy in its answers. The levels of accuracy were low (55%) or high (80%). We also told users what level of accuracy to expect (60% or 100%). This produced a set of 6 combinations of actual accuracy with expected accuracy (including the condition when we did not tell the users which level of accuracy to expect). As expected, when users experience a more reliable IVA (i.e., 80% vs. 55%) their rating of trust is higher, and when actual an IVA with high accuracy and they are expecting accuracy to be high, then their trust rating is higher still. However, expected accuracy seems to outweigh actual accuracy, particularly when the actual performance is less than expected. Counter intuitively, this suggests that participants were not able to judge the actual accuracy of the IVA but relied on the expected accuracy. |
CRIT-UK: A tool to understand contributory factors involved in current cyclist incidents
| Document
|
Author Siobhan E. Merriman, Katie J. Parnell & Katherine L. Plant |
| Abstract Collisions between motorised vehicles and cyclists remain a persistent road safety issue worldwide, however the nature of these collisions remain poorly understood as there are currently few mechanisms available for cyclists to report sufficient detail about their collisions and near-miss incidents. Originally developed in Australia, this paper will describe the expansion of the Cyclists Report of Incidents Tool (CRIT) app to the UK road system, to understand the contributory factors involved in current cyclist collisions and near-miss incidents in the UK. Furthermore, it will explore how these factors may change with the introduction of Levels 2 and 3 Automated Vehicles. |
Using SUS for Current and Future AI
| Document
|
Author Richard Farry |
| Abstract The System Usability Scale (SUS) was assessed for its relevance and ease of use for assessing an AI capable of human-like interaction. Participants used SUS to assess Outlook, a contemporary consumer-grade AI interaction partners (smartphone digital assistants), and human teammates as a proxy ‘system’ for future human-like AI interaction partners. The results show that participants considered SUS to be relevant and easy to use for contemporary consumer-grade AI interaction partners, but not for human teammates. However, there was no meaningful difference in their ability to apply SUS between contemporary digital assistants, human teammates, and an email client. Thus, SUS can be used effectively for all of these kinds of systems. |
Human Factors Integration Strategy: Embedding Human Factors in Practice within Healthcare
| Document
|
Author Eva-Maria Carman, Giulia Miles, Bryn Baxendale, Emma Smith & Owen Bennett |
| Abstract Despite identifying the need for enhancing the use of Human Factors and Ergonomics in healthcare about 20 years ago, progress to date has been slow. A cohesive strategy is required that aligns these methods and expertise with established improvement, transformation and organisational development programmes and which is synergistic with existing work that seeks to address local system and organisational priorities. This paper describes progress to date and proposed future steps for the integration of Human Factors and Ergonomics in one large NHS Hospital Trust. |
The systemic causes of medication problems for hospitalised children
| Document
|
Author Adam Sutherland, Suzanne Grant, Stephen Tomlin, Denham L Phipps, & Darren M Ashcroft |
| Abstract Medication processes are chaotic and complex, and assumed to be undertaken by specific professionals in isolation from other healthcare tasks. However tasks are delivered simultaneously and adaptively because of the complexity of healthcare provision. This study aimed to explore the systemic contributory factors to medication related problems in children’s wards using multiple qualitative methods (230 hours participant observation and 19 semi-structured interviews). There is insufficient resource available to undertake all the processes to ensure safety; decisions about medicines were made with reference to immediate problems only; parents were relied on to administer medicines to children, and; there was widespread non-compliance with interventions to improve safety because they conflicted with day-to-day work. |
Review of UAV Loss of Control In-Flight: Accidents and Incidents
| Document
|
Author Rahma El Safany, Michael A.Bromfield |
| Abstract Sixty Uncrewed Aerial Vehicle accident reports were analysed to identify possible causal and contributory factors leading to loss of control in flight and recovery actions where applicable. Manufacturing and design errors were dominant in 22 causal factors (34% of events) and 18 contributory factors (22% of events) (e.g. ingestion of precipitation). Recovery was not attempted in 35 (55%) events. The relationship between age, total hours experience, hours experience on type, recovery attempts and number of accidents increase with operator age or lack of experience was also analysed. As total experience increases the number of accidents and attempted recovery increases. All this information is presented in a framework adapted from the Accident Route Matrix to recognise loss of control in flight in future accidents and improve recovery response. |
Design Blindspots: User testing clinical IT systems
| Document
|
Author Lauren J Morgan & Paula Pryce |
| Abstract In early 2021, the MHRA launched its guidance on applying human factors and usability engineering to medical devices including drug-device combination products in Great Britain (MHRA, 2021). In its guidance it states: A usability engineering process can, and should, be applied by device manufacturers in the identification, assessment and mitigation of potential patient and user safety risks; also in the analysis of incidents that have occurred, in order to identify learning and put into place corrective actions to improve device design However, experience in hospital healthcare is that many devices and IT systems are often poorly designed and continue to contribute to patient safety risks. A seminar Harvard Business Review paper stated: “to fix physician burnout, we must first fix the electronic patient record”. In everyday work in our hospitals we see examples where poor device and IT design is making clinicians lives harder, and decreasing patient safety as a consequence. We have a workforce cataclysm, of which the state of hospital devices and IT is possibly contributing to rather than helping to fix. We explore a simplified multi-method approach to user testing to identify patient safety and usability risks. We present evaluations of 3 clinical IT systems, showing how user testing conducted correctly easily identifies these safety risks. The MHRA guidance as currently stands is not being used fully by suppliers, we need to consider how to strengthen its impact. |
Digitalisation of HFE in Medical Product Development: Challenges and Opportunities
| Document
|
Author Diego Cortez & Erin Davis |
| Abstract This paper presents current challenges of applying human factors engineering (HFE) throughout medical device development, and the opportunity digitalisation creates for innovation in the field. The paper focuses on describing current HFE challenges and examples of how digital tools and software applications can contribute to the work of HF specialists in the medical industry, noting that there is large, unmet need for more HF expertise. Finally, it briefly presents a case study of a software released by Emergo by UL Solutions which aims to address some of these challenges. |
Human Factors and Procurement: Lessons Learnt from a High-Value Procurement Exercise
| Document
|
Author Eva-Maria Carman, Michael Johnson & Giulia Miles |
| Abstract Human Factors can inform and enhance traditional procurement processes by capturing the users’ input and considering the wider system into which the products will be implemented. Despite this, traditional procurement processes do not typically consider integrating HFE into the process in a systematic way. This paper describes the role of Human Factors in the different phases of a large procurement project and the lessons learnt for the procurement of hospital beds within one large NHS Hospital Trust. The aim of the procurement project was to determine the best solution that includes a variety of products and service contract from one supplier. A total of six different bedframes and two different types of mattress needed to be considered. The role of the HFE team was to provide advice at strategic project meetings, support the specification design, conduct an HFE evaluation of the products and ensuring a system’s perspective was considered throughout the process. Across all the product types, the HFE evaluation included 27 simulation testing sessions followed by feedback from staff on in-situ use on 23 different wards. Key lessons learnt included the value of qualitative data can add to support the decision-making process in procurement projects, the need to understand clinical needs as in this context there is no one perfect product due to the wide range of applications, and the need for HFE specialists to have a better understanding of the procurement process and their involvement across all phases of this type of project. |
Identifying Non-Technical Skills behavioural markers in rail controller and maintenance roles
| Document
|
Author Anisha Tailor, Paul Leach, Kirsten Huysamen & Priya Shah |
| Abstract This paper discusses the use of task observations, semi-structured interviews and thematic analysis to identify and classify Non-Technical Skills (NTS) behavioural data in rail controller and rolling stock maintenance roles. A human-centred design process was applied to develop materials to support rail organisations with observing, evaluating and developing NTS in rail controller and rolling stock maintenance roles. Wider factors influencing human performance in these roles were also identified and are discussed in this paper. |
Using a systems thinking tool to identify work system interactions in healthcare
| Document
|
Author Emma Smith & Stacey Sadler |
| Abstract This paper advances the use of Systems Engineering Initiative for Patient Safety (SEIPS) to provide a visual representation of how work system factors interact with each other to shape processes and outcomes. Healthcare professionals identified that deficiencies in work system factors surrounding the person, task, tools and technology, environmental factors, organisational factors and external factors shaped undesirable outcomes for the patient, professionals or organisation around the discharge process. Improving work system factors may decrease the likelihood of negative outcomes for the patient, professionals or organisation. |
Modelling User Contribution to Capability Within a Supervisory Control System
| Document
|
Author Mike Tainsh |
| Abstract Supervisory control is a common category of system employed for many surveillance applications and is a continuing subject of interest to ergonomists. During their development, following an initial statement of system requirements, capability options need to be assessed to understand the contribution of design features to system effectiveness. One technique that can be employed is capability modelling which aims to generate predictions of outcomes dependent on initiating events. A novel capability modelling technique is proposed based on an integration of ergonomics research results and professional input. |
Making the Right Choices: Behavioural Safety for Designers on a Construction Project
| Document
|
Author Shelley Stiles |
| Abstract Behavioural safety programmes are widely used across the Construction Industry, largely targeted at influencing behaviours of frontline workers and/or leadership behaviours. However, there is limited application of behavioural safety at the pre-construction (design) phase of a construction project, given the importance of the design community in eliminating and mitigating health and safety risks. This paper details a case study for the application of behavioural safety intervention targeted at the design community for a large infrastructure project. |
Key User Factors in a New Style Two-Person Train Cab
| Document
|
Author David Hitchcock, Kimberley Harding & Penny Gazard |
| Abstract The introduction of a new train to Transport for Wales (TfW) was initially faced with reluctance and resistance from key stakeholders. Consistent with previous train driver projects, TfW elected to adopt an ergonomics approach to identify, clarify and resolve issues of acceptance, safety and performance. The reasons for this were to address cab design issues, and to inform the design and delivery of a training programme intended to facilitate a cultural shift in drivers’ ways of working and those who support them. The ergonomics investigative work embraced consultations, literature review, measurements and observations of cab users, and was conducted alongside, and in harmony with, engineering reviews. This integrated approach led to a successful resolution to the concerns of stakeholders, resulting in accepted changes to working practices and deployment of the train. |
Digital Simulation Modelling providing a platform for ETCS Driveability Assessments
| Document
|
Author John Gunnell |
| Abstract The development of a true-to-life, cutting-edge Digital Simulation Model of a train drivers experience operating with a future European Train Control System (ETCS) system has provided a platform for robust Human Factors analysis and has proven to be a success with the driver stakeholders. |
Human factors in emergency management
| Document
|
Author Weixuan Li, Glyn Lawson & Gary Burnett |
| Abstract Good emergency management is essential when human beings face natural disasters. However, there are many human factors challenges in this area, in particular teamwork amongst those involved with the response efforts, but also communication, navigation, and workload. This article presents the findings from nine interviews with emergency management practitioners based in China, with an emphasis on the potential use of Virtual- and Augmented-Reality to address the aforementioned human factors issues. |
Trap and drag incidents on the London Underground – the role played by passenger mental models
| Document
|
Author Jasmine Bayliss, Patrick Waterson & Mark Young |
| Abstract A trap and drag accident occurs when a passenger gets part of their body, or an object trapped between the train doors and gets dragged along the platform as the train departs causing various consequences of varying severity (Roels, 2018). The primary aim of this study was to understand why trap and drag accidents occur, how they come about and, establish how they can be prevented. A secondary aim was to understand how frequently traps occur and go unreported, and to understand if passengers are aware of the risks at the Platform-Train-Interface (PTI). We found that the majority of participants expect the train doors to work in the same way as lift and automatic shop doors and believe that the doors would automatically reopen if there was an obstruction. This study has also found that 40% of in-person survey participants and 27% of online participants have experienced becoming trapped. 6% of in-person survey participants have experienced being trapped and dragged. None of these incidents were reported. This study shows that passenger mental models of train doors are confused and inconsistent. Passenger’s experiences do not correlate with their expectations of how the doors work, their mental models are incomplete, and this can lead to passengers taking risks when boarding or alighting. |
Taking a systems approach to designing national safety policy
| Document
|
Author Tracey A Herlihey, Lauren Mosley & Matthew Fogarty |
| Abstract Any policy developed in a siloed manner and presented for implementation in a straightforward way is limited in its application in complex systems such as healthcare. In this article we describe a process for developing new patient safety policy by taking a user-centred approach and applying a system-based framework. The Patient Safety Incident Response Framework published in 2022 (NHS England, 2022), represents a complete redesign of how the NHS responds to patient safety incidents for the purpose of learning and improvement. The Framework will replace the current Serious Incident Framework (NHS England, 2015). Testing and revision were a formal part of the development cycle. The final version incorporates findings from an early adopter programme and independent evaluation and used SEIPS as a framework specifying the structure of a patient safety incident response system. We found the framework to be a useful tool for informing the revision of PSIRF; however, translating this work into policy form proved difficult and some nuance and direct links to SEIPS may have been lost. |
H-FIT: Assessing the human factors impact of proposed changes to the railway
| Document
|
Author Nora Balfe |
| Abstract This paper describes the background, development, and content of a new tool, H-FIT, to assess the likely human factors impact of proposed railway change projects. The tool provides a structured approach to identifying the scope and requirements for human factors integration at early project stages, around which the human factors activities can be specified and planned. The core of the tool is 14 design scope factors which range from planned changes to the work environment, to the introduction of new HMIs, and changes in working hours. These design scope factors are linked to physical and organisational design outcomes, such as accessibility, usability, and fatigue. Human factors goals can be set for each area of design scope against the related design outcomes. |
A Quantitative Approach to Determining Inclusive Design Features Within UK Railway Depots
| Document
|
Author Kimberley Harding & David Hitchcock |
| Abstract Inclusive design is fast becoming a talking point for engineers, ergonomists, and designers. With holistic views on accessibility and inclusion in railways at the forefront of modern-day design and culture, it is a novel concept which could drastically change how we live and work. However, despite common interest in pursuing inclusive design strategies for railway passengers, there are significant gaps in inclusive thinking for railway workers – namely those who work in depots and trackside. Despite reforming modern system safety techniques, 2022 saw a significant increase in workforce injuries and little to no assistance in returning to work post-injury (Johnson, 2023). Limited accessibility and manoeuvrability in railway depots and trackside sees a workforce of injured staff unable to properly return to their original place of work and having to re-train in a less demanding sector. In response to this, this paper presents findings and methodologies for quantifying the inclusiveness of depot design from a user-centred approach. The data captured emulates how staff experience working in train maintenance and what aspects could be improved to encourage inclusivity in the workplace whilst enabling staff to work to their highest potential. It is hoped that this research could reduce bias in quantifying inclusive design elements in depots, framing a new scope for what is deemed ‘accessible’ or ‘non-accessible’ to make railway depots a better working environment for all. |
Using Systems Thinking to Identify Risks in Telephone Triage: MEAD Study Findings
| Document
|
Author Jill Poots, Jim Morgan, Matteo Curcuruto, Stephen Elliott, & Andrew Catto |
| Abstract This paper presents findings from a modified Macroergonomic Analysis and Design (MEAD) study aiming to identify system components and risks in a telephone triage system. Themes identified included: ‘accessibility and availability’; ‘risks on the part of the telephone triage professional’; ‘risks posed by callers’; and ‘barriers to safety incidents’. |
Human performance and automated operations: A regulatory perspective
| Document
|
Author Linn Iren Vestly Bergh, Kristian Solheim Teigen & Fredrik Dørum |
| Abstract The petroleum industry is becoming increasingly dependent on digital systems, and the companies have ambitious plans for increased use of digital technology – along the entire value chain. Increased levels of digitalisation present major opportunities for efficiency in the oil and gas industry and can also contribute to enhanced levels of resilience to major accident hazards. At the same time, new risks and uncertainties may be introduced. Based on developments in the industry and society in general, the Norwegian Petroleum Safety Authority (PSA) has in recent years pursued targeted knowledge development related to digitalisation and industrial cyber security. The PSA’s follow-up activities related to digitalisation initiatives in the industry have been based on input and experience from several knowledge development projects. In this paper we will give insight into the main regulatory strategies we have used to follow-up initiatives in the industry, present results from audits on automated drilling operations and discuss the results from the follow-up activities in light of current regulatory development. |
Women Are Not Small Men
| Document
|
Author Laird Evans, Robert Pringle & Eluned Lewis |
| Abstract Human Factors Engineers need accurate anthropometric data to design military equipment that is safe, comfortable and enables performance under extreme operational conditions and in the most severe environments. MOD acknowledges that its current anthropometry dataset is becoming increasingly unrepresentative of today’s Armed Forces personnel, particularly women and minority ethnic groups. To address this issue, MOD has launched a new, comprehensive anthropometry survey. This paper describes the requirements underpinning this survey and the planned solution. |
Design induced non-compliance: influences on pedestrian and cyclist behaviour at level crossings
| Document
Insert here |
Author Gemma J. M. Read, Nicole Liddell, Pia Sauer & Paul M. Salmon |
| Abstract Collisions at rail level crossings remain a pressing concern, with the influences on user behaviour a critical area of research. This paper reports the findings of an observational study of pedestrian and cyclist non-compliant behaviours at 10 rail level crossing sites in Australia. The findings illustrate the diversity in crossing designs and how these differences may influence behaviour. General recommendations are provided, alongside the need to consider context-specific risk controls. |
Lessons in the Metaverse: Teaching University students about Virtual Reality from within Virtual Reality
| Document
|
Author Gary Burnett & Catherine Harvey |
| Abstract There is considerable potential in the education sector for the use of immersive virtual worlds to enhance student learning and engagement. This paper outlines five key recommendations for teaching University students in future ‘Metaverse’ contexts. These guidelines are based on video/survey data collected from 266 students during a module that has run predominately in virtual reality for the last three years, as well as the reflective experiences of the teachers. |
Take me home, country road: Comparative optimism, mind-wandering in an automated simulator
| Document
|
Author Rachael A. Wynne, Angus McKerral, Sophie Withers & Kristen M. Pammer |
| Abstract Mind wandering while driving has been shown to factor in distracted driving, a critical cause of road crashes in Australia. With the implementation of autonomous vehicles onto the road network proposed as occurring in the near future, lies the potential for increased mind wandering, as cognitive engagement in driving is lessened. Part of the potential appeal of such vehicles is the ability to perform non-driving related tasks while in an automated driving mode. This study presents an analysis of drivers’ subjective experience of two prolonged drives in a driving simulator set to automated driving. Half of the participants were permitted to engage in non-driving tasks, to simulate potential future features. Participants provided summaries of their experience and preparedness to take-over control following two critical events. This study explores the themes of participants’ subjective experience and how this relates to mind-wandering, comparative optimism of driving behaviour, and readiness to respond to take-over events. |
Clinician perspectives around automating the Emergency Department triage process
| Document
|
Author Katherine L. Plant, Beverley Townsend, & OlTunde Ashaolu |
| Abstract Healthcare has arguably been the sector most impacted by the Covid-19 pandemic, leaving Emergency Department (ED) medical teams overworked and understaffed. An automated system for ED triage has been developed to help alleviate some of these pressures. Eight ED clinicians were interviewed to capture their views of the automated system. Insights were generated around where this system might add value and areas of challenge or concern. These findings will be used to refine the prototype for end-user testing and support the development of training material for clinicians. |
Building a New Hospital: the role of Human Factors
| Document
|
Author Lauren Morgan, David Higgins & Sue Deakin |
| Abstract The HF approach places all stakeholders at the heart of any project to identify their needs and ensure these are being met, ultimately to optimise efficiency and safety. With regards to building a new hospital, this includes not only patients’ needs, but also those of hospital staff, support workers, volunteers, and patients’ contacts. This paper discusses the approaches taken, and benefits realised |
Situation Awareness in Railway PICOPs
| Document
|
Author Abigail Palmer & Nora Balfe |
| Abstract Situation awareness in the railway is extremely important due to its dynamic nature which means that operators have to perceive, understand, and act on continuously changing information. This is true in engineering work, as well as operations. During engineering works, a PICOP (person in charge of possession) needs to keep track of multiple different factors, such as the number of RRVs/OTMs being used, number of worksites, points movements, and train movements, to ensure the safe running of the requiring high levels of situation awareness. Observations were undertaken to gain an understanding of the job as a whole in order to ultimately generate recommendations on how to better support situation awareness. This paper maps the PICOP role to Endsley’s three levels of situation awareness. Factors include track layouts, and the number of worksites |
User centred approach to developing a Concept of Operations
| Document
|
Author Kate Shield & Elaine Thompson |
| Abstract The paper describes a user centred approach to the development of a Concept of Operations and its application to the Synthetic Environment, which is a digital signalling design tool that will be used to develop seamlessly integrated scheme designs for Network Rail. |
Consideration of Stakeholders for Technology Acceptance in Marine Conservation
| Document
|
Author Dr Ella-Mae Hubbard, Melissa Schiele & Prof. Paul Lepper |
| Abstract This paper discusses the importance of understanding various stakeholder perspectives before investigating technology acceptance. Stakeholders are identified, from a systems perspective, with their key requirements and interactions. This is part of wider work towards developing an Augmented Technology Acceptance Model (for example acceptance of drone technology) within the context of marine conservation. |
Developing an Explainable AI Recommender System
| Document
|
Author Prabjot Kandola & Chris Baber |
| Abstract We used a theoretical framework of human-centred explainable artificial intelligence (XAI) as the basis for design of a recommender system. We evaluated the recommender through a user trial. Our primary measures were the degree to which users agreed with the recommendations and the degree to which user decisions changed following the interaction. We demonstrate that, interacting with the recommender system, resulted in users having a clearer understanding of the features that contribute to their decision (even if they did not always agree with the recommender system’s decision or change the decision). We argue that the design illustrates the XAI framework and supports the proposal that explanation involves a two-stage dialogue. |
Human-robot interaction: Assessing the ergonomics of tool handover
| Document
|
Author George V. Papadopoulos & Michail Maniadakis |
| Abstract This work focuses on human-robot collaboration for assembly tasks, examining the position of robot-to-human handover of objects. A simulation environment is implemented to ergonomically evaluate the expected posture of the human arm in hypothetical delivery positions in the 3D space. |
Behavioural intention of e-scooter use: A comparison of users and non-users
| Document
|
Author İbrahim Öztürk & Nazlı Akay |
| Abstract In the present study, the behavioural intention and factors affecting the use of e-scooters were examined among young people in Turkey. Previous users and non-users of e-scooters were compared in relation to the various variables. The relationships of attitudes, subjective norms, perceived behavioural control, and perceived usefulness were investigated regarding the intention to use e-scooters. While users reported positive socio-psychological factors related to e-scooter use, all different factors were positively related to behavioural intention. The results highlighted the importance of socio-psychological factors in predicting behavioural intention, as well as differences between e-scooter users and non-users in these factors. |
Forging Links between Safety Critical Task Analysis and Incident Investigation
| Document
|
Author James Bunn, Simon Dunford, Neil Hunter & Mary Marshall |
| Abstract Safety Critical Task Analysis (SCTA) is an established methodology in high-hazard industries for identifying and managing the human contribution to the risk of major accidents. It seeks to identify which tasks could contribute to the initiation or escalation of a major accident, and then identifies the steps within those tasks where additional controls may be required to mitigate against the effects of human failure. An incident investigation that incorporates examination of human and organisational factors would also be expected to examine where and how failures occurred, the efficacy of control measures in place and where remedial actions are required. This paper describes how the incident investigation process and investigation findings and the SCTA process can support each other and sets out a case for connection and improved alignment between them. This is a connection that has been underexplored to date, yet which has benefits for both risk management and human factors integration. |
Summoning the demon? Identifying risks in a future artificial general intelligence system
| Document
|
Author Paul M Salmon, Brandon King, Gemma J. M Read, Jason Thompson, Tony Carden, Chris Baber, Neville A Stanton & Scott McLean |
| Abstract There are concerns that Artificial General Intelligence (AGI) could pose an existential threat to humanity; however, as AGI does not yet exist it is difficult to prospectively identify risks and develop controls. In this article we describe the use of a many model systems Human Factors and Ergonomics (HFE) approach in which three methods were applied to identify risks in a future ‘envisioned world’ AGI-based uncrewed combat aerial vehicle (UCAV) system. The findings demonstrate that there are many potential risks, but that the most critical arise not due to poor performance, but when the AGI attempts to achieve goals at the expense of other system values, or when the AGI becomes ‘super-intelligent’, and humans can no longer manage it. |
Human factors approach to phlebotomy service review
| Document
|
Author Sharon Beza, Lauren Morgan, Andrea Granger, Joe McCloud 4 & Peter Jeffries |
| Abstract In acute hospital care, sampling a patient’s blood is frequently used to help guide diagnosis, or to understand a patient’s response to treatment. This means many patients will have their blood taken multiple times during an inpatient stay. The work of phlebotomists has been studied before, and acknowledgements made to how they adjust their practice to balance patient safety in the context of fluctuating demands and challenging work environments and equipment (Pickup et al., 2017). A human factors approach was used to analyse the in-patient phlebotomy service within a local National Health Service (NHS) Trust. Multiple systems related issues particularly at organisational level were identified. Recommendations were made on how to improve the safety and reliability of the process. |
Human performance in the rail freight yard
| Document
|
Author David Golightly, James Lonergan & David Ethell |
| Abstract Human performance in the rail freight yard has been identified as source of risk to rail freight operations. It is, however, an area that has received little research attention. Observations and expert elicitation explored work in the freight yard, leading to an understanding of freight yard activities, the impact of freight yard design and environment, and external pressures. Together, these factors make the freight yard a complex and challenging environment, where fluid, cognitive planning optimises physically demanding, cooperative processes. The implications for future management of rail freight operations are discussed. |
PRIME Road Markings for Motorcycle Casualty Reduction in Scotland
| Document
|
Author Prof Alex Stedmon, David McKenzie, Martin Langham, Kevin McKechnie, Richard Perry, Stuart Wlison, Morag Mackay, Stuart Geddes |
| Abstract This paper presents research of new PRIME road markings in which over 32,000 motorcyclists were manually counted and coded. Analyses indicated that these unique road markings produced statistically significant reductions in motorcycle speed, improved lateral lane position, and reduced braking across 22 sites in the West Highlands of Scotland. This work provides insights into the ‘Safe System’ approach to support safer motorcycling and casualty reduction. |
Usability evaluation: an investigation on combination of analytical and empirical methods
| Document
|
Author Setia Hermawati & Glyn Lawson |
| Abstract This study investigated the effect of combining analytical (heuristics and cognitive walkthrough) and empirical methods in usability evaluation. Data from two usability studies were used to simulate the outcomes of different combinations of usability evaluation methods. The findings show that the combined analytical methods significantly reduce the number of participants required in the empirical method without compromising the results of the usability evaluation. |
