Author
Katherine L. Plant, Richard Bye, Katie J. Parnell, Craig K. Allison, Jade Melendez, Neville A. Stanton Abstract
This work presents a collaboration between [an academic] and [industry partner] to help improve safety at level crossings by developing a deeper understanding of how people behave at them. Using the theoretical foundations of the Perceptual Cycle Model (Neisser, 1976) to generate behavioural insights from workshops, interviews and field observations, the work aims to create decision support tools for level crossing managers, engineers, safety teams and investigators. The resulting human factors toolkit will inform hazard analysis, system design and behavioural interventions that will put level crossing users—and their needs, goals and behaviours—at the centre of activities to improve system safety.

Author
Robert J. Houghton and Chris Wragg Abstract
Collaborative interpretation and understanding of complex and uncertain information is a pervasive and growing challenge across many industries and domains from defence and ‘blue light’ services to commerce and government. We carried out two studies to evaluate the Adaptive Report Generation Assistant (ARGA), a piece of collaborative software designed to aid team sensemaking by supporting coding of information inputs and visualisation of outputs. In the first study, ARGA was contrasted with pen and paper processes in laboratory trials and in a second, and more ecologically valid trial, ARGA was contrasted with the use of generic shared electronic documents by two larger teams of expert analysts. In both cases, in addition to usability analysis and evaluation of final report quality, team activity was also analysed with reference to recordings, post-hoc interviews and examination of the cognitive artefacts produced. It was found that by structuring input and interpretation phases of the activity and offering greater flexibility in the rework of both ontologies for input and visualisations of output, groups using ARGA generally produced better quality analyses through avoiding premature fixedness and confirmation bias. However, a persistent problem across all groups lay in maintaining consistent visibility of relative information quality and credibility. The findings imply that sensemaking quality can be enhanced by interventions that reduce the administrative and clerical demands of information management and representation.

Author
Shaun C. Lamb, Mohammad Naiseh, Jediah R. Clark, Sarvapali D. Ramchurn & Timothy J. Norman Abstract
We studied the experimental adoption of an image classifying tool as an organisation plans the adoption within its teams of intelligence analysts. We identified that existing models of expert decision-making and function allocation can be employed to inform the design and adoption of these tools.

Author
Genovefa KEFALIDOU, Robert HOUGHTON Abstract
Sensemaking describes the activity of seeking to understand complex situations and extracting meaning from diverse, and sometimes conflicting, information. We report results from a study of groups attempting to construct narratives from diverse forms of fabricated incident information (reflective of the ever increasing diversity available on the internet) deliberately designed to be ambiguous. We found that groups demonstrated a rich range of convergent and divergent behaviours and manipulations of paper-based stimuli. We conclude by providing discussion of how these insights can be used in the design of web-based collaborative sensemaking tools.

Author
Matthew Woodward, Julie Gore, Fotios Petropoulus & Christos Vasilakis Abstract
Decision-making for hospital patient flow management is a time-constrained task for a dynamic problem, but little is known about the cognitive strategies required for this type of task. The SkillsRules-Knowledge model of cognition was used to study the decision-making strategies of clinical coordinators and patient flow managers in acute medical units in two hospitals. For timeconstrained decisions in an environment with a plethora of dynamic data, a rule-based feedforward strategy was predominant. Additionally, decision makers applied their tacit knowledge of bed demand profiles to project the future situation and to compensate for delays that were inherent in the patient transfer process.

Author
Dhwani Oakley, Esme Flegg, Sam Hodder, Katrina Kemp & Ruth Taylor Abstract
This paper presents a novel methodological approach to safety regulation for the maritime sector. It describes the use of ‘serious gaming’ to explore the impact and support the implementation of emerging and future technologies in the policy-making process. The development and application of the ‘serious gaming’ framework is described and demonstrates the potential benefits of applying a design-led approach to policy development in enabling innovation, contributing to regulatory change, and delivering improved outcomes for seafarers, industry, and the wider society.

Author
Steph Simpson Abstract
This paper presents a series of observations of the hazards and challenges faced as part of the cleanup of the UK’s Nuclear Licensed Sites after decades of electricity production. In some hazardous environments, the dynamic decision making of humans is often preferable to the use of robots, however this does not come without risk. Protective equipment, whilst absolutely necessary to reduce the risks to the operator, can impact their performance when undertaking decommissioning operations. Human Factors and Ergonomics play a critical role in ensuring these tasks are undertaken safely, reliably and efficiently.

Author
Marta Weronika Wronikowska, Verity Westgate, Jody Ede, James Malycha, Lauren Morgan & Peter Watkinson Abstract
Referral to an Intensive Care Unit (ICU) is a complex medical process. The decision making involved can be cognitively challenging and subjective. We aimed to identify variables used by clinicians to make decisions during the ICU referral process, define the requirements for cognitive decision making and to detect commonly repeated errors. Applied Cognitive Task Analysis (ACTA) interviews were carried out with 17 doctors and nurses of varying specialties and levels of seniority to create a high-level task analysis of the participant’s role in the ICU referral process. Interviews were audio recorded, transcribed and analysed by two researchers in NVivo 11 software. We identified 188 variables used for clinical decision-making during an ICU referral. Removal of duplicates created 30 discrete variables. We found that there was not one key variable or piece of information that was significant to clinicians. Instead a ‘big picture’ approach was described, where all the data about a single patient was assembled and cognitively processed. ‘Often missed’ factors in the referral process were also identified. The most common was failure to consult family to discern patient wishes. The 30 variables used in the ICU referral process will inform the development of an interface for the Hospital Alerting Via Electronic Noticeboard project. This aims to identify patients at risk of deterioration in hospitals. Patient wishes were often neglected during the process and mechanisms to address this will form part of future work. We propose the addition of ‘F’ for ‘functional status/family’ to the ‘ABCDE’ acronym that is commonly used to evaluate a patient’s condition.

Author
Chris Baber, Emily McCormick & Ian Apperley Abstract
The issue of ‘explanation’ has become prominent in automated decision aiding, particularly when those aids rely on Artificial Intelligence (AI). In this paper, we propose a formal framework of ‘explanation’ which allows us to define different types of explanation. We provide a use-cases to illustrate how explanation can differ, both in human-human and human-agent interactions. At the heart of our framework is the notion that explanation involves common ground in which two parties are able to align the features to which they attend and the type of relevance that they apply to these features. Managing alignment of features is, for the most part, relatively easy and, in human-human explanation, people might begin an explanation by itemizing the features they are using (and people typically only mention one or two features). However, providing features without an indication of Relevance is unlikely to provide a satisfactory. This implies that explanations that only present features (or Clusters of features) are incomplete. However, most Explainable AI provides output only at the level of Features or Clusters. From this, the user has to infer Relevance by making assumptions as to the beliefs that could have led to that output. But, as the reasoning applied by the human is likely to differ from that of the AI system, such inference is not guaranteed to be an accurate reflection of how the AI system reached its decision. To this end, more work is required to allow interactive explanation to be developed (so that the human is able to define and test the inferences and compare these with the AI system’s reasoning).

Author
Nick Woodier, Paul Davis, Laura Pickup, Kathryn Whitehill & Robert Hutton Abstract
Applied Cognitive Task Analysis is an appropriate method to investigate challenging cognitive tasks and the role of expertise in healthcare contexts. Healthcare needs to support the accelerated development of decision-making skills in its novices and also create the optimum conditions in which to make decisions.

Author
Adrian Wheatley, George Charalambous Abstract
This paper summarises the work undertaken to produce an Allocation of Function method in support of a number of client projects in the nuclear sector. This led to the development of an interactive Excel based tool to support the client in Allocation of Function decision making.

Author
Victoria A. Banks, Katherine L. Plant and Neville A. Stanton Abstract
When designing a new pilot decision aid for the flight deck, it is important to understand ‘how’ pilots make decisions in abnormal operating scenarios so that we can ensure they are provided with appropriate support. This paper provides a decision ladder analysis of an aircraft engine oil leak using data collected from six commercial airline pilot interviews. Traditionally, decision-making models are used reactively as a means to explore why things go wrong. However, we explore whether these models can also be used prospectively. Our analysis yields a number of possible design implications for the design of a pilot decision aid on the flight deck.

Author
Diane POMEROY, Justin FIDOCK, Luke THIELE and Laura CARTER Abstract
The Australian Army recognises personnel need a “cognitive edge” over any adversary. To better understand cognitive performance of military personnel in current and future land operating environments and inform training requirements, we have created an immersive tactical team simulator representing possible elements of the future operating environment, including novel use of technologies by adversaries. The most recent study analysed behaviours of two military teams, each consisting of a three vehicle platoon. Examination of individual and team strategies identified the decision making approaches adopted by the individual teams in response to novel and unexpected threats in a high tempo situation.

Author
Ilinca-Ruxandra Tone & Dr. Amy Irwin Abstract
Livestock operations pose a high risk of injury and fatality in agriculture, especially for lone workers. In other high-risk industries, non-technical skills (NTS) are recognised as important for safe and effective task performance. However, dedicated research ought to be conducted to investigate how these findings apply to farmers, who are suggested to be highly risk tolerant. The current study used the vignette method to investigate farmer risk perception and risk management strategies, including NTS, in four types of cattle-handling risks related to self, equipment, environment, and animal characteristics. A preliminary sample of 50 farmers from the UK and Ireland was recruited through farming forums and organisational contacts to take part in an online qualitative study. Participants were presented with eight scenarios, two per category of risk, and asked to report their reasoning for proceeding or not and to detail any risk management strategies used. Thematic analysis was used to identify patterns. Farmers appeared to evaluate risk in the light of animal welfare and duty. Scenarios concerning faulty equipment and animal characteristics were perceived as too dangerous. Farmers reported using NTS such as task management, situation awareness, and decision-making to reduce risk. Farmers also considered facilities important for safe completion of livestock operations. These findings suggest that future interventions should aim to frame risk based on farmer priorities and to formally raise awareness about the importance of NTS.

Author
Weixuan Li, Glyn Lawson, Setia Hermawati & Kyle Harrington Abstract
This study investigates the application of augmented reality to enhance rescue efficiency, alleviate workload, and improve decision-making in earthquake rescue. The findings demonstrate that AR-based solutions surpass conventional approaches in task completion duration, map utilisation, and decision-making simplicity, underscoring AR's capacity to improve human factors issues in disaster contexts. Future studies should include professional rescue teams as well as more complex simulations to test AR's effectiveness in real-world emergency situations.

Author
Louise Roe Abstract
Local rationality describes how people make decisions, based on what made sense to them at the time (Eurocontrol, 2014). While the importance of understanding local rationality in safety investigations is acknowledged, there is little to support safety investigators with how to effectively gain this knowledge when meeting with staff involved. A local rationality question tool was developed to help healthcare safety investigators understand why an action or inaction made sense at the time, without staff feeling blamed or interrogated.

Author
Hebah Bubakr and Chris Baber Abstract
Sensemaking can involve selecting an appropriate frame to explain a given set of data. The selection of the frame (and the definition of its appropriateness) can depend on the prior experience of the sensemaker as much as on the availability of data. Moreover, artificial intelligence and machine learning systems are dependent on knowledge elicited from human experts, yet, if we trained these systems to perform and think in the same way as a human, most of the tools will be unacceptable to be used as criterion because people consider many personal parameters that a machine should not use. In this paper, we consider how an artificial intelligence system that can be used to filter curriculum vitae (or résumés) might apply frames that result in socially unacceptable decisions.

Author
Nora BALFE, Robert J. HOUGHTON, Jockman CHEUNG and Sarah SHARPLES Abstract
Two simulation methods used to study signaller decision-making strategies are described and compared through experimentation. The first method was a ‘static scenario’ method in which expert participants were presented with printouts from a simulated scenario and asked to give their strategy for routing the trains in the area. The second was a more traditional dynamic simulation undertaken on the same high fidelity simulator. The first method allowed greater control over the experiment and presents interesting opportunities for collecting additional qualitative data from participants, but the second method was more realistic and featured improved participant performance.

Author
Chris Baber and Chris Vance Abstract
A simulated air defence task is used to explore the impact of decision support on operator performance under different levels of task complexity. In this simulation, the status of autonomous air vehicles (drones) is indicated by their colour and by their threat level. Threat level is indicated by a polygon display which is automatically updated when drones fly into areas of attack or areas of risk. From two experiments, we can draw some tentative conclusions on the strategies that participants employ and the role that decision support might play in supporting or thwarting these. Contrary to instructions, participants did not always respond to the cue from the polygon display to engage (particularly when the number of drones was high, which resulted in more dynamic changes to the polygon display). Discussion with participants after the experiments suggested that some of them tried to ‘read’ the changes in polygon display in terms of possible paths that the drones were taking. From this, they might have attempted to anticipate when to respond and rely on their anticipation rather than the decision support. Situation awareness was rated lower when participants monitored two types of drone, and this was sufficient to lead to them performing at levels not far from chance. This is concerning in that it suggests that the decision support (in the polygon display) was not regarded as part of the situation awareness that the participants were using. It also raises the possibility of a difference between awareness of the situation and the awareness of decision options.

Author
Andreas REITER, Xianxu HOU, Genovefa KEFALIDOU, James GOULDING Abstract
Personal data is everywhere. Its complexity grows exponentially as more devices generate data. Understanding and making sense of complex data is fundamental as critical decisions may depend on its interpretation. In this lab-based observation study both novices and experts were exposed to complex medical information. The findings suggest that medical professionals employ different strategies from non-medics during sense-making and task completion. We discuss implications for designing new decision-making tools that support sense-making complex big data.

Author
Setia Hermawati, Lena Cibulski and Glyn Lawson Abstract
Visualisations provide significant support for effective reasoning and decision-making processes. Its value mainly lies in its ability to turn raw data into actionable insights that lead to a decision. This requires appropriate visual representations that are designed with the decision-maker's way of reasoning in mind. Understanding the cognitive aspects underlying decision-making with visualisations is therefore crucial. Cognitive task analysis methods have been used to elicit expert knowledge in a variety of decision-making scenarios, with the Critical Decision Method (CDM) focusing on the cognitive bases in naturalistic non-routine incidents. In this study, we aim to determine the feasibility of CDM for capturing the expert knowledge, strategies, and cues involved with visualisation-based decision-making processes. Based on an analysis of four semi-structured interviews, we evaluate the method’s potential to inform the role of visualisation for human decision-making. We anticipate that our reflections on methodological insights can serve as a starting point for other human factors and visualisation researchers, who aim at studying strategies for higher-level decision-making and problem-solving tasks.

Author
Teemu Suokko & Arto Reiman Abstract
Manufacturing process development is often technology and business oriented. The development of human work in this context is often neglected to a certain extent. With the use of ergonomics tools and methods, it is possible to identify targets for development and justify development investments also from the human work improvement perspective. For that purpose, physical ergonomics data were collected through video observations. In addition, occupational health and safety (OHS) and productivity indicator data were collected. In a series of group interviews, participants representing management and designers discussed the strengths and weaknesses of these data when it comes to decision-making on manufacturing development and related investments. As an outcome, the company decided to invest in production automation in the welding unit to avoid ergonomic problems and to increase productivity.

Author
Kate Kingshott & Anjum Naweed Abstract
Maintenance engineers of aircraft in General Aviation work in a highly time pressured, complex and dynamic environment where errors in decision making could have far reaching consequences. However, few studies have investigated the role and influence of affect on the decision-making process in this setting. Using interviews and a scenario invention method, this study investigated the affective influences in decision making and corresponding responses in General Aviation maintenance engineering work in the Australian context. Preliminary findings based on inductive analysis identified a number of themes including affective response to task interruption specific to work colleagues and customers or management personnel, impact of negative rumination and the role of pride as a safety factor. Findings are discussed in terms of the impact of different affective states with implications for future research directions on crew resource training and non-technical skills development.

Author
Rob HUTTON, Hannah BLACKFORD, Kevin BENNETT, Nigel JONES, and Ade FISHER Abstract
Military commanders are increasingly required to understand more than just the physical terrain. Understanding activities in cyberspace and their impact on operations presents a number of challenges for military personnel, tech-savvy or not. This paper presents a cognitive systems engineering approach to providing visualization solutions to support commander decision making. An Ecological Interface Design (EID) approach was used. Challenges for supporting cyber situational awareness are described.

Author
Suzy Broadbent, Jennifer McVay Abstract
The In the Moment (ITM) Program considers high stakes decision making when there is no “correct” answer, and how Artificial Intelligence (AI) algorithms can be aligned to individuals’ decision-making criteria to improve trust and likelihood of delegation.  Work carried out by in the US identified a number of “Key Decision-Making Attributes” (KDMAs) that may affect an individual’s decision making in the context of Mass Casualty Triage Scenarios. Following successful trials in the US, partnered with in the UK to extend the trial within the UK.  Medically trained participants were assessed on their KDMAs during online text scenarios and Virtual Reality scenarios. They were then asked to review decisions made by “another medic” in similar scenarios, rate them on trust, and decide whether or not they would delegate to that particular medic. These medics were in fact Artificial Intelligence algorithms that were either aligned or deliberately misaligned to the participants’ KDMAs.  This research demonstrates how AI can be tuned to represent human influences beyond competence and support trusted decision making in complex scenarios and investigates differences in decision making between US and UK participants.

Author
David Gibson, Dominic Furniss & Helen Vosper Abstract
The role of the pharmacist is changing, moving from a product focus, centred on the medicine, to a model of delivering person-centred care through the safe and effective use of medicines. This requires the development of enhanced clinical skills. It is recognised that there are significant gaps in current educational programmes, leaving novice pharmacists feeling unprepared for their transition to practice. This situation has been exacerbated by the current Covid-19 pandemic. Of the enhanced clinical skills, one of the most difficult to teach is decision making: often complex and high stakes, it is recognised as one of the hallmarks of the expert practitioner. Despite the importance of this skill in underpinning safe and effective practice, relatively little is known about how experts make such decisions, and there is little support for novices. This case study describes the development of a reflective tool, informed by naturalistic decision making and based on the aviation model of Threat and Error Management. This encourages systems thinking to help novice pharmacists cope with the complexities of decisions relating to real life patient-centred care.