Tools & techniques
Allocation of Function in the era of Artificial Intelligence: a 60-year old paradigm challenged
| Document
|
Author Nick Gkikas |
| Abstract The Fathers of the discipline of Ergonomics and Human Factors used their scientific research and real-life experiences of technological development during WWII and the first years of peace that followed to propose a set of principles for Human-Machine Interaction (HMI). These principles stood the test of time and informed common applications of the discipline, such as allocation of function between human and machine for many years. It is only recently with the advancement and generalisation of certain underlying technologies that forms of Artificial Intelligence (AI), machines and systems with non-deterministic behavioural characteristics became operational. The underlying specification of those machines and systems appear to challenge some of the underlying assumptions made by the Fathers of the discipline. The present article revisits those principles of HMI, identifies the changes in the underlying assumptions and discusses the implications of the changes identified to the discipline of Ergonomics and Human Factors. |
Net-HARMS, AcciNet and SafetyNet: A new safety management toolkit for complex systems
| Document
|
Author Paul M Salmon, Neville A Stanton, Guy H Walker, Patrick Waterson & Adam Hulme |
| Abstract Risk assessment and accident analysis methods based on systems thinking are currently popular, but few can be used together in an integrated manner. This article describes and demonstrates the Systems Thinking Accident and Risk Toolkit (START) which comprises the Networked Hazard Analysis and Risk Management System (Net-HARMS) risk assessment method, the Accident Network (AcciNet) accident analysis method, and the Safety Network (SafetyNet) intervention evaluation method. The three methods were designed to be used in an integrated manner as part of organisational safety management activities. START is described and demonstrated via a case study focussed on autonomous vehicles. The findings highlight the benefits of integrating risk assessment and accident analysis activities, including how accident data can be used to strengthen risk assessment outputs, and how the efficacy of specific risk controls can be considered in accident analysis efforts. Practical guidance on using the methods is offered, as well as recommendations for future research and applications in practice. |
Equality, Diversity and Inclusion: The Forgotten Human Factors
| Document
|
Author Ryan Gamble, Phoebe May Asquith1, David Greeno, Jessica Evans & Phil Morgan |
| Abstract This short paper presents a collection of scenario-based case studies, illustrating how matters of equality, diversity and inclusion (EDI) may operate as human factors. This paper aims to highlight that by placing EDI at the heart of human factors research and practice, future work can ensure a more informed and inclusive practice that avoids reinforcing systemic bias and injustice. |
EEG-based Mental Workload Level Estimation using Deep Learning Models
| Document
|
Author Kunjira Kingphai & Yashar Moshfeghi |
| Abstract Deep learning-based approaches have recently received much attention and managed to accurately capture variance characteristics in the Electroencephalography (EEG) signals. In this paper, we aim to classify the subject’s mental workload (MWL) level from EEG signal by using deep learning models named Stacked Gated Recurrent Unit (GRU), Bidirectional GRU (BGRU), BGRU-GRU, Stacked Long-Short Term Memory (LSTM), Bidirectional LSTM (BLSTM), BLSTM-LSTM and Convolutional Neural Network (CNN). The classification was performed on a publicly available mental workload dataset, STEW. Our encouraging results show the potential of deep learning models for MWL level detection. |
Tool for Estimating Rail Freight Yard Complexity and impact on human performance
| Document
|
Author David Golightly , David Ethell & James Lonergan |
| Abstract Previous research has identified that the complexity of freight yards – layout, capacity, access and local physical context – is one factor that impacts the safe performance of freight yard tasks. We present the rationale, method and contents of a tool that makes explicit the definition of site complexity, and elements that comprise site complexity. Applications of the tool include site risk assessment, impact assessment of operational change, and potential to underpin a design standard. |
Novel mental workload scale application using fuzzy sets theory
| Document
|
Author Hanna J. Neroj & Jim Nixon |
| Abstract A new scale measuring pilot subjective mental workload was developed as part of the Horizon 2020, Future Sky Safety Project: The validity of new measures needs to be established for effective and reliable implementation. This article investigates the validity of this new scale through assessment of psychological distances between linguistic variables on the scale and subsequent transformation from discrete to continuous scale using fuzzy sets theory. Although the new scale has claimed ordinal properties for the items, no evidence exists to support the interval properties of the scale and any subsequent analysis using statistical methods. The results in this article show linear progression of the items on the scale, supporting the order of the items. To establish the interval properties of the scale, the distances between the ordered items were evaluated using fuzzy sets theory using the Fuzzy Logic Designer in MATLAB. Transformation to continuous scale using fuzzy sets may allow the capture of the dispersion of the data around specific points, showing the degree to which each score belongs to different linguistic category of mental workload. Finally, a proposal for incorporating the validated performance measure was presented. |
Human-robot interaction: Assessing the ergonomics of tool handover
| Document
|
Author George V. Papadopoulos & Michail Maniadakis |
| Abstract This work focuses on human-robot collaboration for assembly tasks, examining the position of robot-to-human handover of objects. A simulation environment is implemented to ergonomically evaluate the expected posture of the human arm in hypothetical delivery positions in the 3D space. |
Social network analysis in submarine command and control
| Document
|
Author Neville A. Stanton and Aaron P. J. Roberts |
| Abstract This is a world’s first-of-a-kind study that compares three operational scenarios in a simulated submarine control room: Returning to Periscope Depth (RTPD), Inshore Operations (INSO) and Dived Tracking of Contact (DT). The Event Analysis of Systematic Teamwork (EAST) method was used to model the social networks. 10 teams were recruited for the study. Results indicate that, across all scenarios, the Operations Officer (OPSO) and Sonar Controller (SOC) are particularly loaded, with communication between these operators being revealed as a potential bottleneck. The type of operation being performed affected the type of information used significantly, with a higher reliance on sonar information (and the sonar operators) during a RTPD and a higher reliance on visual information (and the periscope operator) during INSO. Implications are discussed alongside suggestions for future work. |
Critical Assessment of the usability of a New Modular Ward
| Document
|
Author Jonathan M McCloud & Lauren Morgan |
| Abstract An acute District General Trust commissioned a new modular building to house a surgical ward. The process of commissioning and design of the building was in accordance with standard trust practice and established building regulations for hospital buildings. The purpose of this project was to assess the ward layout in terms of safety and efficiency and offer mitigations for any problems found. Observation of the ward layout demonstrated significant problems with sight lines in to bays, layout and design of toileting facilities, and layout and positioning of key ward areas including kitchen facilities, office space, drug and fluids stores, and nursing stations. To further examine the ward layout, Hierarchical Task Analysis and link analysis were performed to review two tasks: one an emergency task (treatment of a deteriorating patient) and a routine common task (serving meals). We also considered the Systems Engineering Initiative in Patient Safety (SEIPS) framework to ensure that our analysis was as thorough as possible. This work demonstrated significant problems with the ward layout, in terms of sight lines, workspace, communication, and contributed to significant extra time doing both emergency and routine duties. This led to a series of recommendations to mitigate on the ward, as well as a recommendation that appropriate staff involvement is included at appropriate times in planning future new build and refurbishment projects. |
Guidance on customising Bowtie Analysis for use in healthcare
| Document
|
Author Ronald W. McLeod and Paul Bowie |
| Abstract Based on the CIEHF white paper ‘Human factors in barrier management’, NHS Education for Scotland (NES) has been exploring the potential application of Bowtie Analysis (BTA) in healthcare. Both an initial workshop-based study in a primary care context, as well as feedback from training and a series of case studies conducted across primary and secondary care and supporting health functions, suggested BTA has significant potential as an approach to identifying and managing risk in healthcare. It seems realistic to expect that existing healthcare professionals should be able to conduct BTAs to a reasonable quality standard making use of an NES Guide guidance document, together with a relatively small amount of training and support. |
Breaking bad systems: using work domain analysis to identify strategies for disrupting terrorist cells
| Document
|
Author Paul M. Salmon, Tony Carden & Nicholas J. Stevens |
| Abstract Terrorism represents a major global issue. Despite decades of research, interventions designed to prevent or disrupt terrorist activities are failing to adequately control the problem. This paper argues that a human factors systems-thinking approach may support the identification of novel, holistic and impactful interventions. To demonstrate, a systems analysis of a generic Islamic State terrorist cell, created using the work domain analysis phase of cognitive work analysis, is presented. The analysis is subsequently used to identify opportunities for disrupting terrorist cells and their activities. In addition to well-known and already applied interventions, the analysis identified a series of other opportunities for disrupting terrorist cell activities. Examples discussed include strategies designed to disrupt propaganda activities and the planning of terrorist attacks. The analysis also showcases the potential use of systems ergonomics for disrupting, as well as optimising, sociotechnical systems. The implications for ergonomics generally as well as counter-terrorism activities are discussed. |
Scoping ergonomics information with User System Architectures to meet HSE’s COMAH requirements
| Document
|
Author Mike TAINSH |
| Abstract The Health and Safety Executive (UK) apply the COntrol of Major Accident Hazards (COMAH) regulations during inspections of designated sites. They pay attention to the ergonomics issues associated with the organisation, the jobs and individual characteristics including competency. The organisation needs to scope the ergonomic information, and integrate it appropriately prior to assessment. A User System Architecture (USA) was used to scope and contain all ergonomics information. This supported an integrated understanding of the ergonomics issues, and traceability. |
Accident Analysis Models used in Practice: Comparison of MAIB and NSIA models
| Document
|
Author Abigail Palmer, Gyuchan Thomas Jun & Will Tutton |
| Abstract The aim of this study is to compare two accident analysis models: Casualty Analysis Diagram (CAD) used by Marine Accident Investigation Branch (MAIB) and a model developed/used by Norwegian Safety Investigation Authority (NSIA). The study explored the similarities and differences between the two models when applied to a real marine accident involving a collision of two vessels. The views of investigators working at MAIB and NSIA were collected using a questionnaire and interviews. This paper discusses various issues on usability of these models and provides insights into effective use of accident analysis models in practice. |
Using digital games to boost HF learning
| Document
|
Author Johnny Mitchell & Lorraine Braben |
| Abstract This paper describes how an online game (‘Control Panel’) was used as part of Human Factors (HF) training to effectively introduce key concepts and engage participants. The game process will be explained and the results compared with available research. Finally, there will be some reflections from a practitioner on the impact the game has made in delivering HF training events. |
Automation and Cyber Security Risks on the Railways – the Human Factors implications
| Document
|
Author Eylem Thron & Shamal Faily |
| Abstract Automation improves rail passenger experience but may reduce cyber resilience because it fails to adequately account for human factors. Preliminary results from a study on signallers and automation confirms this, but judicious use of modelling tools may ensure design for automation considers this. |
Analysing two serious incidents in clinical research from a systems theory perspective
| Document
|
Author Anthony Vacher, Yannick Bardie, Monica Pollina, Myra Daridan and Brian Edwards, for the Safety Analysis Team |
| Abstract Effectively ensuring the safety of volunteers that participate in clinical trials involving healthcare products (drugs, medical devices) is a subject of debate in clinical research. This is especially so when healthy volunteers choose to enter Phase I (first-in-human) trials where any serious incident is unacceptable and undermines the confidence in the whole healthcare industry. As in other industries, safety management of clinical trials rely mainly on a traditional view that aims to avoid serious incidents by the identification of hazards, the development of safety barriers (technological barriers, procedures, regulation, laws) to prevent and mitigate risks, and the strict compliance of operators with these safety barriers. This traditional view of safety management is recognised as no longer sufficient to maintain safety in a dynamic, complex, and competitive environment where changes and perturbations are permanent, and the pace of technological innovations is high. In that respect, a group of individual pharmacology and clinical professionals have argued for the need to introduce the principles and methods from human factors and systems theory into the process of safety investigations following serious incidents occurring in clinical research. In this perspective, an international, interdisciplinary and multi-stakeholder collaboration was established to explore the feasibility to transpose human factors and systems theory methods to the specific context of investigation of serious incidents that occurred during phase I-trials. The Causal Analysis using System Theory method was applied to two emblematic serious incidents, one in London (United Kingdom) in 2006 and one in Rennes (France) in 2016. These two serious incidents have benefited from extensive investigations by both stakeholders and authorities afterwards to identify their root-causes and propose remedial actions to avoid their recurrence. |
Applying the ‘many models’ approach to the successful introduction of advanced automated vehicles
| Document
|
Author Gemma J. M. Read, Paul M. Salmon, Alison O’Brien and Neville A. Stanton |
| Abstract There have been previous calls for a ‘many model’ approach to solving complex challenges in ergonomics (Salmon and Read, 2018). In this work, we demonstrate how the combination of three systems ergonomics methods can provide different, yet compatible insights into the emerging challenge of the successful (safe and effective) introduction of advanced automated vehicles. Specifically, we applied ActorMap, Hierarchical Task Analysis, and the abstraction hierarchy from the Cognitive Work Analysis framework. Each model provides detailed descriptions of the road transport system that can be used as a basis to explore the issues surrounding the introduction of automated vehicles. Importantly, they provide a series of compatible insights from different perspectives regarding the future road transport system that can inform policy response and other initiatives to support the successful introduction of advanced automated vehicles. It is concluded that the application of multiple methods is beneficial to gain more insight than would be available from a single method used in isolation. |
Design of human-machine teams using a modified CoActive Design Method
| Document
|
Author Professor Chris Baber, Chris Vance |
| Abstract Designing Human-Machine Teams not only requires an appreciation of which functions might be appropriately allocated to human or machine, but also how each team member can make sense of the functions performed by it and its team-mates. The aim of this paper is to present an approach to Allocation of Function within Human-Machine Teams (HuMaT) that can be applied across different Levels of Automation and which can explore information management issues in such teams. To do this, we present a modification of the CoActive Design method. A key aspect of the modification lies in the focus on information exchange and issues relating to common ground in HuMaT. In this paper, Cognitive Work Analysis is used as the basis for the CoActive Design Method to explore how different Levels of Automation can be conceptualised. The benefit of such an approach is that is provides a decomposition of functions such that it is possible to see how, even in systems that have high-levels of autonomy, there remains a role for human operators. Taking the example of an in-car navigation system, we illustrate how each member uses information to support the functions allocated to them, and how common ground develops in the team. |
Measuring Field of Vision Using Digital Human Modelling Techniques
| Document
|
Author John Lovegrove |
| Abstract This paper will provide the methodology used to evaluate the field of vision for an entire target population whilst only working in a digital environment using digital human modelling techniques. The context is an evaluation of a small, powered craft at sea, operating at different running trims. The maritime regulations stipulate that the pilot of the boat must be able to see the surface of the water at one boat length. The standard BS EN ISO 11591:2019 ‘Small craft – Field of Vision from the steering position’ provides the practitioner with the target conditions in the horizontal and vertical fields but very little information on the evaluation of the different boats that exist. This paper will take the reader through the entire process for measuring the Field of Vision for any small craft (i.e. less than 24 metres in length), at a range of running trims, using digital human modelling and computer aided drawing techniques. |
Effects of safety gloves used by gardeners on perceived discomfort and performance
 |
Author Payam Khanlari, Fakhradin Ghasemi, Rashid HeidariMoghadam |
| Abstract A shovel is a common tool in agricultural activities. It is very popular among Iranian gardeners and they used it for a variety of purposes. To avoid such damages, some gardeners wear safety gloves. Some other gardeners do not use safety gloves because they maintain that the gloves negatively affect their performance. Accordingly, the aim of the present study was to investigate and compare several safety gloves used by gardeners in terms of comfort and performance. Ten gardeners with at least two years of experience were invited to participate in this study. The participants were asked to plow the ground with two commonly-used gloves and also bare hands for 30 minutes. After completing the task, they were given a hand and fingers map to express their discomfort level in each region. The performance of the participants was determined by measuring the surface area plowed by them. Area P was the one with the highest level of perceived discomfort, followed by TP, MM, and IM. In all areas, the lowest level of discomfort was perceived when the participants used the cotton glove. The average area plowed by participants with bare hands, cotton gloves, and Latex gloves were 1266cm2 (±112.7), 1230cm2 (±80.4), and 1186cm2 (±138.6), respectively. Therefore, wearing any type of safety gloves can negatively affect the performance of gardeners. Safety gloves used by gardeners were different in terms of the perceived discomfort and performance. |
Beyond user-centred design. Crowdsourcing with Serious Games for Design.
| Document
|
Author Edward Oates |
| Abstract Designers rely on direct access to ‘users’ (those who will use the product) to assist in their design process. User-centred design strongly emphasises the full involvement of users in the design process; but what happens when they aren’t available? This study investigates the extent to which Serious Games may offer an asynchronous remote alternative to ‘face-to-face’ design processes through Crowdsourcing. A design process completes with summative usability testing of the product. Again, a lack of access to users is a serious limitation and one that may be ameliorated by remote unmoderated usability testing. The extent to which Serious Games may be the vehicle for remote usability testing is also explored in this research. Results from the Crowdsourcing activity show, from contributed design ideas, that a Serious Game may provide a credible tool for Crowdsourced Design. Remote unmoderated testing has known limitations and the use of Serious Games provides some mitigation, with careful implementation being required. This is a mid-study report on UK Ministry of Defence sponsored research under the Royal Navy ‘DARE Innovation’ initiative. |
Applying barrier analysis to overspeed events in rail operations
| Document
|
Author Nora Balfe, Ken Byrne & Ron McLeod |
| Abstract Irish Rail has applied the Bowtie Analysis method and followed the CIEHF White Paper guidance on Barrier Management to analyse the risk from overspeed events on the national rail network. The analysis was performed in the early days of the COVID19 restrictions, requiring a virtual approach to the analysis, rather than the normal face-to-face workshop format. This paper briefly describes the process followed and the results obtained, focusing on the learnings and potential improvements to the process, rather than the technical details of the analysis. These include: at what point in the bowtie development process to include frontline expertise; prioritising threats and consequences identified so that most effort can be focussed on the most important controls; the complexity of degradation factors relating to human performance; and ensuring presentation of the results matches the expectations of the stakeholder community. |
Measuring Professional Wellbeing in Healthcare
| Document
|
Author Giulia Miles, Eva-Maria Carman,Upasana Topiwala, Benjamin Warren, Sasha Blackwood & Steve Cantellow |
| Abstract Wellbeing of healthcare staff has been highlighted as a key issue across clinical professions and a focus on professional wellbeing allows us to identify and better understand the system performance shaping factors that affect individual clinicians and ultimately patient care. This paper describes the method of selecting and applying a validated wellbeing tool in a large acute NHS hospital trust as part of a wider project looking at system monitoring tools. |
Healthcare Investigation Fatigue Trigger Tool
| Document
|
Author Laura Pickup, Saskia Fursland, Mairi Alexander, Suzy Broadbent, Paul Davis, Kathryn Whitehill & Sian Blanchard |
| Abstract Fatigue and the implications on human performance are well recognised in the literature and many safety critical industries. This is not currently the case in healthcare, where there are no formal approaches to monitor or investigate the impact of fatigue on clinical performance and patient safety. The Healthcare Safety Investigation Branch (HSIB) was set up in 2017 and its core function is to provide independent investigations of healthcare incidents. HSIB adopts a systems approach to investigations and this paper will describe the work in progress to develop a standardised approach to consider fatigue. |
Validating ergonomics methods
| Document
|
Author Neville A Stanton |
| Abstract This paper revisits the challenge laid down over 15 years ago, that Ergonomics needs to report on the reliability and validity of its methods if it is to maintain its standing amongst the Engineering community. Unfortunately, a review of books reporting on Ergonomics methods since that time shows very little change. The theoretical constructs of reliability and validity are presented together with the rationale for conducting studies of training in Ergonomics methods. Revisiting the original study shows the way in which such validation work can be undertaken and data reported. It is hoped that this article provides the stimulus for more studies of this nature. |
Human factors: emergency department suspected heart attack process
| Document
|
Author Frances Ives & Deborah Jackson |
| Abstract Early recognition of a heart attack is essential as delayed treatment can result in death. A Human Factors (HF) review of the process for treating suspected heart attack patients in a busy inner-city Emergency Department (ED) was completed to help decrease risk, improve patient safety, care and patient and staff experience. To provide timely treatment for a heart attack, a diagnostic electrocardiogram (ECG) must be performed within 15 minutes of arrival at ED. However, concerns had been raised by staff relating to patient experience and delayed ECG, due to the number of process steps the patient had to complete from arrival at ED reception to the ECG being taken. Observations of the process were carried out along with multidisciplinary staff focus groups to understand the patient journey. A Hierarchical Task Analysis (HTA) and Failure Modes Effects Analysis (FMEA) were also completed to identify process failures and impact. The FMEA and observations identified that the patient journey was complex due to: the distance they were required to walk; having to enter the department through the ambulance entrance; and the requirement to speak to a specific member of staff in the department to obtain further directions. The process was changed to: reduce the distance the patient had to walk; decrease the likelihood for them to get lost; improve patient visibility for staff; and to facilitate more timely ECGs. As result of the Human Factors review, ED staff felt empowered to make immediate, no-cost and sustainable improvements. |
Allocation of Function Method to support future nuclear reactor plant design
| Document
|
Author Dr Arzoo Naghiyev, Mr John Mount, Mr Anthony Rice and Mrs Caroline Sayce |
| Abstract Current Allocation of Function methods require significant levels of judgement and interpretation and there is an opportunity to develop an improved capability for Allocation of Function for the context of the Nuclear Reactor Plant. This paper presents the development and application of an Allocation of Function method that provides a flexible and configurable set of tools which can be selected in accordance to the design stage and project requirements. The Allocation of Function method has been designed to be used in an iterative manner throughout the different stages of design development and used to engage with different engineering teams. The method draws upon existing and well-established HF methods to investigate and capture human-system interactions associated with function delivery. It also focuses on cognitive tasks to ensure introduction of automation continues to provide support to the operator. Particular emphasis is placed on mapping and understanding the cognitive processes employed in function delivery to ensure that all functionality and information requirements are captured in future automation design. The method also informs assigning and selecting a Level of Automation to a function. The Allocation of Function method enables integration with Systems Engineering to trade HF requirements against the engineering requirements for provision of automation. |
Revolution of Report Writing in Safety Investigations
| Document
|
Author Jenny O’Donnell, Nic Steevenson & Paul Bowie |
| Abstract This paper highlights the presence of reductionism and consequent blame within safety investigation reports across multiple high-reliability industries. It discusses the use of an innovative Learning Response Review and Improvement Tool, (LRRIT, referred to as the ‘Tool’ in this paper) initially developed for healthcare safety improvement (HSSIB, 2023). This Tool helps to shift the focus from the person, at the sharp end, to drive impartiality and systemic learning in safety investigation reports. |
Paediatric Homecare Risk Management: A Functional Resonance Analysis Method Study of Incident and Risk Assessment Management
| Document
|
Author Kevin Michael Hoy, Enda Fallon & Martina Kelly |
| Abstract Paediatric homecare is an advancing field of healthcare and risk management is an integral component of these services. This study is part of a larger study into integrated risk management in paediatric homecare risk management. Through interviews with nursing staff using Grounded Theory methodology, analysis of the risk management components was undertaken using Function Resonance Analysis Method (FRAM). The results indicated a clear mapping of the functionality of the process for incident reporting and the assessment of risks. Resonance was evident in several key functions allowing system changes for organisational improvements to enhance quality of care. KEYWORDS |
Systems Human Factors and Ergonomics methods: applications, outcomes, and future directions
| Document
|
Author Samantha L. Jackson, Gemma J. M. Read, Adam Hulme, & Paul M. Salmon |
| Abstract This systematic literature review identified peer-reviewed applications of systems HFE methods to determine the range of problems examined and how the methods have been applied. The review revealed a growth in applications of systems HFE methods over time. The review suggests that as problem and system complexity continue to intensify, continual evaluation and potential adaption of methods may be required, including using more than one method. |
The quest for the ring: Designing submarine control room work using ComTET
| Document
|
Author Neville A. Stanton, Aaron P. J. Roberts, Daniel Fay and Kiome A. Pope |
| Abstract Research and development for new operational capability in submarines comprises research into human centred design of new capability (where the Command Team-work Experimental Testbed (ComTET) resides), procurement of new capability training capability, and operational evaluation of capability and lessons learnt. The research is given impetus from lessons learnt at sea as well as horizon scanning for technological capability by Defence Science and Technology Laboratory (Dstl). This is fed into the design challenge, which is presented to the ComTET Team, who undertake the research. The ComTET team work with submarine trainers to review current operations and new ways of working. Experiments conducted using the ComTET facilities with serving submariners are able to test and evaluate new technologies and new ways of working. The refined concepts may be taken into the simulators to further refine and test. Once the new operational capabilities have been shown to deliver benefit in ComTET facilities, and the simulators, then they may enter the procurement process. When the new capability enters service on operational platforms, then full evaluation in-service may begin. Any lessons learnt during service may then be fed back into the research cycle. The ComTET team have delivered a guidance document reviewing simulator fidelity, validity and transfer of training to underpin understanding of facility requirements across the different phases of testing. |
