| Document
Insert here |
Author Chris Baber, Patrick Waterson, Paul Salmon |
| Abstract Often ‘responsible AI’ focuses attention on the design of AI systems. We propose that a more pressing need arises when such systems are deployed. There is a need to predict where to place responsibility for outcomes arising from the activity of AI systems. Responsibility cannot be given to the AI system, human oversight must be designed into the broader sociotechnical system in which the AI is deployed, and human oversight must be considered in terms of meaningful control of the AI system. We demonstrate how methods from the Cognitive Work Analysis (CWA) framework allow exploration of allocation of responsibility, AoR, in AI systems. Using a case study of a human-agent team, we show that the AI system does not operate at the level of physical form, i.e., it is not a tool that supports human activity, but operates at higher levels (e.g., physical and generalized functions) in ways that influence abstract functions (i.e., system values). Introducing AI involves redesign of the system to reflect the need for human oversight. As a result, we need to define new tasks for humans in this system and these could involve either changes to existing roles (e.g., the road traffic management role might extend to cover maintenance scheduling) or the introduction of additional roles (e.g., the highway engineers might need a role to work with the AI system and confirm its schedule), and the new roles could have additional requirements for coordination across the other roles in the system. |