Who is Responsible? Explaining Safety Violations in Multi-Agent Cyber-Physical Systems

Multi-agent cyber-physical systems are present in a variety of applications. Agent decision-making can be affected due to errors induced by uncertain, dynamic operating environments or due to incorrect actions taken by an agent. When an erroneous decision that leads to a violation of safety is identified, assigning responsibility to individual agents is a key step towards preventing future accidents. Current approaches to carrying out such investigations require human labor or high degree of familiarity with operating environments. Automated strategies to assign responsibility can achieve significant reduction in human effort and associated cognitive burden.In this paper, we develop an automated procedure to assign responsibility for safety violations to actions of any single agent in a principled manner. We ground our approach on reasoning about safety violations in road safety. When provided with an instance of a safety violation, we use counterfactual reasoning to create alternate scenarios that determine how different outcomes might have been achieved if a specific action or set of actions was replaced by another action or set of actions. We devise a metric called the degree of responsibility (DoR) for each agent. The DoR uses the Shapley value to quantify the relative contribution of each agent to the observed safety violation, thus serving as a basis to explain and justify future decisions. We devise both heuristic techniques and methods based on the structure of agent interactions to improve scalability of our solution as the number of agents increases. We consider three instances of safety violations from the National Highway Traffic Safety Administration (NHTSA). We carry out experiments using representations of the three scenarios using the CARLA urban driving simulator. Our results indicate that the DoR enhances explainability of decision-making and assigning accountability for actions of agents and their consequences.