Many safety-critical systems have a human-in-the-loop for some part of their operation, and rely on the higher cognitive abilities of the human operator for fault diagnosis and risk-management decision-making. Although these operators are often experts on the processes being controlled, they still sometimes misjudge situations or make poor decisions. There is thus potential for Safety Decision Support Systems (SDSS) to help operators, building on past successes with Clinical Decision Support Systems in the health care industry. Such SDSS could help operators more accurately assess the system's state along with any associated risk and uncertainty. However, such a system supporting a safety critical operation inevitably attracts its own safety assurance obligations. This paper will outline those challenges and suggest an initial safety case architecture for SDSS.

Autonomous planning in safety critical systems is a difficult task where decisions must carefully balance optimisation for performance goals of the system while also keeping the system away from safety hazards. These tasks often conflict, and hence present a challenging multi-objective planning problem where at least one of the objectives relates to safety risk. Recasting safety risk into an objective introduces additional requirements on planning algorithms: safety risk cannot be "averaged out" nor can it be combined with other objectives without loss of information and losing its intended purpose as a tool in risk reduction. Thus, existing algorithms for multi-objective planning cannot be used directly as they do not provide any facility to accurately track and update safety risk. A common workaround is to restrict available decisions to those guaranteed safe a priori, but this can be overly conservative and hamper performance significantly. In this paper, we propose a planning algorithm based on multiobjective Monte-Carlo Tree Search to resolve these problems by recognising safety risk as a first class objective. Our algorithm explicitly models the safety of the system separately from the performance of the system, uses safety risk to both optimise and provide constraints for safety in the planning process, and uses an ALARP-based preference selection method to choose an appropriate safe plan from its output. The preference selection method chooses from the set of multiple safe plans to weigh risk against performance. We demonstrate the behaviour of the algorithm using an example representative of safety critical decision-making.