VADR: Discriminative Multimodal Explanations for Situational Understanding

An Experimentation Platform for Explainable Coalition Situational Understanding

“…Multimodal Explanations Figure 4 shows the detail associated with a simple event where a shooting has been detected in the city. This uses the selective relevance technique (Taylor et al 2020) to highlight features in the input that were most salient to the detection of the event. Here we can see red highlighting on the person to the left, who is the active shooter (the other person is merely walking past).…”

Towards human-agent knowledge fusion (HAKF) in support of distributed coalition teams

“…In Section II we define a simple worked example using the selective audio-visual relevance (SAVR) explainability technique -derived from the research reported in [3] -to highlight the video and audio aspects of an incoming sensor feed that most influenced a particular classification.…”

“…In earlier work [3] we have defined a technique for providing explanations for the classification decision of a multimodal (audio and video) activity recognition system. The deep neural network model was trained [3] using a generic activity dataset (UCF-101 [7]) and is therefore able to output classification decisions for a wide range of human activities that can be sensed through audio-visual means such as CCTV or camera sources. These classifications will be for specific activities that are unlikely to be of particular relevance to the domain problem in which they are applied.…”

“…Figure 2 shows an example canvas within the Cogni-Sketch environment in which the user has dragged out the video feed from the 'Nightclub' camera (A UK asset as indicated by the flag), and has then dropped the 'SAVR' explainable analytic service onto this video-audio feed 3 . Immediately upon doing so, any classifications that are predicted by the service are shown as new nodes on the canvas, along with the frequency of classifications for each case.…”

“…Effectively this means that those two classes of 'shotput' and 'hammer throw' will be ignored by the downstream service(s). The green icon on the top right enables the SAVR explanations, highlighting the video and audio saliency to the current scene, or historical footage, but these are not described in this paper (refer to [3]). A user accessing these explanations, along with associated confidence or certainty metrics, may be seeking visibility of the saliency of parts of the video or audio in order to better understand the way in which activity recognition is being influenced.…”