2021
DOI: 10.48550/arxiv.2112.00686
|View full text |Cite
Preprint
|
Sign up to set email alerts
|

CYBORG: Blending Human Saliency Into the Loss Improves Deep Learning

Abstract: Can deep learning models achieve greater generalization if their training is guided by reference to human perceptual abilities? And how can we implement this in a practical manner? This paper proposes a first-ever training strategy to ConveY Brain Oversight to Raise Generalization (CY-BORG). This new training approach incorporates humanannotated saliency maps into a CYBORG loss function that guides the model towards learning features from image regions that humans find salient when solving a given visual task.… Show more

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...
2
2
1

Citation Types

0
5
0

Year Published

2022
2022
2023
2023

Publication Types

Select...
2
1
1

Relationship

0
4

Authors

Journals

citations
Cited by 4 publications
(5 citation statements)
references
References 27 publications
0
5
0
Order By: Relevance
“…Why the model came to this decision, on the other hand, is more complex. The interpretation in this work of why the AI made that decision relates to areas in the image We use pre-trained DenseNet-121 models to detect synthetic faces (Boyd et al 2021), and devise three separate experimental settings, as illustrated in Fig. 3.…”
Section: Ai Cues Shown To Humansmentioning
confidence: 99%
See 2 more Smart Citations
“…Why the model came to this decision, on the other hand, is more complex. The interpretation in this work of why the AI made that decision relates to areas in the image We use pre-trained DenseNet-121 models to detect synthetic faces (Boyd et al 2021), and devise three separate experimental settings, as illustrated in Fig. 3.…”
Section: Ai Cues Shown To Humansmentioning
confidence: 99%
“…CAMs are overlayed on the original image as a heatmap, where red and blue regions represent more and less salient regions, respectively. Because we could use 10 models in each training setting (Boyd et al 2021), we combine the generated CAM heatmaps into a GIF that smoothly transitions between all 10 unique CAMs over a 10s period.…”
Section: Ai Cues Shown To Humansmentioning
confidence: 99%
See 1 more Smart Citation
“…Human-aided DL training is another promising avenue. Indeed, humans and machines cooperating in vision tasks is not new, and this strategy is finding its way into DL as well [14,17]. For example, Boyd et al [14] analyzed the utility of human judgement about salient regions of images to improve generalization of DL models.…”
Section: Open Research Questions In Iris Padmentioning
confidence: 99%
“…Asked about regions that humans deem important for their decision about an image, the work proposed to transform the training data to incorporate such opinions, demonstrating an improvement in accuracy and generalization in leave-one-attack-type-out scenarios. In a similar work, Boyd et al [17] incorporated annotated saliency maps into the loss function to penalize large differences with human judgement.…”
Section: Open Research Questions In Iris Padmentioning
confidence: 99%