A behaviorally inspired fusion approach for computational audiovisual saliency modeling

Tsiami, Antigoni; Koutras, Petros; Katsamanis, Athanasios; Vatakis, Argiro; Maragos, Petros

doi:10.1016/j.image.2019.05.001

Cited by 16 publications

(17 citation statements)

References 86 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…A. Setup 1) Datasets: The proposed method is trained and evaluated on AVAD [53], Coutrot1 [68], [69], Coutrot2 [68], [69], DIEM [70], ETMD [71], [72] and SumMe [72], [73] datasets. These datasets contains various types videos accompanied with audios.…”

Section: Experiments and Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Bio-Inspired Representation Learning for Visual Attention Prediction

Yuan

Ning

2021

IEEE Trans. Cybern.

View full text Add to dashboard Cite

Visual Attention Prediction (VAP) is a significant and imperative issue in the field of computer vision. Most of existing VAP methods are based on deep learning. However, they do not fully take advantage of the low-level contrast features while generating the visual attention map. In this paper, a novel VAP method is proposed to generate visual attention map via bioinspired representation learning. The bio-inspired representation learning combines both low-level contrast and high-level semantic features simultaneously, which are developed by the fact that human eye is sensitive to the patches with high contrast and objects with high semantics. The proposed method is composed of three main steps: 1) feature extraction, 2) bio-inspired representation learning and 3) visual attention map generation. Firstly, the high-level semantic feature is extracted from the refined VGG16, while the low-level contrast feature is extracted by the proposed contrast feature extraction block in a deep network. Secondly, during bio-inspired representation learning, both the extracted low-level contrast and high-level semantic features are combined by the designed densely connected block, which is proposed to concatenate various features scale by scale. Finally, the weightedfusion layer is exploited to generate the ultimate visual attention map based on the obtained representations after bio-inspired representation learning. Extensive experiments are performed to demonstrate the effectiveness of the proposed method.

show abstract

Section: Experiments and Resultsmentioning

confidence: 99%

“…Specifically, 1) the AVAD dataset [53] The eye-tracking data are recorded via 42 participants. The ETMD dataset [71], [72] includes 12 video clips from several hollywood movies. The eye-tracking data are annoted by 10 different people.…”

Section: Experiments and Resultsmentioning

confidence: 99%

Bio-Inspired Representation Learning for Visual Attention Prediction

Yuan

Ning

2021

IEEE Trans. Cybern.

View full text Add to dashboard Cite

show abstract

“…A higher visual attention priority indicates the object of interest is irregular or rare in its surroundings. Recently, a diversity of saliency models has become a popular technical term for human vision attention study in fields such as communication and electronic engineering (e.g., Garnett et al, 2014;Mahdi et al, 2019;Tsiami et al, 2019). However, there is little research that applies saliency modeling to the fashion area.…”

mentioning

confidence: 99%

Where Do Consumers Look When Viewing Fashion Advertisements? Saliency Based Models for Visual Attention Prediction

Mahdi¹,

Qin²,

Lee

et al. 2020

Pivoting for the Pandemic

View full text Add to dashboard Cite

Conceptual Framework. Advertising is a key marketing tool and is costly to the company, so creating ads as effectively as possible to attract potential customers is essential for business success (Simmonds et al., 2020). Many industries, including the fashion industry, use magazine ads to deliver brand images to consumers, increase brand reputation, and encourage consumers to buy their brand products. An understanding of consumers' visual attention to advertising is vital because there is a robust relationship between consumers' visual attention and memory (Pieters et al., 2007). Most marketing communication devices work through the sense of sight. Advertisement content including brand logos, colors, packaging, letters and fonts all attract visual attention to create consumer awareness of a brand (Simmonds et al., 2020).Human beings tend to minimize their neural resources by using eye, head, and body movement to shift their visual attention and gaze behavior toward more informative image spatial locations (Mahdi et al., 2017). In computer vision, a saliency map is a 2D topological map that indicates visual attention priorities using a numerical scale. A higher visual attention priority indicates the object of interest is irregular or rare in its surroundings. Recently, a diversity of saliency models has become a popular technical term for human vision attention study in fields such as communication and electronic engineering (e.g., Garnett et al., 2014;Mahdi et al., 2019;Tsiami et al., 2019). However, there is little research that applies saliency modeling to the fashion area. Thus, it is meaningful to investigate whether saliency modeling can be applied to the fashion area.Purposes of the study. This multidisciplinary study were 1) to explore where consumers look at fashion advertisements using saliency based models, and 2) to investigate which saliency model(s) are the most effective tools in predicting consumers' visual attention when viewing fashion advertisements.Method. Seventy college students were recruited to participate in the study at a mid-western university in US. Participants were from different majors such as fashion and electronic engineering. One hundred fashion images were selected from fashion magazines (e.g., Vogue) and divided into four image groups (25 images in each group). Each participant was seated about 1 meter away from the screen, viewed 25 fashion images with 5 seconds per image, and selected locations on each image that most attracted their attention by clicking a mouse on the screen. The mouse clicking points were saved as human attention data on the 100 fashion images. The experimental setup is shown in Figure 1. Moreover, six saliency models (Ground-truth, AW5,

show abstract

“…H axiolÏghsh twn tri∏n diaforetik∏n sqhmàtwn s'mmeixhc, efarmosmËna gia s'mmeixh akoustik †c emfàneiac me 4 diaforetikà qwro-qronikà montËla optik †c emfàneiac kai h s'gkrish me to optikoakoustikÏ montËlo Min et al [186] pragmatopoio'ntai se 6 diafore-tikËc bàseic dedomËnwn: th DIEM [267], thn AVAD [186] tic Coutrot1, Coutrot2 [181,183], th SumMe [246,268] kai thn ETMD [148,268], oi opo-ec periËqoun arketà per-ploka kai d'skola erej-smata. Sthn axiolÏghsh aut † pou pragmatopoio'me, de stoqe'oume sthn pl †rh beltistopo-hsh twn montËlwn ∏ste na apokt †soume ta kal'tera dunatà apotelËsmata se apÏlutouc arijmo'c, allà kur-wc sto na diereun †soume an h ep-dosh twn optik∏n montËlwn auxànetai kai belti∏netai me thn s'mmeixh thc akoustik †c plhrofor-ac.…”

Section: Peiràmata Me Dedomëna Ofjalmokinhtik †C Parakolo'jhshcmentioning

confidence: 99%

“…H bàsh dedomËnwn SumMe [246,268] periËqei 25 b-nteo qrhst∏n kaj∏c kai tic peril †yeic touc pou Ëqoun pragmatopoihje-apÏ diàforouc anjr∏pouc mËsa se Ëna elegqÏmeno yuqo-logikÏ pe-rama. 'Opwc anafËrjhke prohgoumËnwc, sullËxame dedomËna ofjalmokinhtik †c parakolo'jhshc, ta opo-a qrhsimopoio'me gia thn axiolÏghsh.…”

Section: Bàsh Dedomënwn Summementioning

confidence: 99%

Οπτικοακουστική Μοντελοποίηση Προσοχής Και Πολυ-Αισθητηριακή Κατανόηση Ακουστικής Σκηνής

Τσιάμη¹

View full text Add to dashboard Cite

Στη διατριβή αυτή μελετάται η δυνατότητα ανάπτυξης ενός μοντέλου οπτικοακουστικής προσοχής/εμφάνειας βασισμένου σε ευρήματα από συμπεριφορικά πειράματα, το οποίο προβλέπει τα σημεία εστίασης του ανθρώπινου ματιού σε μια δισδιάστατη οπτικοακουστική σκηνή. Η μελέτη αυτή πραγματοποιείται με δύο τρόπους: με τεχνικές επεξεργασίας σήματος και με βαθιά νευρωνικά δίκτυα. Όσον αφορά τις τεχνικές επεξεργασίας σήματος, διερευνώνται διάφορες τεχνικές σύμμειξης οπτικής και ακουστικής εμφάνειας με ήδη υπάρχοντα μοντέλα. Τα δημιουργούμενα οπτικοακουστικά μοντέλα επαληθεύονται τόσο με ευρήματα από συμπεριφορικά πειράματα, όσο και με δεδομένα οφθαλμοκινητικής παρακολούθησης αλλά και δεδομένα fMRI. Τα αποτελέσματα τόσο από τα συμπεριφορικά πειράματα όσο και από τα πειράματα με οφθαλμοκινητικά δεδομένα υποδεικνύουν ότι η οπτικοακουστική μοντελοποίηση βελτιώνει την επίδοση του συστήματος πρόβλεψης σημείων εστίασης του ματιού. Όσον αφορά τα βαθιά νευρωνικά δίκτυα, παρουσιάζεται ένα χωρο-χρονικό οπτικοακουστικό δίκτυο εμφάνειας, το οποίο περιέχει ένα οπτικό δίκτυο εμφάνειας, ένα δίκτυο ακουστικών αναπαραστάσεων, μία μονάδα εντοπισμού του ήχου στο βίντεο και μία μονάδα σύμμειξης της οπτικής και της ακουστικής εμφάνειας. Όλα αυτά ενσωματώνονται κάτω από ένα ενιαίο δίκτυο το οποίο εκπαιδεύεται end-to-end. Το δίκτυο αξιολογείται σε βάσεις οφθαλμοκινητικών δεδομένων. Τα αποτελέσματα των συγκρίσεων με άλλες μεθόδους υποδεικνύουν την αποτελεσματικότητα της μοντελοποίησης, ανοίγοντας το δρόμο για την εκτίμηση εμφάνειας «in-the-wild». Παράλληλα έχει πραγματοποιηθεί έρευνα στην κατανόηση ακουστικής σκηνής και συγκεκριμένα έχει αναπτυχθεί ένα σύστημα εντοπισμού ομιλητή, ένα σύστημα αναγνώρισης φωνής στα Ελληνικά και στα Αγγλικά και ένα σύστημα διαλόγου και κατανόησης λόγου. Τα συστήματα έχουν προσαρμοστεί και εφαρμοστεί είτε σε περιβάλλοντα έξυπνου σπιτιού ή/και σε περιβάλλοντα αλληλεπίδρασης ανθρώπων/παιδιών και ρομπότ με πολλαπλούς αισθητήρες. Επίσης, γίνεται αξιολόγησή τους σε κατάλληλες βάσεις δεδομένων. Τέλος, εκτός από την ανάπτυξη νέων αλγοριθμικών μεθόδων για τα παραπάνω προβλήματα, σημαντικό μέρος της διατριβής αποτελεί και η συλλογή νέων μεγάλων βάσεων δεδομένων: Συλλέχθηκε μία βάση με δεδομένα οφθαλμοκινητικής παρακολούθησης από 37 βίντεο και 20 συμμετέχοντες, καθώς και μία πολυκαναλική βάση με δεδομένα φωνής στα Ελληνικά από 20 ομιλητές.

show abstract

A behaviorally inspired fusion approach for computational audiovisual saliency modeling

Cited by 16 publications

References 86 publications

Bio-Inspired Representation Learning for Visual Attention Prediction

Bio-Inspired Representation Learning for Visual Attention Prediction

Where Do Consumers Look When Viewing Fashion Advertisements? Saliency Based Models for Visual Attention Prediction

Οπτικοακουστική Μοντελοποίηση Προσοχής Και Πολυ-Αισθητηριακή Κατανόηση Ακουστικής Σκηνής

Contact Info

Product

Resources

About