What People Look at in Multimodal Online Dating Profiles: How Pictorial and Textual Cues Affect Impression Formation

Abstract

This study investigates how online dating profiles, consisting of both pictures and texts, are visually processed, and how both components affect impression formation. The attractiveness of the profile picture was varied systematically, and texts either included language errors or not. By collecting eye tracking and perception data, we investigated whether picture attractiveness determines attention to the profile text and if the text plays a secondary role. Eye tracking results revealed that pictures are more likely to attract initial attention and that more attractive pictures receive more attention. Texts received attention regardless of the picture’s attractiveness. Moreover, perception data showed that both the pictorial and textual cues affect impression formation, but that they affect different dimensions of perceived attraction differently. Based on our results, a new multimodal information processing model is proposed, which suggests that pictures and texts are processed independently and lead to separate assessments of cue attractiveness before impression formation.

Effects of Secondhand Information on Impression Formation in Spoken Communication

“…Previous emotion recognition research has shown that awareness of meaning in ambiguous spoken words is biased toward the emotional tone of voice of speakers [20]. In addition, a recent empirical study on direct impressions reported that moderately attractive faces tend to be more affected by language information (i.e., personal profile of that person) than lesser or highly attractive faces [37]. Given that these previous studies investigated the integration process of multiple information from the same person, and not different people, there is a possibility that the present results are due to secondhand information.…”

There Are Layers to Liars: A Systematic Literature Review of Online Dating Deception

“…One of the main reasons for the prevalence of deception in online dating may be that it is a way to appear more attractive, a highly important aspect in this context that is all about attractiveness. Appearing more attractive on a dating platform may thus be a strategy to encourage others to select their profile (Heino et al, 2010) and to increase the odds of finding a romantic partner Van der Zanden et al, 2022). Though most online daters admit they engage in deceptive behavior to some extent.…”

Analyzing Text and Images in Digital Communication: The Case of Securitization in American White Supremacist Online Discourse

“…The former task assesses content; it detects and labels objects in images, finds people and their faces, infers humans' attributes (e.g., ethnicity, age) and expressions (e.g., emotion, ideology), and so on. It is usually done by building on the manual coding of concepts within and across observations (e.g., Casas and Williams 2019;Trilló and Shifman 2021; van Haperen, Uitermark, and van der Zeeuw 2020), using machine learning (ML) methods (e.g., Cantú 2019;Steinert-Threlkeld, Chan, and Joo 2021;Xi et al 2020;Zhang and Pan 2019), or a combination of the two, such as by first doing manual coding to uncover concepts-and examples of these concepts-that are then used in ML-driven analysis (e.g., Steinert-Threlkeld and Joo 2020;Steinert-Threlkeld et al 2021;van Haperen et al 2020). 2 After identifying and measuring relevant dimensions of images, researchers usually then classify all the images in the database using a framework relevant to their research questions.…”

“…Some of this work does focus on the interaction of text and image, as we do, but stops short of examining how the interplay helps construct meaning in a discourse. 4 For example, studies may analyze how the interpretation of one mode affects the interpretation of the other, such as how an online dating profile’s picture affects the perception of its written portion (Van der Zanden et al 2022). Other research considers each mode as an input of information about the meaning of a discrete object, such a particular meme, but not a general discourse (e.g., Baishya 2021; Milner 2016).…”