Ask Me or Tell Me? Enhancing the Effectiveness of Crowdsourced Design Feedback

Abstract

Crowdsourced design feedback systems are emerging resources for getting large amounts of feedback in a short period of time. Traditionally, the feedback comes in the form of a declarative statement, which often contains positive or negative sentiment. Prior research has shown that overly negative or positive sentiment can strongly influence the perceived usefulness and acceptance of feedback and, subsequently, lead to ineffective design revisions. To enhance the effectiveness of crowdsourced design feedback, we investigate a new approach for mitigating the effects of negative or positive feedback by combining open-ended and thought-provoking questions with declarative feedback statements. We conducted two user studies to assess the effects of question-based feedback on the sentiment and quality of design revisions in the context of graphic design. We found that crowdsourced question-based feedback contains more neutral sentiment than statement-based feedback. Moreover, we provide evidence that presenting feedback as questions followed by statements leads to better design revisions than question-or statement-based feedback alone. CCS CONCEPTS• Human-centered computing → Human computer interaction (HCI).

On the Role of Large Language Models in Crowdsourcing Misinformation Assessment

“…Later, Roitero et al (2021) looked at the longitudinal dimension of crowdsourced truthfulness assessment observing a consistency in the generated labels. La Barbera et al (2020) observed a political bias in crowd-generated truthfulness labels. In summary, the key conclusion that can be drawn from this body of work is that the crowd can be a viable alternative to experts or to automatically generate misinformation labels with the risk of potential bias.…”

“…We divide the selected 120 political statements into 20 tasks/units each of them containing 6 statements. To ensure all units are balanced in terms of truthfulness level and political parties to mitigate the cognitive bias of crowd workers (La Barbera et al 2020;Draws et al 2022), we merge the original 6-level truthfulness scale into a 3-level one (true, in between and false) based on the observations made by Roitero et al (2020). The set of labels given by PolitiFact editors is considered the ground truth in this study.…”

“…An interesting alternative to using experts or ML to deal with misinformation is the use of crowdsourcing. Recent research has looked at the feasibility of crowdsourcing as a way to detect and label misinformation (La Barbera et al 2020;Roitero et al 2020).…”

Socio-Economic Diversity in Human Annotations

“…We set standardised thresholds to classify annotations: a composite score between -0.05 and 0.05 indicates a neutral sentiment; a composite score greater than 0.05 indicates a positive sentiment; a composite score less than -0.05 indicates a negative sentiment. This method of sentiment classification has been widely used [20,25]. Table 6 shows the number of annotations under the distinct segments of abstract annotations.…”