Novice and Expert Sensemaking of Crowdsourced Design Feedback

Foong, Eureka; Gergle, Darren; Gerber, Elizabeth M.

doi:10.1145/3134680

Cited by 19 publications

(7 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, more aspects likely come into play. Also, given that designers with varying expertise make sense of and provide feedback differently [14,16], it would be interesting to determine if question-based feedback is perceived differently by non-professional and professional designers.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Ask Me or Tell Me? Enhancing the Effectiveness of Crowdsourced Design Feedback

Lekschas

Ampanavos

Siangliulue

et al. 2021

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Crowdsourced design feedback systems are emerging resources for getting large amounts of feedback in a short period of time. Traditionally, the feedback comes in the form of a declarative statement, which often contains positive or negative sentiment. Prior research has shown that overly negative or positive sentiment can strongly influence the perceived usefulness and acceptance of feedback and, subsequently, lead to ineffective design revisions. To enhance the effectiveness of crowdsourced design feedback, we investigate a new approach for mitigating the effects of negative or positive feedback by combining open-ended and thought-provoking questions with declarative feedback statements. We conducted two user studies to assess the effects of question-based feedback on the sentiment and quality of design revisions in the context of graphic design. We found that crowdsourced question-based feedback contains more neutral sentiment than statement-based feedback. Moreover, we provide evidence that presenting feedback as questions followed by statements leads to better design revisions than question-or statement-based feedback alone. CCS CONCEPTS• Human-centered computing → Human computer interaction (HCI).

show abstract

Section: Discussionmentioning

confidence: 99%

“…Only participants with an acceptance rate above 97% and more than 500 approved HITs were accepted. The majority of participants (16) were aged between 30-40. Three were between 20-30 years old.…”

Section: Participantsmentioning

confidence: 99%

Ask Me or Tell Me? Enhancing the Effectiveness of Crowdsourced Design Feedback

Lekschas

Ampanavos

Siangliulue

et al. 2021

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…In crowd feedback systems, feedback requesters face the challenge of having to explore and analyse a potentially high number of responses. Crowd feedback systems thus call for processing and aggregating of individual feedback items to support the feedback receiver in the related sensemaking activities [14]. Crowd feedback systems address this issue in a number of different ways.…”

Section: Aggregating Design Feedbackmentioning

confidence: 99%

CrowdUI: Supporting Web Design with the Crowd

Oppenlaender

Tiropanis

Hosio

2020

Proc. ACM Hum.-Comput. Interact.

View full text Add to dashboard Cite

Web design is a complex and challenging task. It involves making many design decisions that materialise preconceived notions of user needs that may or may not be true. In this paper, we investigate supporting the co-design of a website with visual feedback elicited from the website's community of users. Website users can express their needs by rearranging and modifying the website's layout and design. To explore and validate this idea, we present CrowdUI, a web-based tool that enables members of the community of a website to visually express their design improvement ideas, frustrations and needs, and to send this feedback to the person in charge of designing or maintaining the website. CrowdUI is validated in a study with 45 users of a popular social media and networking website. Second, our qualitative evaluation with 60 experienced web developers shows that CrowdUI is able to elicit diverse and meaningful feedback. Put together, our results suggest that CrowdUI's approach constitutes a productive setting for eliciting visual feedback from the user community as a complement to traditional ways of eliciting feedback and participatory design. Finally, based on our experiences, we discuss a design space for crowdsourced web design and provide design recommendations for similar future tools. CCS Concepts: • Human-centered computing → Human computer interaction (HCI); User studies; User interface toolkits; • Information systems → Crowdsourcing.

show abstract

“…They complement existing algorithmic, visual, and crowd debugging systems by detecting and describing complex failures in deployment, like those developers may not have considered due to their own blind spots and biases [6,25,49,56]. Visualizing crowdsourced failure reports continues the emerging theme of distributed or crowd sensemaking [20,21,23,32,33], which has been used to improve clustering [3,13], summarize bug reports [27], and learn model features [17].…”

Section: Introductionmentioning

confidence: 99%

Discovering and Validating AI Errors With Crowdsourced Failure Reports

Cabrera

Druck

Hong

et al. 2021

Proc. ACM Hum.-Comput. Interact.

View full text Add to dashboard Cite

AI systems can fail to learn important behaviors, leading to real-world issues like safety concerns and biases. Discovering these systematic failures often requires significant developer attention, from hypothesizing potential edge cases to collecting evidence and validating patterns. To scale and streamline this process, we introduce crowdsourced failure reports, end-user descriptions of how or why a model failed, and show how developers can use them to detect AI errors. We also design and implement Deblinder, a visual analytics system for synthesizing failure reports that developers can use to discover and validate systematic failures. In semi-structured interviews and think-aloud studies with 10 AI practitioners, we explore the affordances of the Deblinder system and the applicability of failure reports in real-world settings. Lastly, we show how collecting additional data from the groups identified by developers can improve model performance.CCS Concepts: • Human-centered computing → Visual analytics; Social tagging; • Computing methodologies → Machine learning.

show abstract

Novice and Expert Sensemaking of Crowdsourced Design Feedback

Cited by 19 publications

References 26 publications

Ask Me or Tell Me? Enhancing the Effectiveness of Crowdsourced Design Feedback

Ask Me or Tell Me? Enhancing the Effectiveness of Crowdsourced Design Feedback

CrowdUI: Supporting Web Design with the Crowd

Discovering and Validating AI Errors With Crowdsourced Failure Reports

Contact Info

Product

Resources

About