The trouble with social computing systems research

Bernstein, Michael S.; Ackerman, Mark S.; H., Ed; Miller, Robert C.

doi:10.1145/1979742.1979618

Cited by 30 publications

(19 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It has used EV to refer to fieldwork (Carter, Mankoff, Klemmer, & Matthews, 2008;Bernstein, Ackerman, Chi, & Miller, 2011), real-life or naturalistic conditions where experimental tasks are performed with high levels of fidelity (Castro, Favela, & GarciaPena, 2011), and studies "where subjects are totally unaware of being tested, testing being performed during their natural activity on the web" (Guerini, Strapparava, & Stock, 2012). None of these interpretations actually refer to EV.…”

Section: Contributionmentioning

confidence: 99%

ECOVAL: Ecological Validity of Cues and Representative Design in User Experience Evaluations

Kieffer¹

2017

THCI

View full text Add to dashboard Cite

Abstract:Egon Brunswik coined and defined the concepts of ecological validity and representative design, which are both essential to achieve external validity. However, research in HCI has inconsistently and incorrectly used Brunswik's concept of ecological validity, which prevents the field from developing cumulative science and from generalizing the findings of user experience (UX) evaluations. In this paper, I present ECOVAL, a framework I built on Brunswik's ideas. On the one hand, ECOVAL helps HCI researchers describe and assess the ecological validity of cues in UX evaluations. On the other hand, ECOVAL guidelines-formulated as a step-by-step procedure-help HCI researchers achieve representative design and, therefore, increase external validity. An industrial case study demonstrates the relevance of ECOVAL for achieving representative design while conducting formative UX testing. In discussing the case study, I describe how ECOVAL can help HCI researchers assess and increase the validity of UX experiments and generalize UX findings. I also illustrate the trade-offs between internal and external validities and UX resources that inevitably arise when one conducts UX experiments. From the results, I sketch avenues for future research and discuss the related challenges that future work should address.

show abstract

Section: Contributionmentioning

confidence: 99%

ECOVAL: Ecological Validity of Cues and Representative Design in User Experience Evaluations

Kieffer¹

2017

THCI

View full text Add to dashboard Cite

show abstract

“…Yet there are numerous debates and disagreements within HCI about what constitutes an adequate systems evaluation [8,22]. Some scholars have even claimed that the bar for evaluation at UIST, CHI, and other HCI publication venues is increasingly unrealistic for systems papers [1,14] by insisting on having results from controlled studies or large cohorts of representative users.…”

Section: Introductionmentioning

confidence: 99%

“…Specifically, we present an in-depth case study of several evaluation methods applied to LemonAid [4] (Figure 1) over the last five years, from its inception as a university research project to commercialization through a venture capital financed startup called AnswerDash. 1 We trace the motivations for adopting different HCI methods at different stages during the evolution of the research, product, and startup business and the tradeoffs made between user-centered design and what we call adoption-centered design-understanding and designing for product adopters and stakeholders beyond just end users.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

From User-Centered to Adoption-Centered Design

Chilana

Wobbrock

2015

Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

As we increasingly strive for scientific rigor and generalizability in HCI research, should we entertain any hope that by doing good science, our discoveries will eventually be more transferrable to industry? We present an in-depth case study of how an HCI research innovation goes through the process of transitioning from a university project to a revenue-generating startup financed by venture capital. The innovation is a novel contextual help system for the Web, and we reflect on the different methods used to evaluate it and how research insights endure attempted dissemination as a commercial product. Although the extent to which any innovation succeeds commercially depends on a number of factors like market forces, we found that our HCI innovation with user-centered origins was in a unique position to gain traction with customers and garner buy-in from investors. However, since end users were not the buyers of our product, a strong user-centered focus obfuscated other critical needs of the startup and pushed out perspectives of nonuser-centered stakeholders. To make the research-toproduct transition, we had to focus on adoption-centered design, the process of understanding and designing for adopters and stakeholders of the product. Our case study raises questions about how we evaluate the novelty and research contributions of HCI innovations with respect to their potential for commercial impact.

show abstract

“…Nonetheless, snowball sampling is often embraced as the only way to approach hidden or marginalized populations, such as drug users [17], prostitutes [19], pickpockets [15], and low-income respondents in the developing world [13,18,22,23,26]. Researchers have also championed the use of snowball sampling in social computing research, where a global directory of all users is usually unavailable and snowball sampling can be viewed as a form of convenience sample [4].…”

Section: Introductionmentioning

confidence: 99%

Increasing the Reach of Snowball Sampling

Vashistha

Cutrell

Thies

2015

Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work &Amp; Social Computing

View full text Add to dashboard Cite

Though many researchers have studied how to incentivize people to respond to surveys, little is known about how these incentives impact respondents' willingness to recruit others to participate as well. In this paper, we show that the incentives offered for individual survey responses can have a dramatic impact on the overall reach of a survey through a network of peers. In a field experiment in India, we made a survey accessible via mobile phones and offered respondents either a fixed incentive (guaranteed payment of about $0.17) or a lottery incentive (1% chance of winning $17). When asked to choose, a significant fraction of respondents preferred the lottery incentive. However, when encouraged to spread the survey, the fixed incentive spread over 100 times further, reaching about 800 people in a day. We interpret this surprising result and discuss the implications for HCI.

show abstract

The trouble with social computing systems research

Cited by 30 publications

References 27 publications

ECOVAL: Ecological Validity of Cues and Representative Design in User Experience Evaluations

ECOVAL: Ecological Validity of Cues and Representative Design in User Experience Evaluations

From User-Centered to Adoption-Centered Design

Increasing the Reach of Snowball Sampling

Contact Info

Product

Resources

About