Can we crowdsource language design?

Wilson, Preston Tunnell; Pombrio, Justin; Krishnamurthi, Shriram

doi:10.1145/3133850.3133863

Cited by 11 publications

(7 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Chamberlain [17] compared functional-style to literal-style approaches for specifying topology of streaming applications (i.e., pipes-and-filters style applications) using Mechanical Turk, finding that users were more likely to prefer literal-style specifications, and experienced programmers were more likely to understand the literal-style specifications than the functionalstyle ones. Wilson et al [103] investigated crowdsourcing more esoteric language design decisions, finding low consistency (people did not give consistent answers when asked similar questions repeatedly) and low consensus (people did not agree with each other on which design choice was best). Crowdsourcing approaches can scale well, but typically require that the studies be of relatively short duration.…”

Section: Related Workmentioning

confidence: 99%

Pliers

Coblenz

Kambhatla

Koronkevich

et al. 2021

ACM Trans. Comput.-Hum. Interact.

View full text Add to dashboard Cite

Programming language design requires making many usability-related design decisions. However, existing HCI methods can be impractical to apply to programming languages: languages have high iteration costs, programmers require significant learning time, and user performance has high variance. To address these problems, we adapted both formative and summative HCI methods to make them more suitable for programming language design. We integrated these methods into a new process, PLIERS, for designing programming languages in a user-centered way. We assessed PLIERS by using it to design two new programming languages. Glacier extends Java to enable programmers to express immutability properties effectively and easily. Obsidian is a language for blockchains that includes verification of critical safety properties. Empirical studies showed that the PLIERS process resulted in languages that could be used effectively by many programmers and revealed additional opportunities for language improvement.

show abstract

Section: Related Workmentioning

confidence: 99%

Pliers

Coblenz

Kambhatla

Koronkevich

et al. 2021

ACM Trans. Comput.-Hum. Interact.

View full text Add to dashboard Cite

show abstract

“…A number of studies employed workers in order to identify preferences and consensus; for example, researchers have sought to identify preferences about the order of writing expressions such as new byte [10 + length] and new byte[length + 10] [4], about gradual typing semantics [34], and about specific features in a novel programming language [35].…”

Section: Surveysmentioning

confidence: 99%

“…As with much research, it is also unclear here whether the filedrawer effect-not writing up results that contradict researchers' expectations-has resulted in a publication bias that suppresses researchers' negative experiences with crowdsourcing. That being said, the reviewed studies do also report some "negative results," such as identifying that MTurk workers (or programmers in general) should likely not be used for finding a consensus for features of a new programming language [35], and observing that a particular feature of a system developed by the researchers is less efficient than other approaches [22].…”

Section: Study Designs and Data Qualitymentioning

confidence: 99%

See 1 more Smart Citation

Crowdsourcing in Computing Education Research: Case Amazon MTurk

Hellas

Zavgorodniaia

Sorva

2020

Koli Calling '20: Proceedings of the 20th Koli Calling International Conference on Computing Education Research

View full text Add to dashboard Cite

Crowdsourcing platforms such as Amazon MTurk provide access to a human workforce that can be given tasks to complete online for a fee. In this article, we review studies in computing education research (CER) that rely on crowdsourcing; we also describe our own experiences of using Amazon MTurk for a CER study. We discuss challenges in recruiting workers with specific backgrounds-such as no programming experience-and considerations in filtering out unreliable research participants. Combining recommendations from the literature with the lessons that we learned whilst conducting our study, we synthesize advice for researchers in CER who are considering crowdsourcing. In our case study, we did not find widespread foul play by crowdsourced workers and, overall, our experiences and the literature suggest that crowdsourced CER is feasible. It is, however, uncertain to what extent crowdsourced data can produce answers that apply to specific educational contexts. More research is needed before definitive conclusions can be drawn about the validity and generalizability of crowdsourced CER. CCS CONCEPTS• Social and professional topics → Computing education.

show abstract

“…Tunnell Wilson et al [34] have previously tried crowdsourcing language design decisions using Mechanical Turk. They created surveys for a variety of language features to investigate Turkers' expectations and consistency.…”

Section: User Studiesmentioning

confidence: 99%

The behavior of gradual types: a user study

et al. 2020

Self Cite

View full text Add to dashboard Cite

There are several different gradual typing semantics, reflecting different trade-offs between performance and type soundness guarantees. Notably absent, however, are any data on which of these semantics developers actually prefer. We begin to rectify this shortcoming by surveying professional developers, computer science students, and Mechanical Turk workers on their preferences between three gradual typing semantics. These semantics reflect important points in the design space, corresponding to the behaviors of Typed Racket, TypeScript, and Reticulated Python. Our most important finding is that our respondents prefer a runtime semantics that fully enforces statically declared types.

show abstract

Can we crowdsource language design?

Cited by 11 publications

References 24 publications

Pliers

Pliers

Crowdsourcing in Computing Education Research: Case Amazon MTurk

The behavior of gradual types: a user study

Contact Info

Product

Resources

About