Causal Effects of Linguistic Properties

Pryzant, Reid; Card, Dallas; Jurafsky, Dan; Veitch, Victor; Sridhar, Dhanya

doi:10.18653/v1/2021.naacl-main.323

Cited by 20 publications

(17 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is similar to the assumptionsPryzant et al (2021) make for linguistic properties of text as treatment.…”

supporting

confidence: 65%

“…Other work has used causal mediation analysis to better understand components of natural language processing (NLP) models (Vig et al, 2020;Finlayson et al, 2021). However, this work is more closely aligned with studies that focus on causal estimation in which text is one or more causal variables (e.g., Veitch et al, 2020;Roberts et al, 2020;Keith et al, 2020;Zhang et al, 2020;Pryzant et al, 2021).…”

Section: Introductionmentioning

confidence: 77%

See 1 more Smart Citation

Text as Causal Mediators: Research Design for Causal Estimates of Differential Treatment of Social Groups via Language Aspects

Keith¹,

Rice²,

O’Connor³

2021

Proceedings of the First Workshop on Causal Inference and NLP

View full text Add to dashboard Cite

Using observed language to understand interpersonal interactions is important in highstakes decision making.We propose a causal research design for observational (nonexperimental) data to estimate the natural direct and indirect effects of social group signals (e.g. race or gender) on speakers' responses with separate aspects of language as causal mediators. We illustrate the promises and challenges of this framework via a theoretical case study of the effect of an advocate's gender on interruptions from justices during U.S. Supreme Court oral arguments. We also discuss challenges conceptualizing and operationalizing causal variables such as gender and language that comprise of many components, and we articulate technical open challenges such as temporal dependence between language mediators in conversational settings.

show abstract

“…This is similar to the assumptionsPryzant et al (2021) make for linguistic properties of text as treatment.…”

supporting

confidence: 65%

Section: Introductionmentioning

confidence: 77%

Text as Causal Mediators: Research Design for Causal Estimates of Differential Treatment of Social Groups via Language Aspects

Keith¹,

Rice²,

O’Connor³

2021

Proceedings of the First Workshop on Causal Inference and NLP

View full text Add to dashboard Cite

show abstract

“…[56][57][58] Our procedure brings a similar explainable AI strategy to NLP by identifying words and sentences that activate specific artificial neurons tied to ES levels. While there are ongoing efforts to make NLP explainable, 28,59,60 our methodology makes such insights accessible to medical education researchers now. Our methods augment traditional qualitative and mixed-methods analysis by giving medical education researchers AI-mediated assistance to explore themes in prohibitively large narrative databases that may be difficult to assess with traditional qualitative methods (i.e.…”

Section: Context and Taskmentioning

confidence: 99%

Exploring how feedback reflects entrustment decisions using artificial intelligence

et al. 2021

View full text Add to dashboard Cite

Context Clinical supervisors make judgements about how much to trust learners with critical activities in patient care. Such decisions mediate trainees' opportunities for learning and competency development and thus are a critical component of education. As educators apply entrustment frameworks to assessment, it is important to determine how narrative feedback reflecting entrustment may also address learners' educational needs. Methods In this study, we used artificial intelligence (AI) and natural language processing (NLP) to identify characteristics of feedback tied to supervisors' entrustment decisions during direct observation encounters of clerkship medical students (3328 unique observations). Supervisors conducted observations of students and collaborated with them to complete an entrustment‐based assessment in which they documented narrative feedback and assigned an entrustment rating. We trained a deep neural network (DNN) to predict entrustment levels from the narrative data and developed an explainable AI protocol to uncover the latent thematic features the DNN used to make its prediction. Results We found that entrustment levels were associated with level of detail (specific steps for performing clinical tasks), feedback type (constructive versus reinforcing) and task type (procedural versus cognitive). In justifying both high and low levels of entrustment, supervisors detailed concrete steps that trainees performed (or did not yet perform) competently. Conclusions Framing our results in the factors previously identified as influencing entrustment, we find a focus on performance details related to trainees' clinical competency as opposed to nonspecific feedback on trainee qualities. The entrustment framework reflected in feedback appeared to guide specific goal‐setting, combined with details necessary to reach those goals. Our NLP methodology can also serve as a starting point for future work on entrustment and feedback as similar assessment datasets accumulate.

show abstract

“…Since the new prompt did not specifically solicit named entities like it did the year prior, are students making fewer references to them? The emerging causal inference with NLP literature has so far focused on word embedding [45] and topic modeling [46] methods. We extend this rapidly growing literature in our use of a finer grained method, NER, and in our more sociological context (as opposed to political science or traditional computer science).…”

Section: Study Two: Named Entity Recognitionmentioning

confidence: 99%

Linguistic, Cultural, and Narrative Capital: Computational and Human Readings of Transfer Admissions Essays

Alvero¹,

Pal²,

Moussavian³

2022

Preprint

View full text Add to dashboard Cite

Stratification in college application materials is a contentious topic in social science and national discourse in the United States. This line of research has also started to use computational methods to consider qualitative materials, such as personal statements and letters of recommendation. Despite the prominence of this topic, fewer studies have considered a fairly common academic pathway: transferring. Approximately 40\% of all college students in the United States transfer schools at least once. One quirk of the system is that students from community colleges are applying for the same spots for students already enrolled in four year schools and trying to transfer. How might these different institutional experiences and the transfer application itself reflect social stratification? We leverage a dataset of 20,532 transfer admissions essays submitted to the University of California system to describe how transfer applicants are stratified linguistically, culturally, and narratively with respect to academic pathways and essay prompts. These results show different types of stratification that can emerge in educational processes intended to equalize opportunity and how combining computational and human reading might illuminate them.

show abstract

Causal Effects of Linguistic Properties

Cited by 20 publications

References 45 publications

Text as Causal Mediators: Research Design for Causal Estimates of Differential Treatment of Social Groups via Language Aspects

Text as Causal Mediators: Research Design for Causal Estimates of Differential Treatment of Social Groups via Language Aspects

Exploring how feedback reflects entrustment decisions using artificial intelligence

Linguistic, Cultural, and Narrative Capital: Computational and Human Readings of Transfer Admissions Essays

Contact Info

Product

Resources

About