How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study

Piorkowski, David; Park, Soya; Wang, April Yi; Wang, Dakuo; Muller, Michael; Portnoy, Felix

doi:10.48550/arxiv.2101.06098

Cited by 3 publications

(4 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There is a long tradition of research in the HCI and STS communities focusing on workplace studies that situate technology work practices within their organizational contexts [e.g., 68,76]. In recent years, considerable research in CSCW has taken a situated approach to understanding how the work practices of data scientists [52,57], AI developers [60], and software developers are shaped by the organizational contexts in which they are embedded [84][85][86][87][88]. For example, Passi and Sengers described collaborative, interdisciplinary teams of practitioners who "mak[e] data science systems work" [58], identifying organizational factors that empower business actors over other team members and business imperatives that prioritize particular normative goals over other goals.…”

Section: Fairness Work In Organizational Contextsmentioning

confidence: 99%

Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support

Madaio¹,

Egede²,

Subramonyam³

et al. 2021

Preprint

View full text Add to dashboard Cite

Various tools and practices have been developed to support practitioners in identifying, assessing, and mitigating fairness-related harms caused by AI systems. However, prior research has highlighted gaps between the intended design of these tools and practices and their use within particular contexts, including gaps caused by the role that organizational factors play in shaping fairness work. In this paper, we investigate these gaps for one such practice: disaggregated evaluations of AI systems, intended to uncover performance disparities between demographic groups. By conducting semi-structured interviews and structured workshops with thirty-three AI practitioners from ten teams at three technology companies, we identify practitioners' processes, challenges, and needs for support when designing disaggregated evaluations. We find that practitioners face challenges when choosing performance metrics, identifying the most relevant direct stakeholders and demographic groups on which to focus, and collecting datasets with which to conduct disaggregated evaluations. More generally, we identify impacts on fairness work stemming from a lack of engagement with direct stakeholders, business imperatives that prioritize customers over marginalized groups, and the drive to deploy AI systems at scale.

show abstract

Section: Fairness Work In Organizational Contextsmentioning

confidence: 99%

Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support

Madaio¹,

Egede²,

Subramonyam³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Task We follow prior works' practices of recruiting human raters to evaluate model-generated contents (Wang et al, 2021). For each rater, we randomly select two papers, one from the former four papers, and another one from the test set.…”

Section: Human Evaluationmentioning

confidence: 99%

“…From business to education to research, presentations are everywhere as they are visually effective in summarizing and explaining bodies of work to the audience (Bartsch and Cobern, 2003;Wang, 2016;Piorkowski et al, 2021). However, it is tedious and time-consuming to manually create presentation slides (Franco et al, 2016).…”

Section: Introductionmentioning

confidence: 99%

D2S: Document-to-Slide Generation Via Query-Based Text Summarization

Sun

Hou²,

Wang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Presentations are critical for communication in all areas of our lives, yet the creation of slide decks is often tedious and time-consuming. There has been limited research aiming to automate the document-to-slides generation process and all face a critical challenge: no publicly available dataset for training and benchmarking. In this work, we first contribute a new dataset, SciDuet, consisting of pairs of papers and their corresponding slides decks from recent years' NLP and ML conferences (e.g., ACL). Secondly, we present D2S, a novel system that tackles the document-to-slides task with a two-step approach: 1) Use slide titles to retrieve relevant and engaging text, figures, and tables; 2) Summarize the retrieved context into bullet points with long-form question answering. Our evaluation suggests that longform QA outperforms state-of-the-art summarization baselines on both automated ROUGE metrics and qualitative human evaluation.

show abstract

“…In practice it is not always feasible to set up an iML system for domain experts to work with. Currently most ML projects still rely on data scientists to write code and set up the pipelines [61]. Moreover, having data scientists mediate the knowledge input offers the flexibility to apply it to different kinds of ML algorithms, and allow domain experts to provide reusable knowledge not constrained by a particular modeling task.…”

Section: With Domain Expertsmentioning

confidence: 99%

Facilitating Knowledge Sharing from Domain Experts to Data Scientists for Building NLP Models

Park¹,

Wang²,

Kawas³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Data scientists face a steep learning curve in understanding a new domain for which they want to build machine learning (ML) models. While input from domain experts could offer valuable help, such input is often limited, expensive, and generally not in a form readily consumable by a model development pipeline. In this paper, we propose Ziva, a framework to guide domain experts in sharing essential domain knowledge to data scientists for building NLP models. With Ziva, experts are able to distill and share their domain knowledge using domain concept extractors and five types of label justification over a representative data sample. The design of Ziva is informed by preliminary interviews with data scientists, in order to understand current practices of domain knowledge acquisition process for ML development projects. To assess our design, we run a mix-method case-study to evaluate how Ziva can facilitate interaction of domain experts and data scientists. Our results highlight that (1) domain experts are able to use Ziva to provide rich domain knowledge, while maintaining low mental load and stress levels; and (2) data scientists find Ziva's output helpful for learning essential information about the domain, offering scalability of information, and lowering the burden on domain experts to share knowledge. We conclude this work by experimenting with building NLP models using the Ziva output by our case study.CCS Concepts: • Human-centered computing → Empirical studies in HCI ; Interactive systems and tools.

show abstract

How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study

Cited by 3 publications

References 34 publications

Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support

Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support

D2S: Document-to-Slide Generation Via Query-Based Text Summarization

Facilitating Knowledge Sharing from Domain Experts to Data Scientists for Building NLP Models

Contact Info

Product

Resources

About