Putting Tools in Their Place: The Role of Time and Perspective in Human-AI Collaboration for Qualitative Analysis

Feuston, Jessica L.; Brubaker, Jed R.

doi:10.1145/3479856

Cited by 26 publications

(3 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…IE and data labeling are used in a variety of settings, each with a distinct set of challenges. Not all settings are similarly suitable for human-computer collaboration (Jiang et al, 2021), and practitioners often resist solutions which over focus on automation instead of providing assistance (Feuston and Brubaker, 2021). To ground our investigation of human-computer collaboration in IE, we start by exploring two use cases related to the criminal justice domain (Section 2.1).…”

Section: Requirements Of High-precision Information Extractionmentioning

confidence: 99%

Optimising Human-Machine Collaboration for Efficient High-Precision Information Extraction from Text Documents

Butcher¹,

Zilka²,

Cook³

et al. 2023

Preprint

View full text Add to dashboard Cite

While humans can extract information from unstructured text with high precision and recall, this is often too time-consuming to be practical. Automated approaches, on the other hand, produce nearly-immediate results, but may not be reliable enough for high-stakes applications where precision is essential. In this work, we consider the benefits and drawbacks of various human-only, human-machine, and machine-only information extraction approaches. We argue for the utility of a human-in-the-loop approach in applications where high precision is required, but purely manual extraction is infeasible. We present a framework and an accompanying tool for information extraction using weak-supervision labeling with human validation. We demonstrate our approach on three criminal justice datasets. We find that the combination of the computer speed and human understanding yields precision comparable to manual annotation while requiring only a fraction of time, and significantly outperforms fully automated baselines in terms of precision.

show abstract

Section: Requirements Of High-precision Information Extractionmentioning

confidence: 99%

Optimising Human-Machine Collaboration for Efficient High-Precision Information Extraction from Text Documents

Butcher¹,

Zilka²,

Cook³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…They found that, when co-creating with AI, creators were concerned about ownership, leadership, AI feedback and evaluation, accountability, data collection and usage, personified communicative AI, and AI access to public data. In the field of human-AI collaboration, researchers have studied creators' perceptions and practices about the complementary of AI and humans [16,42]. Lubars and Tan [46] argued that designers need to consider what AI should do rather than what it can do.…”

Section: 22mentioning

confidence: 99%

Experiments of arsenic (V) adsorption on birnessite: implications on arsenic cycling

Huang¹,

Yang²,

Huang³

et al. 2023

Preprint

View full text Add to dashboard Cite

<p>Birnessite occurs as a major manganese oxide in sediments. It is characterized by a high adsorption capacity for trace elements, including arsenic. However, the effects of birnessite on arsenic cycling were less intensively investigated than that of goethite, a commonly recognized arsenic host. Therefore, this study utilizes arsenic-bearing solutions containing 0.1&#8211;50 ppm arsenic to synthesize birnessite and uses arsenic sequential extraction procedure (SEP) analysis to quantify its arsenic co-precipitation and adsorption capacity.<br />&#160; &#160; SEPs showed that adsorbed/structural arsenic ratio grew from 0:100 to 60:40, with arsenic concentration increasing from 0.1&#8211;50 ppm, implying saturation of structural arsenic. SEPs quantify the adsorbed and structural As of birnessite being 0.831 and 1.308 mg/g, respectively. However, the low arsenic concentrations of < 1% in residual solutions indicate nonattainment of the maximum adsorption capacity. Subsequent adsorption experiments using higher initial arsenic concentrations reaching 250 ppm determined the maximum arsenic adsorption capacity to be 19.81 mg/g at pH 7, comparable to the values of 15.3&#8211;22.5 mg/g at pH 6.5 (Manning et al., 2002; Singh et al., 2010) and that of 3.79&#8211;15.73 mg/g for goethite. It then appears that birnessite adsorption/desorption is more effective than precipitation/dissolution in controlling arsenic cycling. However, the consideration based on the maximum adsorption capacity might overrate the controls of adsorption/desorption relative to precipitation/dissolution because the natural water generally contains less arsenic than that used for the adsorption experiments. Therefore, we conducted adsorption experiments with low arsenic concentrations of 0.5 ppm and a solid/liquid ratio of 0.01 to 0.001. The results showed that adsorption capacity rose abruptly to 1.416 mg/g in two weeks and then slowly increased to 1.523 mg/g after three months. This feature indicated incomplete filling of the adsorption sites on the surface of birnessite at low As concentration despite its large specific surface area. Another controlling factor was the abundance of birnessite in sediments. If 10% of MnO (0.05&#8211;0.15%) in the Chianan sediments in southern Taiwan occurred as birnessite, the adsorbed arsenic was calculated to be 0.0005&#8211;0.022 mg, corresponding to < 2 % of adsorbed arsenic in 1 g sediments, based on the SEP data of Yang et al. (2016). In the extreme case that all the Mn occurred as birnessite, birnessite could account for up to 25% of the adsorbed arsenic. Apparently, birnessite is not a major contributor to surface arsenic cycling. However, this inference must be evaluated by considering the adsorption and co-precipitation data from goethite and goethite proportions in sediments.</p>

show abstract

“…At the same time, there has been extensive work from the HCI community on how qualitative coders would like automation to assist them during the coding process (Marathe and Toyama, 2018;Feuston and Brubaker, 2021;Chen et al, 2018;Haug et al, 2021). These studies show that while researchers desire automation, they believe the coding agency should remain in their hands.…”

Section: Related Workmentioning

confidence: 99%

Qualitative Code Suggestion: A Human-Centric Approach to Qualitative Coding

Piano,

Rahimi,

Cheung

2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Qualitative coding is a content analysis method in which researchers read through a text corpus and assign descriptive labels or qualitative codes to passages. It is an arduous and manual process which human-computer interaction studies have shown could greatly benefit from NLP techniques to assist qualitative coders. Yet, previous attempts at leveraging language technologies have set up qualitative coding as a fully automatable classification problem. In this work, we take a more assistive approach by defining the task of qualitative code suggestion (QCS) in which a ranked list of previously assigned qualitative codes is suggested from an identified passage. In addition to being usermotivated, QCS integrates previously ignored properties of qualitative coding such as the sequence in which passages are annotated, the importance of rare codes and the differences in annotation styles between coders. We investigate the QCS task by releasing the first publicly available qualitative coding dataset, CVDQuoding, consisting of interviews conducted with women at risk of cardiovascular disease. In addition, we conduct a human evaluation which shows that our systems consistently make relevant code suggestions.

show abstract

Putting Tools in Their Place: The Role of Time and Perspective in Human-AI Collaboration for Qualitative Analysis

Cited by 26 publications

References 42 publications

Optimising Human-Machine Collaboration for Efficient High-Precision Information Extraction from Text Documents

Optimising Human-Machine Collaboration for Efficient High-Precision Information Extraction from Text Documents

Experiments of arsenic (V) adsorption on birnessite: implications on arsenic cycling

Qualitative Code Suggestion: A Human-Centric Approach to Qualitative Coding

Contact Info

Product

Resources

About