Knowledge Graph Programming with a Human-in-the-Loop

Lou, Yuze; Uddin, Mahfus; Brown, Nathaniel; Cafarella, Michael

doi:10.1145/3328519.3329132

Cited by 4 publications

(2 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Work in natural language processing (NLP) has studied how entity resolution can be learned and improved over time through user interactions [20]. A number of tools [21], [22] help people model queries into knowledge graphs [23], [24]; however, there are scale limitations for large, complex, and heterogeneous structures [25]. Other automated approaches feature interactive programming interfaces [26], equi-join-able tables [27], and deep learning approaches to entity resolution [28].…”

Section: Data Integrationmentioning

confidence: 99%

Preliminary Guidelines For Combining Data Integration and Visual Data Analysis

Coscia,

Suh,

Chang

et al. 2024

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

Data integration is often performed to consolidate information from multiple disparate data sources during visual data analysis. However, integration operations are usually separate from visual analytics operations such as encode and filter in both interface design and empirical research. We conducted a preliminary user study to investigate whether and how data integration should be incorporated directly into the visual analytics process. We used two interface alternatives featuring contrasting approaches to the data preparation and analysis workflow: manual file-based ex-situ integration as a separate step from visual analytics operations; and automatic UI-based in-situ integration merged with visual analytics operations. Participants were asked to complete specific and free-form tasks with each interface, browsing for patterns, generating insights, and summarizing relationships between attributes distributed across multiple files. Analyzing participants' interactions and feedback, we found both task completion time and total interactions to be similar across interfaces and tasks, as well as unique integration strategies between interfaces and emergent behaviors related to satisficing and cognitive bias. Participants' time spent and interactions revealed that in-situ integration enabled users to spend more time on analysis tasks compared with ex-situ integration. Participants' integration strategies and analytical behaviors revealed differences in interface usage for generating and tracking hypotheses and insights. With these results, we synthesized preliminary guidelines for designing future visual analytics interfaces that can support integrating attributes throughout an active analysis process.

show abstract

Section: Data Integrationmentioning

confidence: 99%

Preliminary Guidelines For Combining Data Integration and Visual Data Analysis

Coscia,

Suh,

Chang

et al. 2024

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

show abstract

“…It incorporates human feedback at each iteration to improve performance and control the semantic drift using human feedback. Many researchers add humans to natural language processing tasks(such as entity analysis, knowledge graphs, and so on) by using crowdsourcing [54,55,57,59]. Ristoski et al [66] propose a method of extracting instances from various web resources.…”

Section: Data Preprocessingmentioning

confidence: 99%

A Survey of Human-in-the-loop for Machine Learning

Wu,

Xiao,

Sun

et al. 2021

Preprint

View full text Add to dashboard Cite

Human-in-the-loop aims to train an accurate prediction model with minimum cost by integrating human knowledge and experience. Humans can provide training data for machine learning applications and directly accomplish some tasks that are hard for computers in the pipeline with the help of machine-based approaches. In this paper, we survey existing works on human-in-the-loop from a data perspective and classify them into three categories with a progressive relationship: (1) the work of improving model performance from data processing, (2) the work of improving model performance through interventional model training, and (3) the design of the system independent human-in-the-loop. Using the above categorization, we summarize major approaches in the field, along with their technical strengths/ weaknesses, we have simple classification and discussion in natural language processing, computer vision, and others. Besides, we provide some open challenges and opportunities. This survey intends to provide a high-level summarization for human-in-the-loop and motivates interested readers to consider approaches for designing effective human-in-the-loop solutions.

show abstract