A Survey of Domain Knowledge Elicitation in Applied Machine Learning

Kerrigan, Daniel; Hullman, Jessica; Bertini, Enrico

doi:10.3390/mti5120073

Cited by 18 publications

(7 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To achieve this, a documented process of literature search and expert consensus was undertaken to identify, and engineer candidate features and set operational priorities for the intended application. The taxonomy provided by Kerrigan et al ( 7 ) provided a useful framework to represent the elicitation process. In this study, domain knowledge from experts, a literature search and the researcher were integrated to achieve the goals of problem specification and feature selection prior to data collection, feature engineering and pre-processing.…”

Section: Discussionmentioning

confidence: 99%

“…This includes the integration of human knowledge in a field (domain knowledge) as well as knowledge of learning, the human brain, computer science and statistics (general knowledge) ( 6 ). Kerrigan et al have conducted a survey of the elicitation of domain knowledge in applied machine learning and pointed to the need for the documentation of the elicitation process ( 7 ). In doing so, they have developed a taxonomy for the elicitation process that includes the elicitation goal , the elicitation target , the elicitation process and the use of elicited knowledge .…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Elicitation of domain knowledge for a machine learning model for paediatric critical illness in South Africa

Pienaar

Sempa²,

Luwes³

et al. 2023

Front. Pediatr.

View full text Add to dashboard Cite

ObjectivesDelays in identification, resuscitation and referral have been identified as a preventable cause of avoidable severity of illness and mortality in South African children. To address this problem, a machine learning model to predict a compound outcome of death prior to discharge from hospital and/or admission to the PICU was developed. A key aspect of developing machine learning models is the integration of human knowledge in their development. The objective of this study is to describe how this domain knowledge was elicited, including the use of a documented literature search and Delphi procedure.DesignA prospective mixed methodology development study was conducted that included qualitative aspects in the elicitation of domain knowledge, together with descriptive and analytical quantitative and machine learning methodologies.SettingA single centre tertiary hospital providing acute paediatric services.ParticipantsThree paediatric intensivists, six specialist paediatricians and three specialist anaesthesiologists.InterventionsNone.Measurements and main resultsThe literature search identified 154 full-text articles reporting risk factors for mortality in hospitalised children. These factors were most commonly features of specific organ dysfunction. 89 of these publications studied children in lower- and middle-income countries. The Delphi procedure included 12 expert participants and was conducted over 3 rounds. Respondents identified a need to achieve a compromise between model performance, comprehensiveness and veracity and practicality of use. Participants achieved consensus on a range of clinical features associated with severe illness in children. No special investigations were considered for inclusion in the model except point-of-care capillary blood glucose testing. The results were integrated by the researcher and a final list of features was compiled.ConclusionThe elicitation of domain knowledge is important in effective machine learning applications. The documentation of this process enhances rigour in such models and should be reported in publications. A documented literature search, Delphi procedure and the integration of the domain knowledge of the researchers contributed to problem specification and selection of features prior to feature engineering, pre-processing and model development.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Elicitation of domain knowledge for a machine learning model for paediatric critical illness in South Africa

Pienaar

Sempa²,

Luwes³

et al. 2023

Front. Pediatr.

View full text Add to dashboard Cite

show abstract

“…Applications for machine learning research of this nature, invite close interdisciplinary collaboration that allows the integration of machine learning knowledge with clinical domain knowledge, not only of biomedicine but also of clinical context in a fashion that enhances the relevance and applicability of these new methods. The documentation of these processes can be seen to be an important aspect of rigour when publishing reports of such models and demands the investigation of this agenda ( 50 ).…”

Section: Discussionmentioning

confidence: 99%

Development of artificial neural network models for paediatric critical illness in South Africa

et al. 2022

View full text Add to dashboard Cite

ObjectivesFailures in identification, resuscitation and appropriate referral have been identified as significant contributors to avoidable severity of illness and mortality in South African children. In this study, artificial neural network models were developed to predict a composite outcome of death before discharge from hospital or admission to the PICU. These models were compared to logistic regression and XGBoost models developed on the same data in cross-validation.DesignProspective, analytical cohort study.SettingA single centre tertiary hospital in South Africa providing acute paediatric services.PatientsChildren, under the age of 13 years presenting to the Paediatric Referral Area for acute consultations.OutcomesPredictive models for a composite outcome of death before discharge from hospital or admission to the PICU.InterventionsNone.Measurements and main results765 patients were included in the data set with 116 instances (15.2%) of the study outcome. Models were developed on three sets of features. Two derived from sequential floating feature selection (one inclusive, one parsimonious) and one from the Akaike information criterion to yield 9 models. All developed models demonstrated good discrimination on cross-validation with mean ROC AUCs greater than 0.8 and mean PRC AUCs greater than 0.53. ANN1, developed on the inclusive feature-et demonstrated the best discrimination with a ROC AUC of 0.84 and a PRC AUC of 0.64 Model calibration was variable, with most models demonstrating weak calibration. Decision curve analysis demonstrated that all models were superior to baseline strategies, with ANN1 demonstrating the highest net benefit.ConclusionsAll models demonstrated satisfactory performance, with the best performing model in cross-validation being an ANN model. Given the good performance of less complex models, however, these models should also be considered, given their advantage in ease of implementation in practice. An internal validation study is now being conducted to further assess performance with a view to external validation.

show abstract

“…Comprehensive metadata about the meaning and data quality is typically not available and metadata management, including data catalogs, is still in the exploration stage [42]. Areas of related research include: domain knowledge elicitation [51], visual domain modeling tools [7], taxonomy development to aid problem specification, feature engineering, model development, and model evaluation [52]. But, studies exploring collaboration tools used in inter-disciplinary teams found that data scientists and domain experts [51], [53] experience adoption challenges because MDATs tend to settle on using tools that everybody is familiar with -rather than expend effort to use expert tools from their own discipline or use new and innovative tools that may be more effective [53].…”

Section: Literature Related To Findingsmentioning

confidence: 99%

Dealing with Data Challenges when Delivering Data-Intensive Software Solutions

Graetsch¹,

Khalajzadeh²,

Shahin³

et al. 2022

Preprint

View full text Add to dashboard Cite

The predicted increase in demand for data-intensive solution development is driving the need for software, data, and domain experts to effectively collaborate in multi-disciplinary data-intensive software teams (MDSTs). We conducted a socio-technical grounded theory study through interviews with 24 practitioners in MDSTs to better understand the challenges these teams face when delivering data-intensive software solutions. The interviews provided perspectives across different types of roles including domain, data and software experts, and covered different organisational levels from team members, team managers to executive leaders. We found that the key concern for these teams is dealing with data-related challenges. In this paper, we present the theory of dealing with data challenges to explain the challenges faced by MDSTs including gaining access to data, aligning data, understanding data, and resolving data quality issues; the context in and condition under which these challenges occur, the causes that lead to the challenges, and the related consequences such as having to conduct remediation activities, inability to achieve expected outcomes and lack of trust in the delivered solutions. We also identified contingencies or strategies applied to address the challenges including high-level strategic approaches such as implementing data governance, implementing new tools and techniques such as data quality visualisation and monitoring tools, as well as building stronger teams by focusing on people dynamics, communication skill development and cross-skilling. Our findings have direct implications for practitioners and researchers to better understand the landscape of data challenges and how to deal with them.

show abstract

A Survey of Domain Knowledge Elicitation in Applied Machine Learning

Cited by 18 publications

References 67 publications

Elicitation of domain knowledge for a machine learning model for paediatric critical illness in South Africa

Elicitation of domain knowledge for a machine learning model for paediatric critical illness in South Africa

Development of artificial neural network models for paediatric critical illness in South Africa

Dealing with Data Challenges when Delivering Data-Intensive Software Solutions

Contact Info

Product

Resources

About