Learning to Validate the Predictions of Black Box Machine Learning Models on Unseen Data

Redyuk, Sergey; Schelter, Sebastian; Rukat, Tammo; Markl, Volker; Bießmann, Felix

doi:10.1145/3328519.3329126

Cited by 6 publications

(4 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Second of all, the fact that the accuracy for training and testing are virtually equal. Which means its accuracy on unseen data [25] is as good and there is no potential overfitting at all.…”

Section: Discussionmentioning

confidence: 97%

Automatic Data Extraction Utilizing Structural Similarity From A Set of Portable Document Format (PDF) Files

Satria,

Primanita

2023

SJIA

View full text Add to dashboard Cite

Instead of storing data in databases, common computer-aided office workers often choose to keep data related to their work in the form of document or report files that they can conveniently and comfortably access with popular off-the-shelf softwares, such as in Portable Document Format (PDF) format files. Their workplaces may actually use databases but they usually do not possess the privilege nor the proficiency to fully utilize them. Said workplaces likely have front-end systems such as Management Information System (MIS) from where workers get their data containing reports or documents.These documents are meant for immediate or presentational uses but workers often keep these files for the data inside which may come to be useful later on. This way, they can manipulate and combine data from one or more report files to suit their work needs, on the occasions that their MIS were not able to fulfill such needs. To do this, workers need to extract data from the report files. However, the files also contain formatting and other contents such as organization banners, signature placeholders, and so on. Extracting data from these files is not easy and workers are often forced to use repeated copy and paste actions to get the data they want. This is not only tedious but also time-consuming and prone to errors. Automatic data extraction is not new, many existing solutions are available but they typically require human guidance to help the data extraction before it can become truly automatic. They may also require certain expertise which can make workers hesitant to use them in the first place. A particular function of an MIS can produce many report files, each containing distinct data, but still structurally similar. If we target all PDF files that come from such same source, in this paper we demonstrated that by exploiting the similarity it is possible to create a fully automatic data extraction system that requires no human guidance. First, a model is generated by analyzing a small sample of PDFs and then the model is used to extract data from all PDF files in the set. Our experiments show that the system can quickly achieve 100% accuracy rate with very few sample files. Though there are occasions where data inside all the PDFs are not sufficiently distinct from each other resulting in lower than 100% accuracy, this can be easily detected and fixed with slight human intervention. In these cases, total no human intervention may not be possible but the amount needed can be significantly reduced.

show abstract

Section: Discussionmentioning

confidence: 97%

Automatic Data Extraction Utilizing Structural Similarity From A Set of Portable Document Format (PDF) Files

Satria,

Primanita

2023

SJIA

View full text Add to dashboard Cite

show abstract

“…This is due to the fact that larger data sets include more information. This challenge is not insurmountable; nonetheless, mathematics condensing and dimensionality reduction are both required to solve it [2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18]. Athletes may have specialized sensors such as gyroscopes, magnetometers, accelerometers, and infrared sensors and be connected to them so that data can be collected in the field of sports medicine.…”

Section: A Knnmentioning

confidence: 99%

“…Model validation and assessment may take place after training has been completed. Validation and assessment must adhere to a number of prerequisites for success, including the use of separate data sets for training and testing, the use of an acceptable error measure, the use of simulated data when working with smaller data sets, and an awareness of typical errors that might occur when working with ML [11][12][13]. The K-fold cross-validation method is the gold standard for validation right now.…”

Section: Introductionmentioning

confidence: 99%

Injury Prediction in Sports using Artificial Intelligence Applications: A Brief Review

Kumar,

Reddy

et al. 2023

Journal of Robotics and Control

View full text Add to dashboard Cite

Avoiding injuries in sports has always depended on historical records and human experience. This is despite using injuries being a major and unsolvable issue. The development of more precise preventative procedures using the now available approaches has been excruciatingly sluggish. The development of artificial intelligence (AI) and machine learning (ML) as potentially valuable procedures to improve damage prevention and recovery procedures has been made possible by technological advances that have made these areas more accessible. This article presents a detailed summary of ML approaches as they have been used to predict and anticipate sports injuries to this point in time. The research conducted over the last five years has been collated, and its results have been untaken. Assuming the present absence of accessible sources, standardized statistics, and a dependence on obsolete deterioration prototypes, it is impossible to draw any definitive conclusions regarding the real-world effectiveness of machine learning in terms of its application to the prediction of sports injuries. However, it has been hypothesized that resolving these two problems would make it possible to deploy innovative, strong machine-learning architectures, which will hasten the process of increasing the state of this area while also offering proven clinical tools.

show abstract

“…Prior work exists on performance prediction (Guerra, Prudêncio, and Ludermir 2008;Chen et al 2019;Finn et al 2019;Redyuk et al 2019;Schat et al 2020;Talagala, Li, and The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21)…”

Section: Related Workmentioning

confidence: 99%

Learning Prediction Intervals for Model Performance

Elder¹,

Arnold²,

Murthi³

et al. 2021

AAAI

View full text Add to dashboard Cite

Understanding model performance on unlabeled data is a fundamental challenge of developing, deploying, and maintaining AI systems. Model performance is typically evaluated using test sets or periodic manual quality assessments, both of which require laborious manual data labeling. Automated performance prediction techniques aim to mitigate this burden, but potential inaccuracy and a lack of trust in their predictions has prevented their widespread adoption. We address this core problem of performance prediction uncertainty with a method to compute prediction intervals for model performance. Our methodology uses transfer learning to train an uncertainty model to estimate the uncertainty of model performance predictions. We evaluate our approach across a wide range of drift conditions and show substantial improvement over competitive baselines. We believe this result makes prediction intervals, and performance prediction in general, significantly more practical for real-world use.

show abstract

Learning to Validate the Predictions of Black Box Machine Learning Models on Unseen Data

Cited by 6 publications

References 2 publications

Automatic Data Extraction Utilizing Structural Similarity From A Set of Portable Document Format (PDF) Files

Automatic Data Extraction Utilizing Structural Similarity From A Set of Portable Document Format (PDF) Files

Injury Prediction in Sports using Artificial Intelligence Applications: A Brief Review

Learning Prediction Intervals for Model Performance

Contact Info

Product

Resources

About