Data partition methodology for validation of predictive models

Morrison, Rebecca E.; Bryant, Corey; Terejanu, Gabriel; Prudhomme, Serge; Miki, Kenji

doi:10.1016/j.camwa.2013.09.006

Cited by 20 publications

(12 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is followed by Cohen kappa (J) coefficient analysis to compare low-risk group prediction between the criteria from other studies (maximal tumor diameter e2.0 cm with no m-PMI) and the predictive model suggested by this study. 16 All statistical analyses were performed using SPSS version 20.0 (IBM Inc., Chicago, Ill). P values G0.05 were considered statistically significant.…”

Section: Discussionmentioning

confidence: 99%

Assessment of Parametrial Involvement in Early Stages Cervical Cancer With Preoperative Magnetic Resonance Imaging

Chen

et al. 2018

International Journal of Gynecological Cancer

View full text Add to dashboard Cite

show abstract

Section: Discussionmentioning

confidence: 99%

Assessment of Parametrial Involvement in Early Stages Cervical Cancer With Preoperative Magnetic Resonance Imaging

Chen

et al. 2018

International Journal of Gynecological Cancer

View full text Add to dashboard Cite

show abstract

“…Beyond automated driving, there is generally little literature available regarding the split between test environments. Morrison et al [37], Terejanu [38] find an optimal split between physical data to be used for model calibration and validation. Mullins et al [39] similarly find the optimal number of calibration and validation tests considering both the costs at each test condition and the prediction uncertainty of the simulation model.…”

Section: Scenario Assignment Methodsmentioning

confidence: 99%

Model Validation and Scenario Selection for Virtual-Based Homologation of Automated Vehicles

et al. 2020

View full text Add to dashboard Cite

Due to the rapid progress in the development of automated vehicles over the last decade, their market entry is getting closer. One of the remaining challenges is the safety assessment and type approval of automated vehicles, as conventional testing in the real world would involve an unmanageable mileage. Scenario-based testing using simulation is a promising candidate for overcoming this approval trap. Although the research community has recognized the importance of safeguarding in recent years, the quality of simulation models is rarely taken into account. Without investigating the errors and uncertainties of models, virtual statements about vehicle safety are meaningless. This paper describes a whole process combining model validation and safety assessment. It is demonstrated by means of an actual type-approval regulation that deals with the safety assessment of lane-keeping systems. Based on a thorough analysis of the current state-of-the-art, this paper introduces two approaches for selecting test scenarios. While the model validation scenarios are planned from scratch and focus on scenario coverage, the type-approval scenarios are extracted from measurement data based on a data-driven pipeline. The deviations between lane-keeping behavior in the real and virtual world are quantified using a statistical validation metric. They are then modeled using a regression technique and inferred from the validation experiments to the unseen virtual type-approval scenarios. Finally, this paper examines safety-critical lane crossings, taking into account the modeling errors. It demonstrates the potential of the virtual-based safeguarding process using exemplary simulations and real driving tests.

show abstract

“…It includes conventional techniques such as combinatorial testing or statistical design of experiments (DoE), but also sophisticated techniques such as reinforcement learning or rapidly exploring random trees that can generate entire trajectories to test dynamic and formal models. Furthermore, there are partitioning techniques to determine an optimal data split [125,177]. Mullins et al [127] select calibration and validation scenarios so that costs and prediction uncertainties are minimized.…”

Section: Scenario Designmentioning

confidence: 99%

Unified Framework and Survey for Model Verification, Validation and Uncertainty Quantification

Riedmaier

Danquah

Schick

et al. 2020

Arch Computat Methods Eng

View full text Add to dashboard Cite

Simulation is becoming increasingly important in the development, testing and approval process in many areas of engineering, ranging from finite element models to highly complex cyber-physical systems such as autonomous cars. Simulation must be accompanied by model verification, validation and uncertainty quantification (VV&UQ) activities to assess the inherent errors and uncertainties of each simulation model. However, the VV&UQ methods differ greatly between the application areas. In general, a major challenge is the aggregation of uncertainties from calibration and validation experiments to the actual model predictions under new, untested conditions. This is especially relevant due to high extrapolation uncertainties, if the experimental conditions differ strongly from the prediction conditions, or if the output quantities required for prediction cannot be measured during the experiments. In this paper, both the heterogeneous VV&UQ landscape and the challenge of aggregation will be addressed with a novel modular and unified framework to enable credible decision making based on simulation models. This paper contains a comprehensive survey of over 200 literature sources from many application areas and embeds them into the unified framework. In addition, this paper analyzes and compares the VV&UQ methods and the application areas in order to identify strengths and weaknesses and to derive further research directions. The framework thus combines a variety of VV&UQ methods, so that different engineering areas can benefit from new methods and combinations. Finally, this paper presents a procedure to select a suitable method from the framework for the desired application.

show abstract

Data partition methodology for validation of predictive models

Cited by 20 publications

References 9 publications

Assessment of Parametrial Involvement in Early Stages Cervical Cancer With Preoperative Magnetic Resonance Imaging

Assessment of Parametrial Involvement in Early Stages Cervical Cancer With Preoperative Magnetic Resonance Imaging

Model Validation and Scenario Selection for Virtual-Based Homologation of Automated Vehicles

Unified Framework and Survey for Model Verification, Validation and Uncertainty Quantification

Contact Info

Product

Resources

About