Mapping model validation metrics to subject matter expert scores for model adequacy assessment

Teferra, Kirubel; Shields, Michael D.; Hapij, Adam; Daddazio, Raymond P.

doi:10.1016/j.ress.2014.07.010

Cited by 19 publications

(6 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A bias can be introduced if the same experts provide the data to develop the model and perform this validation step. While it is appropriate to have the experts review the graphical model, it is best to have some experts not involved with developing the model review it as well . In this research, experts from the industry 6DOF working group at the Institute of Environmental Sciences and Technology's ESTECH 2016 conference were engaged to provide validation input .…”

Section: Building a Bn Model Based On Expert Priorsmentioning

confidence: 99%

Harnessing expert knowledge: Defining a Bayesian network decision model with limited data–Model structure for the vibration qualification problem

Rizzo

Blackburn

2018

Systems Engineering

View full text Add to dashboard Cite

As systems become more complex, systems engineers rely on experts to inform decisions. There are few experts and limited data in many complex new technologies. This challenges systems engineers as they strive to plan activities such as qualification in an environment where technical constraints are coupled with the traditional cost, risk, and schedule constraints. Bayesian network (BN) models provide a framework to aid systems engineers in planning qualification efforts with complex constraints by harnessing expert knowledge and incorporating technical factors. By quantifying causal factors, a BN model can provide data about the risk of implementing a decision supplemented with information on driving factors. This allows a systems engineer to make informed decisions and examine "what-if" scenarios. This paper discusses a novel process developed to define a BN model structure based primarily on expert knowledge supplemented with extremely limited data (25 data sets or less). The model was developed to aid qualification decisions-specifically to predict the suitability of six degrees of freedom (6DOF) vibration testing for qualification. The process defined the model structure with expert knowledge in an unbiased manner. Validation during the process execution and of the model provided evidence the process may be an effective tool in harnessing expert knowledge for a BN model. K E Y W O R D SBayesian network, decision model, qualification, structural knowledge assessment the effort to harness expert knowledge to create the model becomes critical. This paper discusses a novel approach developed to define a BN model structure. The BN model in this research, called the 6DOF Systems Engineering. 2018;21:285-294.

show abstract

Section: Building a Bn Model Based On Expert Priorsmentioning

confidence: 99%

Harnessing expert knowledge: Defining a Bayesian network decision model with limited data–Model structure for the vibration qualification problem

Rizzo

Blackburn

2018

Systems Engineering

View full text Add to dashboard Cite

show abstract

“…Output: Score, the total assessment score of system behavior Method: (1) while(readFileState == true){ // readFileState is the flag that the data are correctly read (2) x[] ← readFile(D); // read the file of system judging criterion Algorithm Explains: In Algorithm 2, Eq. (3), viz., 1 þ P s i¼2 Q s j¼i r j is computed form steps (1) to (8), where Q s j¼i r j is computed from steps (4) to (6); according to Eq.…”

Section: } the Relative Importance Set Between Objectivesmentioning

confidence: 99%

“…Therefore, as to the behavioral assessment, the research difficult is the complex behavior assessment of multiple elements variable, while the research hot spot of multiple elements variable behavioral assessment is the determination of relative weights of all the variables. The existing methods for determining the relative weights of multiple elements variable are expert assessment method [1,2], experience formula method [3,4], mean-variance method [5,6], and support-confidence method [7,8].…”

Section: Introductionmentioning

confidence: 99%

“…Expert assessment method is the method to conduct complex behavior assessment by expert scoring, whose accuracy is affected by the experience and the knowledge range and quality of the experts [1,2]. Therefore, the expert assessment method is use simple and intuitive, but it is with the drawbacks of too strong subjectivity, low assessment efficiency, lack theory, and insufficient systemization.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dynamic behavioral assessment model based on Hebb learning rule

Yin

Yuan

Zhang

2016

Neural Comput & Applic

View full text Add to dashboard Cite

Behavioral assessment based on computing system is with important value for computer-simulated training and system diagnosis. However, the existing assessment is a static method for ex post evaluation, and the low efficiency and high complexity have been the urgent problems to be solved in the academic field. In this paper, we propose an adaptive dynamic behavioral assessment model based on Hebb learning rule that effectively combines the assessment standard and the weights of factors. The dynamic behavioral assessment considers the relative weights between the assessment indexes, whereas the existing assessment method does not; the dynamic behavioral assessment uses the assessment standard data recursively and can conduct an instant assessment for the objectives. We have built an assessment system for computer-simulated training, and took the pilot behavioral assessment for example to test and verify the dynamic behavioral assessment mode. Experimental results show that the dynamic behavioral assessment model based on Hebb learning rule has more advantage in assessment efficiency and online computing support.

show abstract

“…The literature on methods for comparing time-histories is vast. Without being exhaustive, former studies can be found in field of mechanical systems (Jiang and Mahadevan 2010), cavitation in fluid-structure interaction (Sprague and Geers 2004), seismology (Kristeková et al 2009), geophysics (Willmott et al 1985), economics (Bliemel 1973), shock of floating platforms (Teferra et al 2014), crash simulations in vehicle designs (Sarin et al 2010), etc. For data-driven simulations of downburst winds, Wang et al (2013) have utilized a number of schemes for validation, including metrics like Rényi entropy and fractal dimension.…”

Section: Introductionmentioning

confidence: 99%

Comparison Metrics for Time-Histories: Application to Bridge Aerodynamics

2020

View full text Add to dashboard Cite

Wind effects can be critical for the design of lifelines such as long-span bridges. The existence of a significant number of aerodynamic force models, used to assess the performance of bridges, poses an important question regarding their comparison and validation. This study utilizes a unified set of metrics for a quantitative comparison of time-histories in bridge aerodynamics with a host of characteristics. Accordingly, nine comparison metrics are included to quantify the discrepancies in local and global signal features such as phase, time-varying frequency and magnitude content, probability density, nonstationarity and nonlinearity. Among these, seven metrics available in the literature are introduced after recasting them for time-histories associated with bridge aerodynamics. Two additional metrics are established to overcome the shortcomings of the existing metrics. The performance of the comparison metrics is first assessed using generic signals with prescribed signal features. Subsequently, the metrics are applied to a practical example from bridge aerodynamics to quantify the discrepancies in the aerodynamic forces and response based on numerical and semi-analytical aerodynamic models. In this context, it is demonstrated how a discussion based on the set of comparison metrics presented here can aid a model evaluation by offering deeper insight. The outcome of the study is intended to provide a framework for quantitative comparison and validation of aerodynamic models based on the underlying physics of fluid-structure interaction. Immediate further applications are expected for the comparison of time-histories that are simulated by data-driven approaches.

show abstract

Mapping model validation metrics to subject matter expert scores for model adequacy assessment

Cited by 19 publications

References 36 publications

Harnessing expert knowledge: Defining a Bayesian network decision model with limited data–Model structure for the vibration qualification problem

Harnessing expert knowledge: Defining a Bayesian network decision model with limited data–Model structure for the vibration qualification problem

Dynamic behavioral assessment model based on Hebb learning rule

Comparison Metrics for Time-Histories: Application to Bridge Aerodynamics

Contact Info

Product

Resources

About