Measuring, Quantifying, and Predicting the Cost-Accuracy Tradeoff

Baughman, Matt; Chakubaji, Nifesh; Truong, Hong Linh; Kreics, Krists; Chard, Kyle; Foster, Ian

doi:10.1109/bigdata47090.2019.9006370

Cited by 3 publications

(1 citation statement)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Dealing with trade-offs in ML is one of the important research directions [32,33]. In our previous work, we also have examined QoA for common ML pipelines [34]. The role of robustness, reliability, and elasticity for end-to-end ML pipelines has been studied in [6].…”

Section: Understanding the Quality Trade-offs In End-to-end Bim Objecmentioning

confidence: 99%

Understanding quality of analytics trade-offs in an end-to-end machine learning-based classification system for building information modeling

2021

Self Cite

View full text Add to dashboard Cite

Optimizing quality trade-offs in an end-to-end big data science process is challenging, as not only do we need to deal with different types of software components, but also the domain knowledge has to be incorporated along the process. This paper focuses on methods for tackling quality trade-offs in a common data science process for classifying Building Information Modeling (BIM) elements, an important task in the architecture, engineering, and construction industry. Due to the diversity and richness of building elements, machine learning (ML) techniques have been increasingly investigated for classification tasks. However, ML-based classification faces many issues, w.r.t. vast amount of data with heterogeneous data quality, diverse underlying computing configurations, and complex integration with industrial BIM tools, in an end-to-end BIM data analysis. In this paper, we develop an end-to-end ML classification system in which quality of analytics is considered as the first-class feature across different phases, from data collection, feature processing, training to ML model serving. We present our method for studying the quality of analytics trade-offs and carry out experiments with BIM data extracted from Solibri to demonstrate the automation of several tasks in the end-to-end ML classification. Our results have demonstrated that the quality of data, data extraction techniques, and computing configurations must be carefully designed when applying ML classifications for BIM in order to balance constraints of time, cost, and prediction accuracy. Our quality of analytics methods presents generic steps and considerations for dealing with such designs, given the time, cost, and accuracy trade-offs required in specific contexts. Thus, the methods could be applied to the design of end-to-end BIM classification systems using other ML techniques and cloud services.

show abstract

Section: Understanding the Quality Trade-offs In End-to-end Bim Objecmentioning

confidence: 99%