Modelling the projected separation of microlensing events using systematic time-series feature engineering

Kennedy, Alex. Mills; Nash, Gemma; Rattenbury, N. J.; Kempa-Liehr, Andreas W.

doi:10.1016/j.ascom.2021.100460

Cited by 16 publications

(5 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Loading and integrating these data can be challenging, and it is an ongoing process, even after the machine learning model has been deployed and an initial master database has been constructed (continuous integration). All these data are in raw format and require transformation (cleaning), i.e., treating missing values (i.e., imputation [53,54], complete entry removal, matrix completion [55][56][57][58][59] depending on the randomness in the missing data pattern or by producing the missing data with simulations [60]), correcting for data formats (mathematical representation of numbers, time stamps, inconsistent text format, size and resolution of images, etc. ), de-duplicating [61] (with or without a unique key per real-world entry) or removing redundant data, dealing with contradicting data, removing outliers (point outliers, collective or contextual outliers), etc.…”

Section: Data Pipeline: Extraction Loading and Transformationmentioning

confidence: 99%

Fundamental Components and Principles of Supervised Machine Learning Workflows with Numerical and Categorical Data

Kampezidou,

Tikayat Ray,

Bhat

et al. 2024

Eng

View full text Add to dashboard Cite

This paper offers a comprehensive examination of the process involved in developing and automating supervised end-to-end machine learning workflows for forecasting and classification purposes. It offers a complete overview of the components (i.e., feature engineering and model selection), principles (i.e., bias–variance decomposition, model complexity, overfitting, model sensitivity to feature assumptions and scaling, and output interpretability), models (i.e., neural networks and regression models), methods (i.e., cross-validation and data augmentation), metrics (i.e., Mean Squared Error and F1-score) and tools that rule most supervised learning applications with numerical and categorical data, as well as their integration, automation, and deployment. The end goal and contribution of this paper is the education and guidance of the non-AI expert academic community regarding complete and rigorous machine learning workflows and data science practices, from problem scoping to design and state-of-the-art automation tools, including basic principles and reasoning in the choice of methods. The paper delves into the critical stages of supervised machine learning workflow development, many of which are often omitted by researchers, and covers foundational concepts essential for understanding and optimizing a functional machine learning workflow, thereby offering a holistic view of task-specific application development for applied researchers who are non-AI experts. This paper may be of significant value to academic researchers developing and prototyping machine learning workflows for their own research or as customer-tailored solutions for government and industry partners.

show abstract

Section: Data Pipeline: Extraction Loading and Transformationmentioning

confidence: 99%

Fundamental Components and Principles of Supervised Machine Learning Workflows with Numerical and Categorical Data

Kampezidou,

Tikayat Ray,

Bhat

et al. 2024

Eng

View full text Add to dashboard Cite

show abstract

“…treating missing values (i.e. imputation [52,53], complete entry removal, matrix completion [54][55][56][57][58] depending on the randomness in the missing data pattern or by producing the missing data with simulations [59]), correcting for data formats (mathematical representation of numbers, time stamps, inconsistent text format, size and resolution of images, etc), de-duplicating [60] (with or without a unique key per real-world entry) or removing redundant data, dealing with contradicting data, removing outliers (point outliers, collective or contextual outliers), etc. Current industry practices for data engineering, include ELT tools (Extract, Load, Transform) [61], which offer data extraction, data integration, and data transformation functions via some Application Programming Interface (API).…”

Section: Data Pipeline: Extraction Loading and Transformationmentioning

confidence: 99%

Fundamental Components and Principles of Supervised Machine Learning Workflows with Numerical and Categorical Data

Kampezidou,

Tikayat Ray,

Bhat

et al. 2023

Preprint

View full text Add to dashboard Cite

This paper offers a comprehensive examination of the process involved in developing and automating supervised end-to-end machine learning workflows for forecasting and classification purposes. It offers a complete overview of the components (i.e. feature engineering, model selection, etc), principles (i.e. bias-variance decomposition, model complexity, overfitting, model sensitivity to feature assumptions and scaling, output interpretability, etc), models (i.e. neural networks, regression models, etc), methods (i.e. Cross-Validation, data augmentation, etc), metrics (i.e. Mean Squared Error, F1-score, etc) and tools that rule most supervised learning applications with numerical and categorical data, as well as their integration, automation, and deployment. The end goal and contribution of this paper is the education and guidance of the non-AI expert academic community over complete and rigorous machine learning pipelines and data science practices, from problem scoping to design and state-of-the-art automation tools, including basic principles and reasoning in the choice of methods. The paper delves into the critical stages of supervised machine learning workflow development, many of which are often omitted by researchers due to brevity, and covers foundational concepts essential for understanding and optimizing a functional machine learning workflow, thereby offering a holistic view of task-specific application development for applied researchers who are not AI experts.

show abstract

“…Random Forest has generated great interest in recent astrophysical and cosmological analyses (e.g. Bonjean et al 2019;Hernandez Vivanco et al 2020;Kennedy et al 2021), because it achieves high accuracy and efficiency with large data sets, whilst being quick to implement. Additionally, at the training stage, RF is able to learn in a fast way highly non-linear relations between the inputs and the outputs it is given.…”

Section: Machine Learning Approachmentioning

confidence: 99%

Retrieving cosmological information from small-scale CMB foregrounds

Douspis

Salvati

Gorce

et al. 2022

A&A

View full text Add to dashboard Cite

We propose a new analysis of small-scale cosmic microwave background (CMB) data by introducing the cosmological dependency of the foreground signals, focussing first on the thermal Sunyaev-Zel’dovich (tSZ) power spectrum, derived from the halo model. We analyse the latest observations by the South Pole Telescope (SPT) of the high-ℓ power (cross) spectra at 95, 150, and 220 GHz, as the sum of CMB and tSZ signals, both depending on cosmological parameters and remaining contaminants. In order to perform faster analyses, we propose a new tSZ modelling based on machine learning algorithms (namely Random Forest). We show that the additional information contained in the tSZ power spectrum tightens constraints on cosmological and tSZ scaling relation parameters. We combined for the first time the Planck tSZ data with SPT high-ℓ to derive new constraints. Finally, we show how the amplitude of the remaining kinetic SZ power spectrum varies depending on the assumptions made on both tSZ and cosmological parameters. These results show the importance of a thorough modelling of foregrounds in the cosmological analysis of small-scale CMB data. Reliable constraints on cosmological parameters can only be achieved once other significant foregrounds, such as the kinetic SZ and the cosmic infrared background (CIB), are also properly accounted for.

show abstract

Modelling the projected separation of microlensing events using systematic time-series feature engineering

Cited by 16 publications

References 30 publications

Fundamental Components and Principles of Supervised Machine Learning Workflows with Numerical and Categorical Data

Fundamental Components and Principles of Supervised Machine Learning Workflows with Numerical and Categorical Data

Fundamental Components and Principles of Supervised Machine Learning Workflows with Numerical and Categorical Data

Retrieving cosmological information from small-scale CMB foregrounds

Contact Info

Product

Resources

About