Overfitting Affects the Reliability of Radial Velocity Mass Estimates of the V1298 Tau Planets

Abstract

Mass, radius, and age measurements of young (≲100 Myr) planets have the power to shape our understanding of planet formation. However, young stars tend to be extremely variable in both photometry and radial velocity (RV) measurements, which makes constraining these properties challenging. The V1298 Tau system of four ∼0.5 R J planets transiting a pre-main-sequence star presents an important, if stress-inducing, opportunity to observe and measure directly the properties of infant planets. Suárez Mascareño et al. published radial-velocity-derived masses for two of the V1298 Tau planets using a state-of-the-art Gaussian process regression framework. The planetary densities computed from these masses were surprisingly high, implying extremely rapid contraction after formation in tension with most existing planet-formation theories. In an effort to constrain further the masses of the V1298 Tau planets, we obtained 36 RVs using Keck/HIRES, and analyzed them in concert with published RVs and photometry. Through performing a suite of cross-validation tests, we found evidence that the preferred model of Suárez Mascareño et al. suffers from overfitting, defined as the inability to predict unseen data, rendering the masses unreliable. We detail several potential causes of this overfitting, many of which may be important for other RV analyses of other active stars, and recommend that additional time and resources be allocated to understanding and mitigating activity in active young stars such as V1298 Tau.

The TESS-Keck Survey. XVII. Precise Mass Measurements in a Young, High-multiplicity Transiting Planet System Using Radial Velocities and Transit Timing Variations

“…We perform cross-validation to assess our model's predictive ability on data it has not seen before. Ideally, we would follow the method proposed in Blunt et al (2023), reserving 30% of our RV data as a "test data set" and only training our model on 70%. We could, at fixed intervals, check our model's predictive ability and determine when the test likelihood starts worsening.…”

“…Additionally, our model requires large amounts of RAM to manipulate the long (300,000 steps) and wide (1000 chains) samples object, and our access to specialized high-memory CPUs is additionally limited. Because of these constraints, we make a compromise between a simpler cross-validation utilized in Hara et al (2020) and the more complicated method utilized in Blunt et al (2023).…”

The TESS-Keck Survey. XVI. Mass Measurements for 12 Planets in Eight Systems

“…The AIC favors the one-planet plus GP model over the (adopted) one-planet model, which itself is favored over the two-planet model (all at the ΔAIC >10 level). It should be noted, however, that GPs can be susceptible to overfitting (e.g., Blunt et al 2023), which can muddle the interpretability of Bayesian model comparison statistics. In this context, the AIC's preference for the GP-enabled model is not entirely surprising.…”

“…For completeness, we explored alternative models of the HD 6061 observations to check for model overfitting, which can plague GP-based planet mass measurements (e.g., Blunt et al 2023). A joint model of the photometry and RVs that did not include a GP and did not assume a circular orbit for HD 6061 b finds M b =7.8 ± 2.9 M ⊕ and that e b is consistent with zero.…”

The California-Kepler Survey. XI. A Survey of Chromospheric Activity through the Lens of Precise Stellar Properties

“…In addition to the search for direct correlations between RVs and stellar activity cycles over the timescale of decades (Wright et al 2008;Fulton et al 2015), sophisticated analyses such as the FF′ method, Gaussian processes, and other signal processing algorithms can be employed on data spanning shorter timescales to disentangle the complex relationships that planets have on stars gravitationally from stellar activity due to surface inhomogeneities (Aigrain et al 2012;Howard et al 2013;Pepe et al 2013). However, sophisticated signal processing techniques can sometimes overfit the data, and their results should be interpreted with caution (Blunt et al 2023). When additional information beyond stellar spectra are available, such as spacebased photometry from Kepler, K2, or the Transiting Exoplanet Survey Satellite (TESS; Kosiarek & Crossfield 2020), ever smaller planets can be characterized with RVs from instruments such as the High Resolution Echelle Spectrometer (HIRES; Akana Murphy et al 2021) or refuted with instruments such as Habitable Planet Finder (Lubin et al 2021).…”