Formulating a rigorous system-bath partitioning approach remains an open issue. In this context the famous Caldeira-Leggett model that enables quantum and classical treatment of Brownian motion on equal footing has enjoyed popularity. Although this model is by any means a useful theoretical tool, its ability to describe anharmonic dynamics of real systems is often taken for granted. In this Letter we show that the mapping between a molecular system under study and the model cannot be established in a self-consistent way, unless the system part of the potential is taken effectively harmonic. Mathematically, this implies that the mapping is not invertible. This 'invertibility problem' is not dependent on the peculiarities of particular molecular systems under study and is rooted in the anharmonicity of the system part of the potential.PACS numbers: 78.20. Bh, 33.20.Ea, 05.40.Jc Introduction and Theory. Model systems play an important role for our understanding of complex manybody dynamics. Reducing the description to a few parameters can not only ease the interpretation, but enable the identification of key properties [1]. In condensed phase dynamics the spin-boson [2] and Caldeira-Leggett (CL) [3, 4] models have been the conceptual backbone of countless studies [5]. The latter has been extended to describe linear and non-linear spectroscopy within the second-order cumulant approximation, termed multimode Brownian oscillator model in this context [6,7]. It has become a popular tool for assigning spectroscopic signals in the last two decades.The CL model comprises an arbitrary system (coordinates x, potential V S (x)), which is bi-linearly coupled to a bath of harmonic oscillators (coordinates Q i , potential V B ({Q i })) via a system-bath coupling potential V S−B (x, {Q i }) [3]. Later Caldeira and Leggett extended the model to an arbitrary function of system coordinates in the coupling and motivated the linearity of the coupling on the bath side [4]. Here, we limit ourselves to the bi-linear version for the reasons that will become apparent later. Restricting ourselves to a one-dimensional case yields [5]