The accuracy of spectroscopic ellipsometry studies of transparent polymer films in the visible wavelength range is greatly reduced in the presence of a liquid ambient. The relatively high refractive index of a liquid ambient strongly amplifies the effects of experimental non-idealities, such as window birefringence and an unknowingly inaccurate description of the optical properties of the ambient. In particular for thin films (approximately <100 nm), the implications of even small non-idealities are substantial, and large deviations in the thickness and refractive index are found. The concurrent low root mean square error (MSE) values show that a good MSE does not show that the optical model is correct; it just shows a good agreement between the experimental and fitted data. This implies that quantitative in situ ellipsometry studies for thin films are intricate, and derived properties like sorption behaviour should be treated with caution. Here, we show that quantitative in situ ellipsometry swelling measurements in solvents require a very accurate solvent dispersion relation, a high-quality temperature control, especially to prevent temperature differences between the cell, its content and exterior ambient, and accurate quantification of window effects, in particular for polymer films with thicknesses of approximately below 100 nm, depending on the optical contrast of the polymer-solvent system.