The ab initio thermodynamic simulation of correlated Fermi systems is of central importance for many applications, such as warm dense matter, electrons in quantum dots, and ultracold atoms. Unfortunately, path integral Monte Carlo (PIMC) simulations of fermions are severely restricted by the notorious fermion sign problem (FSP). In this work, we present a hands-on discussion of the FSP and investigate in detail its manifestation with respect to temperature, system size, interactionstrength and -type, and the dimensionality of the system. Moreover, we analyze the probability distribution of fermionic expectation values, which can be non-Gaussian and fat-tailed when the FSP is severe. As a practical application, we consider electrons and dipolar atoms in a harmonic confinement, and the uniform electron gas in the warm dense matter regime. In addition, we provide extensive PIMC data, which can be used as a reference for the development of new methods and as a benchmark for approximations.