Physical foundations for relativistic spacetimes are revisited, in order to check at what extent Finsler spacetimes lie in their framework. Arguments based on inertial observers (as in the foundations of Special Relativity and Classical Mechanics) are shown to correspond with a double linear approximation in the measurement of space and time. While General Relativity appears by dropping the first linearization, Finsler spacetimes appear by dropping the second one. The classical Ehlers-Pirani-Schild approach is carefully discussed and shown to be compatible with the Lorentz-Finsler case. The precise mathematical definition of Finsler spacetime is discussed by using the space of observers. Special care is taken in some issues such as: the fact that a Lorentz-Finsler metric would be physically measurable only on the causal directions for a cone structure, the implications for models of spacetimes of some apparently innocuous hypotheses on differentiability, or the possibilities of measurement of a varying speed of light.MSC: 53C60, 83D05 83A05, 83C05.