The phase problem in transmission electron microscopy emerges through the conventional way of detecting intensities. Consequently, it is not possible to deduce the expected intensity at an arbitrary electron optical plane below the object from the measured intensity at the detector. In case of an elastically scattered electron, additional access to its phase suffices to construct a wave in the sense of a pure state, which can be propagated numerically between different electron optical planes. The case of inelastic scattering complicates the situation much more, because both scattering and detection process lead inevitably to decoherence (see e.g. [1]). Since a wave is fully coherent per definition, it is insufficient to describe the inelastically scattered electron by a wave. Here, the beam electron transfers from a pure state into a mixed state, which has to be described e.g. by a reduced density matrix. If this density matrix is known at the detector plane, it is known for all other planes (below the object). The diagonal elements of this matrix directly correspond to the measured intensity and are accessible by means of conventional TEM. The off-diagonal elements describe the coherence of the scattered electron hence require an interferometric approach for their experimental determination. Finally, the significance of the off-diagonal elements of the density matrix for inelastically scattered electrons is comparable to that of the phases for the case of pure elastic scattering. Off-axis electron holography in combination with energy filtering [2, 3] realizes experimental access to the off-diagonal elements by superposition of different partial beams in dependence on their spatial separation (shear d, see fig. 1a and 1b) [4]. The contrast of the resulting fringe pattern depends on the coherence between the interfering beams. This coherence is determined by the scattering process with the object [5,6], but also depends on the partial coherence of the beam electron ensemble [7]. Fresnel diffraction at the biprism rim [8] as well as aberrations of the objective lens have a share in the formation of the detected fringe pattern. Thus in general, a direct interpretation in terms of reduced density matrix is complicated. To identify special interpretable cases, we derive and analyze a transfer theory for this holographic measurement setup based on a generalization of the transmission cross coefficient. It turns out that for samples homogeneous perpendicular to the biprism and for small scattering angles, the effect of diffraction at the biprism and partial coherence of the electron beam ensemble can be separated from object influences. That allows a direct and model-independent procedure to determine the density matrix of a scattered electron ( fig. 1c and 1d). Limitations are given by the finite biprism diameter providing lower bounds for the shear and, if applicable, by aberrations of the energy filter. We apply this method for the investigation of the influence of surface plasmons on the coherence of the scattere...