We consider elastic reflection and transmission of electrons by a disordered system characterized by a 2N × 2N scattering matrix S. Expressing S in terms of the N radial parameters and of the four N×N unitary matrices used for the standard transfer matrix parametrization, we calculate their probability distributions for the circular orthogonal (COE) and unitary (CUE) Dyson ensembles. In this parametrization, we explicitely compare the COE-CUE distributions with those suitable for quasi-1d conductors and insulators. Then, returning to the usual eigenvalue-eigenvector parametrization of S, we study the distributions of the scattering phase shifts. For a quasi-1d metallic system, microscopic simulations show that the phase shift density and correlation functions are close to those of the circular ensembles. When quasi-1d longitudinal localization breaks S into two uncorrelated reflection matrices, the phase shift form factor b(k) exhibits a crossover from a behavior characteristic of two uncoupled COE-CUE (small k) to a single COE-CUE behavior (large k). Outside quasi-one dimension, we find that the phase shift density is no longer uniform and S remains nonzero after disorder averaging. We use perturbation theory to calculate the deviations to the isotropic Dyson distributions. When the electron dynamics is no longer zero dimensional in the transverse directions, small-k corrections to the COE-CUE behavior of b(k) appear, which are reminiscent of the dimensionality dependent non universal regime of energy level statistics. Using a known relation between the scattering phase shifts and the system energy levels, we analyse those corrections to the universal random matrix behavior of S which result from d-dimensional diffusion on short time scales.