Exceptional points are known in the mathematical literature for many years. They are singular points at which (at least) two eigenvalues of an operator coalesce. In physics, they can be studied best when the Hamiltonian of the system is non-Hermitian. Although the points themselves can not be directly identified in physics, their strong influence onto the neighborhood can be traced. Here, the exceptional points are called mostly crossing points (of the eigenvalue trajectories) or branch points or double poles of the S matrix. In the present paper, first the mathematical basic properties of the exceptional points are discussed. Then, their role in the description of real physical quantum systems is considered (after solving the corresponding equations exactly). The Hamiltonian of these systems is non-Hermitian due to their embedding into an environment (continuum of scattering wavefunctions). Outside the energy window coupled directly to the continuum, the Hamiltonian is Hermitian but with corrections originating from the principal value integral of the coupling term via the continuum. Most interesting value of the non-Hermitian quantum physics is the phase rigidity of the eigenfunctions which varies (as function of a control parameter) between 1 (for distant non-overlapping states) and 0 (at the exceptional point where the resonance states completely overlap). This variation allows the system to incorporate environmentally induced effects. In the very neighborhood of a crossing (exceptional) point, the system can be described well by a conventional nonlinear Schrödinger equation. Here, the entanglement of the different states is large. In the regime of overlapping resonances, many eigenvalue trajectories cross or avoid crossing, and spectroscopic redistribution processes occur in the whole system. As a result, a dynamical phase transition takes place to which all states of the system contribute: a few short-lived resonance states are aligned to the scattering states of the environment by trapping the other states. The trapped resonance states are long-lived, show chaotic features, and are described well by means of statistical ensembles. Due to the alignment of a few states with the states of the environment, observable values (e.g. the transmission through the system) are enhanced. The dynamical phase transition breaks the spectroscopic relation of the short-lived and long-lived resonance states to the original individual states of the system. These results hold also for PT symmetric systems. The dynamical phase transition characteristic of non-Hermitian quantum physics, allows us to understand some experimental results which remained puzzling in the framework of conventional Hermitian quantum physics. The effects caused by the exceptional (crossing) points in physical systems allow us to manipulate them for many different applications.