We discuss the spectral property of unstable dynamical systems in both classical and quantum mechanics. An important class of unstable dynamical systems corresponds to the Large Poincare Systems (LPS). Conventional perturbation technique leads then to divergences. We introduce methods for the elimination of Poincare divergences to obtain a solution of the spectral problem analytic in the coupling constant. To do so, we have to enlarge the class of permissible transformations, to include non-unitary transformations as well as to extend the Hilbert space. A simple example refers to the Friedrichs model, which was studied independently by George Sudarshan and his co-workers. However, our main interest is the irreducible representations in the Liouville space. In these representations the central quantity is the density matrix, and the eigenfunctions of the Liouville operator cannot be expressed in terms of the wave functions. We suggest that this situation corresponds to quantum chaos. Indeed, classical chaos does not mean that Newton's equation becomes "wrong" but that trajectories loose their operational meaning. Similarly, whenever we have an irreducible representation in the Liouville space this means that the wave function description looses its operational meaning. Additional statistical features appear. A simple example corresponds to persistent interactions in the scattering problem which cannot be treated in the frame of usual S-matrix theory.