In a normal human life span, the heart beats about 2 to 3 billion times. Under diseased conditions, a heart may lose its normal rhythm and degenerate suddenly into much faster and irregular rhythms, called arrhythmias, which may lead to sudden death. The transition from a normal rhythm to an arrhythmia is a transition from regular electrical wave conduction to irregular or turbulent wave conduction in the heart, and thus this medical problem is also a problem of physics and mathematics. In the last century, clinical, experimental, and theoretical studies have shown that dynamical theories play fundamental roles in understanding the mechanisms of the genesis of the normal heart rhythm as well as lethal arrhythmias. In this article, we summarize in detail the nonlinear and stochastic dynamics occurring in the heart and their links to normal cardiac functions and arrhythmias, providing a holistic view through integrating dynamics from the molecular (microscopic) scale, to the organelle (mesoscopic) scale, to the cellular, tissue, and organ (macroscopic) scales. We discuss what existing problems and challenges are waiting to be solved and how multi-scale mathematical modeling and nonlinear dynamics may be helpful for solving these problems.