This paper presents a mathematical formalism of non-Hermitian quantum mechanics, following the Dirac-von Neumann formalism of quantum mechanics. In this formalism, the state postulate is the same as in the Dirac-von Neumann formalism, but the observable postulate should be changed to include para-Hermitian operators (spectral operators of scalar type with real spectrum) representing observable, as such both the measurement postulate and the evolution postulate must be modified accordingly. This is based on a Stone type theorem as proved here that the dynamics of non-Hermitian quantum systems is governed by para-unitary time evolution. The novelty of this formalism is the Born formula on the expectation of an observable at a certain state, which is proved to be equal to the usual Born rule for every Hermitian observable, but for a non-Hermitian one it may depend on measurement via the choice of a metric operator associated with the non-Hermitian observable under measurement. This non-Hermitian Born formula satisfies probability conservation for both Hermitian and non-Hermitian observables. Our formalism is nether Hamiltonian-dependent nor basis-dependent, but can recover both PT-symmetric and biorthogonal quantum mechanics, and it reduces to the Dirac-von Neumann formalism of quantum mechanics in the Hermitian setting. As application, we study geometric phases for non-Hermitian quantum systems, focusing on the observable-geometric phase of a time-dependent non-Hermitian quantum system.