Low-temperature magnetic and electric properties in hexagonal multiferroic compound YbMnO3 were studied. The Mn3+ spin moments order at TN = 85 K and reoriented around 43.5 K, leading to the magnetic phase transition from B2(P63cm) → A2(P63cm). The concomitant ferroelectric polarization is observed and explained microscopically by the destruction of initial symmetric relationship of the polarization between the upper and lower half of the magnetic unit cell. The asymmetry of the polarization vs temperature curves under opposite poling voltage revealed the pinning effect of the defects on the electrical polarization.