Atmospheric pressure discharges in Ne, N2 and N2/O2 gases have been investigated with a novel high-voltage pulse power supply. Three characteristic types of discharge modes were observed: (1) diffuse glow discharge seen in Ne gas both with the eye and high-speed camera; (2) diffuse glow-like discharge in N2 gas observed with the eye, but seen as a vague filamentary discharge with a high-speed camera; and (3) clear filamentary discharge in N2/O2 gas mixture both with the eye and high-speed camera. Characteristics in the current and voltage waveforms revealed that the existence of the second voltage with opposite polarity following the first impulse voltage was important to have a diffuse glow-like discharge. This suggests that elimination of the memory effect by non-uniformly distributed charges on the surface is important for obtaining the diffuse mode because the non-uniformity triggers a positive feedback to the current constriction at the points with higher charge density, and results in filamentation of the discharge.