Most matter in the Universe is invisible and unknown, which is called dark matter. A candidate of dark matter is axion, which is an ultra-light particle motivated as a solution for the CP problem. Axions form clouds in a galactic halo, amplify, and delay a part of gravitational waves propagating in the clouds. The Milky Way is surrounded by the dark matter halo composed of a number of axion patches. Thus, the characteristic secondary gravitational waves are always expected right after the reported gravitational-wave signals from compact binary mergers. In this paper, we derive a realistic amplitude of the secondary gravitational waves. Then we search the gravitational waves having the characteristic time delay and duration with a method optimized for them. We find no significant signal. Assuming the axions are dominant component of dark matter, we obtain the constraints on the axion coupling to the parity violating sector of gravity for the mass range, [1.7 × 10 −13 , 8.5 × 10 −12 ] eV, which is at most ∼ 10 times stronger than that from Gravity Probe B.