Recently, deep learning has been gradually applied to digital watermarking, which avoids the trouble of hand-designing robust transforms in traditional algorithms. However, most of the existing deep watermarking algorithms use encoder–decoder architecture, which is redundant. This paper proposes a novel audio watermarking algorithm based on adversarial perturbation, AAW. It adds tiny, imperceptible perturbations to the host audio and extracts the watermark with a pre-trained decoder. Moreover, the AAW algorithm also uses an attack simulation layer and a whitening layer to improve performance. The AAW algorithm contains only a differentiable decoder, so it reduces the redundancy. The experimental results also demonstrate that the proposed algorithm is effective and performs better than existing audio watermarking algorithms.