Anomalies represent rare observations (e.g., data records, messages or events) that are deviating significantly from others. Over the last forty years, researches on anomalies have received intensified interests and the burst of information has attracted more attention on anomalies because of their significance in a wide range of disciplines (e.g., computer science, chemistry, and biology). Anomaly detection, which aims to identify these rare observations, is among the most vital tasks and has shown its power in preventing detrimental events, such as financial fraud, network intrusion, and social spam, from happening. The detection task is typically solved by detecting outlying data points in the features space and inherently overlooks the structural information in real-world data. Graphs have been prevalently used to preserve the structural information, and this raises the graph anomaly detection problem -identifying anomalous graph objects (i.e., nodes, edges and sub-graphs). However, conventional anomaly detection techniques cannot well solve this problem because of the complexity of graph data (e.g., irregular structures, non-independent and large-scale). For the aptitudes of deep learning in breaking these limitations, graph anomaly detection with deep learning has received intensified studies recently. In this survey, we aim to provide a systematic and comprehensive review of the contemporary deep learning techniques for graph anomaly detection. Specifically, our categorization follows a task-driven strategy and classifies existing works according to the anomalous graph objects they can detect. We especially focus on the motivations, key intuitions and technical details of existing works. Moreover, we summarize open-sourced implementations, public datasets, and commonly-used evaluation metrics to provide affluent resources for future studies. Finally, we highlight twelve extensive future research directions according to our survey results covering emerging problems introduced by graph data, anomaly detection and real applications.