In the application of data driven Structural Damage Identification (SDI) based on supervised deep learning technology, valid data demarcation is the foundation; a convolutional neural network model with learning ability and capability of processing rich signal information is the core. Based on this understanding, this work makes three contributions: Firstly, the structural damage location and severity are jointly demarcated, and the SDI problem is transformed into a multi-classification task. Secondly, a 3D Signal processing Convolutional Neural Networks (3DS-CNN) is designed with an attempt to identify the complex and slight damages using the most basic network structure. Thirdly, a “Major and Subsidiary” Data Construction (MSDC) method integrating the key Intrinsic Mode Function (IMF) is proposed to construct 3D data. Then the proposed schemes are verified by two different structures. The results show that the 3DS-CNN has excellent damage identification ability for small-size data with noise pollution. MSDC method can enrich the feature information of the damage signals and help the network with deep feature excavation, even if the vibration signals are heavily polluted. Going one step further, the impact of sensor placement is discussed, and it is found that when external excitation is obvious, better SDI accuracy can be achieved even using a single sensor signal with slight noise. When the noise interference is obvious, the generalization ability and noise robustness of the network can be enhanced by optimizing sensor placement. In this case, the sensor placement criteria and the sensitive nodes of the structure should be comprehensively and carefully considered to avoid mutual “coupling” interference of data between sensors.