Bridges are the essential components in lifeline transportation systems, and their safe operation is of great importance. Information on structural damage could assist in timely repairs and reduce downtime. With the latest advancements in sensing technology, collecting vibration data from bridges has become more accessible. However, effective vibration processing is still a challenge, given the high dimensionality and massive size of vibration data. Existing studies have shown that machine/deep learning techniques can be valuable tools for this task. However, the learning and computational capacities of these models are challenged in the presence of large sensor arrays. We propose Trident as a novel deep learning framework that enables automatic damage feature extraction by simultaneously learning from temporal and three-dimensional (3D) spatial variations of 6D input data in instrumented bridges. Trident is equipped with 3 ConvLSTM3D branches to achieve this goal. A 3D steel truss bridge subject to dynamic traffic loads is monitored for its vibrations to evaluate Trident’s robustness in finding damaged elements. A damage dataset of 52,800 vehicle passing simulations is generated leveraging a database of 528 passenger vehicles in the United States, obtained from the National Highway and Traffic Safety Administration. Bayesian optimization is utilized to tune the model’s hyperparameters, achieving a test Node Average Geometric Mean Accuracy of 86%. This level of performance is promising given the high dimensionality and complexities of the output space in vibration-based monitoring. Trident’s concept can be extended to other vibration monitoring tasks with different time series data and damage labeling strategies.