Road damage such as potholes and cracks may reduce ride comfort and traffic safety. This influence can be prevented by regular, proper monitoring and maintenance of roads. Traditional methods and existing methods of surveying are very time-consuming, expensive, require a lot of human effort, and, thus, cannot be conducted frequently. A more efficient and cost-effective process is required to augment profilometer and traditional road-condition recognition systems. In this study, we propose deep-learning methods using smartphone data to devise a cost-effective and ad-hoc approach. Information from sensors on smartphones such as motion sensors and cameras are harnessed to detect road damage using deep-learning algorithms. In order to give heuristic and accurate information about the road damage, we used a cloud-based collaborative approach to fuse all the data and update a map frequently with these road-surface conditions. During the experiment, the deep-learning models achieved good prediction accuracy on our dataset, and the cloud-based fusion approach was able to group and merge the detections from different vehicles.