In this paper, a decision support system is proposed to assist an analyst in updating the highway roadside asset inventory. The feasibility of the system is tested with assets along an 8 km section of the A27 highway on the south coast of England, UK. Survey data from a vehicle equipped with a single forward-facing camera and a GPS-enabled inertial measurement unit, aerial imagery of the highway, and the asset inventory are fused to develop the system. The camera on the vehicle is calibrated so that assets may be automatically located within the survey images. The assets are then classified by a state-of-the-art convolutional neural network. Therefore, those assets recorded correctly in the inventory and those needing further manual inspection are automatically identified. Three different asset types are considered (traffic signs, matrix signs, and reference marker posts), and overall 91% of the assets in a withheld test set are verified automatically. Thus the analyst is presented with a much smaller set of assets for which the inventory is incorrect and which require further inspection. We therefore demonstrate the value in fusing multiple data sources to develop decision support systems for transportation asset monitoring.