The aim of this paper is to classify and segment roofs using vertical aerial imagery to generate three-dimensional (3D) models. Such models can be used, for example, to evaluate the rainfall runoff from properties for rainwater harvesting and in assessing solar energy and roof insulation options. Aerial orthophotos and building footprints are used to extract individual roofs and bounding boxes, which are then fed into one neural network for classification and then another for segmentation. The approach initially implements transfer learning on a pre-trained VGG16 model. The first step achieves an accuracy of 95.39% as well as a F1 score of 95%. The classified images are segmented using a fully convolutional network semantic segmentation model. The mask of the segmented roof planes is used to extract the coordinates of the roof edges and the nexus points using the Harris corner detector algorithm. The coordinates of the corners are then used to plot a 3D Level of Detail 2 (LOD2) representation of the building and the roof height is determined by calculating the maximum and minimum height of a Digital Surface Model LiDAR point cloud and known building height data. Subsequently the wireframe plot can be used to compute the roof area. This model achieved an accuracy of 80.2%, 96.1%, 96.0%, 85.1% and 91.1% for flat, hip, gable, cross-hip and mansard roofs, respectively.