Machine learning for Non-Destructive Evaluation (NDE) has the potential to bring significant improvements in defect characterization accuracy due to its effectiveness in pattern recognition problems. However, the application of modern machine learning methods to NDE has been obstructed by the scarcity of real defect data to train on. This paper demonstrates how an efficient, hybrid finite element and ray-based simulation can be used to train a Convolutional Neural Network (CNN) to characterize real defects. To demonstrate this methodology, an inline-pipe inspection application is considered. This uses four plane wave images from two arrays, and is applied to the characterization of cracks of length 1-5 mm and inclined at angles of up to 20° from the vertical. A standard image-based sizing technique, the 6 dB drop method, is used as a comparison point. For the 6 dB drop method the average absolute error in length and angle prediction is ±1.1 mm, ±8.6° while the CNN is almost four times more accurate at ±0.29 mm, ±2.9°. To demonstrate the adaptability of the deep-learning approach, an error in sound speed estimation is included in the training and test set. With a maximum error of 10% in shear and longitudinal sound speed the 6 dB drop method has an average error of ±1.5 mm, ±12° while the CNN has ±0.45 mm, ±3.0°. This demonstrates far superior crack characterization accuracy by using deep learning rather than traditional image-based sizing.