Purpose Object classification and localization is a key task of computer-aided diagnosis (CAD) tool. Although there have been numerous generic deep learning (DL) models developed for CAD, there is no work in the literature to evaluate their effectiveness when utilized in diagnosing fractures in proximity of joint implants. In this work, we aim to assess the performance of existing classification systems on binary and multi-class problems (fracture types) using plain radiographs. In addition, we evaluated the performance of object detection systems using the one- and two-stage DL architectures. Methods A data set of 1272 X-ray images of Peri-prosthetic Femur Fracture PFF was collected. The fractures were annotated with bounding boxes and classified according to the Vancouver Classification System (type A, B, C) by two clinical specialists. Four classification models such as Densenet161, Resnet50, Inception, VGG and two object detection models such as Faster RCNN and RetinaNet were evaluated, and their performance compared. Six confusion matrix-based measures were reported to evaluate fracture classification. For localization of the fracture, Average Precision and localization accuracy were reported. Results The Resnet50 showed the best performance with $$95\%$$ 95 % accuracy and $$94\%$$ 94 % F1-score in the binary classification: fracture/normal. In addition, the Resnet50 showed $$90\%$$ 90 % accuracy in multi-classification (normal, Vancouver type A, B and C). Conclusions A large data set of PFF images and the annotations of fracture features by two independent assessments were created to implement a DL-based approach for detecting, classifying and localizing PFFs. It was shown that this approach could be a promising diagnostic tool of fractures in proximity of joint implants.