Multi-task Localization and Segmentation for X-Ray Guided Planning in Knee Surgery

Abstract

X-ray based measurement and guidance are commonly used tools in orthopaedic surgery to facilitate a minimally invasive workflow. Typically, a surgical planning is first performed using knowledge of bone morphology and anatomical landmarks. Information about bone location then serves as a prior for registration during overlay of the planning on intra-operative X-ray images. Performing these steps manually however is prone to intra-rater/inter-rater variability and increases task complexity for the surgeon. To remedy these issues, we propose an automatic framework for planning and subsequent overlay. We evaluate it on the example of femoral drill site planning for medial patellofemoral ligament reconstruction surgery. A deep multi-task stacked hourglass network is trained on 149 conventional lateral X-ray images to jointly localize two femoral landmarks, to predict a region of interest for the posterior femoral cortex tangent line, and to perform semantic segmentation of the femur, patella, tibia, and fibula with adaptive task complexity weighting. On 38 clinical test images the framework achieves a median localization error of 1.50 mm for the femoral drill site and mean IOU scores of 0.99, 0.97, 0.98, and 0.96 for the femur, patella, tibia, and fibula respectively. The demonstrated approach consistently performs surgical planning at expert-level precision without the need for manual correction.

Automatically Diagnosing HIP Conditions from X-Rays using Landmark Detection

“…We experimented with two architectures, a simple U-net architecture [10] with 512 × 512 images as input and output and a stacked hourglass network [11] with 8 hourglasses and 256 × 256 images. The stacked hourglass network was proposed for human-pose detection but has been applied to find landmarks in medical images [12]. We used a batch size of 4 and an ADAM optimizer with a learning rate of 0.001 stepped down by a factor of 0.1 after epoch 40.…”

Contour-based Bone Axis Detection for X-Ray Guided Surgery on the Knee

“…The proposed construction relies on a segmentation mask of the target long bone and ROI encodings for both relevant contour regions. For combined prediction we use a multi-task variant of the hourglass network architecture by Newell et al [9,14]. This network architecture allows to optimize a joint representation of both tasks and benefits execution time and computational footprint upon inference.…”

“…We translate the established two-line/two-circle manual method [6,7,8,10] to a learning based extraction of anatomical features and subsequent geometric construction based on segmentation of the bone cortex outline. With reference to [9], region of interest (ROI) encoding of the relevant contour sections is used to cope with variability in image truncation and arbitrary image rotation. Moreover, the segmentation results can directly be used for registration of the detected axis on fluoroscopic live images.…”

An Annotation Sparsification Strategy for 3D Medical Image Segmentation via Representative Selection and Self-Training

“…3D image segmentation is one of the most important tasks in medical image applications, such as morphological and pathological analysis (Lee et al 2015b;Hou et al 2019), disease diagnosis (Pace et al 2015), and surgical planning (Kordon et al 2019). Recently, 3D deep learning (DL) models have been widely used in medical image segmentation and achieved state-of-the-art performance (Ronneberger, Fischer, and Brox 2015;Yu et al 2017;Liang et al 2019), most of which were trained with fully annotated 3D image stacks.…”