Purpose Our clinical understanding of the relationship between 3D bone morphology and knee osteoarthritis, as well as our ability to investigate potential causative factors of osteoarthritis, has been hampered by the time‐intensive nature of manually segmenting bone from MR images. Thus, we aim to develop and validate a fully automated deep learning framework for segmenting the patella and distal femur cortex, in both adults and actively growing adolescents. Methods Data from 93 subjects, obtained from on institutional review board–approved protocol, formed the study database. 3D sagittal gradient recalled echo and gradient recalled echo with fat saturation images and manual models of the outer cortex were available for 86 femurs and 90 patellae. A deep‐learning–based 2D holistically nested network (HNN) architecture was developed to automatically segment the patella and distal femur using both single (sagittal, uniplanar) and 3 cardinal plane (triplanar) methodologies. Errors in the surface‐to‐surface distances and the Dice coefficient were the primary measures used to quantitatively evaluate segmentation accuracy using a 9‐fold cross‐validation. Results Average absolute errors for segmenting both the patella and femur were 0.33 mm. The Dice coefficients were 97% and 94% for the femur and patella. The uniplanar, relative to the triplanar, methodology produced slightly superior segmentation. Neither the presence of active growth plates nor pathology influenced segmentation accuracy. Conclusion The proposed HNN with multi‐feature architecture provides a fully automatic technique capable of delineating the often indistinct interfaces between the bone and other joint structures with an accuracy better than nearly all other techniques presented previously, even when active growth plates are present.