BackgroundThe delineation of brain arteriovenous malformations (bAVMs) is crucial for subsequent treatment planning. Manual segmentation is time‐consuming and labor‐intensive. Applying deep learning to automatically detect and segment bAVM might help to improve clinical practice efficiency.PurposeTo develop an approach for detecting bAVM and segmenting its nidus on Time‐of‐flight magnetic resonance angiography using deep learning methods.Study TypeRetrospective.Subjects221 bAVM patients aged 7–79 underwent radiosurgery from 2003 to 2020. They were split into 177 training, 22 validation, and 22 test data.Field Strength/Sequence1.5 T, Time‐of‐flight magnetic resonance angiography based on 3D gradient echo.AssessmentThe YOLOv5 and YOLOv8 algorithms were utilized to detect bAVM lesions and the U‐Net and U‐Net++ models to segment the nidus from the bounding boxes. The mean average precision, F1, precision, and recall were used to assess the model performance on the bAVM detection. To evaluate the model's performance on nidus segmentation, the Dice coefficient and balanced average Hausdorff distance (rbAHD) were employed.Statistical TestsThe Student's t‐test was used to test the cross‐validation results (P < 0.05). The Wilcoxon rank test was applied to compare the median for the reference values and the model inference results (P < 0.05).ResultsThe detection results demonstrated that the model with pretraining and augmentation performed optimally. The U‐Net++ with random dilation mechanism resulted in higher Dice and lower rbAHD, compared to that without that mechanism, across varying dilated bounding box conditions (P < 0.05). When combining detection and segmentation, the Dice and rbAHD were statistically different from the references calculated using the detected bounding boxes (P < 0.05). For the detected lesions in the test dataset, it showed the highest Dice of 0.82 and the lowest rbAHD of 5.3%.Data ConclusionThis study showed that pretraining and data augmentation improved YOLO detection performance. Properly limiting lesion ranges allows for adequate bAVM segmentation.Level of Evidence4Technical Efficacy Stage1