Rapid and precise tree Diameter at Breast Height (DBH) measurement is pivotal in forest inventories. While the recent advancements in LiDAR and Structure from Motion (SFM) technologies have paved the way for automated DBH measurements, the significant equipment costs and the complexity of operational procedures continue to constrain the ubiquitous adoption of these technologies for real-time DBH assessments. In this research, we introduce KAN-Forest, a real-time DBH measurement and key point localization algorithm utilizing RGB-D (Red, Green, Blue-Depth) imaging technology. Firstly, we improved the YOLOv5-seg segmentation module with a Channel and Spatial Attention (CBAM) module, augmenting its efficiency in extracting the tree’s edge features in intricate forest scenarios. Subsequently, we devised an image processing algorithm for real-time key point localization and DBH measurement, leveraging historical data to fine-tune current frame assessments. This system facilitates real-time image data upload via wireless LAN for immediate host computer processing. We validated our approach on seven sample plots, achieving bbAP50 and segAP50 scores of: 90.0%(+3.0%), 90.9%(+0.9%), respectively with the improved YOLOv5-seg model. The method exhibited a DBH estimation RMSE of 17.61∼54.96 mm (R2=0.937), and secured 78% valid DBH samples at a 59 FPS. Our system stands as a cost-effective, portable, and user-friendly alternative to conventional forest survey techniques, maintaining accuracy in real-time measurements compared to SFM- and LiDAR-based algorithms. The integration of WLAN and its inherent scalability facilitates deployment on Unmanned Ground Vehicles (UGVs) to improve the efficiency of forest inventory. We have shared the algorithms and datasets on Github for peer evaluations.