Rubber Tire Gantry (RTG) plays a pivotal role in facilitating efficient container handling within port operations. Conventional RTG, highly depending on human operations, is inefficient, labor-intensive and also poses safety issues in adverse environments. This paper introduces a multi-target detection and tracking (MTDT) algorithm specifically tailored for automated port RTG operations. The approach seamlessly integrates enhanced YOLOX for object detection and improved DeepSORT for object tracking to enhance MTDT performance in the complex port settings. In particular, Light-YoloX, an upgraded version of YOLOX incorporating separable convolution and attention mechanism, is introduced to improve real-time capability and small target detection. Subsequently, OSNet-DeepSORT, an enhanced version of DeepSORT, is proposed to mitigate ID switching challenges arising from unreliable data communication or occlusion in real port scenarios. The effectiveness of the proposed method is validated in various real-life port operations. Ablation studies and comparative experiments against typical MTDT algorithms demonstrate noteworthy enhancements in key performance metrics, encompassing small target detection, tracking accuracy, ID switching frequency, and realtime performance.