Traffic light detection demands high real-time performance and a lightweight design, existing methods often encounter issues such as detection delays and limited computational resources. Therefore, reducing computational overhead and increasing detection speed, while maintaining accuracy, becomes a critical challenge. To tackle these, this paper proposes GAD-DETR, an enhanced RT-DETR-based network. First, inspired by the approach of GhostNet to minimize computational redundancy and integrate reparameterized convolution (RepConv), the GRELAN module is developed to restructure the backbone network which significantly decreases model size and parameters while enhancing detection speed. To improve the recognition of small objects, whose features tend to be diluted as the network deepens, ADown is introduced to replace standard convolution for downsampling, enhancing small-object detection capability. Finally, a lightweight feature fusion module, DGSFM, is designed to further reduce computational costs and enhance efficiency. Experimental results indicate that GAD-DETR achieves a detection precision of 95.9%, with a model size reduction of 50.3%, and parameter and computation reductions of 50.8% and 51.2%, respectively. FPS increases from 76.7 to 117.8, demonstrating that the proposed algorithm achieves lightweight, real-time traffic light detection.