There exist two main drawbacks for traffic images in classic image enhancement methods. First is the performance degradation that occurs under frontlight, backlight, and extremely dark conditions. The second drawback is complicated manual settings, such as transform functions and multiple parameter selection mechanisms. Thus, this paper proposes an effective and adaptive parameter optimization enhancement algorithm based on adaptive brightness baseline drift (ABBD) for color traffic images under different luminance conditions. This method consists of two parts: brightness baseline model acquisition and adaptive color image compensation. The brightness baseline model can be attained by analyzing changes in light along a timeline. The adaptive color image compensation involves color space remapping and adaptive compensation specific color components. Our experiments were tested on various traffic images under frontlight, backlight, and during nighttime. The experimental results show that the proposed method achieved better effects compared with other available methods under different luminance conditions, which also effectively reduced the influence of the weather.