Container drayage involves the transportation of containers by trucks. Although the distance is relatively short compared to maritime and rail transport, container drayage accounts for 25% to 40% of the total container transportation costs and significantly contributes to increased fuel consumption and carbon emissions. Thus, the modeling of the container drayage problem (CDP) has received a lot of attention in the last two decades. However, the three fundamental modeling factors, including the combination of trucking operation modes and empty container relocation strategies, as well as empty container constraints and multiple inland depots, have not been simultaneously investigated. Hence, this study addressed a comprehensive CDP that simultaneously incorporates the three modeling factors. The problem was formulated as a novel mixed integer linear programming (MILP) model based on the DAOV graph. Given the complexity of this problem, it was not realistic to find an exact solution for large instances. Therefore, an improved genetic algorithm (GA) was designed by integrating the “sequential insertion” method and “solution re-optimization” operation. The performance of Gurobi and GA was validated and evaluated through randomly generated instances. The results indicate that (1) the proposed algorithm can provide near-optimal solutions for large-scale instances within a reasonable running time, (2) the greatest cost savings from combining trucking operation modes and empty container relocation strategies range from 10.45% to 31.86%, and (3) the three modeling factors significantly influence the fuel consumption and carbon emissions, which can provide managerial insights for sustainable container drayage practices.