In production systems, the buffer capacities have usually been assumed to be fixed during normal operations. Inspired by the observations from the real industrial operations, a novel concept of the Adaptive Buffer Space (ABS) is proposed in this paper. The ABS is a type of equipment, such as movable racks or mobile robots with racks, that can be used to provide extra storage space for a production line to temporarily increase certain buffers' capacities in a real-time fashion. A good strategy to assign and re-assign the ABS can significantly improve real-time production throughput. In order to model the production systems with changing buffer capacities, a data-driven model is developed to incorporate the impact of buffer capacity variation in system dynamics. Based on the model, a real-time ABS assignment strategy is developed by analyzing real-time buffer levels and machine status. The strategy is demonstrated to be effective in improving system throughput. An approximate dynamic programming algorithm, referred to as ABS-ADP, is developed to obtain the optimal ABS assignment policy based on the strategy. A knowledge-guided value function initialization method is proposed in ABS-ADP algorithm to expedite the convergence.