Remaining competitive in future conflicts with technologically-advanced competitors requires us to continue to invest in developing robust artificial intelligence (AI) for wargaming. Although deep reinforcement learning (RL) continues to show promising results in intelligent agent behavior development, it has yet to perform at or above the human level in the long-horizon, complex tasks typically found in combat modeling and simulation. Capitalizing on the proven potential of RL and recent successes of hierarchical reinforcement learning (HRL), our research aims to extend the use of HRL to create intelligent agents capable of performing effectively in these large and complex simulation environments. We plan to do so by developing a scalable HRL agent architecture and training framework, developing a dimension-invariant dynamic abstraction engine, and demonstrating scalability by incorporating our approach into a high-fidelity combat simulation.