Dual‐band optical signals, consisting of visible and infrared wavelengths, hold significant potential in various fields such as night vision, vehicle safety, bioimaging, and optical communications. Consequently, considerable attention has focused on modulation techniques for visible‐infrared dual‐band signals, including various electrochromic devices. However, these devices often suffer from low speeds and complex structures, making them unsuitable for satisfying the current demand for ultrafast on‐chip nanophotonics. Here, an ultrafast, dual‐band selective optical switching is presented in thallium selenide (TlSe), a layered nanomaterial with low‐symmetry. Transient absorption microscopy directly reveals distinct modulation bands in the visible and near‐infrared (NIR) regions. Notably, these bands exhibit orthogonal polarization dependencies, enabling the selective modulation of visible and NIR light via polarization control. Theoretical analysis reveals that the modulation bands in the visible and NIR regions arise from the perpendicularly anisotropic excited‐state absorption of electrons and holes, respectively. This is supported by distinct ultrafast cooling times observed for electrons and holes. The modulations exhibit picosecond‐scale dynamics owing to efficient Auger recombination. These exceptional characteristics highlight the promising nature of low‐symmetry TlSe as a novel material for ultrafast dual‐band nanophotonics. Furthermore, this work provides fundamental insights into anisotropic carrier dynamics, including effective mass‐dependent cooling and many‐body interactions.