2D black phosphorus (BP) possesses an in‐plane anisotropy, offering another degree of freedom to create new optoelectronic and photonic devices, such as waveplates with atomic thickness. Yet, due to its very low energy bandgap, the anisotropy of BP crystal is mainly studied in the near‐infrared region and remains elusive in the visible region. Furthermore, the strength of optical anisotropy of few‐layer BP, which is crucial for practical applications, is still limited. Herein, a BP‐based Fabry–Pérot (FP) cavity that can increase the strength of BP anisotropy twice compared with the bare BP flake is proposed. Notably, the FP cavity composed of BP/Au dual‐layer structure sitting on the Si substrate could realize a reconfigurable color switching over the whole visible region by varying the polarization of the incident light, utilizing polarization‐dependent complex permittivity of the anisotropic BP crystal. The finding not only offers a simple approach to improve the optical anisotropy of 2D layered materials but also advises an efficient method to develop BP optical devices, including a phase plate, a linear polarizer, and a color display.