Optical switches offer higher switching speeds than electronics, however, in most cases utilizing the interband transitions of the active medium for switching. As a result, the signal suffers heavy losses. In this article, we demonstrate a simple and yet efficient ultrafast broadband all-optical switching on ps timescale in the sub-bandgap region of the a-Se thin film, where the intrinsic absorption is very weak. The optical switching is attributed to short-lived transient defects that form localized states in the bandgap and possess a large electron-phonon coupling. We model these processes through first principles simulation that are in agreement with the experiments.