Fifteen new photochromic hybrid materials were synthesized by gas phase loading of fluorinated azobenzenes, namely ortho‐tetrafluoroazobenzene (tF‐AZB), 4H,4H′‐octafluoroazobenzene (oF‐AZB), and perfluoroazobenzene (pF‐AZB), into the pores of the well‐known metal‐organic frameworks MOF‐5, MIL‐53(Al), MIL‐53(Ga), MIL‐68(Ga), and MIL‐68(In). Their composition was analysed by elemental (CHNS) and DSC/TGA. For pF‐AZB0.34@MIL‐53(Al), a structural model based on high‐resolution synchrotron powder diffraction data was developed and the host‐guest and guest‐guest interactions were elucidated from this model. These interactions of O−H⋅⋅⋅F and π⋅⋅⋅π type were confirmed by significant shifts of the O−H frequencies in loaded and unloaded MOFs of the MIL‐53 and MIL‐68 series. Most remarkably, all of the synthesized F‐AZB@MOF systems can be switched with visible light, and some of them show almost quantitative (>95 %) photo‐isomerization between its E and Z forms with no significant fatigue after repeated switching cycles.