316L stainless steel is a promising material candidate for a hydrogen containment system. However, when in contact with hydrogen, the material could be degraded by hydrogen embrittlement (HE). Moreover, the mechanism and the effect of HE on 316L stainless steel have not been clearly studied. This study investigated the effect of hydrogen exposure on the impact toughness of 316L stainless steel to understand the relation between hydrogen charging time and fracture toughness at ambient and cryogenic temperatures. In this study, 316L stainless steel specimens were exposed to hydrogen in different durations. Charpy V-notch (CVN) impact tests were conducted at ambient and low temperatures to study the effect of HE on the impact properties and fracture toughness of 316L stainless steel under the tested temperatures. Hydrogen analysis and scanning electron microscopy (SEM) were conducted to find the effect of charging time on the hydrogen concentration and surface morphology, respectively. The result indicated that exposure to hydrogen decreased the absorbed energy and ductility of 316L stainless steel at all tested temperatures but not much difference was found among the pre-charging times. Another academic insight is that low temperatures diminished the absorbed energy by lowering the ductility of 316L stainless steel.