Aluminum oxide (Al2O3) is a well‐known electrolyte filler for stabilizing the Li‐metal (Li0) anode in all‐solid‐state Li0‐based batteries. However, its strong interactions with lithium polysulfides (PS) hinder the direct application of Al2O3‐added electrolytes in all‐solid‐state lithium‐sulfur batteries (ASSLSBs). Herein, the role of Al2O3 in ASSLSBs both as electrolyte filler and cathode additive is studied. The combination of Al2O3‐added electrolyte and Al2O3‐added S8 cathode with optimum cell configuration could deliver an unprecedented discharge capacity of 0.85 mAh cm−2 (C/10, 30 cycles) for polymer‐based ASSLSBs. These results suggest that the rational incorporation of Al2O3 can lead simultaneously to PS anchoring and Li0 anode stabilizing benefits from the ceramic filler, thus improving the electrochemical performance of ASSLSBs.