Multi‐dimensional multiplexed metasurface holography extends holographic information capacity, promises revolutionary advancements for vivid imaging, information storage and encryption. However, achieving multifunctional metasurface holography by forward design method is still difficult since it relies heavily on Jones matrix engineering, which places high demands on physical knowledge and processing technology. To break these limitations and simplify the design process, here we propose an end‐to‐end inverse design framework. By directly linking the metasurface to the reconstructed images and employing a loss function to guide the update of metasurface, the calculation of hologram can be omitted, thus greatly simplifying the design process. In addition, the requirements on the completeness of meta‐library can also be significantly reduced, allowing multi‐channel hologram to be achieved using meta‐atoms with only two degrees of freedom, which is very friendly to processing. By exploiting the proposed method, metasurface hologram containing up to 12 channels of multi‐wavelength, multi‐plane and multi‐polarization are designed and experimentally demonstrated, which exhibits the state‐of‐the‐art information multiplexing capacity of the metasurface composed of simple meta‐atoms. Our method is conducive to promoting the intelligent design of multifunctional meta‐devices, and it is expected to eventually accelerate the application of meta‐devices in colorful display, imaging, storage and other fields.This article is protected by copyright. All rights reserved