The objective of this study was to develop an efficient drying technology for poria cubes in order to improve product quality. Poria cubes were dried using different methods, including air impingement drying, infrared-assisted air impingement drying, vacuum drying, two-stage vacuum drying, and infrared-assisted air impingement drying. The results were compared with those from hot air drying. For the two-stage drying, the tested conditions were the first stage of vacuum drying with temperatures between 65–85 °C and a switching moisture ratio of 70–90%. The second stage infrared-assisted air impingement drying also had temperatures 65–85 °C. The drying kinetics (effective moisture diffusivity (Deff), Biot number (Bi), and mass transfer coefficient (k) were studied via the product qualities (broken ratio, firmness, microstructure, and water-soluble polysaccharide content) and specific energy consumption (SEC) of the drying processes. The results showed that two-stage drying led to the lowest drying time and energy consumption, and also obtained the best qualities. Box–Behnken experimental design with response surface methodology (RSM) was used to optimize the two-stage operating conditions as 82 °C under vacuum drying until a moisture content of 81% and a temperature of 69 °C with infrared-assisted air impingement drying was achieved. These findings suggested that two-stage vacuum and infrared-assisted air impingement drying is a promising method for producing high quality and energy efficient dried poria cubes.