In this paper, a new solar-assisted heat pump drying system with waste heat recovery and double water tanks (SCAHP) was established and the system optimized by TRNSYS and variable air volume experiment. The annual cumulative efficiency of the SCAHP (COPac), buffer tank heating efficiency (ηBT) and hot water storage tank heating efficiency (ηST) are all considered as optimization objectives, and this paper discusses the relationship between the three optimization objectives and studies the influence of hot air volume (qDR), area of solar collector (ASC), inclination angle of solar collector (ISC) and volume of heat storage water tank (VST) on system efficiency in the drying process. In order to explore the general rule of system optimization, numerical analysis and optimization were carried out in five typical cities in five climatic regions of China. The results show that the mixed variable air volume mode can increase the drying rate with less energy loss, and shorten the drying period by 28.6–33.3%. When the system surface body ratio (SBR) in Nanjing is between 3.1 and 4.1, the COPac, ηBT, and ηST can reach the maximum value simultaneously. It is estimated that the cost can be recovered in 5 years when the system configuration parameters are optimized.