This study analyzes the effect of sample movement on the drying performance based on the drying conditions in a heat pump dryer. The sample movement changes are analyzed by controlling hot air flow into the drum and the mechanical force caused by drum rotation. A drying program is then developed for controlling the drying conditions. Considering the sample movement in the heat pump dryer, for the drum rotating at 50 rpm along with hot air flow, the sample mainly moves at the center of the drum, resulting in more changes in the local movement of the sample than those when only hot air is introduced, or only drum rotation is operated. Moreover, for the abovementioned condition, the drying period was the lowest and the wrinkle recovery is excellent, but the shrinkage increases. Therefore, by identifying the sample movements affecting the drying performance, the drying period decreases as the sample moves, occupying the entire space including the center of the drum. The shrinkage increases when the sample is at the center of the drum. However, it is advantageous for wrinkle recovery when the sample is in the first quadrant and tumbles. On this basis, the drying process is classified into three stages. Hot air flow, drum rotation speed, and drum rotation direction conditions are combined, resulting in efficient wrinkle recovery and shrinkage reduction simultaneously.