Drying is a key step that affects the chemical composition and quality of tea. In the present study, we reported the impacts of drying temperature and time on drying and color change kinetics, phytochemical composition, antioxidant activity, and surface microstructure of coffee leaves during hot-air drying. The results showed that drying temperature was positively (p < 0.05) correlated with the drying rate (DR), color index a* and total color change ΔE, and total soluble sugar (TSS), while negatively correlated with color indexes b* and L*, soluble protein content, and the DPPH scavenging capacity. Drying time has similar impacts on the color indexes and soluble protein as drying temperature. The content of total free amino acid and TSS increased by 62.5% and 47.4%, respectively, when coffee leaves were dried at 160 • C for 24 min, under which the total phenolic content and DPPH and ABTS scavenging capacities reached the maximum of 108.04 mg GAE/g, 515.07 µmol Trolox/g, and 606.70 µmol Trolox/g, respectively. Drying significantly decreased the contents of chlorogenic acids and mangiferin and antioxidant activity, while high-temperature short-time drying helped retain phenolic compounds in coffee leaves. The DR fitted Page kinetic model. The color changes fitted the first-order kinetic models and the activation energies ranged from 16.00 to 31.06 kJ⋅mol −1 . Prolonged drying time caused serious wrinkles on the surface of coffee leaves and the stomata closed.