Woodworking manipulators are applied in wood processing to promote automatic levels in the wood industry. However, traditional trajectory planning results in low operational stability and inefficiency. Therefore, we propose a method combining 3-5-3 piecewise polynomial (composed of cubic and quintic polynomials) interpolation and an improved particle swarm optimization (PSO) algorithm to study trajectory planning and time optimization of woodworking manipulators. In trajectory planning, we conducted the kinematics analysis to determine the position information of joints at path points in joint space and used 3-5-3 piecewise polynomial interpolation to fit a point-to-point trajectory and ensure the stability. For trajectory time optimization, we propose an improved PSO that adapts multiple strategies and incorporates a golden sine optimization algorithm (Gold-SA). Therefore, the proposed improved PSO can be called GoldS-PSO. Using benchmark functions, we compared GoldS-PSO to four other types of PSO algorithms and Gold-SA to verify its effectiveness. Then, using GoldS-PSO to optimize the running time of each joint, our results showed that GoldS-PSO was superior to basic PSO and Gold-SA. The shortest running time obtained by using GoldS-PSO was 47.35% shorter than before optimization, 8.99% shorter than the basic PSO, and 6.23% shorter than the Gold-SA, which improved the running efficiency. Under optimal time for GoldS-PSO, our simulation results showed that the displacement and velocity of each joint were continuous and smooth, and the acceleration was stable without sudden changes, proving the method’s feasibility and superiority. This study can serve as the basis for the motion control system of woodworking manipulators and provide reference for agricultural and forestry engineering optimization problems.