In recent years, soft manipulators have attracted much attention in the field of robotics research due to their dexterity and good environmental adaptability. However, accurately modeling of soft manipulators remains a difficult task due to the uncertainties in their dynamics. Therefore, this paper proposed an method to accurately model the inverse dynamics of a soft manipulator. A BP neural network model was used to establish the inverse dynamics model of the soft manipulator, and then the particle swarm optimization (PSO) algorithm was employed to optimize the initial weights and biases of the BP neural network, and finally the correspondence between the end position of the soft manipulator and the input air pressure was established. In addition, load weights were introduced as one of the network inputs to enhance the control accuracy of the soft manipulator when it carries a load. By trajectory tracking experiments with and without load, the method was proved to achieve an average end position error of 0.839 mm with a relative error of 0.98%. When carrying loads its average end error is 1.800 mm with a relative error of 2.12%. The results demonstrated that the proposed optimized method can effectively improve the accuracy of the soft manipulator under both no-load and load conditions.