Firefly Algorithm (FA) is one of the most recently introduced stochastic, nature-inspired, meta-heuristic approaches used for solving optimization problems. The conventional FA use randomization factor during generation of solution search space and fireflies position changing, which results in imbalanced relationship between exploration and exploitation. This imbalanced relationship causes in incapability of FA to find the most optimum values at termination stage. In the proposed model, this issue has been resolved by incorporating PS at the termination stage of standard FA. The optimized values obtained from the FA are set as the initial starting points for the PS algorithm and the values are further optimized by PS to get the most optimal values or at least better values than the values obtained by conventional FA during its maximum number of iterations. The performance of the newly developed FA-PS model has been tested on eight minimization functions and six maximization functions by considering various performance evaluation parameters. The results obtained have been compared with other optimization algorithms namely genetic algorithm (GA), standard FA, artificial bee colony (ABC), ant colony optimization (ACO), differential equations (DE), bat algorithm (BA), grey wolf optimization (GWO), Self-Adaptive Step Firefly Algorithm (SASFA), and FA-Cross algorithm in terms of convergence rate and various numerical performance evaluation parameters. A significant improvement has been observed in the solution quality by embedding PS in the standard FA at the termination stage. The result behind this improvement is the better exploration and exploitation of the solution search space at this stage.