Nature-inspired metaheuristic optimization algorithms, e.g., the butterfly optimization algorithm (BOA), have become increasingly popular. The BOA, which adapts the food foraging and social behaviors of butterflies, involves randomly defined, algorithmic-dependent parameters that affect the exploration and exploitation strategies, which negatively influences the overall performance of the algorithm. To address this issue and improve performance, this paper proposes a modified BOA, i.e., the quantum chaos BOA (QCBOA), that relies on chaos theory and quantum computing techniques. Chaos mapping of unpredictable and divergent behavior helps tune critical parameters, and the quantum wave concept helps the representative butterflies in the algorithm explore the search space more effectively. The proposed QCBOA also implements a ranking strategy to maintain balance between the exploration and exploitation phases, which is lacking in conventional BOAs. To evaluate reliability and efficiency, the proposed QCBOA is tested against a well-utilized set of 20 benchmark functions and travelling salesman problem which belongs to the class of combinatorial optimization problems. Besides, the proposed method is also adopted to photovoltaic system parameter extraction to demonstrate its application to real-word problems. An extensive comparative study was also conducted to compare the performance of QCBOA with that of the conventional BOAs, fine-tuned particle swarm optimization (PSO) algorithm, differential evolution (DE), and genetic algorithm (GA). The results demonstrate that, chaos functions with the quantum wave concept yield better performance for most tested cases and comparative results in the rest of the cases. The speed of convergence also increased compared to the conventional BOAs. The proposed QCBOA is expected to provide better results in other real-word optimization problems and benchmark functions.