The Quantum Approximate Optimization Algorithm (QAOA), which is a variational quantum algorithm, aims to give sub-optimal solutions of combinatorial optimization problems. It is widely believed that QAOA has the potential to demonstrate application-level quantum advantages in the noisy intermediate-scale quantum(NISQ) processors with shallow circuit depth. Since the core of QAOA is the computation of expectation values of the problem Hamiltonian, an important practical question is whether we can find an efficient classical algorithm to solve quantum mean value in the case of general shallow quantum circuits. Here, we present a novel graph decomposition based classical algorithm that scales linearly with the number of qubits for the shallow QAOA circuits in most optimization problems except for complete graph case. Numerical tests in Max-cut, graph coloring and Sherrington-Kirkpatrick model problems, compared to the state-of-the-art method, shows orders of magnitude performance improvement. Our results are not only important for the exploration of quantum advantages with QAOA, but also useful for the benchmarking of NISQ processors.