We rewrite the (extended) Ooguri-Vafa partition function for colored HOMFLY-PT polynomials for torus knots in terms of the free-fermion (semi-infinite wedge) formalism, making it very similar to the generating function for double Hurwitz numbers. This allows us to conjecture the combinatorial meaning of full expansion of the correlation differentials obtained via the topological recursion on the Brini-Eynard-Mariño spectral curve for the colored HOMFLY-PT polynomials of torus knots.This correspondence suggests a structural combinatorial result for the extended Ooguri-Vafa partition function. Namely, its coefficients should have a quasi-polynomial behavior, where non-polynomial factors are given by the Jacobi polynomials. We prove this quasi-polynomiality in a purely combinatorial way. In addition to that, we show that the (0,1)-and (0,2)-functions on the corresponding spectral curve are in agreement with the extension of the colored HOMFLY-PT polynomials data.