We introduce a generalized Grover matrix of a graph and present an explicit formula for its characteristic polynomial. As a corollary, we give the spectra for the generalized Grover matrix of a regular graph. Next, we define a zeta function and a generalized zeta function of a graph G with respect to its generalized Grover matrix as an analog of the Ihara zeta function and present explicit formulas for their zeta functions for a vertextransitive graph. As applications, we express the limit on the generalized zeta functions of a family of finite vertex-transitive regular graphs by an integral. Furthermore, we give the limit on the generalized zeta functions of a family of finite tori as an integral expression.