In this paper, we study the symmetry of quantum torus with the concept of crossed product algebra. As a classical counterpart, we consider the orbifold of classical torus with complex structure and investigate the transformation property of classical theta function. An invariant function under the group action is constructed as a variant of the classical theta function. Then our main issue, the crossed product algebra representation of quantum torus with complex structure under the symplectic group is analyzed as a quantum version of orbifolding. We perform this analysis with Manin's so-called model II quantum theta function approach. The symplectic group Sp(2n, Z) satisfies the consistency condition of crossed product algebra representation. However, only a subgroup of Sp(2n, Z) satisfies the consistency condition for orbifolding of quantum torus.