There are two ways to unify gravitational field and gauge field. One is to represent gravitational field as principal bundle connection, and the other is to represent gauge field as affine connection. Poincaré gauge theory and metric-affine gauge theory adopt the first approach. This paper adopts the second. In this approach: (i) Gauge field and gravitational field can both be represented by affine connection; they can be described by a unified spatial frame. (ii) Time can be regarded as the total metric with respect to all dimensions of internal coordinate space and external coordinate space. On-shell can be regarded as gradient direction. Quantum theory can be regarded as a geometric theory of distribution of gradient directions. Hence, gauge theory, gravitational theory and quantum theory obtain the same geometric foundation and a unified description of evolution. (iii) Coupling constants, chiral asymmetry, PMNS mixing and CKM mixing can appear spontaneously as geometric properties in affine connection representation. It is remarkable that PMNS mixing and CKM mixing can be interpreted as a spontaneous geometric consequence of symmetry reduction, so they are not necessary to be regarded as direct postulates in the Lagrangian anymore. (iv) The unification theory of gauge fields that are represented by affine connection can avoid the problem that a proton decays into a lepton in theories such as SU(5). (v) There exists a possible geometric interpretation to the color confinement of quarks. In the affine connection representation, we can get better interpretations of the above physical properties, therefore, to represent gauge fields by affine connection is probably a necessary step towards the ultimate theory of physics.