We explain the microscopic origin of linear confinement potential with the Casimir scaling in generic confining gauge theories. In the low-temperature regime of confining gauge theories such as QCD, Polyakov lines are slowly varying Haar random modulo exponentially small corrections with respect to the inverse temperature, as shown by one of the authors (M. H.) and Watanabe. With exact Haar randomness, computation of the two-point correlator of Polyakov loops reduces to the problem of random walk on group manifold. Linear confinement potential with approximate Casimir scaling except at short distances follows naturally from slowly varying Haar randomness. With exponentially small corrections to Haar randomness, string breaking and loss of Casimir scaling at long distance follow. Hence we obtain the Casimir scaling which is only approximate and holds only at intermediate distance, which is precisely needed to explain the results of lattice simulations. For (1 + 1)-dimensional theories, there is a simplification that admits the Casimir scaling at short distances as well.