In this paper, we classify four-point local gluon S-matrices in arbitrary dimensions. This is along the same lines as [1] where four-point local photon S-matrices and graviton S-matrices were classified. We do the classification explicitly for gauge groups SO(N) and SU(N) for all N but our method is easily generalizable to other Lie groups. The construction involves combining not-necessarily-permutation-symmetric four-point S-matrices of photons and those of adjoint scalars into permutation symmetric four-point gluon S-matrix. We explicitly list both the components of the construction, i.e permutation symmetric as well as non-symmetric four point S-matrices, for both the photons as well as the adjoint scalars for arbitrary dimensions and for gauge groups SO(N) and SU(N) for all N. In this paper, we explicitly list the local Lagrangians that generate the local gluon S-matrices for D ≥ 9 and present the relevant counting for lower dimensions. Local Lagrangians for gluon S-matrices in lower dimensions can be written down following the same method. We also express the Yang-Mills four gluon S-matrix with gluon exchange in terms of our basis structures.