Permutation group algebras, and their generalizations called permutation centralizer algebras (PCAs), play a central role as hidden symmetries in the combinatorics of large N gauge theories and matrix models with manifest continuous gauge symmetries. Polynomial functions invariant under the manifest symmetries are the observables of interest and have applications in AdS/CFT. We compute such correlators in the presence of matrix or tensor witnesses, which by definition, can include a matrix or tensor field appearing as a coupling in the action (i.e a spurion) or as a classical (un-integrated) field in the observables, appearing alongside quantum (integrated) fields. In both matrix and tensor cases we find that two-point correlators of general gauge-invariant observables can be written in terms of gauge invariant functions of the witness fields, with coefficients given by structure constants of the associated PCAs. Fourier transformation on the relevant PCAs, relates combinatorial bases to representation theoretic bases. The representation theory basis elements obey orthogonality results for the two-point correlators which generalise known orthogonality relations to the case with witness fields. The new orthogonality equations involve two representation basis elements for observables as input and a representation basis observable constructed purely from witness fields as the output. These equations extend known equations in the super-integrability programme initiated by Mironov and Morozov, and are a direct physical realization of the Wedderburn-Artin decompositions of the hidden permutation centralizer algebras of matrix/tensor models.