Operator fractional Brownian motions (OFBMs) are zero mean, operator self-similar (o.s.s.), Gaussian processes with stationary increments. They generalize univariate fractional Brownian motions to the multivariate context. It is well-known that the so-called symmetry group of an o.s.s. process is conjugate to subgroups of the orthogonal group. Moreover, by a celebrated result of Hudson and Mason, the set of all exponents of an operator self-similar process can be related to the tangent space of its symmetry group.In this paper, we revisit and study both the symmetry groups and exponent sets for the class of OFBMs based on their spectral domain integral representations. A general description of the symmetry groups of OFBMs in terms of subsets of centralizers of the spectral domain parameters is provided. OFBMs with symmetry groups of maximal and minimal types are studied in any dimension. In particular, it is shown that OFBMs have minimal symmetry groups (as thus, unique exponents) in general, in the topological sense. Finer classification results of OFBMs, based on the explicit construction of their symmetry groups, are given in the lower dimensions 2 and 3. It is also shown that the parametrization of spectral domain integral representations are, in a suitable sense, not affected by the multiplicity of exponents, whereas the same is not true for time domain integral representations.