Unitary equivariance is a natural symmetry that occurs in many contexts in physics and mathematics. Optimization problems with such symmetry can often be formulated as semidefinite programs for a $$d^{p+q}$$
d
p
+
q
-dimensional matrix variable that commutes with $$U^{\otimes p} \otimes {\bar{U}}^{\otimes q}$$
U
⊗
p
⊗
U
¯
⊗
q
, for all $$U \in \textrm{U}(d)$$
U
∈
U
(
d
)
. Solving such problems naively can be prohibitively expensive even if $$p+q$$
p
+
q
is small but the local dimension d is large. We show that, under additional symmetry assumptions, this problem reduces to a linear program that can be solved in time that does not scale in d, and we provide a general framework to execute this reduction under different types of symmetries. The key ingredient of our method is a compact parametrization of the solution space by linear combinations of walled Brauer algebra diagrams. This parametrization requires the idempotents of a Gelfand–Tsetlin basis, which we obtain by adapting a general method inspired by the Okounkov–Vershik approach. To illustrate potential applications of our framework, we use several examples from quantum information: deciding the principal eigenvalue of a quantum state, quantum majority vote, asymmetric cloning and transformation of a black-box unitary. We also outline a possible route for extending our method to general unitary-equivariant semidefinite programs.