We show that the Ocneanu algebra of quantum symmetries, for an ADE diagram (or for higher Coxeter-Dynkin systems, like the Di Francesco -Zuber system) can be, in most cases, deduced from the structure of the modular T matrix in the A series. We recover in this way the (known) quantum symmetries of su(2) diagrams and illustrate our method by studying those associated with the three genuine exceptional diagrams of type su(3), namely E5, E9 and E21. This also provides the shortest way to the determination of twisted partition functions in boundary conformal field theory with defect lines.