Recently, perfect state transfer (PST for short) on graphs has attracted great attention due to their applications in quantum information processing and quantum computations. Many constructions and results have been established through various graphs. However, most of the graphs previously investigated are abelian Cayley graphs. Necessary and sufficient conditions for Cayley graphs over dihedral groups having perfect state transfer were studied recently. The key idea in that paper is the assumption of the normality of the connection set. In those cases, viewed as an element in a group algebra, the connection set is in the center of the group algebra, which makes the situations just like in the abelian case. In this paper, we study the non-normal case. In this case, the discussion becomes more complicated. Using the representations of the dihedral group $D_n$, we show that ${\rm Cay}(D_n,S)$ cannot have PST if $n$ is odd. For even integers $n$, it is proved that if ${\rm Cay}(D_n,S)$ has PST, then $S$ is normal.