We present a detailed analysis of the contribution of small-angle Nambu-Goldstone boson (phonon) collisions to the shear viscosity, η, in a superfluid atomic Fermi gas close to the unitarity limit. We show that the experimental values of the shear viscosity coefficient to entropy ratio, η/s, obtained at the lowest reached temperature can be reproduced assuming that phonons give the leading contribution to η. The phonon contribution is evaluated considering 1 ↔ 2 processes and taking into account the finite size of the experimental system. In particular, for very low temperatures, T < ∼ 0.1TF , we find that phonons are ballistic and the contribution of phonons to the shear viscosity is determined by the processes that take place at the interface between the superfluid and the normal phase. This result is independent of the detailed form of the phonon dispersion law and leads to two testable predictions: the shear viscosity should correlate with the size of the optical trap and it should decrease with decreasing temperature. For higher temperatures the detailed form of the phonon dispersion law becomes relevant and, within our model, we find that the experimental data for η/s can be reproduced assuming that phonons have an anomalous dispersion law.