A unitary coupled-channels model is presented for investigating the decays of
heavy mesons and excited meson states into three light pseudoscalar mesons. The
model accounts for the three-mesons final state interactions in the decay
processes, as required by both the three-body and two-body unitarity
conditions. In the absence of the Z-diagram mechanisms that are necessary
consequences of the three-body unitarity, our decay amplitudes are reduced to a
form similar to those used in the so-called isobar-model analysis. We apply our
coupled-channels model to the three-pions decays of a1(1260), pi2(1670),
pi2(2100), and D0 mesons, and show that the Z-diagram mechanisms can contribute
to the calculated Dalitz plot distributions by as much as 30% in magnitudes in
the regions where f0(600), rho(770), and f2(1270) dominate the distributions.
Also, by fitting to the same Dalitz plot distributions, we demonstrate that the
decay amplitudes obtained with the unitary model and the isobar model can be
rather different, particularly in the phase that plays a crucial role in
extracting the CKM CP-violating phase from the data of B meson decays. Our
results indicate that the commonly used isobar model analysis must be extended
to account for the final state interactions required by the three-body
unitarity to reanalyze the three-mesons decays of heavy mesons, thereby
exploring hybrid or exotic mesons, and signatures of physics beyond the
standard model.Comment: 32 pages, 10 figures. Version to appear in PR