For D-brane system with three D-branes on compact Calabi-Yau threefolds, the dual F-theory fourfolds for parallel/complete coincident/part coincident D-brane system is constructed by the type II/F-theory duality. Complete coincident means that the three D-branes coincide and part coincident represents the coincident of two of the three D branes. The low energy effective superpotentials are calculated by mirror symmetry, GKZ-system method and the type II/F-theory duality on the B-model side, respectively. Using the mirror symmetry, A-model superpotentials and the Ooguri-Vafa invariants are obtained from the B-model side. These results indicate that the superpotential contributed by one of three parallel D-branes is identical with the D-brane system with only one D-brane, which is a signal of decoupling of the parallel topological D-branes. However, the superpotential and Oogrui-Vafa invariants are different among parallel, complete coincident and part coincident D-brane system, which show the evidence of the phase transition due to the enhanced gauge symmetry in the low energy theories and the geometrical singularity.