Protein contact maps represent spatial pairwise inter-residue interactions, providing a protein's translationally and rotationally invariant topological representation. Accurate contact map prediction has been a critical driving force for improving protein structure prediction, one of computational biology's most challenging problems in the last half-century. While many computational tools have been developed to this end, most fail to predict accurate contact maps for proteins with insufficient homologous protein sequences, and exhibit low accuracy for long-range contacts. To address these limitations, we develop a novel hybrid model, CGAN-Cmap, that uses a generative adversarial neural network embedded with a series of modified squeeze and excitation residual networks. To exploit features of different dimensions, we build the generator of CGAN-Cmap via two parallel modules: sequential and pairwise modules to capture and interpret distance profiles from 1D sequential and 2D pairwise feature maps, respectively, and combine them during the training process to generate the contact map. This novel architecture helps to improve the contact map prediction by surpassing redundant features and encouraging more meaningful ones from 1D and 2D inputs simultaneously. We also introduce a new custom dynamic binary cross-entropy (BCE) as the loss function to extract essential details from feature maps, and thereby address the input imbalance problem for highly sparse long-range contacts in proteins with insufficient numbers of homologous sequences. We evaluate the performance of CGAN-Cmap on the 11th, 12th, 13th, and 14th Critical Assessment of protein Structure Prediction (CASP 11, 12, 13, and 14) and CAMEO test sets. CGAN-Cmap significantly outperforms state-of-the-art models, and in particular, it improves the precision of medium and long-range contact by at least 3.5%. Furthermore, our model has a low dependency on the number of homologous sequences obtained via multiple sequence alignment, suggesting that it can predict protein contact maps with good accuracy for those proteins that lack homologous templates. These results demonstrate an efficient approach for fast and highly accurate contact map prediction toward construction of protein 3D structure from protein sequence.