Motivation: Identifying chromatin loops from genome-wide interaction matrices like Hi-C data is notoriously difficult. Such kinds of patterns can span through the genome from a hundred kilobases to thousands of kilobases. Most loop patterns are frequently related to biological functions, such as providing contacts between regulatory regions and promoters. They can also affect the cell-specific biological functions of different regulatory regions of DNA, thus leading to disease and tumorigenesis. While most statistical methods failed in the generalization to multiple cell types, recently proposed machine learning-based methods struggled when tested on sparse single-cell Hi-C (scHi-C) contact maps. We notice that there is an urgent need for an algorithm that can handle sparse scHi-C maps, and at the same time, can generate confident loop calls on regular cell lines. Results: Therefore, we propose a novel deep learning-based framework for Hi-C chromatin loop detection (HiC-LDNet) and provide the corresponding downstream analysis. HiC-LDNet can give relatively more accurate predictions in multiple tissue types and contact technologies. Compared to other loop calling algorithms, such as HiCCUPS, Peakachu, and Chromosight, HiC-LDNet recovers a higher number of loop calls in multiple experimental platforms (Hi-C, ChIA-PET, DNA-SPRITE, and HiChIP), and achieves higher confidence scores in multiple cell types (Human GM12878, K562, HAP1, and H1-hESC). For example, in genome-wide loop detection on the human GM12878 cell line, HiC-LDNet successfully recovered 82.5% of loops within only 5 pixels of 10k bp resolution. Furthermore, in the sparse scHi-C ODC tissue, HiC-LDNet achieves superior performance by recovering 93.5% of ground truth loops with high confidence scores, compared with that of Peakachu (31.5%), Chromosight(69.6%), and HiCCUPS(9.5%). Therefore, our method is a robust and general pipeline for genome-wide chromatin loop detection for both bulk Hi-C and scHi-C data.