A comprehensive evaluation of generalizability of deep-learning based Hi-C resolution improvement methods

Abstract

HiC is a widely used technique to study the 3D organization of the genome. Due to its high sequencing cost, most of the generated datasets are of coarse quality, consequently limiting the quality of the downstream analyses. Recently, multiple deep learning-based methods have been proposed to improve the quality of these data sets by increasing their resolution through upscaling. However, the existing works do not thoroughly evaluate these methods using HiC reproducibility metrics across different HiC experiments to establish their applicability in real-world scenarios. This study extensively compares deep learning-based HiC upscaling methods on real-world, low-quality HiC datasets. We evaluate these models using HiC reproducibility metrics on data from three cell lines - GM12878 (lymphoblastoid), K562 (human erythroleukemic), and IMR90 (lung fibroblast) - obtained from different HiC experiments. We show that the deep-learning techniques evaluated in this study, trained on downsampled data, cannot upscale real-world, low-quality HiC matrices effectively. More importantly, we also show that retraining these methods on examples of real-world data improves their performance and similarity on target experimental data sets. However, even with retraining, our downstream analyses on the output of these methods suggest that these methods fail to capture the biological signals in the real-world inputs with high sparsity. Therefore, our study highlights the need for rigorous evaluation and identifies specific areas that need improvement concerning current deep learning-based HiC upscaling methods.

GrapHiC: An integrative graph based approach for imputing missing Hi-C reads

“…We compare the performance of GrapHiC against two relevant state-of-the-art methods. First, we compare our performance with HiCNN [8] because a recent evaluation [14] showed that HiCNN provides the best imputation generalizability across a broad range of sparse real-world Hi-C datasets. Second, we compare our performance with HiCReg [30], a method that utilizes the same 1D genomic signals without a sparse Hi-C contact map to impute Hi-C reads.…”

“…Similar to our Hi-C data, we binned all the data in 10kbp resolution, normalized them in the 0 to 99.75th percentile range, and cropped them into sub-ranges of size 200 that aligned with Hi-C submatrices. We divided chromosome [1][2][3][4][5][6][7][8][12][13][14][15][16][17][18] as training, [8,10,19,22] as validation and [9,11,20,21] as testing chromosome sets. We have summarized our inputs' pre-processing workflow in the Figure 1 A and their statistics in the Supplementary Tables S1, S2, and S3.…”

“…A recent work by Murtaza et al . [14] evaluated these methods on real-world sparse Hi-C datasets and found that the performance of these methods degraded substantially as the sparsity of the input increased. We have included a result from this work in the Supplementary Section S1 for ease of reference.…”

“…Our results show that GrapHiC outperforms HiCNN (the best state-of-the-art method [14]) on a Hi-C similarity metric by 3.4% on average on GM12878 Hi-C datasets, with the most significant improvement of 24% for a dataset downsampled by a factor of 1 / 100. Similarly, the 3D chromatin models reconstructed with GrapHiC show better accuracy in 4 out of 5 sparse GM12878 datasets.…”

“…Subsequent models to HiCPlus [7] added additional model parameters [8, 9] or tailored the loss objective [10, 11, 12] to assist the model in extracting better features for reconstruction. A recent work [14] has shown that these methods struggle to generalize to real-world sparse Hi-C contact maps. Specifically, the performance of these methods degrades as the sparsity of the input contact maps increases.…”