K-mer clustering algorithm using a MapReduce framework: application to the parallelization of the Inchworm module of Trinity

Abstract

BackgroundDe novo transcriptome assembly is an important technique for understanding gene expression in non-model organisms. Many de novo assemblers using the de Bruijn graph of a set of the RNA sequences rely on in-memory representation of this graph. However, current methods analyse the complete set of read-derived k-mer sequence at once, resulting in the need for computer hardware with large shared memory.ResultsWe introduce a novel approach that clusters k-mers as the first step. The clusters correspond to small sets of gene products, which can be processed quickly to give candidate transcripts. We implement the clustering step using the MapReduce approach for parallelising the analysis of large datasets, which enables the use of compute clusters. The computational task is distributed across the compute system, and no specialised hardware is required. Using this approach, we have re-implemented the Inchworm module from the widely used Trinity pipeline, and tested the method in the context of the full Trinity pipeline. Validation tests on a range of real datasets show large reductions in the runtime and per-node memory requirements, when making use of a compute cluster.ConclusionsOur study shows that MapReduce-based clustering has great potential for distributing challenging sequencing problems, without loss of accuracy. Although we have focussed on the Trinity package, we propose that such clustering is a useful initial step for other assembly pipelines.

Improving in-silico normalization using read weights

“…Kim et al . 7 proposed a modification to the Trinity assembler by pre-clustering the input k -mers using the MapReduce framework, which works if a suitable infrastructure is available. Techniques such as entropy based compression 8 can also be used to reduce the memory required to store k -mers.…”

Modern transcriptome data processing algorithms: a review of methods and results of approbation

“…2) или выполнить de novo сборку. Для этой цели в биоинформатике применяют программное обеспечение rnaSPADes [24], Trinity [25][26][27][28], Oases [29,30], SOAPdenovo-trans, Abyss [31][32][33], NextGene Floton и другие. Более длинные прочтения или прочтения с парными концами (как при секвенировании обеих цепей ДНК) способствуют получению лучших результатов de novo сборки.…”