BackgroundSelf-incompatibility (SI) is under genetic control and prevents inbreeding depression in angiosperms. SI mechanisms are quite complicated and still poorly understood in many plants. Tea (Camellia sinensis L.) belonging to the family of Theaceae, exhibits high levels of SI and high heterozygosity. Uncovering the molecular basis of SI of the tea plant may enhance breeding and simplify genomics research for the whole family.ResultsThe growth of pollen tubes following selfing and crossing was observed using fluorescence microscopy. Self-pollen tubes grew slower than cross treatments from 24 h to 72 h after pollination. RNA-seq was employed to explore the molecular mechanisms of SI and to identify SI-related genes in C. sinensis. Self and cross-pollinated styles were collected at 24 h, 48 h and 72 h after pollination. Six RNA-seq libraries (SP24, SP48, SP72, CP24 CP48 and CP72; SP = self-pollinated, CP = cross-pollinated) were constructed and separately sequenced. In total, 299.327 million raw reads were generated. Following assembly, 63,762 unigenes were identified, and 27,264 (42.76 %) unigenes were annotated in five public databases: NR, KOG, KEGG, Swiss-Port and GO. To identify SI-related genes, the fragments per kb per million mapped reads (FPKM) values of each unigene were evaluated. Comparisons of CP24 vs. SP24, CP48 vs. SP48 and CP72 vs. SP72 revealed differential expression of 3,182, 3,575 and 3,709 genes, respectively. Consequently, several ubiquitin-mediated proteolysis, Ca2+ signaling, apoptosis and defense-associated genes were obtained. The temporal expression pattern of genes following CP and SP was analyzed; 6 peroxidase, 1 polyphenol oxidase and 7 salicylic acid biosynthetic process-related genes were identified. The RNA-seq data were validated by qRT-PCR of 15 unigenes. Finally, a unigene (CL25983Contig1) with strong homology to the S-RNase was analyzed. It was mainly expressed in styles, with dramatically higher expression in self-pollinated versus cross-pollinated tissues at 24 h post-pollination.ConclusionsThe present study reports the transcriptome of styles after cross- and self-pollination in tea and offers novel insights into the molecular mechanism behind SI in C. sinensis. We believe that this RNA-seq dataset will be useful for improvement in C. sinensis as well as other plants in the Theaceae family.Electronic supplementary materialThe online version of this article (doi:10.1186/s12864-016-2703-5) contains supplementary material, which is available to authorized users.