Water deficit caused a serious threat to crops, especially panicle development at reproductive growth phase. We investigated grain yield components and gene expression profiles of panicle among tolerant and sensitive rice in response to drought stress. Panicle morphologies exhibited that secondary branches per panicle were more severely affected as compared to primary branches per panicle. Moreover, grain weight per panicle showed significant decrease for both tolerant and sensitive varieties except for MILT1444. Expression profile analysis revealed that 783 differentially expressed genes (DEGs) were identified to be drought-induced from young panicles in 2 cm length. Hierarchical clustering indicated that 76.8% of DEGs were up-regulated for all six rice varieties, and the percentage of down-regulated genes was higher in sensitive group than tolerant group. Biological process category revealed that the shared Gene Ontology (GO) terms were involved in response to abiotic stimulus and stress, whereas the specific GO terms in tolerant group were identified as regulation of biological quality, homeostatic process, cell growth, anatomical structure morphogenesis and development, and the unique terms in sensitive varieties were identified as lipid metabolic process and secondary metabolic process. Furthermore, the gene-based association analysis narrowed down list of DEGs, and four genes common to all six varieties were selected as candidate for breeders. Together, we found several shared and distinct biological processes between tolerant and sensitive varieties, and candidate stress-responsive genes. These findings provided insight into functional mechanisms regulating drought stress response in panicle development and may also help to crop tolerant improvement.