High-quality reconstruction under a low sampling rate is very important for ghost imaging. How to obtain perfect imaging results from the low sampling rate has become a research hotspot in ghost imaging. In this paper, inspired by matrix optimization in compressed sensing, an optimization scheme of speckle patterns via measurement-driven framework is introduced to improve the reconstruction quality of ghost imaging. According to this framework, the sampling matrix and sparse basis are optimized alternately using the sparse coefficient matrix obtained from the low-dimension pseudo-measurement process and the corresponding solution is obtained analytically, respectively. The optimized sampling matrix is then dealt with non-negative constraint and binary quantization. Compared to the developed optimization schemes of speckle patterns, simulation results show that the proposed scheme can achieve better reconstruction quality with the low sampling rate in terms of peak signal-to-noise ratio (PSNR) and mean structural similarity index (MSSIM). In particular, the lowest sampling rate we use to achieve a good performance is about 6.5%. At this sampling rate, the MSSIM and PSNR of the proposed scheme can reach 0.787 and 17.078 dB, respectively.