The quantization technique with a low-bit resolution can significantly reduce the cost and power consumption of analog-to-digital converter (ADC). It will play an important role in energy conservation and cost reduction for the incoming B5G millimeter-wave (MMW) communication systems. In this paper, we propose and demonstrated experimentally a low-bit Gaussian mixture model (GMM) based non-uniform adaptive vector quantization (AVQ) scheme for the low-cost intensity modulated envelope detection photonics-assisted 28 GHz MMW communication system for the first time. The principles of GMMbased one-dimensional adaptive scalar quantization (ASQ) and multi-dimensional AVQ are first introduced, and then are used to realize the low-bit non-uniform adaptive quantization for reducing the ADC bit resolution of MMW receiver. Furthermore, the performance of traditional uniform quantization, the present K-means and proposed GMM-based non-uniform ASQ/AVQ schemes are evaluated and compared in detail. Utilizing the proposed GMM-based AVQ scheme, the ADC quantization resolution in our MMW receiver can be reduced from 5 bits of the traditional uniform quantization to as low as 2 bits, without noticeable performance penalty. Moreover, as compared with the K-means-based quantization scheme, the MMW receiver enabled by GMM-based ASQ/AVQ scheme can save about half of the quantization time under similar performance. This is mainly because the clustering based on probability converges faster than the Euclidean distance, which significantly reduces the number of iterations required. Therefore, the GMM-based AVQ scheme is a promising solution to realize high performance ADCs with low-bit resolution for future MMW-enabled optical fiber wireless access networks.