Hand gesture recognition is one of emerging Human Computer Interaction (HCI) technologies for the next generation of mobile devices. However, conventional softwareoriented approaches spend a considerable time and require a large memory size for hand segmentation, which fails to give real-time interactions between users and mobile devices. Therefore, in this paper, we present a high-throughput and memoryefficient hand segmentation processor. To obtain both of high throughput and high memory-efficiency, we propose a parallelized hand candidate decision and a compressed feedback histogram. As a result, it achieves 124.9 fps with only 26.9 KB on-chip memory, which are 1.39 times faster and 92 time smaller, respectively, compared to the state-of-the-art.