Iterative decoding is an effective technique to approach the channel capacity for very large block sizes with enough iterations. However, due to the limitation of bandwidth and delay, small blocks of data are much more commonly applied in practical communications, and low iteration counts are usually preferred for both decoding complexity and delay consideration. In such cases, the design rules of near capacity decoding-which is generally asymptotic with respect to the block size-may cause inferior performance. To overcome this problem for 8-phase shift keying (8PSK) modulated variable length codes (VLCs), an irregular mapping scheme for the transmission system of bit-interleaved coded modulation with iterative decoding (BICM-ID) is studied in this paper. A submapping searching algorithm and an irregular mapping optimization algorithm are proposed aiming at maximizing the extrinsic mutual information after a target number of iterations. Simulation results show that for small data block size with a low iteration count, our scheme has advantages with respect to the existing near capacity systems optimized by the asymptotic tools.