Image steganalysis is the task of detecting a secret message hidden in an image. Deep steganalysis using end-to-end deep learning has been successful in recent years, but previous studies focused on improving detection performance rather than designing a lightweight model for practical applications. This caused a deep steganalysis model to be heavy and computationally costly, making the model infeasible to deploy in real-world applications. To address this issue, we study an effective model design strategy for lightweight image steganalysis. Considering the domain-specific characteristics of steganalysis, we propose a simple yet effective block removal strategy that progressively removes a sequence of blocks from deep classification networks. This method involves the gradual removal of convolutional neural network blocks, starting from deeper ones. By doing so, the number of parameters and FLOPs are decreased without compromising the detection performance. Experimental results show that our removal strategy makes the EfficientNet-B0 variants 9.58 $$\times$$
×
smaller and has 2.16 $$\times$$
×
fewer FLOPs than the baseline while retaining detection accuracy of 90.73% and 82.40% that are on par with the baseline on BOSSBase and ALASKA#2 datasets, respectively. Backed by our in-depth analyses, the results indicate that only a few early layers are sufficient for effective image steganalysis.