Structure of convolutional neural network (CNN) applied for image recognition requires large numbers of tuning for designated datasets in practice. It is a time-consuming process to finally come up with a feasible structure for specific requirement. This paper proposes a method based on Taguchi method which can efficiently determine the optimal structure of hyperparameters combination. Five hyperparameters with four levels are defined as control factors and two indicators are chosen to measure the performance of CNN structure. L16 (45) orthogonal array is used to arrange the experiment. S/N ratio and main effect plot are used to identify the optimal structure (hyperparameter combination) of CNN. The classic case of MNIST is employed to verify the practicability of the proposed method. Results show that the proposed method can identify the optimal CNN structure efficiently and also rank the significance priority of hyperparameters.