A novel system which is specially designed for measuring effectiveness of small-scale magnetic shields in a liquid helium Dewar is introduced in this paper. This system is mainly applied to test the shielding performance of multilayer magnetic shielding structure which is used in the field of single-flux-quantum (SFQ) electronics testing at liquid helium temperature. This system allows using a fluxgate magnetometer to measure the magnetic flux instead of the superconducting quantum interference device (SQUID) method. The fluxgate magnetometer can be put into the system through a circular channel which is connected with the ambient environment. Thus, both the shielding effectiveness and the absolute magnetic flux can be obtained by this system. In addition, this system is designed with the shielding of the interference of environmental magnetic field, which improves the measurement accuracy. By utilizing this system, the magnetic shield effectiveness of a four-layer magnetic shields manufactured with low-temperature permalloy material and superconducting material was measured. The experimental results shows that the axial shielding effectiveness (SE) is 58.3 dB. The comparison between the experimental and theoretical results verified the effectiveness of this system.