The interaction of extreme ultraviolet (XUV) photon with matter is a meaningful way to understand the electronic structure of microscopic particles. In this paper, the electron angular distributions of single ionization and double ionization of Ne and Xe atoms interacting with XUV photons are investigated by utilizing a reaction microscope. The asymmetric parameters of 2p electrons of Ne atoms, and 5p, 5s electrons of Xe atoms combined with the reported experimental data are compared with different theoretical models. The result shows that the electron correlation effect can be ignored in the ionization of 2p electron of Ne atoms. While the ionization of 5p electron of Xe atoms is strongly influenced by the electron correlation effect, but not affected by the relativistic effect. Both of these two effects play an important role in 5s shell ionization of Xe atom. In addition, this study also finds that direct double ionization and indirect double ionization exist simultaneously during the ionization of Xe and the photoelectron angular distributions and asymmetry parameters of the first and second steps in indirect double ionization are given.