Based on virtual reality technology, landscape 3D modeling provides users with the possibility to construct a simulated garden landscape environment design effect online, so it has high requirements for accuracy. With the continuous improvement of precision requirements, the number of people involved in the construction of 3D models is also increasing, which puts forward higher requirements for modeling. Based on this, this paper studies the optimization strategy of landscape space 3D model based on big data analysis. Based on the analysis of the establishment of the 3D model and the related algorithm research, this paper analyzes the optimal design of the 3D model under the background of big data. In the 3D modeling of the edge folded area, it is based on the traditional quadratic error measurement grid simplification algorithm, combined with the vertex error matrix to simplify, so as to shorten the modeling time. Based on an efficient search algorithm, an adaptive nonsearch fractal image compression and decoding method is proposed in the image compression and decoding stage of 3D modeling. The search is performed by specifying the defined area block. Finally, an experiment is designed to analyze the performance of the optimization algorithm. The results show that the improved edge folding region algorithm can reduce errors on the basis of ensuring image quality, and the adaptive search algorithm can shorten the search time and improve the compression rate. This method provides a technical reference for the visualization experience and simulation system of garden landscape design and improves the presentation quality of virtual garden landscape design scenes.