In drilling engineering, resonant wireless energy transmission technology can effectively solve the problem of frequent charging or battery replacement of cable-less electronic tools downhole. However, due to the narrow space, high temperature, and high pressure inside the oil well, as well as the complex environment where oil, water, gas, and sand coexist, the traditional coil shape is not applicable. Therefore, spiral coils with inner and outer casing are usually used. However, in practice, due to the complex downhole environment, resonant wireless energy transmission coupling devices often encounter difficulties when nested downhole, thus affecting the transmission efficiency. To solve these problems and ensure transmission efficiency, a novel coil coupling model is proposed in this paper, and the power and efficiency expressions of the coupled electrical energy transmission system are derived. Subsequently, the spiral cylinder and the novel coil were simulated and analyzed using COMSOL Multiphysics software. The experimental results show that the model maintains a good transmission efficiency and improves the nesting effect in a complex downhole environment. This study provides a valuable reference for the optimization and design of wireless energy transmission systems.