To take the sphericity of the Earth into account, tesseroids are often utilized as grid elements in large-scale gravitational forward modeling. However, such elements in a latitude–longitude mesh suffer from degenerating into poorly shaped triangles near poles. Moreover, tesseroids have limited flexibility in describing laterally variable density distributions with irregular boundaries and also face difficulties in achieving completely equivalent division over a spherical surface that may be desired in a gravity inversion. We develop a new method based on triangular spherical prisms (TSPs) for 3D gravitational modeling in spherical coordinates. A TSP is defined by two spherical surfaces of triangular shape, with one of which being the radial projection of the other. Due to the spherical triangular shapes of the upper and lower surfaces, TSPs enjoy more advantages over tesseroids in describing mass density with different lateral resolutions. In addition, such an element also allows subdivisions with nearly equal weights in spherical coordinates. To calculate the gravitational effects of a TSP, we assume the density in each element to be polynomial along radial direction so as to accommodate a complex density environment. Then, we solve the Newton’s volume integral using a mixed Gaussian quadrature method, in which the surface integral over the spherical triangle is calculated using a triangle-based Gaussian quadrature rule via a radial projection that transforms the spherical triangles into linear ones. A 2D adaptive discretization strategy and an extension technique are also combined to improve the accuracy at observation points near the mass sources. The numerical experiments based on spherical shell models show that the proposed method achieves good accuracy from near surface to a satellite height in the case of TSPs with various dimensions and density variations. In comparison with the classical tesseroid-based method, the proposed algorithm enjoys better accuracy and much higher flexibility for density models with laterally irregular shapes. It shows that to achieve the same accuracy, the number of elements required by the proposed method is much less than that of the tesseroid-based method, which substantially speeds up the calculation by more than 2 orders. The application to the tessellated LITHO1.0 model further demonstrates its capability and practicability in realistic situations. The new method offers an attractive tool for gravity forward and inverse problems where the irregular grids are involved.