Accurate estimation of current position and attitude of a vehicle is one of the key technologies for autonomous driving. Due to the defect of LiDAR intrinsic parameter and the sparsity of LiDAR beam in the vertical direction, current LiDAR-based simultaneous localization and mapping (SLAM) system generally suffers from the problem of inaccurate height positioning. In this study, a LiDAR and inertial measurement unit (IMU) tightly coupled localization algorithm considering ground constraint is proposed, which is developed based on a pose graph optimization framework. At the front end, the ground segmentation algorithm Patchwork is improved to obtain a point cloud with higher verticality, which is added to the LiDAR inertial odometry. Moreover, constraints are constructed by using current frame ground points and world map ground points, which are added to factor map optimization to limit elevation errors. At the back end, SC++ descriptors are used to construct loop constraints to eliminate accumulated errors. Verifications based on KITTI dataset show that the height positioning accuracy will be improved through introducing ground constraint factor and loop detection factor. Real vehicle tests indicate that the proposed algorithm has better height positioning accuracy and better robustness compared with the LeGO-LOAM algorithm.