To achieve the ability of associating continuous-time laser frames is of vital importance but challenging for hand-held or backpack simultaneous localization and mapping (SLAM). In this study, the complex associating and mapping problem is investigated and modeled as a multilayer optimization problem to realize low drift localization and point cloud map reconstruction without the assistance of the GNSS/INS navigation systems. 3D point clouds are aligned among consecutive frames, submaps, and closed-loop frames using the normal distributions transform (NDT) algorithm and the iterative closest point (ICP) algorithm. The ground points are extracted automatically, while the non-ground points are automatically segmented to different point clusters with some noise point clusters omitted before 3D point clouds are aligned. Through the three levels of interframe association, submap matching and closed-loop optimization, the continuous-time laser frames can be accurately associated to guarantee the consistency of 3D point cloud map. Finally, the proposed method was evaluated in different scenarios, the experimental results showed that the proposed method could not only achieve accurate mapping even in the complex scenes, but also successfully handle sparse laser frames well, which is critical for the scanners such as the new Velodyne VLP-16 scanner’s performance.