This paper proposes a density-based boundary evolving algorithm for continuum-based topology optimization. The boundary of voids in the design domain is described by RBF (radial basis function) function controlled by RBF knots in polar coordinate, where the voids are projected onto a fixed grid using Heaviside function. For merging overlapped multiple voids, the p-norm function is introduced to describe composite density field. The differentiability of the projection-based boundary description algorithm allows for the sensitivity analysis via the chain rule, and therefore, it enables an efficient gradient-based optimization method. The goal of this paper is to optimize the initial design to generate buckling-induced mechanism under large deformation, and without loss of generality, the hyperelastic material model is chosen to describe the material behavior. Notably, this method possesses the merit of level set method, where the intermediate density only exists at the boundary of topology shape. At the same time, the proposed method is still in the density-based optimization framework and standard sensitivity analysis of density-based methods can be directly derived based on the chain rule. Several numerical examples are presented and discussed in detail to demonstrate the effectiveness of the proposed density-based boundary evolving method.