Virtual hand usually simulates human hands by mapping the actual shapes of hands to virtual environment and modeling the hand-object manipulations in human machine interactions. For describing virtual hand and its manipulation in a unified framework, a method based on conformal geometric algebra (CGA) is proposed to solve the problems of virtual hand modeling and interaction in this paper. With the vertex blending based on CGA, the artifacts on the finger joints are improved, which enhances the realism of the virtual hand model. With the same tool, the collision detection between the virtual hand and the manipulated objects is implemented. The gesture of grasp, pinch, and hold are recognized, and the corresponding manipulation rules are established by CGA calculation. To test these three typical manipulations, the manipulated objects are imported and the manipulation effectiveness is evaluated.