This paper presents a new feature point pairs based technique for object pose estimation and structure recovery from a single view. It first estimates rotational matrix independently, then computes translation vector and recovers the 3D structure of the object directly. Linear and nonlinear strategies are presented to estimate the rotational matrix. One is for small rotational motion and the other is used to estimate large rotational parameters. When the nonlinear technique is applied, its initial guesses are given automatically by the proposed linear estimation method. On the other hand, the presented structure recovery method is not sensitive to the rotational matrix estimation results. The proposed method is applicable to three, four or more feature points and has no constraints, such as collinear or coplanar, on their relative positions. As the number of feature points increases, the estimation results are improved while the computation cost is almost unchanged. Many experiments are performed on synthetic data and real images to demonstrate the presented technique.