A precision reducer with a high ratio and a small size is essential for precise control of the position of a robotic arm. The rotate vector (RV) is a reducer widely applied in robotic joints. However, its complex structure, over-positioning and difficult assembly restrict its application industrial robots. Herein, a novel abnormal cycloidal gear (ACG) reducer is proposed. In comparison to the RV reducer, the proposed reducer has the characteristics of compact structure, less over-positioning and high ratio. The compound tooth profile of “epicycloid-involute hypocycloid” is used as the driving teeth for improving the performance of the reducer. The operating principle, mechanism design and reduction ratio of the proposed reducer is investigated, and the design method of the ACG tooth profile is also introduced. A dynamic characteristic model is established to verify the correctness of the design method of the reducer. A prototype of the ACG reducer is fabricated by using the computer numerical control (CNC) machining technology. Results show that the ACG reducer is capable to cover a wide range of reduction ratios with a simple mechanism. The numerical and theoretical results are in agreement, which depict the structural feasibility and correctness of the design method of the reducer. The adaptation of the ACG tooth profile as the driving teeth helps in improving the performance of the reducer.