Ammonium polyphosphate (APP) with different polymerization degrees were modified by a novel phosphorus-containing organosilicon compound (PCOC), and the products obtained were coded as MAPP-30 and MAPP-1000. Then they were applied to prepare flame-retardant rigid polyurethane foam (RPUF) separately. The impact of modified APP (MAPP) on the flame-retardant properties of RPUF was investigated by the limited oxygen index (LOI) test, horizontal burning test, and cone calorimeter test. The morphologies of the char residues were observed by SEM. Furthermore, the mechanical properties of RPUF composites were measured by the compressive strength test. The results showed that whether the degree of polymerization of MAPP is 30 or 1000, they both had greater charring ability and better flame-retardant properties than unmodified APP. The residual char yield of RPUF/MAPP-30 (37.3%) and RPUF/MAPP-1000 (36.5%) were both significantly higher than RPUF/APP-30 (22.8%) and RPUF/APP-1000 (24.9%). The peak heat release rate value of RPUF/MAPP-30 was 29.9% lower than that of RPUF/APP-30, and the drop of RPUF/MAPP-1000 was 50.9% compared to RPUF/APP-1000. Moreover, the total heat release of RPUF/MAPP-1000 (9.7 MJ/m2) was much lower than that of RPUF/MAPP-30 (11.3 MJ/m2). In summary, MAPP-1000 has the best flame-retardant properties among all RPUF composites. In addition, the results also showed that flame-retardant performance and the mechanical properties dramatically decreased with the increase in the addition of MAPP-1000, and the RPUF composite had the best comprehensive performance with 20% content of MAPP-1000.