Reliable multicast distribution is essential for some applications such as Internet of Things (IoT) alarm information and important file distribution. Traditional IP reliable multicast usually relies on multicast source retransmission for recovery losses, causing huge recovery delay and redundancy. Moreover, feedback implosion tends to occur towards multicast source as the number of receivers grows. Information-Centric Networking (ICN) is an emerging network architecture that is efficient in content distribution by supporting multicast and in-network caching. Although ubiquitous in-network caching provides nearby retransmission, the design of cache strategy greatly affects the performance of loss recovery. Therefore, how to recover losses efficiently and quickly is an urgent problem to be solved in ICN reliable multicast. In this paper, we first propose an overview architecture of ICN-based reliable multicast and formulate a problem using recovery delay as the optimization target. Based on the architecture, we present a Congestion-Aware Probabilistic Cache (CAPC) strategy to reduce recovery delay by caching recently transmitted chunks during multicast transmission. Then, we propose NACK feedback aggregation and recovery isolation scheme to decrease recovery overhead. Finally, experimental results show that our proposal can achieve fully reliable multicast and outperforms other approaches in recovery delay, cache hit ratio, transmission completion time, and overhead.