The agglomeration of silver nanoparticles (AgNPs) results in poor antibacterial performance, and the accumulation of silver in the human body threatens human health. Preparing a matrix is a technique worth considering as it not only prevents the aggregation of AgNPs but also reduces deposition of AgNPs in the human body. In this paper, carboxy-cellulose nanocrystals (CCNC) were prepared by a simple one-step acid hydrolysis method. Chito-oligosaccharides (CSos) were grafted onto the surface of CCNC to form CSos-CCNC composite nanoparticles. CCNC and CSos-CCNC were used as stabilizers for deposing AgNPs and two types of complexes—AgNPs-CCNC and AgNPs-CSos-CCNC—were obtained, respectively. The influence of the two stabilizer matrices—CCNC and CSos-CCNC—on the morphology, thermal behavior, crystal structure, antibacterial activity, and cell compatibility of AgNPs-CCNC and AgNPs-CSos-CCNC were examined. The results showed that the AgNPs deposited on the CSos-CCNC surface had a smaller average diameter and a narrower particle size distribution compared with the ones deposited on CCNC. The thermal stability of AgNPs-CSos-CCNC was better than that of AgNPs-CCNC. AgNPs did not affect the crystalline structure of CCNC and CSos-CCNC. The antibacterial activity of AgNPs-CSos-CCNC was better than that of AgNPs-CCNC based on antibacterial studies using Escherichia coli, Staphylococcus aureus, and Klebsiella pneumoniae. The cytotoxicity of AgNPs-CSos-CCNC was remarkably lower than that of AgNPs-CCNC.