In this study, composite anion exchange membranes (AEMs) were synthesized by cross-linking poly(phenylene oxide) (PPO) with cellulose functionalized by 1,4-diazabicyclo[2.2.2]-octane (DABCO) or di-guanidine (DG). The structural and morphological characteristics of the synthesized AEMs were characterized by FTIR, 1H-NMR, SEM, TEM, and AFM, while their performance was evaluated in terms of ionic conductivity, water uptake, ion exchange capacity, and tensile strength with respect to the loading of the quaternized cellulose in the quaternized PPO (qPPO) matrix. The composite AEMs exhibited considerably enhanced mechanical and alkaline stability as well as good anion conductivity. The composite AEM with 7 wt% of cellulose functionalized with DG in the qPPO matrix (qPPO/DG-Cel7) exhibited a maximum hydroxide conductivity of 0.164 S cm−1. Furthermore, a urea/O2 fuel cell prepared using this composite membrane showed a maximum power density of 12.3 mW cm−2. The results indicated that the cellulose-based composite membranes showed a satisfactory performance in alkaline fuel cell applications.