A rigid carboxylate ligand with a nitro functional group was selected to coordinate with Tb(III) cation, and Tb-MOF ({[Tb 4 (L) 4 (OH) 4 (H 2 O) 3 ]•8H 2 O} n , H 2 L = 2-nitroterephthalic acid) with large porous and excellent hydrophilicity was obtained successfully. The obtained Tb-MOF was filled into the Nafion matrix to improve its proton conduction performance. The Tb-MOF/ Nafion composite membrane was characterized by PXRD, IR, and thermogravimetry (TG) and for water uptake, area swelling, and proton conductivity. The activity energy, E a , value of the composite membrane, which is a very important factor affecting the proton conduction performance of the membrane, was fitted and calculated. It was revealed that Tb-MOF can improve the proton conductivities of composite membranes, and the improvement degree and E a value were both affected by Tb-MOF content. When Tb-MOF content was 5%, the proton conductivity of the composite membrane was 1.53 × 10 −2 S•cm −1 at 100% RH and 80 °C, which is 1.81 times that of the pure Nafion membrane. A MOF containing a nitro functional group was first doped into Nafion in this study and exhibited excellent performance for improving composite membrane proton conductivity. This study will provide a valuable reference for designing different functionalized MOFs to promote the proton conductivities of proton exchange membranes.