Two zinc-diphosphonates driven by 4,4'-bipyridine (4,4'-bipy), [Zn(HEDP)(4,4'-bipy)]·4.5(HO) (1) and (H-4,4'-bipy)·[Zn(HEDP)(HEDP)(HO)]·1.5(HO) (2), were solvothermally prepared (HEDP = CHC(OH)(PO), 1-hydroxyethylidenediphosphonate). Compound 1 features a layered structure with the Zn-HEDP chains as supramolecular units and 4,4'-bipy as interchain linkers. Compound 2 possesses a Zn-HEDP layer with protonated 4,4'-bipy as template. The structure differences between the two compounds are mainly ascribed to the distinct role of 4,4'-bipy in the process of in situ assembly. In compound 1, the 4,4'-bipy moiety acts as coordinative unit to participate in the formation of the resulting architecture. However, in compound 2, the protonated 4,4'-bipy serves as template to direct the formation of the final structure. The luminescence and proton conduction of the two compounds were investigated. Compounds 1 and 2 exhibit intense blue fluorescence; compound 2 has a proton conductivity of 2.61 × 10 S cm at 100% relative humidity and 70 °C.