Membrane gas separation is an environmentally friendly and economical method used to separate valuable gases, industrial process gas wastes, and carbon dioxide from mixed gases. The most important part of this method is the membranes. Gas separation membranes are expected to have high separation and permeability performance, high mechanical strength, easy and fast production capability, and low prices. Polymer-based membranes are mostly preferred depending on the ease of modification capability. In this study, a zirconium-based metal organic framework (Zr-MOF, MIL-140 A) was synthesized and used as a filler within polyvinyl alcohol (PVA) matrix for the selective separation of hydrogen (H2) from carbon dioxide (CO2). The effect of MIL-140 A addition on the mechanical, structural, and morphological properties of PVA was evaluated. The MIL-140 A significantly improved the mechanical strength of the membrane. According to the gas separation results, the increasing concentration of MIL-140 A increased the selective separation performance of the nanocomposite membrane. The highest mechanical strength (43.1 MPa) and best film-forming ability were obtained with 3 wt% MIL-140 A loaded membrane. The ideal H2/CO2 selectivity and hydrogen permeability were obtained as 5.6 and 944 Barrer, respectively at 2 bar feed pressure and room temperature. The highest ideal H2/CO2 selectivity was obtained as 6.3 with the H2 permeability of 959 Barrer when the MIL-140 A ratio was 4 wt%.