Two‐dimensional covalent organic frameworks (2D COFs) are attractive candidates for next‐generation membrane active layers due to their robust linkages and uniform, tunable pores. Many publications have claimed to achieve selective molecular transport through 2D COF membranes, but reported performance metrics for similar networks vary dramatically, and in several cases the reported experiments are inadequate to support such conclusions. These issues require a reevaluation of the literature. Published examples of 2D COF membranes for liquid‐phase separations can be broadly divided into two categories, each with common performance characteristics: polycrystalline COF films (most >1 μm thick) and weakly crystalline or amorphous films (most <500 nm thick). The former exhibit high solvent permeance, and most, if not all, function as selective adsorbents rather than membranes. The latter behave as membranes with lower permeance in line with conventional reverse osmosis and nanofiltration membranes but have amorphous or ambiguous long‐range order that precludes conclusions about separations occurring via selective transport through the COF pores. So far, neither category has demonstrated consistent relationships between the designed COF pore structure and separation performance, suggesting that these imperfect materials do not sieve molecules through uniform pores. In this perspective, we describe rigorous characterization practices that should be applied to both COF membrane structure and separation performance, which will facilitate their development toward molecularly precise membranes capable of performing previously unrealized chemical separations. In the absence of this more rigorous standard of proof, reports of COF‐based membranes should be treated with skepticism. As methods to control 2D polymerization and 2D polymer processing improve, we anticipate that precise 2D polymer membranes will exhibit exquisite and energy efficient performance relevant for contemporary separation challenges.This article is protected by copyright. All rights reserved