With the fast development of agriculture, industrialization and urbanization, large amounts of different (in)organic pollutants are inevitably discharged into the ecosystems. The efficient decontamination of the (in)organic contaminants is crucial to human health and ecosystem pollution remediation. Covalent organic frameworks (COFs) and metal–organic frameworks (MOFs) have attracted multidisciplinary research interests because of their outstanding physicochemical properties like high stability, large surface areas, high sorption capacity or catalytic activity. In this review, we summarized the recent works about the elimination/extraction of organic pollutants, heavy metal ions, and radionuclides by MOFs and COFs nanomaterials through the sorption-catalytic degradation for organic chemicals and sorption-catalytic reduction-precipitation-extraction for metals or radionuclides. The interactions between the (in)organic pollutants and COFs/MOFs nanomaterials at the molecular level were discussed from the density functional theory calculation and spectroscopy analysis. The sorption of organic chemicals was mainly dominated by electrostatic attraction, π-π interaction, surface complexation and H-bonding interaction, whereas the sorption of radionuclides and metal ions was mainly attributed to surface complexation, ion exchange, reduction and incorporation reactions. The porous structures, surface functional groups, and active sites were important for the sorption ability and selectivity. The doping or co-doping of metal/nonmetal, or the incorporation with other materials could change the visible light harvest and the generation/separation of electrons/holes (e−/h+) pairs, thereby enhanced the photocatalytic activity. The challenges for the possible application of COFs/MOFs nanomaterials in the elimination of pollutants from water were described in the end.