Carbohydrate esterases family 4 (CE4 enzymes) includes chitin and peptidoglycan deacetylases, acetylxylan esterases, and poly-N-acetylglucosamine deacetylases that act on structural polysaccharides, altering their physicochemical properties, and participating in diverse biological functions. Chitin and peptidoglycan deacetylases are not only involved in cell wall morphogenesis and remodeling in fungi and bacteria, but they are also used by pathogenic microorganisms to evade host defense mechanisms. Likewise, biofilm formation in bacteria requires partial deacetylation of extracellular polysaccharides mediated by poly-N-acetylglucosamine deacetylases. Such biological functions make these enzymes attractive targets for drug design against pathogenic fungi and bacteria. On the other side, acetylxylan esterases deacetylate plant cell wall complex xylans to make them accessible to hydrolases, making them attractive biocatalysts for biomass utilization. CE4 family members are metal-dependent hydrolases. They are highly specific for their particular substrates, and show diverse modes of action, exhibiting either processive, multiple attack, or patterned deacetylation mechanisms. However, the determinants of substrate specificity remain poorly understood. Here, we review the current knowledge on the structure, activity, and specificity of CE4 enzymes, focusing on chitin deacetylases and related enzymes active on N-acetylglucosamine-containing oligo and polysaccharides.