Microplastic fibers from synthetic textiles have been indicated as a major source of pollution because millions of fibers are released into the environment by laundry washing. In fact, such types of microplastics usually bypass wastewater treatment processes and filters, thus reaching the oceans and other water natural reservoirs in huge quantities. Nowadays, several approaches are available for characterizing microplastics, but unfortunately, there is no unique and standard method for this aim. Among the various methodologies, several microscopy techniques such as optical microscopy (OM), scanning electron microscopy coupled with energy dispersive X-ray (SEM−EDX) analysis, or OM combined with molecular spectroscopy can be used for visual detection and measurements. Our proposal is a non-destructive method based on a digital holographic microscope in a configuration fully sensitive to the polarization of light transmitted by the fibers with a micron dimension. Our aim is to prove that the proposed approach allows for the precise characterization of synthetic and natural fibers in water. The method exploits all the advantages of digital holography such as numerical refocusing, non-invasive testing, and quantitative measurements of the complex wave field by adding access to the polarization of light that conveys meaningful information about materials. We intend to show that a unique all-optical fingerprint can be retrieved using the Jones-matrix formalism for the major classes of synthetic microfilaments used in the textile industry (i.e., polyamide 6, polyamide 6.6, polypropylene, and polyester) and for the most common natural fibers (i.e., wool and cotton) prepared according to previously developed appropriate protocols. Our results show that the proposed technology identifies new features for micron-sized fibers based on optical anisotropy through quantitative digital holography that could open future routes for automatic and all-optical identification of textile contaminants in water.