Due to its distinct pore structure consisting of micropores and ultramicropores, carbon molecular sieve (CMS) is a promising material for creating membranes. However, the pyrolysis of an asymmetric polymer precursor to produce CMS membranes would compromise its inherent porous structure, resulting in a thick selective layer that might lead to a low flux membrane. Instead, thin composite membranes with selective CMS layers coated on porous substrates can provide a short transport length for enhanced permeability. In this work, a polymer precursor was dip-coated onto porous alumina hollow fibers, followed by pyrolysis to yield composite hollow fiber membranes with a very thin (∼2 μm), selective CMS layer. Using the composite membranes, xylene mixtures were separated by organic solvent reverse osmosis mode, and enhanced flux of the composite membrane (up to 10 times) was obtained compared to other CMS membranes. It is expected that organic liquid mixtures can be separated at a lower cost through membranes than through the conventional thermal process.