We consider rare semileptonic decays of a heavy D-meson into a light vector meson in the framework of QCD factorization. In contrast to the corresponding B-meson decays, the naive factorization hypothesis does not even serve as a first approximation. Rather, the decay amplitudes appear to be dominated by non-factorizable dynamics, e.g. through annihilation topologies, which are particularly sensitive to long-distance hadronic contributions. We therefore pay particular attention to intermediate vector-meson resonances appearing in quark-loop and annihilation topologies. Compared to the analogous B-meson decays, we identify a number of effects that result in very large theoretical uncertainties for differential decay rates. Some of these effects are found to cancel in the ratio of partially integrated decay rates for transversely and longitudinally polarized ρ mesons. On the phenomenological side this implies a very limited potential to constrain physics beyond the Standard Model by means of these decays.