Lens-free digital in-line holography (LDIH) produces cellular diffraction patterns (holograms) with a large field of view that lens-based microscopes cannot offer. It is a promising diagnostic tool allowing comprehensive cellular analysis with high-throughput capability. Holograms are, however, far more complicated to discern by the human eye, and conventional computational algorithms to reconstruct images from hologram limit the throughput of hologram analysis. To efficiently and directly analyze holographic images from LDIH, we developed a novel deep learning architecture called a holographical deep learning network (HoloNet) for cellular phenotyping. The HoloNet uses holo-branches that extract large features from diffraction patterns and integrates them with small features from convolutional layers. Compared with other state-of-the-art deep learning methods, HoloNet achieved better performance for the classification and regression of the raw holograms of the breast cancer cells stained with well-known breast cancer markers, ER/PR and HER2. Moreover, we developed the HoloNet dual embedding model to extract high-level diffraction features related to breast cancer cell types and their marker intensities of ER/PR and HER2 to identify previously unknown subclusters of breast cancer cells. This hologram embedding allowed us to identify rare and subtle subclusters of the phenotypes overlapped by multiple breast cancer cell types. We demonstrate that our HoloNet efficiently enables LDIH to perform a more detailed analysis of heterogeneity of cell phenotypes for precise breast cancer diagnosis.