With the advent of medical endoscopes, earth observation satellites and personal phones, content-based image retrieval (CBIR) has attracted considerable attention, triggered by its wide applications, e.g., medical image analytics, remote sensing, and person re-identification. However, constructing effective feature extraction is still recognized as a challenging problem. To tackle this problem, we first propose the five-level color quantizer (FLCQ) to acquire a color quantization map (CQM). Secondly, according to the anatomical structure of the human visual system, the color quantization map (CQM) is amalgamated with a local binary pattern (LBP) map to construct a local ternary cross structure pattern (LTCSP). Third, the LTCSP is further converted into the uniform local ternary cross structure pattern (LTCSPuni) and the rotation-invariant local ternary cross structure pattern (LTCSPri) in order to cut down the computational cost and improve the robustness, respectively. Finally, through quantitative and qualitative evaluations on face, objects, landmark, textural and natural scene datasets, the experimental results illustrate that the proposed descriptors are effective, robust and practical in terms of CBIR application. In addition, the computational complexity is further evaluated to produce an in-depth analysis.