An image texture was defined in terms of pixel intensities and directionality. However, most of the current texture representation methods did not consider the two key factors simultaneously. To effectively capture the directional and pixel intensity information of texture, in this paper, we propose a novel and robust local descriptor, named locally directional and extremal pattern (LDEP), for texture classification. It extracts directional local difference count pattern (DLDCP) being made up of DLDCP in the odd positions and DLDCP in the even positions to express directional information in the local area in the first place. Furthermore, to acquire the extremum information remained by DLDCP, by concatenating extremum location pattern (ELP), extremum difference pattern (EDP), and extremum compression pattern (ECP) from the sampling points, we extract a neighbors extremum related local pattern (NERLP). The experimental results obtained from four representative texture databases (Prague, Stex, UIUC, Kth-tips2-a, Brodatz, and CUReT) demonstrate that our proposed LDEP descriptor can achieve comparable accurate classification rates in different conditions (rotation, illumination, scale variation, viewpoint variation, and noise) with ten typical texture classification methods. INDEX TERMS Directional pattern, extremal pattern, local pattern, texture representation, texture classification.