Spectral domain optical coherence tomography (SD-OCT) is a non-invasive imaging modality for assessing retinal diseases, such as diabetic retinopathy (DR), which is one of the most prevalent diseases responsible for visual impairment and blindness in the world. The main manifestations of DR are retinal deformation and fluid masses, termed diabetic macular edema (DME), which is the primary biomarker for assessing and diagnosing diseases. In the clinic, ophthalmologists can manually segment retinal layers and fluids to get quantitative and diagnostic information, which is the basement of the final diagnosis. However, this manual segmentation is time-consuming and labor-intensive. To facilitate and promote it, researchers have proposed many automated methods, where most of them usually ignore the priorities in ophthalmology and just regard this task as a standard semantic segmentation task. In this study, we consider the priority of the mutex relationship among different layers and introduce it into the dice loss function to build a novel one, named mutex dice loss (MDL). Besides, we propose a novel fully convolutional network based on our proposed depth max pooling (DMP) to segment retinal layers and fluids in SD-OCT images. Experimental results of the proposed method on two public datasets demonstrate promising performance, which also shows the potential to help ophthalmologists in the diagnostic process of DR or other related diseases. INDEX TERMS Diabetic macular edema (DME), retinal layer segmentation, fully convolutional network, mutex dice loss.