Glaucoma is a chronic eye disease that can cause permanent visual loss and is difficult to detect early. Retinal nerve fiber layer defect (RNFLD) is clinical evidence for the diagnosis of glaucoma. Classical deep learning based methods can be used to segment RNFLD from fundus images. However, the segmentation results of these methods do not have the specific geometry of RNFLD, and the segmentation errors of fundus images with special styles are large. In this paper, we present a novel conditional adversarial shuffle U-shaped network (CASU-Net) to segment RNFLD, which consists of a generator and a discriminator. For the generator, a mixed loss is designed, which consists of an adaptive weighted segmentation loss and an adversarial loss. This adaptive weighted segmentation loss can balance the segmentation accuracy of the target and background region, and assign more attention to the hard samples, thus ensuring the consistent improvement of the segmentation accuracy of all fundus images. The adversarial loss not only helps to improve the pixel-wise segmentation accuracy but also makes the geometry of the RNFLD segmentation closer to the ground truth. In addition, in the generator, a shuffle module was designed to fully mine the information of all channels to improve the feature extraction capability of the model. The proposed CASU-Net is verified on a RNFLD dataset from Beijing Tongren Hospital. The experiments show that the CASU-Net achieves state-of-the-art results on this dataset. INDEX TERMS Glaucoma, retinal nerve fiber layer defect segmentation, deep learning.