Glaucoma is an eye disease that can cause loss of vision by damaging the optic nerve. It is the world's second leading cause of blindness after cataracts. Early diagnosis of glaucoma is a key to prevent permanent blindness as it has no noticeable symptoms in its early stages. Color fundus photography is used for examining the optic disc (OD) which is an important step in the diagnoses of glaucoma. This is done by estimating the cup-to-disc ratio (CDR). In this paper, we proposed a Cup Disc Encoder Decoder Network (CDED-Net) for the joint segmentation of optic disc (OD) and optic cup (OC). We have eradicated the preprocessing and post-processing steps to reduce the computational cost of the overall system. Segmentation of (OD) and OC is modeled as a semantic pixel-wise labeling problem. The model was trained on the DRISHTI-GS, RIM-ONE and REFUGE datasets. Experiments show that our CDED-Net system achieves state-of-the-art OD and OC segmentation results on these datasets.