This paper describes a scenario where D2D assisted cooperative relaying downlink transmission utilizing non-orthogonal multiple access (DC-NOMA) in the same time or frequency resource. Aiming to enhance the performance including ergodic sum rate (ESR) of the whole system and output block error rate (BER) of detection receiver. Firstly, we investigate the convexity of ESR maximization optimization problem and then prove its non-strictly concave property. Unlike existing works of optimizing the power coefficient only in one-stage, a jointly two-stage power allocation scheme based on iterative search is proposed for the formulated ESR maximizing problem. It is proved that the proposed algorithm converges to a stable state within limited numbers of iteration. Secondly, in order to improve the detection performance of the NOMA system downlink receiver, we propose a novel receiving detection algorithm based on the max-log-map algorithm and design a new receiver structure adapted to this algorithm. Finally we confirm the feasibility and verify the enhancement of the receiver design. The numerical results illustrate that: i) the jointly two-stage power allocation scheme can achieve up to 45.6% of performance gains in term of ESR in comparison of with the one-stage optimizing scheme; and ii) the delay and BER of the receiver both decrease significantly using novel receiver detection algorithm in the case of high-order signal modulation and multiuser multiplexing. INDEX TERMS NOMA, D2D, cooperative communication, power allocation, receiver design.