The importance of downside risk cannot be denied. In this study, we have replaced beta in the five-factor model of using downside beta and have added a momentum factor to suggest a new six-factor downside beta capital asset pricing model (CAPM). Two models are tested—a beta- and momentum-based six-factor model and a downside-beta- (proxy of downside risk) and momentum-based six-factor model. Beta and downside beta are highly correlated; therefore, portfolios are double-sorted to disentangle the correlation. Factor loadings, i.e., size, value, momentum, profitability, and investment, are constructed. The standard methodologies are applied. Data for sample stocks from different non-financial sectors listed in the Pakistan Stock Exchange (PSX) are taken from January 2000 to December 2018. The PSX-100 index and three-month T-bills are taken as proxies for market and risk-free returns. The study uses three subsamples for robustness—period of very high volatility, period of stability, and period of stability and growth with volatility. The results show that the value factor is redundant in both models. The momentum factor is rejected in the beta-based six-factor model only. The beta-based six-factor model shows very low R2 in periods of highly volatility. The R2 is high for the other periods. In contrast, the downside beta six-factor model captures the downside trend of the market in an effective manner with a relatively high R2. The risk–return relationship is stronger for the downside beta model. These reasons lead us to believe that, overall, the downside beta six-factor model is a better option for investors as compared to the beta-based six-factor model in the area of asset pricing models.