In many marine ecosystems, diatoms dominate in nutrient-rich coastal waters while coccolithiophores are found offshore in areas where nutrients may be limiting. In lab-controlled batch cultures, mixed-species competition between the diatom Phaeodactylum tricornutum and the coccolithophore Emiliana huxleyi and the response of each species were examined under nitrate (N) and phosphate (P) starvation. Based on the logistic growth model and the Lotka-Volterra competition model, E. huxleyi showed higher competitive abilities than P. tricornutum under N and P starvation. For both species, cell growth was more inhibited by P starvation, while photosynthetic functions (chl a fluorescence parameters) and cellular constituents (pigments) were impaired by N starvation. The decline of photosynthetic functions occurred later in E. huxleyi (day 12) than in P. tricornutum (day 9); this time difference was associated with greater damage of the photosynthetic apparatus in P. tricornutum compared with E. huxleyi. Xanthophyll cycle pigment accumulation and the transformation from diadinoxanthin to diatoxanthin was more active in E. huxleyi than P. tricornutum, under similar N and P starvation. We concluded that E. huxleyi and P. tricornutum have different mechanisms to allocate resources and energy under nutrient starvation. It appears that E. huxleyi has a more economic strategy to adapt to nutrient depleted environments than P. tricornutum. These findings provided additional evidence explaining how N versus P limitation differentially support diatom and coccolithophore blooms in natural environments.