Melatonin, secreted nocturnally by the pineal gland, can bind to human benign prostate epithelial cells and attenuate their growth and viability. In the present study, melatonin binding and responses were explored in the human steroid-independent PC3 prostatic tumor cells. PC3 cells bound 125I-melatonin with low affinity (Kd ca. 0.9 nM) at high as well as low cell density. Melatonin enhanced cGMP and 3H-thymidine incorporation at low, but attenuated them at high cell density. In addition, melatonin inhibited cAMP at low, but augmented it at high cell density. These effects were associated with an increase in cell count at low- but not high-density cultures. Pertussis toxin treatment suppressed 125I-melatonin binding and ablated all the effects of melatonin on 3H-thymidine incorporation, cAMP, and cGMP at both cell densities. Cholera toxin treatment failed to block the effects of melatonin on 3H-thymidine incorporation, but prevented the modulation by melatonin of cAMP at low and cGMP at high cell density. The cGMP analog 8-Br-cGMP, inhibited melatonin's effects on 3H-thymidine incorporation at both cell densities. H89, a protein kinase A inhibitor, prevented melatonin's effects on 3H-thymidine incorporation at low but not high cell density. These results provide the first demonstration of direct interaction of melatonin with hormone-insensitive prostate tumor cells. The melatonin receptors in the PC3 cells are coupled to pertussis toxin-sensitive G proteins to induce cell density-dependent changes in cGMP, cAMP, and cell growth.