BackgroundBesides its neurotransmitter and vasoconstriction functions, serotonin is an important mediator of numerous biological processes in peripheral tissues including cell proliferation, steatosis, and fibrogenesis. Recent reports indicate that serotonin may promote tumor growth in liver cancer, however, the molecular mechanisms remain elusive. n this study, we investigated the role and molecular signaling mechanisms mediated by serotonin in liver cancer cell survival, drug resistance, and steatosis.MethodsEffect of serotonin on modulation of cell survival/proliferation was determined by MTT/WST1 assay. Effect of serotonin on the regulation of autophagy biomarkers and lipid/fatty acid proteins expression, AKT/mTOR and Notch signaling was evaluated by immunoblotting. The role of serotonin in normal human hepatocytes and liver cancer cell steatosis was analyzed by Oil Red O staining. The mRNA expression levels of lipid/fatty acid proteins and serotonin receptors were validated by qRT-PCR. The important roles of autophagy, Notch signaling, serotonin receptors and serotonin re-uptake proteins on serotonin-mediated cell steatosis were investigated by using selective inhibitors or antagonists. The association of peripheral serotonin, autophagy, and hepatic steatosis was also investigated using chronic EtOH fed mouse model.ResultsExposure of liver cancer cells to serotonin induced Notch signaling and autophagy, independent of AKT/mTOR pathway. Also, serotonin enhanced cancer cell proliferation/survival and drug resistance. Furthermore, serotonin treatment up-regulated the expression of lipogenic proteins and increased steatosis in liver cancer cells. Inhibition of autophagy or Notch signaling reduced serotonin-mediated cell steatosis. Treatment with serotonin receptor antagonists 5-HTr1B and 5-HTr2B reduced serotonin-mediated cell steatosis; in contrast, treatment with selective serotonin reuptake inhibitors (SSRIs) increased steatosis. In addition, mice fed with chronic EtOH resulted in increased serum serotonin levels which were associated with the induction of hepatic steatosis and autophagy.ConclusionsSerotonin regulates liver cancer cell steatosis, cells survival, and may promote liver carcinogenesis by activation of Notch signaling and autophagy.Electronic supplementary materialThe online version of this article (10.1186/s12964-018-0282-6) contains supplementary material, which is available to authorized users.