In the online social networks, blogs, microblogs, social bookmarking services and sharing sites, and various web forum pages; the sharing of knowledge, opinions, ideas, etc. are spreading very quickly. This situation brings very dangerous problems in social networks. One of these problems is hate speech detection (HSD) problem which is covering issues such as insults, swearing, humiliation, discrimination, exclusion, detest, abhor, blast, damn, and intolerance. These can be reactions to a person, a group, an organization, an order, or an event. Although few machine learning methods have been used in the literature to solve this important problem in online social media, the performance of the HSD models in terms of many metrics needs to be increased. In this study, an automatic HSD system based on metaheuristic methodology was proposed for better results in this new and important problem. In the proposed optimization approach, Ant Lion Optimization (ALO) algorithm and Moth Flame Optimization (MFO) algorithm were designed for the HSD problem. This is the first attempt to use optimization algorithms as solution search strategies for automatic HSD. An efficient representation scheme and flexible fitness function were designed for this purpose. Many metrics can easily be embedded into the designed fitness function in order to be simultaneously optimized. Firstly, the basic natural language processing (NLP) steps were carried out. Feature extraction was performed using Bag of Words (BoW), Term Frequency (TF), and document vector (Word2Vec). Then, the performances of the proposed novel approaches were analyzed in detail on the three different real-world data. The obtained results were also checked against eight popular supervised machine learning algorithms, Social Spider Optimization (SSO) algorithm, and state-of-the-art Tunicate Swarm Algorithm (TSA). Considering the evaluation criteria for three sets of experiments, it was observed that the accuracy, sensitivity, precision, and f-score results of the ALO and MFO algorithms were superior to machine learning methods. As a result of the experimental studies, the highest accuracy value was 92.1% for ALO, while this value was 90.7% for MFO. Other numerical values obtained in the study were given in the experiments and results section with tables and graphics in detail. Due to the promising results of the proposed approaches, they are anticipated to be used in the solution of many social media and networking problems.INDEX TERMS Hate speech detection, metaheuristic optimization, natural language processing, social network analysis, text mining.