Data Mining applied on Web Robots Detection: A Systematic Mapping

Abstract

Browsing on Internet is part of the world population’s daily routine. The number of web pages is increasing and so is the amount of published content (news, tutorials, images, videos) provided by them. Search engines use web robots to index web contents and to offer better results to their users. However, web robots have also been used for exploiting vulnerabilities in web pages. Thus, monitoring and detecting web robots’ accesses is important in order to keep the web server as safe as possible. Data Mining methods have been applied to web server logs (used as data source) in order to detect web robots. Then, the main objective of this work was to observe evidences of definition or use of web robots detection by analyzing web server-side logs using Data Mining methods. Thus, we conducted a systematic Literature mapping, analyzing papers published between 2013 and 2020. In the systematic mapping, we analyzed 34 studies and they allowed us to better understand the area of web robots detection, mapping what is being done, the data used to perform web robots detection, the tools, and algorithms used in the Literature. From those studies, we extracted 33 machine learning algorithms, 64 features, and 13 tools. This study is helpful for researchers to find machine learning algorithms, features, and tools to detect web robots by analyzing web server logs.

Análise de Técnicas de Aumento de Dados para Processamento de Linguagem Natural

“…Aplicac ¸ões interessantes envolvendo PLN podem ser notadas em artigos como [3], [4] que desenvolvem algoritmos para coleta de conteúdos com opiniões e informac ¸ões relevantes da rede social Twitter. No trabalho [5] foi realizada, com auxílio de PLN, uma minerac ¸ão de dados em logs de servidores Web para detecc ¸ão de robôs de mecanismos de busca. Já nos artigos [6] e [7], uma arquitetura inovadora chamada Generative Pre-trained Transformer (GPT) mostra o quão impactante as implementac ¸ões em PLN podem se tornar, pois o GPT foi o primeiro passo para criac ¸ão do famoso modelo de linguagem ChatGPT.…”