Application of speech recognition and autoreference models for logging tasks

Abstract

Protocols play an important role in decision-making across many fields. The effectiveness of the work of various organizations and teams directly depends on their quality and speed of writing. Therefore, automating this process is of utmost importance today. This article proposes an approach to address the issue of logging by processing existing audio recordings of meetings or events and using an ensemble of artificial intelligence models, including a pre-trained acoustic speech recognition model based on the "Quartznet" architecture, a language N-gram model implemented using the "KenLM" tool, and a Russian-language model "RuBERT" retrained for extractive abstracting. The algorithms of these models' operation are examined, and the processes of data collecting and retraining, as well as the implementation of the selected models, are described. Quality metrics for the speech recognition system are compared, and an analysis of the implemented systems is conducted. To utilize the developed system an HTTP web server was deployed via an API. The final results of the developed automatic logging system, capable of extracting timestamps of spoken words, highlighting speakers in audio recordings, and reducing the resulting text to a specified percentage are demonstrated.