Approaches to visualizing big text data at the stage of collection and pre-processing

Makarova, E. A.; Lagerev, D. G.; Lozbinev, Fedor

doi:10.26583/sv.11.4.02

Cited by 3 publications

(2 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In previous works of the authors [8] concerning the process of collecting and processing semi-structured data, much attention was paid to the "hybrid" approach in the field of developing data analysis systems. In this approach, human expertise is used in conjunction with automatic analysis methods, which allows, on the one hand, to improve the quality of the system on tasks that cannot be solved without human intervention, and on the other, to relieve the expert from solving typical, routine tasks.…”

Section: Management Decision-making In Healthcarementioning

confidence: 99%

“…In this approach, human expertise is used in conjunction with automatic analysis methods, which allows, on the one hand, to improve the quality of the system on tasks that cannot be solved without human intervention, and on the other, to relieve the expert from solving typical, routine tasks. This was achieved using various methodologies for pre-processing and data visualization, which helped the expert make faster decisions about the inclusion / exclusion of a particular data source [8].…”

Section: Management Decision-making In Healthcarementioning

confidence: 99%

See 1 more Smart Citation

Methodology for Preprocessing Semi-Structured Data for Making Managerial Decisions in the Healthcare

Makarova

Lagerev

2020

Proceedings of the 30th International Conference on Computer Graphics and Machine Vision (GraphiCon 2020). Part 2

Self Cite

View full text Add to dashboard Cite

This paper describes the process of supporting management decisionmaking in healthcare based on data mining. The authors described various problems and specifics of data in medical information systems, leading to the complexity of their analysis and integration, such as: the presence of a large number of specific abbreviations, errors in the data and their poor structure. The paper demonstrates an approach to the search and further disclosure of abbreviations in texts, built on a combination of machine and human processing. A method for extracting features from semi-structured fields using an expert in the subject area and using various visualizations is proposed. The proposed abbreviation search and disclosure methods, based on a hybrid approach combining the strengths of processing with the help of a machine and an expert, can increase the number of abbreviations found automatically and significantly reduce the time spent by experts on processing the remaining reductions. In addition, the method for automated feature extraction during integration can significantly increase the amount of useful input data, while reducing the time of the expert.

show abstract

Section: Management Decision-making In Healthcarementioning

confidence: 99%

Section: Management Decision-making In Healthcarementioning

confidence: 99%

Methodology for Preprocessing Semi-Structured Data for Making Managerial Decisions in the Healthcare

Makarova

Lagerev

2020

Proceedings of the 30th International Conference on Computer Graphics and Machine Vision (GraphiCon 2020). Part 2

Self Cite

View full text Add to dashboard Cite

show abstract

Features of Preliminary Processing of Semi-Structured Medical Data in Russian for Use in Ensembles of Data Mining Models

Lagerev¹,

Makarova²

2020

vkit

View full text Add to dashboard Cite

The paper considers the problem of integration, processing and mining of poorly structured data of medical information systems in order to make managerial decisions in healthcare. The problems of medical data are described, such as the lack of a sufficient structure, a large number of abbreviations characteristic of specific nosologies, the complexity of the automatic semantic interpretation of some fields. The authors demonstrated an approach to the search and disclosure of abbreviation in texts, based on a combination of machine and human processing. The method proposed by the authors, based on a hybrid approach combining the strengths of machine and human processing, made it possible to increase the number of abbreviations found by automatic methods by 21 %, and also opened up to 55 % of cases in the automated mode (with a probability of correctness above 70 %) and significantly reduce the time spent by specialists in processing the remaining reductions. Further research will be aimed at solving the following problems associated with the processing and specificity of medical data, such as a large number of spelling errors, specific grammatical constructions. Using a hybrid approach to preprocessing poorly structured data will increase the efficiency of management decisions in the field of healthcare by reducing the time spent by experts on their creation and support. The hybrid approach to the preprocessing of text data in Russian can be applied in other subject areas. However, it may be necessary to adjust the technique to the specifics of the processed data.

show abstract

Determining the Semantic Proximity of News Messages Based on Titles Analysis

Lagerev¹,

Makarova²

2021

vkit

View full text Add to dashboard Cite

The paper is devoted to the peculiarities of using data from unstructured sources, such as social networks, online media, etc. for the development of management decisions. The issue of analyzing such sources in the process of developing and making managerial decisions in the socio-economic sphere is considered. The difficulties of processing unstructured data are described, such as the impossibility of fully automatic evaluation of the semantics of data, the presence of a large amount of duplicate information. Various approaches to determining ratings and various metrics of sources and messages are described. The problem of identifying duplicate messages is considered using the example of online media both by full texts and by titles. Under the duplicate of a news message in this context refers a repetition of a significant amount of information from it in another article. If the text of a news message (article) is not repeated verbatim, it is impossible to determine the degree of duplication of the meaning of the article without involving human expertise. There are various metrics for assessing the similarity (semantic proximity) of textual information that can help in this matter. Some of them are described in the article. An adaptation of the Word Mover Distance method for the Russian language is proposed and the Word2Vec model is trained for its use. A hybrid approach to identifying and eliminating duplicate messages as part of the preprocessing of unstructured data in the context of managerial decision-making is proposed. According to the results of the experiments, depending on the chosen method, it was possible to automatically determine based on the publication time and the analysis of titles from 43 to 74 % duplicates.

show abstract

Approaches to visualizing big text data at the stage of collection and pre-processing

Cited by 3 publications

References 9 publications

Methodology for Preprocessing Semi-Structured Data for Making Managerial Decisions in the Healthcare

Methodology for Preprocessing Semi-Structured Data for Making Managerial Decisions in the Healthcare

Features of Preliminary Processing of Semi-Structured Medical Data in Russian for Use in Ensembles of Data Mining Models

Determining the Semantic Proximity of News Messages Based on Titles Analysis

Contact Info

Product

Resources

About