VID2XML: Automatic Extraction of a Complete XML Data From Mobile Programming Screencasts

Optimizing OCR Performance for Programming Videos: The Role of Image Super-Resolution and Large Language Models

“…It is important to note that much of the existing research utilizes the source code displayed in videos for various purposes, including identifying actions [27,34,35], classifying the source code [14], and tagging videos [29]. This underscores the necessity of selecting the most effective OCR (optical character recognition) engines and LLMs (large language models) tailored for extracting code from videos of varying quality, thereby facilitating the reuse of code for diverse applications.…”

“…Similarly, Malkadi et al [13] utilized a Bert-based model to denoise the extracted code. Recent work used Google Vision OCR to extract source code from videos for restructuring complete Android XML files [14], extracting meta information [15], and linking code to the relevant video fragment [30].…”

“…This method extracts the textual content from the images, facilitating the transformation of pixelated code into a format that developers can easily integrate into their projects. Over the last decade, many efforts have been dedicated to facilitating the process of extracting source code from video programming tutorials [7][8][9][10][11][12][13][14][15][16]. Several approaches have been proposed to clean the extracted code, such as by (i) using statistical language models [10,16] for Java language, (ii) verifying the extracted tokens using a dictionary-based approach [14,15,17], and (iii) using the CodeT5 model [18] to correct Java code [13].…”

“…Over the last decade, many efforts have been dedicated to facilitating the process of extracting source code from video programming tutorials [7][8][9][10][11][12][13][14][15][16]. Several approaches have been proposed to clean the extracted code, such as by (i) using statistical language models [10,16] for Java language, (ii) verifying the extracted tokens using a dictionary-based approach [14,15,17], and (iii) using the CodeT5 model [18] to correct Java code [13]. Other work focused on developing an ensemble technique that improves OCR performance on specific domains, such as extracting texts from (i) geology papers using Softmax regression and SVM [19], (ii) raster images with Arabic script using SVM [20], and (iii) an IAM handwritten dataset using CRNN, LSTM, and CTC [21].…”

“…Yet, previous work has extracted the code from images, then created a custom model for a specific programming language (e.g., Java) to detect and correct errors [10,13,16], without initially addressing image quality or employing image denoising techniques. Finally, previous work has relied on off-the-shelf OCR engines such as Tesseract [7,22] and Google Vision [14,15,23] for extracting source code without investigating the state-of-the-art visionbased large language models (LLMs), which could significantly enhance code extraction performance dramatically.…”