A deep learning approach to identifying source code in images and video

Abstract

While substantial progress has been made in mining code on an Internet scale, efforts to date have been overwhelmingly focused on data sets where source code is represented natively as text. Large volumes of source code available online and embedded in technical videos have remained largely unexplored, due in part to the complexity of extraction when code is represented with images. Existing approaches to code extraction and indexing in this environment rely heavily on computationally intense optical character recognition. To improve the ease and efficiency of identifying this embedded code, as well as identifying similar code examples, we develop a deep learning solution based on convolutional neural networks and autoencoders. Focusing on Java for proof of concept, our technique is able to identify the presence of typeset and handwritten source code in thousands of video images with 85.6%-98.6% accuracy based on syntactic and contextual features learned through deep architectures. When combined with traditional approaches, this provides a more scalable basis for video indexing that can be incorporated into existing software search and mining tools. CCS CONCEPTS • Information systems → Video search; • Computing methodologies → Machine learning approaches; • Computer systems organization → Neural networks; • Software and its engineering → Software libraries and repositories;

Exploring the applicability of low-shot learning in mining software repositories

“…The VGG network shown in Fig. 1b is a very popular model used across many domains, and was chosen because it has recently been applied to software mining [2]. The VGG model has a convenient architecture in which multiple convolutional operations occur in succession, followed by a max pooling layer for down-sampling.…”

“…1 School of Information and Computer Science, University of California, Irvine, Irvine, CA, USA. 2 Fowler School of Engineering, Chapman University, One University Dr., Orange, CA 92866, USA.…”

“…

IntroductionIn the past couple of years, applications of deep learning to mining software repositories have grown in number and diversity of methods [1][2][3][4][5]. Fueled in part by easy-touse libraries and graphics processing unit (GPU) computing, deep architectures have facilitated new avenues for research, often producing results that far surpass previous techniques.

…”

FVT: A Fragmented Video Tutor for "Dubbing" Software Development Tutorials

“…Yadid et al [3] present ACE, a tool that combines language models and image processing techniques to extract source code from software development videos. Ott et al [2] further improved the identification of code fragments using deep learning algorithm. Ponzanelli et al [4,14] employed the extracted code as features to segment development videos, which can efficiently assist developers to focus on the key point in video.…”

“…Many efforts have been taken to enhance software development tutorials [2,3,4,5], which could be classified into two categories: text tutorials and video tutorials. However, they may not be sufficient to meet the targeted learning needs if used individually.…”