Hierarchical Clustering of OSS License Statements toward Automatic Generation of License Rules

Abstract

Reusing open source software (OSS) components for one's own software products has become common in the modern software development. Automated license identification tools have been proposed to help developers identify OSS licenses, since a large number of licenses sometimes must be checked before attempting to reuse. Of the existing tools, Ninka [1] can most correctly identify licenses of each source file by using regular expressions. In case Ninka does not have license identification rules for unknown licenses, Ninka reports these as "unknown licenses" which must be checked by developers manually. Since completely-new or derived OSS licenses appear nearly every year, a license identification tool should be appropriately maintained by adding regular expressions corresponding to the new licenses. The final goal of our study is to construct a method to automatically create candidate license rules to be added to a license identification tool such as Ninka. Toward achieving the goal, files identified as unknown licenses must be classified by license firstly. In this paper, we propose a hierarchical clustering which divides unknown licenses into clusters of files with a single license. We conduct a case study to confirm the usefulness of our clustering method when it is applied for classifying 2,801, 1,230 and 2,446 unknown license statement files for Linux Kernel v4.4.6, FreeBSD v10.3.0 and Debian v7.8.0 respectively. As a result, it is confirmed that our method can create clusters which are suitable as candidates for generating license rules automatically.

Automating License Rule Generation to Help Maintain Rule-based OSS License Identification Tools

“…Figure 2 shows an overview of the proposed method consisting of the following 5 major processes. 3.1 First, the clustering method of our previous studies [2], [3] is used to classify the unknown license statements according to the similarity of the word vectors (Bag-of-Words). 3.2 Next, the license statements that cannot be classified due to slight differences in minor versions are flittered out as outliers by the similarity based on the Levenshtein distance.…”

“…(Step 3) Generating license rules: Tokenize license statements as regular expressions and generate license rules that can be matched to new licenses. In our previous study [2], [3], we proposed a clustering method to classify the license statements of detected unknown licenses to automate Step 1. In this paper, we focus on Step 2 and Step 3 to automatically generate license rules from each cluster created by the clustering method.…”

“…The goal of our study is to automatically generate candidate license rules to support the creation of license rules. In our previous study [3], we proposed a clustering method to classify unknown license statements by license name. In this paper, we extract expression patterns from clusters classified by our clustering method and generate regular expressions.…”

“…In our previous study [2], [3], we proposed a clustering method to automate the grouping of license statements in Step (I) described in Section 1. It consists of the three parts as follows.…”

“…In the previous study [2], [3], we calculated and evaluated the per-centage of clusters consisting of a single license. The ratio of the single license clusters was high enough for FreeBSD v10.3.0 (91.7%) and Linux Kernel v4.4.6 (90.7%) respectively, but it was not so high for Debian v7.8.0 (69.1%).…”