The Convolutional Tsetlin Machine

Abstract

Deep neural networks have obtained astounding successes for important pattern recognition tasks, but they suffer from high computational complexity and the lack of interpretability. The recent Tsetlin Machine (TM) attempts to address this lack by using easy-to-interpret conjunctive clauses in propositional logic to solve complex pattern recognition problems. The TM provides competitive accuracy in several benchmarks, while keeping the important property of interpretability. It further facilitates hardware-near implementation since inputs, patterns, and outputs are expressed as bits, while recognition and learning rely on straightforward bit manipulation. In this paper, we exploit the TM paradigm by introducing the Convolutional Tsetlin Machine (CTM), as an interpretable alternative to convolutional neural networks (CNNs). Whereas the TM categorizes an image by employing each clause once to the whole image, the CTM uses each clause as a convolution filter. That is, a clause is evaluated multiple times, once per image patch taking part in the convolution. To make the clauses location-aware, each patch is further augmented with its coordinates within the image. The output of a convolution clause is obtained simply by ORing the outcome of evaluating the clause on each patch. In the learning phase of the TM, clauses that evaluate to 1 are contrasted against the input. For the CTM, we instead contrast against one of the patches, randomly selected among the patches that made the clause evaluate to 1. Accordingly, the standard Type I and Type II feedback of the classic TM can be employed directly, without further modification. The CTM obtains a peak test accuracy of 99.51% on MNIST, 96.21% on Kuzushiji-MNIST, 89.56% on Fashion-MNIST, and 100.0% on the 2D Noisy XOR Problem, which is competitive with results reported for simple 4-layer CNNs, BinaryConnect, and a recent FPGA-accelerated Binary CNN.

A relational tsetlin machine with applications to natural language understanding

“…We finally introduce a novel convolution scheme that effectively manages multiple possible mappings from constants to variables. While the mechanism of convolution remains the same as in the original [10], what we wish to attain from using it in a Relational TM context is completely different, as explained in the following.…”

“…To avoid redundant rules, the Relational TM produces all possible permutations of variable assignments. To process the different permutations, we finally perform a convolution over the permutations in Step 4, employing a TM convolution operator [10]. The target value of the convolution is the truth value of the consequent (Step 5).…”

“…While the idea of Tsetlin automaton (TA) [4] have been around since 1960s, using them in pattern recognition is relatively new. TMs have successfully addressed several machine learning tasks, including natural language understanding [5], [6], [7], [8], [9], image analysis [10], classification [11], regression [12], and speech understanding [13]. The propositional clauses constructed by a TM have high discriminative power and constitute a global description of the task learnt [8], [14].…”

Coalesced Multi-Output Tsetlin Machines with Clause Sharing

“…TMs [19] have recently demonstrated competitive accuracy-, memory footprint-, energy-, and learning speed on several benchmarks, spanning tabular data [2,40], images [20,37], regression [4], natural language [6,45,46,9,44], and speech [27]. By not relying on minimizing output error, TMs are less prone to overfitting.…”

On the Convergence of Tsetlin Machines for the AND and the OR Operators

“…There are many variations of TMs with two main architectures: the convolutional TM (CTM) [10] and the regression TM (RTM) [11,12]. The TM, together with its variations, has been employed in several applications, such as word sense disambiguation [13], aspect-based sentiment analysis [14], novalty detection [15], and interpretable NLP [16].…”