A Tree-Based Structure-Aware Transformer Decoder for Image-To-Markup Generation

Zhong, Shuhan; Song, Sizhe; Li, Guanyao; Chan, S.-H. Gary

doi:10.1145/3503161.3548424

Cited by 20 publications

(15 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To further improve the performance of encoder, DenseWAP [14] improved on the basis of WAP, used a more efficient DenseNet as the encoder, and proposed a multi-scale attention model, which can handle mathematical symbols of different scales well, and retain more details lost due to pooling operations. Later, he proposed DenseWAP-TD [15] model, which replaced string decoder with tree structure decoder based on DenseWAP to improve the parsing ability of complex structures. Li et al [16] proposed a HMER method with scale enhancement and drop attention.…”

Section: Hmer Methodsmentioning

confidence: 99%

“…At the inference stage, beam search is applied to find the output with the maximum probability, and the beam size is set to 4. In order to verify the effectiveness of the ClipMath we propose, we will compare it with other state-of-the-art methods, including DenseWAP [14], PAL-v2 [30], WS-WAP [31], DenseWAP-MSA [14], DenseWAP-TD [15], BTTR [11], and DATWAP [32]. We take DenseWAP as the baseline, and do not use any data enhancement in the experiment.…”

Section: Experimental Configurationmentioning

confidence: 99%

See 1 more Smart Citation

Offline Handwritten Mathematical Expression Recognition based on CLIP enhancement

2022

JRSE

View full text Add to dashboard Cite

Handwritten Mathematical Expression Recognition (HMER) aims to convert complex mathematical expression images into LaTeX expressions, which is of great significance in electronic documents and electronic education. The attention-based encoder-decoder architecture is very popular in HMER tasks. However, in the absence of large and medium-sized handwritten mathematical expression datasets, it is impossible to establish a pre-training task, and it is difficult to train a strong encoder from scratch. In addition, usually the decoder directly uses the high-level semantic features extracted by the encoder for decoding, and the low-level details are filtered out in the pooling layer, so the decoder pays little attention to the low-level information. In order to alleviate these two problems, this paper proposes a new ClipMath model. Its encoder is extended to a formula-text matching network based on CLIP-enhancement, including a formula-head and a text-head, which are used to cross-modal alignment of the image-state formula and the textual LaTeX label. The decoder uses the proposed Cascaded Multi-scale Decoder, which can incorporate features of different scales in the decoding process and improve the attention to detail information. Experiments show that the ExpRate of the proposed method on CROHME 2014, CROHME2016 and CROHME 2019 is 1.22%, 0.64% and 2.01% higher than the state-of-the-art model respectively.

show abstract

Section: Hmer Methodsmentioning

confidence: 99%

Section: Experimental Configurationmentioning

confidence: 99%

Offline Handwritten Mathematical Expression Recognition based on CLIP enhancement

2022

JRSE

View full text Add to dashboard Cite

show abstract

“…To improve the encoder's effectiveness, we apply sequence self-attention and ASG self-attention to the encoder to highlight dependencies between every node pair and every adjacent ASG node pair. Following existing work [10], [18]- [20], we use two vectors/matrices ("ASG Node Sequence" and "Adjacency Matrix" in Figure 2) to represent each ASG, which are the input formats that a deep learning encoder can take. The ASG Node Sequence, notated by {n i } = (n 1 , n 2 , .…”

Section: Encodermentioning

confidence: 99%

KNOD: Domain Knowledge Distilled Tree Decoder for Automated Program Repair

Jiang¹,

Lutellier²,

Lou³

et al. 2023

Preprint

View full text Add to dashboard Cite

Automated Program Repair (APR) improves software reliability by generating patches for a buggy program automatically. Recent APR techniques leverage deep learning (DL) to build models to learn to generate patches from existing patches and code corpora. While promising, DL-based APR techniques suffer from the abundant syntactically or semantically incorrect patches in the patch space. These patches often disobey the syntactic and semantic domain knowledge of source code and thus cannot be the correct patches to fix a bug.We propose a DL-based APR approach KNOD, which incorporates domain knowledge to guide patch generation in a direct and comprehensive way. KNOD has two major novelties, including (1) a novel three-stage tree decoder, which directly generates Abstract Syntax Trees of patched code according to the inherent tree structure, and (2) a novel domain-rule distillation, which leverages syntactic and semantic rules and teacher-student distributions to explicitly inject the domain knowledge into the decoding procedure during both the training and inference phases.We evaluate KNOD on three widely-used benchmarks. KNOD fixes 72 bugs on the Defects4J v1.2, 25 bugs on the QuixBugs, and 50 bugs on the additional Defects4J v2.0 benchmarks, outperforming all existing APR tools.

show abstract

“…Context matching loss is adapted to constrain the intra-class distance and enhance the discriminative power of model. Lately, Zhang et al [46] devised a tree-based decoder to parse mathematical expression. At each step, a parent and child node pair was generated and the relation between parent node and child node reflects the structure type.…”

Section: Related Workmentioning

confidence: 99%

“…As shown in tab. 1 and 2, we compare our prosposed method with DWAP [42], DWAP-TD [46], BTTR [51], ABM [3], SAN [39] and CAN [17] on HME100K dataset. It is clear to notice that SAM-DWAP and SAM-CAN achieves the best performance.…”

Section: Comparison With State-of-the-artmentioning

confidence: 99%

Semantic Graph Representation Learning for Handwritten Mathematical Expression Recognition

Liu,

Yuan,

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Handwritten mathematical expression recognition (HMER) has attracted extensive attention recently. However, current methods cannot explicitly study the interactions between different symbols, which may fail when faced similar symbols. To alleviate this issue, we propose a simple but efficient method to enhance semantic interaction learning (SIL). Specifically, we firstly construct a semantic graph based on the statistical symbol co-occurrence probabilities. Then we design a semantic aware module (SAM), which projects the visual and classification feature into semantic space. The cosine distance between different projected vectors indicates the correlation between symbols. And jointly optimizing HMER and SIL can explicitly enhances the model's understanding of symbol relationships. In addition, SAM can be easily plugged into existing attention-based models for HMER and consistently bring improvement. Extensive experiments on public benchmark datasets demonstrate that our proposed module can effectively enhance the recognition performance. Our method achieves better recognition performance than prior arts on both CROHME and HME100K datasets.

show abstract

A Tree-Based Structure-Aware Transformer Decoder for Image-To-Markup Generation

Cited by 20 publications

References 18 publications

Offline Handwritten Mathematical Expression Recognition based on CLIP enhancement

Offline Handwritten Mathematical Expression Recognition based on CLIP enhancement

KNOD: Domain Knowledge Distilled Tree Decoder for Automated Program Repair

Semantic Graph Representation Learning for Handwritten Mathematical Expression Recognition

Contact Info

Product

Resources

About