Simulated annealing for optimization of graphs and sequences

Liu, Xianggen; Li, Pengyong; Meng, Fandong; Zhou, Hao; Zhong, Huasong; Zhou, Jie; Mou, Lili; Song, Sen

doi:10.1016/j.neucom.2021.09.003

Cited by 20 publications

(17 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our work on unsupervised summarization follows the recent progress of search-based text generation (Liu et al, , 2021aKumar et al, 2020). Schumann et al (2020) formulate summarization as word-level extraction (with order preserved), and apply edit-based discrete local search to maximize a heuristically designed objective.…”

Section: Search-based Summarizationmentioning

confidence: 99%

Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization

Liu¹,

Huang²,

Mou³

2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Self Cite

View full text Add to dashboard Cite

Text summarization aims to generate a short summary for an input text. In this work, we propose a Non-Autoregressive Unsupervised Summarization (NAUS) approach, which does not require parallel data for training. Our NAUS first performs edit-based search towards a heuristically defined score, and generates a summary as pseudo-groundtruth. Then, we train an encoder-only non-autoregressive Transformer based on the search result. We also propose a dynamic programming approach for length-control decoding, which is important for the summarization task. Experiments on two datasets show that NAUS achieves state-of-the-art performance for unsupervised summarization, yet largely improving inference efficiency. Further, our algorithm is able to perform explicit length-transfer summary generation. 1

show abstract

Section: Search-based Summarizationmentioning

confidence: 99%

Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization

Liu¹,

Huang²,

Mou³

2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In this work, we propose a heuristic, non-convex optimization algorithm, namely simulated annealing (SA), for the structure optimization of partially connected neural networks after pruning [47]. The choice of simulated annealing has been motivated by the success of the algorithm in various problems involving network/graph structures with a large number of configurations and complicated cost surfaces with various local minima [48][49][50].…”

Section: Network Optimization Using Simulated Annealingmentioning

confidence: 99%

“…After updating the temperature for approximately 100 to 150 iterations, the network performance is observed to converge. The time consumption of Algorithm 1 for different MLLs (1,10,20,30,40,50,60,70,80,90) are measured for each one of three different fractions of gradual pruning percentage p (2%, 5%, 10%) in Figure 10. Even though the extreme cases are included in the time consumption evaluation; MLLs of 20 to 50 seem to be enough, as can be observed in Figure 7.…”

Section: Time Complexity Of the Sa-based Pruning Processmentioning

confidence: 99%

Neural Network Structure Optimization by Simulated Annealing

Kuo

Kuruoğlu

Chan

2022

Entropy

View full text Add to dashboard Cite

A critical problem in large neural networks is over parameterization with a large number of weight parameters, which limits their use on edge devices due to prohibitive computational power and memory/storage requirements. To make neural networks more practical on edge devices and real-time industrial applications, they need to be compressed in advance. Since edge devices cannot train or access trained networks when internet resources are scarce, the preloading of smaller networks is essential. Various works in the literature have shown that the redundant branches can be pruned strategically in a fully connected network without sacrificing the performance significantly. However, majority of these methodologies need high computational resources to integrate weight training via the back-propagation algorithm during the process of network compression. In this work, we draw attention to the optimization of the network structure for preserving performance despite compression by pruning aggressively. The structure optimization is performed using the simulated annealing algorithm only, without utilizing back-propagation for branch weight training. Being a heuristic-based, non-convex optimization method, simulated annealing provides a globally near-optimal solution to this NP-hard problem for a given percentage of branch pruning. Our simulation results have shown that simulated annealing can significantly reduce the complexity of a fully connected network while maintaining the performance without the help of back-propagation.

show abstract

“…To improve the dissimilarity between the outputs and input sentences, Lin and Wan [2021] leverage multi-round paraphrase generation and back translation. Also, Liu et al [2021] propose multi-round modification in hope of lexically different sequences (e.g., paraphrases). In addition, Li et al [2019] generate paraphrases of a sentence at different levels of granularity in a disentangled way.…”

Section: Related Workmentioning

confidence: 99%

“…The abstract rule was first introduced to study the language acquisition of infants [Marcus et al, 1999]. In this paper, we use an abstract rule to represent a type of paraphrase transformation.…”

Section: Abstract Rules In Paraphrase Generationmentioning

confidence: 99%

Abstract Rule Learning for Paraphrase Generation

Wang

Fang

et al. 2022

Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence

Self Cite

View full text Add to dashboard Cite

Data-free knowledge distillation (DFKD) conducts knowledge distillation via eliminating the dependence of original training data, and has recently achieved impressive results in accelerating pre-trained language models. At the heart of DFKD is to reconstruct a synthetic dataset by inverting the parameters of the uncompressed model. Prior DFKD approaches, however, have largely relied on hand-crafted priors of the target data distribution for the reconstruction, which can be inevitably biased and often incompetent to capture the intrinsic distributions. To address this problem, we propose a prompt-based method, termed as PromptDFD, that allows us to take advantage of learned language priors, which effectively harmonizes the synthetic sentences to be semantically and grammatically correct. Specifically, PromptDFD leverages a pre-trained generative model to provide language priors and introduces a reinforced topic prompter to control data synthesis, making the generated samples thematically relevant and semantically plausible, and thus friendly to downstream tasks. As shown in our experiments, the proposed method substantially improves the synthesis quality and achieves considerable improvements on distillation performance. In some cases, PromptDFD even gives rise to results on par with those from the data-driven knowledge distillation with access to the original training data.

show abstract

Simulated annealing for optimization of graphs and sequences

Cited by 20 publications

References 55 publications

Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization

Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization

Neural Network Structure Optimization by Simulated Annealing

Abstract Rule Learning for Paraphrase Generation

Contact Info

Product

Resources

About