Incremental Multilayer Broad Learning System With Stochastic Configuration Algorithm for Regression

Ding, Shifei; Zhang, Chenglong; Zhang, Jian; Guo, Lili; Ding, Ling

doi:10.1109/tcds.2022.3192536

Cited by 5 publications

(2 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this research, we restrict the architecture of each expert to a fully connected layer with a maximum of three hidden layers (corresponding to the number of experts). While recognizing the potential for enhanced diversity in submodels 80 , exploring a wider array of model architectures encompassing various versions of compressed GPTs with diverse attention layers might be advantageous 81 . Nonetheless, it is crucial to note that such exploration may involve trade-offs, particularly in computational time.…”

Section: Experimental Evaluationmentioning

confidence: 99%

Defense against adversarial attacks: robust and efficient compressed optimized neural networks

Kraidia,

Ghenai,

Belhaouari

2024

Sci Rep

View full text Add to dashboard Cite

In the ongoing battle against adversarial attacks, adopting a suitable strategy to enhance model efficiency, bolster resistance to adversarial threats, and ensure practical deployment is crucial. To achieve this goal, a novel four-component methodology is introduced. First, introducing a pioneering batch-cumulative approach, the exponential particle swarm optimization (ExPSO) algorithm was developed for meticulous parameter fine-tuning within each batch. A cumulative updating loss function was employed for overall optimization, demonstrating remarkable superiority over traditional optimization techniques. Second, weight compression is applied to streamline the deep neural network (DNN) parameters, boosting the storage efficiency and accelerating inference. It also introduces complexity to deter potential attackers, enhancing model accuracy in adversarial settings. This study compresses the generative pre-trained transformer (GPT) by 65%, saving time and memory without causing performance loss. Compared to state-of-the-art methods, the proposed method achieves the lowest perplexity (14.28), the highest accuracy (93.72%), and an 8 × speedup in the central processing unit. The integration of the preceding two components involves the simultaneous training of multiple versions of the compressed GPT. This training occurs across various compression rates and different segments of a dataset and is ultimately associated with a novel multi-expert architecture. This enhancement significantly fortifies the model's resistance to adversarial attacks by introducing complexity into attackers' attempts to anticipate the model's prediction integration process. Consequently, this leads to a remarkable average performance improvement of 25% across 14 different attack scenarios and various datasets, surpassing the capabilities of current state-of-the-art methods.

show abstract

Section: Experimental Evaluationmentioning

confidence: 99%

Defense against adversarial attacks: robust and efficient compressed optimized neural networks

Kraidia,

Ghenai,

Belhaouari

2024

Sci Rep

View full text Add to dashboard Cite

show abstract

“…Due to its strong generalization capability and superior learning efficiency, BLS has attracted extensive attentions from both academia and industry, and various improvements and expansions have been made to it [4]. As a novel neural network model, BLS has much room for network structure expansion, and many BLS-based structural variants have been constantly proposed, such as cascade BLS [5], deep cascade BLS [6], intergroup cascade BLS [7], stacked BLS [8], recurrent BLS and gated BLS [9,10], cascade BLS with dropout or dense connection [11], incremental multilayer BLS [12], etc. By introducing Takagi-Sugeno fuzzy systems into BLS, a new fuzzy BLS was established [13] and studied deeply [14][15][16][17][18].…”

Section: Related Workmentioning

confidence: 99%

RCBLS: An Outlier-Robust Broad Learning Framework with Compact Structure

Wei

Zhou

et al. 2023

Electronics

View full text Add to dashboard Cite

Recently, the broad learning system (BLS) has been widely developed in virtue of its excellent performance and high-computational efficiency. However, two deficiencies still exist in BLS and preclude its deployment in real applications. First, the standard BLS performs poorly in outlier environments because the least squares loss function it uses to train the network is sensitive to outliers. Second, the model structure of BLS is likely to be redundant since the hidden nodes in it are randomly generated. To address the above two issues, a new robust and compact BLS (RCBLS), based on M-estimator and sparsity regularization, is proposed in this paper. The RCBLS develops from the BLS model and maintains its excellent characteristics, but replaces the conventional least squares learning criterion with an M-estimator-based loss function that is less sensitive to outliers, in order to suppress the incorrect feedback of the model to outlier samples, and hence enhance its robustness in the presence of outliers. Meanwhile, the RCBLS imposes the sparsity-promoting l2,1 -norm regularization instead of the common l2-norm regularization for model reduction. With the help of the row sparsity of l2,1-norm regularization, the unnecessary hidden nodes in RCBLS can be effectively picked out and removed from the network, thereby resulting in a more compact network. The theoretical analyses on outlier robustness, structural compactness and computational complexity of the proposed RCBLS model are provided. Finally, the validity of the RCBLS is verified by regression, time series prediction and image classification tasks. The experimental results demonstrate that the proposed RCBLS has stronger anti-outlier ability and more compact network structure than BLS and other representative algorithms.

show abstract