Locker: Locally Constrained Self-Attentive Sequential Recommendation

He, Zhankui; Zhao, Handong; Lin, Zhe; Wang, Zhaowen; Kale, Ajinkya; McAuley, Julian

doi:10.1145/3459637.3482136

Cited by 46 publications

(25 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, we examine the inference efficiency for varying input lengths, see Figure 4. We observe that when sequences are shorter than 2 12 , ELTransformer performs on average similarly to CNN models (O(l) computational complexity) and outperforms BERT4NILM. For longer sequences, the inference time of BERT4NILM starts growing quadratically, while ELTransformer remains efficient, suggesting that ELTransformer has similar efficiency to CNN while being much more scalable with only 1.91M parameters.…”

Section: Implementation 1) Preprocessingmentioning

confidence: 85%

“…For the seq2point setting in this paper, another drawback of the transformer model is the lack of localness modeling. Although self-attention is designed to capture longterm semantics from input sequences, they often fail to capture short-term and local signal patterns [12]. In energy disaggregation, lacking local dependency can lead to mismatches and performance drops for multi-status appliances [11].…”

Section: Transformer Models In Nilmmentioning

confidence: 99%

“…For instance, NILM methods often fail to detect status changes on multi-status appliances (e.g., standby/on in washing machine). This can be traced back to the lack of inductive bias in local context, as self-attention is designed to be a global operator [12]. To alleviate this issue, local attention heads are introduced.…”

Section: Localness Modelingmentioning

confidence: 99%

“…Additionally, transformer models are first designed to capture long-term global context from long sequences. In this sense, they often fail to fully exploit shortterm and local signal patterns, which can result in performance deterioration on multi-status appliances [11], [12]. As a result, current transformer models for NILM only achieve marginal performance gains compared to previous methods [9], [10].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Efficient Localness Transformer for Smart Sensor-Based Energy Disaggregation

Yue¹,

Zeng²,

Kou³

et al. 2022

Preprint

View full text Add to dashboard Cite

Modern smart sensor-based energy management systems leverage non-intrusive load monitoring (NILM) to predict and optimize appliance load distribution in real-time. NILM, or energy disaggregation, refers to the decomposition of electricity usage conditioned on the aggregated power signals (i.e., smart sensor on the main channel). Based on real-time appliance power prediction using sensory technology, energy disaggregation has great potential to increase electricity efficiency and reduce energy expenditure. With the introduction of transformer models, NILM has achieved significant improvements in predicting device power readings. Nevertheless, transformers are less efficient due to O(l 2 ) complexity w.r.t. sequence length l. Moreover, transformers can fail to capture local signal patterns in sequenceto-point settings due to the lack of inductive bias in local context. In this work, we propose an efficient localness transformer for non-intrusive load monitoring (ELTransformer). Specifically, we leverage normalization functions and switch the order of matrix multiplication to approximate self-attention and reduce computational complexity. Additionally, we introduce localness modeling with sparse local attention heads and relative position encodings to enhance the model capacity in extracting short-term local patterns. To the best of our knowledge, ELTransformer is the first NILM model that addresses computational complexity and localness modeling in NILM. With extensive experiments and quantitative analyses, we demonstrate the efficiency and effectiveness of the the proposed ELTransformer with considerable improvements compared to state-of-the-art baselines.

show abstract

Section: Implementation 1) Preprocessingmentioning

confidence: 85%

Section: Transformer Models In Nilmmentioning

confidence: 99%

Section: Localness Modelingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Efficient Localness Transformer for Smart Sensor-Based Energy Disaggregation

Yue¹,

Zeng²,

Kou³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…SSE-PT [22], extends SASRec by introducing explicit user representations. LOCKER [8], enhances short-term user dynamics via local self-attention. Intent-aware Methods: NOVA [14], uses non-invasive self-attention to leverage side information.…”

Section: Experiments 41 Experimental Settingmentioning

confidence: 99%

Coarse-to-Fine Sparse Sequential Recommendation

Li¹,

Zhao²,

Li³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Sequential recommendation aims to model dynamic user behavior from historical interactions. Self-attentive methods have proven effective at capturing short-term dynamics and long-term preferences. Despite their success, these approaches still struggle to model sparse data, on which they struggle to learn high-quality item representations. We propose to model user dynamics from shopping intents and interacted items simultaneously. The learned intents are coarse-grained and work as prior knowledge for item recommendation. To this end, we present a coarse-to-fine self-attention framework, namely CaFe, which explicitly learns coarse-grained and fine-grained sequential dynamics. Specifically, CaFe first learns intents from coarse-grained sequences which are dense and hence provide high-quality user intent representations. Then, CaFe fuses intent representations into item encoder outputs to obtain improved item representations. Finally, we infer recommended items based on representations of items and corresponding intents. Experiments on sparse datasets show that CaFe outperforms state-of-the-art self-attentive recommenders by 44.03% NDCG@5 on average.

show abstract

Basket Booster for Prototype-based Contrastive Learning in Next Basket Recommendation

Chen

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Locker: Locally Constrained Self-Attentive Sequential Recommendation

Cited by 46 publications

References 12 publications

Efficient Localness Transformer for Smart Sensor-Based Energy Disaggregation

Efficient Localness Transformer for Smart Sensor-Based Energy Disaggregation

Coarse-to-Fine Sparse Sequential Recommendation

Basket Booster for Prototype-based Contrastive Learning in Next Basket Recommendation

Contact Info

Product

Resources

About