Speaker-Adaptive Lip Reading with User-Dependent Padding

Kim, Minsu; Kim, Hyunjun; Ro, Yong Man

doi:10.1007/978-3-031-20059-5_33

Cited by 15 publications

(4 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The key idea of prompting is to modify the inputs rather than network parameters (Li and Liang 2021). Following (Jia et al 2022;Kim, Kim, and Ro 2022), we propose to transform the padding, an input to convolutional layers, to be our prompt. In convolutional layers, padding is usually employed to maintain or control the size of the output feature maps, such as zero padding and reflect padding.…”

Section: Prompting For Test-time Personalizationmentioning

confidence: 99%

Test-Time Personalization with Meta Prompt for Gaze Estimation

Liu,

Qi,

et al. 2024

AAAI

View full text Add to dashboard Cite

Despite the recent remarkable achievement in gaze estimation, efficient and accurate personalization of gaze estimation without labels is a practical problem but rarely touched on in the literature. To achieve efficient personalization, we take inspiration from the recent advances in Natural Language Processing (NLP) by updating a negligible number of parameters, "prompts", at the test time. Specifically, the prompt is additionally attached without perturbing original network and can contain less than 1% of a ResNet-18's parameters. Our experiments show high efficiency of the prompt tuning approach. The proposed one can be 10 times faster in terms of adaptation speed than the methods compared. However, it is non-trivial to update the prompt for personalized gaze estimation without labels. At the test time, it is essential to ensure that the minimizing of particular unsupervised loss leads to the goals of minimizing gaze estimation error. To address this difficulty, we propose to meta-learn the prompt to ensure that its updates align with the goal. Our experiments show that the meta-learned prompt can be effectively adapted even with a simple symmetry loss. In addition, we experiment on four cross-dataset validations to show the remarkable advantages of the proposed method.

show abstract

Section: Prompting For Test-time Personalizationmentioning

confidence: 99%

Test-Time Personalization with Meta Prompt for Gaze Estimation

Liu,

Qi,

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Model Reprogramming, also known as Adversarial Reprogramming, is a form of adversarial attack on ML models that transforms the inputs for model to change the model's behavior to a desired adversarial behavior [Elsayed et al, 2018]. Transforming the inputs for images, also known as visual prompting, is usually achieved by adding values to whole or parts of the image or padding the image with custom values [Elsayed et al, 2018;Zheng et al, 2021;Jia et al, 2022;Kim et al, 2022;Bahng et al, 2022;Zhang et al, 2022].…”

Section: Model Reprogrammingmentioning

confidence: 99%

Attribute inference privacy protection for pre-trained models

Abedi Khorasgani,

Mohammed,

Wang

2024

Int. J. Inf. Secur.

View full text Add to dashboard Cite

With the increasing popularity of machine learning in image processing, privacy concerns have emerged as a significant issue in deploying and using ML services. However, current privacy protection approaches often require computationally expensive training from scratch or extensive fine-tuning of models, posing significant barriers to the development of privacy-conscious models, particularly for smaller organizations seeking to comply with data privacy laws. In this thesis, we address the privacy challenges in computer vision by investigating the effectiveness of two recent fine-tuning methods, Model Reprogramming and Low-Rank Adaptation. We adapt these techniques to provide attribute privacy protection for pre-trained models, minimizing computational overhead and training time. We integrate these methods into an adversarial min-max framework, allowing us to conceal sensitive information from feature outputs without extensive modifications to the pre-trained model, but rather focusing on a small set of new parameters. We demonstrate the effectiveness of our methods by conducting experiments on the CelebA dataset, achieving state-of-the-art performance while significantly reducing computational complexity and cost. Our research provides a valuable contribution to the field of computer vision and privacy, offerii

show abstract

“…Prompt-based learning, widely recognized in natural language processing (NLP), employs specific text inputs to amplify the proficiency of NLP models (Lester, Al-Rfou, and Constant 2021), (Li and Liang 2021). This concept has been extended to computer vision as visual prompts, showcasing promising applications (Kim, Kim, and Ro 2022), (Lin et al 2023), (Chen et al 2023). While methods like fine-tuning pre-trained vision models with visual prompts have demonstrated competitive results across various tasks with minimal resources (Jia et al 2022).…”

Section: Prompt Learningmentioning

confidence: 99%

Improving Open Set Recognition via Visual Prompts Distilled from Common-Sense Knowledge

Kim,

2024

AAAI

View full text Add to dashboard Cite

Open Set Recognition (OSR) poses significant challenges in distinguishing known from unknown classes. In OSR, the overconfidence problem has become a persistent obstacle, where visual recognition models often misclassify unknown objects as known objects with high confidence. This issue stems from the fact that visual recognition models often lack the integration of common-sense knowledge, a feature that is naturally present in language-based models but lacking in visual recognition systems. In this paper, we propose a novel approach to enhance OSR performance by distilling common-sense knowledge into visual prompts. Utilizing text prompts that embody common-sense knowledge about known classes, the proposed visual prompt is learned by extracting semantic common-sense features and aligning them with image features from visual recognition models. The unique aspect of this work is the training of individual visual prompts for each class to encapsulate this common-sense knowledge. Our methodology is model-agnostic, capable of enhancing OSR across various visual recognition models, and computationally light as it focuses solely on training the visual prompts. This research introduces a method for addressing OSR, aiming at a more systematic integration of visual recognition systems with common-sense knowledge. The obtained results indicate an enhancement in recognition accuracy, suggesting the applicability of this approach in practical settings.

show abstract

Speaker-Adaptive Lip Reading with User-Dependent Padding

Cited by 15 publications

References 54 publications

Test-Time Personalization with Meta Prompt for Gaze Estimation

Test-Time Personalization with Meta Prompt for Gaze Estimation

Attribute inference privacy protection for pre-trained models

Improving Open Set Recognition via Visual Prompts Distilled from Common-Sense Knowledge

Contact Info

Product

Resources

About