Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models

Na, Clara; Mehta, Sanket Vaibhav; Strubell, Emma

doi:10.48550/arxiv.2205.12694

Cited by 2 publications

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also note that while some recent works (Foret et al, 2020;Na et al, 2022;Rangwani et al, 2022) propose methods that may be utilized to better train quantized models on LT data, actual training of quantized models on LT data is seldom explored.…”

Section: Related Workmentioning

confidence: 99%

North Korean Defectors and Human Resource Development in South Korea

Kim

You

Choi

et al. 2020

Human Resource Development in South Korea

View full text Add to dashboard Cite

This report introduces EEVE-Korean-v1.0, a Korean adaptation of large language models that exhibit remarkable capabilities across English and Korean text understanding. Building on recent highly capable but Englishcentric LLMs, such as SOLAR-10.7B and Phi-2, where non-English texts are inefficiently processed with English-centric tokenizers, we present an efficient and effective vocabulary expansion (EEVE) method, which encompasses parameter freezing and subword initialization. In contrast to previous efforts that believe new embeddings require trillions of training tokens, we show that our method can significantly boost non-English proficiency within just 2 billion tokens. Surpassing most instruction-tuned LLMs on the Open Ko-LLM Leaderboard, as of January 2024, our model EEVE-Korean-10.8B-v1.0 ranks as the leading Korean pre-trained model in the open-source community, according to Hugging Face's leaderboard. We open-source our models on Huggingface to empower the open research community in various languages.

show abstract

Section: Related Workmentioning

confidence: 99%