Today, transformer language models serve as a core component for majority of natural language processing tasks. Industrial application of such models requires minimization of computation time and memory footprint. Knowledge distillation is one of approaches to address this goal. Existing methods in this field are mainly focused on reducing the number of layers or dimension of embeddings/hidden representations. Alternative option is to reduce the number of tokens in vocabulary and therefore the embeddings matrix of the student model. The main problem with vocabulary minimization is mismatch between input sequences and output class distributions of a teacher and a student models. As a result, it is impossible to directly apply KL-based knowledge distillation. We propose two simple yet effective alignment techniques to make knowledge distillation to the students with reduced vocabulary. Evaluation of distilled models on a number of common benchmarks for Russian such as Russian SuperGLUE, SberQuAD, RuSentiment, ParaPhaser, Collection-3 demonstrated that our techniques allow to achieve compression from 17× to 49×, while maintaining quality of 1.7× compressed student with the full-sized vocabulary, but reduced number of Transformer layers only. We make our code and distilled models available.

In this paper, we perform a microeconomic analysis of positive and negative imbalances in the maturity structure of Russian banks' transactions. In particular, using Heckman selection models at the cross-section of Russian banks, we test the ability of such imbalances to predict the probability of the detection of banks' negative net worth and its expected magnitude in advance (three months before negative worth detection). The estimation results show that, first, certain indicators of imbalances do offer 'value added' in predicting 'holes' in banks' capital: taking into account these imbalances in banks' short-and medium-term transactions with households and shortterm transactions with enterprises improves the quality of out-ofsample forecasts. Second, the very division into positive and negative imbalances makes sense: the effects are in many cases found to be opposite with respect to the size and likelihood of negative net worth detection at banks. Third, a separate analysis of banking transactions with households and those with businesses is also of great importance: the effect of imbalances in transactions similar in maturity structure but with different types of economic agents is in many cases opposite in sign. The results may be useful for the Bank of Russia in identifying potentially fragile banks as part of its prudential policy.

No abstract

Магистрант кафедры экологии, природопользования, землеустройства и БЖД, Астраханский государственный университет пл. Шаумяна, 1, 414000 г. Астрахань, Российская Федерация aqueous runoff low volume for the second quarter and its increase in the years with overflow high level in relation to the deep subsoil water occurrence and exudative hydrological regime predomination under flushing on the high level grounds. Besides, there is derating CL/SO 4 and soil solution toxic level reduction under aqueous runoff volume for the second quarter less than 80 cubic km in soil ground of this level. Vegetation community productivity directive increase was specified from the beginning of attendance till 2006, however in the following years there was some high soluble salts in the ground, soil solution toxic increase thus vegetation productivity reduction in relation to annual average air temperature growth, rainfall amount and aqueous runoff volume reduction.