Fashion e-commerce has enjoyed an exponential growth in the last few years. A key challenge of the market players is to offer customers a personalized experience and to suggest relevant articles. In that respect, although product recommendation is a well-studied field, size and fit recommendation is still in its infancy. The size and fit topic is a very challenging problem as data is extremely sparse and noisy. Most approaches so far have exploited traditional machine learning techniques. In this work, we bring forward a meta-learning approach using an underlying deep neural network. The advantage of such an approach lies in its ability to exploit large scale data, learn across fashion categories, and absorb new data efficiently without retraining. We benchmark our method against 3 recent methods proven successful in the domain, and demonstrate various strengths of the proposed approach. To that end, we use a large-scale anonymized dataset of about 9.4 million customer-size interactions, collected over 5 years from around 384k customers.