Sentiment Analysis is becoming an essential task for academics, as well as for commercial companies. However, most current approaches only identify the overall polarity of a sentence, instead of the polarity of each aspect mentioned in the sentence. Aspect-Based Sentiment Analysis (ABSA) identifies the aspects within the given sentence, and the sentiment that was expressed for each aspect. Recently, the use of pre-trained models such as BERT has achieved state-of-the-art results in the field of natural language processing. In this paper, we propose two ensemble models based on multilingual-BERT, namely, mBERT-E-MV and mBERT-E-AS. Using different methods, we construct an auxiliary sentence from this aspect and convert the ABSA problem to a sentence-pair classification task. We then fine-tune different pre-trained BERT models and ensemble them for a final prediction based on the proposed model; we achieve new, state-of-the-art results for datasets belonging to different domains in the Hindi language.