Amblyopia is a neurodevelopmental visual disorder that affects approximately 3-5% of children globally and it can lead to monocular vision loss if it is not diagnosed and treated early. Traditional diagnostic methods, which rely on subjective assessments and expert interpretation of eye movement recordings presents challenges in resource-limited eye care clinics. This study introduces a new approach that integrates the Gemini large language model (LLM) with eye-tracking data to develop a classification tool for diagnosis of patients with amblyopia. The study demonstrates that, (i) LLMs can be used to analyze fixation eye movement data to diagnose patients with amblyopia; and (ii) integration of medical subject matter expertise improves the performance of LLMs in medical applications. Our LLM-based classification tool achieves an accuracy of 80% in diagnosing patients with amblyopia using a combination of few shot learning and multiview prompting with expert input from pediatric ophthalmologist. The model classifies amblyopic patients with moderate or severe amblyopia from control subjects with an accuracy of 83% and mild or treated amblyopic patients from control subjects with an accuracy of 81%. Finally, the model achieves an accuracy of 85% for classifying amblyopic patients with nys-tagmus from control subjects. The results of this study demonstrate the feasibility of using LLMs in ophthalmology application and highlights the essential role of medical expert input in LLM-based medical applications.