Face authentication is a widely used technique for verifying identity, but current approaches encounter limitations due to their reliance on extensive computing resources, large datasets, and well-lit environments. Additionally, these approaches often lack adaptability to accommodate new individuals and continuously improve performance. These constraints make them impractical for various edge applications such as smart home security, bio-metric, surveillance system, etc. To address these challenges, this paper introduces a novel technique called FewFaceNet, which leverages a very lightweight few-shot learning-based incremental face authentication. Unlike existing methods, FewFaceNet employs a shallow lightweight backbone model that can start work with just one face image and also can handle infrared images in dark environments. These features make it highly suitable for deployment on small-edge cameras like door security cameras. We curated a diverse dataset from various reliable sources, including our own infrared camera to train and evaluate the model. Through extensive experimentation, we assessed the performance of FewFaceNet with different backbone ablation studies across one-shot to five-shot scenarios. The experimental results convincingly demonstrate the effectiveness of FewFaceNet in overcoming the limitations of existing approaches. The code and data available at: https://github.com/Sufianlab/FewFaceNet.