This thesis presents a series of neural network classifiers for object detection in both optical and infrared images. The focus of this work is on efficient and accurate solutions. The thesis discusses the evolution of the highly efficient and tiny network Binary Classification Vision Transformer (BC-ViT) and how through thoughtful modifications and improvements, the BC-ViT can be utilized for tasks of increasing complexity. Chapter 2 discusses the creation of BC-ViT and its initial use case for underwater image classification of optical images. The BC-ViT is able to complete its task with an accuracy of 99.29\% while being comprised of a mere 15,981 total trainable parameters. Chapter 3, Waste Multi-Class Vision Transformer (WMC-ViT), introduces the usefulness of mindful algorithm design for the realm of multi-class classification on a mutually exclusive dataset. WMC-ViT shows that the task oriented design strategy allowed for a network to achieve an accuracy score of 94.27\% on a five class problem while still maintaining a tiny parameter count of 35,492. The final chapter demonstrates that by utilizing functional blocks of BC-ViT, a simple and effective target detection algorithm for infrared images can be created. The Edge Infrared Vision Transformer (EIR-ViT) showed admirable results with a high IoU of 79\% coupled with a massive reduction in parameters compared to current state of the art solutions at roughly 120,000. This thesis ends with summary of Chapters 2, 3, and 4 that further discusses the implications this design strategy offers.