Predicting agricultural yields is imperative for effective planning to sustain the growing global population. Traditionally, regression-based, simulation-based, and hybrid methods were employed for yield prediction. In recent times, there has been a notable shift towards the adoption of Machine Learning (ML) methods, with Deep Learning (DL), particularly Convolutional Neural Networks (CNNs) and Long-Short Term Memory (LSTM) networks, emerging as popular choices for their enhanced predictive accuracy. This research introduces a cost-effective DL architecture tailored for corn yield prediction, considering computational efficiency in processing time, data size, and NN architecture complexity. The proposed architecture, named SEDLA (Simple and Efficient Deep Learning Architecture), leverages the spatial and temporal learning capabilities of CNNs and LSTMs, respectively, with a unique emphasis on exploring the impact of kernel size in CNNs. Simultaneously, the study aims to exclusively employ satellite and yield data, strategically minimizing input variables to enhance the model's simplicity and efficiency. Notably, the study demonstrates that employing larger kernel sizes in CNNs, especially when processing histogram-based Surface Reflectance (SR) and Land Surface Temperature (LST) data from Moderate Resolution Imaging Spectroradiometer (MODIS), allows for a reduction in the number of hidden layers. The efficacy of the architecture was evaluated through extensive testing on corn yield prediction across 13 states in the United States (U.S.) Corn Belt at county-level. The experimental results showcase the superiority of the proposed architecture, achieving a Mean Absolute Percentage Error (MAPE) of 6.71 and Root Mean Square Error (RMSE) of 14.34, utilizing a single-layer CNN with a 15x15 kernel in conjunction with LSTM. These outcomes surpass existing benchmarks in the literature, affirming the efficacy and potential of the suggested DL framework for accurate and efficient crop yield predictions.