Agriculture is known as one of the main factor for a growth of a country. Paddy plantation is the most widely planted crop in Malaysia. The rice produced is the main food source to Malaysian's people and source of income to this country as well. However, a disease known as Brown Spot (BS) attacks the paddy plants and threats their quality. This disease caused by bipolar fungus, which represent by the development of an oval, dark brown to purplish-brown spot on leaf. This disease observed as among the hazardous disease that may result in degradation of paddy production. Brown Spot disease could spread through airborne spores from plant to plant on the field. In this research, a system that could help people, especially farmers, to detect the disease at early stage is developed. The real image capture at paddy field is processed in the MATLAB software with image enhancement, background removal as well as binary and RGB image masking process. To determine the Brown Spot area, pixel intensity between the infected and noninfected areas is calculated. The severity level table developed by Horsfall and Heuberger is then used as reference to classify the severity level of Brown Spot disease. A GUI is created to detect the Brown Spot disease automatically. From the study conducted, the accuracy of Brown Spot detection is approximately 89% accurate compared to manual evaluation by plant pathology.