Unlike most other rocks, coal is a sedimentary rock composed primarily of organic matter derived from plant debris that accumulated in peat mires during different geological periods. Coal is also an essential economic resource in many countries, having been the main driving force behind the industrial revolution. Coal is still widely used industrially for many different purposes: carbonization and coke production, iron/steel making, thermal coal to generate electricity, liquefaction, and gasification. The utility of the coal is dictated by its properties which are commonly referred to as its rank, type, and grade. Coal composition, in terms of its macerals, and its rank determination are determined manually by a petrographer due to its complex nature. This study aimed to develop an automatic method based on machine learning capable of maceral segmentation at group level followed by a module for rank reflectance determination on petrographic images of coal that can improve the efficiency of this process and decrease operator subjectivity. Firstly, a Mask R-CNN-based architecture deep learning approach was developed to identify and segment the vitrinite maceral group, which is fundamental for rank analysis, as rank is determined by collotelinite reflectance (one of its individual macerals). Secondly, an image processing method was developed to analyze the vitrinite segmented images and determine coal rank by associating the grey values with the reflectance. For the maceral (group) segmentation, five samples were used to train the network, 174 images were used for training, and 86 were used for testing, with the best results obtained for the vitrinite, inertinite, liptinite, and collotelinite models (89.23%, 68.81%, 37.00% and 84.77% F1-score, respectively). Those samples were used alongside another eight samples to determine the rank results utilizing collotelinite reflectance. The samples ranged from 0.97% to 1.8% reflectance. This method should help save time and labor for analysis if implemented into a production model. The root mean square calculated between the proposed method and the reference reflectance values was 0.0978.