For better understanding compositions and evolutions of residual oil of shale and coal, a grain-based microscale sealed vessel (MSSV) pyrolysis method to whole rock was used to investigate the residual oil contents and its fractional compositions from shale, coal, and coaly shale samples, as well as their variations at different temperatures and maturities. Quantities of extracted oil from residuals and yields of C1 to C5 gases were used to define oil, wet gas, and dry gas windows using equivalent Ro (%) calculated through EasyRo (%) method. Oil windows are defined as 0.6−1.3%Ro for shale and 0.5−1.2%Ro for coal; wet gas windows are 0.9−3.0%Ro for shale and 0.8−2.7%Ro for coal, and dry gas window are 1.3−4.0%Ro for shale and 1.2−4.0%Ro for coal, respectively. Coal shows relatively wider oil window than shale but similar gas window to shale. The maximum residual oil can reach 133.44 mg/g TOC, 69.84 mg/g TOC for marine and lacustrine shale, 10.03 mg/g TOC for coal, and 83.79 mg/g TOC for coaly shale, respectively. Comparing with natural residual oil, the laboratory residual oil of shale is much higher, while the residual oil in coal is mainly retained due to its unique structures. The results show that, in oil window, marine and lacustrine shale residual oil show mainly saturates, aromatics, resins but less asphaltenes, while coal residual oil are mainly asphaltenes, aromatics, resins but less saturates. In the wet and dry gas window, marine and lacustrine shale residual oil is mainly made up of saturates, aromatics, and resins, while coal residual oil is mainly made up of asphaltenes and resins. These results suggest that residual oil contents of marine shale, lacustrine shale, and coaly shale are higher with high proportions of saturated and aromatic hydrocarbons in low maturities which show high shale oil prospective than coal, while in high maturities the residual oil contents decrease quickly but still have higher potential for cracking gases which might become the source of shale gas. The residual oil in coal is low mainly in forms of aromatics, resins, and asphaltenes, which can only be the source of coal-bed methane as maturity increases.