Agricultural by-products have several disadvantages as fuel, such as low calorific values and high ash contents. To address these disadvantages, this study examined the mixing of agricultural by-products and spent coffee grounds, for use as a solid fuel, and the improvement of fuel characteristics through torrefaction. Pepper stems and spent coffee grounds were first dried to moisture contents of <15% and then combined, with mixing ratios varying from 9:1 to 6:4. Fuel pellets were produced from these mixtures using a commercial pelletiser, evaluated against various standards, and classified as grade A, B, or Bio-SRF. The optimal ratio of pepper stems to spent coffee grounds was determined to be 8:2. The pellets were torrefied to improve their fuel characteristics. Different torrefaction temperatures improved the mass yields of the pellets to between 50.87% and 88.27%. The calorific value increased from 19.9% to 26.8% at 290 °C. The optimal torrefaction temperature for coffee ground pellets was 230 °C, while for other pellets, it was 250 °C. This study provides basic information on the potential enhancement of agricultural by-products for fuel applications.