Plant fine root turnover is a continuous process both spatially and temporally, and fine root decomposition is affected by many biotic and abiotic factors. However, the effect of the living roots and the associated mycorrhizal fungal mycelia on fine root decomposition remains unclear. The objective of this study is to explore the influence of these biotic factors on fine root decomposition in a semi-arid ecosystem. In this study, we investigated the effect of fine roots and mycelia on fine root decomposition of a pioneer shrub (Artemisia halodendron) in Horqin sandy land, northeast China, by the ingrowth core method combined with the litterbag method. Litterbags were installed in cores. Results showed that core a allowed the growth of both fine roots and mycelia (treatment R + M), core b only allowed the growth of mycelia (treatment M), and in core c the fine root and mycelia growth were restricted and only bulk soil was present (treatment S). These findings suggest that the process of root decomposition was significantly affected by the living roots and mycelia, and carbon (C) and nitrogen (N) concentration dynamics during root decomposition differed among treatments. Mycelia significantly stimulated the mass loss and C and N release during root decomposition. Treatment R + M significantly stimulated the accumulation of soil total C, total N, and organic N under litterbags. The mycelia significantly stimulated the accumulation of the inorganic N (ammonium-N and nitrate-N) but the presence of fine roots weakened nitrate-N accumulation. The presence of living roots and associated mycelia strongly affected the process of root decomposition and matter release in the litter-soil system. The results of this study should strengthen the understanding of root-soil interactions.