The rapid expansion of palm oil cultivation has remarkably influenced the social structure in rural areas, particularly the socioeconomic and ecological aspects. Social interaction in the palm oil area has certain characteristics and patterns. The unresourceful pattern within the community around valuable natural resources often creates social conflict among stakeholders. This study describes the relationship pattern between the key stakeholders in Keban Agung, South Sumatera Province, Indonesia. It also explores how social structure and people within this area influence each other. We use a qualitative approach to collect a broad perspective, including in-depth interviews and focus group discussions. Hence, this study is beneficial to investigate a suitable approach that can mitigate diverse interests, especially in economic, social, and environmental aspects. Our objective is to study the structuration and agency process throughout palm oil expansion. In addition, we examine the current collective process in mitigating social-environmental conflicts. We found that the social structure in our research site is strong, remarkably influencing the overall circumstances. However, it poses a challenge given the inflexibility of the current agency process, especially for locals, who seek to modify key aspects within the structure.