Coordination in cooperative multi-agent systems is one of the important issues in multi-agent learning and has been broadly studies in the literature. Much work in this field has been carried out yet. However, there are still some coordination issues that are required to be improved. In this work, we look over the multi-agent coordination problems in cooperative environments under the networked multi-agent learning framework using some social network structures and will try to improve coordination efficiency. In our framework, we talk about two types of learners, i.e. individual action learner and joint action learner. We are considering hierarchical multi-agent learning framework to accelerate the coordination efficiency. Our research direction is to consider utilizing the characteristics of different neighboring agents (e.g., the past interaction histories of different nodes, the relative degree of different nodes in the neighborhood) while performing the multi-agent learning to improve coordination performance.