Task-oriented social systems involve workers and tasks, which can typically be modeled by bipartite networks. The absence of workers may cause some tasks cannot be finished in time, thus hampering the smoothing operation of the whole system. In this study, we focus on the robustness analysis of such bipartite task-oriented social networks under the absence of some workers. In particular, taking hospital logistics systems as an example, we propose four attack strategies: efficiency-based attack, centrality-based attack, diversity-based attack, and influence-based attack. The experiments on nine real hospital logistics systems show that these systems are especially vulnerable to the diversity-based attack. Different hospitals may present different patterns: some hospital logistics systems are relatively robust against these attacks while others are more vulnerable. The former hospitals are relatively small in size and adopt a global task assignment mode in which the workers can be easily replaced by others. On the contrary, the workers in the latter ones are highly professional and irreplaceable due to the large scale of the hospitals and the adoption of a regional task assignment mode. We also design two task reassignment mechanisms to model the cascading failure of such systems. In this case, the working load plays an important role in the robustness of these systems. Moreover, we find that the robustness and efficiency seem to be negatively correlated with each other, and how to balance the two in real systems still remains an open problem.