The consequences of physical neglect on retardation in the development of adaptive behaviors and the increased risk of poor physical and mental health are well documented. As physical neglect is a phenomenon found almost exclusively among socially deprived people, it is important to distinguish the health effects caused by neglect from those caused by poverty. The objective of this study was to compare the effects of poverty and physical neglect on the development of problematic externalizing and internalizing behaviors, adaptive skills, and school problems among school children between the ages of 3 and 12. A group of 157 children were chosen from 28 Andalusian schools and classified in three homogeneous groups. Children in group 1 (n = 53) had two target conditions: living in slums (poverty) and suffering from neglect. Children in group 2 (n = 52) had one target condition: living in the same slums as the children in group 1, but not suffering from neglect. Group 3 (n = 52) consisted of children from other (non-slum) neighborhoods who did not suffer from neglect. Adaptive and maladaptive behaviors were evaluated with the Behavior Assessment System for Children (BASC). Significant differences were found between group 1 and group 2, but there were no important differences between group 2 and group 3. The conclusion was that externalizing and internalizing problems, school problems, and low adaptive skills found in neglected children were associated with neglect rather than with poverty or socially deprived environments.