Background: Fall-related psychological concerns are common among older adults, potentially contributing to functional decline as well as to restriction of activities and social participation. To effectively prevent such negative consequences, it is important to understand how even very low concern about falling could affect physical activity behavior in everyday life. We hypothesized that concern about falling is associated with a reduction in diversity, dynamics, and performance of daily activities, and that these features can be comprehensively quantified in terms of complexity of physical activity patterns. Methods: A sample of 40 community-dwelling older adults were assessed for concern about falling using the Falls Efficacy Scale-International (FES-I). Free-living physical activity was assessed using a set of metrics derived from data recorded with a chest-worn tri-axial accelerometer. The devised metrics characterized physical activity behavior in terms of endurance (total locomotion time, longest locomotion period, usual walking cadence), performance (cadence of longest locomotion period, locomotion periods with at least 30 steps and 100 steps/min), and complexity of physical activity patterns. Complexity was quantified according to variations in type, intensity, and duration of activities, and was considered as an adaptive response to environmental exigencies over the course of the day. Results: Based on FES-I score, participants were classified into two groups: not concerned at all/fully confident (n = 25) and concerned/less confident (n = 15). Demographic and health-related variables did not differ significantly between groups. Comparison of physical activity behavior indicated no significant differences for endurance-related metrics. In contrast, performance and complexity metrics were significantly lower in the less confident group compared to the fully confident group. Among all metrics, complexity of physical activity patterns appeared as the most discriminative feature between fully confident and less confident participants (p = 0.001, non-parametric Cliff’s delta effect size = 0.63). Conclusions: These results extend our understanding of the interplay between low concern about falling and physical activity behavior of community-dwelling older persons in their everyday life context. This information could serve to better design and evaluate personalized intervention programs in future prospective studies.