Whether foreign language writing is rated using analytic rating scales or holistically, the organization of ideas is invariably one of the aspects assessed. However, it is unclear what raters are sensitive to when rating writing for organization. Which do they value more highly, the physical aspects of organization, such as paragraphing and the existence of organization markers, or deeper textual aspects, such as the coherent flow of ideas? This study investigates whether raters of timed essays value paragraphing, cohesive devices or coherence more when assigning a score for organization. The current study used multiple regression to ascertain what raters of the writing section of an in-house proficiency test were sensitive to when rating writing for organization using an analytic rating scale. The number of paragraphs, number of cohesive devices and coherence within 116 timed essays were evaluated and it was found that raters seem to value the physical aspects of organization more than the deeper textual aspects. Specifically, the number of paragraphs and the number of cohesive devices predicted differences in scores assigned for organization. In addition, the scores assigned for content were significantly predictive of scores for organization.