Spatial patterns in microbial colonies are the consequence of cell-division dynamics coupled with cell-cell interactions on a physical media. Agent-based models (ABMs) are a powerful tool for understanding the emergence of large scale structure from these individual cell processes. However, most ABMs have focused on fission, a process by which cells split symmetrically into two daughters. The yeast, Saccharomyces cerevisiae, is a model eukaryote which commonly undergoes an asymmetric division process called budding. The resulting mother and daughter cells have unequal sizes and the daughter cell does not inherit the replicative age of the mother. In this work, we develop and analyze an ABM to study the impact of budding cell division and nutrient limitation on yeast colony structure. We find that while budding division does not impact large-scale properties of the colony (such as shape and size), local spatial organization of cells with respect to spatial layout of mother-daughter cell pairs and connectivity of subcolonies is greatly impacted. In addition, we find that nutrient limitation further promotes local spatial organization of cells and changes global colony organization by driving variation in subcolony sizes. Moreover, resulting differences in spatial organization, coupled with differential growth rates from nutrient limitation, create distinct sectoring patterns within growing yeast colonies. Our findings offer novel insights into mechanisms driving experimentally observed sectored yeast colony phenotypes. Furthermore, our work illustrates the need to include relevant biophysical mechanisms when using ABMs to compare to experimental studies.