Cancer is a widespread class of diseases that each year affects millions of people. It is mostly treated with chemotherapy, surgery, radiation therapy, or combinations thereof. High doserate (HDR) brachytherapy (BT) is one modality of radiation therapy, which is used to treat for example prostate cancer and gynecologic cancer. In BT, catheters (i.e., hollow needles) or applicators are used to place a single, small, but highly radioactive source of ionizing radiation close to or within a tumour, at dwell positions. An emerging technique for HDR BT treatment is intensity modulated brachytherapy (IMBT), in which static or dynamic shields are used to further shape the dose distribution, by hindering the radiation in certain directions.The topic of this thesis is the application of mathematical optimization to model and solve the treatment planning problem. The treatment planning includes decisions on catheter placement, that is, how many catheters to use and where to place them, as well as decisions for dwell times. Our focus is on the latter decisions. The primary treatment goals are to give the tumour a sufficiently high radiation dose while limiting the dose to the surrounding healthy organs, to avoid severe side effects. Because these aims are typically in conflict, optimization models of the treatment planning problem are inherently multiobjective. Compared to manual treatment planning, there are several advantages of using mathematical optimization for treatment planning. First, the optimization of treatment plans requires less time, compared to the time-consuming manual planning. Secondly, treatment plan quality can be improved by using optimization models and algorithms. Finally, with the use of sophisticated optimization models and algorithms the requirements of experience and skill level for the planners are lower. The use of optimization for treatment planning of IMBT is especially important because the degrees of freedom are too many for manual planning.The contributions of this thesis include the study of properties of treatment planning models, suggestions for extensions and improvements of proposed models, and the development of new optimization models that take clinically relevant, but uncustomary aspects, into account in the treatment planning. A common theme is the modelling of constraints on dosimetric indices, each of which is a restriction on the portion of a volume that receives at least a specified dose, or on the lowest dose that is received by a portion of a volume. Modelling dosimetric indices explicitly yields mixed-integer programs which are computationally demanding to solve. We have therefore investigated approximations of dosimetric indices, for example using smooth non-linear functions or convex functions. Contributions of this thesis are also a literature review of proposed treatment planning models for HDR BT, including mathematical analyses and comparisons of models, and a study of treatment planning for IMBT, which shows how robust optimization can be used to mitigate the risks fro...