Fragment-based drug design (FBDD), where fragments act as starting points for molecular generation, is an effective way to constrain chemical space and improve generation for biologically active molecules. Key challenges in this process is to navigate through the vast molecular space, and produce promising molecules. Here, we propose a controllable FBDD model, CLigOpt, which can generate molecules with desired properties from a given fragment pair. CLigOpt narrows down the size of the generated molecule set by restricting sampling from the latent space, and only allowing seeds with desired predicted properties to continue the generation procedure. Results show that CLigOpt achieves consistently strong performance in generating structurally and chemically valid molecules, as evaluated across six metrics. Applicability of the method is illustrated through ligand candidates for hDHFR and it is shown that the proportion of feasible active molecules from the generated set is increased by 10%. Molecular docking and synthesisability prediction tasks are conducted to prioritise generated molecules, with low docking energy and ease of synthesis indicating potential lead compounds.