Colloidal particles may be considered as building blocks for materials, just like atoms are the bricks of molecules, macromolecules, and crystals. Periodic arrays of colloids (colloidal crystals) have attracted much interest over the last two decades, largely because of their unique photonic properties. The archetype opal structures are based on close-packed arrays of spheres of submicrometer diameter. Interest in structuring materials at this length scale, but with more complex features and ideally by self-assembly processes, has led to much progress in controlling features of both building blocks and assemblies. The necessary ingredients include colloids, colloidal clusters, and colloidal "molecules" which have special shapes and the ability to bind directionally, the control over short-range and long-range interactions, and the capability to place and orientate these bricks. This Review highlights recent experimental and theoretical progress in the assembly of colloids larger than 50 nm.