Thermally conductive adhesives were prepared by incorporating magnesium oxide (MgO) and boron nitride (BN) into fluorosilicone resins. The effects of filler type, size, and shape on thermal conductivity and adhesion properties were analyzed. Higher thermal conductivity was achieved when larger fillers were used, but smaller ones were advantageous in terms of adhesion strength. Bimodal adhesives containing spherical MgOs with an average particle size of 120 μm and 90 μm exhibited the highest conductivity value of up to 1.82 W/mK. Filler shape was also important to improve the thermal conductivity as the filler type increased. Trimodal adhesives revealed high adhesion strength compared to unimodal and bimodal adhesives, which remained high after aging at 85 °C and 85% relative humidity for 168 h. It was found that the thermal and adhesion properties of fluorosilicone composites were strongly affected by the packing efficiency and interfacial resistance of the particles.