We investigate the effects of chiral three-nucleon force (3NF) on proton scattering at 65 MeV and 4 He scattering at 72 MeV/nucleon from heavier targets, using the standard microscopic framework composed of the Brueckner-Hartree-Fock (BHF) method and the g-matrix folding model. For nuclear matter, the g matrix is evaluated from chiral two-nucleon force (2NF) of N 3 LO and chiral 3NF of NNLO by using the BHF method. Since the g matrix thus obtained is numerical and nonlocal, an optimum local form is determined from the onshell and near-on-shell components of g matrix that are important for elastic scattering. For elastic scattering, the optical potentials are calculated by folding the local chiral g matrix with projectile and target densities. This microscopic framework reproduces the experimental data without introducing any adjustable parameter. Chiral-3NF effects are small for proton scattering, but sizable for 4 He scattering at middle angles where the data are available. Chiral 3NF, mainly in the 2π-exchange diagram, makes the folding potential less attractive and more absorptive for all the scattering.