An analytical and cold model study of a prototype extended vane rod-type RFQ (radio frequency quadrupole linac) is presented. The bead-pull measurement allowed us to experimentally determine the effect of shaping of the vane-supporting posts and, most interestingly, the size of the vacuum enclosure in the dipolar and quadrupolar field asymmetry. Disk shaped beads of BaTiO3 have been tailor made for this purpose. The special shape of the beads has allowed precise measurements of frequency shifts all along the length of the RFQ. The measured quadrupolar asymmetry and dipole component at a radial distance of 7 mm from the beam axis are within ±1% for the optimized structure. Particle tracking of a 10 mA proton beam with a simulated 3D field for the optimized full scale RFQ with modulated vanes has been carried out. Simulation shows vertical and horizontal shifts of only 0.01 mm and 0.03 mm, respectively, for the accelerated beam at the exit of RFQ. The percentage of accelerated particles lying within an energy width of ±40 keV about 800 keV is increased from 86% for the original enclosure to 88% for the reduced enclosure, whereas the transmission efficiency is 100% for both the cases.