Physical Unclonable Functions (PUFs) are used for authentication and generation of secure cryptographic keys. However, recent research work has shown that PUFs, in general, are vulnerable to machine learning modeling attacks. From a subset of Challenge-Response Pairs (CRPs), the remaining CRPs can be effectively predicted using different machine learning algorithms. In this work, Artificial Neural Networks (ANNs) using swarm intelligence-based modeling attacks are used against different silicon-based PUFs to test their resiliency to these attacks. Amongst the swarm intelligence algorithms, the Gravitational Search Algorithm (GSA), Cuckoo Search Algorithm (CS), Particle Swarm Optimizer (PSO) and the Grey Wolf Optimizer (GWO) are used. The attacks are extensively performed on six different types of PUFs; namely, Configurable Ring Oscillator, Inverter Ring Oscillator, XOR-Inverter Ring Oscillator, Arbiter, Modified XOR-Inverter Ring Oscillator, and Hybrid Delay Based PUF. From the results, it can be concluded that the first four PUFs under study are vulnerable to ANN swarm intelligence-based models, and their responses can be predicted with an average accuracy of 71.1% to 88.3 % for the different models. However, for the Hybrid Delay Based PUF and the Modified XOR-Inverter Ring Oscillator PUF, which are especially designed to thwart machine learning attacks, the prediction accuracy is much lower and in the range of 9.8 % to 14.5 %.