Web Crawler: Design And Implementation For Extracting Article-Like Contents

Hien, Ngo Le Huy; Tien, Thai Quang; Hieu, Nguyen Van

doi:10.35470/2226-4116-2020-9-3-144-151

Cited by 7 publications

(8 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Therefore, the intelligent rules of rejection are designed to prevent the crawler from falling into infinite crawling loops. This technique outperforms the web crawler presented in [43]. On comparing accuracy and recall, in testing phase crawler in [43] has accuracy 81.06% and precision 84.62%, while both performance measures have reached above 95% in technique.…”

Section: Comparative Advantagesmentioning

confidence: 84%

“…This technique outperforms the web crawler presented in [43]. On comparing accuracy and recall, in testing phase crawler in [43] has accuracy 81.06% and precision 84.62%, while both performance measures have reached above 95% in technique. Figures 9 and 10 show the comparison of precision and recall delivered by the proposed crawler.…”

Section: Comparative Advantagesmentioning

confidence: 84%

See 1 more Smart Citation

IHWC: intelligent hidden web crawler for harvesting data in urban domains

Kaur

Singh

Geetha³

et al. 2021

Complex Intell. Syst.

View full text Add to dashboard Cite

Due to the massive size of the hidden web, searching, retrieving and mining rich and high-quality data can be a daunting task. Moreover, with the presence of forms, data cannot be accessed easily. Forms are dynamic, heterogeneous and spread over trillions of web pages. Significant efforts have addressed the problem of tapping into the hidden web to integrate and mine rich data. Effective techniques, as well as application in special cases, are required to be explored to achieve an effective harvest rate. One such special area is atmospheric science, where hidden web crawling is least implemented, and crawler is required to crawl through the huge web to narrow down the search to specific data. In this study, an intelligent hidden web crawler for harvesting data in urban domains (IHWC) is implemented to address the relative problems such as classification of domains, prevention of exhaustive searching, and prioritizing the URLs. The crawler also performs well in curating pollution-related data. The crawler targets the relevant web pages and discards the irrelevant by implementing rejection rules. To achieve more accurate results for a focused crawl, ICHW crawls the websites on priority for a given topic. The crawler has fulfilled the dual objective of developing an effective hidden web crawler that can focus on diverse domains and to check its integration in searching pollution data in smart cities. One of the objectives of smart cities is to reduce pollution. Resultant crawled data can be used for finding the reason for pollution. The crawler can help the user to search the level of pollution in a specific area. The harvest rate of the crawler is compared with pioneer existing work. With an increase in the size of a dataset, the presented crawler can add significant value to emission accuracy. Our results are demonstrating the accuracy and harvest rate of the proposed framework, and it efficiently collect hidden web interfaces from large-scale sites and achieve higher rates than other crawlers.

show abstract

Section: Comparative Advantagesmentioning

confidence: 84%

Section: Comparative Advantagesmentioning

confidence: 84%

IHWC: intelligent hidden web crawler for harvesting data in urban domains

Kaur

Singh

Geetha³

et al. 2021

Complex Intell. Syst.

View full text Add to dashboard Cite

show abstract

“…Hasat Oranı Kesinlik/ Hassasiyet(%) Geri Çağırma Oranı Taranan Sayfa Sayısı [43] 0.389 36.00 0.611 5000 [44] 0.411 ------1000 [45] 0.810 ------5000 [46] 0.850 ------5000 [47] 0.500 ---0.600 6500 [48] ---92.01 0.590 495 [49] 0.890 ------1000 [50] 0.500 32.00 ---3200 [51] 0.830 ------1200 [26] ---92.00 0.300 2000 [25] 0.750 ---0.400 10000 [52] ------0.820 400 [53] 0.850 ------6000 [54] 0.700 ---0.470 13377 Tablo 2' de görüldüğü gibi odaklı web tarayıcılarında en çok kullanılan performans ölçütü hasat oranıdır. Hasat oranı taranan sayfa sayıları arttıkça genel olarak artmaktadır.…”

Section: Kaynakunclassified

Web Tarayıcılarında Tohum URL Seçimi ve Performans Analizi: Kapsamlı Bir İnceleme

ALANOĞLU

Akçayol

2023

Düzce Üniversitesi Bilim Ve Teknoloji Dergisi

View full text Add to dashboard Cite

Web, İnternet üzerinde yayınlanan çeşitli türden bilgilerin bulunduğu bir veri deposudur. Bu bilgileri üzerinde bulunduran ve birbirlerine köprülerle bağlı olan yapılara web sayfaları denir. Web tarayıcıları, web sayfaları üzerindeki köprüleri kullanarak Web’i tarayan ve sayfaları indiren programlardır. Bir arama motorunun performansı da web tarayıcısının performansına bağlıdır. Web tarayıcılarının performans metrikleri, kapsamı ve tohum URL seçim yöntemleri performansı etkileyen en önemli faktörlerdir. Bu çalışmada, genel, odaklanmış, artırılmış, gizli, mobil ve dağıtılmış olmak üzere altı kategoride sınıflandırdığımız web tarayıcılarının performansları, kapsamları ve tohum URL kullanım yöntemleri hakkında kapsamlı bir inceleme ve analiz yapılmıştır. Ayrıca her bir tarayıcının çeşitli çalışmalarda yapılmış performans ölçütleri karşılaştırılmıştır.

show abstract

“…[41,42] has been employed in this study to estimate the total CO 2 emissions and fuel consumption of vehicles from multiple inputs. CNN is a form of deep neural network that is often used to explore visual imagery [37,43]. The deep learning model has been built using Google Collab and results are presented in Figure 19 and Table 14.…”

Section: Linear Regression and Univariate Polynomial Regressionmentioning

confidence: 99%

Analysis and Prediction Model of Fuel Consumption and Carbon Dioxide Emissions of Light-Duty Vehicles

Hien

Kor

2022

Applied Sciences

Self Cite

View full text Add to dashboard Cite

Due to the alarming rate of climate change, fuel consumption and emission estimates are critical in determining the effects of materials and stringent emission control strategies. In this research, an analytical and predictive study has been conducted using the Government of Canada dataset, containing 4973 light-duty vehicles observed from 2017 to 2021, delivering a comparative view of different brands and vehicle models by their fuel consumption and carbon dioxide emissions. Based on the findings of the statistical data analysis, this study makes evidence-based recommendations to both vehicle users and producers to reduce their environmental impacts. Additionally, Convolutional Neural Networks (CNN) and various regression models have been built to estimate fuel consumption and carbon dioxide emissions for future vehicle designs. This study reveals that the Univariate Polynomial Regression model is the best model for predictions from one vehicle feature input, with up to 98.6% accuracy. Multiple Linear Regression and Multivariate Polynomial Regression are good models for predictions from multiple vehicle feature inputs, with approximately 75% accuracy. Convolutional Neural Network is also a promising method for prediction because of its stable and high accuracy of around 70%. The results contribute to the quantifying process of energy cost and air pollution caused by transportation, followed by proposing relevant recommendations for both vehicle users and producers. Future research should aim towards developing higher performance models and larger datasets for building APIs and applications.

show abstract

Web Crawler: Design And Implementation For Extracting Article-Like Contents

Cited by 7 publications

References 16 publications

IHWC: intelligent hidden web crawler for harvesting data in urban domains

IHWC: intelligent hidden web crawler for harvesting data in urban domains

Web Tarayıcılarında Tohum URL Seçimi ve Performans Analizi: Kapsamlı Bir İnceleme

Analysis and Prediction Model of Fuel Consumption and Carbon Dioxide Emissions of Light-Duty Vehicles

Contact Info

Product

Resources

About