The Method of Keyword Based Crawler Load Balancing

Wei, Moji; Zhao, Yanqing; Zhu, Shiwei; Yang, An‐Gang

doi:10.12783/dtcse/ceic2018/24546

Search citation statements

Order By: Relevance

Paper Sections

Select...

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2019

Publication Types

Select...

Article1

Other1

Relationship

Self Cite0

Independent2

Authors

Journals

Cited by 2 publications

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Research and Design of Theme Image Crawler Based on Difference Hash Algorithm

Wang

Liang

2019

IOP Conf. Ser.: Mater. Sci. Eng.

View full text Add to dashboard Cite

For the problem of high repetition rate of image resources collected by general theme crawler, a theme image crawler system is designed to reduce image similarity. The main contents of the design include the main function modules of the crawler, the workflow of the system and the implementation method of the key modules. The difference hash algorithm is used to solve the problem of image similarity effectively. Combined with Web text cosine correlation algorithm and link PageRank algorithm, the paper comprehensively evaluates the relevance between Web resources and topics. The experimental results show that the subject image crawler can effectively reduce the similarity of the collected images and improve the efficiency of crawler image resources acquisition.

show abstract