Research and Implementation of Campus Network Search Engine Based on Scrapy Framework and Elasticsearch

Xue, Tian; Zhang, Tiancheng; Zhuang, Xufei; He, Xue Dong

doi:10.1109/ccdc49329.2020.9164582

Cited by 4 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From the perspective of the paper "Analyzing Open Source GitHub Repositories Towards Technology Acceptance Model" [17], a proposal for analyzing public data from GitHub repositories available to the public in Big Query was presented. The article investigated the correlations and anomalies between the trends and the most used languages using the ELK suite [18]. The research concluded with an architecture that integrates data from Big Query and is analyzed with Elasticsearch, Logstash, and Kibana, which determine the correlation between trends and anomalies in developing technologies.…”

Section: Related Workmentioning

confidence: 99%

“…Elasticsearch is a distributed real-time analytics engine, first released in 2010 [29] and designed to organize data to make it easily accessible [30]. It is developed as open-source on Apache Lucene [31], in Java [29] and is part of the ELK suite [18], where we can find other tools for analytics such as Logstash and Kibana [29]. Elasticsearch works like a distributed document warehouse, saving them in JSON format [30].…”

Section: ) Elasticsearchmentioning

confidence: 99%

“…The main function of Elasticsearch is search [18]. The general process begins with indexing a document when it is saved to the system.…”

Section: ) Elasticsearchmentioning

confidence: 99%

“…Elasticsearch keeps two fields by default: the document's original content. The other is the inverted index, which is generated by a series of processes such as word segmentation and filtering during indexing [18]. It should be noted that documents in Elasticsearch are indexed by default and do not have a schema [33], so it's not necessary defining fields for data types before adding data [30].…”

Section: ) Elasticsearchmentioning

confidence: 99%

“…Now, for searching, the user enters a keyword of the document he intends to access, so Elasticsearch executes a search in the inverted index table, after which the document corresponding to the keyword is related. The result is returned to the user [18].…”

Section: ) Elasticsearchmentioning

confidence: 99%

See 4 more Smart Citations

A Cloud Pub/Sub Architecture to Integrate Google Big Query with Elasticsearch using Cloud Functions

Gutiérrez¹,

Vera²

2022

IJC

View full text Add to dashboard Cite

In recent years, the need for analytics on large volumes of data has become increasingly important. It turns out to be extremely useful in making strategic decisions about different applications. In this way, appropriate mechanisms must be designed to carry out data processing and integration with different platforms to take advantage of their best features. In this work, an architecture that works on cloud services is shown to migrate data stored in Big Query to an analytics engine such as Elasticsearch and take advantage of its potential in query, insert and display operations. This is accomplished through the use of Cloud Functions and Pub / Sub. The integration of these platforms through the proposed architecture showed 100% effectiveness when transferring data to another, maintaining an insertion rate of 4,138.30 documents per second, demonstrating its robustness, efficiency, and versatility when performing a data migration. This pretends to establish an architecture solution when it comes about handling a large amount of data as in the real world.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: ) Elasticsearchmentioning

confidence: 99%

“…The main function of Elasticsearch is search [18]. The general process begins with indexing a document when it is saved to the system.…”

Section: ) Elasticsearchmentioning

confidence: 99%

Section: ) Elasticsearchmentioning

confidence: 99%

Section: ) Elasticsearchmentioning

confidence: 99%

See 3 more Smart Citations

A Cloud Pub/Sub Architecture to Integrate Google Big Query with Elasticsearch using Cloud Functions

Gutiérrez¹,

Vera²

2022

IJC

View full text Add to dashboard Cite

show abstract

Design of telecom operators IDC resale analysis system based on Spark

Song

Jia

Huang

et al. 2022

2022 International Conference on Computing, Robotics and System Sciences (ICRSS)

View full text Add to dashboard Cite

Design of Artificial Intelligence Companion Chatbot

Chen,

Kang,

2024

JNM

View full text Add to dashboard Cite

With the development of cities and the prevalence of networks, interpersonal relationships have become increasingly distant. When people crave communication, they hope to find someone to confide in. With the rapid advancement of deep learning and big data technologies, an enabling environment has been established for the development of intelligent chatbot systems. By effectively combining cutting-edge technologies with humancentered design principles, chatbots hold the potential to revolutionize our lives and alleviate feelings of loneliness. A multi-topic chat companion robot based on a state machine has been proposed, which can engage in fluent dialogue with humans and meet different functional requirements. It can chat with users about movies, music, and other related topics, and recommend movies and music that may interest them to alleviate their loneliness and provide companionship. The interaction platform of the companion robot is realized through the QQ communication platform, with two chat modes: Conversation mode and recommendation mode. First, the KdConv open-source corpus was selected, and Python was used to crawl information on movies and music from Douban and QQ Music to establish and pre-process the dataset. Then, the dialogue function was implemented using generative language models and retrieval systems, while the recommendation function was achieved using user profiling and collaborative filtering. Finally, a state machine algorithm was used to achieve real-time switching between the two chat modes of the companion robot. In conclusion, test participants gave high ratings for the accuracy of the companion robot's responses and the satisfaction with its content recommendations. Compared to traditional large-scale integrated models, this robot employs a state-machine framework to achieve diverse functions through seamless state transitions, thereby enhancing computational speed and precision. Additionally, the robot can recommend movies and music, providing companionship and alleviating loneliness for users, which is of great significance in modern society where interpersonal relationships are increasingly alienated.

show abstract

Research and Implementation of Campus Network Search Engine Based on Scrapy Framework and Elasticsearch

Cited by 4 publications

References 0 publications

A Cloud Pub/Sub Architecture to Integrate Google Big Query with Elasticsearch using Cloud Functions

A Cloud Pub/Sub Architecture to Integrate Google Big Query with Elasticsearch using Cloud Functions

Design of telecom operators IDC resale analysis system based on Spark

Design of Artificial Intelligence Companion Chatbot

Contact Info

Product

Resources

About