Survey of pre-processing techniques for mining big data

Hariharakrishnan, Jayaram; Mohanavalli, S.; Srividya,; Kumar, Krishna B.

doi:10.1109/icccsp.2017.7944072

Cited by 34 publications

(19 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To solve this problem and clean the data, a double smoothing method (DSM) is utilized, in which the binning method is followed by a clustering based technique. The binning method is used for data smoothing to eliminate noisy data [20], and it consists of two main steps: (1) sorting the data and partitioning them into (equal frequency) bins; and (2) smoothing the data (the boundary based method is used in this study [21]). The goal of the clustering method is to detect and eliminate the outliers, which are considered as the most negative type of noises that can be located within the data.…”

Section: B Used Data Setmentioning

confidence: 99%

Intelligent System for Price Premium Prediction in Online Auctions

Mohamed¹,

Kamel²

2020

IJACSA

View full text Add to dashboard Cite

The use of data mining techniques in the field of auctions has attracted considerable interest from the research community. In auctions, the users try to achieve the highest gain and avoid loss as much as possible. Therefore, data mining techniques can be implemented in the auctioning domain to develop an intelligent method that can be used by the users in online auctions. However, determining the factors that affect the result of an auction, especially the initial price, is critical. In addition, the intelligent system must be established based on clean data to ensure the accuracy of the results. In this paper, we propose an intelligent system (classifier) to predict the initial price of auctions. The proposed system uses the double smoothing method (DSM) for data cleaning in terms of preprocessing. This system is implemented on a data set collected from the eBay website and cleaned using the proposed DSM. In the training phase, the CART technique is employed for the classifier construction. Compared to similar techniques, the proposed system exhibits a better performance in terms of the accuracy and robustness against noisy data, as determined using ROC curves.

show abstract

Section: B Used Data Setmentioning

confidence: 99%

Intelligent System for Price Premium Prediction in Online Auctions

Mohamed¹,

Kamel²

2020

IJACSA

View full text Add to dashboard Cite

show abstract

“…However, in the present paper, the authors pay an attention only on remarkable findings of techniques and technologies involved in preparing the big web data suitable to analytics. In this direction, some of the authors [10,11,12,14,27] described various methodologies discovered by both research and industry community to pre-process the weblog data efficiently in Big Data environment. According to the research works made by the authors [10,20,27,31] are endorsed that big data storage, big data cleansing, unique user identification, session identification and so on are important and crucial tasks in the big data preprocessing model.…”

Section: Related Workmentioning

confidence: 99%

“…Only the records having 200 as the status code is treated as relevant data. Many authors in the literature [10,11,12,14,20,27] are also prominently expressed that removal of failure requests, irrelevant file requests, inappropriate access method requests, web robots requests, internal dummy connection requests and irrelevant log fields to extract more relevant data for the analysis of weblog.…”

Section: Architecture Of Hadoop For Processing Of Web Logsmentioning

confidence: 99%

A Complete Research on Techniques & Technologies of Big Web Data Preparation to Web User Usage Behaviour

2019

IJRTE

View full text Add to dashboard Cite

The rapid advancements in data digitization, the most powerful inventions of learning methodologies in data collection and reduced cost of data storage further enabled the World Wide Web with immense amount of data at significant rate in all the key domains. The generated web data is non-scalable, high dimensional, widely distributed, heterogeneous, dynamic in nature and having useful insights, and thus, it evolved as big data. This situation creates inevitably increasing opportunities in extracting structured solutions from unstructured weblog data for the present big data researchers. Moreover, to provide value addition to any key domain and derive actionable knowledge for various applications, such as, web usage analysis for improvements in fraud detection, product analysis and customer segmentation, got the focus in big data era by the web analysts. To improve operational performance and to discover hidden insights accurately, a comprehensive process is required to investigate the web user usage behavior by analyzing big web data. Towards this, the authors concentrate on reviewing the techniques and technologies of web data collection and preparation for investigating web user usage behavior effectively. In the present paper, the researchers initially pay an attention to explore web log data preparation methods in the traditional approach. Later, the review emphasizes on Hadoop approach for big data preparation and processing. This approach able to concentrate comprehensively on both the stages: distributed data storage and parallel processing of weblog data and to leverage the strengths of techniques and technologies of individual stages. Moreover, the authors deliberately review the possible potential research paths that results in an improved methodologies for data storage and optimized processing speed in the era of big web data

show abstract

“…In last few years data cleaning and preprocessing is viewed as an important action and topic of research, as various supervised and unsupervised methods have been experimented for number of domains for analysis purpose. Hariharakrishnan, J. et al [15] reviewed numerous techniques of text preprocessing and highlighted the significance and multi-aspects of preprocessing such as Noise reduction, outlier identification, and inconsistent data. They raised a very logical point that most of the experiments for text preprocessing are performed either in data collection phase or for homogeneous data.…”

Section: Related Workmentioning

confidence: 99%

Normalization of Unstructured and Informal Text in Sentiment Analysis

Javed¹,

Kamal²

2018

ijacsa

View full text Add to dashboard Cite

Sentiment Analysis is problem of natural language processing which deals with the extraction and analysis of public sentiments shared about target entities over microbloging websites. This field has gained great attention due to the huge availability of decision making textual contents. Sentiment Analysis has enormous application areas such as; Market Analysis, Service Analysis, Showbiz analysis, Movies, sports and even the popularity and acceptance rate of political policies can also be predicted via sentiment analysis systems. Although tremendous volume of opinionative text is available but it is unstructured and noisy due to which sentiment classifiers can't achieve good outcomes. Normalization is the process used to clean noise from unstructured text for sentiment analysis. In this study we have proposed a mechanism for the normalization of informal and unstructured text. Proposed mechanism is comprised of four essential phases; Noise Reduction, Part of Speech Tagging, Stop Word Removal stemming and Lemmatization. Numerous experiments are performed on twitter data set with unsupervised lexicons and dictionaries. Python and Natural language toolkit is used for performing all four essential steps. This study demonstrates that utilization and normalization of informal tokens in tweets improved the overall classification accuracy from 75.42 to 82.357.

show abstract

Survey of pre-processing techniques for mining big data

Cited by 34 publications

References 16 publications

Intelligent System for Price Premium Prediction in Online Auctions

Intelligent System for Price Premium Prediction in Online Auctions

A Complete Research on Techniques & Technologies of Big Web Data Preparation to Web User Usage Behaviour

Normalization of Unstructured and Informal Text in Sentiment Analysis

Contact Info

Product

Resources

About