Assessing user-specific difficulty of documents

Paukkeri, Mari-Sanna; Ollikainen, Marja; Honkela, Timo

doi:10.1016/j.ipm.2012.04.001

Cited by 3 publications

(3 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By building on the CHV project, several studies have proposed predictive models for measuring the average familiarity of various consumer health vocabularies based on term occurrence in text corpora [ 14 ], demographics factors [ 15 ], and contextual features [ 16 , 17 ]. In attempts to provide more consumer-friendly health materials, other researchers have developed automated tools for assessing the readability of health texts by substituting difficult terms with easier synonyms and simplifying long sentences [ 18 ] or by comparing the terms appeared in a document and terms known by the user [ 19 ]. Another study to improve the availability of consumer-friendly information is the consumer health educational project by European Patients’ Academy on Therapeutic Innovation (EUPATI) [ 20 ].…”

Section: Introductionmentioning

confidence: 99%

Effects of Individual Health Topic Familiarity on Activity Patterns During Health Information Searches

Puspitasari¹,

Moriyama²,

Fukui³

et al. 2015

JMIR Med Inform

View full text Add to dashboard Cite

BackgroundNon-medical professionals (consumers) are increasingly using the Internet to support their health information needs. However, the cognitive effort required to perform health information searches is affected by the consumer’s familiarity with health topics. Consumers may have different levels of familiarity with individual health topics. This variation in familiarity may cause misunderstandings because the information presented by search engines may not be understood correctly by the consumers.ObjectiveAs a first step toward the improvement of the health information search process, we aimed to examine the effects of health topic familiarity on health information search behaviors by identifying the common search activity patterns exhibited by groups of consumers with different levels of familiarity.MethodsEach participant completed a health terminology familiarity questionnaire and health information search tasks. The responses to the familiarity questionnaire were used to grade the familiarity of participants with predefined health topics. The search task data were transcribed into a sequence of search activities using a coding scheme. A computational model was constructed from the sequence data using a Markov chain model to identify the common search patterns in each familiarity group.ResultsForty participants were classified into L1 (not familiar), L2 (somewhat familiar), and L3 (familiar) groups based on their questionnaire responses. They had different levels of familiarity with four health topics. The video data obtained from all of the participants were transcribed into 4595 search activities (mean 28.7, SD 23.27 per session). The most frequent search activities and transitions in all the familiarity groups were related to evaluations of the relevancy of selected web pages in the retrieval results. However, the next most frequent transitions differed in each group and a chi-squared test confirmed this finding (P<.001). Next, according to the results of a perplexity evaluation, the health information search patterns were best represented as a 5-gram sequence pattern. The most common patterns in group L1 were frequent query modifications, with relatively low search efficiency, and accessing and evaluating selected results from a health website. Group L2 performed frequent query modifications, but with better search efficiency, and accessed and evaluated selected results from a health website. Finally, the members of group L3 successfully discovered relevant results from the first query submission, performed verification by accessing several health websites after they discovered relevant results, and directly accessed consumer health information websites.ConclusionsFamiliarity with health topics affects health information search behaviors. Our analysis of state transitions in search activities detected unique behaviors and common search activity patterns in each familiarity group during health information searches.

show abstract

Section: Introductionmentioning

confidence: 99%

Effects of Individual Health Topic Familiarity on Activity Patterns During Health Information Searches

Puspitasari¹,

Moriyama²,

Fukui³

et al. 2015

JMIR Med Inform

View full text Add to dashboard Cite

show abstract

“…An experiment to rerank search results for people with lower topic familiarity showed that the classifier was effective: the portion of introductory pages at the top 5 and top 10 result sets using this method were significantly higher than those in the baseline run using a default search engine ranking. Paukkeri, Ollikainen, and Honkela () recognize that different users have different levels of DK, and documents are written differently for people with such differences. Documents intended for professionals may not be understandable at all by a novice user and documents for novice users may not contain all the detailed, specific, and in‐depth information needed by an expert.…”

Section: Literature Reviewmentioning

confidence: 99%

The role of domain knowledge in document selection from search results

Liu

Zhang

2019

Asso for Info Science & Tech

View full text Add to dashboard Cite

It is a frequently seen scenario that when people are not familiar with their search topics, they use a simple keyword search, which leads to a large amount of search results in multiple pages. This makes it difficult for users to pick relevant documents, especially given that they are not knowledgeable of the topics. To explore how systems can better help users find relevant documents from search results, the current research analyzed document selection behaviors of users with different levels of domain knowledge (DK). Data were collected in a laboratory study with 35 participants each searching on four tasks in the genomics domain. The results show that users with high and low DK levels selected different sets of documents to view; those high in DK read more documents and gave higher relevance ratings for the viewed documents than those low in DK did. Users with low DK tended to select documents ranking toward the top of the search result lists, and those with high in DK tended to also select documents ranking down the search result lists. The findings help design search systems that can personalize search results to users with different levels of DK.

show abstract

“…However, Gan et al [8] focus on modeling the difficulty of labeling tasks in crowdsourcing instead of single documents. Paukkeri et al [15] propose a method to estimate a document's subjective difficulty for each user separately based on comparing a document's terms with the known vocabulary of an individual. Sameki et al model tweet difficulty in the context of crowdsourcing [19] where they devise a system that minimizes the labeling costs for micro-tasks by allocating more budget to difficult tweets and less to easy ones.…”

Section: Related Workmentioning

confidence: 99%

How Does Tweet Difficulty Affect Labeling Performance of Annotators?

Räbiger,

Saygın,

Spiliopoulou

2018

Preprint

View full text Add to dashboard Cite

Crowdsourcing is a popular means to obtain labeled data at moderate costs, for example for tweets, which can then be used in text mining tasks. To alleviate the problem of low-quality labels in this context, multiple human factors have been analyzed to identify and deal with workers who provide such labels. However, one aspect that has been rarely considered is the inherent difficulty of tweets to be labeled and how this affects the reliability of the labels that annotators assign to such tweets. Therefore, we investigate in this preliminary study this connection using a hierarchical sentiment labeling task on Twitter. We find that there is indeed a relationship between both factors, assuming that annotators have labeled some tweets before: labels assigned to easy tweets are more reliable than those assigned to difficult tweets. Therefore, training predictors on easy tweets enhances the performance by up to 6% in our experiment. This implies potential improvements for active learning techniques and crowdsourcing.

show abstract

Assessing user-specific difficulty of documents

Cited by 3 publications

References 33 publications

Effects of Individual Health Topic Familiarity on Activity Patterns During Health Information Searches

Effects of Individual Health Topic Familiarity on Activity Patterns During Health Information Searches

The role of domain knowledge in document selection from search results

How Does Tweet Difficulty Affect Labeling Performance of Annotators?

Contact Info

Product

Resources

About