Retrieving records from a gigabyte of text on a minicomputer using statistical ranking

Harman, Donna; Candela, Gerald T.

doi:10.1002/(sici)1097-4571(199012)41:8<581::aid-asi4>3.0.co;2-u

Cited by 99 publications

(65 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In an inverted file document database, each distinct word in the database is held in a vocabulary [4,8,11,19]. The vocabulary might be an array, or a search structure such as a B-tree that can be efficiently updated.…”

Section: Document Databasesmentioning

confidence: 99%

“…Another method for determining the ranking is to directly use the inverted file structure and document lengths [4,8,11], as we choose to do. In this method, the index entry for each word in the query is retrieved in turn.…”

Section: Document Databasesmentioning

confidence: 99%

“…Lewit [4]; Harman and Candela [8]; and Lucarella [11] similarly assume that it is possible to perform random access upon a set of accumulators.…”

Section: Bounding the Accumulator Spacementioning

confidence: 99%

“…Although the cosine measure is poor with regard to update, other ranking measures do not have this limitation [4,8], and moreover, as we have seen, only crude approximations to length are needed for effective ranking. It would probably be sufficient to recompute lengths each time the database doubled in size, requiring perhaps ten full passes over the data in the lifetime of the database.…”

Section: Dynamic Databasesmentioning

confidence: 99%

“…When the documents are stored in a database that is indexed by an inverted file [4,8,19], several structures must be used if ranked query evaluation is to be acceptably fast. These structures include the vocabulary of the document collection; the inverted file entries that record, for each term, which documents contain that term; the word weights; the document lengths; and address tables into the inverted file and the text itself.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Memory efficient ranking

Moffat¹,

Zobel²,

Sacks-Davis³

1994

Information Processing & Management

View full text Add to dashboard Cite

Section: Document Databasesmentioning

confidence: 99%

Section: Document Databasesmentioning

confidence: 99%

“…Lewit [4]; Harman and Candela [8]; and Lucarella [11] similarly assume that it is possible to perform random access upon a set of accumulators.…”

Section: Bounding the Accumulator Spacementioning

confidence: 99%

Section: Dynamic Databasesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Memory efficient ranking

Moffat¹,

Zobel²,

Sacks-Davis³

1994

Information Processing & Management

View full text Add to dashboard Cite

User-friendly systems instead of user-friendly front-ends

Harman

1992

J. Am. Soc. Inf. Sci.

Self Cite

View full text Add to dashboard Cite

Most commercial online retrieval systems are not designed to service end users and, therefore, have often built "front-ends" to their systems specifically to serve the end-user market. These front-ends have not been well accepted, mostly because the underlying systems are still difficult for end users to use successfully in searching. New techniques, based on statistical methods, that allow natural language input and return lists of records in order of likely relevance, have long been available from research laboratories.This article presents four prototype implementations of these statistical retrieval systems that demonstrate their potential as powerful and easily used retrieval systems able to service all users.In an article entitled "In search of the elusive end user" (Summit, 1989), Roger Summit, founder, and president of Dialog Information Services, Inc., estimated that the current usage of DIALOG by end users was about 12% of overall usage, and that that number was growing only at about 20% a year. He mentioned many problems encountered by end users, and described some of the necessary features of user-friendly front-ends. In a subsequent letter to the editor (Cleverdon, 1990), Cyril Cleverdon, however, commented:I do not argue with the points he raises, but I find it astonishing that he makes no mention of the problem which far outweighs all others. I refer to the insistence on Boolean searching.A look at most user-friendly front-ends shows many important features, such as easy dialup and logins, easy file management, and help in selecting databases. All these are clearly necessary and useful, but at the heart of these retrieval systems is a Boolean search engine requiring users to supply AND's and OR's (either directly or indirectly using graphic interfaces), and, more critically, returning an unranked list of document or record titles. At least one study (Miller, Kirby, & Templeton, 1988) found that 45% of search statements issued using a front-end to MEDLINE documents. Equally problematic are searches that are so broad that a very large number of documents are returned; with no ranking, the perusal of this list is intimidating to many users. In addition to the problem of submitting adequate initial queries, there is the problem of improving these queries. Boolean systems seldom have any mechanism for aiding the user in producing a better query, and never have any mechanism for automatically improving that query. A viable alternative to these Boolean retrieval engines are the statistical retrieval engines that take a natural language query and return a list of titles ranked in order of likely relevance to that query.These systems have been developed over the past 30 years in various information retrieval research laboratories, but have had very little influence on commercial online systems. Several operational prototypes (or small operational systems) have been built, particularly recently, to demonstrate the strengths of these statistical retrieval engines, and several small companies are beginning to ...

show abstract

Data compression in full-text retrieval systems

Bell

Moffat

Nevill-Manning

et al. 1993

J. Am. Soc. Inf. Sci.

View full text Add to dashboard Cite

When data compression is applied to full-text retrieval systems, intricate relationships emerge between the amount of compression, access speed, and computing resources required. We propose compression methods, and explore corresponding tradeoffs, for all components of static full-text systems such as text databases on CD-ROM. These components include lexical indexes, inverted files, bitmaps, signature files, and the main text itself. Results are reported on the application of the methods to several substantial full-text databases, and show that a large, unindexed text can be stored, along with indexes that facilitate fast searching, in less than half its original size-at some appreciable cost in primary memory requirements.

show abstract

Retrieving records from a gigabyte of text on a minicomputer using statistical ranking

Cited by 99 publications

References 19 publications

Memory efficient ranking

Memory efficient ranking

User-friendly systems instead of user-friendly front-ends

Data compression in full-text retrieval systems

Contact Info

Product

Resources

About