Loading...
Efficient Update of Indexes for Dynamically Changing Web Documents
Lim, Lipyeow Wang, Min Padmanabhan, Sriram Vitter, Jeffrey Scott Agarwal, Ramesh
Lim, Lipyeow
Wang, Min
Padmanabhan, Sriram
Vitter, Jeffrey Scott
Agarwal, Ramesh
Citations
Altmetric:
Abstract
Recent work on incremental crawling has enabled the indexed document collection of a
search engine to be more synchronized with the changing World Wide Web. However, this
synchronized collection is not immediately searchable, because the keyword index is rebuilt
from scratch less frequently than the collection can be refreshed. An inverted index is usually
used to index documents crawled from the web. Complete index rebuild at high frequency is
expensive. Previous work on incremental inverted index updates have been restricted to adding
and removing documents. Updating the inverted index for previously indexed documents that
have changed has not been addressed.
In this paper, we propose an efficient method to update the inverted index for previously
indexed documents whose contents have changed. Our method uses the idea of landmarks
together with the diff algorithm to significantly reduce the number of postings in the inverted
index that need to be updated. Our experiments verify that our landmark-diff method results
in significant savings in the number of update operations on the inverted index.
Description
The original publication is available at www.springerlink.com
Date
2007
Journal Title
Journal ISSN
Volume Title
Publisher
Springer Verlag
Research Projects
Organizational Units
Journal Issue
Keywords
Web search, Indexing methods, Document management
Citation
L. Lim, M. Wang, S. Padmanabhan, J. S. Vitter, and R. Agarwal. “Efficient Update of Indexes for Dynamically Changing Web Documents,” World Wide Web, 10(1), March 2007, 37–69. An extended abstract appears in “Dynamic Maintenance of Web Indexes Using Landmarks,” Proceedings of the 12th International World Wide Web Conference (WWW ’03), Budapest, May 2003, 102–111. http://dx.doi.org/10.1007/s11280-006-0009-2