Efficient query evaluation using a two-level retrieval process
CIKM '03 Proceedings of the twelfth international conference on Information and knowledge management
Information Processing and Management: an International Journal
Performance comparison of clustered and replicated information retrieval systems
ECIR'07 Proceedings of the 29th European conference on IR research
Query efficiency prediction for dynamic pruning
Proceedings of the 9th workshop on Large-scale and distributed informational retrieval
Learning to predict response times for online query scheduling
SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
Hybrid query scheduling for a replicated search engine
ECIR'13 Proceedings of the 35th European conference on Advances in Information Retrieval
Load-sensitive selective pruning for distributed search
Proceedings of the 22nd ACM international conference on Conference on information & knowledge management
| Hi-index | 0.00 |
For increased efficiency, an information retrieval system can split its index into multiple shards, and then replicate these shards across many query servers. For each new query, an appropriate replica for each shard must be selected, such that the query is answered as quickly as possible. Typically, the replica with the lowest number of queued queries is selected. However, not every query takes the same time to execute, particularly if a dynamic pruning strategy is applied by each query server. Hence, the replica's queue length is an inaccurate indicator of the workload of a replica, and can result in inefficient usage of the replicas. In this work, we propose that improved replica selection can be obtained by using query efficiency prediction to measure the expected workload of a replica. Experiments are conducted using 2.2k queries, over various numbers of shards and replicas for the large GOV2 collection. Our results show that query waiting and completion times can be markedly reduced, showing that accurate response time predictions can improve scheduling accuracy and attesting the benefit of the proposed scheduling algorithm.