A re-examination of relevance: toward a dynamic, situational definition
Information Processing and Management: an International Journal
Variations in relevance judgments and the measurement of retrieval effectiveness
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Information Processing and Management: an International Journal
Modern Information Retrieval
Using graded relevance assessments in IR evaluation
Journal of the American Society for Information Science and Technology
The concept of relevance in IR
Journal of the American Society for Information Science and Technology
Experimenting with the automatic assignment of educational standards to digital library content
Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries
TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing)
Fedora: an architecture for complex objects and their relationships
International Journal on Digital Libraries
Relationship between index term specificity and relevance judgment
Information Processing and Management: an International Journal
Standards or semantics for curriculum search?
Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries
Text Categorization for Aligning Educational Standards
HICSS '07 Proceedings of the 40th Annual Hawaii International Conference on System Sciences
Journal of the American Society for Information Science and Technology
Journal of the American Society for Information Science and Technology
User rankings of search engine results
Journal of the American Society for Information Science and Technology
Contexts of relevance for information retrieval system design
CoLIS'05 Proceedings of the 5th international conference on Context: conceptions of Library and Information Sciences
Searching for relevance in the relevance of search
CoLIS'05 Proceedings of the 5th international conference on Context: conceptions of Library and Information Sciences
World vs. method: educational standard formulation impacts document retrieval
Proceedings of the 11th annual international ACM/IEEE joint conference on Digital libraries
Journal of the American Society for Information Science and Technology
Characterizing and Predicting the Multifaceted Nature of Quality in Educational Web Resources
ACM Transactions on Interactive Intelligent Systems (TiiS)
Hi-index | 0.00 |
With over 60,000 US K-12 science and mathematics education standards and a rapid proliferation of Web-enabled curriculum, retrieving curriculum that aligns with the standards to which teachers must teach is a key objective for educational digital libraries. However, previous studies of such alignment use single-dimensional and binary measures of the alignment concept. As a consequence, they suffer from low inter-rater reliability (IRR), with experts agreeing about alignments only some 20-40% of the time. We present the results of an experiment in which the alignment variable was operationalized using the Saracevic model of relevance 'clues' taken from the everyday practice of K-12 teaching. Results show high IRR across all clues with IRR on several specific alignment dimensions significantly higher than on overall alignment. In addition, a model of overall alignment is derived and estimated. The structure and explanatory power of the model as well as the relationships between alignment clues differ significantly between alignments of curriculum found by users themselves and curriculum found by others. These results illustrate the usefulness of clue-based relevance measures for information retrieval and have important consequences for both the formulation of automated retrieval mechanisms and the construction of a gold standard or benchmark set of standard-curriculum alignments.