The vocabulary problem in human-system communication
Communications of the ACM
Measuring relevance judgements
Information Processing and Management: an International Journal
Development of an instrument measuring user satisfaction of the human-computer interface
CHI '88 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Formative design evaluation of superbook
ACM Transactions on Information Systems (TOIS)
Evaluation of the user interface in an information retrieval system: a model
Information Processing and Management: an International Journal
Information Processing and Management: an International Journal
Information retrieval interaction
Information retrieval interaction
Measuring the informativeness of a retrieval process
SIGIR '92 Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval
Evaluation issues in information retrieval
Information Processing and Management: an International Journal - Special issue on evaluation issues in information retrieval
The state of retrieval system evaluation
Information Processing and Management: an International Journal - Special issue on evaluation issues in information retrieval
On the evaluation of IR systems
Information Processing and Management: an International Journal - Special issue on evaluation issues in information retrieval
The pragmatics of information retrieval experimentation, revisited
Information Processing and Management: an International Journal - Special issue on evaluation issues in information retrieval
Evaluation measures for interactive information retrieval
Information Processing and Management: an International Journal - Special issue on evaluation issues in information retrieval
The experience of flow in computer-mediated and in face-to-face groups
ICIS '91 Proceedings of the twelfth international conference on Information systems
Wizard of Oz studies: why and how
IUI '93 Proceedings of the 1st international conference on Intelligent user interfaces
Evaluating hypermedia and learning: methods and results from the Perseus Project
ACM Transactions on Information Systems (TOIS)
Measuring usability: preference vs. performance
Communications of the ACM
Journal of the American Society for Information Science
Task complexity affects information seeking and use
Information Processing and Management: an International Journal
Evaluation of evaluation in information retrieval
SIGIR '95 Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval
Towards new measures of information retrieval evaluation
SIGIR '95 Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval
CHI '95 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Multiple search sessions model of end-user behavior: an exploratory study
Journal of the American Society for Information Science
Some perspectives on the evaluation of information retrieval systems
Journal of the American Society for Information Science - Special issue: evaluation of information retrieval systems
Journal of the American Society for Information Science - Special issue: evaluation of information retrieval systems
Evaluation of a tool for visualization of information retrieval results
SIGIR '96 Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval
User analysis in HCI—the historical lessons from individual differences research
International Journal of Human-Computer Studies
Time, relevance and interaction modelling for information retrieval
Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval
Effectiveness of a graphical display of retrieval results
Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval
Journal of the American Society for Information Science - Special topic issue on the history of documentation and information science: part II
Affective computing
Given a context by any other name: methodological tools for taming the unruly beast
ISIC '96 Proceedings of an international conference on Information seeking in context
Information needs: a person-in-situation approach
ISIC '96 Proceedings of an international conference on Information seeking in context
The Cranfield tests on index language devices
Readings in information retrieval
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Measures of relative relevance and ranked half-life: performance indicators for interactive IR
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
A study of the use of variables in information retrieval user studies
Journal of the American Society for Information Science
Towards the identification of the optimal number of relevance categories
Journal of the American Society for Information Science
Interaction in information retrieval: trends over time
Journal of the American Society for Information Science - Special issue on the 50th anniversary of the Journal of The American Society for Information Science: part 2: paradigms, models and methods of information science
User interactions with everyday applications as context for just-in-time information access
Proceedings of the 5th international conference on Intelligent user interfaces
Hedonic and ergonomic quality aspects determine a software's appeal
Proceedings of the SIGCHI conference on Human Factors in Computing Systems
Helping people find what they don't know
Communications of the ACM
Do batch and user evaluations give the same results?
SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
IR evaluation methods for retrieving highly relevant documents
SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
Regions and levels: measuring and mapping users' relevance judgments
Journal of the American Society for Information Science and Technology
Cookies and Web browser design: toward realizing informed consent online
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Why batch and user evaluations do not give the same results
Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
Placing search in context: the concept revisited
ACM Transactions on Information Systems (TOIS)
Cognitive and task influences on Web searching behavior
Journal of the American Society for Information Science and Technology
Web searching: a process-oriented experimental study of three interactive search paradigms
Journal of the American Society for Information Science and Technology
Judgement of information quality and cognitive authority in the Web
Journal of the American Society for Information Science and Technology
Collaborative relevance judgment: a group consensus method for evaluating user search performance
Journal of the American Society for Information Science and Technology
Automated monitoring to support the analysis and evaluation of information systems
SIGIR '79 Proceedings of the 2nd annual international ACM SIGIR conference on Information storage and retrieval: information implications into the eighties
End user behavior on an online information retrieval system: a computer monitoring study
SIGIR '83 Proceedings of the 6th annual international ACM SIGIR conference on Research and development in information retrieval
Information and information sources in tasks of varying complexity
Journal of the American Society for Information Science and Technology
The role of individual differences in Internet searching: an empirical study
Journal of the American Society for Information Science and Technology
Information Retrieval Experiment
Information Retrieval Experiment
Liberal relevance criteria of TREC -: counting on negligible documents?
SIGIR '02 Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval
Information seeking and mediated searching. Part 4: cognitive styles in information seeking
Journal of the American Society for Information Science and Technology
Multitasking information seeking and searching processes
Journal of the American Society for Information Science and Technology
Cumulated gain-based evaluation of IR techniques
ACM Transactions on Information Systems (TOIS)
Research ethics in Internet-enabled research: Human subjects issues and methodological myopia
Ethics and Information Technology
What is special about the ethical issuesin online research?
Ethics and Information Technology
Evaluating Interactive Cross-Language Information Retrieval: Document Selection
CLEF '00 Revised Papers from the Workshop of Cross-Language Evaluation Forum on Cross-Language Information Retrieval and Evaluation
Using graded relevance assessments in IR evaluation
Journal of the American Society for Information Science and Technology
Client-side monitoring for web mining
Journal of the American Society for Information Science and Technology
Stuff I've seen: a system for personal information retrieval and re-use
Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval
Using terminological feedback for web search refinement: a log-based study
Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval
Query length in interactive information retrieval
Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval
The concept of relevance in IR
Journal of the American Society for Information Science and Technology
What we talk about when we talk about context
Personal and Ubiquitous Computing
A diary study of task switching and interruptions
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
The perfect search engine is not enough: a study of orienteering behavior in directed search
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Lost in memories: interacting with photo collections on PDAs
Proceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries
WiIRE: the web interactive information retrieval experimentation system prototype
Information Processing and Management: an International Journal
Seeking information in order to produce information: an empirical study at Hewlett Packard labs
Journal of the American Society for Information Science and Technology - Special issue: Part II: Information seeking research
Flow experiences in information technology use
International Journal of Human-Computer Studies
The influence of relevance levels on the effectiveness of interactive information retrieval
Journal of the American Society for Information Science and Technology
Journal of the American Society for Information Science and Technology
Journal of the American Society for Information Science and Technology
How users assess web pages for information seeking
Journal of the American Society for Information Science and Technology
Conceptual framework for tasks in information studies: Book Reviews
Journal of the American Society for Information Science and Technology
Interfaces for staying in the flow
Ubiquity
Accurately interpreting clickthrough data as implicit feedback
Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
Personalizing search via automated analysis of interests and activities
Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
Assessing learning outcomes in two information retrieval learning environments
Information Processing and Management: an International Journal
Evaluating implicit feedback models using searcher simulations
ACM Transactions on Information Systems (TOIS)
The Turn: Integration of Information Seeking and Retrieval in Context (The Information Retrieval Series)
TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing)
Exploratory search: from finding to understanding
Communications of the ACM - Supporting exploratory search
Cross-Evaluation: A new model for information system evaluation
Journal of the American Society for Information Science and Technology
The validity of the stimulated retrospective think-aloud method as measured by eye tracking
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Written versus spoken queries: A qualitative and quantitative comparative analysis
Journal of the American Society for Information Science and Technology - Research Articles
How are we searching the world wide web?: a comparison of nine search engine transaction logs
Information Processing and Management: an International Journal - Special issue: Formal methods for information retrieval
Current practice in measuring usability: Challenges to usability studies and research
International Journal of Human-Computer Studies
The influence of task and gender on search and evaluation behavior using Google
Information Processing and Management: an International Journal
User performance versus precision measures for simple search tasks
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Improving web search ranking by incorporating user behavior information
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Elicitation of term relevance feedback: an investigation of term source and context
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Proceedings of the 1st international conference on Information interaction in context
IIiX Proceedings of the 1st international conference on Information interaction in context
Glass box: capturing, archiving, and retrieving workstation activities
Proceedings of the 3rd ACM workshop on Continuous archival and retrival of personal experences
On GMAP: and other transformations
CIKM '06 Proceedings of the 15th ACM international conference on Information and knowledge management
Evaluation by comparing result sets in context
CIKM '06 Proceedings of the 15th ACM international conference on Information and knowledge management
Theories of Information Behavior (Asist Monograph)
Theories of Information Behavior (Asist Monograph)
Relationships between categories of relevance criteria and stage in task completion
Information Processing and Management: an International Journal
Meta-analysis of correlations among usability measures
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Journal of the American Society for Information Science and Technology
An analysis of two approaches in information retrieval: From frameworks to study designs
Journal of the American Society for Information Science and Technology
A field study characterizing Web-based information-seeking tasks
Journal of the American Society for Information Science and Technology
Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries
Overview of the TREC 2006 ciQA task
ACM SIGIR Forum
Towards task-based personal information management evaluations
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
Studying the use of popular destinations to enhance web search interaction
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
SearchTogether: an interface for collaborative web search
Proceedings of the 20th annual ACM symposium on User interface software and technology
Information Processing and Management: an International Journal
Editorial: Introduction to the special issue on evaluating interactive information retrieval systems
Information Processing and Management: an International Journal
Information Processing and Management: an International Journal
On the role of user-centred evaluation in the advancement of interactive information retrieval
Information Processing and Management: an International Journal
Controlling the complexity in comparing search user interfaces via user studies
Information Processing and Management: an International Journal
Evaluation of a mobile information system in context
Information Processing and Management: an International Journal
Questionnaire mode effects in interactive information retrieval experiments
Information Processing and Management: an International Journal
Effectiveness of additional representations for the search result presentation on the web
Information Processing and Management: an International Journal
Journal of the American Society for Information Science and Technology
Journal of the American Society for Information Science and Technology
End-user privacy in human-computer interaction
Foundations and Trends in Human-Computer Interaction
Meeting of the MINDS: an information retrieval research agenda
ACM SIGIR Forum
Journal of the American Society for Information Science and Technology
What is user engagement? A conceptual framework for defining user engagement with technology
Journal of the American Society for Information Science and Technology
Eye tracking and online search: Lessons learned and challenges ahead
Journal of the American Society for Information Science and Technology
Mining the search trails of surfing crowds: identifying relevant websites from user activity
Proceedings of the 17th international conference on World Wide Web
On the history of evaluation in IR
Journal of Information Science
How do users find things with PubMed?: towards automatic utility evaluation with user simulations
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
User adaptation: good results from poor systems
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Affective feedback: an investigation into the role of emotions in the information seeking process
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Relevance assessment: are judges exchangeable and does it matter
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Effects of performance feedback on users' evaluations of an interactive IR system
Proceedings of the second international symposium on Information interaction in context
Comparing collaborative and independent search in a recall-oriented task
Proceedings of the second international symposium on Information interaction in context
A faceted approach to conceptualizing tasks in information seeking
Information Processing and Management: an International Journal
On test collections for adaptive information retrieval
Information Processing and Management: an International Journal
Crowdsourcing for relevance evaluation
ACM SIGIR Forum
User-centered evaluation of interactive question answering systems
IQA '06 Proceedings of the Interactive Question Answering Workshop at HLT-NAACL 2006
Collaborative information seeking and retrieval
Annual Review of Information Science and Technology
Interactive information retrieval
Annual Review of Information Science and Technology
Discounted cumulated gain based evaluation of multiple-query IR sessions
ECIR'08 Proceedings of the IR research, 30th European conference on Advances in information retrieval
The interactive track at INEX 2005
INEX'05 Proceedings of the 4th international conference on Initiative for the Evaluation of XML Retrieval
The effects on topic familiarity on online search behaviour and use of relevance criteria
ECIR'06 Proceedings of the 28th European conference on Advances in Information Retrieval
A user behavior model for average precision and its generalization to graded judgments
Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
When is system support effective?
Proceedings of the third symposium on Information interaction in context
Proceedings of the third symposium on Information interaction in context
Proceedings of the third symposium on Information interaction in context
Proceedings of the 73rd ASIS&T Annual Meeting on Navigating Streams in an Information Ecosystem - Volume 47
Developing a corpus of plagiarised short answers
Language Resources and Evaluation
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Find it if you can: a game for modeling different types of web search success using interaction data
Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
CLEF'11 Proceedings of the Second international conference on Multilingual and multimodal information access evaluation
Discounted cumulative gain and user decision models
SPIRE'11 Proceedings of the 18th international conference on String processing and information retrieval
Use cases as a component of information access evaluation
Proceedings of the 2011 workshop on Data infrastructurEs for supporting information retrieval evaluation
A user interaction model based on the principle of polyrepresentation
Proceedings of the 4th workshop on Workshop for Ph.D. students in information & knowledge management
Who, what, why: examining annotations in mobile content sharing games
ICADL'11 Proceedings of the 13th international conference on Asia-pacific digital libraries: for cultural heritage, knowledge dissemination, and future creation
Evaluating tag-based information access in image collections
Proceedings of the 23rd ACM conference on Hypertext and social media
An extensible personal photograph collection for graded relevance assessments and user simulation
Proceedings of the 2nd ACM International Conference on Multimedia Retrieval
A subjunctive exploratory search interface to support media studies researchers
SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
A permeable expert search strategy approach to multimodal retrieval
Proceedings of the 4th Information Interaction in Context Symposium
Looking for genre: the use of structural features during search tasks with Wikipedia
Proceedings of the 4th Information Interaction in Context Symposium
What does time spent on searching indicate?
Proceedings of the 4th Information Interaction in Context Symposium
Unobtrusive mobile browsing behaviour tracking tool
Proceedings of the 4th Information Interaction in Context Symposium
SCAMP: a tool for conducting interactive information retrieval experiments
Proceedings of the 4th Information Interaction in Context Symposium
Search tactics as means of examining search processes in collaborative exploratory web search
Proceedings of the 5th Ph.D. workshop on Information and knowledge
WNavis: Navigating Wikipedia semantically with an SNA-based summarization technique
Decision Support Systems
Evaluating semantic search query approaches with expert and casual users
ISWC'12 Proceedings of the 11th international conference on The Semantic Web - Volume Part II
Evaluating web archive search systems
WISE'12 Proceedings of the 13th international conference on Web Information Systems Engineering
Human-Computer interaction view on information retrieval evaluation
PROMISE'12 Proceedings of the 2012 international conference on Information Retrieval Meets Information Visualization
User-Oriented evaluation in IR
PROMISE'12 Proceedings of the 2012 international conference on Information Retrieval Meets Information Visualization
Influence of timeline and named-entity components on user engagement
ECIR'13 Proceedings of the 35th European conference on Advances in Information Retrieval
A pilot study on using profile-based summarisation for interactive search assistance
ECIR'13 Proceedings of the 35th European conference on Advances in Information Retrieval
Exploiting click logs for adaptive intranet navigation
ECIR'13 Proceedings of the 35th European conference on Advances in Information Retrieval
Crowdsourcing and the crisis-affected community
Information Retrieval
How do users' search tactic selections influence search outputs in exploratory search tasks?
Proceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries
How query cost affects search behavior
Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval
Proceedings of the 10th Conference on Open Research Areas in Information Retrieval
Will we be lost without paper maps in the digital age?
Journal of Information Science
Examining the generalizability of the User Engagement Scale (UES) in exploratory search
Information Processing and Management: an International Journal
Modeling behavioral factors ininteractive information retrieval
Proceedings of the 22nd ACM international conference on Conference on information & knowledge management
Supporting exploratory people search: a study of factor transparency and user control
Proceedings of the 22nd ACM international conference on Conference on information & knowledge management
Keyword search and evaluation over relational databases: an outlook to the future
Proceedings of the 7th International Workshop on Ranking in Databases
An entropy-based query expansion approach for learning researchers' dynamic information needs
Knowledge-Based Systems
You have e-mail, what happens next? Tracking the eyes for genre
Information Processing and Management: an International Journal
VideoSkip: event detection in social web videos with an implicit user heuristic
Multimedia Tools and Applications
Hi-index | 0.00 |
This paper provides overview and instruction regarding the evaluation of interactive information retrieval systems with users. The primary goal of this article is to catalog and compile material related to this topic into a single source. This article (1) provides historical background on the development of user-centered approaches to the evaluation of interactive information retrieval systems; (2) describes the major components of interactive information retrieval system evaluation; (3) describes different experimental designs and sampling strategies; (4) presents core instruments and data collection techniques and measures; (5) explains basic data analysis techniques; and (4) reviews and discusses previous studies. This article also discusses validity and reliability issues with respect to both measures and methods, presents background information on research ethics and discusses some ethical issues which are specific to studies of interactive information retrieval (IIR). Finally, this article concludes with a discussion of outstanding challenges and future research directions.