An empirical validation of software cost estimation models
Communications of the ACM
Calibrating a software cost estimation model: why and how
Information and Software Technology
Software engineering metrics and models
Software engineering metrics and models
Evaluation of current software costing tools
ACM SIGSOFT Software Engineering Notes
Parametric estimation of programming effort: an object-oriented model
Journal of Systems and Software
A Pattern Recognition Approach for Software Engineering Data Analysis
IEEE Transactions on Software Engineering - Special issue on software measurement principles, techniques, and environments
IEEE Transactions on Software Engineering - Special issue on software reliability
Robust regression for developing software estimation models
Journal of Systems and Software
Machine Learning Approaches to Estimating Software Development Effort
IEEE Transactions on Software Engineering
Estimating Software Project Effort Using Analogies
IEEE Transactions on Software Engineering
A Procedure for Analyzing Unbalanced Datasets
IEEE Transactions on Software Engineering
Effort estimation and prediction of object-oriented systems
Journal of Systems and Software
COBRA: a hybrid method for software cost estimation, benchmarking, and risk assessment
Proceedings of the 20th international conference on Software engineering
Explaining the cost of European space and military projects
Proceedings of the 21st international conference on Software engineering
An assessment and comparison of common software cost estimation modeling techniques
Proceedings of the 21st international conference on Software engineering
A Controlled Experiment to Assess the Benefits of Estimating with Analogy and Regression Models
IEEE Transactions on Software Engineering
A replicated assessment and comparison of common software cost modeling techniques
Proceedings of the 22nd international conference on Software engineering
Software Cost Estimation with Incomplete Data
IEEE Transactions on Software Engineering
An Empirical Study of Analogy-based Software Effort Estimation
Empirical Software Engineering
Empirical Software Engineering
A Simulation Tool for Efficient Analogy Based Cost Estimation
Empirical Software Engineering
Experience With the Accuracy of Software Maintenance Task Effort Prediction Models
IEEE Transactions on Software Engineering
A Further Empirical Investigation of the Relationship Between MRE and Project Size
Empirical Software Engineering
Some Misconceptions About Lines of Code
METRICS '97 Proceedings of the 4th International Symposium on Software Metrics
Human Performance Estimating with Analogy and Regression Models: An Empirical Validation
METRICS '98 Proceedings of the 5th International Symposium on Software Metrics
The Predictive Validity Criterion for Evaluating Binary Classifiers
METRICS '98 Proceedings of the 5th International Symposium on Software Metrics
Benchmarking COTS Projects Using Data Envelopment Analysis
METRICS '99 Proceedings of the 6th International Symposium on Software Metrics
Using Simulation to Evaluate Prediction Techniques
METRICS '01 Proceedings of the 7th International Symposium on Software Metrics
Using Public Domain Metrics To Estimate Software Development Effort
METRICS '01 Proceedings of the 7th International Symposium on Software Metrics
Reliability and Validity in Comparative Studies of Software Prediction Models
IEEE Transactions on Software Engineering
A Comparison of Software Project Overruns-Flexible versus Sequential Development Models
IEEE Transactions on Software Engineering
Optimal Project Feature Weights in Analogy-Based Cost Estimation: Improvement and Limitations
IEEE Transactions on Software Engineering
Predicting object-oriented software maintainability using multivariate adaptive regression splines
Journal of Systems and Software
A Systematic Review of Software Development Cost Estimation Studies
IEEE Transactions on Software Engineering
Cross versus Within-Company Cost Estimation Studies: A Systematic Review
IEEE Transactions on Software Engineering
How Long Will It Take to Fix This Bug?
MSR '07 Proceedings of the Fourth International Workshop on Mining Software Repositories
Predictive accuracy comparison of fuzzy models for software development effort of small programs
Journal of Systems and Software
An empirical analysis of software effort estimation with outlier elimination
Proceedings of the 4th international workshop on Predictor models in software engineering
Confidence in software cost estimation results based on MMRE and PRED
Proceedings of the 4th international workshop on Predictor models in software engineering
Improving analogy software effort estimation using fuzzy feature subset selection algorithm
Proceedings of the 4th international workshop on Predictor models in software engineering
A comparative evaluation on the accuracies of software effort estimates from clustered data
Information and Software Technology
Evaluation of preliminary data analysis framework in software cost estimation based on ISBSG R9 Data
Software Quality Control
Software development cost estimation using wavelet neural networks
Journal of Systems and Software
Segmented software cost estimation models based on fuzzy clustering
Journal of Systems and Software
Capture-recapture in software unit testing: a case study
Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement
Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement
Empirical evaluation of analogy-x for software cost estimation
Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement
An over-sampling method for analogy-based software effort estimation
Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement
Predicting Software Metrics at Design Time
PROFES '08 Proceedings of the 9th international conference on Product-Focused Software Process Improvement
Information and Software Technology
A study of project selection and feature weighting for analogy based software cost estimation
Journal of Systems and Software
Expert Systems with Applications: An International Journal
Why comparative effort prediction studies may be invalid
PROMISE '09 Proceedings of the 5th International Conference on Predictor Models in Software Engineering
Conceptual data model-based software size estimation for information systems
ACM Transactions on Software Engineering and Methodology (TOSEM)
A study of the non-linear adjustment for analogy based software cost estimation
Empirical Software Engineering
An effort prediction framework for software defect correction
Information and Software Technology
An application of Bayesian network for predicting object-oriented software maintainability
Information and Software Technology
Visual comparison of software cost estimation models by regression error characteristic analysis
Journal of Systems and Software
Empirical Software Engineering
Information and Software Technology
The relationship between search based software engineering and predictive modeling
Proceedings of the 6th International Conference on Predictive Models in Software Engineering
Modeling the relationship between software effort and size using deming regression
Proceedings of the 6th International Conference on Predictive Models in Software Engineering
Analogy-based software effort estimation using Fuzzy numbers
Journal of Systems and Software
Prediction of business process model quality based on structural metrics
ER'10 Proceedings of the 29th international conference on Conceptual modeling
Comparison of weighted grey relational analysis for software effort estimation
Software Quality Control
Quality assessment of business process models based on thresholds
OTM'10 Proceedings of the 2010 international conference on On the move to meaningful internet systems - Volume Part I
An investigation of using neuro-fuzzy with software size estimation
WOSQ'09 Proceedings of the Seventh ICSE conference on Software quality
A principled evaluation of ensembles of learning machines for software effort estimation
Proceedings of the 7th International Conference on Predictive Models in Software Engineering
Customization support for CBR-based defect prediction
Proceedings of the 7th International Conference on Predictive Models in Software Engineering
ACM SIGSOFT Software Engineering Notes
MCPR'11 Proceedings of the Third Mexican conference on Pattern recognition
Validity and reliability of evaluation procedures in comparative studies of effort prediction models
Empirical Software Engineering
A replicated assessment and comparison of adaptation techniques for analogy-based effort estimation
Empirical Software Engineering
Empirical Software Engineering
Search-based approaches for software development effort estimation
Proceedings of the 12th International Conference on Product Focused Software Development and Process Improvement
Investigating effort prediction of web-based applications using CBR on the ISBSG dataset
EASE'10 Proceedings of the 14th international conference on Evaluation and Assessment in Software Engineering
A systematic review of cross- vs. within- company cost estimation studies
EASE'06 Proceedings of the 10th international conference on Evaluation and Assessment in Software Engineering
Evaluating prediction systems in software project estimation
Information and Software Technology
Evaluating defect prediction approaches: a benchmark and an extensive comparison
Empirical Software Engineering
On using planning poker for estimating user stories
Journal of Systems and Software
StatREC: a graphical user interface tool for visual hypothesis testing of cost prediction models
Proceedings of the 8th International Conference on Predictive Models in Software Engineering
Alternative methods using similarities in software effort estimation
Proceedings of the 8th International Conference on Predictive Models in Software Engineering
Size doesn't matter?: on the value of software size features for effort estimation
Proceedings of the 8th International Conference on Predictive Models in Software Engineering
Empirical Software Engineering
Towards an early software estimation using log-linear regression and a multilayer perceptron model
Journal of Systems and Software
ICONIP'12 Proceedings of the 19th international conference on Neural Information Processing - Volume Part III
Software effort prediction using fuzzy clustering and functional link artificial neural networks
SEMCCO'12 Proceedings of the Third international conference on Swarm, Evolutionary, and Memetic Computing
The impact of parameter tuning on software effort estimation using learning machines
Proceedings of the 9th International Conference on Predictive Models in Software Engineering
Proceedings of the 9th International Conference on Predictive Models in Software Engineering
Building a second opinion: learning cross-company data
Proceedings of the 9th International Conference on Predictive Models in Software Engineering
On the value of outlier elimination on software effort estimation research
Empirical Software Engineering
Information and Software Technology
Software effort estimation as a multiobjective learning problem
ACM Transactions on Software Engineering and Methodology (TOSEM) - Testing, debugging, and error handling, formal methods, lifecycle concerns, evolution and maintenance
A systematic review of systematic review process research in software engineering
Information and Software Technology
LMES: A localized multi-estimator model to estimate software development effort
Engineering Applications of Artificial Intelligence
Finding conclusion stability for selecting the best effort predictor in software effort estimation
Automated Software Engineering
Information and Software Technology
Hi-index | 0.01 |
The Mean Magnitude of Relative Error, MMRE, is probably the most widely used evaluation criterion for assessing the performance of competing software prediction models. One purpose of MMRE is to assist us to select the best model. In this paper, we have performed a simulation study demonstrating that MMRE does not always select the best model. Our findings cast some doubt on the conclusions of any study of competing software prediction models that used MMRE as a basis of model comparison. We therefore recommend not using MMRE to evaluate and compare prediction models. At present, we do not have any universal replacement for MMRE. Meanwhile, we therefore recommend using a combination of theoretical justification of the models that are proposed together with other metrics proposed in this paper.