Using Decision Trees for Agent Modeling: Improving Prediction Performance

• Authors: Bark Cheung Chiu;Geoffrey I. Webb
• Venue: User Modeling and User-Adapted Interaction
• Year: 1998

A modeling system may be required topredict an agent‘s future actions under constraints ofinadequate or contradictory relevant historicalevidence. This can result in low predictionaccuracy, or otherwise, low prediction rates, leavinga set of cases for which no predictions are made. Aprevious study that explored techniques for improvingprediction rates in the context of modeling students‘subtraction skills using Feature Based Modeling showeda tradeoff between prediction rate and predicationaccuracy. This paper presents research that aims toimprove prediction rates without affecting predictionaccuracy. The FBM-C4.5 agent modeling system was usedin this research. However, the techniques exploredare applicable to any Feature Based Modeling system,and the most effective technique developed isapplicable to most agent modeling systems. Thedefault FBM-C4.5 system models agents‘ competencieswith a set of decision trees, trained on allhistorical data. Each tree predicts one particularaspect of the agent‘s action. Predictions frommultiple trees are compared for consensus. FBM-C4.5makes no prediction when predictions from differenttrees contradict one another. This strategy tradesoff reduced prediction rates for increased accuracy. To make predictions in the absence of consensus, threetechniques have been evaluated. They include usingvoting, using a tree quality measure and using a leafquality measure. An alternative technique that mergesmultiple decision trees into a single tree provides anadvantage of producing models that are morecomprehensible. However, all of these techniquesdemonstrated the previous encountered trade-offbetween rate of prediction and accuracy of prediction,albeit less pronounced. It was hypothesized thatmodels built on more current observations wouldoutperform models built on earlier observations. Experimental results support this hypothesis. ADual-model system, which takes this temporal factorinto account, has been evaluated. This fifth approachachieved a significant improvement in prediction ratewithout significantly affecting prediction accuracy.