A multilingual named entity recognition system using boosting and c4.5 decision tree learning algorithms

  • Authors:
  • György Szarvas;Richárd Farkas;András Kocsor

  • Affiliations:
  • Department of Informatics, University of Szeged, Szeged, Hungary;Research Group on Artificial Intelligence, MTA-SZTE, Szeged, Hungary;Research Group on Artificial Intelligence, MTA-SZTE, Szeged, Hungary

  • Venue:
  • DS'06 Proceedings of the 9th international conference on Discovery Science
  • Year:
  • 2006

Quantified Score

Hi-index 0.00

Visualization

Abstract

In this paper we introduce a multilingual Named Entity Recognition (NER) system that uses statistical modeling techniques. The system identifies and classifies NEs in the Hungarian and English languages by applying AdaBoostM1 and the C4.5 decision tree learning algorithm. We focused on building as large a feature set as possible, and used a split and recombine technique to fully exploit its potentials. This methodology provided an opportunity to train several independent decision tree classifiers based on different subsets of features and combine their decisions in a majority voting scheme. The corpus made for the CoNLL 2003 conference and a segment of Szeged Corpus was used for training and validation purposes. Both of them consist entirely of newswire articles. Our system remains portable across languages without requiring any major modification and slightly outperforms the best system of CoNLL 2003, and achieved a 94.77% F measure for Hungarian. The real value of our approach lies in its different basis compared to other top performing models for English, which makes our system extremely successful when used in combination with CoNLL modells.