Automatic feature extraction for question classification based on dissimilarity of probability distributions

  • Authors:
  • David Tomás;José L. Vicedo;Empar Bisbal;Lidia Moreno

  • Affiliations:
  • Departamento de Lenguajes y Sistemas Informáticos, Universidad de Alicante, Spain;Departamento de Lenguajes y Sistemas Informáticos, Universidad de Alicante, Spain;Departamento de Sistemas Informáticos y Computación, Universidad Politécnica de Valencia, Spain;Departamento de Sistemas Informáticos y Computación, Universidad Politécnica de Valencia, Spain

  • Venue:
  • FinTAL'06 Proceedings of the 5th international conference on Advances in Natural Language Processing
  • Year:
  • 2006

Quantified Score

Hi-index 0.00

Visualization

Abstract

Question classification is one of the first tasks carried out in a Question Answering system. In this paper we present a multilingual question classification system based on machine learning techniques. We use Support Vector Machines to classify the questions. All the features needed to train and test this method are automatically extracted through statistical information in an unsupervised way, comparing Poisson distributions of single words in two plain corpora of questions and documents. Thus, we need nothing but plain text to train the system, obtaining a flexible approach easy to adapt to new languages and domains. We have tested it on a bilingual corpus of questions in English and Spanish.