An evaluation of existing and new feature selection metrics in text categorization
2008 23rd International Symposium on Computer and Information Sciences, ISCIS 2008, İstanbul, Türkiye, 27 - 29 Ekim 2008, (Tam Metin Bildiri)
- Yayın Türü: Bildiri / Tam Metin Bildiri
- Doi Numarası: 10.1109/iscis.2008.4717900
- Basıldığı Şehir: İstanbul
- Basıldığı Ülke: Türkiye
- Boğaziçi Üniversitesi Adresli: Evet
Özet
categorization is widely used for organizing and manipulating the documents in the electronic medium. Since the data in text categorization field are high-dimensional, feature selection is crucial to make the task more efficient and precise. In this paper, we make an extensive evaluation of the feature selection metrics used in text categorization by using local and global policies. For the experiments, we use three datasets which vary in size, complexity and skewness. We use SVM as the classifier and tf-idf weighting for term weighting. We observed that almost in all metrics, local policy outperforms when the number of keywords is low and global policy outperforms as the number of keywords increases. In addition to the evaluation of the existing feature selection metrics, we propose new metrics, which have shown high success rates especially in datasets with a low number of keywords. Moreover, we propose a keyword selection policy called Adaptive Keyword Selection (AKS). It is based on selecting different number of keywords for different classes and it improved the performance significantly in skew datasets. © 2008 IEEE.