Elephant: Sequence Labeling for Word and Sentence Segmentation

Kilian Evang, Valerio Basile, Grzegorz Chrupala, Johan Bos

    Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

    21 Citations (Scopus)

    Abstract

    Tokenization is widely regarded as a solved problem due to the high accuracy that rule-based tokenizers achieve. But rule-based tokenizers are hard to maintain and their rules language specific. We show that high-accuracy word and sentence segmentation can be achieved by using supervised sequence labeling on the character level combined with unsupervised feature learning. We evaluated our method on three languages and obtained error rates of 0.27 ‰ (English), 0.35 ‰ (Dutch) and 0.76 ‰ (Italian) for our best models.
    Original languageEnglish
    Title of host publicationProceedings of the 2013 Conference on Empirical Methods in Natural Language Processing
    Place of PublicationSeattle, Washington, USA
    PublisherAssociation for Computational Linguistics (ACL)
    Pages1422-1426
    ISBN (Electronic)978-1-937284-97-8
    Publication statusPublished - 2013
    EventEMNLP 2013: Conference on Empirical Methods in Natural Language Processing - Seattle, United States
    Duration: 18 Oct 201321 Oct 2013

    Conference

    ConferenceEMNLP 2013: Conference on Empirical Methods in Natural Language Processing
    CountryUnited States
    CitySeattle
    Period18/10/1321/10/13

    Fingerprint Dive into the research topics of 'Elephant: Sequence Labeling for Word and Sentence Segmentation'. Together they form a unique fingerprint.

    Cite this