Elephant: Sequence Labeling for Word and Sentence Segmentation

Kilian Evang; Valerio Basile; Grzegorz Chrupala; Johan Bos

Elephant: Sequence Labeling for Word and Sentence Segmentation

Kilian Evang
, Valerio Basile
, Grzegorz Chrupala
, Johan Bos

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

Abstract

Tokenization is widely regarded as a solved problem due to the high accuracy that rule-based tokenizers achieve. But rule-based tokenizers are hard to maintain and their rules language specific. We show that high-accuracy word and sentence segmentation can be achieved by using supervised sequence labeling on the character level combined with unsupervised feature learning. We evaluated our method on three languages and obtained error rates of 0.27 ‰ (English), 0.35 ‰ (Dutch) and 0.76 ‰ (Italian) for our best models.

Original language	English
Title of host publication	Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing
Place of Publication	Seattle, Washington, USA
Publisher	Association for Computational Linguistics (ACL)
Pages	1422-1426
ISBN (Electronic)	978-1-937284-97-8
Publication status	Published - 2013
Event	EMNLP 2013: Conference on Empirical Methods in Natural Language Processing - Seattle, United States Duration: 18 Oct 2013 → 21 Oct 2013

Conference

Conference	EMNLP 2013: Conference on Empirical Methods in Natural Language Processing
Country/Territory	United States
City	Seattle
Period	18/10/13 → 21/10/13

Access to Document

http://www.aclweb.org/anthology/D13-1146

Elephant: Sequence labeling for word and sentence segmentation
Evang, K., Basile, V., Chrupala, G. & Bos, J., 2013
Research output: Online publication or Non-textual form › Software

Cite this

@inproceedings{20178076b1fe4ed497f245a2ffcc97de,

title = "Elephant: Sequence Labeling for Word and Sentence Segmentation",

abstract = "Tokenization is widely regarded as a solved problem due to the high accuracy that rule-based tokenizers achieve. But rule-based tokenizers are hard to maintain and their rules language specific. We show that high-accuracy word and sentence segmentation can be achieved by using supervised sequence labeling on the character level combined with unsupervised feature learning. We evaluated our method on three languages and obtained error rates of 0.27 ‰ (English), 0.35 ‰ (Dutch) and 0.76 ‰ (Italian) for our best models. ",

author = "Kilian Evang and Valerio Basile and Grzegorz Chrupala and Johan Bos",

year = "2013",

language = "English",

pages = "1422--1426",

booktitle = "Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing",

publisher = "Association for Computational Linguistics (ACL)",

note = "EMNLP 2013: Conference on Empirical Methods in Natural Language Processing ; Conference date: 18-10-2013 Through 21-10-2013",

}

Evang, K, Basile, V, Chrupala, G & Bos, J 2013, Elephant: Sequence Labeling for Word and Sentence Segmentation. in Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics (ACL), Seattle, Washington, USA, pp. 1422-1426, EMNLP 2013: Conference on Empirical Methods in Natural Language Processing, Seattle, United States, 18/10/13. <http://www.aclweb.org/anthology/D13-1146>

Elephant: Sequence Labeling for Word and Sentence Segmentation. / Evang, Kilian; Basile, Valerio; Chrupala, Grzegorz et al.
Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Seattle, Washington, USA: Association for Computational Linguistics (ACL), 2013. p. 1422-1426.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Elephant: Sequence Labeling for Word and Sentence Segmentation

AU - Evang, Kilian

AU - Basile, Valerio

AU - Chrupala, Grzegorz

AU - Bos, Johan

PY - 2013

Y1 - 2013

N2 - Tokenization is widely regarded as a solved problem due to the high accuracy that rule-based tokenizers achieve. But rule-based tokenizers are hard to maintain and their rules language specific. We show that high-accuracy word and sentence segmentation can be achieved by using supervised sequence labeling on the character level combined with unsupervised feature learning. We evaluated our method on three languages and obtained error rates of 0.27 ‰ (English), 0.35 ‰ (Dutch) and 0.76 ‰ (Italian) for our best models.

AB - Tokenization is widely regarded as a solved problem due to the high accuracy that rule-based tokenizers achieve. But rule-based tokenizers are hard to maintain and their rules language specific. We show that high-accuracy word and sentence segmentation can be achieved by using supervised sequence labeling on the character level combined with unsupervised feature learning. We evaluated our method on three languages and obtained error rates of 0.27 ‰ (English), 0.35 ‰ (Dutch) and 0.76 ‰ (Italian) for our best models.

M3 - Conference contribution

SP - 1422

EP - 1426

BT - Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing

PB - Association for Computational Linguistics (ACL)

CY - Seattle, Washington, USA

T2 - EMNLP 2013: Conference on Empirical Methods in Natural Language Processing

Y2 - 18 October 2013 through 21 October 2013

ER -

Elephant: Sequence Labeling for Word and Sentence Segmentation

Abstract

Conference

Access to Document

Fingerprint

Research output

Elephant: Sequence labeling for word and sentence segmentation

Cite this