Towards Framework-Independent Evaluation of Deep
Linguistic Parsers
Yusuke Miyao, Kenji Sagae, and Jun'ichi Tsujii
Abstract
This paper describes practical issues in framework-independent evaluation of deep and shallow parsers. We focus on the use of two dependency-based syntactic representation formats in parser evaluation, namely, Carroll et al. (1998)'s Grammatical Relations and de Marneffe et al. (2006)'s Stanford Dependency scheme. Our approach is to convert the output of parsers into these two formats, and measure the accuracy of the resulting converted output. Through the evaluation of an HPSG parser and Penn Treebank phrase structure parsers, we found that mapping between different representation schemes is a non-trivial task that results in lossy conversions that may obscure important differences between different parsing approaches. We discuss sources of disagreements in the representation of syntactic structures in the two dependency-based formats, indicating possible directions for improved framework-independent parser evaluation.
Proceedings of GEAF07; CSLI Publications On-line
Proceedings TOC
Proceedings as a single large pdf file