Autonlu: Detecting, root-causing, and fixing nlu model errors
arXiv preprint arXiv:2110.06384, 2021•arxiv.org
Improving the quality of Natural Language Understanding (NLU) models, and more
specifically, task-oriented semantic parsing models, in production is a cumbersome task. In
this work, we present a system called AutoNLU, which we designed to scale the NLU quality
improvement process. It adds automation to three key steps: detection, attribution, and
correction of model errors, ie, bugs. We detected four times more failed tasks than with
random sampling, finding that even a simple active learning sampling method on an …
specifically, task-oriented semantic parsing models, in production is a cumbersome task. In
this work, we present a system called AutoNLU, which we designed to scale the NLU quality
improvement process. It adds automation to three key steps: detection, attribution, and
correction of model errors, ie, bugs. We detected four times more failed tasks than with
random sampling, finding that even a simple active learning sampling method on an …
Improving the quality of Natural Language Understanding (NLU) models, and more specifically, task-oriented semantic parsing models, in production is a cumbersome task. In this work, we present a system called AutoNLU, which we designed to scale the NLU quality improvement process. It adds automation to three key steps: detection, attribution, and correction of model errors, i.e., bugs. We detected four times more failed tasks than with random sampling, finding that even a simple active learning sampling method on an uncalibrated model is surprisingly effective for this purpose. The AutoNLU tool empowered linguists to fix ten times more semantic parsing bugs than with prior manual processes, auto-correcting 65% of all identified bugs.
arxiv.org
以上显示的是最相近的搜索结果。 查看全部搜索结果