We present a fine-tuned RoBERTa model for English named entity recognition, achieving high performance on both formal and informal datasets. Our approach uses a simplified version of the CONLL2003 dataset and removes unnecessary prefixes for improved efficiency. The resulting model shows superiority over other models, especially on entities that do not begin with uppercase letters, and can be used for various applications such as email signature detection.
We present a fine-tuned RoBERTa model for English named entity recognition, achieving high performance on both formal and informal datasets. Our approach uses a simplified version of the CONLL2003 dataset and removes unnecessary prefixes for improved efficiency. The resulting model shows superiority over other models, especially on entities that do not begin with uppercase letters, and can be used for various applications such as email signature detection.
[roberta-large-ner-english] is an english NER model that was fine-tuned from roberta-large on conll2003 dataset. Model was validated on emails/chat data and outperformed other models on this type of data specifically. In particular the model seems to work better on entity that don't start with an upper case.
Training data was classified as follow:
Abbreviation | Description |
---|---|
O | Outside of a named entity |
MISC | Miscellaneous entity |
PER | Person’s name |
ORG | Organization |
LOC | Location |
In order to simplify, the prefix B- or I- from original conll2003 was removed. I used the train and test dataset from original conll2003 for training and the "validation" dataset for validation. This resulted in a dataset of size:
Train | Validation |
---|---|
17494 | 3250 |
Model performances computed on conll2003 validation dataset (computed on the tokens predictions)
entity | precision | recall | f1 |
---|---|---|---|
PER | 0.9914 | 0.9927 | 0.9920 |
ORG | 0.9627 | 0.9661 | 0.9644 |
LOC | 0.9795 | 0.9862 | 0.9828 |
MISC | 0.9292 | 0.9262 | 0.9277 |
Overall | 0.9740 | 0.9766 | 0.9753 |
On private dataset (email, chat, informal discussion), computed on word predictions:
entity | precision | recall | f1 |
---|---|---|---|
PER | 0.8823 | 0.9116 | 0.8967 |
ORG | 0.7694 | 0.7292 | 0.7487 |
LOC | 0.8619 | 0.7768 | 0.8171 |
By comparison on the same private dataset, Spacy (en_core_web_trf-3.2.0) was giving:
entity | precision | recall | f1 |
---|---|---|---|
PER | 0.9146 | 0.8287 | 0.8695 |
ORG | 0.7655 | 0.6437 | 0.6993 |
LOC | 0.8727 | 0.6180 | 0.7236 |
For those who could be interested, here is a short article on how I used the results of this model to train a LSTM model for signature detection in emails: https://medium.com/@jean-baptiste.polle/lstm-model-for-email-signature-detection-8e990384fefa