Models Accuracy¶
Entities Recognition¶
Trained on 80% of dataset, tested on 20% of dataset. Link to download dataset available inside the notebooks. All training sessions stored in session/entities
from IPython.core.display import Image, display
display(Image('ner-accuracy.png', width=500))
Concat¶
precision recall f1-score support
OTHER 0.971261 0.995777 0.983367 5160854
event 0.991603 0.181505 0.306844 143787
law 0.989329 0.879490 0.931181 146950
location 0.848477 0.960757 0.901133 428869
organization 0.960967 0.761301 0.849560 694150
person 0.850705 0.969984 0.906437 507960
quantity 0.996606 0.972120 0.984211 88200
time 0.879509 0.986763 0.930054 179880
accuracy 0.951052 7350650
macro avg 0.936057 0.838462 0.849098 7350650
weighted avg 0.953613 0.951052 0.945046 7350650
Bahdanau¶
precision recall f1-score support
OTHER 0.974847 0.994647 0.984648 5160854
event 0.984159 0.230737 0.373830 143787
law 0.981267 0.869745 0.922146 146950
location 0.790109 0.969399 0.870619 428869
organization 0.950195 0.736809 0.830007 694150
person 0.894418 0.951801 0.922218 507960
quantity 0.873435 0.996122 0.930753 88200
time 0.830533 0.994663 0.905218 179880
accuracy 0.948443 7350650
macro avg 0.909871 0.842990 0.842430 7350650
weighted avg 0.951745 0.948443 0.943289 7350650
Luong¶
precision recall f1-score support
OTHER 0.970056 0.989573 0.979718 5160854
event 0.992172 0.065228 0.122409 143787
law 0.988701 0.878278 0.930224 146950
location 0.818681 0.963229 0.885092 428869
organization 0.890460 0.761138 0.820736 694150
person 0.868039 0.956424 0.910091 507960
quantity 0.859562 0.983662 0.917434 88200
time 0.934577 0.973788 0.953780 179880
accuracy 0.943410 7350650
macro avg 0.915281 0.821415 0.814935 7350650
weighted avg 0.945269 0.943410 0.935230 7350650
BERT-Multilanguage¶
precision recall f1-score support
OTHER 0.897421 0.996914 0.944555 5160854
PAD 0.999999 0.999999 0.999999 7606936
X 0.999640 0.929736 0.963421 5330208
event 0.983410 0.011955 0.023623 143787
law 0.986088 0.880749 0.930446 146950
location 0.842292 0.986024 0.908508 428869
organization 0.966130 0.827488 0.891451 694150
person 0.947691 0.931636 0.939595 507960
quantity 0.979431 0.992290 0.985819 88200
time 0.975794 0.933417 0.954135 179880
accuracy 0.964354 20287794
macro avg 0.957789 0.849021 0.854155 20287794
weighted avg 0.967486 0.964354 0.961233 20287794
BERT-Base¶
precision recall f1-score support
OTHER 0.963803 0.997365 0.980297 5160854
PAD 0.999818 0.999991 0.999905 8134246
X 0.999613 0.995673 0.997639 2744716
event 1.000000 0.001384 0.002764 143787
law 0.908430 0.936774 0.922385 146950
location 0.819429 0.990137 0.896731 428869
organization 0.994678 0.794566 0.883432 694150
person 0.939039 0.967610 0.953110 507960
quantity 0.980706 0.995295 0.987947 88200
time 0.978965 0.991166 0.985028 179880
accuracy 0.981145 18229612
macro avg 0.958448 0.866996 0.860924 18229612
weighted avg 0.982425 0.981145 0.977152 18229612
BERT-Small¶
precision recall f1-score support
OTHER 0.964207 0.998268 0.980942 5160854
PAD 0.999818 1.000000 0.999909 8134246
X 0.999607 0.999790 0.999698 2744716
event 1.000000 0.036109 0.069701 143787
law 0.999070 0.899592 0.946725 146950
location 0.900732 0.988383 0.942524 428869
organization 0.990733 0.865072 0.923648 694150
person 0.951168 0.972149 0.961544 507960
quantity 0.997732 0.987630 0.992656 88200
time 0.967680 0.994196 0.980759 179880
accuracy 0.984762 18229612
macro avg 0.977075 0.874119 0.879811 18229612
weighted avg 0.985340 0.984762 0.981195 18229612
Language Detection¶
Trained on 80% of dataset, tested on 20% of dataset. All training sessions stored in session/language-detection
display(Image('language-detection-accuracy.png', width=500))
XGB¶
precision recall f1-score support
OTHER 0.98 0.99 0.99 9424
eng 1.00 0.99 0.99 9972
ind 1.00 0.99 0.99 11511
zlm 1.00 1.00 1.00 10679
micro avg 0.99 0.99 0.99 41586
macro avg 0.99 0.99 0.99 41586
weighted avg 0.99 0.99 0.99 41586
Multinomial¶
precision recall f1-score support
OTHER 1.00 0.97 0.99 9424
eng 0.99 1.00 0.99 9972
ind 1.00 1.00 1.00 11511
zlm 0.99 1.00 0.99 10679
micro avg 0.99 0.99 0.99 41586
macro avg 0.99 0.99 0.99 41586
weighted avg 0.99 0.99 0.99 41586
SGD¶
precision recall f1-score support
OTHER 0.97 0.99 0.98 9424
eng 0.99 0.99 0.99 9972
ind 1.00 0.99 0.99 11511
zlm 1.00 1.00 1.00 10679
micro avg 0.99 0.99 0.99 41586
macro avg 0.99 0.99 0.99 41586
weighted avg 0.99 0.99 0.99 41586
Deep learning¶
precision recall f1-score support
other 1.00 0.99 0.99 9445
english 1.00 1.00 1.00 9987
indonesian 1.00 1.00 1.00 11518
malay 1.00 1.00 1.00 10636
micro avg 1.00 1.00 1.00 41586
macro avg 1.00 1.00 1.00 41586
weighted avg 1.00 1.00 1.00 41586
POS Recognition¶
Trained on 80% of dataset, tested on 20% of dataset. Link to download dataset available inside the notebooks. All training sessions stored in session/pos
display(Image('pos-accuracy.png', width=500))
Concat¶
precision recall f1-score support
ADJ 0.817403 0.689528 0.748040 45666
ADP 0.946743 0.960281 0.953464 119589
ADV 0.862745 0.790976 0.825303 47760
AUX 0.992753 1.000000 0.996363 10000
CCONJ 0.980811 0.917167 0.947922 37171
DET 0.922089 0.927882 0.924976 38839
NOUN 0.825955 0.933548 0.876462 268329
NUM 0.917225 0.927107 0.922139 41211
PART 0.879739 0.881818 0.880777 5500
PRON 0.967885 0.938671 0.953054 48835
PROPN 0.941781 0.858129 0.898011 227608
PUNCT 0.999666 0.998906 0.999286 182824
SCONJ 0.710932 0.839604 0.769929 15150
SYM 0.989079 0.981111 0.985079 3600
VERB 0.943542 0.908367 0.925620 124518
X 0.000000 0.000000 0.000000 150
accuracy 0.913031 1216750
macro avg 0.856147 0.847068 0.850402 1216750
weighted avg 0.915901 0.913031 0.912964 1216750
Bahdanau¶
precision recall f1-score support
ADJ 0.840047 0.621951 0.714731 45666
ADP 0.964556 0.949134 0.956783 119589
ADV 0.807150 0.835846 0.821247 47760
AUX 0.980583 0.999900 0.990147 10000
CCONJ 0.973852 0.910791 0.941266 37171
DET 0.952105 0.917197 0.934325 38839
NOUN 0.789860 0.935113 0.856371 268329
NUM 0.920680 0.936206 0.928378 41211
PART 0.933212 0.835818 0.881834 5500
PRON 0.977711 0.935968 0.956384 48835
PROPN 0.944440 0.816074 0.875577 227608
PUNCT 0.997880 0.999076 0.998478 182824
SCONJ 0.740312 0.796898 0.767563 15150
SYM 0.999425 0.965556 0.982198 3600
VERB 0.931810 0.917996 0.924851 124518
X 0.000000 0.000000 0.000000 150
accuracy 0.903527 1216750
macro avg 0.859601 0.835845 0.845633 1216750
weighted avg 0.909183 0.903527 0.903409 1216750
Luong¶
precision recall f1-score support
ADJ 0.831224 0.617002 0.708269 45666
ADP 0.968751 0.943582 0.956001 119589
ADV 0.797609 0.804690 0.801134 47760
AUX 0.990094 0.999500 0.994775 10000
CCONJ 0.961630 0.924377 0.942635 37171
DET 0.928473 0.923119 0.925788 38839
NOUN 0.808552 0.923039 0.862010 268329
NUM 0.959908 0.846473 0.899629 41211
PART 0.875569 0.873818 0.874693 5500
PRON 0.954275 0.938896 0.946523 48835
PROPN 0.935033 0.833543 0.881376 227608
PUNCT 0.998556 0.998425 0.998490 182824
SCONJ 0.608801 0.862046 0.713622 15150
SYM 0.976771 0.946111 0.961197 3600
VERB 0.908930 0.926011 0.917391 124518
X 0.000000 0.000000 0.000000 150
accuracy 0.901504 1216750
macro avg 0.844011 0.835039 0.836471 1216750
weighted avg 0.906385 0.901504 0.901539 1216750
BERT-Multilanguage¶
precision recall f1-score support
ADJ 0.841830 0.736237 0.785501 45666
ADP 0.965676 0.948323 0.956921 119589
ADV 0.855381 0.832224 0.843644 47760
AUX 0.994721 0.998700 0.996707 10000
CCONJ 0.946915 0.930483 0.938627 37171
DET 0.914014 0.938747 0.926215 38839
NOUN 0.900014 0.905474 0.902736 268329
NUM 0.925285 0.944796 0.934939 41211
PAD 1.000000 0.999988 0.999994 913117
PART 0.905887 0.892545 0.899167 5500
PRON 0.979693 0.934555 0.956592 48835
PROPN 0.913676 0.937669 0.925517 227608
PUNCT 0.995142 0.998299 0.996718 182824
SCONJ 0.705528 0.843234 0.768259 15150
SYM 1.000000 0.959722 0.979447 3600
VERB 0.941861 0.939350 0.940604 124518
X 0.998458 0.997520 0.997989 498463
accuracy 0.966343 2628180
macro avg 0.928475 0.925757 0.926446 2628180
weighted avg 0.966508 0.966343 0.966292 2628180
BERT-Base¶
precision recall f1-score support
ADJ 0.818816 0.758354 0.787426 45666
ADP 0.965894 0.946065 0.955877 119589
ADV 0.843652 0.823409 0.833408 47760
AUX 0.995014 0.997800 0.996405 10000
CCONJ 0.961230 0.915122 0.937609 37171
DET 0.906354 0.938721 0.922254 38839
NOUN 0.894673 0.906182 0.900391 268329
NUM 0.938875 0.893766 0.915765 41211
PAD 0.997943 0.999998 0.998969 885531
PART 0.911302 0.906000 0.908643 5500
PRON 0.982959 0.933142 0.957403 48835
PROPN 0.909701 0.935477 0.922409 227608
PUNCT 0.997954 0.997790 0.997872 182824
SCONJ 0.635332 0.883300 0.739072 15150
SYM 0.960405 0.896111 0.927145 3600
VERB 0.957721 0.920879 0.938939 124518
X 0.999599 0.998108 0.998853 501714
accuracy 0.964343 2603845
macro avg 0.922201 0.920602 0.919908 2603845
weighted avg 0.965078 0.964343 0.964494 2603845
BERT-Small¶
precision recall f1-score support
ADJ 0.846381 0.716660 0.776137 45666
ADP 0.962181 0.947144 0.954603 119589
ADV 0.851804 0.811265 0.831040 47760
AUX 0.994221 0.997800 0.996007 10000
CCONJ 0.949824 0.922789 0.936112 37171
DET 0.927095 0.935426 0.931242 38839
NOUN 0.898681 0.897626 0.898153 268329
NUM 0.921416 0.928660 0.925024 41211
PAD 0.997934 0.999998 0.998965 885531
PART 0.920861 0.918182 0.919519 5500
PRON 0.968619 0.937975 0.953051 48835
PROPN 0.909923 0.932590 0.921117 227608
PUNCT 0.996126 0.998468 0.997296 182824
SCONJ 0.615736 0.881782 0.725126 15150
SYM 0.973931 0.923611 0.948104 3600
VERB 0.941730 0.938017 0.939870 124518
X 0.999188 0.998324 0.998756 501714
accuracy 0.963971 2603845
macro avg 0.922097 0.922724 0.920595 2603845
weighted avg 0.964521 0.963971 0.963981 2603845
Sentiment Analysis¶
Trained on 80% of dataset, tested on 20% of dataset. All training sessions stored in session/sentiment
Graph based on F1-score.
display(Image('sentiment-accuracy.png', width=500))
Bahdanau¶
precision recall f1-score support
negative 0.79 0.82 0.80 70381
positive 0.79 0.76 0.78 64624
avg / total 0.79 0.79 0.79 135005
Luong¶
precision recall f1-score support
negative 0.79 0.80 0.80 70329
positive 0.78 0.77 0.78 64676
avg / total 0.79 0.79 0.79 135005
Multinomial¶
precision recall f1-score support
negative 0.78 0.84 0.81 70720
positive 0.80 0.74 0.77 64129
micro avg 0.79 0.79 0.79 134849
macro avg 0.79 0.79 0.79 134849
weighted avg 0.79 0.79 0.79 134849
Self-Attention¶
precision recall f1-score support
negative 0.77 0.82 0.80 70708
positive 0.79 0.74 0.76 64297
avg / total 0.78 0.78 0.78 135005
XGB¶
precision recall f1-score support
negative 0.81 0.80 0.81 70356
positive 0.79 0.80 0.79 64493
micro avg 0.80 0.80 0.80 134849
macro avg 0.80 0.80 0.80 134849
weighted avg 0.80 0.80 0.80 134849
BERT-Multilanguage¶
precision recall f1-score support
negative 0.851888 0.746347 0.795633 70691
positive 0.754612 0.857372 0.802716 64314
accuracy 0.799237 135005
macro avg 0.803250 0.801859 0.799175 135005
weighted avg 0.805548 0.799237 0.799007 135005
BERT-Base¶
precision recall f1-score support
negative 0.826599 0.797648 0.811865 70323
positive 0.788071 0.818079 0.802795 64682
accuracy 0.807437 135005
macro avg 0.807335 0.807864 0.807330 135005
weighted avg 0.808140 0.807437 0.807520 135005
BERT-Small¶
precision recall f1-score support
negative 0.835787 0.771196 0.802193 70663
positive 0.768377 0.833592 0.799657 64342
accuracy 0.800933 135005
macro avg 0.802082 0.802394 0.800925 135005
weighted avg 0.803660 0.800933 0.800985 135005
Toxicity Analysis¶
Trained on 80% of dataset, tested on 20% of dataset. All training sessions stored in session/toxic
Labels are,
{0: 'toxic', 1: 'severe_toxic', 2: 'obscene', 3: 'threat', 4: 'insult', 5: 'identity_hate'}
Graph based on F1-score.
display(Image('toxic-accuracy.png', width=500))
Bahdanau¶
precision recall f1-score support
toxic 0.77 0.67 0.72 3650
severe_toxic 0.45 0.40 0.43 395
obscene 0.82 0.65 0.73 1985
threat 0.54 0.30 0.38 142
insult 0.71 0.62 0.66 1856
identity_hate 0.65 0.35 0.45 357
avg / total 0.75 0.62 0.68 8385
Logistic Regression¶
precision recall f1-score support
0 0.98 0.27 0.43 805
1 0.50 0.02 0.04 88
2 0.99 0.30 0.46 460
3 0.00 0.00 0.00 32
4 0.87 0.22 0.35 420
5 0.00 0.00 0.00 68
avg / total 0.88 0.24 0.38 1873
Multinomial¶
precision recall f1-score support
0 0.81 0.52 0.63 805
1 0.44 0.35 0.39 88
2 0.76 0.49 0.59 460
3 0.00 0.00 0.00 32
4 0.68 0.47 0.56 420
5 0.15 0.09 0.11 68
avg / total 0.71 0.47 0.56 1873
Luong¶
precision recall f1-score support
toxic 0.77 0.70 0.74 3678
severe_toxic 0.58 0.14 0.23 430
obscene 0.80 0.66 0.72 2014
threat 0.53 0.21 0.30 127
insult 0.72 0.60 0.65 1905
identity_hate 0.67 0.27 0.38 338
avg / total 0.75 0.62 0.67 8492
Self-Attention¶
precision recall f1-score support
toxic 0.80 0.53 0.64 3806
severe_toxic 0.55 0.17 0.26 417
obscene 0.80 0.55 0.65 2106
threat 0.43 0.02 0.05 122
insult 0.73 0.46 0.56 1989
identity_hate 0.54 0.12 0.20 343
avg / total 0.76 0.48 0.58 8783
BERT-Multilanguage¶
precision recall f1-score support
toxic 0.758527 0.744709 0.751554 3733
severe_toxic 0.514184 0.375648 0.434132 386
obscene 0.768908 0.729811 0.748849 2006
threat 0.742857 0.201550 0.317073 129
insult 0.690093 0.737226 0.712881 1918
identity_hate 0.555172 0.449721 0.496914 358
micro avg 0.728267 0.702227 0.715010 8530
macro avg 0.671623 0.539777 0.576901 8530
weighted avg 0.725752 0.702227 0.710601 8530
samples avg 0.066491 0.066235 0.063811 8530
BERT-Base¶
precision recall f1-score support
toxic 0.772569 0.767225 0.769888 3759
severe_toxic 0.604520 0.256595 0.360269 417
obscene 0.787863 0.717009 0.750768 2046
threat 0.666667 0.380165 0.484211 121
insult 0.683317 0.728919 0.705382 1933
identity_hate 0.569579 0.531722 0.550000 331
micro avg 0.741476 0.707447 0.724062 8607
macro avg 0.680752 0.563606 0.603420 8607
weighted avg 0.738723 0.707447 0.718538 8607
samples avg 0.068567 0.068068 0.065612 8607
BERT-Small¶
precision recall f1-score support
toxic 0.844565 0.676247 0.751092 3688
severe_toxic 0.661538 0.219949 0.330134 391
obscene 0.813200 0.672515 0.736196 2052
threat 0.592593 0.285714 0.385542 112
insult 0.801894 0.580696 0.673600 1896
identity_hate 0.664671 0.331343 0.442231 335
micro avg 0.816442 0.614114 0.700970 8474
macro avg 0.729743 0.461077 0.553133 8474
weighted avg 0.808535 0.614114 0.693682 8474
samples avg 0.061611 0.055738 0.056286 8474
Subjectivity Analysis¶
Trained on 80% of dataset, tested on 20% of dataset. All training sessions stored in session/subjectivity
Graph based on F1-score.
display(Image('subjectivity-accuracy.png', width=500))
Bahdanau¶
precision recall f1-score support
negative 0.90 0.68 0.77 975
positive 0.75 0.93 0.83 1018
avg / total 0.82 0.81 0.80 1993
Luong¶
precision recall f1-score support
negative 0.79 0.84 0.82 998
positive 0.83 0.78 0.80 995
avg / total 0.81 0.81 0.81 1993
Multinomial¶
precision recall f1-score support
negative 0.91 0.85 0.88 999
positive 0.86 0.92 0.89 994
micro avg 0.89 0.89 0.89 1993
macro avg 0.89 0.89 0.89 1993
weighted avg 0.89 0.89 0.89 1993
Self-Attention¶
precision recall f1-score support
negative 0.84 0.70 0.76 1023
positive 0.73 0.86 0.79 970
avg / total 0.79 0.78 0.77 1993
Xgboost¶
precision recall f1-score support
negative 0.86 0.85 0.85 1003
positive 0.85 0.86 0.85 990
micro avg 0.85 0.85 0.85 1993
macro avg 0.85 0.85 0.85 1993
weighted avg 0.85 0.85 0.85 1993
BERT-Multilanguage¶
precision recall f1-score support
negative 0.958380 0.859738 0.906383 991
positive 0.874094 0.963074 0.916429 1002
accuracy 0.911691 1993
macro avg 0.916237 0.911406 0.911406 1993
weighted avg 0.916005 0.911691 0.911434 1993
BERT-Base¶
precision recall f1-score support
negative 0.906977 0.936000 0.921260 1000
positive 0.933403 0.903323 0.918117 993
accuracy 0.919719 1993
macro avg 0.920190 0.919662 0.919688 1993
weighted avg 0.920143 0.919719 0.919694 1993
BERT-Small¶
precision recall f1-score support
negative 0.916185 0.907443 0.911793 1048
positive 0.898429 0.907937 0.903158 945
accuracy 0.907677 1993
macro avg 0.907307 0.907690 0.907475 1993
weighted avg 0.907766 0.907677 0.907699 1993
Emotion Analysis¶
Trained on 80% of dataset, tested on 20% of dataset. All training sessions stored in session/emotion
Graph based on F1-score.
display(Image('emotion-accuracy.png', width=500))
Bahdanau¶
precision recall f1-score support
anger 0.91 0.92 0.92 14943
fear 0.87 0.86 0.87 7630
joy 0.94 0.89 0.92 16570
love 0.94 0.92 0.93 15729
sadness 0.73 0.91 0.81 19849
surprise 0.77 0.47 0.58 9383
avg / total 0.86 0.86 0.85 84104
Luong¶
precision recall f1-score support
anger 0.93 0.90 0.91 14883
fear 0.89 0.83 0.86 7680
joy 0.93 0.89 0.91 16640
love 0.91 0.94 0.92 15621
sadness 0.78 0.79 0.78 19766
surprise 0.66 0.72 0.69 9514
avg / total 0.86 0.85 0.85 84104
Multinomial¶
precision recall f1-score support
anger 0.84 0.83 0.83 14746
fear 0.83 0.44 0.58 7661
joy 0.74 0.87 0.80 16560
love 0.87 0.79 0.83 15829
sadness 0.61 0.86 0.71 19839
surprise 0.77 0.27 0.39 9467
avg / total 0.76 0.74 0.72 84102
Self-attention¶
precision recall f1-score support
anger 0.90 0.90 0.90 14869
fear 0.83 0.85 0.84 7682
joy 0.87 0.90 0.89 16658
love 0.92 0.90 0.91 15767
sadness 0.77 0.74 0.76 19866
surprise 0.64 0.67 0.66 9262
avg / total 0.83 0.83 0.83 84104
Xgboost¶
precision recall f1-score support
anger 0.91 0.90 0.91 14898
fear 0.86 0.84 0.85 7589
joy 0.89 0.91 0.90 16554
love 0.91 0.92 0.91 15694
sadness 0.73 0.73 0.73 19869
surprise 0.57 0.57 0.57 9498
avg / total 0.82 0.82 0.82 84102
BERT-Multilanguage¶
precision recall f1-score support
anger 0.926929 0.921493 0.924203 15005
fear 0.895782 0.857369 0.876155 7579
joy 0.931087 0.915800 0.923380 16627
love 0.959995 0.913981 0.936423 15543
sadness 0.739468 0.955388 0.833674 19860
surprise 0.841078 0.453952 0.589652 9490
accuracy 0.868449 84104
macro avg 0.882390 0.836330 0.847248 84104
weighted avg 0.877101 0.868449 0.862842 84104
BERT-Base¶
precision recall f1-score support
anger 0.947061 0.905025 0.925566 14667
fear 0.875731 0.879734 0.877728 7658
joy 0.920258 0.925339 0.922791 16662
love 0.923129 0.950992 0.936854 15671
sadness 0.776396 0.855665 0.814105 19912
surprise 0.731197 0.579190 0.646377 9534
accuracy 0.866689 84104
macro avg 0.862295 0.849324 0.853903 84104
weighted avg 0.865921 0.866689 0.864726 84104
Bert-Small¶
precision recall f1-score support
anger 0.936102 0.922657 0.929331 14830
fear 0.886564 0.860382 0.873277 7585
joy 0.944103 0.904864 0.924067 16818
love 0.956539 0.919631 0.937722 15628
sadness 0.781855 0.847184 0.813210 19867
surprise 0.679429 0.685154 0.682279 9376
accuracy 0.868615 84104
macro avg 0.864099 0.856645 0.859981 84104
weighted avg 0.871982 0.868615 0.869811 84104
Relevancy¶
Trained on 80% of dataset, tested on 20% of dataset. All training sessions stored in session/relevancy
Graph based on F1-score.
display(Image('relevancy-accuracy.png', width=500))
BERT-Multilanguage¶
precision recall f1-score support
relevant 0.942377 0.817411 0.875457 5422
not relevant 0.732577 0.909152 0.811369 2983
accuracy 0.849970 8405
macro avg 0.837477 0.863281 0.843413 8405
weighted avg 0.867917 0.849970 0.852711 8405
BERT-Base¶
precision recall f1-score support
relevant 0.925108 0.881827 0.902949 5365
not relevant 0.807353 0.874013 0.839362 3040
accuracy 0.879001 8405
macro avg 0.866230 0.877920 0.871155 8405
weighted avg 0.882517 0.879001 0.879950 8405
Similarity¶
Trained on 80% of dataset, tested on 20% of dataset. All training sessions stored in session/similarity
Graph based on F1-score.
display(Image('similarity-accuracy.png', width=500))
BERT-Multilanguage¶
precision recall f1-score support
not similar 0.86 0.86 0.86 50757
similar 0.77 0.76 0.76 30010
avg / total 0.83 0.83 0.83 80767
BERT-Base¶
precision recall f1-score support
not similar 0.838849 0.783363 0.810157 50767
similar 0.670304 0.745333 0.705830 30000
accuracy 0.769237 80767
macro avg 0.754577 0.764348 0.757994 80767
weighted avg 0.776245 0.769237 0.771406 80767
Dependency parsing¶
Trained on 90% of dataset, tested on 10% of dataset. All training sessions stored in session/dependency
display(Image('dependency-accuracy.png', width=500))
Bahdanau¶
precision recall f1-score support
PAD 1.0000 1.0000 1.0000 843055
acl 0.9406 0.9296 0.9351 2983
advcl 0.8924 0.8613 0.8766 1175
advmod 0.9549 0.9482 0.9515 4712
amod 0.9296 0.9100 0.9197 4135
appos 0.9312 0.9570 0.9439 2488
aux 1.0000 1.0000 1.0000 5
case 0.9809 0.9823 0.9816 10557
cc 0.9676 0.9795 0.9735 3170
ccomp 0.8598 0.8045 0.8312 404
compound 0.9201 0.9464 0.9331 6605
compound:plur 0.9597 0.9630 0.9613 594
conj 0.9600 0.9572 0.9586 4158
cop 0.9670 0.9720 0.9695 966
csubj 0.8929 0.8333 0.8621 30
csubj:pass 0.8000 0.6667 0.7273 12
dep 0.8189 0.9259 0.8691 459
det 0.9558 0.9369 0.9463 4041
fixed 0.9337 0.8953 0.9141 535
flat 0.9724 0.9714 0.9719 10479
iobj 0.9286 0.7222 0.8125 18
mark 0.9210 0.9491 0.9349 1376
nmod 0.9355 0.9324 0.9340 3921
nsubj 0.9430 0.9538 0.9484 6345
nsubj:pass 0.9458 0.9053 0.9251 1985
nummod 0.9762 0.9787 0.9775 3854
obj 0.9495 0.9465 0.9480 5162
obl 0.9458 0.9543 0.9500 5599
parataxis 0.9268 0.8283 0.8748 367
punct 0.9978 0.9968 0.9973 16549
root 0.9743 0.9643 0.9693 5037
xcomp 0.8878 0.9039 0.8958 1217
avg / total 0.9953 0.9953 0.9953 951993
precision recall f1-score support
0 1.0000 1.0000 1.0000 843055
1 0.9718 0.9633 0.9675 5037
2 0.9604 0.9459 0.9531 4285
3 0.9474 0.9557 0.9515 4971
4 0.9575 0.9647 0.9611 6594
5 0.9534 0.9665 0.9599 5880
6 0.9648 0.9632 0.9640 6037
7 0.9512 0.9654 0.9582 5548
8 0.9611 0.9623 0.9617 5542
9 0.9729 0.9498 0.9612 4877
10 0.9614 0.9621 0.9617 4559
11 0.9495 0.9588 0.9541 4316
12 0.9547 0.9573 0.9560 3698
13 0.9664 0.9506 0.9584 3600
14 0.9652 0.9590 0.9621 3294
15 0.9619 0.9541 0.9580 3179
16 0.9604 0.9573 0.9589 3117
17 0.9634 0.9587 0.9610 2831
18 0.9406 0.9594 0.9499 2392
19 0.9657 0.9582 0.9619 2176
20 0.9656 0.9615 0.9635 2102
21 0.9523 0.9577 0.9550 1960
22 0.9519 0.9586 0.9552 1859
23 0.9605 0.9555 0.9580 1732
24 0.9649 0.9474 0.9561 1540
25 0.9399 0.9503 0.9451 1349
26 0.9680 0.9333 0.9503 1199
27 0.9246 0.9604 0.9422 1111
28 0.9491 0.9561 0.9526 956
29 0.9578 0.9646 0.9612 989
30 0.9365 0.9513 0.9438 1007
31 0.9483 0.9592 0.9537 784
32 0.9352 0.9545 0.9448 726
33 0.9468 0.9290 0.9378 690
34 0.9575 0.9464 0.9519 690
35 0.9480 0.9231 0.9354 533
36 0.9532 0.9432 0.9481 475
37 0.9511 0.9340 0.9425 500
38 0.9455 0.9139 0.9294 418
39 0.9326 0.9708 0.9513 342
40 0.9361 0.9338 0.9350 408
41 0.9260 0.9602 0.9428 352
42 0.9649 0.9615 0.9632 286
43 0.9418 0.9487 0.9453 273
44 0.9125 0.9389 0.9255 311
45 0.9406 0.9556 0.9480 315
46 0.9703 0.9655 0.9679 203
47 0.9662 0.9542 0.9602 240
48 0.9065 0.9065 0.9065 214
49 0.9455 0.9720 0.9585 214
50 0.9315 0.9189 0.9252 148
51 0.9356 0.9265 0.9310 204
52 0.9257 0.9580 0.9416 143
53 0.9496 0.9231 0.9362 143
54 0.9381 0.9430 0.9406 193
55 0.9837 0.9237 0.9528 131
56 0.8532 0.9688 0.9073 96
57 0.9604 0.9510 0.9557 102
58 0.9633 0.9459 0.9545 111
59 0.9870 0.8837 0.9325 86
60 1.0000 0.9559 0.9774 68
61 0.9429 0.9519 0.9474 104
62 0.9726 0.8875 0.9281 80
63 0.9459 0.9589 0.9524 73
64 0.9385 0.9531 0.9457 64
65 1.0000 0.8833 0.9381 60
66 0.8676 0.9516 0.9077 62
67 0.9020 0.8519 0.8762 54
68 0.9683 0.9242 0.9457 66
69 0.9474 0.9351 0.9412 77
70 0.8406 0.8923 0.8657 65
71 0.9474 0.9818 0.9643 55
72 0.9722 0.9459 0.9589 37
73 0.9796 0.9600 0.9697 50
74 0.9630 0.9630 0.9630 27
75 0.9750 1.0000 0.9873 39
76 0.9655 1.0000 0.9825 28
77 0.9655 0.9333 0.9492 30
78 1.0000 1.0000 1.0000 24
79 0.9677 1.0000 0.9836 30
80 0.9608 0.9074 0.9333 54
81 0.9167 1.0000 0.9565 11
82 0.9074 0.9423 0.9245 52
83 0.9259 1.0000 0.9615 25
84 0.9677 1.0000 0.9836 30
85 1.0000 1.0000 1.0000 14
86 1.0000 0.9412 0.9697 34
87 1.0000 1.0000 1.0000 22
88 1.0000 1.0000 1.0000 8
89 1.0000 1.0000 1.0000 14
90 1.0000 1.0000 1.0000 18
91 0.9677 0.8824 0.9231 34
92 0.8182 1.0000 0.9000 9
93 0.9444 0.9444 0.9444 18
94 1.0000 0.9444 0.9714 18
95 0.9259 0.9615 0.9434 26
96 1.0000 1.0000 1.0000 8
97 1.0000 1.0000 1.0000 2
98 1.0000 1.0000 1.0000 16
99 0.9697 0.8649 0.9143 37
100 1.0000 1.0000 1.0000 2
101 1.0000 1.0000 1.0000 44
102 1.0000 1.0000 1.0000 15
103 0.8889 1.0000 0.9412 8
104 0.8269 0.9773 0.8958 44
105 1.0000 1.0000 1.0000 6
106 1.0000 1.0000 1.0000 7
107 1.0000 1.0000 1.0000 10
108 0.9412 1.0000 0.9697 32
109 1.0000 1.0000 1.0000 13
110 1.0000 1.0000 1.0000 9
111 1.0000 1.0000 1.0000 1
112 1.0000 0.7826 0.8780 23
113 1.0000 1.0000 1.0000 16
114 0.8333 1.0000 0.9091 5
115 1.0000 1.0000 1.0000 1
116 0.9130 0.9545 0.9333 22
117 1.0000 1.0000 1.0000 5
118 0.0000 0.0000 0.0000 0
119 1.0000 1.0000 1.0000 3
120 1.0000 1.0000 1.0000 15
122 1.0000 1.0000 1.0000 8
123 1.0000 1.0000 1.0000 4
125 1.0000 1.0000 1.0000 10
126 1.0000 1.0000 1.0000 2
129 1.0000 1.0000 1.0000 8
133 1.0000 1.0000 1.0000 4
135 1.0000 1.0000 1.0000 3
136 1.0000 1.0000 1.0000 2
139 1.0000 1.0000 1.0000 1
142 1.0000 1.0000 1.0000 2
146 1.0000 1.0000 1.0000 1
151 1.0000 1.0000 1.0000 1
avg / total 0.9951 0.9951 0.9951 951993
Luong¶
precision recall f1-score support
PAD 1.0000 1.0000 1.0000 840905
acl 0.9249 0.9392 0.9320 3094
advcl 0.8952 0.8478 0.8709 1209
advmod 0.9629 0.9475 0.9551 4952
amod 0.9288 0.9246 0.9267 4218
appos 0.9535 0.9204 0.9367 2426
aux 1.0000 1.0000 1.0000 1
case 0.9796 0.9795 0.9796 10991
cc 0.9686 0.9739 0.9713 3298
ccomp 0.8426 0.8501 0.8463 447
compound 0.9170 0.9477 0.9321 6787
compound:plur 0.9428 0.9744 0.9584 626
conj 0.9539 0.9581 0.9560 4251
cop 0.9625 0.9809 0.9716 993
csubj 0.9655 0.8750 0.9180 32
csubj:pass 1.0000 0.9167 0.9565 12
dep 0.8905 0.8320 0.8603 518
det 0.9503 0.9364 0.9433 4088
fixed 0.9113 0.8899 0.9005 554
flat 0.9596 0.9792 0.9693 10272
iobj 1.0000 0.6000 0.7500 15
mark 0.9396 0.9217 0.9305 1417
nmod 0.9086 0.9475 0.9277 4155
nsubj 0.9524 0.9547 0.9535 6483
nsubj:pass 0.9402 0.9108 0.9252 1916
nummod 0.9747 0.9761 0.9754 4022
obj 0.9559 0.9468 0.9513 5337
obl 0.9622 0.9242 0.9428 5727
parataxis 0.8072 0.8910 0.8470 376
punct 0.9972 0.9984 0.9978 16581
root 0.9646 0.9688 0.9667 5037
xcomp 0.9225 0.8364 0.8774 1253
avg / total 0.9950 0.9950 0.9950 951993
precision recall f1-score support
0 1.0000 1.0000 1.0000 840905
1 0.9709 0.9726 0.9717 5037
2 0.9310 0.9534 0.9420 4271
3 0.9543 0.9485 0.9514 5148
4 0.9587 0.9514 0.9551 6220
5 0.9471 0.9631 0.9550 5984
6 0.9593 0.9585 0.9589 5827
7 0.9597 0.9554 0.9576 5789
8 0.9657 0.9527 0.9592 5559
9 0.9548 0.9517 0.9532 5088
10 0.9565 0.9492 0.9528 4427
11 0.9458 0.9631 0.9544 4280
12 0.9584 0.9540 0.9562 3910
13 0.9481 0.9586 0.9533 3791
14 0.9385 0.9563 0.9473 3272
15 0.9577 0.9389 0.9482 3306
16 0.9383 0.9560 0.9471 3023
17 0.9629 0.9417 0.9522 2815
18 0.9384 0.9548 0.9465 2409
19 0.9463 0.9391 0.9427 2103
20 0.9349 0.9617 0.9481 2166
21 0.9712 0.9354 0.9530 2090
22 0.9525 0.9450 0.9487 1763
23 0.9512 0.9512 0.9512 1742
24 0.9624 0.9475 0.9549 1619
25 0.9439 0.9460 0.9449 1333
26 0.9584 0.9333 0.9457 1260
27 0.9443 0.9231 0.9336 1158
28 0.9384 0.9414 0.9399 955
29 0.9313 0.9417 0.9365 1080
30 0.9332 0.9323 0.9327 1004
31 0.9240 0.9404 0.9322 789
32 0.9500 0.9226 0.9361 762
33 0.9292 0.9502 0.9396 843
34 0.9553 0.9468 0.9510 677
35 0.9284 0.9396 0.9339 662
36 0.9238 0.9287 0.9262 561
37 0.9213 0.9152 0.9183 448
38 0.8978 0.9114 0.9045 395
39 0.8991 0.9114 0.9052 440
40 0.9262 0.9446 0.9353 505
41 0.9289 0.9098 0.9193 388
42 0.9544 0.9181 0.9359 342
43 0.9119 0.9308 0.9212 289
44 0.9106 0.9006 0.9056 362
45 0.8525 0.9091 0.8799 286
46 0.9283 0.8859 0.9066 263
47 0.9068 0.8924 0.8995 316
48 0.9282 0.9095 0.9188 199
49 0.9648 0.9202 0.9419 238
50 0.9274 0.9583 0.9426 120
51 0.9167 0.9585 0.9371 241
52 0.9507 0.9415 0.9461 205
53 0.9248 0.9179 0.9213 134
54 0.9200 0.9306 0.9253 173
55 0.9329 0.8910 0.9115 156
56 0.9073 0.8954 0.9013 153
57 0.9304 0.9469 0.9386 113
58 0.9417 0.9576 0.9496 118
59 0.8947 0.8500 0.8718 100
60 0.9770 0.8095 0.8854 105
61 0.8020 0.9576 0.8729 165
62 0.8767 0.8889 0.8828 72
63 0.9355 0.8365 0.8832 104
64 0.8852 0.8308 0.8571 65
65 0.9375 0.8955 0.9160 67
66 0.8690 0.8588 0.8639 85
67 0.9839 0.8472 0.9104 72
68 0.9223 0.9500 0.9360 100
69 0.9367 0.9250 0.9308 80
70 0.8442 0.9701 0.9028 67
71 0.8462 0.8462 0.8462 65
72 0.9200 0.8734 0.8961 79
73 0.8909 0.8596 0.8750 57
74 0.9487 0.8810 0.9136 42
75 0.9296 0.8919 0.9103 74
76 0.8333 0.9677 0.8955 31
77 0.8056 0.9062 0.8529 32
78 0.8750 0.8077 0.8400 26
79 0.7636 0.9333 0.8400 45
80 0.9180 0.8889 0.9032 63
81 0.7188 0.8214 0.7667 28
82 0.8983 0.9298 0.9138 57
83 1.0000 0.8571 0.9231 28
84 0.8605 0.9487 0.9024 39
85 0.9474 0.9474 0.9474 19
86 0.8919 0.9706 0.9296 34
87 0.9231 0.8571 0.8889 14
88 0.9474 0.7826 0.8571 23
89 1.0000 0.8571 0.9231 14
90 0.8929 0.8621 0.8772 29
91 0.8462 0.9429 0.8919 35
92 0.9333 0.7568 0.8358 37
93 0.7895 0.8333 0.8108 18
94 1.0000 0.8000 0.8889 20
95 0.9048 0.9500 0.9268 20
96 0.9412 0.9412 0.9412 17
97 0.9583 1.0000 0.9787 23
98 0.9000 1.0000 0.9474 9
99 1.0000 0.9643 0.9818 28
100 0.8333 1.0000 0.9091 5
101 1.0000 0.9231 0.9600 13
102 1.0000 1.0000 1.0000 13
103 0.8750 1.0000 0.9333 14
104 1.0000 0.9231 0.9600 26
105 1.0000 0.9167 0.9565 12
106 0.9444 0.8500 0.8947 20
107 1.0000 0.8571 0.9231 21
108 1.0000 1.0000 1.0000 20
109 1.0000 1.0000 1.0000 6
110 0.8750 1.0000 0.9333 7
111 1.0000 1.0000 1.0000 4
112 0.9200 0.9583 0.9388 24
113 0.8889 1.0000 0.9412 8
114 1.0000 0.6667 0.8000 3
115 1.0000 1.0000 1.0000 5
116 0.9474 0.8571 0.9000 21
117 0.6667 1.0000 0.8000 2
119 1.0000 1.0000 1.0000 3
120 0.8824 0.9375 0.9091 16
121 1.0000 0.8000 0.8889 5
122 0.8889 1.0000 0.9412 8
123 0.0000 0.0000 0.0000 2
124 1.0000 0.6667 0.8000 3
125 1.0000 1.0000 1.0000 8
126 1.0000 0.8000 0.8889 10
127 1.0000 1.0000 1.0000 3
128 1.0000 1.0000 1.0000 1
129 1.0000 1.0000 1.0000 5
130 1.0000 0.8333 0.9091 12
131 1.0000 1.0000 1.0000 2
132 1.0000 1.0000 1.0000 1
133 1.0000 1.0000 1.0000 9
134 1.0000 1.0000 1.0000 6
136 1.0000 1.0000 1.0000 3
137 1.0000 1.0000 1.0000 10
138 1.0000 1.0000 1.0000 10
140 1.0000 1.0000 1.0000 4
141 1.0000 1.0000 1.0000 2
142 0.4000 0.5000 0.4444 4
144 0.5714 1.0000 0.7273 4
146 0.7500 0.7500 0.7500 4
147 1.0000 0.6000 0.7500 5
149 1.0000 1.0000 1.0000 2
150 0.6667 0.6667 0.6667 3
151 0.5000 0.5000 0.5000 2
152 0.5000 0.5000 0.5000 2
153 1.0000 0.5000 0.6667 2
156 1.0000 1.0000 1.0000 2
158 0.8889 1.0000 0.9412 8
160 0.8000 1.0000 0.8889 4
164 1.0000 1.0000 1.0000 4
avg / total 0.9941 0.9941 0.9941 951993
Concat¶
precision recall f1-score support
PAD 1.0000 1.0000 1.0000 841717
acl 0.9501 0.9110 0.9301 2965
advcl 0.8127 0.8719 0.8413 1249
advmod 0.9423 0.9329 0.9376 4846
amod 0.9141 0.9104 0.9123 4208
appos 0.9282 0.9266 0.9274 2412
case 0.9757 0.9756 0.9756 10896
cc 0.9613 0.9726 0.9669 3171
ccomp 0.8115 0.7094 0.7570 437
compound 0.9176 0.9350 0.9263 6804
compound:plur 0.9172 0.9767 0.9460 601
conj 0.9504 0.9493 0.9498 4119
cop 0.9621 0.9761 0.9690 962
csubj 0.8095 0.7083 0.7556 24
csubj:pass 0.7500 0.6000 0.6667 10
dep 0.8712 0.8333 0.8519 552
det 0.9288 0.9339 0.9313 4082
fixed 0.9229 0.8288 0.8733 549
flat 0.9619 0.9712 0.9666 10328
iobj 0.7273 0.8000 0.7619 10
mark 0.9059 0.9260 0.9159 1487
nmod 0.9159 0.9318 0.9238 4105
nsubj 0.9284 0.9550 0.9415 6316
nsubj:pass 0.9367 0.8999 0.9179 2007
nummod 0.9743 0.9617 0.9680 4024
obj 0.9428 0.9340 0.9384 5184
obl 0.9598 0.9292 0.9442 5776
parataxis 0.8301 0.7537 0.7900 337
punct 0.9957 0.9984 0.9971 16529
root 0.9654 0.9694 0.9674 5037
xcomp 0.8955 0.8575 0.8761 1249
avg / total 0.9943 0.9943 0.9943 951993
precision recall f1-score support
0 1.0000 1.0000 1.0000 841717
1 0.9638 0.9676 0.9657 5037
2 0.9526 0.9295 0.9409 4367
3 0.9410 0.9395 0.9403 4942
4 0.9544 0.9516 0.9530 6440
5 0.9453 0.9514 0.9484 6035
6 0.9376 0.9633 0.9503 6024
7 0.9456 0.9491 0.9473 5398
8 0.9506 0.9438 0.9472 5482
9 0.9488 0.9455 0.9472 4977
10 0.9331 0.9578 0.9453 4430
11 0.9453 0.9468 0.9460 4583
12 0.9364 0.9420 0.9392 3673
13 0.9495 0.9298 0.9395 3719
14 0.9425 0.9343 0.9384 3316
15 0.9460 0.9197 0.9327 3065
16 0.9125 0.9443 0.9281 3071
17 0.9350 0.9228 0.9289 2667
18 0.9377 0.9198 0.9286 2469
19 0.9167 0.9267 0.9217 2197
20 0.9076 0.9286 0.9180 2031
21 0.9355 0.8701 0.9016 1917
22 0.8985 0.8980 0.8983 1834
23 0.9038 0.9011 0.9025 1689
24 0.9066 0.8968 0.9017 1667
25 0.8782 0.9227 0.8999 1320
26 0.8769 0.9204 0.8982 1169
27 0.9041 0.9049 0.9045 1094
28 0.9054 0.8825 0.8938 987
29 0.9352 0.8799 0.9067 1099
30 0.8952 0.9110 0.9031 910
31 0.8745 0.8951 0.8847 810
32 0.8978 0.8772 0.8874 741
33 0.8782 0.9206 0.8989 705
34 0.9467 0.8692 0.9063 818
35 0.8893 0.8745 0.8819 542
36 0.9258 0.8794 0.9020 539
37 0.8603 0.9259 0.8919 459
38 0.9019 0.8458 0.8729 402
39 0.8577 0.9035 0.8800 487
40 0.8374 0.9071 0.8709 420
41 0.9148 0.8496 0.8810 379
42 0.8424 0.9393 0.8882 313
43 0.8852 0.8415 0.8628 284
44 0.9130 0.8571 0.8842 245
45 0.8829 0.9009 0.8918 343
46 0.8036 0.8654 0.8333 208
47 0.8803 0.8834 0.8818 283
48 0.9158 0.7699 0.8365 226
49 0.9074 0.8376 0.8711 234
50 0.7014 0.9136 0.7936 162
51 0.8268 0.9080 0.8655 163
52 0.8539 0.8889 0.8711 171
53 0.9136 0.8457 0.8783 175
54 0.8881 0.8581 0.8729 148
55 0.9073 0.8354 0.8698 164
56 0.8456 0.9200 0.8812 125
57 0.9000 0.8250 0.8609 120
58 0.9027 0.8430 0.8718 121
59 0.7947 0.9231 0.8541 130
60 0.7705 0.7833 0.7769 60
61 0.9315 0.8774 0.9037 155
62 0.8611 0.8493 0.8552 73
63 0.8172 0.9048 0.8588 84
64 0.8571 0.7273 0.7869 66
65 0.9130 0.8750 0.8936 72
66 0.7500 0.9398 0.8342 83
67 0.8409 0.8315 0.8362 89
68 0.9545 0.7590 0.8456 83
69 0.8916 0.8810 0.8862 84
70 0.7727 0.8644 0.8160 59
71 0.8679 0.8846 0.8762 52
72 0.8876 0.8404 0.8634 94
73 0.9298 0.8833 0.9060 60
74 0.9273 0.8226 0.8718 62
75 0.9070 0.8298 0.8667 47
76 0.7885 0.8723 0.8283 47
77 0.8000 0.8780 0.8372 41
78 0.8542 1.0000 0.9213 41
79 0.8696 0.9091 0.8889 44
80 0.9375 0.8571 0.8955 70
81 0.8667 0.7222 0.7879 36
82 0.8514 0.9130 0.8811 69
83 0.9024 0.9250 0.9136 40
84 0.9444 1.0000 0.9714 34
85 0.9189 0.9444 0.9315 36
86 0.8810 0.9487 0.9136 39
87 0.9310 0.8710 0.9000 31
88 0.8857 1.0000 0.9394 31
89 0.9200 0.9200 0.9200 25
90 0.8667 0.8125 0.8387 32
91 0.8519 0.9200 0.8846 25
92 0.8913 0.9535 0.9213 43
93 0.8500 0.9444 0.8947 18
94 0.9231 0.8571 0.8889 28
95 0.7500 0.8571 0.8000 7
96 0.9375 0.7143 0.8108 21
97 0.9688 0.8158 0.8857 38
98 0.9091 0.8696 0.8889 23
99 0.8462 1.0000 0.9167 33
100 1.0000 0.7778 0.8750 9
101 0.9744 0.9744 0.9744 39
102 0.8636 0.8636 0.8636 22
103 0.9677 0.9677 0.9677 31
104 1.0000 1.0000 1.0000 7
105 1.0000 0.6471 0.7857 17
106 0.9600 1.0000 0.9796 24
107 0.9750 1.0000 0.9873 39
108 0.8947 1.0000 0.9444 17
109 1.0000 1.0000 1.0000 14
110 0.9524 1.0000 0.9756 20
111 0.9091 0.8333 0.8696 12
112 0.9259 0.9259 0.9259 27
113 0.8889 1.0000 0.9412 16
114 0.8000 0.9231 0.8571 13
115 0.8235 1.0000 0.9032 14
116 1.0000 0.8095 0.8947 21
117 1.0000 0.8571 0.9231 7
118 0.7692 0.8333 0.8000 12
119 1.0000 1.0000 1.0000 4
120 0.9500 1.0000 0.9744 19
121 1.0000 1.0000 1.0000 7
122 0.8235 0.9333 0.8750 15
123 1.0000 1.0000 1.0000 6
124 1.0000 0.3333 0.5000 3
125 1.0000 0.8889 0.9412 18
126 1.0000 0.9667 0.9831 30
127 0.8750 1.0000 0.9333 7
128 0.8333 0.8333 0.8333 6
129 0.9412 0.9412 0.9412 17
130 0.9333 1.0000 0.9655 14
131 1.0000 1.0000 1.0000 9
132 1.0000 1.0000 1.0000 3
133 1.0000 1.0000 1.0000 11
134 0.9412 1.0000 0.9697 16
135 1.0000 1.0000 1.0000 6
136 1.0000 0.8000 0.8889 10
137 1.0000 0.8000 0.8889 10
138 1.0000 1.0000 1.0000 22
139 0.0000 0.0000 0.0000 1
140 1.0000 1.0000 1.0000 2
141 1.0000 1.0000 1.0000 2
142 1.0000 1.0000 1.0000 4
144 1.0000 1.0000 1.0000 4
146 1.0000 1.0000 1.0000 3
147 0.8889 1.0000 0.9412 8
149 1.0000 1.0000 1.0000 4
150 0.7500 1.0000 0.8571 3
151 1.0000 1.0000 1.0000 2
152 1.0000 1.0000 1.0000 1
153 1.0000 1.0000 1.0000 1
154 1.0000 1.0000 1.0000 2
156 1.0000 0.8333 0.9091 6
157 1.0000 1.0000 1.0000 1
158 1.0000 1.0000 1.0000 5
159 1.0000 1.0000 1.0000 1
160 1.0000 1.0000 1.0000 2
162 1.0000 1.0000 1.0000 3
163 0.0000 0.0000 0.0000 2
164 1.0000 1.0000 1.0000 2
167 0.6667 1.0000 0.8000 4
174 1.0000 1.0000 1.0000 2
176 1.0000 0.7500 0.8571 4
177 1.0000 1.0000 1.0000 2
178 1.0000 1.0000 1.0000 1
179 1.0000 1.0000 1.0000 1
182 1.0000 1.0000 1.0000 4
183 1.0000 1.0000 1.0000 4
avg / total 0.9921 0.9920 0.9920 951993
Attention is all you need¶
precision recall f1-score support
PAD 1.0000 1.0000 1.0000 841796
acl 0.8768 0.8849 0.8809 3016
advcl 0.8290 0.7943 0.8113 1196
advmod 0.9043 0.9163 0.9102 4754
amod 0.9121 0.8773 0.8943 4149
appos 0.8934 0.8983 0.8958 2547
aux 1.0000 1.0000 1.0000 6
case 0.9593 0.9670 0.9631 10888
cc 0.9523 0.9606 0.9564 3198
ccomp 0.7984 0.7385 0.7673 413
compound 0.8677 0.8956 0.8815 6679
compound:plur 0.9073 0.9255 0.9163 550
conj 0.8625 0.9330 0.8964 4162
cop 0.9296 0.9679 0.9484 996
csubj 0.9000 0.4091 0.5625 22
csubj:pass 0.8462 0.8462 0.8462 13
dep 0.8274 0.7377 0.7800 507
det 0.8897 0.9196 0.9044 4094
fixed 0.8851 0.7966 0.8385 580
flat 0.9468 0.9198 0.9331 10333
iobj 1.0000 0.6000 0.7500 20
mark 0.8535 0.8447 0.8491 1359
nmod 0.8749 0.8907 0.8827 4107
nsubj 0.8746 0.8881 0.8813 6471
nsubj:pass 0.8478 0.7116 0.7738 1949
nummod 0.9568 0.9524 0.9546 3884
obj 0.9082 0.8946 0.9013 5274
obl 0.9203 0.8854 0.9025 5740
parataxis 0.7980 0.7980 0.7980 391
punct 0.9933 0.9957 0.9945 16561
root 0.8974 0.9200 0.9085 5037
xcomp 0.8580 0.8593 0.8587 1301
avg / total 0.9906 0.9906 0.9906 951993
precision recall f1-score support
0 1.0000 1.0000 1.0000 841796
1 0.9486 0.9277 0.9381 5037
2 0.9157 0.9547 0.9348 4325
3 0.9505 0.9137 0.9318 4856
4 0.9439 0.9311 0.9374 6309
5 0.9422 0.9396 0.9409 6540
6 0.9314 0.9516 0.9414 5697
7 0.9468 0.9461 0.9464 5414
8 0.9524 0.9394 0.9458 5559
9 0.9432 0.9421 0.9427 5028
10 0.9308 0.9544 0.9425 4300
11 0.9623 0.9323 0.9471 4358
12 0.9449 0.9493 0.9471 3903
13 0.9338 0.9442 0.9390 3497
14 0.9444 0.9475 0.9459 3445
15 0.9445 0.9487 0.9466 3177
16 0.9411 0.9589 0.9500 3068
17 0.9350 0.9589 0.9468 2774
18 0.9527 0.9352 0.9439 2499
19 0.9767 0.9207 0.9478 2319
20 0.9445 0.9558 0.9501 2013
21 0.9321 0.9374 0.9347 2124
22 0.9337 0.9423 0.9380 1749
23 0.9508 0.9175 0.9339 1685
24 0.9608 0.9240 0.9421 1540
25 0.8654 0.9661 0.9130 1358
26 0.9511 0.9245 0.9376 1179
27 0.9416 0.9367 0.9392 1154
28 0.8961 0.9549 0.9245 975
29 0.9260 0.9383 0.9321 1054
30 0.9342 0.9551 0.9445 1025
31 0.9482 0.9146 0.9311 761
32 0.9549 0.9126 0.9333 835
33 0.9235 0.9506 0.9368 749
34 0.9492 0.9465 0.9478 710
35 0.9323 0.9649 0.9483 599
36 0.9750 0.9458 0.9602 535
37 0.9363 0.9620 0.9490 474
38 0.9099 0.9815 0.9443 432
39 0.9462 0.9342 0.9401 395
40 0.9170 0.9535 0.9349 452
41 0.9446 0.9214 0.9328 407
42 0.9452 0.9452 0.9452 292
43 0.9731 0.9031 0.9368 320
44 0.9030 0.9767 0.9384 343
45 0.9343 0.9812 0.9572 319
46 0.9943 0.7955 0.8838 220
47 0.9420 0.9684 0.9550 285
48 0.9160 0.9745 0.9443 235
49 0.9113 0.9893 0.9487 187
50 0.9568 0.8636 0.9078 154
51 0.9706 0.9538 0.9621 173
52 0.9554 0.9934 0.9740 151
53 0.9116 0.9515 0.9311 206
54 0.9008 0.9833 0.9402 120
55 0.9371 0.9371 0.9371 159
56 0.9179 0.9535 0.9354 129
57 0.9091 0.8824 0.8955 102
58 0.9350 0.9127 0.9237 126
59 0.9725 0.7910 0.8724 134
60 0.9576 0.9826 0.9700 115
61 0.9200 0.9485 0.9340 97
62 0.9200 0.9079 0.9139 76
63 0.9551 0.9770 0.9659 87
64 0.9878 0.9310 0.9586 87
65 0.9103 0.9861 0.9467 72
66 0.9474 0.9863 0.9664 73
67 1.0000 0.9667 0.9831 60
68 0.9855 0.8831 0.9315 77
69 0.8889 0.9231 0.9057 52
70 0.9524 1.0000 0.9756 80
71 0.9241 0.9605 0.9419 76
72 0.9870 0.9870 0.9870 77
73 0.9531 1.0000 0.9760 61
74 1.0000 0.9667 0.9831 30
75 0.9412 1.0000 0.9697 64
76 1.0000 0.8571 0.9231 28
77 0.9487 1.0000 0.9737 37
78 0.9677 0.9677 0.9677 31
79 1.0000 1.0000 1.0000 25
80 1.0000 0.9348 0.9663 46
81 1.0000 0.9756 0.9877 41
82 1.0000 0.9302 0.9639 43
83 0.9474 1.0000 0.9730 18
84 0.8846 1.0000 0.9388 23
85 0.9583 1.0000 0.9787 23
86 1.0000 0.8636 0.9268 44
87 1.0000 1.0000 1.0000 10
88 0.9412 0.9412 0.9412 17
89 1.0000 0.8750 0.9333 8
90 0.9167 0.9565 0.9362 23
91 1.0000 1.0000 1.0000 15
92 1.0000 1.0000 1.0000 34
93 0.8571 1.0000 0.9231 6
94 0.9231 1.0000 0.9600 12
95 1.0000 1.0000 1.0000 9
96 1.0000 0.9333 0.9655 15
97 1.0000 1.0000 1.0000 30
98 1.0000 1.0000 1.0000 8
99 1.0000 0.9200 0.9583 25
100 0.8571 1.0000 0.9231 6
101 1.0000 0.9744 0.9870 39
102 1.0000 1.0000 1.0000 7
103 0.8889 1.0000 0.9412 16
104 1.0000 0.9500 0.9744 20
105 1.0000 0.9000 0.9474 10
106 0.9500 1.0000 0.9744 19
107 0.7500 1.0000 0.8571 27
108 1.0000 1.0000 1.0000 15
109 1.0000 1.0000 1.0000 3
110 1.0000 1.0000 1.0000 14
111 1.0000 1.0000 1.0000 9
112 0.9474 1.0000 0.9730 18
113 0.8571 1.0000 0.9231 6
114 1.0000 1.0000 1.0000 10
115 1.0000 1.0000 1.0000 7
116 1.0000 0.9375 0.9677 16
117 1.0000 0.5000 0.6667 2
118 1.0000 1.0000 1.0000 12
119 1.0000 1.0000 1.0000 4
120 1.0000 0.9231 0.9600 13
121 1.0000 1.0000 1.0000 6
122 1.0000 1.0000 1.0000 3
123 1.0000 0.8333 0.9091 6
124 1.0000 1.0000 1.0000 2
125 1.0000 1.0000 1.0000 2
126 0.8846 1.0000 0.9388 23
127 1.0000 1.0000 1.0000 6
128 1.0000 1.0000 1.0000 5
129 1.0000 0.8333 0.9091 6
130 1.0000 1.0000 1.0000 12
131 1.0000 0.7143 0.8333 7
132 1.0000 1.0000 1.0000 2
133 1.0000 1.0000 1.0000 4
134 0.9000 0.9000 0.9000 10
135 0.8571 1.0000 0.9231 6
136 1.0000 1.0000 1.0000 7
137 1.0000 1.0000 1.0000 8
138 1.0000 1.0000 1.0000 12
139 1.0000 1.0000 1.0000 1
140 1.0000 1.0000 1.0000 2
141 1.0000 1.0000 1.0000 2
142 1.0000 1.0000 1.0000 4
144 1.0000 1.0000 1.0000 4
146 1.0000 1.0000 1.0000 3
147 1.0000 1.0000 1.0000 7
149 1.0000 1.0000 1.0000 2
150 1.0000 1.0000 1.0000 2
151 1.0000 1.0000 1.0000 2
152 1.0000 1.0000 1.0000 1
153 1.0000 1.0000 1.0000 1
154 1.0000 1.0000 1.0000 2
156 1.0000 1.0000 1.0000 6
157 1.0000 1.0000 1.0000 1
158 1.0000 1.0000 1.0000 5
159 1.0000 1.0000 1.0000 1
160 1.0000 1.0000 1.0000 2
162 0.6667 0.6667 0.6667 3
163 0.6667 1.0000 0.8000 2
164 1.0000 1.0000 1.0000 2
167 1.0000 0.7500 0.8571 4
174 1.0000 1.0000 1.0000 2
176 1.0000 1.0000 1.0000 4
177 1.0000 1.0000 1.0000 2
178 1.0000 1.0000 1.0000 1
179 1.0000 1.0000 1.0000 1
182 1.0000 1.0000 1.0000 4
183 1.0000 1.0000 1.0000 4
avg / total 0.9933 0.9932 0.9932 951993
CRF¶
precision recall f1-score support
case 0.9584 0.9687 0.9635 11014
obl 0.8045 0.8274 0.8158 5810
flat 0.9469 0.9551 0.9510 10648
cc 0.9538 0.9652 0.9595 3336
conj 0.8684 0.8482 0.8582 4560
punct 0.9848 0.9963 0.9905 17017
nsubj:pass 0.8336 0.7640 0.7973 2059
root 0.7960 0.8453 0.8199 5037
nummod 0.9334 0.9359 0.9347 4088
mark 0.8739 0.8865 0.8802 1392
advcl 0.7649 0.6508 0.7033 1200
advmod 0.8932 0.8924 0.8928 4769
nmod 0.7762 0.7355 0.7553 4215
nsubj 0.8600 0.8835 0.8716 6388
det 0.9020 0.8868 0.8943 4142
compound 0.8776 0.8974 0.8874 6869
amod 0.8677 0.8530 0.8602 4128
obj 0.8749 0.8765 0.8757 5256
acl 0.8375 0.8094 0.8232 3075
xcomp 0.8082 0.8070 0.8076 1264
parataxis 0.7636 0.6208 0.6848 385
appos 0.8221 0.8177 0.8199 2425
cop 0.9350 0.9498 0.9423 1015
fixed 0.8569 0.8056 0.8305 602
ccomp 0.7516 0.5576 0.6402 434
compound:plur 0.9154 0.9498 0.9323 638
dep 0.7820 0.5275 0.6300 510
csubj 0.8750 0.8400 0.8571 25
iobj 0.9375 0.6818 0.7895 22
csubj:pass 1.0000 0.8000 0.8889 5
aux 0.5000 0.2500 0.3333 4
avg / total 0.8953 0.8961 0.8953 112332
precision recall f1-score support
5 0.5452 0.5875 0.5656 5964
2 0.6193 0.7164 0.6643 4365
1 0.8839 0.9031 0.8934 4942
7 0.5181 0.5460 0.5317 5505
9 0.5569 0.5504 0.5536 4804
12 0.5421 0.5309 0.5364 3760
15 0.5556 0.5105 0.5321 3181
4 0.5195 0.6219 0.5661 6241
6 0.5346 0.5571 0.5456 5942
11 0.5350 0.5581 0.5463 4150
14 0.5425 0.5109 0.5262 3251
8 0.5463 0.5414 0.5438 5395
10 0.5705 0.5252 0.5469 4682
13 0.5506 0.5199 0.5348 3537
3 0.5871 0.6077 0.5972 5068
18 0.5613 0.5232 0.5415 2504
20 0.5772 0.5315 0.5534 2109
23 0.6065 0.5814 0.5937 1689
26 0.5820 0.5861 0.5841 1138
29 0.6089 0.5874 0.5980 1047
32 0.6459 0.6241 0.6348 798
35 0.6659 0.5931 0.6274 521
36 0.6312 0.6406 0.6359 537
40 0.6039 0.6620 0.6316 426
17 0.5513 0.5303 0.5406 2674
22 0.5889 0.5238 0.5545 1827
25 0.5898 0.5967 0.5932 1381
27 0.5802 0.5588 0.5693 1088
28 0.6101 0.6082 0.6092 970
34 0.6011 0.6029 0.6020 695
39 0.6711 0.5884 0.6270 430
37 0.6675 0.5876 0.6250 468
42 0.6975 0.6323 0.6633 310
43 0.6504 0.6584 0.6544 243
44 0.7205 0.6221 0.6677 344
47 0.6667 0.7077 0.6866 260
49 0.6903 0.7290 0.7091 214
51 0.6829 0.7368 0.7089 190
53 0.7483 0.6730 0.7086 159
55 0.7143 0.6936 0.7038 173
57 0.7093 0.6224 0.6630 98
59 0.7652 0.6779 0.7189 149
60 0.7253 0.7174 0.7213 92
61 0.7658 0.7516 0.7586 161
62 0.6500 0.5571 0.6000 70
63 0.7257 0.7736 0.7489 106
64 0.8730 0.7971 0.8333 69
65 0.8533 0.6667 0.7485 96
66 0.7097 0.8354 0.7674 79
67 0.5965 0.6415 0.6182 53
72 0.9362 0.6769 0.7857 65
70 0.9024 0.5968 0.7184 62
75 0.9348 0.7414 0.8269 58
77 0.7838 0.8286 0.8056 35
78 0.8750 0.8750 0.8750 16
80 0.7200 0.8000 0.7579 45
82 0.7027 0.7222 0.7123 36
83 0.6923 0.7200 0.7059 25
84 0.7407 0.5000 0.5970 40
85 0.6923 0.8571 0.7660 21
86 0.9091 0.6061 0.7273 33
87 0.5833 0.5000 0.5385 28
88 0.8333 0.4412 0.5769 34
89 0.7619 0.9412 0.8421 17
90 0.9143 0.7805 0.8421 41
91 0.6923 0.8182 0.7500 22
92 1.0000 0.8519 0.9200 27
93 1.0000 0.7273 0.8421 22
94 0.9333 0.6364 0.7568 22
95 1.0000 0.6250 0.7692 16
96 0.8000 0.6857 0.7385 35
97 0.9500 0.8261 0.8837 23
100 1.0000 0.6667 0.8000 6
103 1.0000 0.7857 0.8800 14
104 1.0000 0.6000 0.7500 15
101 0.9574 0.8491 0.9000 53
107 0.8846 0.8214 0.8519 28
112 0.8000 0.8000 0.8000 10
115 1.0000 0.5556 0.7143 9
120 0.6667 0.6667 0.6667 6
122 0.5556 0.7143 0.6250 7
124 1.0000 0.2857 0.4444 7
125 0.2857 0.4000 0.3333 5
126 0.5455 0.3529 0.4286 17
127 0.6667 1.0000 0.8000 4
128 1.0000 0.3500 0.5185 20
129 0.7000 0.7778 0.7368 9
130 0.8667 0.9286 0.8966 14
132 1.0000 0.7143 0.8333 7
133 0.5714 1.0000 0.7273 4
134 1.0000 1.0000 1.0000 2
138 0.9091 0.7692 0.8333 13
147 1.0000 0.5789 0.7333 19
149 0.6667 1.0000 0.8000 2
150 0.6667 1.0000 0.8000 2
21 0.5698 0.5552 0.5624 1940
24 0.5665 0.5503 0.5583 1501
30 0.6101 0.5757 0.5924 905
19 0.5310 0.5163 0.5236 2142
16 0.5312 0.5369 0.5340 2917
38 0.6839 0.6005 0.6395 418
33 0.6255 0.5951 0.6100 741
41 0.6913 0.6341 0.6614 399
0 0.4083 0.0926 0.1510 529
31 0.5924 0.5625 0.5771 752
48 0.6432 0.6010 0.6214 198
50 0.7320 0.6222 0.6727 180
52 0.6685 0.6538 0.6611 182
54 0.7024 0.6705 0.6860 176
68 0.7791 0.6262 0.6943 107
79 0.9020 0.8214 0.8598 56
46 0.8037 0.6187 0.6992 278
56 0.7721 0.7095 0.7394 148
98 0.8000 0.5926 0.6809 27
45 0.6513 0.6804 0.6655 291
73 0.8261 0.7451 0.7835 51
105 0.8571 0.7500 0.8000 8
108 0.9091 0.8333 0.8696 12
110 0.8462 0.7857 0.8148 14
114 0.7778 0.4375 0.5600 16
123 0.7500 0.5000 0.6000 6
135 1.0000 0.5625 0.7200 16
139 0.0000 0.0000 0.0000 1
142 1.0000 0.7500 0.8571 4
146 1.0000 1.0000 1.0000 3
151 1.0000 0.5000 0.6667 2
76 0.8400 0.7000 0.7636 30
58 0.6838 0.7207 0.7018 111
69 0.6824 0.7838 0.7296 74
74 0.8605 0.8043 0.8315 46
71 0.8077 0.7778 0.7925 81
109 0.8889 0.7273 0.8000 11
99 0.8889 0.6667 0.7619 12
117 1.0000 0.1429 0.2500 7
116 0.6000 0.6667 0.6316 9
113 0.5833 0.2917 0.3889 24
121 0.7500 0.5000 0.6000 6
131 0.8333 1.0000 0.9091 5
137 1.0000 0.7500 0.8571 4
81 0.9375 0.6522 0.7692 46
118 0.5000 0.5000 0.5000 6
111 0.6000 0.6000 0.6000 5
102 1.0000 0.7143 0.8333 7
106 1.0000 0.7727 0.8718 22
136 0.7778 0.2800 0.4118 25
140 1.0000 0.5000 0.6667 2
141 0.0000 0.0000 0.0000 5
144 1.0000 0.3000 0.4615 10
152 0.0000 0.0000 0.0000 1
153 0.0000 0.0000 0.0000 1
158 1.0000 0.6250 0.7692 8
156 0.9091 0.4762 0.6250 21
160 1.0000 1.0000 1.0000 2
164 1.0000 1.0000 1.0000 2
143 0.0000 0.0000 0.0000 0
155 0.0000 0.0000 0.0000 0
157 0.5000 0.2500 0.3333 4
161 0.0000 0.0000 0.0000 0
162 1.0000 0.2500 0.4000 12
166 0.0000 0.0000 0.0000 0
175 0.0000 0.0000 0.0000 0
173 0.0000 0.0000 0.0000 0
176 1.0000 1.0000 1.0000 16
177 1.0000 1.0000 1.0000 8
178 1.0000 1.0000 1.0000 4
181 0.0000 0.0000 0.0000 0
182 1.0000 1.0000 1.0000 16
119 1.0000 0.7143 0.8333 21
148 0.0000 0.0000 0.0000 0
154 1.0000 0.7500 0.8571 8
159 1.0000 0.2500 0.4000 4
163 0.0000 0.0000 0.0000 8
167 1.0000 1.0000 1.0000 16
174 1.0000 1.0000 1.0000 8
179 1.0000 1.0000 1.0000 4
183 1.0000 1.0000 1.0000 16
145 0.0000 0.0000 0.0000 0
avg / total 0.5859 0.5847 0.5836 109699