Table 5 Comparison of system predictions and human evaluation of 100 new trials and 100 clinical cases

EudraCT

UMLS entities

0.908

0.879

0.893

0.965

0.928

0.946

(± 0.037)

(± 0.049)

(± 0.042)

(± 0.018)

(± 0.029)

(± 0.020)

Temporal entities

0.901

0.899

0.899

0.933

0.933

0.931

(± 0.098)

(± 0.086)

(± 0.085)

(± 0.062)

(± 0.066)

(± 0.049)

Medication information

0.838

0.793

0.803

0.934

0.879

0.893

(± 0.155)

(± 0.158)

(± 0.140)

(± 0.122)

(± 0.161)

(± 0.112)

Negation / speculation

0.785

0.819

0.795

0.892

0.912

0.899

(± 0.153)

(± 0.095)

(± 0.125)

(± 0.126)

(± 0.056)

(± 0.090)

Miscellaneous entities

0.801

0.714

0.754

0.872

0.776

0.820

(± 0.075)

(± 0.073)

(± 0.068)

(± 0.064)

(± 0.061)

(± 0.055)

Experiencer / temporality

0.897

0.854

0.873

0.939

0.908

0.918

(± 0.069)

(± 0.093)

(± 0.074)

(± 0.113)

(± 0.091)

(± 0.093)

Overall

0.876

0.842

0.858

0.942

0.897

0.919

(± 0.029)

(± 0.039)

(± 0.032)

(± 0.020)

(± 0.028)

(± 0.019)

Clinical cases

UMLS entities

0.946

0.929

0.937

0.971

0.949

0. 960

(± 0.027)

(± 0.023)

(± 0.024)

(± 0.020)

(± 0.014)

(± 0.015)

Temporal entities

0.955

0.952

0.953

0.988

0.986

0.987

(± 0.063)

(± 0.056)

(± 0.059)

(± 0.018)

(± 0.016)

(± 0.014)

Medication information

0.948

0.945

0.946

0.989

0.990

0.989

(± 0.905)

(± 0.104)

(± 0.096)

(± 0.033)

(± 0.033)

(± 0.027)

Negation / speculation

0.938

0.968

0.952

0.962

0.992

0.976

(± 0.063)

(± 0.031)

(± 0.042)

(± 0.059)

(± 0.014)

(± 0.034)

Miscellaneous entities

0.757

0.878

0.812

0.796

0.933

0.858

(± 0.085)

(± 0.060)

(± 0.072)

(± 0.068)

(± 0.033)

(± 0.048)

Experiencer / temporality

0.759

0.880

0.811

0.803

0.932

0.859

(± 0.126)

(± 0.110)

(± 0.106)

(± 0.117)

(± 0.094)

(± 0.092)

Overall

0.896

0.924

0. 910

0.926

0.955

0.940

(± 0.023)

(± 0.018)

(± 0.019)

(± 0.018)

(± 0.010)

(± 0.012)