MedCAT model used in validation about medcat HOT 3 CLOSED

The "UMLS full" model is certainly not the one reference in the paper. It has been trained relatively recently.

I wasn't part of the team back then, but the model they would have used would almost certainly have been a "v0.x" model. I don't think these are still publicly available anywhere. Though I could be wrong.

from medcat.

Thank you for the answer, @mart-r,
Ah ok, yea I understand that those early models are not available anymore. But it was nice to see some performance metrics of the model.

Are the available models validated on any annotated corpora? I would be nice to see their metrics.

I am running the models against ShARe/CLEF and MedMentions and get a lower performance than the original model presented in the publication.

ShareClef:

MedMentions:

An extraction was correct if the span overlapped and the CUI was correct, as reported in the publication: " For each manual annotation we check whether it was detected and linked to the correct Unified Medical Language System (UMLS) concept.".

What do you think is the reason for this drop in performance?

from medcat.

The model(s) used in the paper were trained in self-supervised as well as supervised capacity.
However, no supervised training was performed for the models publicly available through the link in the README.
This may not account for all the performance drop, but probably most of it.

The MedMentions model that's used within the tutorials (e.g Part 3.2) did receive some supervised training (on the MedMentions dataset). You can try that one as well.
But it also won't be identical to the models within the paper.

As with most machine learning models, a model is best at the task it's produced to perform. These public models weren't produced to break any performance records. They are there for use case demonstration.

Though in general, for discussion, I recommend the discourse:
https://discourse.cogstack.org/
Not many people monitor the GitHub repo issues.
There's also some discussion about public models and their training in there already.

from medcat.