yds12 / gphelper Goto Github PK

Sometimes token insertion fails.
Error message:

Failed to insert token <name_token> ID=XXX. Error: Document insert conflict

Maybe related to #5, this headline:

Agência de classificação de risco Fitch rebaixa perspectiva da França de estável para negativa arz/soe/dlm/pc/cc

was inserted with the following tokens:

    311,agencia
    308,fitch
    310,risco
    308,fitch
    313,rebaixa
    307,perspectiva
    239,frança
    307,perspectiva
    312,negativa
    309,arzsoedlmpccc

Token "classificação" is one that fell in the case of #5. It seems that token 308 took its place.

During the process of inserting new tokens/news items, some tokens are being inserted multiple times.

If one provider fails to give a response, we should just log the error and go ahead with the other providers.

One file for each provider will not be practical for maintenance. There should be just one provider module/class and all providers should be instantiated from it.

For example, the text for EM is not showing correctly and the website has the tag:

<meta charset="ISO-8859-1" />