biomedicalinformaticsgroup / cadmus Goto Github PK

I generated the API token for NCBI and CrossRef, then I tried running the colab notebook, but I was getting HTTP Error 400: Bad Request. Is there something wrong with my search query,

bioscraping(
    '"proteins"[MeSH Terms] OR "proteins"[All Fields] OR "protein"[All Fields]',
    email,
    NCBI_API_KEY,
    CROSSREF_API_KEY
)

Since Crossref is retired, we need to use Wiley and Elsevier API directly instead

There are libraries that will allow us to read and write within compression files. It might considerably reduce the size of cadmus output.

It seems the parser is not extracting the text for Elsevier XML correctly. More work is needed to better parse it.

The retrieval system was built for moderate-sized corpora (<10k) initially. As corpora have scaled, there is an issue with the speed of reading and writing large data frames into pickle objects. The system becomes very slow and can freeze during these steps. To overcome this, we propose to move to simple text files for storing data for each record. I've created a new branch called "new_output" for working on transfer from pickled output to structured directories of text files for the raw files, retrieved text and metadata.

It makes the installation of cadmus more practical

Shared servers seem to have an issue with the tika module.