thepushkarp / nalcos Goto Github PK

Instead of the --look-past flag, the user should be given the choice to enter a date range using --from and --to flags.

Do this before doing #27 #11

Do #28 before this

Currently, the project is using an unauthenticated GH API which is capped to 60 requests per hour from an IP address.

We can add the option to add a user's personal API access token to increase this limit.

Do #28 before this

Original idea from this tweet

We can use some Python Wrapper of the GitHub API such as ghapi or PyGitHub instead of using the requests library.

Additional Reference: https://docs.github.com/en/rest/overview/libraries#python

This can help with #11

AUtomate the testing of the module (preferably with GitHub Actions for CI).

NOTE: Considering the large installation size and time of the Torch and HuggingFace modules, the resources allocated may go over the GH Actions limit. This is something we have to take care of.

Follow up of #13

Currently, we are using multi-qa-MiniLM-L6-cos-v1, which has a speed (sentences encoded/sec on 1 V100 GPU) of 14200 and a model size of 80 MB. We should try out other models to see if we can get better performance and speed out of them.

Additionally, we can also try using other types of tokenizers.

Further reading:

• https://www.sbert.net/docs/pretrained_models.html
• https://huggingface.co/sentence-transformers
• https://huggingface.co/transformers/tokenizer_summary.html

Currently, if the model is not downloaded, the program downloads it during the first run, in the middle of the "Retrieving the commits ..." status mesage.

This can be improved by adding a flag through which the user can download/redownload the model when they need it.

Additionally, the program should prompt the user when it is run without downloading the weights with a choice to download it now or to abort the program.

Write unit tests and add them as pre-commit hook