Last commit date about ghs HOT 6 CLOSED

Aah okay, I think I see what's going on. I think this is a mining issue that stems from how language mining ranges are created, split and iterated. If a repository is updated very frequently like PyTorch then it's constantly moved to the end of the very search results that we are mining. This, coupled with the fact that we take some precautions to avoid over-mining, causes this particular issue, where a very active repository is missed in updates. Thank you for bringing this to my attention, I'll devise a fix for this.

from ghs.

Hi @egor-bogomolov

If I understood correctly, you would like to be able to filter both by the last commit and last update date? In its current state, we only allow the former, but we do include all the information in the export. We can also introduce support for filtering by non-code updates by adding another date range input group. The reason for its initial omission was that the date of the last repository page update is not as interesting to researchers as the date of the last commit. Not only that, but people also typically confuse the two, so we only gave filtering options by the one that was more in demand.

To answer your question regarding updates: we crawl each language by the last repository push date. Operating under the assumption that active projects are regularly pushed to, this implies that new data will eventually be encountered by the crawler if there are pushes to the remote. However, detecting non-code updates (such as changes to issue labels and the repository list of topics) would most likely entail some form of background job in order for our data to maintain complete consistency with GitHub. As of right now, we already make use of two scheduled tasks, one for checking if repositories in our database still exist on GitHub, and another that computes line information through static analysis of the last commit on the default branch. I'll give this some more thought, but as far as this "data maintenance" task is concerned, it will either replace one of the aforementioned jobs, or all three might be merged together into one.

from ghs.

I would like to filter by last commit, but it turns out that I miss quite some repositories this way. For example, PyTorch repo has not been updated for two years, so it does not pass the filter "last commit after 01.01.2023). However, PyTorch repo is regularly updated so I expect it to be found among the results

from ghs.

Thanks a lot! 🚀

from ghs.

I merged the changes just now. Results should take effect after some time.

from ghs.

from ghs.