From this <a href="https://github.com/Niger-Volta-LTI/yoruba-text/commit/a9eb853b43fbb

[FIX] training and test text from Tolúwaṣẹ lang-id task about yoruba-text HOT 1 CLOSED

niger-volta-lti commented on August 29, 2024

[FIX] training and test text from Tolúwaṣẹ lang-id task

from yoruba-text.

Comments (1)

ruohoruotsi commented on August 29, 2024

We did the following analysis on the top level of [Tolúwaṣẹ's repo]:(https://github.com/Toluwase/Word-Level-Language-Identification-for-Resource-Scarce-)

$ cat Yoruba_training_corpus\(part\).txt | wc
    4708  130914  824905
$ cat EngYor_test_corpus.txt | wc
     616   12077   76069
$ cat EngYor_test_corpus.txt Yoruba_training_corpus\(part\).txt | wc
    5324  142991  900974
$ cat EngYor_test_corpus.txt Yoruba_training_corpus\(part\).txt | sort | uniq -d | wc 
     417    5621   37125
$ cat EngYor_test_corpus.txt Yoruba_training_corpus\(part\).txt | sort | uniq | wc 
    3915  129775  811647

it shows us that we only have 417 repeated lines comprising 5621 words out of 130k total words. Given how small this is, I think it is okay for there to be these duplicates, for simplicity's sake we can just combine the text and rename it correctly. Closing.

from yoruba-text.

Related Issues (13)

Recommend Projects