shailja-thakur / codegen-fine-tuning Goto Github PK

According to the paper, there a 2 datasets for finetuning which come from Github and books seperately. But it seems that only Github Corpus is released under your huggingface account. Could you share another one?

Another question is that I' ve noticed that there is another dataset in your huggingface account called CodeGen_RE_data. Could you explain the source of this dataset?

Thanks!

Traceback (most recent call last):
File "/wjl/transformers/examples/pytorch/language-modeling/run_clm.py", line 51, in
from transformers.testing_utils import CaptureLogger
File "/wjl/tutorial-env/lib/python3.10/site-packages/transformers/testing_utils.py", line 39, in
from pytest.doctest import (
ModuleNotFoundError: No module named 'pytest'
Traceback (most recent call last):
File "/wjl/transformers/examples/pytorch/language-modeling/run_clm.py", line 51, in
from transformers.testing_utils import CaptureLogger
File "/wjl/tutorial-env/lib/python3.10/site-packages/transformers/testing_utils.py", line 39, in
from pytest.doctest import (
ModuleNotFoundError: No module named 'pytest'

Hi,

Could you please share the dataset pre-processing script or the path to it, that was used for fine-tuning.

Thanks

Hi,
as per paper It is give as like below for fine tuning." For CodeGen models, the data format has to be in a loose json format with one json per line followed by a new line as follows:"
is there any place we have json file available. we need to have date_sets json file for fine tuning.

Thanks
Kishore

where is run_clm.py?