xlang-ai / binder Goto Github PK

Hi there,

I was wondering where I can find "binder_program_wikitq_test_exec.json" file, mentioned in the following command.

Thanks a lot.

=======================
"wikitq nsql execution command"
os.system(fr"""{TOKENIZER_FALSE}python {ROOT_DIR}/scripts/execute_binder_program.py --dataset wikitq
--dataset_split test
--qa_retrieve_pool_file templates/qa_retrieve_pool/qa_retrieve_pool.json
--input_program_file binder_program_wikitq_test.json
--output_program_execution_file binder_program_wikitq_test_exec.json
--vote_method simple
""")

I tried to create a proper env using

conda env create -f py3.7binder.yaml
pip install records==0.5.3

however, the conda command fails.

I tried it over Mac OS, Unbuntu ec2 machine, Windows. Could you please verify and update the yaml file?

Hi there,
When I try to run python run.py, I encountered the error /scripts/annotate_binder_program.py.

After I change the third line of run.py to ROOT_DIR = './', the error message becomes:
File ".//scripts/annotate_binder_program.py", line 15, in
from generation.generator import Generator
ModuleNotFoundError: No module named 'generation'

Would you please take a look for me? Thanks.

Thanks for the nice work!

Also are we still expecting updated results rerun on GPT-3.5-turbo? Looking forward to it.

Would it be possible to share the result files (both gpt-3.5 and codex maybe?) for the WTQ dataset (test set)? I'm planning on sampling a few data points for testing and would appreciate having the same subset for a comparison. Thank you.

Hi! It's me again.🤣

When I run the annotate script on wikitq dataset (Here are my cmd and args)

python -u scripts/annotate_binder_program.py --dataset wikitq
--dataset_split test
--prompt_file templates/prompts/wikitq_binder.txt
--n_parallel_prompts 1
--n_processes 2
--max_generation_tokens 512
--temperature 0.4
--sampling_n 20
-v

Error generation error: list index out of range occurred in 76 samples. I took the first 5 samples (wtqid#nu-30, 208, 263, 279, 367) for debug, and found that the input still exceeded the length limit even when n_shot was reduced to 0.

scripts/annotate_binder_program.py

The list few_shot_prompt_list is empty and this step will throw the exception.

generation/generator.py

Will the missing results of these 76 samples have any effect on the execution stage? How should I solve this problem?

Thanks!

@Timothyxxx @BlankCheng Thanks for sharing the code!

I noticed that the default option in the code is to list out all rows for the testing table. However, in the paper, it seems you select the first 3 rows. I wonder what is the setting that corresponds to the results reported in the paper? Thanks!

Hi, when I run the "wikitq nsql annotation command" with following commands,

cd Binder

export TOKENIZERS_PARALLELISM=false

nohup  python scripts/annotate_binder_program.py --dataset wikitq \
 --dataset_split test \
 --prompt_file templates/prompts/wikitq_binder.txt \
 --n_parallel_prompts 1 \
 --max_generation_tokens 512 \
 --temperature 0.4 \
 --sampling_n 20 \
 -v \
> run-wikitq-annotation.out 2>&1 &

tail -fn 5000 run-wikitq-annotation.out

(I didn't run run.py directly, because it reports an error: python: can't open file '/scripts/annotate_binder_program.py': [Errno 2] No such file or directory)

the console outputs the following message:

I located the code that threw the error:
self.records_conn = self.db.get_connection() at line 100, nsql/database.py

I haven't completed the test yet, but I'm wondering if this whole bunch of 'Database' object has no attribute 'get_connection' errors is normal? How should I fix it?