Source code of our EMNLP 2018 paper: SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-DomainText-to-SQL Task .
👍 03/20/2022: We open-sourced a simple but SOTA model (just T5) for 20 tasks including text-to-SQL! Please check out our code in the UnifiedSKG repo!!
@InProceedings{Yu&al.18.emnlp.syntax,
author = {Tao Yu and Michihiro Yasunaga and Kai Yang and Rui Zhang and Dongxu Wang and Zifan Li and Dragomir Radev},
title = {SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-Domain Text-to-SQL Task},
year = {2018},
booktitle = {Proceedings of EMNLP},
publisher = {Association for Computational Linguistics},
}
- The code uses Python 2.7 and Pytorch 0.2.0 GPU.
- Install Python dependency:
pip install -r requirements.txt
- Download the dataset from the Spider task website to be updated, and put
tables.json,train.json, anddev.jsonunderdata/directory. - Download the pretrained Glove, and put it as
glove/glove.%dB.%dd.txt - Download
evaluation.pyandprocess_sql.pyfrom the Spider github page - Download preprocessed train/dev datasets and pretrained models from here. It contains:
-
generated_datasets/generated_datafor original Spider training datasets, pretrained models can be found atgenerated_data/saved_modelsgenerated_data_augmentfor original Spider + augmented training datasets, pretrained models can be found atgenerated_data_augment/saved_models
You could find preprocessed train/dev data in generated_datasets/.
To generate them by yourself, update dirs under TODO in preprocess_train_dev_data.py, and run the following command to generate training files for each module:
python preprocess_train_dev_data.py train|dev
data/contains raw train/dev/test data and table filegenerated_datasets/described as abovemodels/contains the code for each module.evaluation.pyis for evaluation. It usesprocess_sql.py.train.pyis the main file for training. Usetrain_all.shto train all the modules (see below).test.pyis the main file for testing. It usessupermodel.shto call the trained modules and generate SQL queries. In practice, and usetest_gen.shto generate SQL queries.generate_wikisql_augment.pyfor cross-domain data augmentation
Run train_all.sh to train all the modules.
It looks like:
python train.py \
--data_root path/to/generated_data \
--save_dir path/to/save/trained/module \
--history_type full|no \
--table_type std|no \
--train_component <module_name> \
--epoch <num_of_epochs>
Run test_gen.sh to generate SQL queries.
test_gen.sh looks like:
SAVE_PATH=generated_datasets/generated_data/saved_models_hs=full_tbl=std
python test.py \
--test_data_path path/to/raw/test/data \
--models path/to/trained/module \
--output_path path/to/print/generated/SQL \
--history_type full|no \
--table_type std|no \
Follow the general evaluation process in the Spider github page.
You could find preprocessed augmented data at generated_datasets/generated_data_augment.
If you would like to run data augmentation by yourself, first download wikisql_tables.json and train_patterns.json from here, and then run python generate_wikisql_augment.py to generate more training data. Second, run get_data_wikisql.py to generate WikiSQL augment json file. Finally, use merge_jsons.py to generate the final spider + wikisql + wikisql augment dataset.
The implementation is based on SQLNet. Please cite it too if you use this code.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent