NaturalCC is a sequence modeling toolkit that allows researchers and developers to train custom models for many software engineering tasks, e.g., code summarization, code retrieval, code completion, code clone detection and type inference. Our vision is to bridge the gap between programming language and natural language through machine learning techniques.

• [May 10] We have merged the source code of "What Do They Capture? - A Structural Analysis of Pre-Trained Language Models for Source Code" into NaturalCC.

• A collection of code corpus with data preprocessing
• Performance benchmark
• Mixed precision training
  • Nvidia APEX
  • Automatic Mixed Precision
• Multi-GPU training
• Better logging output
• Various Implementations:
  • tensorflow gradient clipping
  • optimizers or learning schedulers
  • baseline models
  • binary data formats

• PyTorch version >= 1.6.0
• Python version >= 3.6
• GCC/G++ > 5.0
• For training new models, you'll also need an NVIDIA GPU and NCCL
• (optional) For faster training, you need to install NVIDIA's apex library.

git clone https://github.com/CGCL-codes/naturalcc && cd naturalcc
pip install -r requirements.txt

Once you installed prerequisite libraries, you can check them via python -m env_test

Export your NaturalCC cache directory (data and models will be saved in this directory) to user variables(~/.bashrc or ~/.zshrc).

echo "export NCC=/data/ncc_data" >> ~/.bashrc

Note: PyCharm cannot get environment variables and, therefore, we recommend you to register your NCC variable at ncc/__init__.py.

Compile Cython files to accelerate programs and register NaturalCC into your pip list

# compile for debug
# python setup.py build_ext --inplace
# install 
pip install --editable ./

NaturalCC supports half precision training.

• If your Pytorch.__version__ < 1.6.0 and nvcc -V is runnable, please install apex.
• Otherwise, use Automatic Mixed Precision (AMP). Available Now (set amp: 1 in yaml file, An example).

Since NCC is build via Cython, your GCC/G++ version should be greater than 4.9. If you have the root permission, update GCC/G++; otherwise, install GCC/G++ with conda.

# install GCC/G++ with conda
conda install -c anaconda gxx_linux-64
conda install -c conda-forge gcc_linux-64
cd ~/anaconda/envs/XXX/bin
ln -s x86_64-conda_cos6-linux-gnu-gcc gcc
ln -s x86_64-conda_cos6-linux-gnu-g++ g++
# check
conda deactivate
conda activate XXX
>> type "gcc/g++ -v" in terminals

Currently, we have processed the following datasets:

• Python (Wan et al.)
• CodeSearchNet (Husain et al.)
• CodeXGlue (Feng et al.)
• Py150 (official processed) (raw)
• OpenCL (Grewe et al.)
• Java (Hu et, al.)
• Stack Overflow
• DeepCS (Gu et al.)
• AVATAR (Ahmad et al.)
• StackOverflow (Iyer et al.)

• NBOW
• BiRNN
• 1D-CNN
• SelfAttn
• Deepcs

• SeqRNN
• GPT2

• static mapping
• decision tree
• deeptune
• inst2vec

• Naive Copy
• CodeNN
• DeepCom
• Seq2Seeq + Attention
• Nary-/ChildSum-Tree2Seq
• Code2Seq
• Transformer + (Sinusoidal/Relative/Learned Position Encoding)
• CodeBERT
• GraphCodeBERT
• PLBART

• ICSE 2022

Dataset: Python (Wan et al.)

Dataset: CodeSearchNet (Husain et al.)

Dataset: Py150 (official processed) (raw)

Dataset: CodeSearchNet-Java (Husain et al.)

Dataset: OpenCL (Grewe et al.)

All the running commands here should be executed in the root of project folder (the path of your naturalcc). For example, in my environment I will stay at /data/wanyao/Dropbox/ghproj-v100/naturalcc.

We also have more detailed READMEs to start your tutorial of NaturalCC.

# ref: dataset/python_wan/README.md
# download dataset
bash dataset/python_wan/download.sh
# clean data
python -m dataset.python_wan.clean
# cast data attributes into different files
python -m dataset.python_wan.attributes_cast

# ref: dataset/python_wan/summarization/README.md
# save code tokens and docstirng tokens into MMAP format
python -m dataset.python_wan.summarization.preprocess

• If you want to create a new model, please add your model at ncc/models and ncc/modules.

• If your training policy are more complex than we thought, you should update your criterions and training procedure at ncc/criterions and ncc/trainers, respectively.
  

  Do not forget to update your self defined module at ncc/XX/__init__.py.

• Select a task and a model from task list and follow the instructions in its README.md to start your learning.

# ref: run/summarization/transformer/README.md
# train
CUDA_VISIBLE_DEVICES=0,1,2,3 nohup python -m run.summarization.transformer.train -f config/python_wan/python > run/summarization/transformer/config/python_wan/python.log 2>&1 &
# inference
CUDA_VISIBLE_DEVICES=0 python -m run.summarization.transformer.eval -f config/python_wan/python -o run/summarization/transformer/config/python_wan/python.txt

Please fell free to contact me if you have any troubles.

NaturalCC is MIT-licensed. The license applies to the pre-trained models as well. This project is also highly inspired by Fairseq and AllenNLP.

Paper
NaturalCC-demo
About us: XCodeMind

Please cite as:

@inproceedings{wan2022naturalcc,
              author    = {Yao Wan and
                           Yang He and
                           Zhangqian Bi and
                           Jianguo Zhang and
                           Yulei Sui and
                           Hongyu Zhang and
                           Kazuma Hashimoto and
                           Hai Jin and
                           Guandong Xu and
                           Caiming Xiong and
                           Philip S. Yu},
              title     = {NaturalCC: An Open-Source Toolkit for Code Intelligence},
              booktitle   = {Proceedings of 44th International Conference on Software Engineering, Companion Volume},
              publisher = ACM,
              year      = {2022}
            }