This project is developed by Cybercore AI for AI City Challenge 2022 Track 3. The project is based on the open source mmaction2 and action-former

Please follow the steps in docs/Env_setup.md. We provide both Docker and Conda Env.

Please follow the steps in docs/Pretrain_backbone.md

Please follow the steps in docs/second_stage_classifier.md

Please follow the steps in docs/Proposal_Generation.md

IN_DIR='path/to/videos'
OUT_DIR="data/raw_frames/full_video/B"
reproduce_scripts/detector/extract_rawframes_fullvideo.sh $IN_DIR $OUT_DIR

where:

• IN_DIR is the directory of testing videos . For A2 test set as an example, set IN_DIR=data/raw_video/A2. For B test set, set IN_DIR=data/raw_video/B.
• The extracted frames for each video is saved at OUT_DIR folder, and procecced in the step 1.2

IN_DIR='path/to/raw_frames/full_video/B'
OUT_DIR="tsp_features/round_b/"
CKPT="http://118.69.233.170:60001/open/AICity/track3/detector/ckpt/round2_tsp_67.5.pth"
./reproduce_scripts/detector/extract_tsp_b.sh  $CKPT $IN_DIR $OUT_DIR

where:

• IN_DIR is the directory of testing raw frames. For A2 test set as an example, set IN_DIR=data/raw_frames/full_video/A2. For B test set, set IN_DIR=data/raw_frames/full_video/B.
• CKPT is our pretrained checkpoint.
• The extracted feature for each video is saved at OUT_DIR folder, and procecced in the step 1.3

Using the feature extracted in step 1.1, run the following command to create proposals:

CONFIG="configs/aicity/actionformer/track3_actionformer_B.yaml"
CKPT="http://118.69.233.170:60001/open/AICity/track3/detector/ckpt/round2_map_31.55.pth.tar"
PROPOSAL_RESULT="proposals.json"

./reproduce_scripts/detector/val_actionformer.sh $CONFIG $CKPT $PROPOSAL_RESULT 

where:

• CONFIG is the model's config file. We assume the extracted feature in Step 1.1. is saved as tsp_features/round_b/, which is specified in the config yaml. If you change the directory or testing video, please adjust the config file accordingly.
• CKPT is the model's checkpoint.
• PROPOSAL_RESULT is the output file, which is used in the step 2.

TEST_VIDEO_DIR=<path/to/test_video>
PROPOSAL_RESULT="proposals.json"
OUTPUT="result_submission.json"
./reproduce_scripts/second_stage_classifier/inference.sh $TEST_VIDEO_DIR $PROPOSAL_RESULT $OUTPUT

For example:

./reproduce_scripts/second_stage_classifier/inference.sh \
    ./data/raw_video/A2 \ 
    proposals.json \
    result_submission.json

After running this script: we have result file: result_submission.json which is used for post-processing in the next step.

INPUT_FILE=result_submission.json
./reproduce_scripts/gen_submit.sh $INPUT_FILE

where:

• INPUT_FILE is the output from step 2.
• This will generate the file submit.txt for submission.
• NOTE: For submission, we need the video_ids file, such as one in tools/detector/video_ids_A2clean.csv. To run inference on B dataset (unknown at this time), we need a similar file. See input of tools/detector/convert_json2txt.py.

We thank mmaction2, action-former, and ConvNeXt code base. Please cite their works should you find our code is helpful.