[BUG] .CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmpecd6su1w/main.c' about qwen HOT 3 CLOSED

For the newer versions of Flash Attention v2, the additional rotary pos ops are dependent on the triton library. However, it appears there's an issue with triton compiling the CUDA kernel. Unfortunately, the error messages from this compilation process are not included in the currently provided logs; they should ideally be located above the Python error messages.

As a temporary workaround, I recommend uninstalling Triton. This will cause it to fallback to the non--Flash Attention v2 implementation.

To troubleshoot the issue, the version of triton and related logs will be needed.

from qwen.

For the newer versions of Flash Attention v2, the additional rotary pos ops are dependent on the triton library. However, it appears there's an issue with triton compiling the CUDA kernel. Unfortunately, the error messages from this compilation process are not included in the currently provided logs; they should ideally be located above the Python error messages.

As a temporary workaround, I recommend uninstalling Triton. This will cause it to fallback to the non--Flash Attention v2 implementation.

To troubleshoot the issue, the version of triton and related logs will be needed.

Hi thanks for your prompt reply. I think I have figured out the problem, it is because I am using V100 for finetune and flash-attention does not support V100 currently. After I uninstall V100, I could run the finetune.py as normal.

I have also noticed that V100 does not support training with BF16, do you have a benchmark for FP16? (Coz I only see the comparison between BF16, INT8 and INT4.) I am curious how much performance will be degraded if I finetune the baseline Qwen-7B model using FP16 for full parameters?. Or in this case, it would be preferred to just fine-tune use LORA only? (The magnitude of my dataset is around 100K single conversations)

Thanks for your help in advance!!

from qwen.

bf16 and fp16 should have similar performance (as in speed) on devices where both are supported. if accuracy is concerned, bf16 could enable training that are more stable for larger models but if both can train the model succesfully, the resulted models may not differ significantly.

from qwen.