forjiuzhou / spherical-knowledge-distillation Goto Github PK

Hi there @forjiuzhou,

The idea of normalising logits is an interesting one.

However, I am unable to replicate the results using the codes provided in this respo. I have set the teacher network to be ResNet50 with the student network as ResNet18.

The other change I made is to change the Pipeline from reading TFRecords to raw data for ImageNet as discussed here https://github.com/NVIDIA/DALI/blob/eb712f593d98afb87ea56700be7cfd83f512a5f8/docs/examples/use_cases/pytorch/resnet50/main.py

All other hyper-parameters are set to match those in the paper.

In Training, there is a --LN argument in the README.md for training ResNet18, but I cannot find it in the code. Besides, -a should be ResNet18, but not ResNet50. Can u address the issue? @forjiuzhou

Hi there, I've read your interesting paper. and thank you for sharing the codes.
I'm rewriting code to adequate RepDistiller format. and I have a question.

As mentioned in the paper, a Teacher's norm(l_avg) is needed to apply your idea.
then we can make our new logit (f^_i(x) * l_avg, f^_j(x) * l_avg)
but your simplified code doesn't seem to reflect these operations.

Is it replacing this operation by multiplying a constant between 2 and 3?

Please understand that I did not fully understand the paper. :)
thanks.

I find that the setting of learning rate and kD-loss weight in your code is quite special. May I ask if it is consistent with your setting? Then please provide the specific operation parameters of your Resnet18/ Resnet50?

        if epoch < 30:
            args.alpha = 0.9
        elif epoch < 60:
            args.alpha = 0.9
        elif epoch < 80:
            args.alpha = 0.5
        elif epoch < 100:
            args.alpha = 0.1

   factor = epoch // 30
    # factor = epoch // 100
    if epoch >= 80:
        factor = factor + 1
    # if epoch >= 90:
    #     factor = factor + 1
    lr = args.lr * (0.1 ** factor)

Hello, your work, ATKD is very interesting and amazing.
Reducing the Teacher-Student Gap via Adaptive Temperatures.
But I find it in openreview, so maybe you have modify it.
Does you plan to modify it by those reviewers' advice to upload it into arxiv?

I am looking forward to your reply! Thank you.