Official Review about qualcommai-micronet-submission-mixnet HOT 10 OPEN

Hi Trevor,
Thanks for the comment, I tried my best to keep the code well organized and reproducible. :)

Your understanding of the code is correct. We scale both the addition and multiplication operations inside the kernel by the reduced precision factor. This is because - All the input activations and weights to the kernel are quantized to 8-bits and hence, all the operations in the kernel (both multiplication and addition) are done in 8-bits. This can be explained by this diagram better (taken from here):

All the matrix operations (even the additions) are performed in low precision before the output is multiplied again by the scaling factor(s).

Thanks for all the hardwork you are putting into reviewing the submissions. Highly appreciate it.

Yash

from qualcommai-micronet-submission-mixnet.

Hi Yash,

Perhaps I'm missing something. AFAICT the paper doesn't mention doing reduced precision accumulation. Given the variable bit-width results, they're likely doing "fake quantization" to evaluate the approach, in which case the additions are full width (32-bits, I assume).

If both the inputs and activations to the kernel are rounded to 8-bit then the multiplications can be counted as being performed in 8-bit. However, after the multiplications the result will be a full 32-bit floating point value, and all accumulation will be performed in 32-bit floating-point.

Trevor

from qualcommai-micronet-submission-mixnet.

Hi Trevor,

Your analysis is correct. The accumulation is not happening in reduced precision, so I will have to calculate the additions in 32 bit. Sorry for the confusion.

I assume that this will change the score of all my submissions. After a rough calculation, the score of this submission increases from 0.181 to 0.31. I will update my flops_counting code and update the scores of the submissions by today midnight. Is that ok?

Thanks,
Yash

from qualcommai-micronet-submission-mixnet.

Hi Yash,

Yes, that sounds good. Thanks for being so responsive!

Trevor

from qualcommai-micronet-submission-mixnet.

Hi Trevor,

I have fixed the bug in flops_counting script in all my submissions. I have also updated the scores in the reports of all my submissions. The updated scores of the submissions are as follows:

If you remember, I made a submission much earlier named "QualcommAI-M0" which is currently on the leaderboard. Since the scores of my later submissions are much better, I would like to scrap the submission QualcommAI-M0. (Honestly, this is because I don't have the bandwidth currently to correct the score of that submission.)

Let me know if you have any more questions. :)
Yash

from qualcommai-micronet-submission-mixnet.

Hi Yash,

Thanks so much for the updates! Everything about this submissions checks out now.

With regard to your M0 entry, if you get a chance to update it in the next few days we'd be happy to review it still (it's very little overhead since we have 3 others from you as well). Just let us know!

Trevor

from qualcommai-micronet-submission-mixnet.

Hi Trevor,

Sure thing, I will update it by today midnight. You can check back on that submission (link) tomorrow morning. :)

from qualcommai-micronet-submission-mixnet.

from qualcommai-micronet-submission-mixnet.

Hi Trevor,
I have updated the QualcommAI-M0 (link) submission with the corrected score. The score for that submission is now 0.5488.

Thanks,
Yash

from qualcommai-micronet-submission-mixnet.

from qualcommai-micronet-submission-mixnet.