how to use smoothnet for online inference？ about smoothnet HOT 5 CLOSED

Hi @mangguoxia,

Similar to other sliding-window-based filters, SmoothNet is also a near-online method. To modify it to an online inference, you can first take the first T frame as the latency, and you will get the first T smoothed poses. After that, you can input the following frames one by one (input the [1, 2, ..., T, T+1] poses), and obtain the smoothed (T+1)_th pose from the last frame from the output of SmoothNet.

Hope it will be helpful!

from smoothnet.

Hi @mangguoxia,

Similar to other sliding-window-based filters, SmoothNet is also a near-online method. To modify it to an online inference, you can first take the first T frame as the latency, and you will get the first T smoothed poses. After that, you can input the following frames one by one (input the [1, 2, ..., T, T+1] poses), and obtain the smoothed (T+1)_th pose from the last frame from the output of SmoothNet.

Hope it will be helpful!

栓Q so much !

Actually, I used a queue of size T. I copied the first frame T times to fullfill the queue, then put the queue into smoothnet, and obtain the smoothed (T)_th pose from the last frame output by smoothnet.
After that , I delete the first frame in the queue and add a new frame from the video into queue.

But the result is very poor, even I trained the model using my own dataset. (I also tested using pre-train model you provided, the result is similiar.)

I want to know whether there is a problem with my inference method or whether my task (facial landmark detection) is not suitable for smoothnet ?
If input the [1, 2, ..., T, T+1] poses as you said, the length of input will increase gradually , is this correct ?

from smoothnet.

Hi @mangguoxia ,

We have tested SmoothNet on the whole body 3d poses (including face, hand, and body detection), and the results look good.
If input the [1, 2, ..., T, T+1] poses, we always keep the fixed length T as a sliding window.
I'm concerned about the normalization of your data since we need to normalize the data into [-1, 1].

from smoothnet.

Thanks ! I will try to normalize the data into [-0.5, 0.5] (right? I checked the training code for dataloader)

BTW, which pretrained model is suggested for the facial landmark detection ? And the best windowsize ?

from smoothnet.

The performance depends on the movement features and jitter degrees of your data. You can try all these models.
For the offline test, the longer the sliding window length, the smoother the result.

from smoothnet.