LASR fails for sequence of a person about lasr HOT 2 CLOSED

Hey Gengshan, I went about addressing the case without symmetry and you were exactly correct! I reinit the CNN weights for the non-root bones when reiniting the bones (code included below) and got the following results:

The code added when reiniting the bones (inside the if statement you pointed to) was:

# Quat weights: (4*n_hypo) per bone
quat_weights = self.model.module.code_predictor.quat_predictor.pred_layer.weight
quat_weights_shape = quat_weights.shape
nfeat = quat_weights_shape[-1] # same for all predictors
quat_weights = torch.cat(
    [
        quat_weights.view(self.opts.n_bones, self.opts.n_hypo*4, nfeat)[:1], # leave root bone estimator unchanged
        torch.randn_like(quat_weights.view(self.opts.n_bones, self.opts.n_hypo*4, nfeat)[1:]) # reinit other weights
    ],
    dim=0
).view(quat_weights_shape)
self.model.module.code_predictor.quat_predictor.pred_layer.weight = torch.nn.Parameter(quat_weights)

# Trans weights: 2 per bone
trans_weights = self.model.module.code_predictor.trans_predictor.pred_layer.weight
trans_weights_shape = trans_weights.shape
trans_weights = torch.cat(
    [
        trans_weights.view(self.opts.n_bones, 2, nfeat)[:1], # leave root bone estimator unchanged
        torch.randn_like(trans_weights.view(self.opts.n_bones, 2, nfeat)[1:]) # reinit other weights
    ],
    dim=0
).view(trans_weights_shape)
self.model.module.code_predictor.trans_predictor.pred_layer.weight = torch.nn.Parameter(trans_weights)

# Depth weights: 1 per bone
depth_weights = self.model.module.code_predictor.depth_predictor.pred_layer.weight
depth_weights_shape = depth_weights.shape
depth_weights = torch.cat(
    [
        depth_weights.view(self.opts.n_bones, 1, nfeat)[:1], # leave root bone estimator unchanged
        torch.randn_like(depth_weights.view(self.opts.n_bones, 1, nfeat)[1:]) # reinit other weights
    ],
    dim=0
).view(depth_weights_shape)
self.model.module.code_predictor.depth_predictor.pred_layer.weight = torch.nn.Parameter(depth_weights)

This seems to address the problem of the mesh being dragged out of screen, although I am implicitly making the assumption that all the root body quaternion predictor weights for every hypothesis come before any other bones. Thanks a lot for the excellent help!

from lasr.

Hi Erik, thanks for sharing the details, which helps a lot for understanding your problem.
Here are my thoughts on the two scenarios (1) symmetry (2) without symmetry:

1. In short, when symmetry is enabled for your video, LASR ended up finding a wrong symmetry plane, which poses a wrong constraint on the shape via the soft-symmetry loss, Eq(11). Specifically, at the first stage, LASR will hypothesize 16 symmetry planes whose normal is evenly distributed on a hemisphere. At the second stage, LASR finds the symmetry plane with the lowest accumulated reconstruction error and use it in the subsequent stages for construction symmetry loss Eq (11).
2. When symmetry is disabled, the problem encountered is different. Based on my experience, the misalignment of GT silhouette and renderings is sometimes caused by the re-initialization of bones here, where the rest bone positions are re-intialized but the CNN predictor for bone transforms Eq(13) is not re-initalized. Sometimes, a large and misaligned bone transformation can drag the whole mesh out of screen.

To solve 1, you may manually select the best symmetry plane by modifying the optim_cam variable here. I don't have a piece of code right now, but I usually save reconstructed meshes corresponding to all hypothesis and visually inspect them. Vertices are from states['mean_v'], faces are from states['faces'] and texture are from states['tex'].

To solve 2, re-initializing the corresponding weights of CNN predictor for the bone transformations would work. It is not implemented. In case you want to implement it, these lines could help. Basically you want to maintain the weights for root body pose (the first set of parameters for quat, trans and depth) and assign random values to weights of the second to last set of parameters.

from lasr.