mishig25 / 3d-posenet Goto Github PK

A javascript error will occur when opening index.html.
Please tell me how to do it.

Great Job! Thanks for sharing it!
I am running PoseNet on PC and saw that whatever people pose, the model can't detect the legs of the person, does this is something that is happening only to me, or is it a general problem. If so is there any way I can improve the accuracy of PoseNet?
Running on PC: Micro-Star (Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz 2.20 GHz)
Also why it's so unstable, I mean I stay in one place and points keep changing places
Can it work better on black and white frames?
Should I have a dark background image?

The only thing that is being updated on the model is the rotation of the head, the shoulders and arms are not moving as they are in the example video. It looks as if that part of the code might have been left out. Any way to get that added back or updated?

Hi,

Thanks for the awesome code! I found a few small problems.

On https://github.com/mishig25/3d-posenet/blob/master/posenet.js, line 132 (ish) I think you should be including something like:
`
// hip, shoulder, elbow

// Replace keypoint indices with joint names if needed

const angleRight = self.transform.rotateJoint(24, 12, 14);

const angleLeft = self.transform.rotateJoint(23, 11, 13);
`

In my own implementation I actually rewrote much of your rotateJoint function. It turns out you really only need two keypoints per joint. If you want to rotate the shoulder, you can simply pass in the shoulder and elbow joint. The BabylonJS skeleton model can be thought of actually acting on joints, even though the data structure is called skeleton.bones (to be more precise, the "bone" directly maps to the "parent joint", which, for the forearm for example, maps to the elbow).

Finally, I found using bone.setYawPitchRoll to be the most effective way to move the skeleton. This function rotates the skeleton in absolute coordinates, which more closely follows the data you receive from the pose detector. Some trigonometry can be used to convert the direction vector of the bone to pitch, roll, and yaw.

For reference, here is my implementation:
`
// Within the Transform Class
rotateJoint(jointA, jointB){
if (this.keypoints[jointA] && this.keypoints[jointB])
{
let a = this.keypoints[jointA]
let b = this.keypoints[jointB]

	if (isNaN(a.x + a.y + a.z + b.x + b.y + b.z))
	{
		this.joints.update({ "x": 0, "y": 0, "z": 0 })
		return
	}

	let dir = { "x": (b.x - a.x), "y": (b.y - a.y), "z": -(b.z - a.z) * this.zFactor}
	
	// Divide all values by the "size" of the vector (so that all values lie between 0-1)
	dir = this.normalize(dir)
	
	// Note that this is no longer returning angles
	// Rather it is returning a 3D vector of the direction the bone faces
	// A little bit of re-orientation might be needed when rendering to the model (described below)
	this.joints.update(jointA, dir);
}

}

// Within the Babylon Scene. Called from this.scene.registerBeforeRender
rotate(joint, data, mesh, flipAxis, isLeg)
{
let x = data.x
let y = data.y
let z = data.z

if (isNaN(x + y + z))
	return

// We use triginometry to convert the normalized direction vector into a pitch, roll, and yaw
// ... Well technically it only gives you pitch and yaw, the roll is another problem to solve

// How far to move the part forward and backward (along z axis)
let yaw = Math.asin(z) * Math.abs(x)
if (yaw > Math.PI / 2)
	yaw = Math.PI / 2
if (yaw < -Math.PI / 2)
	yaw = Math.PI / 2


if (isLeg)
{
	// How far to move the part up and down (along y axis)
	let pitch = Math.PI / 2 - Math.atan2(y, x)

	let roll = -Math.PI / 2
	yaw = -(yaw + Math.PI / 2)

	// Order of the rotations is flipped because of the default world orientation compared to the orientation of the bone
	joint.setYawPitchRoll(roll, pitch, yaw, BABYLON.Space.WORLD, mesh)
}
else
{
	let pitch = Math.atan2(y, x)
	let roll = Math.PI

	// Order of the rotations is flipped because of the default world orientation
	joint.setYawPitchRoll(yaw, roll, pitch, BABYLON.Space.WORLD, mesh)
}

}
`

Thanks!

Only the head position is getting mapped to model and the rest of model stays in the Tpose.

Just so You know
the head is spazzing a bit, but thats probably because of poor lighting.
Hands and legs are not moving at all.

Uncaught SyntaxError: Unexpected identifier (app.js:1)
Any idea?

for some reason the model's arms don't move. The head seems to be reacting.

Any ideas why?