Hips Orthogonal basis and arm flip

[Eagleton]

Hi @TemugeB! Read your post and it has helped me a ton. When I wan't to use this way to other works, like MotionBert, it comes to 2 problem:

1. When you calculate the hips rotation, you select u = hips-lefthip, v = hips-neck, w = cross(u, v), and then decompose rotation matrix. But in many case, u and v are not orthogonal. What should I do in that case? Reserve u or v and use Gram-Schmidt Orthogonalization?
   
2. I'm having trouble when I try to retarget to a standard skeleton in maya
   My skeleton is here, it's shoulder is connected to spine, and neck is connected to spine
   
   Follow to you, I first calculate the local matrix, then calculate the world matrix by get_rotation_chain function, and then turn to euler in world space, finally, I assign the world euler to maya joints.
   
   The problem is the arm will flip when I retarget to the lafan1 model
   
   Is there any problem when convert euler or calculate joint rotation matrix? Maybe the sqrt in Decompose Function?

[TemugeB]

Hi,

Thanks for reading my post. The skeleton in my example code is defined in that way because of the keypoints that Mediapipe detects. But the ideas from my post should still carry over to other skeleton definitions.

1. The purpose of w = cross(u ,v) is to define a forward direction. In your case, u and v are not orthogonal. But w = cross(u,v) will still define a forward direction, but not a unit vector. So just convert it to a unit vector.
   Assuming v = pelvis_to_spine and w = forward_unit_vector, then recalculate u. So u = cross(v, w). This will create the unit vectors that you need to make the first rotation matrix.

2. I think the skeleton you showed is already taking it into account, but just in case, you are changing the offsets of each joint position right? For example, some of these offsets in this function will not be single direction vectors:
   

   joint_angles_calculate/calculate_joint_angles.py

   Line 128 in caec22c

   def get_base_skeleton(kpts, normalization_bone = 'neck'):

I would suggest that you feed in all 0s angles to reproduce the Tpose first and then feed single angles to specific joints to make sure they are rotated as you expect. This usually helps me debug.

If things still don't work, please leave another comment. I might be overlooking something and need to think more about it before I comment.

[TemugeB]

By the way, are the hip joints rigidly attached to the pelvis? If not, then the formulation above will not work. Take a look at this drawing. In the first figure, how would you define the forward direction? In the second figure, the pelvis and hips are rigidly attached so you can define a forward direction.

In the first figure, it is still possible to define the forward direction if you take the vector between the hips as u, not vector from pelvis to one hip. Keep this in mind.

[Eagleton]

I just rename the hips to pelvis, it's the same joint

[TemugeB]

Yeah I understand the joint names. If you see the left figure above, you can see that you might get very strange forward directions if you try to use pelvis_to_hips as u. So my suggestion is to use u = right_hip - left_hip.

[Eagleton]

Actually, I already did that

where RHip index = 1 and LHip index = 4

[TemugeB]

frame_pos[7] is spine?

In any case, this looks correct. If you pass in all 0 angles, do you get the correct T pose?

[Eagleton]

spine index = 7
From the point of view of the skeleton，It seems to a correct T other except pelvis.

But some frame, the spine seems not very good

[TemugeB]

Looks like the hips are flat, unlike the 2nd image in your original post. Could you try to find out why the hips are not as expected?
Did you correctly set the offset for the hips?

[Eagleton]

When I calculate, I use the offset of my standard joints in the 2nd image in my original post. In my original post, I set hips world position and joints local euler, so if I pass in all 0 angles, I will always get correct T pose. So I just apply the rotation result and all the world pos to the origin bone from motionbert result to debug T pose, maybe the debug is not correct

[TemugeB]

I'm not sure what you mean but these vectors are not in your T pose reconstruction. Can you check?

[Eagleton]

I mean I have two skeleton. One is the defined skeleton(left), and the other it the debug skeleton now

[TemugeB]

Well, try to find out why this is happening. After that, set joint angles one by one to make sure they are rotating as you expect.

[Eagleton]

It seems that in current frame the x is not correct
the result

modified x