[Question] TMRobot + Robotiq85 rotating/drifting

[reinaldoyang]

• Isaac Lab Version: [2.3.0]
• Isaac Sim Version: [5.1]

I already checked other issues that are similar to mine, but still can't seem to solve this issue.
I'm working on the techman robot + robotiq-85 gripper.
I'm following this tutorial, used the franka example, and created my own folder
https://isaac-sim.github.io/IsaacLab/main/source/overview/imitation-learning/teleop_imitation.html

When I run this command, the robot just starts rotating/drifting out of nowhere

./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py --task TM7S-Original-Cube-Stack --num_envs 1 --teleop_device keyboard

I got the USD file from the official tmrobot company, from this repo
https://github.com/TechmanRobotInc/tm_ros1_isaac_example

As for the robotic config:
tm7s_original.py

Stack joint position config
stack_joint_pos_env_cfg.py

Inverse kinematics config
stack_ik_rel_tm7s_env_cfg.py

I tried to print out the robot.num_joints and also the robot_joint_names, it seems that my both robot arm and gripper joints are defined in robot_joint_names
I tried modifying the stiffness and damping, trying out stiffness one magnitude larger than damping, it all doesn't work well, after a lot of trial and error, I decided to come here to look for help

This is the video of the robot turning
https://github.com/user-attachments/assets/74a7c845-11b3-4b8e-ab5f-30f58a6878c4

[RandomOakForest]

Thank you for posting this. This behavior almost always comes from a mismatch between how the TM7S+Robotiq are modeled in USD and how the Isaac Lab task/controller assumes the robot is structured, plus a few typical physics pitfalls.¹²

Below is a concrete summary to consider, though still under review.

---

1. Confirm base and gravity setup

Unintended “drifting” often means gravity or the base joint is not configured as expected.¹

1. In Isaac Sim (or headless, via a quick script), check the TM7S root joint:
   • The joint between the world and the base link must be:
     • Fixed if you want a fixed-base arm, or
     • A single prismatic/revolute if you intentionally want it mobile (not the case here).
   • If the USD has a 6-DoF articulation root or multiple free joints, the controller will inject torques that cause rotation.
2. For a sanity test, temporarily disable gravity on all TM7S links:
   • In Isaac Sim GUI, select the robot rigid bodies, check Disable Gravity.¹
   • With gravity disabled and your teleop script running but with zero command (no keys pressed), the arm should stay almost perfectly still.
   • If it still drifts, the issue is not gravity but the drive/joint config mismatch (see next sections).

---

2. Match joint ordering and DoF between USD and cfg

Teleop SE(3) task assumes a specific mapping from robot DoFs to the controller’s joint indices, derived from the env cfg.³

Do the following checks:

1. Print robot metadata at runtime (you already did some of this, but do it more systematically):
   • Right after env.step(0) in your task (or after env._post_initialize() in the env class), log:
     • robot.num_joints
     • robot.joint_dof_count
     • robot.data.joint_names
     • robot.data.joint_pos shape.
   • Compare this to the number and order of joints in tm7s_original.py and in your two env cfgs:
     • stack_joint_pos_env_cfg.py
     • stack_ik_rel_tm7s_env_cfg.py.
2. Make sure:
   • The arm action space only covers arm joints (same size and order as those you intend).
   • Gripper joints are either:
     • Controlled separately by the gripper action head, or
     • Completely excluded from the teleop action vector.

If arm and gripper joints are all in robot_joint_names, but the env cfg assumes a different slicing (e.g., first 7 are arm, last 2 are gripper), any off-by-N indexing will manifest as weird rotations and drifting when you press keys.

---

3. Check drive mode, stiffness, and damping

For position control, every controlled revolute/prismatic joint must have:

• Drive type: position drive.
• Stiffness > 0.
• Damping ≥ O(0.1 * stiffness) as a starting point.¹

Actionable steps:

1. In Isaac Sim, open the TM7S USD and inspect joint drives:
   • Ensure they are position-controlled (non-zero stiffness).
   • Check there are no leftover velocity drives on arm joints unless the Isaac Lab config explicitly expects velocity control.
2. Start from a known stable baseline similar to industrial arms:
   • For example (ballpark):
     • Arm joints: stiffness ≈ 2000–5000, damping ≈ 50–200 (adjust per joint mass and length).
   • Once that is stable in a simple joint-position test (see next section), plug it back into teleop.

If you set damping one order of magnitude larger than stiffness (instead of smaller), the system can become sluggish and numerically unstable, leading to strange drift instead of steady holding.¹

---

4. Verify a simple joint-position env first

Before using SE(3) teleop, validate the robot behaves correctly under the simplest joint-position control.⁴

1. Make a minimal task (or modify yours) with JointPositionActionCfg:
   • Initialize robot in your desired home pose.
   • Set target joint positions equal to initial positions.
   • No teleop, just run the sim for a few seconds.
2. Expected behavior:
   • Robot should stay essentially still, aside from very tiny numerical noise.
   • If it slowly collapses or rotates, your drive gains or USD constraints are inconsistent.

Once the simple joint-position control is solid, move to RelativeJointPositionActionCfg and then SE(3) IK teleop. If the robot is unstable even in this simplest setting, teleop will only amplify the problem.⁴

---

5. Inspect the gripper coupling and inertia

The TM7S + Robotiq85 combination can cause rotation if:

• The gripper is attached via a free/floating joint with misconfigured inertia.
• The Robotiq fingers are not symmetrical or have incorrect mass/inertia, inducing a torque on close/open.²

Recommendations:

1. In the USD:
   • Confirm the Robotiq base link is rigidly fixed to the TM7S flange (fixed joint, not loose).
   • Verify center of mass and inertia tensors of gripper links:
     • They should be roughly symmetric w.r.t. the flange axis.
   • If in doubt, temporarily reduce gripper link masses by 10–100× to see if the rotation disappears; this is a good diagnostic to isolate a mass/inertia issue.
2. In the env cfg:
   • Make sure gripper joints are not included in the IK teleop’s controlled DoFs.
   • If the action space includes gripper joints in the same vector as arm joints, a non-zero command on “gripper” indices may actually act on arm joints (or vice versa), causing rotation when you open/close.

---

6. Align IK frame and end-effector definition

The SE(3) teleop script assumes an end-effector frame (e.g., tool frame) consistent with the IK config.³

Check in stack_ik_rel_tm7s_env_cfg.py:

1. The end_effector_frame_name (or equivalent) must match the link name in the USD where you intend to control pose (usually the tool flange or gripper base link).
2. The IK solver’s initial target should be the current EE pose:
   • At reset, compute ee_pose = robot.data.body_state[ee_index] and set the IK target to this pose.
   • If you start with an IK target far from the current pose, the controller will immediately generate large motions that look like “drift” even with no keyboard input.

As a test, hard-code a fixed EE target equal to the reset pose and run teleop with no key presses; the arm should remain stable.

---

7. Keyboard teleop scaling and deadzone

With SE(3) keyboard control, each key press generates incremental pose commands.³

Ensure:

1. The teleop command scaling in teleop_se3_agent.py is not too large for your robot’s workspace.
   • If the pose increments per key press are very big, IK may produce large joint changes that appear as sudden rotations.
2. There is no non-zero default command:
   • After reset, verify the SE(3) command buffer is zeros until you press keys.
   • Add a temporary print in the teleop script to log the commanded pose deltas each step; they should be exactly zero with no key pressed.

---

8. Concrete debug path for your setup

Given your description, this is the order I’d go:

1. Run a pure joint-position holding test (no teleop, target = initial pose). Fix any collapse/drift there (likely drive gains or base joint/constraints).
2. Validate joint indexing:
   • Print joint names, indices, and the indices used in your action and IK configs.
   • Ensure arm and gripper segments are correctly sliced.
3. Inspect TM7S+Robotiq USD:
   • Fixed base to world, fixed gripper to flange, reasonable inertias.
4. Confirm IK EE frame and reset target are aligned.
5. Finally, re-enable teleop with small SE(3) scaling and log the commanded pose deltas to ensure zero-command = zero motion.

Footnotes

1. https://docs.isaacsim.omniverse.nvidia.com/4.5.0/robot_setup/joint_tuning.html ↩ ↩² ↩³ ↩⁴ ↩⁵

2. https://github.com/isaac-sim/IsaacLab/discussions/4503 ↩ ↩²

3. https://isaac-sim.github.io/IsaacLab/main/source/overview/imitation-learning/teleop_imitation.html ↩ ↩² ↩³

4. https://github.com/isaac-sim/IsaacLab/issues/2614 ↩ ↩²

[reinaldoyang]

Thank you for the detailed explanation, here are the detailed process of me trying to find the issue. Currently the issue has not been solved yet
This is the tm7s+robotiq USD file : https://drive.google.com/file/d/1PWXDqCVX6zRtVWfLMxf8PoGmU_Pe_xAO/view?usp=sharing.

I'll also include my mdp file, although it may have less or no impact on the teleop SE(3) script:
tm7s_stack_events.py
terminations.py
observations.py

1. Confirm base and gravity setup

• In isaac sim, I confirmed that my base arm are fixed to the world by pressing play, it doesn’t fall off
• the base joint has rigid body enabled with no kinematic (fixed to the world)
• Sanity test: in isaac sim, I selected all rigid body of the robot and in the properties tab, select disable gravity and then press play, the robot won’t move and stay static

2. Match joint ordering and DoF between USD and cfg

1. This is my current robot metadata
   I use print to debug, this is my teleop_se3_agent.py code containing the print statement
   Below is the printed robot metadata

• Robot Metadata:
  • Joint Names:
    • Arm (tm7s): shoulder_1_joint, shoulder_2_joint, elbow_joint, wrist_1_joint, wrist_2_joint, wrist_3_joint
    • Gripper (Robotiq 85 mimic joint from isaac sim assets): finger_joint, right_outer_knuckle_joint, left_inner_finger_joint, right_inner_finger_joint, left_inner_finger_knuckle_joint, right_inner_finger_knuckle_joint
  • Actuators:
    • Arm: tm7s_shoulder, tm7s_wrist
    • Gripper: gripper, gripper_finger, gripper_passive

[DEBUG] ===== ROBOT METADATA =====
[DEBUG] robot.num_joints: 12
[DEBUG] robot.joint_names: ['shoulder_1_joint', 'shoulder_2_joint', 'elbow_joint', 'wrist_1_joint', 'wrist_2_joint', 'wrist_3_joint', 'finger_joint', 'right_outer_knuckle_joint', 'left_inner_finger_joint', 'right_inner_finger_joint', 'left_inner_finger_knuckle_joint', 'right_inner_finger_knuckle_joint']
[DEBUG] robot.num_bodies: 19
[DEBUG] robot.body_names: ['base', 'link_0', 'link_1', 'link_2', 'link_3', 'link_4', 'link_5', 'link_6', 'flange_link', 'tool0', 'base_link', 'left_outer_knuckle', 'right_outer_knuckle', 'left_outer_finger', 'right_outer_finger', 'left_inner_finger', 'right_inner_finger', 'left_inner_knuckle', 'right_inner_knuckle']
[DEBUG] robot.data.joint_pos.shape: torch.Size([1, 12])
[DEBUG] robot.data.joint_pos: tensor([[-4.5731e-05, -1.3138e-05,  7.1033e-06,  8.7084e-04, -3.7414e-03,
1.0347e-03,  1.5311e-02,  1.7608e-04, -1.5869e-02, -8.3784e-03,
1.6929e-02, -4.5956e-02]], device='cuda:0')
[DEBUG] robot.data.joint_names: ['shoulder_1_joint', 'shoulder_2_joint', 'elbow_joint', 'wrist_1_joint', 'wrist_2_joint', 'wrist_3_joint', 'finger_joint', 'right_outer_knuckle_joint', 'left_inner_finger_joint', 'right_inner_finger_joint', 'left_inner_finger_knuckle_joint', 'right_inner_finger_knuckle_joint']
Initial joint positions (all 12 joints): tensor([-4.5731e-05, -1.3138e-05,  7.1033e-06,  8.7084e-04, -3.7414e-03,
1.0347e-03,  1.5311e-02,  1.7608e-04, -1.5869e-02, -8.3784e-03,
1.6929e-02, -4.5956e-02], device='cuda:0')
[DEBUG] ===== ACTUATORS =====
[DEBUG]   actuator 'arm': joint_names=['shoulder_1_joint', 'shoulder_2_joint', 'elbow_joint', 'wrist_1_joint', 'wrist_2_joint', 'wrist_3_joint'], num_joints=6
[DEBUG]   actuator 'gripper': joint_names=['finger_joint'], num_joints=1
[DEBUG]   actuator 'gripper_finger': joint_names=['left_inner_finger_joint', 'right_inner_finger_joint'], num_joints=2
[DEBUG]   actuator 'gripper_passive': joint_names=['right_outer_knuckle_joint', 'left_inner_finger_knuckle_joint', 'right_inner_finger_knuckle_joint'], num_joints=3
[DEBUG] ===== ACTION MANAGER =====
[DEBUG] action_manager.total_action_dim: 7

we can see that the arm action space only covers arm joints.
In the env cfg file:
stack_ik_rel_tm7s_env_cfg.py
stack_joint_pos_env_cfg.py
The IK is as defined:

arm_action = DifferentialInverseKinematicsActionCfg(
    joint_names=[
        "shoulder_1_joint",    # 0
        "shoulder_2_joint",    # 1
        "elbow_joint",         # 2
        "wrist_1_joint",       # 3
        "wrist_2_joint",       # 4
        "wrist_3_joint",       # 5
    ],
    body_name="flange_link",
    controller=DifferentialIKControllerCfg(
        command_type="pose",      # 7D unless relative
        use_relative_mode=True,   # -> 6D: dx,dy,dz,droll,dpitch,dyaw
    ),
)

and for the gripper, binary joint position control was used

gripper_action = BinaryJointPositionActionCfg(
    joint_names=["finger_joint"],  # 1 joint
    ...
)

so robot definition would be:

Arm joints (6): shoulder_1_joint, shoulder_2_joint, elbow_joint, 
                wrist_1_joint, wrist_2_joint, wrist_3_joint
Gripper drive (1): finger_joint

3. Check drive mode, stiffness, and damping

• I opened my tm7s+robotiq USD, it’s in position control (non-zero siffnes)
• no leftover velocity drives
• I use the default stiffness and damping from the official robot manufacturer, I tried changing to your recommended ballpark numbers but when i run teleoperate SE(3) it seems worse. In the simple joint-position env it has no difference

4. Verify a simple joint-position env first

I created a minimal task and initialize my robot in desired home pose, with no teleoperation, the robot will stay still and won’t drift/rotate, it seems to be correct, and doesn't drift away
arm_test_env_cfg.py

5. Inspect the gripper coupling and inertia

based on my differential ik, i don't include the gripper inside of it, only arm joints

1. the robotiq link is fixed to the tm7s flange_link
2. tried reducing the mass of gripper link, the rotation/drifting still exist
   1. left_outer_finger: 0.03918 → 0
   2. left_inner_finger: 0.03925→0
   3. left_inner_knuckle: 0.02731→0
   4. right_outer_finger: 0.03918→0
   5. right_inner_finger: 0.03925→0
   6. right_inner_knuckle: 0.03925→0

6. Align IK frame and end-effector definition

I tried to modify the stack_ik_rel_tm7s_env_cfg.py , to first get current ee pose right after spawning, and then set the IK target to this pose, the robot will note move at all and remain stable

7. Keyboard teleop scaling and deadzone

I tried reducing the sensitivity level for the teleoperate SE(3), the rotation/drifting still exist
I also tried a temporary print in the teleop script to log the commanded pose deltas each step, it is zero when no key is pressed

[reinaldoyang]

@RandomOakForest Attached is the flattened USD file, let me know if you have problem accessing it
https://drive.google.com/file/d/1eZ1j2nXFwfJlS9gRyEw0-2Eh4tJTck1U/view?usp=sharing

[reinaldoyang]

@RandomOakForest I tried disabling self-collision for the articulation, but this caused the rotation to stop completely. This suggests that the issue might be related to the collision mesh. However, it introduced a new type of rotation: as shown in Video 1, the arm rotates and then stops whenever I open or close the gripper.

Next, I attempted to reduce the damping of the robot arm. This decreased the rotation somewhat, as shown in Video 2, but the rotation is still present. Moreover, lowering the damping makes the arm’s movements very stiff, which is undesirable.

https://github.com/user-attachments/assets/dccb8ddd-0dab-40b9-828a-e006f43081ab
https://github.com/user-attachments/assets/7b7424a6-9ede-4381-948a-8dc9fa856122

Also when the gripper grasp something, it returns to the turning issue as I posted before

[RandomOakForest]

Following up, apologies for the delayed response. If you haven't resolved this issue, let us know know. I'll move this post to our Discussions section for follow up.

[reinaldoyang]

I haven't resolved this issue, help would be appreciated

[Marooncoloredchair]

Hey, I’ve been following your drift issue with the TMRobot. We ran into almost the exact same 'ghost rotation' when grasping objects in Isaac Lab.

We found that manual gain tuning (stiffness/damping) is a losing battle because the simulation's collision residuals change the moment you add the mass of a grasped object. We actually ended up building a small Adaptive Control SDK that wraps the act/update loop. It uses a model-free layer to 'learn' these drift artifacts and compensate for them in real-time.

It’s much more stable than trying to fix the USD meshes. If you’re still stuck, I’d be happy to let you try the SDK on your TMRobot setup to see if it kills that drift.

[reinaldoyang]

I haven't solved the issue yet, please let me know how I can use that SDK