Skip to content

bimanual dexterous manipulation state_space and action_space issue #122

@tj-chen-1209

Description

@tj-chen-1209

Dear authors,
In my setup, I have a bimanual dexterous robot with:
Right arm: 6 DoFs
Left arm: 6 DoFs
Right hand: 12 DoFs
Left hand: 12 DoFs
I would like to finetune RDT-1B on this system. To embed my action/state representation into your 128-dimensional workspace, I am considering the following layout:
Right arm joint positions (6 dims): indices 0–5 (a subset of your [0, 10) block)
Left arm joint positions (6 dims): indices 50–55 (a subset of your [50, 60) block)
Right + left hand joint positions (12 + 12 dims): indices 103–126 (24 dims within the reserved [103, 128) block)
The remaining arm-related joint slots, velocities, and some reserved indices would simply be left unused or filled with zeros.

My questions are:

  1. From the design perspective of the unified 128-D workspace, is it reasonable to reuse the reserved range [103, 128) to store additional joint positions for dexterous hands in this way?
  2. During pretraining of RDT-1B, did you make any implicit assumptions about the reserved indices (e.g., that they are always zero or ignored by the model)? In other words, could populating [103, 127] with non-zero hand joint values during finetuning cause a significant distribution shift that you would be concerned about?
  3. Are there any recommended practices or constraints for extending the joint-angle part of the vector when moving from a single-arm setup to a bimanual dexterous setup?

Any feedback on whether this index assignment is compatible with your unified representation and pretraining assumptions would be greatly appreciated.
Thank you very much for your time and for releasing this excellent work.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions