Project files

Roboparty XR Teleop

Modified based on Unitree xr_teleoperate (opens in a new tab), designed for dual-arm teleoperation of RPO/Roboto robots with PICO VR.

Environment Requirements

• Ubuntu 20.04 / 22.04
• Python 3.10
• ROS2 Python environment
• PICO VR device
• RPO/Roboto robot control link

Installation

1. Create the environment

It is recommended to create a separate environment first:

conda create -n roboparty_xr python=3.10 -y
conda activate roboparty_xr

2. Installing IK Dependencies

It is recommended to install pinocchio and casadi via conda-forge:

conda install -c conda-forge pinocchio casadi numpy=1.26.4 -y

3. Install repository dependencies

Run the following in the repository root directory:

pip install -r requirements.txt

4. Prepare ROS2

Before running this project, ensure that the Python environment matches the minor version of Python used by ROS2.

The currently recommended/tested combination is:

• Python 3.10
• ROS2 Python environment matching Python 3.10

Before running, confirm that the terminal has sourced the corresponding ROS2 environment and that the following can be imported normally:

• rclpy
• sensor_msgs

You can perform a self-check with the following command:

python -c "import rclpy; from sensor_msgs.msg import JointState; print('ROS2 Python OK')"

Notes:

• If ROS2 is installed or built based on Python 3.12, and the current project runs in a Python 3.10 environment, rclpy usually cannot be imported normally.
• The reverse is also true: a Python 3.12 environment usually cannot directly use ROS2 Python packages based on Python 3.10.
• Do not mix ROS2 and project runtime environments with different Python minor versions.

Certificate Configuration

PICO needs an HTTPS certificate when accessing the teleoperation page via a browser.

Run the following in the root directory of the repository:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout key.pem -out cert.pem

If the host has a firewall enabled, allow port 8012:

sudo ufw allow 8012

Robot-Side Balance Policy

This project is responsible for the VR -> Upper Body Teleoperation control link. When running on the real robot, the humanoid robot still requires the underlying policy for standing and balance control, so it usually needs to be used together with roboparty_deploy (opens in a new tab).

If you need to switch the robot's underlying policy model, first modify the src/inference/launch/inference.launch.py in roboparty_deploy and change the loaded configuration file to the target policy:

configs = [
    os.path.join(
        get_package_share_directory("inference"),
        "config",
        "inference_interrupt.yaml",
    ),
]

Common optional configuration files include:

• inference_amp.yaml
• inference_attn_enc.yaml
• inference_beyondmimic.yaml
• inference_getup.yaml
• inference_interrupt.yaml

The current recommended configuration for this project is:

• inference_interrupt.yaml

After making the changes, re-run ./tools/start_robot.sh. When the robot starts, it will load the corresponding configuration and use the relevant underlying policy.

For detailed instructions, please refer to the roboparty_deploy README_CN (opens in a new tab).

For robot-side controller button operations, please also refer to this document. Combined with this project, the following buttons will mainly be used:

• X: Enable / Disable motors
• A: Reset
• B: Enter inference mode. After entering, the robot can maintain balance and walk normally, but the upper body control interface is not yet exposed to VR teleoperation
• LB: Expose the upper body control interface to VR teleoperation

The recommended real-machine joint debugging process is as follows:

1. Start roboparty_deploy on the robot side and confirm that inference_interrupt.yaml has been loaded
2. Use the robot controller to press X to enable the motors, and press A to reset the robot
3. Press the controller's B to enter inference mode and confirm that the robot can stand stably, maintain balance, and walk normally
4. Start the VR teleoperation program in this repository and follow the steps below to complete the PICO connection so that the simulated robot begins to follow the operator's movements
5. Do not immediately expose the real robot to VR; first observe the upper body posture of the simulated robot and the real robot, and try to make them as similar as possible
6. After the simulated robot and the real robot's posture are basically aligned, press LB on the robot controller to expose the upper body control interface to VR
7. Finally, continue executing on the VR teleoperation side until the A button on the VR controller is pressed to start sending upper body control commands to the real robot

In other words, for the real robot to enter upper body teleoperation, both of the following conditions must be met:

• The robot side has pressed LB on the controller to allow VR to take over the upper body interface
• The teleoperation side has pressed the A button on the VR controller to start sending control commands

Startup

Execute in the root directory of the repository:

python teleop/xr_control_rpo.py --xr-mode controller

Common parameters:

• --xr-mode {hand,controller}: Select the XR tracking mode, default controller
• --frequency: Control frequency, default 30
• --headless: Disable IK visualization. It is recommended to enable this during real-machine debugging to reduce the additional overhead caused by Meshcat
• --profile-loop: Print control loop timing statistics to facilitate analysis of the time taken by each segment (XR -> IK -> ROS)

If you want to check performance before debugging with the real machine, it is recommended to first use:

python teleop/xr_control_rpo.py --xr-mode controller --headless --profile-loop

The terminal will print statistical information similar to the following:

Loop timing avg over N iters: motion=... ms, ik=... ms, send=... ms, loop=... ms, rate=... Hz, ipopt_iter_avg=..., ipopt_iter_max=..., ipopt_hit_max=.../N, ipopt_last_status=...

Field descriptions:

• motion: Time taken for XR data reading and coordinate transformation
• ik: Time taken for arm_ik.solve_ik()
• send: Time taken to publish ROS joint commands
• loop: Time taken for the entire control loop
• rate: Actual control frequency
• ipopt_iter_*: Ipopt iteration statistics for a single IK solution

PICO Usage Process

The following process assumes:

• The PICO is on the same local network as the host running the program
• The host IP is 192.168.123.2
• The controller mode is being used

If your host IP is different, replace 192.168.123.2 in the URL below with the actual address.

1. Start the control program

python teleop/xr_control_rpo.py --xr-mode controller

After the program starts, it will wait for a start signal, and the browser will pop up the Meshcat simulation interface.

2. Open the page in the PICO browser

Access the following URL in the PICO browser:

https://192.168.123.2:8012/?ws=wss://192.168.123.2:8012

If the browser warns that the certificate is unsafe, click Continue to proceed.

3. Entering VR

After entering the page:

1. Click Pass-Through
2. Accept the permission requests from the browser and device
3. Wait for the XR session to be established

Once the connection log appears in the terminal, it indicates that the PICO has been successfully connected.

4. Starting Teleoperation

It is recommended to use the following command to start on a real machine:

python teleop/xr_control_rpo.py --xr-mode controller --headless

Terminal key bindings:

• r: Start the main loop. After pressing, the program starts receiving data transmitted by the VR, and the robot in Meshcat will follow the operator's movements, but the real machine will not move at this time.
• a: Allow sending arm commands. After pressing, the real machine starts receiving joint positions sent from the computer.
• q: Exit the program.

It is recommended to align the robot's posture in Meshcat as close as possible to the real machine's reset posture before pressing a.

VR controller key bindings:

• Right hand A: Start sending commands. The effect is the same as keyboard a.
• Right hand B: Stop sending commands. After stopping, you can press A again to resume.

It is recommended to have the operator's arm posture close to the robot's real machine posture before starting to send control commands.

Running Recommendations

• When using for the first time, perform small-range motion tests in a safe space
• Before sending commands, confirm that no one is near the robot
• Before exiting, it is recommended to first return the robotic arm to a more natural position

Directory Structure

roboparty_xr_teleop/
├── assets/
│   └── Atom01_urdf/  # compatibility asset path
├── teleop/
│   ├── robot_control/
│   │   └── robot_arm_ik.py
│   ├── utils/
│   │   └── weighted_moving_filter.py
│   ├── xr_control_rpo.py
│   └── xr_Control_Atom.py  # compatibility entrypoint
├── televuer/
├── LICENSE
├── LICENSES/
├── LICENSES.md
├── THIRD_PARTY_NOTICES.md
├── README.md
└── requirements.txt

License

This repository is released under the GPL-3.0-only license. Content based on Unitree xr_teleoperate (opens in a new tab) retains its original copyright and Apache-2.0 notice. The URDF, STL, and documentation images of Atom01 are sourced from roboto_origin, copyrighted by RoboParty. For the scope of the license, see LICENSES.md, and for third-party attributions, see THIRD_PARTY_NOTICES.md.