Project files

THING is a Human-Mimetic Robot Hand based on ROS 2 that recognizes the user's hand gestures in real time and reproduces them with a 7-axis tendon-driven robot hand. MediaPipe detects 21 hand landmarks, which are converted into a 7-axis command consisting of the flexion of the four fingers and the flexion, opposition, and abduction of the thumb.

The Jetson Orin Nano handles vision recognition and motion recording, while the Raspberry Pi 5 handles command arbitration, safety validation, and DYNAMIXEL control. By connecting a React control web app and an EC2 data portal, we implemented a single flow from real-time control to storing and querying experiment data.

Current implementation status

Vision recognition, command arbitration, safety validation, 7-axis motor actuation, the Web Bridge, and the data recording and upload path are integrated. Hand-gesture mimicry and object grasping demonstrations have been completed on the actual tendon-driven robot hand.

Demo

Final Integration Demo

A final demonstration showing real-time imitation control of a 7-axis tendon-driven robotic hand based on recognition of the user's hand gestures.

https://github.com/user-attachments/assets/3f23d3e1-5532-4831-a1eb-fe4c024d04cb (opens in a new tab)

Open original final demo video

Movement Highlights

The preview below briefly shows nine imitation, grasping, and independent-joint movements in sequence.

Individual demo videos

Integration & Imitation

• Final hand gesture imitation demo
• 7-axis drive demo
• Multiple hand gesture imitation
• Wave imitation after latency improvements

Grasping and Sequences

• Can grasp imitation
• Grasping flexible objects
• Wrist brace grasp
• Thumb 3-axis motion
• Finger wave
• Countdown sequence

The original MP4 videos are managed with Git LFS.

Core Implementation

System Architecture

사용자 손
  → Camera / MediaPipe / 7축 목표 생성                (Jetson Orin Nano)
  → command_manager / command_guard / safety_manager (Raspberry Pi 5)
  → DYNAMIXEL XL330-M288-T × 7
  → 텐던 로봇 손

카메라·로봇 상태
  → Web Bridge (WebSocket + MJPEG)
  → React 관제·제어 웹                               (Laptop)

동작 데이터
  → rosbag2 → exporter → uploader                    (Jetson Orin Nano)
  → Django API / SQLite / 파일 저장                  (AWS EC2)

The web interface does not control the motors directly. Every general command passes through command arbitration and safety validation on the Raspberry Pi, and the system is configured so that even when the network is restored, it will not move automatically without explicit user input.

직접 구현한 부분

Translation:

Parts I Implemented Myself

• Using Codex's Blender skill, I designed a model that mimics the joint structure of the human hand and fabricated it with 3D printing.
• I tuned the control values for seven DYNAMIXEL motors and implemented synchronized motion and keyboard-based control.
• I set up a motor communication module based on the U2D2 and implemented a ROS bridge that converts data sent from the control node into motor commands.

Troubleshooting

• Motor control range tuning: I checked the current limit, torque settings, rotation range, and per-control-mode behavior in the DYNAMIXEL SDK. Comparing how each parameter affects actuation, I adjusted the control values to suit the robot hand's mechanical conditions.
• Tendon drive and return structure: I had to consider both the finger-bending motion and the motion that returns it to its original position. I calculated the joint's force application points and reviewed the tendon routing, then changed the structure to a return method that uses tension.
• Improving ROS communication cycle: To improve the transmission cycle of control commands, I adjusted the ROS communication settings and used Cyclone DDS. I configured the communication environment for real-time control while checking both motor actuation and data transfer.

회고

Although I did not get as far as simulation-based learning for Physical AI, I learned the basics of connecting a robot's mechanical structure with motor control. While building it, I ran into communication errors and hardware limits, which helped me understand conditions that are hard to solve with software alone. Based on this, I plan to improve the mechanical and communication structure, and I am developing a project that will connect simulation-based learning with real-world control later on.

Actuator Implementation and Design Decisions

I was responsible for Blender modeling and 3D printing, U2D2 communication, synchronous driving of 7 motors and keyboard control, ROS bridge and Cyclone DDS communication setup, and integration of the fixtures, spools, and tendons. Hand gesture recognition, command arbitration, guard and safety states, and the control UI and logging were implemented by my teammates. The table below summarizes the detailed handling in the motor check code I wrote.

Joint angle, filter and motor calibration formulas, torque application order, and the actual verification scope are summarized in the Drive and Mechanism Integration Technical Note.

Tech Stack

Quick check

Clone the repository

Git LFS is required.

git lfs install
git clone https://github.com/SeMinKong/THING.git
cd THING
git lfs pull

Running the control UI without a robot

cd web/frontend
npm ci
npm run mock

In another terminal, run the following command.

cd web/frontend
npm run dev

Building the ROS 2 Package

This assumes an Ubuntu 22.04 environment with ROS 2 Humble.

source /opt/ros/humble/setup.bash
cd thing_ws
rosdep install --from-paths src --ignore-src -r -y
colcon build --symlink-install
source install/setup.bash
colcon test
colcon test-result --verbose

Before running the physical hardware, check the device-specific settings and the safety documentation first.

Repository Structure

Key Documents

• Drive and Mechanism Integration Technical Notes
• System Architecture
• ROS 2, Web, and EC2 Interfaces
• Requirements Specification V7.1
• Porting Manual
• Safety Manager
• Ubuntu Development Environment
• DYNAMIXEL Configuration
• Internal Monitoring Web
• Web Bridge
• Logger and Landmark JSON Contract
• Mechanical Materials
• Electronics and Wiring Materials
• Test Procedures
• EC2 Data Portal

Contributors

Safety

Before applying torque to a real motor, confirm the motor ID, the range of motion, the current limit, the tendon tension, and the means of emergency stop and power disconnection.

• DYNAMIXEL tools and basic checks
• Motor control safety guidelines
• Raspberry Pi field control guide

License

This project is distributed under the Apache License 2.0.

Original Project README

This repository is a portfolio-style version of the final snapshot of the SSAFY team GitLab project main. The technical and collaboration-focused README used in the team development repository is preserved in README_LEGACY.md.