🎮 Unity Hand Tracking Project – README

Overview

This project is a Unity-based application that connects computer vision (MediaPipe via Python) with a 3D hand in Unity and optionally a Raspberry Pi controlling servos.

The system captures hand movements via webcam, processes them in Python, and sends data to Unity to animate a rigged 3D hand in real time.

---

Requirements

Unity Version

• Unity 6000.3.10f1

External Dependencies

• Python (with MediaPipe setup)
• A working webcam
• (Optional) Raspberry Pi + servos

Unity Packages

Some required .NET packages are NOT native to Unity and must be installed manually.
Make sure to import any missing dependencies if you encounter build/runtime errors.

---

Project Structure (C# Scripts)

1. WebCamBackground.cs

Displays the webcam feed directly inside Unity.

Purpose:

• Captures webcam input
• Renders it as a background in the scene

---

2. VerticalLineFromPoint.cs

Draws vertical lines from fingertip positions.

** Important Setup (Inspector):**

• You MUST assign the finger tip transforms
• The hand model must be rigged properly

---

3. PythonLaunchQuit.cs

Handles launching and closing Python scripts via terminal commands when scenes change.

Purpose:

• Starts MediaPipe tracking
• Stops processes when leaving a scene
• Enables communication between:
  • Python → Unity
  • Python → Raspberry Pi (servos)

⚠️ Important Setup (Inspector):

• Specify the exact Python script name to launch/kill per scene
• Change the path in the script accordingly to your setup

---

4. MakeHandMove.cs

Animates the 3D hand based on data received from Python.

Purpose:

• Receives hand tracking data
• Applies rotations to the hand rig

** Important Setup (Inspector):**

• Assign ALL hand bones
• Set the correct port for Python communication

** IMPORTANT NOTE:**

• The function MoveBone() is calibrated for the provided hand model
• If you use a different hand model, you MUST:
  • Recalibrate
  • Modify the MoveBone() function accordingly

---

5. LaunchMenu.cs

Loads the menu scene from other scenes.

Purpose:

• Scene navigation via UI buttons

---

6. LaunchGame.cs

Loads selected scenes from the menu.

** Important Setup (Inspector):**

• Provide the exact scene names

---

Unity Setup Instructions

🔹 GameObjects

All scripts must be attached to Empty GameObjects in your scenes.

---

🔹 Hand Model

• The hand MUST be:
  • Rigged
  • Properly structured with bones
• If replaced:
  • Update references in MakeHandMove.cs
  • Rework MoveBone() logic

---

🔹 Scene Management

• Ensure all scenes are added to:

• File → Build Settings → Scenes In Build

• Scene names must match exactly with those used in inspectors

---

🔹 Python Integration

• Python scripts are launched via terminal commands
• Ensure:
• Python environment is correctly configured
• Script paths are valid
• Required libraries (MediaPipe, OpenCV, etc.) are installed

---

Data Flow

Webcam → Python (MediaPipe) → Hand tracking data → Unity (via socket) → 3D Hand Animation

(Optional) → Raspberry Pi → Servos

---

Common Issues

Nothing moves in Unity

• Check port configuration
• Ensure Python script is running
• Verify socket connection

Lines not appearing

• Check fingertip references in VerticalLineFromPoint

Hand behaves incorrectly

• Likely due to:
  • Wrong bone assignment
  • Different hand model → requires recalibration

Python not launching

• Verify script name in PythonLaunchQuit
• Check terminal permissions

---

Notes

• This project is not plug-and-play
• It requires:
  • Proper calibration
  • Correct rigging
  • Careful inspector setup

---

Included Assets

• Pre-configured hand model (calibrated for current system)
• C# scripts for Unity integration
• Python communication pipeline (external)

---

Final Advice

If you modify:

• The hand model
• The tracking system
• The communication pipeline

Expect to adapt parts of the code, especially:

• MakeHandMove.cs
• MoveBone() function

---

Contact / Contribution

Feel free to fork, improve, or adapt this system for your own projects.

---