Skip to content

README.md

Latest commit

 

History

History
198 lines (136 loc) · 4.23 KB

README.md

File metadata and controls

198 lines (136 loc) · 4.23 KB

🎮 Unity Hand Tracking Project – README

Overview

This project is a Unity-based application that connects computer vision (MediaPipe via Python) with a 3D hand in Unity and optionally a Raspberry Pi controlling servos.

The system captures hand movements via webcam, processes them in Python, and sends data to Unity to animate a rigged 3D hand in real time.

Requirements

Unity Version

  • Unity 6000.3.10f1

External Dependencies

  • Python (with MediaPipe setup)
  • A working webcam
  • (Optional) Raspberry Pi + servos

Unity Packages

Some required .NET packages are NOT native to Unity and must be installed manually.
Make sure to import any missing dependencies if you encounter build/runtime errors.

Project Structure (C# Scripts)

1. WebCamBackground.cs

Displays the webcam feed directly inside Unity.

Purpose:

  • Captures webcam input
  • Renders it as a background in the scene

2. VerticalLineFromPoint.cs

Draws vertical lines from fingertip positions.

** Important Setup (Inspector):**

  • You MUST assign the finger tip transforms
  • The hand model must be rigged properly

3. PythonLaunchQuit.cs

Handles launching and closing Python scripts via terminal commands when scenes change.

Purpose:

  • Starts MediaPipe tracking
  • Stops processes when leaving a scene
  • Enables communication between:
    • Python → Unity
    • Python → Raspberry Pi (servos)

⚠️ Important Setup (Inspector):

  • Specify the exact Python script name to launch/kill per scene
  • Change the path in the script accordingly to your setup

4. MakeHandMove.cs

Animates the 3D hand based on data received from Python.

Purpose:

  • Receives hand tracking data
  • Applies rotations to the hand rig

** Important Setup (Inspector):**

  • Assign ALL hand bones
  • Set the correct port for Python communication

** IMPORTANT NOTE:**

  • The function MoveBone() is calibrated for the provided hand model
  • If you use a different hand model, you MUST:
    • Recalibrate
    • Modify the MoveBone() function accordingly

5. LaunchMenu.cs

Loads the menu scene from other scenes.

Purpose:

  • Scene navigation via UI buttons

6. LaunchGame.cs

Loads selected scenes from the menu.

** Important Setup (Inspector):**

  • Provide the exact scene names

Unity Setup Instructions

🔹 GameObjects

All scripts must be attached to Empty GameObjects in your scenes.

🔹 Hand Model

  • The hand MUST be:
    • Rigged
    • Properly structured with bones
  • If replaced:
    • Update references in MakeHandMove.cs
    • Rework MoveBone() logic

🔹 Scene Management

  • Ensure all scenes are added to:

  • File → Build Settings → Scenes In Build

  • Scene names must match exactly with those used in inspectors

🔹 Python Integration

  • Python scripts are launched via terminal commands
  • Ensure:
  • Python environment is correctly configured
  • Script paths are valid
  • Required libraries (MediaPipe, OpenCV, etc.) are installed

Data Flow

Webcam → Python (MediaPipe) → Hand tracking data → Unity (via socket) → 3D Hand Animation

(Optional) → Raspberry Pi → Servos

Common Issues

Nothing moves in Unity

  • Check port configuration
  • Ensure Python script is running
  • Verify socket connection

Lines not appearing

  • Check fingertip references in VerticalLineFromPoint

Hand behaves incorrectly

  • Likely due to:
    • Wrong bone assignment
    • Different hand model → requires recalibration

Python not launching

  • Verify script name in PythonLaunchQuit
  • Check terminal permissions

Notes

  • This project is not plug-and-play
  • It requires:
    • Proper calibration
    • Correct rigging
    • Careful inspector setup

Included Assets

  • Pre-configured hand model (calibrated for current system)
  • C# scripts for Unity integration
  • Python communication pipeline (external)

Final Advice

If you modify:

  • The hand model
  • The tracking system
  • The communication pipeline

Expect to adapt parts of the code, especially:

  • MakeHandMove.cs
  • MoveBone() function

Contact / Contribution

Feel free to fork, improve, or adapt this system for your own projects.