Interactive Sports · Input System

Phi Step

A smartphone-camera foot interaction system for a five-zone rhythm game, from tracking and calibration to step events.

Product Prototyping & ImplementationInternal research prototype2026 · Ongoing research

Context

A camera-based rhythm experience needs more than visible feet. It needs a consistent interpretation of where the player is standing, which interaction zone a foot enters, and when that movement should count as a new input.

My responsibility

Designed

The separation between foot observation, calibrated interaction, and game input.

Implemented

Camera and tracking integration, player-space calibration, step-event interpretation, and an internal game slice.

Validation scope

Deterministic coordinate and event scenarios, tracking diagnostics, and the documented review of supplied iPhone recordings.

Reused foundations

Platform camera and model runtimes, and existing pose-model families. The contribution is the interaction system and its integration.

Constraints

Setup

Explore interaction with a phone camera and ordinary footwear, without requiring a physical step mat.

Uncertain input

Feet can disappear, overlap, or reach the image edge. Uncertain observations must not invent new hits.

Consistent coordinates

The visual overlay and input interpretation must describe the same player-relative zones.

Research scope

The current model and game slice remain internal research; broad device and participant reliability is unresolved.

System architecture

Capture & observation
Ownership boundaryPlatform services + tracking integration
Turn camera input into foot observations with an explicit uncertainty state.
1. Camera input
2. Foot observation
3. Anchor representation
Foot position and observation quality
Interaction interpretation
Ownership boundaryMy calibration and event boundary
Interpret observations in the player's calibrated space before deciding whether a step occurred.
1. Calibration
2. Interaction zones
3. Step events
Discrete interaction events
Experience
Ownership boundaryMy internal game integration
Consume accepted events for judgement and feedback, with explicit behavior when tracking becomes unavailable.
1. Input adapter
2. Game judgement
3. Player feedback

Why these boundaries exist

Observation answers where a foot appears to be. Interaction answers what that position means for the player. The game consumes events, so changing a tracking approach does not require rebuilding the game's input semantics.

Engineering decisions

Validate interaction before investing in a specialized model

Context: The first uncertainty was whether camera input could support a useful zone-to-event interaction loop.
Options: Begin with a specialized perception model
Use visible markers to test interaction first
Decision: Start with a marker-assisted baseline, then evaluate markerless alternatives.
Why: It made calibration and event behavior observable before perception became the main research problem.
Tradeoff: The baseline required markers. That setup burden became a reason to investigate body pose and later foot-specific observation.

Keep event semantics stable as perception changes

Context: Marker-assisted, general pose, and foot-specific approaches produce different observation quality.
Decision: Keep calibrated zone interpretation and step events behind a shared observation boundary.
Why: The experience needs consistent inputs even when the source of foot observations changes.
Tradeoff: Each perception path still needs its own uncertainty and identity checks; a shared boundary does not make all providers equally reliable.

What broke

Correct tracking, wrong player direction

Observation: Supplied device recordings showed plausible foot tracking while the virtual mat's directional interpretation was wrong.
Cause: Camera coordinates and player-relative directions had been treated as the same convention.
Change: Make the coordinate boundary explicit and derive the overlay and input classification from one logical mat.
Still open: Directional behavior and a complete game session still require physical acceptance on the current candidate.

Markerless did not mean framing-independent

Observation: The general body-pose comparison was unreliable with only the lower body visible.
Cause: The interaction needed foot observations under a narrower framing constraint than the general model handled well.
Change: Evaluate a task-specific foot-observation path while retaining earlier approaches as comparison baselines.
Still open: Crossed feet, prolonged occlusion, and diverse participants need broader evidence.

Iterations

01
Marker-assisted baseline
Make camera input, calibration, and step events testable.
02
General markerless pose
Remove marker dependency and discover framing limitations through comparison.
03
Foot-specific research
Investigate lower-body observation while preserving downstream interaction semantics.

Verification

Automated scenarios
RECORDED
The source contains deterministic tests for calibration, coordinate mapping, zone behavior, and step events. The records establish behavior in the tested scenarios; physical acceptance is assessed separately.
Device observations
PARTIAL
Supplied iPhone recordings support a limited lower-body tracking and interaction slice. They do not cover the complete edge-case and game-experience matrix.
Artifacts
RESEARCH
Annotated recordings, frame-level outputs, diagnostics, and session summaries help distinguish observation quality from game judgement.

Current state

INTERNAL RESEARCH PROTOTYPE

The internal path connects camera-based foot observation to calibrated step events and a game slice. Production readiness, broad compatibility, and complete physical acceptance remain open.

Discuss a relevant project