We gave a gym app eyes — 30 FPS rep counting, zero wearables, fully on-device.
A fitness startup needed a phone camera that could recognize exercises and count reps in real time. No cloud. No hardware. Just computer vision that works inside a sweaty gym.
Gym tech relies on hardware. We were asked to remove it.
A consumer fitness startup had a simple — and genuinely hard — product brief: a phone propped on a gym shelf that recognises your exercise and counts every rep, with no wearables, no chest strap, and no internet connection.
Their existing options were either cloud-dependent (unworkable in gyms with patchy Wi-Fi) or wearable-dependent (hardware cost killed conversion). What they needed was a computer vision system capable of real-time inference on a mid-range Android — a device with roughly a hundredth of a server GPU's compute budget.
Every approach they had tried before either ran too slowly to be useful in-session, broke under real gym conditions (poor lighting, busy backgrounds, varied clothing), or required per-user calibration that killed the out-of-box experience.
A two-stage on-device pipeline, purpose-built for the constraints.
We designed a pipeline that separates what the person is doing from how many times they've done it — two distinct inference problems that each have a right-sized answer. Decoupling them meant we could tune each independently without touching the other, and swap components as the hardware targets changed.
- 01Replaced raw video with skeleton sequences as the model input. Running pose estimation first strips away everything the gym environment does to confuse a model — bad lighting, moving backgrounds, varied clothing. The result is an input that's stable across locations and people.
- 02Ran action recognition asynchronously on a background thread. The recogniser sees a rolling window of skeleton frames and emits a stable exercise label 4–6 times per second — fast enough to react, slow enough not to thrash. It never blocks the camera thread.
- 03Used a cycle-detection approach with linear — not quadratic — time complexity. Most repetition counters compare every frame against every other frame, which gets expensive fast. Our counter uses a smarter representation of motion cycles that stays fast even during a 45-minute session.
- 04Built a graceful degradation layer for low-end devices. On chipsets below a defined benchmark, the system steps down to a lighter pose model and adjusts the frame budget — staying useful at 24 FPS rather than dropping to something unusable.
- 05Fine-tuned the entire pipeline on client-collected gym footage. Benchmark datasets capture controlled lab conditions. We collected ~40k clips of real gym-environment reps and used them to close the gap between academic accuracy and production accuracy.
Data Foundation
Audit of available data, gap analysis, data collection protocol for gym-environment clips, annotation pipeline.
Model Development
Pose backbone selection, exercise classifier fine-tuning, rep-counting module development, thread-budget profiling.
On-Device Optimisation
Model quantisation, runtime export, device-tier benchmarking, thermal stress testing across 20+ handsets.
SDK & Integration
Native mobile SDK (iOS + Android), API surface for form feedback, workout summary module, beta instrumentation.
98% accuracy. Real gyms. Mid-range phones. No cloud.
The app passed the client's internal "shelf test" — propped 8 feet away against gym equipment, counting through 12 exercises across six testers without manual recalibration. Thermal headroom after a 45-minute continuous session stayed within spec on every test device. The beta hit their internal accuracy threshold (≥95% within ±1 rep) in the first week of user testing.
How the system is structured under the hood.
We don't publish model names or exact architectures publicly — the stack is a competitive advantage for the client. What we can share is the structural logic, which applies to any project in this class.
Action Classification
A compact video-understanding model fine-tuned on skeleton sequences. Optimised for the project’s exercise taxonomy — not a general-purpose benchmark model running on a server.
Temporal Cycle Detection
A counting module that detects motion cycle boundaries with complexity that scales linearly — not quadratically — with session length. Critical for sessions over 10 minutes.
Async Inference Orchestration
A producer-consumer architecture that keeps the camera thread at 30 FPS while model inference runs asynchronously on a background thread — the standard pattern for mobile CV.
Native Mobile SDK
Exported to a mobile-optimised runtime, quantised to 8-bit where precision allowed, and wrapped in a clean SDK surface the client's iOS and Android engineers could integrate in a sprint.
Pose and tracking in motion.
These related movement recordings show tracking and keypoint overlays. They are separate projects, not footage of the gym app or evidence of its phone-performance and rep-count results.
Keypoints follow the horse’s body and legs through the frame, making the pose representation visible.
Video unavailable here. Try the original recording link above.
Tracks every player through a basketball play. This example shows tracking in a busy scene rather than exercise classification.
Video unavailable here. Try the original recording link above.
Building something similar?
We work with teams shipping computer vision and on-device AI into real hardware constraints. Start with a free 30-minute scoping call — no pitch deck required.
Book a call →Speak directly with an engineer about your project.