Floradex
Point your phone at a plant; get a live ID and a Pokédex-style card.
A curiosity about the living world, made tactile — point your phone at a plant and watch a box find it, name it, and remember it. Underneath the toy is a clean engineering idea: real-time detection and accurate identification are two different problems, and the app is fast and accurate precisely because it refuses to conflate them.
In Motion
The app running on a phone — the full capture flow end to end, then a closer look at the live detection loop.
What it is
Floradex is a native iOS (Swift/SwiftUI) computer-vision app. Aim it at a plant and a bounding box tracks it live at ~30 fps; tap the shutter and it identifies the species; save it, and Claude writes a small structured card — family, Latin name, relatives, a fact or two. Every catch lands in a Pokédex-style collection you can browse later. Native was a deliberate choice: real-time camera CV is dramatically better with Vision, AVFoundation, and Core ML than anything cross-platform.
Architecture — a three-stage pipeline
The pipeline deliberately splits “real-time” from “accurate.” The latency-critical loop stays on the device; the accuracy-critical work lives in the cloud, off the hot path.
Detect
On-device · ~30 fps
Apple Vision + AVFoundation
A live bounding box tracks the plant right on the viewfinder. Latency-critical — no network round-trip survives a 30 fps loop, so it must run locally.
Classify
Cloud · on capture
Pl@ntNet API
A still frame goes to a fine-grained classifier for global species ID + confidence. Off the real-time path, so ~1–2 s of latency is fine.
Enrich
Cloud · on save
Claude Haiku 4.5
The species name becomes a tight, schema-valid info card — family, Latin name, notable relatives, fun facts — via structured outputs.
The key insight
“Identify a plant with a YOLO-style box” secretly bundles two different CV problems: object detection (real-time, few classes, on-device) and fine-grained classification (thousands of species, needs a large model or an API). Separating them per stage is the whole reason the app can be both fast and accurate.
What it took
Debug the invisible with an on-screen HUD
Vision returns normalized, bottom-left-origin rects; SwiftUI is top-left; the preview crops aspect-fill. The box was mis-scaled, offset, and its axes transposed. Rather than guess, I put the raw numbers on screen and plotted them against the object's true position across five field screenshots — which revealed a clean X↔Y transpose, then hand-wrote the scale-to-cover transform against all five data points before shipping.
One source of truth for orientation
The bounding box double-rotated because orientation was applied twice — once on the output connection and again as a Vision hint. Two places defining the same fact fought each other. Fixed by making the Vision hint the single source of truth.
A failure mode that reshaped the roadmap
Field-testing showed Apple's saliency only isolates high-contrast subjects — it detects contrast, not “plant-ness,” and structurally struggles with the core case: a green plant against green foliage. That empirically-discovered limit is exactly why Phase 2 is a domain-specific trained detector.
Where it's going
Phase 1 — the vertical slice. Real-time on-device detection → capture → global species ID (Pl@ntNet) → AI enrichment (Claude) → a durable local collection (SwiftData). A complete path, working end to end on real plants.
Phase 2 — the ML deep-dive. Train a custom plant detector (fine-tune YOLO on a plant dataset, evaluate, export to Core ML) and swap it in for Apple's saliency — directly fixing the low-contrast-foliage failure found in the field.