Engineering Console
every layer, one capture · -
Live system pipeline
sample trace
One photo, on-device: two CNNs and a gradient-boosted arbiter, no server in the loop. Capture & trace for live numbers. Tap any stage above to jump to how it works.
What was builtby the numbers
145food classes
132 global + 13 israeli
132 global + 13 israeli
2+1CNN experts + a learned arbiter
88.2%global test accuracy
leakage-free split
leakage-free split
86.2%full-system top-1
+8.8 pts over baseline
+8.8 pts over baseline
<1 son-device recognition
2 CNNs + arbiter, timed live
2 CNNs + arbiter, timed live
20XGBoost arbiter features
fp16ONNX, torch-free
runs in the browser
runs in the browser
100%offline recognition
models cached on-device
models cached on-device
reverse-engineered BLE scale protocol
ensemble solves catastrophic forgetting
leakage-free de-duplicated splits
Gemini vision fallback when unsure
USDA smart-filter nutrition
pure-logic lib, 27 unit tests
Timing, honestly: the recognition (preprocess + both CNNs + arbiter) runs entirely on the phone and is timed live on every capture (the chip shows the real ms for your device, well under a second). The nutrition lookup that follows is a separate call over the network (about a second via the USDA Worker), so a full capture-to-calories flow is roughly 1 to 1.5 s, most of it that network round-trip, not the models.
1 Preprocessing · image pipeline
Every step below is classical image processing done in code before the CNN sees a single number - crop, resample, channel split, and normalisation. Identical to the desktop pipeline; verified 0% top-1 mismatch.
2 Expert models
3 Arbiter feature vector (20)
| feature | value |
|---|
Amber = interaction features derived from the base signals.
4 XGBoost arbiter → routing
5 Nutrition lookup
Worked example - how a USDA query is chosen. For a food not in the local DB, the app sends the label to the Worker, which queries USDA and runs the smart filter (reject, then score, then pick the top). The request:
GET /search?query=apple&dataType=Foundation,SR Legacy&pageSize=10
USDA returns up to 10 rows; the filter scores each - a query word in the primary name (before the first comma) is worth ×100 vs ×15 for a trailing qualifier, +8 for "raw", with penalties for length and ALL-CAPS - and rejects processed-product rows (juice, dried, flavored...). Real result:
| candidate USDA returned | score | verdict |
|---|---|---|
| Apples, raw, without skin | 141 | ✓ chosen → 48 kcal |
| Apples, gala, with skin, raw | 140.5 | kept |
| Croissants, apple | 14 | kept ("apple" only a qualifier) |
| Babyfood, juice, apple | - | ✗ 'juice' |
| Apples, dried, sulfured | - | ✗ 'dried' |
Why some foods are pinned in the local DB instead. USDA's own naming can defeat the primary-name rule. Query almonds and USDA files the nut as "Nuts, almonds, whole, raw" - so the primary name is "Nuts" and "almonds" is only a qualifier (score 21), while "Almond butter, creamy" has "almond" as its primary name (score 123.5) and "butter" is not a junk word - so the filter picks almond butter → 645 kcal. Rather than special-case every such food, almonds and the bell-pepper colors are pinned in the local DB with correct values, so they resolve deterministically and never reach this search.
6 Weight decode · BLE scale
6b Scale calibration · why the app shows a different number than the scale
The number in the app is not
what the scale's own LCD shows, and that is deliberate. This BLE kitchen scale has a
faulty load cell that reads a steady ~16% low. Because calories scale
linearly with weight, we correct it in software before it ever reaches a
calorie figure. Here is exactly what we measured and the formula we derived.
The full calibration study (scatter, error-vs-mass, formula-output-vs-truth and zoom charts) lives in the main report’s BLE Scale tab.
We fit on 10 reference masses (21-1062 g), 5 reads each, centred on a spreader plate, scoring each
candidate by mean absolute error MAE = (1/n)Σ|ŷ − true|, n=10:
• raw (no fix): MAE 50.7 g (16.1%)
• two-parameter line ŷ = a + b·raw: a = +2.846 g, b = 1.1728 → MAE 4.9 g
• through-origin ŷ = k·raw: k = Σ(raw·true)/Σ(raw²) = 1 775 157 / 1 506 877 = 1.178 → MAE 5.1 g
The two are tied on MAE, but the line's +2.846 g intercept over-reads light items (a load-cell span error is multiplicative), so we ship the through-origin model corrected_g = 1.178 · raw.
Validation: applying the unchanged 1.178 to 10 unseen objects (phone, battery, glass, remote, cardboard, camera, toys) predicts them to 1.9 g mean error, so it captures the scale's real span, not a water-specific fit; refitting on all 20 barely moves the slope (1.17804 → 1.17825).
Result: mean error 16% → ~3%, max error ~13 g; the residual is now the scale's own repeatability (~10 g spread), i.e. the hardware noise floor. The app shows this corrected value; the LCD shows the raw one.
Hard limit ~5 g: a 3 g object read 0 on all 5 tries - below a ~4 g detection floor the scale outputs zero, which no multiplicative correction can recover. So sub-5 g weights (and their calories) are flagged as unreliable.
• raw (no fix): MAE 50.7 g (16.1%)
• two-parameter line ŷ = a + b·raw: a = +2.846 g, b = 1.1728 → MAE 4.9 g
• through-origin ŷ = k·raw: k = Σ(raw·true)/Σ(raw²) = 1 775 157 / 1 506 877 = 1.178 → MAE 5.1 g
The two are tied on MAE, but the line's +2.846 g intercept over-reads light items (a load-cell span error is multiplicative), so we ship the through-origin model corrected_g = 1.178 · raw.
Validation: applying the unchanged 1.178 to 10 unseen objects (phone, battery, glass, remote, cardboard, camera, toys) predicts them to 1.9 g mean error, so it captures the scale's real span, not a water-specific fit; refitting on all 20 barely moves the slope (1.17804 → 1.17825).
Result: mean error 16% → ~3%, max error ~13 g; the residual is now the scale's own repeatability (~10 g spread), i.e. the hardware noise floor. The app shows this corrected value; the LCD shows the raw one.
Hard limit ~5 g: a 3 g object read 0 on all 5 tries - below a ~4 g detection floor the scale outputs zero, which no multiplicative correction can recover. So sub-5 g weights (and their calories) are flagged as unreliable.
Source:
scripts/eval/weights_cal.csv ·
fit by scripts/eval/weight_calibration.py (in the main CalEyeZ repo).7 Compatibility & requirements
CalEyeZ runs the full pipeline on-device
(no cloud inference), so the browser and hardware set the limits. It does not
run identically on every phone - here is exactly what it needs and why.
| requirement | needs | why |
|---|---|---|
| WebAssembly + SIMD | any modern browser (2021+) | onnxruntime-web executes the two CNNs in WASM; SIMD is what makes ~0.6 s inference possible |
| Camera | getUserMedia over HTTPS | secure-context only; the ROI crop is read straight off the video frame |
| BLE scale (SWAN) | Web Bluetooth - Android Chrome/Edge only | iOS (Safari and Chrome-iOS, both WebKit) and Firefox block Web Bluetooth → on iPhone the scale can't pair, so weight falls back to manual grams |
WebGPU (optional, ?gpu=1) | WebGPU-capable browser + GPU | faster fp16 compute path; default is WASM fp32 because some mobile GPUs lose accuracy in fp16 |
| Memory / download | ~50 MB models, ≳2 GB RAM | global fp16 ≈25 MB (320², 132 cls) + israeli fp16 ≈25 MB (224², 13+bg); WASM upcasts fp16→fp32 at run time, so a few hundred MB is touched during inference |
| Network | first load only | models fetch once from GitHub Pages then HTTP-cache; recognition then works offline - only USDA nutrition lookups need the network |
Bottom line: best experience is
Android Chrome (full feature set incl. BLE scale). iPhone/Safari runs recognition +
camera + manual weight, but not the Bluetooth scale. Desktop Chrome/Edge runs everything except the phone camera use-case.