CalEyeZ

Then vs Now presented in class current

This tab compares exactly what was shown at the review conference against the rebuilt system. Every "then" number is recomputed from the original results file FINAL_REPORT_v3_results.csv (16,720 images); every "now" number is from the current leakage-free evaluation.

Headline comparison

Why the old router barely used the second expert

In the class version the router chose the old model on 95.0% of images (15,877 of 16,720) and the new model only 5.0% of the time. The new expert was contributing only 6.54% correct answers, so the router had almost nothing to gain by switching. In other words it was not really routing, it was defaulting to the larger model. The current arbiter is a genuine decision maker, explained in full in the Arbiter tab.

The current arbiter is trained as an explicit domain detector (route_to_israeli = (domain == israeli)), reaches Israeli recall of 79% with global recall 98% in V2 (up from 61% / 98% after the open-set background retrain), and the Arbiter tab explains exactly which features drive each decision. It is also why the current component is correctly called an arbiter, not a router: it adjudicates between two experts that have already run, rather than forwarding by a rule.

What changed structurally

Source files: FINAL_REPORT_v3_results.csv (then), datasets/system_evaluation.csv and arbiter_dataset.csv (now).

Dataset

The global dataset was consolidated from the inconsistent 158-class set into a clean, leakage-free 132-class label space, then re-split 70/20/10. This is where two of the reviewer's points are answered directly: class balance and leakage. Pipeline: scripts/flatten_general_dataset.py.

Where the classes went (158 to 132, fully accounted)

The reduction is not deletion of useful classes. Most removed labels either moved to the Israeli expert or were folded into a class they were visually identical to. Here is every change.

What each model is trained to recognize

The two experts cover disjoint label spaces. The global model knows 132 international foods; the Israeli model knows 13 local dishes. No class appears in both, which is why routing reduces cleanly to a domain decision.

De-duplication: how leakage was removed

Two stages run before splitting, so the split unit is a content cluster and never a raw file:

• Exact: SHA-1 hash of file bytes. Byte-identical copies are dropped (981 found).
• Near-duplicate: 64-bit dHash with Hamming distance ≤ 5. Visual twins are clustered and the whole cluster is kept inside a single split.

Class balance (the reviewer's point)

The classes are not perfectly balanced, and that is reported honestly rather than hidden. Dish classes hold roughly 750 to 1,000 images each; ingredient classes (single fruits, nuts) hold far fewer. The effect is confined to the smallest classes and is a small-sample effect, not a systematic weakness. The training uses class-aware sampling and heavy augmentation to compensate, and per-class recall is reported in the System Analysis tab so the imbalance is visible, not papered over.

Global Model 88.18% clean test

YOLO11l-cls (Ultralytics, ImageNet-1k pretrained, about 13M parameters), fine-tuned on the 132-class set. Run directory: runs/general_model_flattened/.

Why this model, backed by data

Training configuration and why we chose each setting

Robustness to lighting and blur: the trade-off we chose on purpose

A model that scores well in a studio but fails on a darker plate is useless as a product. We therefore treated harsh, variable lighting as a first-class requirement and trained for it directly with aggressive photometric augmentation. On every training batch each image is re-lit at random, so the network never sees the "one true lighting" of a dish and is forced to learn features that are invariant to it.

How it works. Per image, Ultralytics samples a random gain and rescales the colour channels: V ← V·U(0.5,1.5), S ← S·U(0.25,1.75), H ← (H·U(0.98,1.02)) mod 180. Over the ~116 training epochs each photo is seen ~116 times, each time under a different random lighting, white balance and exposure.

Training code

# scripts/training/train_general_model.py  (essentials)
from ultralytics import YOLO

model = YOLO("yolo11l-cls.pt")            # ImageNet-1k pretrained
model.train(
    data="datasets/general_model_flattened",  # 132 classes, 70/20/10
    imgsz=320, batch=16, epochs=150, patience=30,
    optimizer="AdamW", lr0=1e-3, lrf=0.01, cos_lr=True,
    dropout=0.2, label_smoothing=0.1,
    augment=True, hsv_h=0.02, hsv_s=0.75, hsv_v=0.50,  # lighting/temp/exposure
    degrees=20, translate=0.15, scale=0.60, flipud=0.15, fliplr=0.50, erasing=0.40,
    # + custom Albumentations blur (motion/defocus/gaussian) on 35% of batches via a train-batch callback
    project="runs", name="general_model_flattened",
)
# validate on the held-out TEST split
model.val(data="datasets/general_model_flattened", split="test")

Results (best.pt, epoch 73)

Most-confused class pairs (test)

Every meaningful confusion is a plausible visual overlap, not a failure:

Israeli Model 91.7% on its domain

YOLO11l-cls, 13 Israeli classes (baklava, bourekas_cheese, falafel, hummus, jachnun, malawach, meorav_yerushalmi, sabich, samosa, schnitzel, shakshuka, shawarma, sufganiyah). Run: runs/israeli_food_yolo11l/, imgsz 224.

Why a second, separate expert

The first attempt added Israeli food to the global model and retrained. The combined model got confused: it lost accuracy on both the international classes it already knew and the new local ones. This is catastrophic forgetting, and it was the trigger for the whole arbiter architecture. Keeping the Israeli dishes in their own small expert means the global model never degrades, and the local model can specialise hard on a narrow domain where it reaches about 92% while the global model scores 0% on those same classes.

Training configuration and why we chose each setting

Results

The accuracy curve is noisier than the global model's because the dataset is much smaller, so each epoch sees fewer images and the validation estimate has higher variance. The trend is still a clean climb to about 92%.

Known behaviour the arbiter must handle

Arbiter acc 96.1%, ROC-AUC 0.97 (V2)

Why we call it an arbiter, not a router

A router would forward the image to one model based on a fixed rule before seeing any result. Our component does the opposite: it runs both experts first, then weighs their competing evidence (confidence, entropy, margin) and adjudicates which one to trust. That is arbitration, not routing. The distinction matters because the decision is made on the models' actual behaviour for this specific image, not on a prior guess about where the image should go.

What the arbiter is, in one paragraph

The arbiter is not a third food classifier. It never looks at the image. It looks only at the two experts' output behaviour for that image (their confidences, how spread out their guesses are, how decisive each one is) and answers a single yes/no question: does this image belong to the Israeli expert's domain? If yes, the system returns the Israeli model's label; if no, it returns the global model's label. Formally the target is route_to_israeli = (domain == israeli). Because the two label spaces do not overlap, routing reduces cleanly to domain detection.

The 20 features and their math

The arbiter input is a 20-number vector: 7 numbers summarising the Global model's output, 8 for the Israeli model (V2 adds i_p_background), and 5 interaction terms that compare the two. Below, p_k is the k-th highest softmax probability and K the number of classes. The "g_" prefix is the Global model, "i_" the Israeli model.

Seeing the math live (per image)

The demo does not just print a probability. For each capture it computes the exact TreeSHAP contribution of every feature, that is, how many log-odds each one added toward "route to Israeli", and shows the top contributors under the decision panel. A typical readout for a global food:

WHY (top feature contributions):
conf_gap     =  0.640  +1.31 -> Global    # global far more confident than israeli
entropy_gap  = -0.220  +0.74 -> Global    # israeli more uncertain than global
i_conf1      =  0.180  +0.42 -> Global    # israeli top-1 is weak

This is the faithful per-prediction explanation (not a global average), so during the demo you can point at the exact numbers that produced the routing decision.

The arbiter decides on relative confidence between the experts, not on the Israeli model's raw confidence, which is exactly why it survives that model's out-of-domain overconfidence.

The softmax behind the features (with code proof)

Every confidence the arbiter consumes is a softmax probability. Softmax turns a model's raw logits z into a probability distribution over classes:

softmax(z)_i = e^(z_i) / Σ_j e^(z_j)        # non-negative, sums to 1

Where it runs: we do not hand-write softmax. Each YOLO11l-cls expert applies it inside its classification head (Ultralytics / PyTorch), so results.probs is already a softmax distribution. Our pipeline consumes that output and derives the features from it. Proof, straight from scripts/arbiter/generate_arbiter_dataset.py (identical in the live demo's expert_features):

probs     = r.probs                          # the Classify head's softmax output
top5_conf = [float(c) for c in probs.top5conf.tolist()]   # top-5 softmax probabilities
full      = probs.data.cpu().numpy()         # the full softmax vector (sums to 1)
full      = np.clip(full, 1e-12, 1.0)        # guard log(0)
entropy   = float(-(full * np.log(full)).sum())           # H = -Σ p ln p
margin    = float(top5_conf[0] - top5_conf[1])        # p1 - p2

So g_conf1..5 / i_conf1..5 are softmax probabilities, *_entropy is the entropy of the softmax vector, *_margin is the gap between its top two, and i_p_background is the softmax mass on the background class. All 20 features are functions of these two softmax distributions, never the raw logits.

Why XGBoost, backed by data

A Random Forest looked great on training data but collapsed on held-out data: it memorised. XGBoost gave essentially the same test accuracy with a tiny generalisation gap, so it was chosen.

Those figures are from the earlier router study; the current V2 arbiter keeps XGBoost and reaches routing accuracy 96.07% and ROC-AUC 0.973 on the rebuilt dataset (with the i_p_background feature).

Training code

# scripts/arbiter/train_arbiter_xgb.py  (essentials)
import xgboost as xgb, pandas as pd

df = pd.read_csv("datasets/arbiter_dataset.csv")   # both models over val+test
y  = (df.domain == "israeli").astype(int)        # route_to_israeli target

# features: each model's conf/entropy/margin + interaction deltas
X = df[["g_conf1","g_entropy","g_margin",
        "i_conf1","i_entropy","i_margin",
        "conf_gap","conf_ratio","entropy_gap","margin_gap"]]

train = df.split == "val"                          # train on val rows
test  = df.split == "test"                         # evaluate on test rows

clf = xgb.XGBClassifier(
    n_estimators=400, max_depth=4, learning_rate=0.05,
    subsample=0.8, colsample_bytree=0.8,
    scale_pos_weight=0.5,                         # Israeli minority ~7%
    eval_metric="auc", early_stopping_rounds=30)
clf.fit(X[train], y[train], eval_set=[(X[test], y[test])])
clf.save_model("scripts/arbiter/arbiter_xgb.json")

Results (V2 arbiter)

ROC-AUC in one line: the probability that the arbiter gives a randomly chosen Israeli image a higher P(israeli) than a randomly chosen global image. 1.0 is perfect separation, 0.5 is a coin flip. 0.973 means it ranks the two domains correctly 97% of the time. Recall = the share of each domain it actually catches.

CNN Visualizer: what the layers do to one image

This is a live, step-by-step view of how a convolutional classifier turns pixels into a class. Pick an input, then press Play. The same logic runs inside the YOLO11 backbone, just with many more filters and layers.

System Error Analysis

Every error of the full routed system was decomposed on the held-out test set (11,265 images). This separates errors a better arbiter could fix from errors no arbiter can fix. Reproduce with scripts/arbiter/system_analysis.py.

Error decomposition

Accuracy by domain

How to read this chart. Each bar is the system's top-1 accuracy on a slice of the test set. "Global domain" = international foods, "Israeli domain" = the 13 local dishes, "Overall system" = everything combined. The reference levels are the always-global baseline (78.0%, what you get without an arbiter) and the oracle ceiling (88.7%, a perfect arbiter). The overall system at 83.8%, between those, is the real arbiter's contribution. Israeli is lower (60.8%) because those dishes are the hardest to route.

The numbers behind the bars (test split, 11,265 images)

How the math works. Each bar is simply correct images divided by total images for that slice:

Global   : 8,641 / 9,960  = 86.8%
Israeli  :   794 / 1,305  = 60.8%
Overall  : 9,435 / 11,265 = 83.8%   (= (8,641 + 794) / (9,960 + 1,305))

Always-global baseline : 8,784 / 11,265 = 78.0%   (global model alone on every image;
                                                    it scores 0 of 1,305 Israeli images)
Oracle ceiling         : 9,988 / 11,265 = 88.7%   (= 8,784 global + 1,204 Israeli that
                                                    at least one expert got right)

For each domain, three bars: the always-global baseline (the old global model used alone), the realized system (global + Israeli experts + arbiter), and the oracle ceiling (the best either expert could do). The story is visible at a glance: on the Israeli slice the baseline is 0% (the global model knows no Israeli dishes), so the Israeli expert plus the arbiter is what creates the 60.8%; on the global slice the system pays a tiny 1.4-point cost (86.8% vs 88.2%) for occasionally misrouting; overall the system beats the baseline by 5.8 points and sits 4.9 points below the oracle.

What still limits the overall number. The gap from the green system bars up to the gold oracle line is the routing error (a correct expert existed but was not chosen), worth about 4.9 points; the gap from the oracle up to 100% is the both-experts-wrong pool, which no router can fix. The Israeli domain has the largest oracle gap (60.8% to 92.3%), so it is where a better arbiter would help most.

How decisively the arbiter separates the two domains

How to read this chart. For every test image the arbiter outputs one number: P(israeli), its estimated probability that the image belongs to the Israeli expert (0 on the left, 1 on the right). The histogram counts how many images fall at each value, on a log scale. Blue bars are images that are truly global, purple are truly Israeli. A good arbiter pushes blue to the left (low P) and purple to the right (high P). The dashed line at 0.5 is the decision threshold: anything to its right is sent to the Israeli model. The clean separation (blue piled near 0, purple piled near 1) is what a 0.93 ROC-AUC looks like; the small amount of blue and purple that overlap in the middle is exactly the routing the arbiter cannot get perfectly right. (This histogram illustrates the V1 arbiter; V2's i_p_background feature sharpened the separation further, lifting AUC to 0.973 and shrinking that middle overlap.)

Case study: the sufganiyah vs donuts confusion, solved

At the previous review this pair was one of the worst failures in the whole system. A sufganiyah (Israeli jam doughnut) and an American donut look almost identical, and the old single-model setup constantly mixed them up. The two-expert split plus a clean visual cue fixed it.

Source: old confusion from FINAL_REPORT_v3_results.csv, current from datasets/system_evaluation.csv (test split). This is a concrete lesson from the prior tests carried into the final model, exactly the kind of improvement the review asked us to surface.

Where the arbiter still misses

It handles most Israeli dishes well (sufganiyah 89%, jachnun 83%, shakshuka 76%, baklava 72%, hummus 69%) but struggles where a dish strongly resembles a global food: sabich 14%, shawarma 26%, falafel 43%. The dominant single confusion is falafel → crab_cakes (60 images). The limiting factor is visual feature overlap, not the arbiter logic.

Validation Plan and Risk Coverage

Risk → test → evidence

Product vs engineering validation

Weight estimation: approaches measured and rejected

BLE packet format: weight = low_byte + high_byte × 256, verified by a checksum over bytes 2 to 6. Example: AC 05 00 02 14 00 CE E4 decodes to 532 g. The full reverse-engineering story is in the BLE Scale tab.

Reverse-Engineering the BLE Scale

The weight is read straight off the scale's Bluetooth Low Energy broadcast, so there is no camera, no OCR, and no vision error in the measurement path. The scale is a generic commercial unit with no public protocol, so we recovered the packet format ourselves. This tab documents how.

Step 1 · Find the device

Using the nRF Connect app on a phone, we scanned for nearby BLE advertisers. The scale shows up only under a generic name, so we identified it by signal strength (RSSI): with the phone held against the scale, the correct device is the one whose RSSI jumps toward 0 dBm while every other advertiser stays weak. Stepping away and watching that one value drop confirmed the match.

The device advertises as SWAN and streams weight on the notify characteristic 0000ffb2-0000-1000-8000-00805f9b34fb. Subscribing to it delivers a fresh 8-byte packet every time the reading changes.

Step 2 · Make it talk by changing the input

With the notifications logging, we treated the scale as a black box and changed one variable at a time: place a known mass, record the packet; change the mass, record again. The bytes that moved with weight, and the way they moved, revealed the encoding.

Step 3 · The packet layout we recovered

low  = packet[4]                          # grams 0..255
high = packet[5] | packet[3]               # number of 256s (carry sits in byte 3)
weight = low + high * 256                  # little-endian, grams

Worked example for AC 05 00 02 14 00 CE E4: low = 0x14 = 20, high = 0x00 | 0x02 = 2, weight = 20 + 2 × 256 = 532 g.

Step 4 · Robust decode in software (a lesson from a real bug)

A live scale flickers as it settles, and the high byte can momentarily read wrong during a refresh, which would make a 532 g reading drop to 20 g for one frame. Our first attempt held the high byte with a sticky latch. That was a mistake: a single transient overshoot (pressing down so it briefly reads above 255 g) latched a permanent +256 g offset that never cleared, so a load removed and replaced read double (200 g, then 400 g). The latch also refused to return to 0 if the scale stopped sending a clean zero packet.

from bleak import BleakScanner, BleakClient
from collections import deque
NOTIFY_UUID = "0000ffb2-0000-1000-8000-00805f9b34fb"
raw = deque(maxlen=5)

def on_notify(sender, data):
    b = list(data)
    if len(b) < 8: return
    low  = b[4]
    high = b[5] | b[3]                   # full high byte, no mask (see odd-256 bug)
    raw.append(low + high * 256)
    weight = sorted(raw)[len(raw) // 2]   # median: no latch, returns to 0, rejects spikes
    # ... push to UI, mark STABLE when last 10 samples vary ≤ 2 g

async def run():
    dev = await BleakScanner.find_device_by_name("SWAN")
    async with BleakClient(dev) as c:
        await c.start_notify(NOTIFY_UUID, on_notify)

The weight-calculation state machine (per packet)

Every notification runs through the same fixed pipeline before it becomes a gram value shown to the user. This is the state machine that computes the weight: validate → decode → median → calibrate → stability → emit. A frame that fails validation is dropped, not shown.

Step 4b · Calibrating the scale: from a 16% span error to a specified instrument

A digital scale is only trustworthy if its reading matches true mass, and calories scale linearly with weight, so any weight error passes straight into the calorie figure. Testing this BLE scale against reference masses exposed a fault: it read a steady ~16% low across the whole range. Crucially the error was systematic, not random: repeated placements of the same mass agreed to within a few grams, which means it is a multiplicative span error, and systematic errors are correctable in software.

The experiment. Ten reference masses from 21 g to 1062 g (water measured on an accurate scale) were each read five times on the scale's own LCD, at a fixed centred placement (a spreader plate removes the single-load-cell eccentricity effect). To test that the fit generalises rather than curve-fitting water, we then measured ten completely different everyday objects (phone, battery, glass, remote, cardboard, camera, toys) as a held-out validation set. Both are plotted below: blue = fit set, amber = held-out objects. A perfect scale would lie on the dashed y = x line; every point sits above it, because the scale reads low, and it does so by a constant proportion - which is why both sets fall on the same straight line through the origin.

The regression (exactly what we computed). We want a correction that maps each raw reading ŷ onto the true mass. Every candidate is scored by its mean absolute error (MAE) over the n = 10 reference masses - the average gap between the corrected reading and the truth:

We fit two candidates to the same 10 masses by least squares and compared their MAE:

The two fitted models are tied on error (4.9 vs 5.1 g), but the two-parameter line carries a spurious +2.846 g intercept that over-reads light items (it would turn a true 0 g into 2.85 g, and a 53 g portion into 64 g). A load-cell span error is physically multiplicative, so the honest model is forced through the origin (a = 0), and its single gain k is the least-squares through-origin slope - literally the two sums from our data divided:

The result. The correction collapses the error from a 16% span to a flat few grams across the entire range. The chart below shows absolute error before and after: the raw error grows linearly with load (the signature of a span error, reaching 150 g at 1 kg), while the corrected error stays near zero and no longer trends with mass.

Formula output vs the truth (the whole calibration in one picture). The most honest way to show the engineering is to plot, for every object, three numbers at once: the raw reading the scale actually gave (before any software touched it), the exact output of our formula 1.178 × raw, and the true weight (the dashed diagonal, where reading = truth). The raw readings sit visibly below the diagonal - the scale under-reports - while the formula output lands on it. This is measured, corrected, and verified against ground truth in a single view: it is a real correction of a physical instrument, not a number we invented.

Zoomed to the light end (0 to 130 g). On the full-range plot above the lightest items are crushed into the corner, so here is the same data zoomed in. Even a 5 g object follows the exact same relationship - raw below the truth, formula output on it - and the 3 g item is the dead-zone case: the scale reads 0, so both the raw point and the formula output sit on the axis while the true weight is 3 g (no multiplicative fix can lift a zero).

Held-out validation: the water-fit constant generalises to unseen objects

The constant above was fit on water and food only. To prove it captures the scale's true physical span error rather than a water-specific curve, we applied the unchanged k = 1.178 to ten objects it never saw - a phone, a battery, a drinking glass, a remote, cardboard, a camera and some toys. It predicts their true mass to a mean absolute error of just 1.9 g (mostly within ±1-3%), across metals, glass, plastic and card. This is a genuine generalisation test, and refitting the slope on all 20 objects barely moves it (1.17804 → 1.17825), so the shipped constant is left unchanged.

Step 5 · The connection state machine

The BLE driver is a finite state machine: it discovers the scale, subscribes, streams and decodes packets, judges stability, and self-heals if the stream goes silent. Every packet is validated (length ≥ 8, AC header, and the additive checksum over bytes 2..6) before it is decoded, so a corrupt frame is dropped rather than shown.

Live Demo: the End-to-End Application

The demo ties everything together: a webcam frame becomes a food class, a BLE weight, and a full nutrition breakdown, live. It runs the current Global and Israeli models behind the XGBoost arbiter, with a Gemini vision fallback for the cases the experts cannot handle. Every design choice below is made for one goal: a correct result in the room, under any camera and any light, as often as the score promises. Source: scripts/demo/caleyez_demo.py.

The pipeline per capture

Preprocessing, step by step (animated)

This is exactly what happens to the pixels between the webcam and the network. Press play to watch one frame travel through the four preprocessing stages and into both experts.

What we deliberately do NOT do (lessons from testing)

The preprocessing is intentionally minimal. Several "smart" steps were tried and then removed because measurement showed they hurt. Keeping the record is the honest engineering story.

Why a fixed center ROI

The crop is a fixed central square (the on-screen guide box), not an automatically detected plate.

The deeper finding: camera angle, not background

Testing exposed something more fundamental than clutter. The Global model classifies a bell pepper at 99.97% even when it fills only a quarter of a dark-scale or wood background, so background and crop size are not what break it. What breaks it is the viewing angle: the training images are almost all side or eye-level views, while a top-down demo camera sees a red disc with a central stem, which is genuinely out of distribution and can read as chocolate cake.

Wiring the real arbiter (not a re-implementation)

The demo runs the V2 Israeli model (14 classes: the 13 dishes plus background) and computes the same 20 features the arbiter was trained on, in the same order, then loads the exact saved model. The 20th feature is i_p_background, the Israeli model's "this is not one of my dishes" probability, which is now the arbiter's single most important feature.

g = expert(global_model, roi, imgsz=320)   # top-5 conf, entropy, margin
i = expert(israeli_model_v2, roi, imgsz=224) # also reads P(background)
X = [g_conf1..5, g_entropy, g_margin,
     i_conf1..5, i_entropy, i_margin, i_p_background,
     conf_gap, conf_ratio, entropy_gap, margin_gap, both_unsure]
p_israeli = arbiter.predict_proba(X)[1]     # scripts/arbiter/arbiter_xgb.json (V2, 20 feats)
route_to_israeli = p_israeli >= 0.5

The Gemini fallback: covering the cases no expert can win

About 11% of system errors are images where both experts are wrong (the both-wrong pool from the System Analysis tab). No amount of routing fixes those. The demo detects that situation and asks a cloud vision model instead, which is the honest way to push realized accuracy past the local ceiling on hard inputs.

The two gates differ on purpose: the Global model spreads probability over 132 classes, so a modest top-1 is still meaningful, while the Israeli model has only 13 classes and must clear a higher bar to be believed. When either trigger fires, the ROI crop is sent to Gemini and its label drives the nutrition lookup. The UI marks these results so the source is never ambiguous.

UI and UX choices

The Web App: Edge AI in the Browser

The same two-expert pipeline that runs on the desktop also runs, unchanged, inside a phone browser - no install, no app store, no server doing the inference. A judge scans a QR code and the two CNNs plus the XGBoost arbiter execute on their own device. This tab documents how a browser becomes an edge-AI runtime, how it uses the device's CPU/GPU and memory, where the cloud is and is not used, and the hard limits of the platform. Repo: caleyez-web (public, GitHub Pages). Live: raz-dv-ee.github.io/caleyez-web · engineering console: /engineering.html · latency bench: /bench.html.

How a browser runs a neural network on the edge

The two YOLO11l-cls experts are exported to ONNX (the same models the desktop ONNX build uses) and executed with ONNX Runtime Web (onnxruntime-web). ORT-Web ships the runtime as a WebAssembly (WASM) module, so the network's operators run as near-native compiled code inside the browser sandbox - this is what makes real CNN inference on a phone practical rather than a toy.

fp16 vs fp32, and why it matters here

A neural network is millions of numbers (the weights). fp32 ("single precision") stores each weight as a 32-bit floating-point number; fp16 ("half precision") stores it in 16 bits. Half the bits means half the memory and faster movement through the processor, but lower numeric precision: fp16 can represent far fewer distinct values, so very small differences get rounded away. We publish the models in fp16 so each download is roughly half the size (about 25 MB instead of about 50 MB) and they sit comfortably in phone memory. The two backends then treat those fp16 weights differently, and that is the crux of our default:

So the phone downloads small fp16 files either way; on the default CPU path each weight is widened to fp32 for the computation. Widening does not recover the bits fp16 dropped - a widened fp16 value is the same rounded number with trailing zeros, no new information - it just stops the millions of multiply-adds from re-rounding at every step, which is where fp16 compute actually loses accuracy. The only difference left from desktop is a ~10⁻³ probability drift from rounding the weights to fp16 once at export; the decision is identical. That single distinction is why CPU/WASM is the default and WebGPU is opt-in.

What a “mantissa” is, and the multiply that overflows it

A floating-point number is stored as sign × mantissa × 2^exponent. The mantissa holds the significant digits, and the number of mantissa bits is what sets the precision. fp16 has 10 mantissa bits (~3-4 decimal digits); fp32 has 23 (~7 digits). The exponent handles magnitude, so fp16 isn’t short on range (it reaches 65,504) - it’s short on digits. That is the “room to grow” you need: not room for bigger numbers, room for more significant digits.

Watch it in one multiply-accumulate - the operation a CNN does millions of times. Multiply a weight by an activation, each stored to ~4 digits (fp16):

weight 1.234  ×  activation 5.678  =  7.006652   ← the exact product needs 7 digits
   fp16 (10-bit mantissa) → rounds back to 7.007     ← the tail .000652 is chopped
   fp32 (23-bit mantissa) → keeps       7.006652     ← all 7 digits fit

In bits: two 10-bit mantissas multiply into a result up to ~20 bits wide - fp32’s 23-bit mantissa has room to hold it, fp16’s 10-bit does not, so fp16 rounds it away at every step. That is exactly your intuition - the product of two 16-bit numbers wants ~32 bits, and only fp32 keeps them. A layer sums thousands of these products, so in fp16 the chopped tails compound; in fp32 they stay. Widening the fp16 weights to fp32 before computing gives each intermediate product that room - which is the whole point of computing at higher precision than you store.

One weight, followed end to end

The whole story on a single number - how an fp32 weight is cut to fp16, padded with zeros, and how the calculation refills those zeros and still lands on the same answer as the full model:

So the light fp16-shipped model reaches the identical decision as the heavy fp32 model - the only thing that differs is the third decimal of the probability, which never decides an answer. That is why we can ship the small file and still be desktop-exact where it counts.

The options we actually weighed

Two independent choices: what format to ship the file in, and what precision to do the math in. Here is every end-to-end option and what it would cost:

Memory and CPU: what the phone actually does

Two fp16 ONNX models are downloaded once: Global ≈25 MB (320², 132 classes) and Israeli ≈25 MB (224², 13 classes + background) - ≈50 MB total, then HTTP-cached so later runs are instant. At run time WASM upcasts fp16→fp32, so a few hundred MB of working memory is touched during inference; the app targets devices with ≥2 GB RAM.

The CPU matters as much as the memory. Because the default path runs on WebAssembly, the actual matrix maths happens on the phone's CPU, so the processor's speed sets the inference time directly: a recent flagship CPU finishes both models in about 0.6 s, while an older or budget CPU is proportionally slower for the exact same work. This is why the models are sized deliberately (fp16 weights, 320/224 inputs, single-threaded WASM) to stay interactive on mid-range phones rather than only flagships, and why the app prints the measured time on every result so the device's real capability is visible rather than assumed. A phone with plenty of RAM but a weak CPU will still run correctly, just with a longer wait per photo.

function arbiterP(f){              // f = the 20 features
  let m = ARB.base;                // start at the base log-odds (-1.2795)
  for (const tree of ARB.trees){
    let n = tree["0"];             // root node
    while (n.length > 1)           // length 4 = internal, length 1 = leaf
      n = tree[ Math.fround(f[n[0]]) < n[1] ? n[2] : n[3] ];  // [feature,threshold,yes,no]
    m += n[0];                     // add this tree's vote (the leaf value)
  }
  return 1 / (1 + Math.exp(-m));   // sigmoid -> P(israeli)
}

Points Raised in Review and How We Addressed Them

The points raised at the interim review, each paired with the concrete change made in response. Nothing here is rhetorical; every response points at a tab with data.

Model and system architecture

Testing and validation

Israeli Model V2 (data + open-set retrain)

V2 attacks the system's biggest weakness: on the Israeli domain the model can reach 92.3% (its test-set ceiling), but the system only realizes 60.8%. The ~31-point gap is almost all routing error, because the original Israeli model is overconfident on non-Israeli food, so the arbiter cannot trust its confidence. V2 fixes the data, not the hyperparameters.

System results: before vs after V2 (the bottom line)

Same two models' label spaces, same test protocol, same oracle ceiling (~88.8%). The entire gain comes from the open-set retrain making the Israeli model's confidence trustworthy, recovering images the arbiter used to misroute. Full method below; the same numbers also appear in the Arbiter tab.

Change 1 · More data for the data-starved weak classes

We targeted classes by accuracy, not image count (jachnun had the fewest images yet scored 100%, so it was not the problem). Real images were collected, then de-duplicated byte-for-byte.

Change 2 · A new "background" class (open-set recognition)

The decisive change. We added a 14th class, background, built from the non-Israeli foods the model used to mistake for Israeli dishes. It lets the model answer "this is not one of my dishes" instead of confidently forcing one of the 13. Crucially it is sourced split-aligned: background train from the global train split, background val from global val, background test from global test. So a background test image is held out from both models (the global model never trained on it either), keeping every evaluation honest.

The look-alikes were chosen from the data: across the test set, 132 of the global images the arbiter misrouted to the Israeli expert were called hummus (creamy/beige foods like risotto, chicken_curry, macaroni_and_cheese), with smaller "sinks" at samosa (fried/dough), falafel (fried clusters), sufganiyah (donuts) and baklava (pastry). The background class is weighted toward those confusers (25 global classes). It is kept modest (400 train / 75 val / 75 test) so it cannot swallow real dishes.

Change 3 · Byte-for-byte dataset hygiene

The whole 14-class set was hashed (SHA-256) before training. This surfaced and removed a real, pre-existing problem.

Training configuration and the reason for every parameter

Full retrain from the ImageNet-pretrained yolo11l-cls.pt (a fresh head over a pretrained backbone), not a fine-tune. Source: scripts/training/train_israeli_food_v2.py, run israeli_food_yolo11l_v2 (a new directory, so the V1 production weights are never overwritten).

How the background class feeds the arbiter (and why it is NOT a routing row)

The background class is an open-set / calibration device for the Israeli model, not a routing target. Two design rules keep it correct and leakage-free in the arbiter:

The mandatory pipeline after retraining

python scripts/training/train_israeli_food_v2.py        # -> runs/israeli_food_yolo11l_v2
python scripts/arbiter/generate_arbiter_dataset.py     # rebuild features with the new model
python scripts/arbiter/train_arbiter_xgb.py            # retrain the arbiter (now includes the i_p_background feature)
python scripts/arbiter/evaluate_system.py              # honest end-to-end re-evaluation

Honest expectation

Two separate effects, neither of which raises the global model or the 88.4% oracle:

• More weak-class data raises the Israeli model's own ceiling (the 92.3%).
• The background class lets the arbiter realize that ceiling by trusting Israeli confidence, recovering routing-lost images (the ~410-image gap).

We will report per-class accuracy, the routing recall/precision, and the threshold trade-off curve so the gain is shown to be a measured operating point, not an arbitrary default.

Measured results (V2, on the held-out test set)

The retrain completed (best epoch 112). Per-class test accuracy on the classes we targeted improved sharply, and crucially the new background signal cleanly separates the domains.

The decisive signal: the Israeli model's mean i_p_background is 0.497 on global food vs 0.029 on real Israeli food, a near-clean "not-mine" separator. Feeding it to the arbiter lifted the whole ROC and recovered routing-lost images.

V2 training curves and confusion matrix

Reading the train-loss vs val-loss graph (what it actually means)

These two curves are the single most important diagnostic in training, and they tell a precise story. Both are the cross-entropy loss (with label smoothing): a number that is small when the model puts high probability on the correct class and large when it is wrong or unsure. They are computed on two different sets of images, and the comparison between them is what matters.

• Train loss (top-left) is measured on the images the model is actively learning from. It falls steadily from about 1.6 toward nearly 0. That only says the model is successfully memorising and fitting its training data, it says nothing on its own about whether it learned anything useful.
• Validation loss (top-right) is measured on held-out images the model never trains on. It is the honest signal of generalisation. Here it drops sharply for the first ~20 epochs, then plateaus around 0.5 and stays flat for the remaining 100 epochs.

This is exactly why we do not deploy the last epoch: best.pt is saved at the epoch with the highest validation top-1 (epoch 112), not the lowest train loss. Selecting on the held-out metric, not the training metric, is what guarantees the shipped model is the best generaliser, and the patience=25 early-stopping rule exists to cut the run if val ever did start to degrade. The jagged look of the val curves is ordinary epoch-to-epoch noise on a finite validation set, not instability; the dotted "smooth" line is just a moving average to make the trend readable.

What the confusion matrix shows

Each cell is the fraction of images of a true class (column) that the model predicted as a given class (row); a perfect model is a pure diagonal. V2's diagonal is strong across all 13 real dishes (0.85 to 1.00: bourekas and jachnun at 1.00, sabich 0.94, malawach 0.93). The one deliberately weaker cell is background at 0.64: it is a broad catch-all of many non-Israeli foods, so it is the hardest class, and about a third of background images are still read as a look-alike dish (creamy foods leak toward hummus, fried toward samosa/falafel). That residual is acceptable because the arbiter does not need background to be the top-1 guess, it uses the probability i_p_background, which is high on those foods even when another class edges it out, and that is enough to route them to the global expert.

System Limitations and Future Work (V2 Architecture)

Limitation 1 · Classification backbone is not Pareto-optimal

The Global and Israeli experts use YOLO11l-cls. In its classification variant this is a legitimate classifier (the detection head is removed), but the C3k2 backbone is co-designed for multi-scale detection feature reuse, which is not the accuracy-per-FLOP optimum for single-label classification. On ImageNet-1k, hierarchical and isotropic classifiers occupy a stronger frontier.

Limitation 2 · The arbiter is information-starved (logits-only features)

The current arbiter consumes only scalar summaries of each expert's softmax: confidence, entropy, and margin. These are a deterministic, lossy function of the penultimate 1280-D embedding. By the data processing inequality, the embedding carries at least as much label-relevant information as any function of it:

I(domain ; conf, entropy, margin)  ≤  I(domain ; embedding_1280)

The information we discard is exactly the visual-feature overlap that defeats margin-based routing (falafel vs crab_cakes, sabich vs global dishes). A multimodal arbiter on the concatenated penultimate embeddings of both experts routes on visual evidence, not probability shape alone.

Limitation 3 · No native Open Set Recognition for out-of-domain inputs

The Israeli expert is overconfident on food it never saw, and we delegated the entire correction to the downstream arbiter. This treats the symptom. The Open Set Recognition literature provides root-cause fixes that force low confidence on non-Israeli inputs natively:

A real attempt: fine-tuning on tagged demo images (and why it failed)

The data-tagging flywheel in the live demo produces labelled real-world captures, so the obvious next step is to fine-tune the Israeli expert on them. We tried it four times on the worst real-world class, hummus, with a safety harness that never touches the production weights, holds out part of the tagged set, and re-validates on the original clean test split. Every run made the model worse, and the regression guard correctly refused to recommend all four.

What would actually fix it

V2 synthesis: which lever moves which number

System Analysis V2 (after the open-set retrain)

This is the full system evaluation after the Israeli Model V2 retrain (the open-set background class plus the i_p_background arbiter feature). The original System Analysis tab is kept unchanged as the V1 baseline; this tab is the current production system, measured on the held-out test set of 11,352 images (regenerated datasets/system_evaluation.csv).

Error decomposition (V2)

Every outcome on the test set, split into the three mutually exclusive buckets that tell you where the remaining errors live.

Accuracy by domain (V2)

V1 vs V2, side by side

Top remaining system confusions (V2)

The misses left after V2, true class to predicted class on the test set. They are dominated by genuine visual look-alikes, which is the both-models-wrong pool, not routing.

Approaches We Tried and Rejected

Before settling on a BLE gravimetric scale for weight, we built and measured two other ways to get the mass of the food. Both worked in the lab and both were rejected for concrete, measured reasons. Keeping this record is the honest engineering story, and it is the direct answer to "why a scale and not just the camera?"

Approach A · Estimating weight from a single photo (volume to mass)

The idea: put a coin of known size in frame as a fiducial (an Israeli 1 shekel coin, 18 mm), use it to convert pixels to centimetres, recover the food's volume, then multiply by density to get grams. We implemented two independent volume methods and validated both against a gold standard.

Method 1 · Monocular depth (MiDaS)

A neural depth network estimates the height of every pixel above the table; summing height over the food's area gives a volume. It got the apple to 139.3 cm³.

Method 2 · Shadow geometry

With a 45° light, an object's height equals the length of its cast shadow. Multiplying the perspective-corrected top area by that height gives a volume without any depth network, purely geometry.

Validation: Archimedes water displacement (the gold standard)

The methods were actually accurate. Against displacement, MiDaS landed within about 7 percent. So the failure below is not that we could not measure volume, we could. The failure is what comes after volume.

Approach B · Reading the kitchen scale's display by OCR

The next idea kept a real scale but avoided pairing it electronically: point the camera at the scale's 7-segment LCD and read the number with OCR (pytesseract, in food_app.py / new_gui_with_ocr.py).

The decision: read mass at the source over BLE

Both rejected routes share one mistake: inferring a physical quantity the hardware already measures exactly. The scale computes grams internally, so instead of photographing its screen we reverse-engineered its Bluetooth Low Energy protocol and read the mass directly (see the BLE Scale tab). No density assumption, no fiducial, no lighting or glare dependence, gram-accurate every time.

Training Lab: how learning actually happens

Three interactive toys that make the training terms concrete: how a model learns by gradient descent (and why the learning rate matters), why batch size changes the path, and what a backbone, a head, and freezing really mean. These are the exact levers behind the Israeli Model V2 retrain and the fine-tune failures.

1 · Gradient descent and the learning rate

A model "learns" by minimising a loss (how wrong it is). It repeatedly nudges its weights downhill along the slope (the gradient) of the loss. The step size is the learning rate (LR). The curve below is a simple loss bowl; the dot is the current weight. Press play and change the LR.

2 · Batch size: why it changes the path

The true downhill direction is computed by averaging over data. A batch is how many images you average before each step. A small batch gives a noisy estimate of the slope (the path jitters); a large batch gives a smooth estimate (a cleaner path) but each step costs more memory. Drag the batch size and watch the descent path on a 2-D loss (the rings are contours).

3 · Backbone, head, and freezing

A classifier has two parts. The backbone is the deep stack that turns an image into a compact feature vector (it holds the general visual knowledge). The head is the small final layer that maps that vector to class scores. Learning flows forward (prediction) then backward (the gradient updates the weights). Freezing the backbone locks it so only the head learns. Toggle freeze and run a step.

4 · Logits, softmax, and what the arbiter reads

The final layer of each model outputs one raw score per class, the logits. They are unbounded (any real number) and do not sum to anything. Softmax turns them into probabilities that sum to 1, by exponentiating and normalising: p_i = e^(z_i/T) / Σ e^(z_j/T). The model's "confidence" is just the largest probability. Drag the logits and watch the probabilities, and the three numbers the arbiter actually consumes: top-1 confidence, margin, and entropy.

Real-World Image Validation

47 real photos of food taken with our own phones, sorted into class folders and run through the exact deployed pipeline offline. The folder name is the ground truth; none of these images were in training. Each tile is the evidence card the pipeline emits (the ROI it saw, both experts’ top-5, the arbiter’s decision). Click any card to enlarge.

The 7 errors, by root cause

falafel✓📷 Apple iPhone 13 · 2023-05-24

french fries✓📷 Google Pixel 10 Pro · 2026-06-07

french fries✓📷 Google Pixel 10 Pro · 2026-06-07

hamburger✓📷 samsung SM-G965F · 2018-09-15

hamburger✓📷 Samsung Galaxy S22 Ultra · 2022-12-06

hamburger✓📷 Samsung Galaxy S22 Ultra · 2022-12-08

hamburger✓📷 Samsung Galaxy S22 Ultra · 2022-12-09

hamburger✓📷 Samsung Galaxy S22 Ultra · 2022-12-09

hamburger✓📷 Samsung Galaxy S22 Ultra · 2023-04-28

hamburger✓📷 Samsung Galaxy S22 Ultra · 2023-04-28

hamburger✓📷 Samsung Galaxy S22 Ultra · 2024-01-02

hamburger✓Forwarded image (EXIF stripped)

hamburger✓Forwarded image (EXIF stripped)

hamburger✓📷 samsung SM-G935F · 2016-11-12

hamburger✓Forwarded image (EXIF stripped)

sabich✓📷 Apple iPhone 13 · 2023-06-25

bourekas cheese✓📷 Google Pixel 10 Pro · 2026-06-06

bourekas cheese✓📷 Google Pixel 10 Pro · 2026-06-06

bourekas cheese✓📷 Google Pixel 10 Pro · 2026-06-06

bourekas cheese✓📷 Google Pixel 10 Pro · 2026-06-06

bourekas cheese✓📷 Google Pixel 10 Pro · 2026-06-06

malawach✗Within-domain look-alike: bourekas and malawach are both golden fried dough; the Israeli expert split two similar pastries.📷 Google Pixel 10 Pro · 2026-06-06

bourekas cheese✓📷 Google Pixel 10 Pro · 2026-06-06

bourekas cheese✓📷 Google Pixel 10 Pro · 2026-06-06

bourekas cheese✓📷 Google Pixel 10 Pro · 2026-06-06

bourekas cheese✓📷 Google Pixel 10 Pro · 2026-06-06

donuts✗Old low-resolution photo; rounded scoops read as ringed donuts.📷 LG G3 ·

ice cream✓Forwarded image (EXIF stripped)

ice cream✓Forwarded image (EXIF stripped)

ice cream✓Forwarded image (EXIF stripped)

canned tuna✓📷 Google Pixel 10 Pro · 2026-06-06

canned tuna✓📷 Google Pixel 10 Pro · 2026-06-06

canned tuna✓📷 Google Pixel 10 Pro · 2026-06-06

ice cream✗Far, top-down shot: the round open can reads as a bowl of ice cream. A framing/scale issue, not lighting.📷 Google Pixel 10 Pro · 2026-06-06

ginger✗Distant, low-contrast pale contents; borderline 0.58 confidence.📷 Google Pixel 10 Pro · 2026-06-06

canned tuna✓📷 Google Pixel 10 Pro · 2026-06-06

canned tuna✓📷 Google Pixel 10 Pro · 2026-06-06

canned tuna✓📷 Google Pixel 10 Pro · 2026-06-06

canned tuna✓📷 Google Pixel 10 Pro · 2026-06-06

canned tuna✓📷 Google Pixel 10 Pro · 2026-06-06

meorav yerushalmi✗Routing miss on a re-shared file: the arbiter sent it to the Israeli expert, which forced a mixed-grill label.📷 Google Pixel 10 Pro · 2026-06-06

chicken quesadilla✗Deliberately hue/temperature-shifted and dim: a cheesy triangle reads as a quesadilla.📷 Google Pixel 10 Pro · 2026-06-06

pizza✓📷 Google Pixel 10 Pro · 2026-06-06

chicken quesadilla✗Same induced degradation (strong blue cast).📷 Google Pixel 10 Pro · 2026-06-06

pizza✓📷 Google Pixel 10 Pro · 2026-06-06

pizza✓📷 Google Pixel 10 Pro · 2026-06-06

pizza✓📷 Google Pixel 10 Pro · 2026-06-06

Failure modes and how I would fix them

CalEyeZ vs the Market - a calorie face-off

We put CalEyeZ head-to-head against the three tools people actually use, on the same physical meals: MyFitnessPal (manual database logging), Cal.ai and FoodVisor (AI photo estimators). Ground truth is the product label wherever one exists. Every number below is a real trial recorded in scripts/eval/calorie_comparison.xlsx. This is an honest pilot (small n), and we say exactly where each number comes from.

Mean absolute calorie error over the 8 meals all three tools were tested on. CalEyeZ is the most accurate - despite MyFitnessPal being handed the true weight and a hand-picked database entry, and Cal.ai being a giant cloud model.

The decisive test: does the calorie number follow the actual portion?

The whole game is portion. The photo apps emit a typical serving; MyFitnessPal offers preset units ("1 slice", "1 can"). Both are right only when your food happens to match their assumption. CalEyeZ measures the grams on a scale, so it tracks the real portion every time. We proved it with one food, two portions:

Why CalEyeZ Is Electronic Engineering, Not Just Software

A common first reaction is "this is a software project with a camera." It is not. CalEyeZ is a closed loop from the physical world, through silicon, back to a physical-world decision. Take away any one layer and a real subsystem disappears. Here is that argument laid out.

The pipeline, read as electronic-engineering stages

Each course → its project → the CalEyeZ code it produced

This is the hard proof. For every course: the project I built in it, and the actual code in our final project that the course put there. Not a name and a checkmark, the real lines.

# scripts/demo/caleyez_demo.py - the module library + a hardware device
import cv2, numpy as np, pandas as pd
from PIL import Image, ImageTk
from bleak import BleakScanner, BleakClient   # BLE hardware, in pure Python
...
self.cap = cv2.VideoCapture(0)   # a physical camera
self._tick_camera()                # live frame loop
self._tick_weight()                # live weight loop from the BLE scale

# scripts/demo/caleyez_demo.py - a class that inherits and overrides, and the app object
class AutocompleteEntry(tk.Entry):                 # inherits a base class
    def __init__(self, parent, textvar, options, **kw):
        super().__init__(parent, textvariable=textvar, **kw)
class CalEyeZDemo:
    def __init__(self, root):
        self.frames = deque(maxlen=6)   # fixed-size buffer = the array/struct discipline from C

# scripts/training/train_general_model.py - CUDA GPU + FP16
results = model.train(
    device = 0,          # the CUDA GPU
    amp    = True,       # mixed precision = the FP16 path from the floating-point lecture
)
# scripts/demo/caleyez_demo.py - a GPU forward pass (the "Deep Learning on GPU" lecture)
g = model_g.embed(roi, imgsz=GLOBAL_IMGSZ)[0]  # 512-D penultimate features, on the GPU

// attention.cu - the optimized fused QKᵀ kernel
#define TILE 32                          // = NVIDIA warp size → coalesced access
__global__ void qkt_fused_opt(const float* Q, const float* K, float* S, float scale){
  __shared__ float Qs[TILE][TILE], Ks[TILE][TILE];
  Ks[ty][tx] = (col < N && kK < D) ? K[col * D + kK] : 0.0f;  // virtual transpose + guard
  for (int k = 0; k < TILE; ++k) acc += Qs[ty][k] * Ks[k][tx];
  if (row < N && col < N) S[row * N + col] = acc * scale;   // KERNEL FUSION
}

# scripts/training/train_israeli_food_v2.py - supervised deep learning + calibration
model = YOLO("yolo11l-cls.pt")
results = model.train(
    data=DATASET_DIR, task="classify",
    epochs=120, patience=25,        # early-stop on val top-1 = no overfitting
    label_smoothing=0.1,           # calibrated, trustworthy probabilities
)

# scripts/arbiter/train_arbiter_xgb.py - a boosted decision-tree ensemble + honest eval
clf = xgb.XGBClassifier(n_estimators=400, max_depth=5, learning_rate=0.05,
    scale_pos_weight=spw, eval_metric="auc")        # class imbalance handling
auc        = roc_auc_score(yte, proba)            # characterise the model
routed_acc = routed_correct.mean()                # vs an always-global baseline…
oracle_acc = (g_corr | i_corr).mean()             # …and the perfect-router ceiling

# scripts/ble/scale_reader.py - raw transducer bytes → grams, with a stability spec
low_byte  = hex_data[4]
high_byte = hex_data[5] | hex_data[3]          # full high byte = number of 256s (no mask)
weight = sorted(raw_buffer)[len(raw_buffer)//2]      # MEDIAN filter = noise vs responsiveness
delta  = max(stability_buffer) - min(stability_buffer)
software_stable = delta <= STABILITY_THRESHOLD        # ≤2 g over 10 samples ⇒ STABLE

# scripts/demo/caleyez_demo.py - 2-D image filtering before recognition
def center_roi(bgr):
    h, w = bgr.shape[:2]
    s = int(min(h, w) * ROI_FRAC)        # deterministic 2-D crop
    y0, x0 = (h - s)//2, (w - s)//2
    return bgr[y0:y0+s, x0:x0+s].copy()
# scripts/ble/scale_reader.py - the SAME course's 1-D filtering, on the sensor signal
weight = sorted(raw_buffer)[len(raw_buffer)//2]   # median = a non-linear 1-D digital filter

Recap: course → subsystem → proof

Per-Class Performance

The 86.16% system figure is an average. This view breaks it down to every one of the 145 classes: its recall on the held-out test set, the food it is most confused with, and why. This is where the real limitations live.

Reading it: recall = the share of that class's test images identified correctly, so 75% recall means 25% were missed. But recall on a tiny test set is noisy: a class with only 8 images moves 12.5% per single miss, so those numbers carry a wide 95% confidence interval and should not be ranked literally. Use reliable only (n ≥ 50) to see the weaknesses that are statistically real.

The Voice-Fingerprint Analogy

CalEyeZ is a convolutional neural network, but you can understand exactly what it does without any deep-learning vocabulary - just signals and filters. Imagine building a system that recognises who is speaking. That system and CalEyeZ are the same machine; only the input changes from a voiceprint to a food photo.

The story, step by step

Adding a second accent - the catastrophic-forgetting problem

Now suppose we want the same system to also recognise Israeli speakers. They use the same physics of speech, but the statistics differ - louder on average, different cadence and formants. We have three options:

CalEyeZ took option 3. The Israeli specialist even carries an extra "background / not-mine" class, so when an English voice reaches it, it can say "this isn't mine" - which is the strongest signal the router uses to send the input back to the English expert.

The reverse case is just as important: feed an English food and the Israeli expert returns a high P(background) ("not mine") with low confidence, so the router sends the decision to the Global expert instead. That single "not-mine" signal is the top feature in the real arbiter and is why routing reaches ROC-AUC 0.973.

How this maps onto CalEyeZ - exactly

Want to see the softmax math live (drag the logits and watch the probabilities, entropy and margin update)? Open the Training Lab. Want the routing decision explained per image? See the Arbiter tab.