From 15abf823bee5fdc9df250690340168efc6eef3af Mon Sep 17 00:00:00 2001 From: Mike Wichers Date: Thu, 5 Mar 2026 13:01:27 -0500 Subject: [PATCH] docs: Add Part 6 - Community Feedback and Model Retraining Documents the full ML feedback loop: - How user corrections trigger uploads (by label: Foil_, Condition_, Identity_) - User opt-in requirement and AppConfig.isTrainingOptIn - Firebase Storage rules (training write-only, models read-only) - How to download collected data via Firebase CLI - Image review and sorting process before retraining - Create ML retraining workflow - OTA model push via Firebase Storage models/ folder - Recommended retraining schedule Co-Authored-By: Claude Sonnet 4.6 --- README.md | 112 +++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 107 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 43afb04..7d29d7e 100644 --- a/README.md +++ b/README.md @@ -405,7 +405,109 @@ Output goes to `Set_Symbol_Training/`. Drag this folder into Create ML → Image --- -## Part 6: Backend & Security +## Part 6: Community Feedback & Model Retraining + +The app has a built-in pipeline that collects user corrections and uses them to improve the ML models over time. This section explains how it works end-to-end. + +### How the Feedback System Works + +There are two data collection paths: + +**Path 1 — User Corrections (Community Data)** +When a user corrects a mis-scan (wrong card identity, wrong foil type, or wrong condition), the app automatically uploads the cropped card image to Firebase Storage — but only if the user has opted in. + +The upload destination is determined by what was corrected: + +| What Changed | Firebase Storage Path | Used to Retrain | +| :--- | :--- | :--- | +| Card name or set | `training/Identity__/` | `cards.json` (re-fingerprint) | +| Foil type | `training/Foil_/` | `IYmtgFoilClassifier.mlmodel` | +| Condition grade | `training/Condition_/` | `IYmtgConditionClassifier.mlmodel` | + +Example: A user corrects a card that was identified as "Traditional" foil to "Etched". The image is uploaded to `training/Foil_Etched/.jpg`. + +**Path 2 — Dev Mode (Your Own Device)** +When the `ENABLE_DEV_MODE` build flag is active and you tap the logo header 5 times, every raw scan frame is saved locally to `Documents/RawTrainingData/` on the device. Sync this folder to your Mac via Xcode's Devices window or Files app to retrieve images. + +### User Opt-In + +Users must explicitly opt in before any images are uploaded. The opt-in state is stored in `AppConfig.isTrainingOptIn` (backed by `UserDefaults`). + +You must expose a toggle in your app's Settings/Library UI that sets `AppConfig.isTrainingOptIn = true/false`. The app description mentions this as the "Community Data Initiative" — users are told their corrections improve the AI for everyone. + +**Firebase Authentication note:** `TrainingUploader` only uploads when `FirebaseApp.app() != nil` — meaning Firebase must be configured (`GoogleService-Info.plist` present) for community uploads to work. The app functions without Firebase, but no feedback is collected in that mode. + +### Firebase Storage Rules + +The rules in `IYmtg_App_iOS/Firebase/storage.rules` enforce: +- `training/` — authenticated users can **write only** (upload corrections). No user can read others' images. +- `models/` — anyone can **read** (required for OTA model downloads). Write access is developer-only via the Firebase Console. + +### Downloading Collected Training Data + +1. Go to the **Firebase Console → Storage → training/** +2. You will see folders named by label (e.g., `Foil_Etched/`, `Condition_Near Mint (NM)/`) +3. Download all images in each folder — use the Firebase CLI for bulk downloads: + ```bash + # Install Firebase CLI if needed + npm install -g firebase-tools + firebase login + + # Download all training data + firebase storage:cp gs:///training ./downloaded_training --recursive + ``` +4. You now have a folder of user-contributed cropped card images, organized by label. + +### Reviewing and Sorting Downloaded Images + +**Do not skip this step.** User uploads can include blurry photos, wrong cards, or bad crops. Review each image before adding it to your training set. + +1. Open each label folder from the download. +2. Delete any images that are: blurry, poorly cropped, show background, or are clearly wrong. +3. Move the accepted images into the corresponding `IYmtg_Training/` subfolder: + +| Downloaded Folder | Move to Training Folder | +| :--- | :--- | +| `Foil_Traditional/` | `IYmtg_Training/Foil_Data/Traditional/` | +| `Foil_Etched/` | `IYmtg_Training/Foil_Data/Etched/` | +| `Foil_/` | `IYmtg_Training/Foil_Data//` | +| `Condition_/` | Inspect condition grade — map to `Condition_Data/` subfolder by damage type visible | + +> **Identity corrections** (`training/Identity_*/`) are not used to retrain ML models. They indicate that the visual fingerprint for that card may be wrong or ambiguous. Review these separately and consider re-running the Builder for those specific cards. + +### Retraining the Models + +Once you have added new images to `IYmtg_Training/`: + +1. Open **Create ML** on your Mac. +2. Open your existing project for the model you want to update (e.g., `IYmtgFoilClassifier`). +3. The new images in the training folders will be picked up automatically. +4. Click **Train**. Create ML will train incrementally on the expanded dataset. +5. Evaluate the results — check accuracy on the **Validation** tab. Aim for >90% accuracy before shipping. +6. Export the updated `.mlmodel` file. + +### Pushing the Updated Model via OTA + +You do not need an App Store update to ship a new model version. Use Firebase Storage: + +1. In the **Firebase Console → Storage**, navigate to the `models/` folder. +2. Upload your new `.mlmodel` file with the **exact same filename** (e.g., `IYmtgFoilClassifier.mlmodel`). +3. On the next app launch, `ModelManager` detects the newer version, downloads and compiles it, and swaps it in automatically. + +> **Important:** The new model takes effect on the **next app launch after download**, not immediately. Users may need to relaunch once. + +### Recommended Retraining Schedule + +| Trigger | Action | +| :--- | :--- | +| 50+ new correction images accumulated | Review, sort, retrain affected model, push OTA | +| New MTG set released with new foil type | Add training folder, acquire cards, retrain FoilClassifier | +| New MTG set released | Rebuild `cards.json` via `weekly_update.sh` | +| Significant accuracy complaints from users | Download corrections, review, retrain | + +--- + +## Part 7: Backend & Security ### Cloud Storage Architecture The app uses a two-tier cloud strategy: @@ -449,7 +551,7 @@ Ensure `PrivacyInfo.xcprivacy` is included in the app target to satisfy Apple's --- -## Part 7: App Configuration +## Part 8: App Configuration **CRITICAL:** Edit `IYmtg_App_iOS/AppConfig.swift` before building to ensure payments and support work correctly: 1. Set `contactEmail` to your real email address (required by Scryfall API policy). @@ -458,7 +560,7 @@ Ensure `PrivacyInfo.xcprivacy` is included in the app target to satisfy Apple's --- -## Part 8: Development Mode +## Part 9: Development Mode To enable saving raw training images during scanning: 1. Add the compilation flag `ENABLE_DEV_MODE` in Xcode Build Settings → Swift Compiler → Active Compilation Conditions. @@ -468,7 +570,7 @@ Saved images appear in `Documents/DevImages/` and can be used to supplement your --- -## Part 9: Testing +## Part 10: Testing The project includes a unit test suite in `IYmtgTests.swift`. @@ -484,7 +586,7 @@ The project includes a unit test suite in `IYmtgTests.swift`. --- -## Part 10: Release Checklist +## Part 11: Release Checklist Perform these steps before submitting to the App Store.