docs: Add Part 6 - Community Feedback and Model Retraining

Documents the full ML feedback loop:
- How user corrections trigger uploads (by label: Foil_, Condition_, Identity_)
- User opt-in requirement and AppConfig.isTrainingOptIn
- Firebase Storage rules (training write-only, models read-only)
- How to download collected data via Firebase CLI
- Image review and sorting process before retraining
- Create ML retraining workflow
- OTA model push via Firebase Storage models/ folder
- Recommended retraining schedule

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-03-05 13:01:27 -05:00
parent d6498d7c14
commit 15abf823be

112
README.md
View File

@@ -405,7 +405,109 @@ Output goes to `Set_Symbol_Training/`. Drag this folder into Create ML → Image
--- ---
## Part 6: Backend & Security ## Part 6: Community Feedback & Model Retraining
The app has a built-in pipeline that collects user corrections and uses them to improve the ML models over time. This section explains how it works end-to-end.
### How the Feedback System Works
There are two data collection paths:
**Path 1 — User Corrections (Community Data)**
When a user corrects a mis-scan (wrong card identity, wrong foil type, or wrong condition), the app automatically uploads the cropped card image to Firebase Storage — but only if the user has opted in.
The upload destination is determined by what was corrected:
| What Changed | Firebase Storage Path | Used to Retrain |
| :--- | :--- | :--- |
| Card name or set | `training/Identity_<SETCODE>_<CollectorNum>/` | `cards.json` (re-fingerprint) |
| Foil type | `training/Foil_<FoilType>/` | `IYmtgFoilClassifier.mlmodel` |
| Condition grade | `training/Condition_<Grade>/` | `IYmtgConditionClassifier.mlmodel` |
Example: A user corrects a card that was identified as "Traditional" foil to "Etched". The image is uploaded to `training/Foil_Etched/<UUID>.jpg`.
**Path 2 — Dev Mode (Your Own Device)**
When the `ENABLE_DEV_MODE` build flag is active and you tap the logo header 5 times, every raw scan frame is saved locally to `Documents/RawTrainingData/` on the device. Sync this folder to your Mac via Xcode's Devices window or Files app to retrieve images.
### User Opt-In
Users must explicitly opt in before any images are uploaded. The opt-in state is stored in `AppConfig.isTrainingOptIn` (backed by `UserDefaults`).
You must expose a toggle in your app's Settings/Library UI that sets `AppConfig.isTrainingOptIn = true/false`. The app description mentions this as the "Community Data Initiative" — users are told their corrections improve the AI for everyone.
**Firebase Authentication note:** `TrainingUploader` only uploads when `FirebaseApp.app() != nil` — meaning Firebase must be configured (`GoogleService-Info.plist` present) for community uploads to work. The app functions without Firebase, but no feedback is collected in that mode.
### Firebase Storage Rules
The rules in `IYmtg_App_iOS/Firebase/storage.rules` enforce:
- `training/` — authenticated users can **write only** (upload corrections). No user can read others' images.
- `models/` — anyone can **read** (required for OTA model downloads). Write access is developer-only via the Firebase Console.
### Downloading Collected Training Data
1. Go to the **Firebase Console → Storage → training/**
2. You will see folders named by label (e.g., `Foil_Etched/`, `Condition_Near Mint (NM)/`)
3. Download all images in each folder — use the Firebase CLI for bulk downloads:
```bash
# Install Firebase CLI if needed
npm install -g firebase-tools
firebase login
# Download all training data
firebase storage:cp gs://<your-bucket>/training ./downloaded_training --recursive
```
4. You now have a folder of user-contributed cropped card images, organized by label.
### Reviewing and Sorting Downloaded Images
**Do not skip this step.** User uploads can include blurry photos, wrong cards, or bad crops. Review each image before adding it to your training set.
1. Open each label folder from the download.
2. Delete any images that are: blurry, poorly cropped, show background, or are clearly wrong.
3. Move the accepted images into the corresponding `IYmtg_Training/` subfolder:
| Downloaded Folder | Move to Training Folder |
| :--- | :--- |
| `Foil_Traditional/` | `IYmtg_Training/Foil_Data/Traditional/` |
| `Foil_Etched/` | `IYmtg_Training/Foil_Data/Etched/` |
| `Foil_<Type>/` | `IYmtg_Training/Foil_Data/<Type>/` |
| `Condition_<Grade>/` | Inspect condition grade — map to `Condition_Data/` subfolder by damage type visible |
> **Identity corrections** (`training/Identity_*/`) are not used to retrain ML models. They indicate that the visual fingerprint for that card may be wrong or ambiguous. Review these separately and consider re-running the Builder for those specific cards.
### Retraining the Models
Once you have added new images to `IYmtg_Training/`:
1. Open **Create ML** on your Mac.
2. Open your existing project for the model you want to update (e.g., `IYmtgFoilClassifier`).
3. The new images in the training folders will be picked up automatically.
4. Click **Train**. Create ML will train incrementally on the expanded dataset.
5. Evaluate the results — check accuracy on the **Validation** tab. Aim for >90% accuracy before shipping.
6. Export the updated `.mlmodel` file.
### Pushing the Updated Model via OTA
You do not need an App Store update to ship a new model version. Use Firebase Storage:
1. In the **Firebase Console → Storage**, navigate to the `models/` folder.
2. Upload your new `.mlmodel` file with the **exact same filename** (e.g., `IYmtgFoilClassifier.mlmodel`).
3. On the next app launch, `ModelManager` detects the newer version, downloads and compiles it, and swaps it in automatically.
> **Important:** The new model takes effect on the **next app launch after download**, not immediately. Users may need to relaunch once.
### Recommended Retraining Schedule
| Trigger | Action |
| :--- | :--- |
| 50+ new correction images accumulated | Review, sort, retrain affected model, push OTA |
| New MTG set released with new foil type | Add training folder, acquire cards, retrain FoilClassifier |
| New MTG set released | Rebuild `cards.json` via `weekly_update.sh` |
| Significant accuracy complaints from users | Download corrections, review, retrain |
---
## Part 7: Backend & Security
### Cloud Storage Architecture ### Cloud Storage Architecture
The app uses a two-tier cloud strategy: The app uses a two-tier cloud strategy:
@@ -449,7 +551,7 @@ Ensure `PrivacyInfo.xcprivacy` is included in the app target to satisfy Apple's
--- ---
## Part 7: App Configuration ## Part 8: App Configuration
**CRITICAL:** Edit `IYmtg_App_iOS/AppConfig.swift` before building to ensure payments and support work correctly: **CRITICAL:** Edit `IYmtg_App_iOS/AppConfig.swift` before building to ensure payments and support work correctly:
1. Set `contactEmail` to your real email address (required by Scryfall API policy). 1. Set `contactEmail` to your real email address (required by Scryfall API policy).
@@ -458,7 +560,7 @@ Ensure `PrivacyInfo.xcprivacy` is included in the app target to satisfy Apple's
--- ---
## Part 8: Development Mode ## Part 9: Development Mode
To enable saving raw training images during scanning: To enable saving raw training images during scanning:
1. Add the compilation flag `ENABLE_DEV_MODE` in Xcode Build Settings → Swift Compiler → Active Compilation Conditions. 1. Add the compilation flag `ENABLE_DEV_MODE` in Xcode Build Settings → Swift Compiler → Active Compilation Conditions.
@@ -468,7 +570,7 @@ Saved images appear in `Documents/DevImages/` and can be used to supplement your
--- ---
## Part 9: Testing ## Part 10: Testing
The project includes a unit test suite in `IYmtgTests.swift`. The project includes a unit test suite in `IYmtgTests.swift`.
@@ -484,7 +586,7 @@ The project includes a unit test suite in `IYmtgTests.swift`.
--- ---
## Part 10: Release Checklist ## Part 11: Release Checklist
Perform these steps before submitting to the App Store. Perform these steps before submitting to the App Store.