docs: Add Part 6 - Community Feedback and Model Retraining
Documents the full ML feedback loop: - How user corrections trigger uploads (by label: Foil_, Condition_, Identity_) - User opt-in requirement and AppConfig.isTrainingOptIn - Firebase Storage rules (training write-only, models read-only) - How to download collected data via Firebase CLI - Image review and sorting process before retraining - Create ML retraining workflow - OTA model push via Firebase Storage models/ folder - Recommended retraining schedule Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
112
README.md
112
README.md
@@ -405,7 +405,109 @@ Output goes to `Set_Symbol_Training/`. Drag this folder into Create ML → Image
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Part 6: Backend & Security
|
## Part 6: Community Feedback & Model Retraining
|
||||||
|
|
||||||
|
The app has a built-in pipeline that collects user corrections and uses them to improve the ML models over time. This section explains how it works end-to-end.
|
||||||
|
|
||||||
|
### How the Feedback System Works
|
||||||
|
|
||||||
|
There are two data collection paths:
|
||||||
|
|
||||||
|
**Path 1 — User Corrections (Community Data)**
|
||||||
|
When a user corrects a mis-scan (wrong card identity, wrong foil type, or wrong condition), the app automatically uploads the cropped card image to Firebase Storage — but only if the user has opted in.
|
||||||
|
|
||||||
|
The upload destination is determined by what was corrected:
|
||||||
|
|
||||||
|
| What Changed | Firebase Storage Path | Used to Retrain |
|
||||||
|
| :--- | :--- | :--- |
|
||||||
|
| Card name or set | `training/Identity_<SETCODE>_<CollectorNum>/` | `cards.json` (re-fingerprint) |
|
||||||
|
| Foil type | `training/Foil_<FoilType>/` | `IYmtgFoilClassifier.mlmodel` |
|
||||||
|
| Condition grade | `training/Condition_<Grade>/` | `IYmtgConditionClassifier.mlmodel` |
|
||||||
|
|
||||||
|
Example: A user corrects a card that was identified as "Traditional" foil to "Etched". The image is uploaded to `training/Foil_Etched/<UUID>.jpg`.
|
||||||
|
|
||||||
|
**Path 2 — Dev Mode (Your Own Device)**
|
||||||
|
When the `ENABLE_DEV_MODE` build flag is active and you tap the logo header 5 times, every raw scan frame is saved locally to `Documents/RawTrainingData/` on the device. Sync this folder to your Mac via Xcode's Devices window or Files app to retrieve images.
|
||||||
|
|
||||||
|
### User Opt-In
|
||||||
|
|
||||||
|
Users must explicitly opt in before any images are uploaded. The opt-in state is stored in `AppConfig.isTrainingOptIn` (backed by `UserDefaults`).
|
||||||
|
|
||||||
|
You must expose a toggle in your app's Settings/Library UI that sets `AppConfig.isTrainingOptIn = true/false`. The app description mentions this as the "Community Data Initiative" — users are told their corrections improve the AI for everyone.
|
||||||
|
|
||||||
|
**Firebase Authentication note:** `TrainingUploader` only uploads when `FirebaseApp.app() != nil` — meaning Firebase must be configured (`GoogleService-Info.plist` present) for community uploads to work. The app functions without Firebase, but no feedback is collected in that mode.
|
||||||
|
|
||||||
|
### Firebase Storage Rules
|
||||||
|
|
||||||
|
The rules in `IYmtg_App_iOS/Firebase/storage.rules` enforce:
|
||||||
|
- `training/` — authenticated users can **write only** (upload corrections). No user can read others' images.
|
||||||
|
- `models/` — anyone can **read** (required for OTA model downloads). Write access is developer-only via the Firebase Console.
|
||||||
|
|
||||||
|
### Downloading Collected Training Data
|
||||||
|
|
||||||
|
1. Go to the **Firebase Console → Storage → training/**
|
||||||
|
2. You will see folders named by label (e.g., `Foil_Etched/`, `Condition_Near Mint (NM)/`)
|
||||||
|
3. Download all images in each folder — use the Firebase CLI for bulk downloads:
|
||||||
|
```bash
|
||||||
|
# Install Firebase CLI if needed
|
||||||
|
npm install -g firebase-tools
|
||||||
|
firebase login
|
||||||
|
|
||||||
|
# Download all training data
|
||||||
|
firebase storage:cp gs://<your-bucket>/training ./downloaded_training --recursive
|
||||||
|
```
|
||||||
|
4. You now have a folder of user-contributed cropped card images, organized by label.
|
||||||
|
|
||||||
|
### Reviewing and Sorting Downloaded Images
|
||||||
|
|
||||||
|
**Do not skip this step.** User uploads can include blurry photos, wrong cards, or bad crops. Review each image before adding it to your training set.
|
||||||
|
|
||||||
|
1. Open each label folder from the download.
|
||||||
|
2. Delete any images that are: blurry, poorly cropped, show background, or are clearly wrong.
|
||||||
|
3. Move the accepted images into the corresponding `IYmtg_Training/` subfolder:
|
||||||
|
|
||||||
|
| Downloaded Folder | Move to Training Folder |
|
||||||
|
| :--- | :--- |
|
||||||
|
| `Foil_Traditional/` | `IYmtg_Training/Foil_Data/Traditional/` |
|
||||||
|
| `Foil_Etched/` | `IYmtg_Training/Foil_Data/Etched/` |
|
||||||
|
| `Foil_<Type>/` | `IYmtg_Training/Foil_Data/<Type>/` |
|
||||||
|
| `Condition_<Grade>/` | Inspect condition grade — map to `Condition_Data/` subfolder by damage type visible |
|
||||||
|
|
||||||
|
> **Identity corrections** (`training/Identity_*/`) are not used to retrain ML models. They indicate that the visual fingerprint for that card may be wrong or ambiguous. Review these separately and consider re-running the Builder for those specific cards.
|
||||||
|
|
||||||
|
### Retraining the Models
|
||||||
|
|
||||||
|
Once you have added new images to `IYmtg_Training/`:
|
||||||
|
|
||||||
|
1. Open **Create ML** on your Mac.
|
||||||
|
2. Open your existing project for the model you want to update (e.g., `IYmtgFoilClassifier`).
|
||||||
|
3. The new images in the training folders will be picked up automatically.
|
||||||
|
4. Click **Train**. Create ML will train incrementally on the expanded dataset.
|
||||||
|
5. Evaluate the results — check accuracy on the **Validation** tab. Aim for >90% accuracy before shipping.
|
||||||
|
6. Export the updated `.mlmodel` file.
|
||||||
|
|
||||||
|
### Pushing the Updated Model via OTA
|
||||||
|
|
||||||
|
You do not need an App Store update to ship a new model version. Use Firebase Storage:
|
||||||
|
|
||||||
|
1. In the **Firebase Console → Storage**, navigate to the `models/` folder.
|
||||||
|
2. Upload your new `.mlmodel` file with the **exact same filename** (e.g., `IYmtgFoilClassifier.mlmodel`).
|
||||||
|
3. On the next app launch, `ModelManager` detects the newer version, downloads and compiles it, and swaps it in automatically.
|
||||||
|
|
||||||
|
> **Important:** The new model takes effect on the **next app launch after download**, not immediately. Users may need to relaunch once.
|
||||||
|
|
||||||
|
### Recommended Retraining Schedule
|
||||||
|
|
||||||
|
| Trigger | Action |
|
||||||
|
| :--- | :--- |
|
||||||
|
| 50+ new correction images accumulated | Review, sort, retrain affected model, push OTA |
|
||||||
|
| New MTG set released with new foil type | Add training folder, acquire cards, retrain FoilClassifier |
|
||||||
|
| New MTG set released | Rebuild `cards.json` via `weekly_update.sh` |
|
||||||
|
| Significant accuracy complaints from users | Download corrections, review, retrain |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part 7: Backend & Security
|
||||||
|
|
||||||
### Cloud Storage Architecture
|
### Cloud Storage Architecture
|
||||||
The app uses a two-tier cloud strategy:
|
The app uses a two-tier cloud strategy:
|
||||||
@@ -449,7 +551,7 @@ Ensure `PrivacyInfo.xcprivacy` is included in the app target to satisfy Apple's
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Part 7: App Configuration
|
## Part 8: App Configuration
|
||||||
|
|
||||||
**CRITICAL:** Edit `IYmtg_App_iOS/AppConfig.swift` before building to ensure payments and support work correctly:
|
**CRITICAL:** Edit `IYmtg_App_iOS/AppConfig.swift` before building to ensure payments and support work correctly:
|
||||||
1. Set `contactEmail` to your real email address (required by Scryfall API policy).
|
1. Set `contactEmail` to your real email address (required by Scryfall API policy).
|
||||||
@@ -458,7 +560,7 @@ Ensure `PrivacyInfo.xcprivacy` is included in the app target to satisfy Apple's
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Part 8: Development Mode
|
## Part 9: Development Mode
|
||||||
|
|
||||||
To enable saving raw training images during scanning:
|
To enable saving raw training images during scanning:
|
||||||
1. Add the compilation flag `ENABLE_DEV_MODE` in Xcode Build Settings → Swift Compiler → Active Compilation Conditions.
|
1. Add the compilation flag `ENABLE_DEV_MODE` in Xcode Build Settings → Swift Compiler → Active Compilation Conditions.
|
||||||
@@ -468,7 +570,7 @@ Saved images appear in `Documents/DevImages/` and can be used to supplement your
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Part 9: Testing
|
## Part 10: Testing
|
||||||
|
|
||||||
The project includes a unit test suite in `IYmtgTests.swift`.
|
The project includes a unit test suite in `IYmtgTests.swift`.
|
||||||
|
|
||||||
@@ -484,7 +586,7 @@ The project includes a unit test suite in `IYmtgTests.swift`.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Part 10: Release Checklist
|
## Part 11: Release Checklist
|
||||||
|
|
||||||
Perform these steps before submitting to the App Store.
|
Perform these steps before submitting to the App Store.
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user