bookapp/docs/experiment_design.md

# Experiment Design: A/B Tests for BookApp Optimization

**Date:** 2026-02-22
**Status:** Completed — fulfills Action Plan Step 3

---

## Methodology

All experiments follow a controlled A/B design. We hold all variables constant except the single variable under test. Success is measured against three primary metrics:

- **Cost per chapter (CPC):** Total token cost / number of chapters written
- **Human Quality Score (HQS):** 1–10 score from a human reviewer blind to which variant generated the chapter
- **Continuity Error Rate (CER):** Number of plot/character contradictions per 10 chapters (lower is better)

Each experiment runs on the same 3 prompts (one each of short story, novella, and novel length). Results are averaged across all 3.

**Baseline:** Current production configuration as of 2026-02-22.

---

## Experiment 1: Persona Caching

**Alt Reference:** Alt 3-D
**Hypothesis:** Caching persona per book reduces I/O overhead with no quality impact.

### Setup

| Parameter | Control (A) | Treatment (B) |
|-----------|-------------|---------------|
| Persona loading | Re-read from disk each chapter | Load once per book run, pass as argument |
| Everything else | Identical | Identical |

### Metrics to Measure

- Token count per chapter (to verify savings)
- Wall-clock generation time per book
- Chapter quality scores (should be identical)

### Success Criterion

- Token reduction ≥ 2,000 tokens/chapter on books with sample files
- HQS difference < 0.1 between A and B (no quality impact)
- Zero new errors introduced

### Implementation Notes

- Modify `cli/engine.py`: call `style_persona.load_persona_data()` once before chapter loop
- Modify `story/writer.py`: accept optional `persona_info` parameter, skip disk reads if provided
- Estimated implementation: 30 minutes

---

## Experiment 2: Skip Beat Expansion for Detailed Beats

**Alt Reference:** Alt 3-E
**Hypothesis:** Skipping `expand_beats_to_treatment()` when beats exceed 100 words saves tokens with no quality loss.

### Setup

| Parameter | Control (A) | Treatment (B) |
|-----------|-------------|---------------|
| Beat expansion | Always called | Skipped if total beats > 100 words |
| Everything else | Identical | Identical |

### Metrics to Measure

- Percentage of chapters that skip expansion (expected: ~30%)
- Token savings per book
- HQS for chapters that skip vs. chapters that don't skip
- Rate of beat-coverage failures (chapters that miss a required beat)

### Success Criterion

- ≥ 25% of chapters skip expansion (validating hypothesis)
- HQS difference < 0.2 between chapters that skip and those that don't
- Beat-coverage failure rate unchanged

### Implementation Notes

- Modify `story/writer.py` `write_chapter()`: add `if sum(len(b) for b in beats) > 100` guard before calling expansion
- Estimated implementation: 15 minutes

---

## Experiment 3: Outline Validation Gate

**Alt Reference:** Alt 2-B
**Hypothesis:** Pre-generation outline validation prevents costly Phase 3 rewrites by catching plot holes at the outline stage.

### Setup

| Parameter | Control (A) | Treatment (B) |
|-----------|-------------|---------------|
| Outline validation | None | Run `validate_outline()` after `create_chapter_plan()`; block if critical issues found |
| Everything else | Identical | Identical |

### Metrics to Measure

- Number of critical outline issues flagged per run
- Rewrite rate during Phase 3 (did validation prevent rewrites?)
- Phase 3 token cost difference (A vs B)
- CER difference (did validation reduce continuity errors?)

### Success Criterion

- Validation blocks at least 1 critical issue per 3 runs
- Phase 3 rewrite rate drops ≥ 15% when validation is active
- CER improves ≥ 0.5 per 10 chapters

### Implementation Notes

- Add `validate_outline(events, chapters, bp, folder)` to `story/planner.py`
- Prompt: "Review this chapter plan for: (1) missing required plot beats, (2) character deaths/revivals without explanation, (3) severe pacing imbalances, (4) POV character inconsistency. Return: {issues: [...], severity: 'critical'|'warning'|'ok'}"
- Modify `cli/engine.py`: call `validate_outline()` and log issues before Phase 3 begins
- Estimated implementation: 2 hours

---

## Experiment 4: Adaptive Scoring Thresholds

**Alt Reference:** Alt 3-B
**Hypothesis:** Lowering SCORE_PASSING for early setup chapters reduces refinement cost while maintaining quality on high-stakes scenes.

### Setup

| Parameter | Control (A) | Treatment (B) |
|-----------|-------------|---------------|
| SCORE_AUTO_ACCEPT | 8.0 (all chapters) | 8.0 (all chapters) |
| SCORE_PASSING | 7.0 (all chapters) | 6.5 (ch 1–20%), 7.0 (ch 20–70%), 7.5 (ch 70–100%) |
| Everything else | Identical | Identical |

### Metrics to Measure

- Refinement pass count per chapter position bucket
- HQS per chapter position bucket (A vs B)
- CPC for each bucket
- Overall HQS for full book (A vs B)

### Success Criterion

- Setup chapters (1–20%): ≥ 20% fewer refinement passes in B
- Climax chapters (70–100%): HQS improvement ≥ 0.3 in B
- Full book HQS unchanged or improved

### Implementation Notes

- Modify `story/writer.py` `write_chapter()`: accept `chapter_position` (0.0–1.0 float)
- Compute adaptive threshold: `passing = 6.5 + position * 1.0` (linear scaling)
- Modify `cli/engine.py`: pass `chapter_num / total_chapters` to `write_chapter()`
- Estimated implementation: 1 hour

---

## Experiment 5: Mid-Generation Consistency Snapshots

**Alt Reference:** Alt 4-B
**Hypothesis:** Running `analyze_consistency()` every 10 chapters reduces post-generation CER without significant cost increase.

### Setup

| Parameter | Control (A) | Treatment (B) |
|-----------|-------------|---------------|
| Consistency check | Post-generation only | Every 10 chapters + post-generation |
| Everything else | Identical | Identical |

### Metrics to Measure

- CER post-generation (A vs B)
- Number of issues caught mid-generation vs post-generation
- Token cost difference (mid-gen checks add ~25K × N/10 tokens)
- Generation time difference

### Success Criterion

- Post-generation CER drops ≥ 30% in B
- Issues caught mid-generation prevent at least 1 expensive post-gen ripple propagation per run
- Additional cost ≤ $0.01 per book (all free on Pro-Exp)

### Implementation Notes

- Modify `cli/engine.py`: every 10 chapters, call `analyze_consistency()` on written chapters so far
- If issues found: log warning and optionally pause for user review
- Estimated implementation: 1 hour

---

## Experiment 6: Iterative Persona Validation

**Alt Reference:** Alt 1-C
**Hypothesis:** Validating the initial persona with a sample passage reduces voice-drift rewrites in Phase 3.

### Setup

| Parameter | Control (A) | Treatment (B) |
|-----------|-------------|---------------|
| Persona creation | Single-pass, no validation | Generate persona → generate 200-word sample → evaluate → accept if ≥ 7/10, else regenerate (max 3 attempts) |
| Everything else | Identical | Identical |

### Metrics to Measure

- Initial persona acceptance rate (how often does first-pass persona pass the check?)
- Phase 3 persona-related rewrite rate (rewrites where critique mentions "voice inconsistency" or "doesn't match persona")
- HQS for first 5 chapters (voice is most important early on)

### Success Criterion

- Phase 3 persona-related rewrite rate drops ≥ 20% in B
- HQS for first 5 chapters improves ≥ 0.2

### Implementation Notes

- Modify `story/style_persona.py`: after `create_initial_persona()`, call a new `validate_persona()` function
- `validate_persona()` generates 200-word sample, evaluates with `evaluate_chapter_quality()` (light version)
- Estimated implementation: 2 hours

---

## Experiment 7: Two-Pass Drafting (Draft + Polish)

**Alt Reference:** Alt 3-A
**Hypothesis:** A cheap rough draft followed by a polished revision produces better quality than iterative retrying.

### Setup

| Parameter | Control (A) | Treatment (B) |
|-----------|-------------|---------------|
| Drafting strategy | Single draft → evaluate → retry | Rough draft (Flash) → polish (Pro) → evaluate → accept if ≥ 7.0 (max 1 retry) |
| Max retry attempts | 3 | 1 (after polish) |
| Everything else | Identical | Identical |

### Metrics to Measure

- CPC (A vs B)
- HQS (A vs B)
- Rate of chapters needing retry (A vs B)
- Total generation time per book

### Success Criterion

- HQS improvement ≥ 0.3 in B with no cost increase
- OR: CPC reduction ≥ 20% in B with no HQS decrease

### Implementation Notes

- Modify `story/writer.py` `write_chapter()`: add polish pass using Pro model after initial draft
- Reduce max_attempts to 1 for final retry (after polish)
- This requires Pro model to be available (handled by auto-selection)

---

## Experiment Execution Order

Run experiments in this order to minimize dependency conflicts:

1. **Exp 1** (Persona Caching) — independent, 30 min, no risk
2. **Exp 2** (Skip Beat Expansion) — independent, 15 min, no risk
3. **Exp 4** (Adaptive Thresholds) — independent, 1 hr, low risk
4. **Exp 3** (Outline Validation) — independent, 2 hrs, low risk
5. **Exp 6** (Persona Validation) — independent, 2 hrs, low risk
6. **Exp 5** (Mid-gen Consistency) — requires stable Phase 3, 1 hr, low risk
7. **Exp 7** (Two-Pass Drafting) — highest risk, run last; 3 hrs, medium risk

---

## Success Metrics Definitions

### Cost per Chapter (CPC)

```
CPC = (total_input_tokens × input_price + total_output_tokens × output_price) / num_chapters
```

Measure in both USD and token-count to separate model-price effects from efficiency effects.

### Human Quality Score (HQS)

Blind evaluation by a human reviewer:
1. Read 3 chapters from treatment A and 3 from treatment B (same book premise)
2. Score each on: prose quality (1–5), pacing (1–5), character consistency (1–5)
3. HQS = average across all dimensions, normalized to 1–10

### Continuity Error Rate (CER)

After generation, manually review character states and key plot facts across chapters. Count:
- Character location contradictions
- Continuity breaks (held items, injuries, time-of-day)
- Plot event contradictions (character alive vs. dead)

Report as errors per 10 chapters.