# Current State Analysis: BookApp AI Pipeline

**Date:** 2026-02-22
**Scope:** Mapping existing codebase to the four phases defined in `ai_blueprint.md`
**Status:** Completed — fulfills Action Plan Step 1

---

## Overview

BookApp is an AI-powered novel generation engine using Google Gemini. The pipeline is structured into four phases that map directly to the review framework in `ai_blueprint.md`. This document catalogues the current implementation, identifies efficiency metrics, and surfaces limitations in each phase.

---

## Phase 1: Foundation & Ideation ("The Seed")

**Primary File:** `story/planner.py` (lines 1–86)
**Supporting:** `story/style_persona.py` (lines 81–104), `core/config.py`

### What Happens

1. User provides a minimal `manual_instruction` (can be a single sentence).
2. `enrich(bp, folder, context)` calls the Logic model to expand this into:
   - `book_metadata`: title, genre, tone, time period, structure type, formatting rules, content warnings
   - `characters`: 2–8 named characters with roles and descriptions
   - `plot_beats`: 5–7 concrete narrative beats
3. If the project is part of a series, context from previous books is injected.
4. `create_initial_persona()` generates a fictional author persona (name, bio, age, gender).

### Costs (Per Book)

| Task | Model | Input Tokens | Output Tokens | Cost (Pro-Exp) |
|------|-------|-------------|---------------|----------------|
| `enrich()` | Logic | ~10K | ~3K | FREE |
| `create_initial_persona()` | Logic | ~5.5K | ~1.5K | FREE |
| **Phase 1 Total** | — | ~15.5K | ~4.5K | **FREE** |

### Known Limitations

| ID | Issue | Impact |
|----|-------|--------|
| P1-L1 | `enrich()` silently returns original BP on exception (line 84) | Invalid enrichment passes downstream without warning |
| P1-L2 | `filter_characters()` blacklists keywords like "TBD", "protagonist" — can cull valid names | Characters named "The Protagonist" are silently dropped |
| P1-L3 | Single-pass persona creation — no quality check on output | Generic personas produce poor voice throughout book |
| P1-L4 | No validation that required `book_metadata` fields are non-null | Downstream crashes when title/genre are missing |

---

## Phase 2: Structuring & Outlining

**Primary File:** `story/planner.py` (lines 89–290)
**Supporting:** `story/style_persona.py`

### What Happens

1. `plan_structure(bp, folder)` maps plot beats to a structural framework (Hero's Journey, Three-Act, etc.) and produces ~10–15 events.
2. `expand(events, pass_num, ...)` iteratively enriches the outline. Called `depth` times (1–4 based on length preset). Each pass targets chapter count × 1.5 events as ceiling.
3. `create_chapter_plan(events, bp, folder)` converts events into concrete chapter objects with POV, pacing, and estimated word count.
4. `get_style_guidelines()` loads or refreshes the AI-ism blacklist and filter-word list.

### Depth Strategy

| Preset | Depth | Expand Calls | Approx Events |
|--------|-------|-------------|---------------|
| Flash Fiction | 1 | 1 | 1 |
| Short Story | 1 | 1 | 5 |
| Novella | 2 | 2 | 15 |
| Novel | 3 | 3 | 30 |
| Epic | 4 | 4 | 50 |

### Costs (30-Chapter Novel)

| Task | Calls | Input Tokens | Cost (Pro-Exp) |
|------|-------|-------------|----------------|
| `plan_structure` | 1 | ~15K | FREE |
| `expand` × 3 | 3 | ~12K each | FREE |
| `create_chapter_plan` | 1 | ~14K | FREE |
| `get_style_guidelines` | 1 | ~8K | FREE |
| **Phase 2 Total** | 6 | ~73K | **FREE** |

### Known Limitations

| ID | Issue | Impact |
|----|-------|--------|
| P2-L1 | Sequential `expand()` calls — each call unaware of final state | Redundant inter-call work; could be one multi-step prompt |
| P2-L2 | No continuity validation on outline — character deaths/revivals not detected | Plot holes remain until expensive Phase 3 rewrite |
| P2-L3 | Static chapter plan — cannot adapt if early chapters reveal pacing problem | Dynamic interventions in Phase 4 are costly workarounds |
| P2-L4 | POV assignment is AI-generated, not validated against narrative logic | Wrong POV on key scenes; caught only during editing |
| P2-L5 | Word count estimates are rough (~±30% actual variance) | Writer overshoots/undershoots target; word count normalization fails |

---

## Phase 3: The Writing Engine (Drafting)

**Primary File:** `story/writer.py`
**Orchestrated by:** `cli/engine.py`

### What Happens

For each chapter:
1. `expand_beats_to_treatment()` — Logic model expands sparse beats into a "Director's Treatment" (staging, sensory anchors, emotional arc, subtext).
2. `write_chapter()` constructs a ~310-line prompt injecting:
   - Author persona (bio, sample text, sample files from disk)
   - Filtered characters (only those named in beats + POV character)
   - Character tracking state (location, clothing, held items)
   - Lore context (relevant locations/items from tracking)
   - Style guidelines + genre-specific mandates
   - Smart context tail: last ~1000 tokens of previous chapter
   - Director's Treatment
3. Writer model generates first draft.
4. Logic model evaluates on 13 rubrics (1–10 scale). Automatic fail conditions apply for filter-word density, summary mode, and labeled emotions.
5. Iterative quality loop (up to 3 attempts):
   - Score ≥ 8.0 → Auto-accept
   - Score ≥ 7.0 → Accept after max attempts
   - Score < 7.0 → Refinement pass (Writer model)
   - Score < 6.0 → Full rewrite (Pro model)
6. Every 5 chapters: `refine_persona()` updates author bio based on actual written text.

### Key Innovations

- **Dynamic Character Injection:** Only injects characters named in chapter beats (saves ~5K tokens/chapter).
- **Smart Context Tail:** Takes last ~1000 tokens of previous chapter (not first 1000) — preserves handoff point.
- **Auto Model Escalation:** Low-scoring drafts trigger switch to Pro model for full rewrite.

### Costs (30-Chapter Novel, Mixed Model Strategy)

| Task | Calls | Input Tokens | Output Tokens | Cost Estimate |
|------|-------|-------------|---------------|---------------|
| `expand_beats_to_treatment` × 30 | 30 | ~5K | ~2K | FREE (Logic) |
| `write_chapter` draft × 30 | 30 | ~25K | ~3.5K | ~$0.087 (Writer) |
| Evaluation × 30 | 30 | ~20K | ~1.5K | FREE (Logic) |
| Refinement passes × 15 (est.) | 15 | ~20K | ~3K | ~$0.090 (Writer) |
| `refine_persona` × 6 | 6 | ~6K | ~1.5K | FREE (Logic) |
| **Phase 3 Total** | ~111 | ~1.9M | ~310K | **~$0.18** |

### Known Limitations

| ID | Issue | Impact |
|----|-------|--------|
| P3-L1 | Persona files re-read from disk on every chapter | I/O overhead; persona doesn't change between reads |
| P3-L2 | Beat expansion called even when beats are already detailed (>100 words) | Wastes ~5K tokens/chapter on ~30% of chapters |
| P3-L3 | Full rewrite triggered at score < 6.0 — discards entire draft | If draft scores 5.9, all 25K output tokens wasted |
| P3-L4 | No priority weighting for climax chapters | Ch 28 (climax) uses same resources/attempts as Ch 3 (setup) |
| P3-L5 | Previous chapter context hard-capped at 1000 tokens | For long chapters, might miss setup context from earlier pages |
| P3-L6 | Scoring thresholds fixed regardless of book position | Strict standards in early chapters = expensive refinement for setup scenes |

---

## Phase 4: Review & Refinement (Editing)

**Primary Files:** `story/editor.py`, `story/bible_tracker.py`
**Orchestrated by:** `cli/engine.py`

### What Happens

**During writing loop (every chapter):**
- `update_tracking()` refreshes character state (location, clothing, held items, speech style, events).
- `update_lore_index()` extracts canonical descriptions of locations and items.

**Every 2 chapters:**
- `check_pacing()` detects if story is rushing or repeating beats; triggers ADD_BRIDGE or CUT_NEXT interventions.

**After writing completes:**
- `analyze_consistency()` scans entire manuscript for plot holes and contradictions.
- `harvest_metadata()` extracts newly invented characters not in the original bible.
- `check_and_propagate()` cascades chapter edits forward through the manuscript.

### 13 Evaluation Rubrics

1. Engagement & tension
2. Scene execution (no summaries)
3. Voice & tone
4. Sensory immersion
5. Show, Don't Tell / Deep POV (**auto-fail trigger**)
6. Character agency
7. Pacing
8. Genre appropriateness
9. Dialogue authenticity
10. Plot relevance
11. Staging & flow
12. Prose dynamics (sentence variety)
13. Clarity & readability

**Automatic fail conditions:** filter-word density > 1/120 words → cap at 5; summary mode detected → cap at 6; >3 labeled emotions → cap at 5.

### Costs (30-Chapter Novel)

| Task | Calls | Input Tokens | Cost (Pro-Exp) |
|------|-------|-------------|----------------|
| `update_tracking` × 30 | 30 | ~18K | FREE |
| `update_lore_index` × 30 | 30 | ~15K | FREE |
| `check_pacing` × 15 | 15 | ~18K | FREE |
| `analyze_consistency` | 1 | ~25K | FREE |
| `harvest_metadata` | 1 | ~25K | FREE |
| **Phase 4 Total** | 77 | ~1.34M | **FREE** |

### Known Limitations

| ID | Issue | Impact |
|----|-------|--------|
| P4-L1 | Consistency check is post-generation only | Plot holes caught too late to cheaply fix |
| P4-L2 | Ripple propagation (`check_and_propagate`) has no cost ceiling | A single user edit in Ch 5 can trigger 100K+ tokens of cascading rewrites |
| P4-L3 | `rewrite_chapter_content()` uses Logic model instead of Writer model | Less creative rewrite output — Logic model optimizes reasoning, not prose |
| P4-L4 | `check_pacing()` sampling only looks at recent chapters, not cumulative arc | Slow-building issues across 10+ chapters not detected until critical |
| P4-L5 | No quality metric for the evaluator itself | Can't confirm if 13-rubric scores are calibrated correctly |

---

## Cross-Phase Summary

### Total Costs (30-Chapter Novel)

| Phase | Token Budget | Cost Estimate |
|-------|-------------|---------------|
| Phase 1: Ideation | ~20K | FREE |
| Phase 2: Outline | ~73K | FREE |
| Phase 3: Writing | ~2.2M | ~$0.18 |
| Phase 4: Review | ~1.34M | FREE |
| Imagen Cover (3 images) | — | ~$0.12 |
| **Total** | **~3.63M** | **~$0.30** |

*Assumes quality-first model selection (Pro-Exp for Logic, Flash for Writer)*

### Efficiency Frontier

- **Best case** (all chapters pass first attempt): ~$0.18 text + $0.04 cover = ~$0.22
- **Worst case** (30% rewrite rate with Pro escalations): ~$0.45 text + $0.12 cover = ~$0.57
- **Budget per blueprint goal:** $2.00 total — current system is 15–29% of budget

### Top 5 Immediate Optimization Opportunities

| Priority | ID | Change | Savings |
|----------|----|--------|---------|
| 1 | P3-L1 | Cache persona per book (not per chapter) | ~90K tokens |
| 2 | P3-L2 | Skip beat expansion for detailed beats | ~45K tokens |
| 3 | P2-L2 | Add pre-generation outline validation | Prevent expensive rewrites |
| 4 | P1-L1 | Fix silent failure in `enrich()` | Prevent silent corrupt state |
| 5 | P3-L6 | Adaptive scoring thresholds by chapter position | ~15% fewer refinement passes |