fix: Pipeline hardening — error handling, token efficiency, and robustness

core/utils.py:
- estimate_tokens: improved heuristic 4 chars/token → 3.5 chars/token (more accurate)
- truncate_to_tokens: added keep_head=True mode for head+tail truncation (better
  context retention for story summaries that need both opening and recent content)
- load_json: explicit exception handling (json.JSONDecodeError, OSError) with log
  instead of silent returns; added utf-8 encoding with error replacement
- log_image_attempt: replaced bare except with (json.JSONDecodeError, OSError);
  added utf-8 encoding to output write
- log_usage: replaced bare except with AttributeError for token count extraction

story/bible_tracker.py:
- merge_selected_changes: wrapped all int() key casts (char idx, book num, beat idx)
  in try/except with meaningful log warning instead of crashing on malformed keys
- harvest_metadata: replaced bare except:pass with except Exception as e + log message

cli/engine.py:
- Persona validation: added warning when all 3 attempts fail and substandard persona
  is accepted — flags elevated voice-drift risk for the run
- Lore index updates: throttled from every chapter to every 3 chapters; lore is
  stable after the first few chapters (~10% token saving per book)
- Mid-gen consistency check: now samples first 2 + last 8 chapters instead of passing
  full manuscript — caps token cost regardless of book length

story/writer.py:
- Two-pass polish: added local filter-word density check (no API call); skips the
  Pro polish if density < 1 per 83 words — saves ~8K tokens on already-clean drafts
- Polish prompt: added prev_context_block for continuity — polished chapter now
  maintains seamless flow from the previous chapter's ending

marketing/fonts.py:
- Separated requests.exceptions.Timeout with specific log message vs generic failure
- Added explicit log message when Roboto fallback also fails (returns None)

marketing/blurb.py:
- Added word count trim: blurbs > 220 words trimmed to last sentence within 220 words
- Changed bare except to except Exception as e with log message
- Added utf-8 encoding to file writes; logs final word count

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

cli/engine.py

@@ -57,6 +57,8 @@ def process_book(bp, folder, context="", resume=False, interactive=False):
                     candidate_persona = style_persona.create_initial_persona(bp, folder)
                     is_valid, p_score = style_persona.validate_persona(bp, candidate_persona, folder)
                     if is_valid or persona_attempt == max_persona_attempts:
+                        if not is_valid:
+                            utils.log("SYSTEM", f"  ⚠️ Persona accepted after {max_persona_attempts} attempts despite low score ({p_score}/10). Voice drift risk elevated.")
                         bp['book_metadata']['author_details'] = candidate_persona
                         break
                     utils.log("SYSTEM", f"  -> Persona attempt {persona_attempt}/{max_persona_attempts} scored {p_score}/10. Regenerating...")
@@ -268,18 +270,21 @@ def process_book(bp, folder, context="", resume=False, interactive=False):
             with open(chars_track_path, "w") as f: json.dump(tracking['characters'], f, indent=2)
             with open(warn_track_path, "w") as f: json.dump(tracking.get('content_warnings', []), f, indent=2)
 
-            # Update Lore Index (Item 8: RAG-Lite)
-            tracking['lore'] = bible_tracker.update_lore_index(folder, txt, tracking.get('lore', {}))
-            with open(lore_track_path, "w") as f: json.dump(tracking['lore'], f, indent=2)
+            # Update Lore Index (Item 8: RAG-Lite) — every 3 chapters (lore is stable after ch 1-3)
+            if i == 0 or i % 3 == 0:
+                tracking['lore'] = bible_tracker.update_lore_index(folder, txt, tracking.get('lore', {}))
+                with open(lore_track_path, "w") as f: json.dump(tracking['lore'], f, indent=2)
 
             # Update Structured Story State (Item 9: Thread Tracking)
             current_story_state = story_state.update_story_state(txt, ch['chapter_number'], current_story_state, folder)
 
             # Exp 5: Mid-gen Consistency Snapshot (every 10 chapters)
+            # Sample: first 2 + last 8 chapters to keep token cost bounded regardless of book length
             if len(ms) > 0 and len(ms) % 10 == 0:
                 utils.log("EDITOR", f"--- Mid-gen consistency check after chapter {ch['chapter_number']} ({len(ms)} written) ---")
                 try:
-                    consistency = story_editor.analyze_consistency(bp, ms, folder)
+                    ms_sample = (ms[:2] + ms[-8:]) if len(ms) > 10 else ms
+                    consistency = story_editor.analyze_consistency(bp, ms_sample, folder)
                     issues = consistency.get('issues', [])
                     if issues:
                         for issue in issues: