thethreemagi/bookapp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Add evaluation report pipeline for prompt tuning feedback

Browse Source 
Adds a full per-chapter evaluation logging system that captures every
score, critique, and quality decision made during writing, then renders
a self-contained HTML report shareable with critics or prompt engineers.

New file — story/eval_logger.py:
- append_eval_entry(folder, entry): writes per-chapter eval data to
  eval_log.json in the book folder (called from write_chapter() at
  every return point).
- generate_html_report(folder, bp): reads eval_log.json and produces a
  self-contained HTML file (no external deps) with:
    • Summary cards (avg score, auto-accepted, rewrites, below-threshold)
    • Score timeline bar chart (one bar per chapter, colour-coded)
    • Score distribution histogram
    • Chapter breakdown table with expand-on-click critique details
      (attempt number, score, decision badge, full critique text)
    • Critique pattern frequency table (keyword mining across all critiques)
    • Auto-generated prompt tuning observations (systemic issues, POV
      character weak spots, pacing type analysis, climax vs. early
      chapter comparison)

story/writer.py:
- Imports time and eval_logger.
- Initialises _eval_entry dict (chapter metadata + polish flags + thresholds)
  after all threshold variables are set.
- Records each evaluation attempt's score, critique (truncated to 700 chars),
  and decision (auto_accepted / full_rewrite / refinement / accepted /
  below_threshold / eval_error / refinement_failed) before every return.

web/routes/run.py:
- Imports story_eval_logger.
- New route GET /project/<run_id>/eval_report/<book_folder>: loads
  eval_log.json, calls generate_html_report(), returns the HTML as a
  downloadable attachment named eval_report_<title>.html.
  Returns a user-friendly "not yet available" page if no log exists.

templates/run_details.html:
- Adds "Eval Report" (btn-outline-info) button next to "Check Consistency"
  in each book's artifact section.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Mike Wichers
2026-02-24 08:03:32 -05:00
parent d2c65f010a
commit f869700070
4 changed files with 578 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

41
web/routes/run.py
View File

@@ -10,7 +10,7 @@ from core import utils
from ai import models as ai_models
from ai import setup as ai_setup
from story import editor as story_editor
from story import bible_tracker, style_persona
from story import bible_tracker, style_persona, eval_logger as story_eval_logger
from export import exporter
from web.tasks import huey, regenerate_artifacts_task, rewrite_chapter_task
@@ -434,6 +434,45 @@ def delete_run(id):
return redirect(url_for('project.view_project', id=project_id))
@run_bp.route('/project/<int:run_id>/eval_report/<string:book_folder>')
@login_required
def eval_report(run_id, book_folder):
"""Generate and download the self-contained HTML evaluation report."""
run = db.session.get(Run, run_id) or Run.query.get_or_404(run_id)
if run.project.user_id != current_user.id:
return "Unauthorized", 403
if not book_folder or "/" in book_folder or "\\" in book_folder or ".." in book_folder:
return "Invalid book folder", 400
run_dir = os.path.join(run.project.folder_path, "runs", f"run_{run.id}")
book_path = os.path.join(run_dir, book_folder)
bp = utils.load_json(os.path.join(book_path, "final_blueprint.json")) or \
utils.load_json(os.path.join(book_path, "blueprint_initial.json"))
html = story_eval_logger.generate_html_report(book_path, bp)
if not html:
return (
"<html><body style='font-family:sans-serif;padding:40px'>"
"<h2>No evaluation data yet.</h2>"
"<p>The evaluation report is generated during the writing phase. "
"Start a generation run and the report will be available once chapters have been evaluated.</p>"
"</body></html>"
), 200
from flask import Response
safe_title = utils.sanitize_filename(
(bp or {}).get('book_metadata', {}).get('title', book_folder) or book_folder
)[:40]
filename = f"eval_report_{safe_title}.html"
return Response(
html,
mimetype='text/html',
headers={'Content-Disposition': f'attachment; filename="{filename}"'}
)
@run_bp.route('/run/<int:id>/download_bible')
@login_required
def download_bible(id):