Auto-commit: v2.14 — Stuck job robustness (heartbeat, retry, stale watcher, granular logging)
- web/db.py: Add last_heartbeat column to Run model - core/utils.py: Add set_heartbeat_callback() and send_heartbeat() - web/tasks.py: Add _robust_update_run_status() with 5-retry exponential backoff; add db_heartbeat_callback(); remove all bare except:pass on DB status updates; set start_time + last_heartbeat when marking run as 'running' - web/app.py: Add last_heartbeat column migration; add _stale_job_watcher() background thread (checks every 5 min, 15-min heartbeat threshold, 2-hr start_time threshold) - cli/engine.py: Add phase-level logging banners and try/except wrappers in process_book(); add utils.send_heartbeat() after each chapter save; add start/finish logging in run_generation() Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -33,6 +33,7 @@ class Run(db.Model):
|
||||
log_file = db.Column(db.String(300), nullable=True)
|
||||
cost = db.Column(db.Float, default=0.0)
|
||||
progress = db.Column(db.Integer, default=0)
|
||||
last_heartbeat = db.Column(db.DateTime, nullable=True)
|
||||
|
||||
logs = db.relationship('LogEntry', backref='run', lazy=True, cascade="all, delete-orphan")
|
||||
|
||||
|
||||
Reference in New Issue
Block a user