From d7972784133dccb5239d9cff338fca8ec60d81c3 Mon Sep 17 00:00:00 2001 From: Mike Wichers Date: Fri, 20 Feb 2026 22:36:39 -0500 Subject: [PATCH] Blueprint v1.0.1: Rewrite README with code-verified modular architecture docs Co-Authored-By: Claude Sonnet 4.6 --- README.md | 362 ++++++++++++++++++++++++++++-------------------- ai_blueprint.md | 2 +- 2 files changed, 216 insertions(+), 148 deletions(-) diff --git a/README.md b/README.md index 5691379..89ace00 100644 --- a/README.md +++ b/README.md @@ -1,196 +1,264 @@ -# 📚 BookApp: AI-Powered Series Engine +# BookApp: AI-Powered Series Engine -An automated pipeline for planning, drafting, and publishing novels using Google Gemini. +An automated pipeline for planning, drafting, and publishing novels using Google Gemini. Supports both a browser-based Web UI and an interactive CLI Wizard. -## 🚀 Quick Start -1. **Install:** `pip install -r requirements.txt` -2. **Web Dependencies:** `pip install -r modules/requirements_web.txt` -3. **Setup:** Add your API key to `.env`. -4. **Launch Dashboard:** `python -m modules.web_app` -5. **Open Browser:** Go to `http://localhost:5000` to create projects, manage personas, and generate books. +## Quick Start -### Alternative: CLI Mode -If you prefer the command line: -1. Run `python wizard.py` to create or edit your book settings interactively. -2. Run `python main.py ` to generate the book(s). +1. **Install core dependencies:** `pip install -r requirements.txt` +2. **Install web dependencies:** `pip install -r web/requirements_web.txt` +3. **Configure:** Copy `.env.example` to `.env` and add your `GEMINI_API_KEY`. +4. **Launch:** `python -m web.app` +5. **Open:** `http://localhost:5000` -## 🛡️ Admin Access -The application includes a protected Admin Dashboard at `/admin` for managing users and performing factory resets. Access is password-protected and restricted to users with the Admin role. +### CLI Mode (No Browser) -**Method 1: Environment Variables (Recommended for Docker)** -Set `ADMIN_USERNAME` and `ADMIN_PASSWORD` in your environment (or `docker-compose.yml`). The system will automatically create this admin account on startup. +```bash +python -m cli.wizard +``` -**Method 2: Manual Promotion** -Register a normal account, then manually update the database row (`is_admin = 1`) or use a CLI script if available. +The wizard guides you through creating or loading a project, defining characters and plot beats, and launching a generation run directly from the terminal. It auto-detects incomplete runs and offers to resume them. -## � Docker Setup (Recommended for Raspberry Pi) -This is the best way to run the Web Dashboard on a server using Portainer. +## Admin Access + +The `/admin` panel allows managing users and performing factory resets. It is restricted to accounts with the Admin role. + +**Via environment variables (recommended for Docker):** Set `ADMIN_USERNAME` and `ADMIN_PASSWORD` — the account is auto-created on startup. + +**Via manual promotion:** Register a normal account, then set `is_admin = 1` in the database. + +## Docker Setup (Recommended for Servers) ### 1. Git Setup -1. Create a new Git repository (GitHub/GitLab). - - For local servers like Gitea, your URL will be something like `http://10.0.0.102:3000/thethreemagi/bookapp.git`. -2. Push this project code to the repository. - - **IMPORTANT:** Ensure `.env`, `token.json`, `credentials.json`, and the `data/` folder are in your `.gitignore`. Do **not** commit secrets to the repo. -### 2. Server Preparation (One-Time Setup) +Push this project to a Git repository (GitHub, GitLab, or a self-hosted Gitea). Ensure `.env`, `token.json`, `credentials.json`, and `data/` are in `.gitignore`. -Since secrets and database files shouldn't be in Git, you need to place them on your server manually. +### 2. Server Preparation (One-Time) -1. **Authenticate Locally:** Run the app on your PC first (`python wizard.py`) to generate the `token.json` file (Google Login). -2. **SSH into your server** and create a folder for your app data: +Place secrets on the server manually — they must not be in Git. + +1. Run `python -m cli.wizard` locally to generate `token.json` (Google OAuth). +2. SSH into your server and create a data folder: ```bash - mkdir -p /opt/bookapp # Or any other path you prefer + mkdir -p /opt/bookapp ``` -3. **Upload Files:** Use WinSCP or SCP to upload these two files from your PC to the folder you just created (e.g., `/opt/bookapp`): - - `token.json` (Generated in step 1) - - `credentials.json` (Your Google Cloud OAuth file) - The `data` subfolder, which stores your database and projects, will be created automatically by Docker when the container starts. +3. Upload `token.json` and `credentials.json` to `/opt/bookapp`. The `data/` subfolder is created automatically on first run. ### 3. Portainer Stack Setup -1. Log in to **Portainer**. -2. Go to **Stacks** > **Add stack**. -3. Select **Repository**. - - **Repository URL:** `http://10.0.0.102:3000/thethreemagi/bookapp.git` - - **Compose path:** `docker-compose.yml` -4. **Enable Authentication:** Since your Gitea repository is private, you will need to provide credentials. - - Toggle on **Authentication**. - - **Username:** Your Gitea username (`thethreemagi`). - - **Password:** Use a **Personal Access Token** from Gitea, not your actual password. - - In Gitea, go to your **User Settings > Applications** and generate a new token with repository read access. -5. Under **Environment variables**, add the following: - - `HOST_PATH`: `/opt/bookapp` (The folder you created in Step 2) - - `GEMINI_API_KEY`: `` - - `ADMIN_USERNAME`: `admin` (Or your preferred username) - - `ADMIN_PASSWORD`: `` - - `FLASK_SECRET_KEY`: `` - - `FLASK_DEBUG`: `False` (Set to `True` only for troubleshooting) - - **Optional (Advanced / Vertex AI):** - - `GCP_PROJECT`: Your Google Cloud Project ID (Required for Imagen 3/Vertex AI). - - `GCP_LOCATION`: `us-central1` (Default). - - `MODEL_LOGIC`: Override the logic model (e.g., `models/gemini-1.5-pro-latest`). - - `MODEL_WRITER`: Override the writer model. -6. Click **Deploy the stack**. -Portainer will pull the code from Git, build the image, and mount the secrets/data from your server folder. +1. Go to **Stacks** > **Add stack** > **Repository**. +2. Set **Repository URL** and **Compose path** (`docker-compose.yml`). +3. Enable **Authentication** and supply a Gitea Personal Access Token if your repo is private. +4. Add **Environment variables**: + + | Variable | Description | + | :--- | :--- | + | `HOST_PATH` | Server folder for persistent data (e.g., `/opt/bookapp`) | + | `GEMINI_API_KEY` | Your Google Gemini API key (**required**) | + | `ADMIN_USERNAME` | Admin account username | + | `ADMIN_PASSWORD` | Admin account password | + | `FLASK_SECRET_KEY` | Random string for session encryption | + | `FLASK_DEBUG` | `False` in production | + | `GCP_PROJECT` | Google Cloud Project ID (required for Imagen / Vertex AI) | + | `GCP_LOCATION` | GCP region (default: `us-central1`) | + | `MODEL_LOGIC` | Override the reasoning model (e.g., `models/gemini-1.5-pro-latest`) | + | `MODEL_WRITER` | Override the writing model | + | `MODEL_ARTIST` | Override the visual-prompt model | + | `MODEL_IMAGE` | Override the image generation model | + +5. Click **Deploy the stack**. ### 4. Updating the App -To update the code: -1. Run the app **on your PC** first (using `python wizard.py` or `main.py`). -2. Push changes to Git. -3. In Portainer, go to your Stack. -4. Click **Editor** > **Pull and redeploy**. -### 📂 How to Manage Files (Input/Output) -The Docker setup uses a **Volume** to map the container's internal `/app/data` folder to a folder on your server. This path is defined by the `HOST_PATH` variable you set in Portainer. +1. Make changes locally and push to Git. +2. In Portainer: Stack > **Editor** > **Pull and redeploy**. -- **To Add Personas/Fonts:** On your server, place files into the `${HOST_PATH}/data/personas/` or `${HOST_PATH}/data/fonts/` folders. The app will see them immediately. -- **To Download Books:** You can download generated EPUBs directly from the Web Dashboard. -- **To Backup:** Just create a backup of the entire `${HOST_PATH}` directory on your server. It contains the database, all projects, and generated books. +### Managing Files -## 🐍 Native Web Setup (Alternative) -If you prefer to run the web app without Docker: +The Docker volume maps `/app/data` in the container to `HOST_PATH` on your server. -1. **Install Web Dependencies:** - ```bash - pip install -r modules/requirements_web.txt - ``` -2. **Start the App:** - ```bash - python -m modules.web_app - ``` -3. **Access:** Open `http://localhost:5000` in a browser. +- **Add personas/fonts:** Drop files into `${HOST_PATH}/data/personas/` or `${HOST_PATH}/data/fonts/`. +- **Download books:** Use the Web Dashboard download links. +- **Backup:** Archive the entire `${HOST_PATH}` directory. -## �️ Features -- **Interactive Wizard:** Create new books, series, or sequels. Edit existing blueprints with natural language commands. -- **Modular Architecture:** Logic is split into specialized modules for easier maintenance and upgrades. -- **Smart Resume:** If a run crashes, simply run the script again. It detects progress and asks to resume. -- **Marketing Assets:** Automatically generates a blurb, back cover text, and a cover image. -- **Rich Text:** Generates EPUBs with proper formatting (Bold, Italics, Headers). -- **Consistency Checker:** Scans the manuscript for plot holes and continuity errors. -- **Manual Editing:** Read and edit chapters directly in the browser; changes are preserved during file regeneration. -- **Metadata Sync:** Updates the "Story Bible" based on your manual edits to the text. -- **Dynamic Structure:** Automatically adapts the plot structure (e.g., "Hero's Journey" vs "Single Scene") based on the book length. -- **Series Support:** Automatically carries context, characters, and plot threads from Book 1 to Book 2, etc. +## Native Setup (No Docker) -## 📂 Project Structure +```bash +pip install -r requirements.txt +pip install -r web/requirements_web.txt +python -m web.app +``` -### Core Files -- **`wizard.py`**: The interactive command-line interface for creating projects, managing personas, and editing the "Book Bible". -- **`main.py`**: The execution engine. It reads the Bible JSON and orchestrates the generation process using the modules. -- **`config.py`**: Central configuration for API keys, file paths, and model settings. -- **`utils.py`**: Shared utility functions for logging, JSON handling, and file I/O. +Open `http://localhost:5000`. -### Modules (`/modules`) -- **`ai.py`**: Handles authentication and connection to Google Gemini and Vertex AI. -- **`story.py`**: Contains the creative logic: enriching ideas, planning structure, and writing chapters. -- **`marketing.py`**: Generates cover art prompts, images, and blurbs. -- **`export.py`**: Compiles the final manuscript into DOCX and EPUB formats. +## Features -### Data Folders -- **`data/personas/`**: Stores author personas and writing samples. -- **`data/fonts/`**: Caches downloaded fonts for cover art. +### Web UI (`web/`) +- **Project Dashboard:** Create and monitor generation jobs from the browser. +- **Real-time Logs:** Console output is streamed to the browser and stored in the database. +- **Chapter Editor:** Edit chapters directly in the browser; manual edits are preserved across artifact regenerations and synced back to character/plot tracking state. +- **Cover Regeneration:** Submit written feedback to regenerate the cover image iteratively. +- **Admin Panel:** Manage all users, view spend, and perform factory resets at `/admin`. +- **Per-User API Keys:** Each user can supply their own Gemini API key; costs are tracked per account. -## � Length Settings Explained -The **Length Settings** control not just the word count, but the **structural complexity** of the story. +### CLI Wizard (`cli/`) +- **Interactive Setup:** Menu-driven interface (via Rich) for creating projects, managing personas, and defining characters and plot beats. +- **Smart Resume:** Detects in-progress runs via lock files and prompts to resume. +- **Interactive Mode:** Optionally review and approve/reject each chapter before generation continues. +- **Stop Signal:** Create a `.stop` file in the run directory to gracefully abort a long run without corrupting state. -| Type | Approx Words | Chapters | Description | -| :--- | :--- | :--- | :--- | -| **Flash Fiction** | 500 - 1.5k | 1 | A single scene or moment. | -| **Short Story** | 5k - 10k | 5 | One conflict, few characters. | -| **Novella** | 20k - 40k | 15 | Developed plot, A & B stories. | -| **Novel** | 60k - 80k | 30 | Deep subplots, slow pacing. | -| **Epic** | 100k+ | 50 | Massive scope, world-building focus. | +### Story Generation (`story/`) +- **Adaptive Structure:** Chooses a narrative framework (Hero's Journey, Three-Act, Single Scene, etc.) based on the selected length preset and expands it through multiple depth levels. +- **Dynamic Pacing:** Monitors story progress during writing and inserts bridge chapters to slow a rushing plot or removes redundant ones detected mid-stream — without restarting. +- **Series Continuity:** When generating Book 2+, carries forward character visual tracking, established relationships, plot threads, and a cumulative "Story So Far" summary. +- **Persona Refinement Loop:** Every 5 chapters, analyzes actual written text to refine the author persona model, maintaining stylistic consistency throughout the book. +- **Consistency Checker (`editor.py`):** Scores chapters on 8 rubrics (engagement, voice, sensory detail, scene execution, etc.) and flags AI-isms ("tapestry", "palpable tension") and weak filter verbs ("felt", "realized"). -> **Note:** This engine is designed for **linear fiction**. It does not currently support branching narratives like "Choose Your Own Adventure" books. +### Marketing Assets (`marketing/`) +- **Cover Art:** Generates a visual prompt from book themes and tracking data, then calls Imagen (Gemini or Vertex AI) to produce the cover. Evaluates image quality with multimodal AI critique before accepting. +- **Back-Cover Blurb:** Writes 150–200 word marketing copy in a 4-part structure (Hook, Stakes, Tension, Close) with genre-specific tone (thriller=urgent, romance=emotional, etc.). -## 📂 Data Structure & File Dictionary +### Export (`export/`) +- **EPUB:** eBook file with cover image, chapter structure, and formatted text (bold, italics, headers). Ready for Kindle / Apple Books. +- **DOCX:** Word document for manual editing. -The application stores all data in the `data/` directory. This makes backup and migration easy. +### AI Infrastructure (`ai/`) +- **Resilient Model Wrapper:** Wraps every Gemini API call with up to 3 retries and exponential backoff, handles quota errors and rate limits, and can switch to an alternative model mid-stream. +- **Auto Model Selection:** On startup, a bootstrapper model queries the Gemini API and selects the optimal models for Logic, Writer, Artist, and Image roles. Selection is cached for 24 hours. +- **Vertex AI Support:** If `GCP_PROJECT` is set and OAuth credentials are present, initializes Vertex AI automatically for Imagen image generation. + +### Cost Tracking +Every AI call logs input/output token counts and estimated USD cost (using cached pricing per model). Cumulative project cost is stored in the database and displayed per user and per run. + +## Project Structure + +```text +BookApp/ +├── ai/ # Gemini/Vertex AI authentication and resilient model wrapper +│ ├── models.py # ResilientModel class with retry logic +│ └── setup.py # Model initialization and auto-selection +├── cli/ # Terminal interface and generation orchestrator +│ ├── engine.py # Full generation pipeline (plan → write → export) +│ └── wizard.py # Interactive menu-driven setup wizard +├── core/ # Central configuration and shared utilities +│ ├── config.py # Environment variable loading, presets, AI safety settings +│ └── utils.py # Logging, JSON cleaning, usage tracking, filename utils +├── export/ # Manuscript compilation +│ └── exporter.py # EPUB and DOCX generation +├── marketing/ # Post-generation asset creation +│ ├── assets.py # Orchestrates cover + blurb creation +│ ├── blurb.py # Back-cover marketing copy generation +│ ├── cover.py # Cover art generation and iterative refinement +│ └── fonts.py # Google Fonts downloader/cache +├── story/ # Core creative AI pipeline +│ ├── bible_tracker.py # Character state and plot event tracking +│ ├── editor.py # Chapter quality scoring and AI-ism detection +│ ├── planner.py # Story structure and chapter plan generation +│ ├── style_persona.py # Author persona creation and refinement +│ └── writer.py # Chapter-by-chapter writing with persona/context injection +├── templates/ # Jinja2 HTML templates for the web application +├── web/ # Flask web application +│ ├── app.py # App factory, blueprint registration, admin auto-creation +│ ├── db.py # SQLAlchemy models: User, Project, Run, LogEntry +│ ├── helpers.py # admin_required decorator, project lock check, CSRF utils +│ ├── tasks.py # Huey background task queue (generate, rewrite, regenerate) +│ ├── requirements_web.txt +│ └── routes/ +│ ├── admin.py # User management and factory reset +│ ├── auth.py # Login, register, session management +│ ├── persona.py # Author persona CRUD and sample file upload +│ ├── project.py # Project creation wizard and job queuing +│ └── run.py # Run status, logs, downloads, chapter editing, cover regen +├── docker-compose.yml +├── Dockerfile +├── requirements.txt # Core AI/generation dependencies +└── README.md +``` + +## Environment Variables + +All variables are loaded from a `.env` file in the project root (never commit this file). + +| Variable | Required | Description | +| :--- | :---: | :--- | +| `GEMINI_API_KEY` | Yes | Google Gemini API key | +| `FLASK_SECRET_KEY` | No | Session encryption key (default: insecure dev value — change in production) | +| `ADMIN_USERNAME` | No | Auto-creates an admin account on startup | +| `ADMIN_PASSWORD` | No | Password for the auto-created admin account | +| `GCP_PROJECT` | No | Google Cloud Project ID (enables Vertex AI / Imagen) | +| `GCP_LOCATION` | No | GCP region (default: `us-central1`) | +| `GOOGLE_APPLICATION_CREDENTIALS` | No | Path to OAuth2 credentials JSON for Vertex AI | +| `MODEL_LOGIC` | No | Override the reasoning model | +| `MODEL_WRITER` | No | Override the creative writing model | +| `MODEL_ARTIST` | No | Override the visual-prompt model | +| `MODEL_IMAGE` | No | Override the image generation model | +| `FLASK_DEBUG` | No | Enable Flask debug mode (`True`/`False`) | + +## Length Presets + +The **Length** setting controls structural complexity, not just word count. It determines the narrative framework, chapter count, and the number of depth-expansion passes the planner performs. + +| Preset | Approx Words | Chapters | Depth | Description | +| :--- | :--- | :--- | :--- | :--- | +| **Flash Fiction** | 500 – 1.5k | 1 | 1 | A single scene or moment. | +| **Short Story** | 5k – 10k | 5 | 1 | One conflict, few characters. | +| **Novella** | 20k – 40k | 15 | 2 | Developed plot, A & B stories. | +| **Novel** | 60k – 80k | 30 | 3 | Deep subplots, slower pacing. | +| **Epic** | 100k+ | 50 | 4 | Massive scope, world-building focus. | + +> **Note:** This engine is designed for **linear fiction**. Branching narratives ("Choose Your Own Adventure") are not currently supported. + +## Data Structure & File Dictionary + +All data is stored in `data/`, making backup and migration simple. ### Folder Hierarchy + ```text data/ ├── users/ -│ └── {user_id}/ # User-specific data -│ └── {Project_Name}/ # Project Root -│ ├── bible.json # Project Source of Truth -│ └── runs/ # Generation History -│ └── run_{id}/ # Specific Job/Run +│ └── {user_id}/ +│ └── {Project_Name}/ +│ ├── bible.json # Project source of truth +│ └── runs/ +│ └── run_{id}/ │ ├── web_console.log -│ └── Book_{N}_{Title}/ # Generated Book Artifacts +│ └── Book_{N}_{Title}/ │ ├── manuscript.json │ ├── tracking_events.json +│ ├── tracking_characters.json +│ ├── chapters.json +│ ├── events.json +│ ├── final_blueprint.json +│ ├── usage_log.json +│ ├── cover_art_prompt.txt │ ├── {Title}.epub -│ └── ... -├── personas/ # Global Author Personas +│ └── {Title}.docx +├── personas/ │ └── personas.json -├── fonts/ # Cached Google Fonts -└── style_guidelines.json # Global AI Writing Rules +├── fonts/ # Cached Google Fonts +└── style_guidelines.json # Global AI writing rules ``` ### File Dictionary | File | Scope | Description | | :--- | :--- | :--- | -| **`bible.json`** | Project | The master plan. Contains the series title, author metadata, full character list, and the high-level plot outline for every book in the series. | -| **`manuscript.json`** | Book | **The Book.** A JSON array containing the raw text of every chapter written so far. Used to resume generation if interrupted. | -| **`events.json`** | Book | The structural outline (e.g., Hero's Journey beats) generated by the Architect. | -| **`chapters.json`** | Book | The detailed writing plan. Lists every chapter with its title, POV character, pacing, and estimated word count. | -| **`tracking_events.json`** | Book | The "Story So Far". Contains a cumulative summary of the plot and a log of key events to ensure continuity. | -| **`tracking_characters.json`** | Book | Tracks the current state of characters (e.g., "wearing a red dress", "lost an arm") to ensure visual consistency. | -| **`final_blueprint.json`** | Book | The final metadata state after the book is finished. Includes any new characters or plot points invented during writing. | -| **`usage_log.json`** | Book | A detailed report of AI token usage and estimated cost for this specific book. | -| **`cover_art_prompt.txt`** | Book | The exact prompt sent to the image generator (Imagen/Vertex) to create the cover. | -| **`{Title}.epub`** | Book | The compiled eBook file, ready for Kindle/Apple Books. | -| **`{Title}.docx`** | Book | The compiled Word document for editing. | +| `bible.json` | Project | Master plan: series title, author metadata, character list, and high-level plot outline for every book. | +| `manuscript.json` | Book | Every written chapter in order. Used to resume generation if interrupted. | +| `events.json` | Book | Structural outline (e.g., Hero's Journey beats) produced by the planner. | +| `chapters.json` | Book | Detailed writing plan: title, POV character, pacing, estimated word count per chapter. | +| `tracking_events.json` | Book | Cumulative plot summary and chronological event log for continuity. | +| `tracking_characters.json` | Book | Current state of every character (appearance, clothing, location, injuries, speech patterns). | +| `final_blueprint.json` | Book | Post-generation metadata snapshot: captures new characters and plot points invented during writing. | +| `usage_log.json` | Book | AI token counts and estimated USD cost per call, per book. | +| `cover_art_prompt.txt` | Book | Exact prompt submitted to Imagen / Vertex AI for cover generation. | +| `{Title}.epub` | Book | Compiled eBook, ready for Kindle / Apple Books. | +| `{Title}.docx` | Book | Compiled Word document for manual editing. | -## 📖 JSON Data Schemas - -Details on the internal structure of key data files. +## JSON Data Schemas ### `bible.json` -Located at the root of the project folder. + ```json { "project_metadata": { @@ -205,7 +273,7 @@ Located at the root of the project folder. }, "characters": [ { - "name": "John Doe", + "name": "Jane Doe", "role": "Protagonist", "description": "Physical and personality details..." } @@ -222,27 +290,27 @@ Located at the root of the project folder. ``` ### `manuscript.json` -Located in each `Book_X` folder. + ```json [ { "num": 1, "title": "Chapter Title", - "pov_character": "John Doe", + "pov_character": "Jane Doe", "content": "# Chapter 1\n\nThe raw markdown text of the chapter..." } ] ``` ### `tracking_characters.json` -Located in each `Book_X` folder. Updates dynamically. + ```json { - "Character Name": { + "Jane Doe": { "descriptors": ["Blue eyes", "Tall"], "likes_dislikes": ["Loves coffee"], "last_worn": "Red dress (Ch 4)", "major_events": ["Injured leg in Ch 2"] } } -``` \ No newline at end of file +``` diff --git a/ai_blueprint.md b/ai_blueprint.md index 3b0f5db..df104c7 100644 --- a/ai_blueprint.md +++ b/ai_blueprint.md @@ -1,6 +1,6 @@ # AI Blueprint: Update README.md -**Version:** 1.0.0 +**Version:** 1.0.1 ## System Rules: AI Blueprint Execution Protocol **CRITICAL:** Every time an AI reads or uses this blueprint to perform a task, it MUST adhere to the following strict protocol before finishing: