Add unlisted visibility: served by link, off the index, noindex

Internal papers need a durable link before member login exists. 'unlisted'
serves a document at its slug but keeps it off /documents and sends
X-Robots-Tag: noindex so it stays out of search results.

Serving and listing are now separate questions: visible() decides whether a
document can be served at all and stays the seam login attaches to, listed()
decides whether it shows on the index. 'members' remains hidden AND
unservable, so the weaker state cannot be mistaken for the gate.

This is obscurity, not access control, and both the README and the manifest
say so. An unlisted link is forwardable.

README.md

@@ -44,8 +44,13 @@ rebuild, no redeploy.** The app re-reads the manifest whenever its mtime changes
 - `slug` is the permanent URL: `greenlanescouts73.org/documents/<slug>`. **Never change one that has
   been printed or emailed.** To publish a new version, point the same slug at the new filename.
 - `category` matches an id in the manifest's `categories`; anything else lands under "Everything else".
-- `visibility`: `public`, or `members` for later. `members` documents are hidden from the index and
+- `visibility`:
-  return 404 - **this is not a working gate yet**, it is the seam login will attach to.
+  - `public` - listed on `/documents`, open to anyone.
+  - `unlisted` - served at its slug to anyone holding the link, kept off the index, sent with
+    `X-Robots-Tag: noindex`. For internal papers that need a durable link before login exists.
+    **This is obscurity, not access control.** An unlisted link is forwardable; assume it will be.
+  - `members` - hidden and unservable, returns 404. **This is not a working gate yet**, it is the
+    seam login will attach to.
 - `updated` optional; without it the file's own mtime is shown.
 
 Documents are served **through the app** (`/documents/{slug}`), never from a static mount. Anything

app/app.py

@@ -594,8 +594,10 @@ def document_file(slug: str):
 </div></div>"""
         return HTMLResponse(page("Not found · Pack & Troop 73", body, "docs"), status_code=404)
     _, mime = documents.kind(d)
+    headers = {"X-Robots-Tag": "noindex, nofollow"} if documents.noindex(d) else None
     return FileResponse(d["path"], media_type=mime, filename=d["path"].name,
-                        content_disposition_type=documents.disposition(d))
+                        content_disposition_type=documents.disposition(d),
+                        headers=headers)
 
 
 FAQ = [

app/documents.py

@@ -17,10 +17,15 @@ member login exists, gating a document is one change in visible() and not a
 single URL moves.
 
 visibility:
-    "public"   anyone, which is everything today
+    "public"    anyone, and listed on /documents.
-    "members"  reserved for the login that does not exist yet. Until it does,
+    "unlisted"  served at its slug to anyone holding the link, but kept off the
-               these are hidden from the index and return 404 rather than 403,
+                index and sent with X-Robots-Tag: noindex. For internal papers
-               because a 403 advertises a document we cannot actually gate yet.
+                that need a durable link before member login exists. This is
+                obscurity, not access control: treat an unlisted link as
+                forwardable, because it is.
+    "members"   reserved for the login that does not exist yet. Until it does,
+                these are hidden AND unservable, returning 404 rather than 403,
+                because a 403 advertises a document we cannot actually gate yet.
 """
 
 import datetime
@@ -109,7 +114,7 @@ def _clean(raw):
             "description": str(d.get("description") or "").strip(),
             "category": str(d.get("category") or "").strip(),
             "unit": unit if unit in ("pack", "troop", "both") else "both",
-            "visibility": vis if vis in ("public", "members") else "members",
+            "visibility": vis if vis in ("public", "unlisted", "members") else "members",
             "updated": str(d.get("updated") or "").strip(),
         })
     return {"categories": cats, "documents": docs}
@@ -133,14 +138,24 @@ def manifest():
 
 
 def visible(doc):
-    """The single gate. Member login plugs in here and nowhere else."""
+    """The single gate on SERVING. Member login plugs in here and nowhere else."""
-    return doc.get("visibility") == "public" and doc["path"].is_file()
+    return doc.get("visibility") in ("public", "unlisted") and doc["path"].is_file()
+
+
+def listed(doc):
+    """The separate, weaker question of whether it appears on the index."""
+    return doc.get("visibility") == "public" and visible(doc)
+
+
+def noindex(doc):
+    """Unlisted documents should not turn up in a search result."""
+    return doc.get("visibility") == "unlisted"
 
 
 def listing():
-    """Visible documents grouped into their categories, in manifest order."""
+    """Listed documents grouped into their categories, in manifest order."""
     m = manifest()
-    docs = [d for d in m["documents"] if visible(d)]
+    docs = [d for d in m["documents"] if listed(d)]
     known = {c["id"] for c in m["categories"]}
     groups = []
     for cat in m["categories"]: