E2E all-in: NBA (reference dogfood)

First shipped dogfood for the generic E2E stack: topic NBA, effort low, collection e2e/NBA. For any other topic or tier, use scripts/e2e_stack.py or /e2e-watch.

This is not in default CI. Sci-Hub stays off. Paperful never sends mail.

Honesty

Rule

Detail

Opt-in

PAPERFUL_E2E=1 or scripts/e2e_stack.py --force

Isolated writes

Only -C e2e/NBA (created on first snowball --gate auto)

Soft-skip

Twenty, SearXNG, headed tabs, Scholar/EZProxy when not ready

No Sci-Hub

Not in the harness; see scihub if you opt in yourself

Watch

Every live run is babysat; see Babysitting

Preconditions

  1. Zotero 10+ local write API; paperful doctor (exit 0 or 1).

  2. Real Unpaywall email in config.toml.

  3. Prefer OPENALEX_API_KEY (shared no-key pool is easy to exhaust).

  4. For summarize + browser-agent: [llm].enabled,
    uv sync --extra llm --extra browser-agent (Python ≥ 3.11), Ollama up.

  5. Optional: [twenty].enabled + TWENTY_API_KEY + base_url;
    [searxng].base_url or SEARXNG_BASE_URL (JSON API).

  6. Optional headed tabs: PAPERFUL_E2E_TABS=1.

Profiles

File

Kind

Role

profiles/e2e-nba-search.toml

snowball

Same caps as harness (gate = auto, fetch_pdfs = full)

profiles/e2e-nba-run.toml

run

all --profile e2e-nba-run slice on e2e/NBA

One command

make e2e-nba
# or:
PAPERFUL_E2E=1 uv run python scripts/e2e_stack.py --topic NBA --effort low

Resume after a failed phase (keep the same --topic and --effort):

PAPERFUL_E2E=1 uv run python scripts/e2e_stack.py --topic NBA --effort low \
  --from-phase run_fetch --run-id <id>

Machine witness: state/e2e/<run-id>/report.json and REPORT.md (older runs may be under state/e2e-nba/).

Phases

#

Phase

What runs

Required

0

doctor

paperful doctor --no-guide

yes (exit 2 fails)

1

snowball_search

snowball search NBA … --gate auto --fetch-pdfs full -C e2e/NBA (optional --dry-run-search prelude)

yes

2

snowball_orcid

ORCIDs from candidates.jsonl → snowball orcid … --author-site-preflight

yes if ≥1 ORCID, else skip

3

run_fetch

run -C e2e/NBA --try-all --retry-failed --upgrade-linked --browser-agent

yes

4

hygiene

all … --steps lint,fix-metadata,summarize --apply

yes (summarize soft if LLM off)

5

authors_pack

authors --apply → snowball packs promote e2e-nba → dry-run

soft

6

twenty

twenty lookup/sync --apply --limit 5

soft

7

reachout

reachout -C e2e/NBA --lookup --to …/reachout.csv

yes (CSV may have 0 data rows)

8

handoff_tabs

gaps --list-missing --handoff list (or tabs if PAPERFUL_E2E_TABS=1)

yes

9

report

report.json + REPORT.md

yes

Lane matrix (fetch)

Default sources from config (unpaywall … htmlpdf), then late author_site (when a pack/contact exists), optional Scholar (session), browser_agent when LLM + extra are ready. No Sci-Hub.

SearXNG is only used inside snowball --author-site-preflight, not as a run source. Twenty feeds listings/emails; it never sends mail.

Pass signals

  • ≥1 snowball candidate for NBA in 2025–2026

  • Parents under e2e/NBA after gate auto

  • After fetch: at least one attach or explicit miss rows (not a silent empty run)

  • reachout.csv exists

  • report.json has "ok": true for required phases

  • Soft-skips logged when Twenty / SearX / tabs / LLM are off

Babysitting a live run

Do not fire-and-forget. Use Cursor /watch (or the same discipline by hand):

  1. Use /e2e-watch or copy templates/e2e-watch-assessment.md into assessments/YYYY-MM-DD-e2e-<topic>-watch.md (gitignored).

  2. Record command, terminal path, start time.

  3. Poll the terminal. Progress (new phase lines) is not a hang.

  4. Hang quiet window: 10 minutes (longer for one browser-agent item or a long summarize batch).

  5. On failure: diagnose → backup once before first code edit → minimal fix → focused test (uv run pytest -q tests/test_e2e_nba.py or narrower) → resume with --from-phase / --run-id.

  6. On hang: kill only that PID; then the failure path.

  7. Finalize the assessment: completed or blocked, duration, git diff --stat.

Harness log: state/e2e/<run-id>/watch.log.
Assessment file: human/agent narrative. Keep both.

Hermetic tests

uv run pytest -q tests/test_e2e_nba.py

Live marker (skipped unless env set):

PAPERFUL_E2E=1 uv run pytest -q tests/test_e2e_nba.py -m e2e_live

Manual copy-paste (without the harness)

SCOPE=(-C e2e/NBA --year-from 2025 --year-to 2026)

uv run paperful snowball search NBA --year-from 2025 --year-to 2026 \
  --max-candidates 12 --depth 1 --gate dry-run

uv run paperful snowball search NBA --year-from 2025 --year-to 2026 \
  --max-candidates 12 --depth 1 --gate auto --fetch-pdfs full -C e2e/NBA

# then ORCID seeds from state/snowball/<run>/candidates.jsonl author_records
uv run paperful run "${SCOPE[@]}" --try-all --retry-failed --upgrade-linked --browser-agent
uv run paperful all "${SCOPE[@]}" --steps lint,fix-metadata,summarize --apply
uv run paperful reachout -C e2e/NBA --lookup --to reachout-nba.csv
uv run paperful gaps -C e2e/NBA --list-missing --handoff list

Cleanup

Review e2e/NBA in Zotero. Trash or keep. Author pack e2e-nba under state/author-packs/ can stay for reruns. Do not point production BBNJ profiles at this collection.