Workflows¶

Operator sequences for one collection slice. A run config (also called a profile) saves that slice. paperful all runs the usual chain in one process. Individual commands stay available and can still be chained with &&.

Run a recipe again when you want another pass. Recipes below are copy-paste CLI commands (paperful all and named run configs).

Workbench first? Click-by-click jobs (Preview → Grab → Attach, Discover, Repair) live in Walkthroughs. The same verbs are on docker compose up / paperful serve: gui.md. The research-pack (cited works → parents → PDFs) is research-pack.

Do not conflate¶

Word	What it is	Where it lives
Run config / profile	SCOPE plus fetch/write policy for a repeated slice (collection, years, item types, `try_all`, …)	`[profiles.*]` in `config.toml`, or `profiles/<name>.toml` beside that file
`all`	One command that runs a fixed sequence of existing verbs	`paperful all`
`--preset eoi`	Source policy: open access plus campus EZProxy, no Scholar, no Sci-Hub	CLI flag or `preset` in a profile
Playbook	URL → PDF rule (`rewrite` / `scrape` / `synthesize`), hand-written or learned from fetch wins	`[[grey_playbooks]]`, `packs/*.toml`, `packs/learned.toml`
Pack	Witness for one executed sequence. Lists child reports. Not a template you re-run	`state/packs/<id>.json`
`authorwatch`	Named people list → their new papers (ORCID / OpenAlex). Optional `suggest` from `-C`; `briefing` adds OpenAlex recent works / co-authors. Not a crawl hop	`state/authorwatch/<name>/`
`snowball trends`	OpenAlex publication-year counts for a query or search/hybrid profile. Read-only (no library)	CLI + MCP `snowball_trends`; see snowball.md
`authors`	Creator frequency in `-C`; `--apply` seeds a proposed field author pack	`state/reports/*-authors.json`, `state/author-packs/`
`reachout`	Missing PDFs → CSV / RG tabs for author contact. Never fetches, never sends mail	`paperful reachout`
`snowball watch`	Re-run a saved snowball profile; propose unseen works. The rollup is `digest.md` (Watch)	`state/snowball/watches/<name>/`
`inbox watch`	PDF drop folder → attach to matching items; `[inbox].route` can pick a collection for create-on-unmatched when `-C` is omitted	`[inbox].dir`

Grey-lit packs/ and run-config profiles/ are different directories. state/packs/ is the witness, and it is excluded from backups of the checkout. Profiles sit next to config.toml so they are kept with config.

1. The shell sequence¶

This is the pattern all copies. SCOPE is shared so every verb sees the same collection, years, and item types. run adds fetch flags. fix-metadata and summarize add --apply.

SCOPE=(-C BBNJ -T journalArticle --year-from 2021 --year-to 2026)

uv run paperful gaps "${SCOPE[@]}" \
  && uv run paperful run "${SCOPE[@]}" --try-all --retry-failed --upgrade-linked \
  && uv run paperful lint "${SCOPE[@]}" \
  && uv run paperful fix-metadata "${SCOPE[@]}" --apply \
  && uv run paperful summarize "${SCOPE[@]}" --apply

&& stops on the first non-zero exit. lint is not --strict, so findings do not abort the chain. summarize needs [llm].enabled; if the model is off, that step fails and the earlier steps have already finished.

The same flags work one verb at a time. See Commands.

2. Same chain, one process¶

uv run paperful all -C BBNJ -T journalArticle --year-from 2021 --year-to 2026

Default steps, in order:

gaps
run --try-all --retry-failed --upgrade-linked
lint (not --strict)
fix-metadata --apply
summarize --apply

Those run flags and --apply are the builtin all policy. They apply when the command is all and the profile does not set the key. paperful run alone does not turn --try-all on.

The chain stops on the first failing step, same as &&.

Dry run¶

uv run paperful all -C BBNJ -T journalArticle --year-from 2021 --year-to 2026 --dry-run

Step	What `--dry-run` does
`gaps`	Counts, as usual
`run`	Would-hit table. No downloads
`lint`	Read-only findings
`fix-metadata`	Proposes patches. Does not pass `--apply`
`summarize`	Skipped. Summarize has no dry-run and the default destination writes notes

run --dry-run exits before it writes state/runs/…-run.json, so an auto-opened pack may list gaps, lint, and fix-metadata and omit run. That matches a standalone dry run.

LLM off¶

If [llm].enabled is false, all skips summarize and continues. The line says llm.enabled is false. Pass --require-summarize (or set require_summarize = true in the profile) to exit 1 instead.

all --dry-run skips summarize even when the LLM is on.

Pack¶

If no pack is open, all opens one (label: --label, otherwise the profile name, otherwise all) and closes it when the command finishes, including when a step fails. If a pack is already open, all appends to it and leaves it open.

PAPERFUL_PACK=off still keeps each child report out of a pack. all does not open one in that case.

paperful pack show reads the pack (paperful.pack.v1). Each step can include witness_id when the child report under state/runs/ carried a witness block. It does not re-run the steps.

Narrow the chain¶

uv run paperful all -C BBNJ --steps gaps,run,lint
uv run paperful all -C BBNJ --skip summarize

--steps replaces the list. --skip removes names from it. Unknown names exit 1. Optional names you can add, which are not in the default chain: snapshot, dedupe, synthesize, ocr, gaps_handoff.

dedupe honors apply only when all is running it. A direct paperful dedupe --profile … does not trash unless you also pass --apply.

3. Save the slice¶

Profiles live beside config.toml, never under state/.

uv run paperful profile save bbnj-journal \
  -C BBNJ -T journalArticle \
  --year-from 2021 --year-to 2026 \
  --try-all --retry-failed --upgrade-linked \
  --apply \
  --description "BBNJ journal articles 2021–2026"

That writes profiles/bbnj-journal.toml next to the config file in use (--config if you passed it, otherwise the file Paperful loaded). It does not rewrite config.toml. A second save exits 1 until --force.

The same keys can live in config instead of a file:

[profiles.bbnj-journal]
description = "BBNJ journal articles 2021–2026"
collections = ["BBNJ"]
types = ["journalArticle"]
year_from = 2021
year_to = 2026
try_all = true
retry_failed = true
upgrade_linked = true
apply = true

If both exist, the file overlays the table key by key. collections in the file replaces collections in the table; keys only in the table remain.

Then:

uv run paperful all --profile bbnj-journal
uv run paperful profile list
uv run paperful profile show bbnj-journal

profile show prints the merge paperful all would use, including builtin defaults the file left out. profile show with no name prints those defaults alone.

A one-off file that is not in profiles/ works too:

uv run paperful all -f ./bbnj-journal.toml

-f / --run-config overlays --profile when both are set.

Use the profile on one verb¶

--profile works on gaps, run, lint, fix-metadata, summarize, ocr, dedupe, snapshot, restore, and synthesize. A profile that only stores SCOPE does not turn on try_all for a bare run:

uv run paperful gaps --profile bbnj-journal \
  && uv run paperful run --profile bbnj-journal \
  && uv run paperful lint --profile bbnj-journal \
  && uv run paperful fix-metadata --profile bbnj-journal \
  && uv run paperful summarize --profile bbnj-journal

Put try_all, retry_failed, and upgrade_linked in the profile if run should use them without extra flags. apply = true makes fix-metadata --profile and summarize --profile write. It does not make dedupe --profile trash.

Turn a saved bool off for one invocation:

uv run paperful run --profile bbnj-journal --no-try-all
uv run paperful all --profile bbnj-journal --no-apply

Pairs: --try-all/--no-try-all, --retry-failed/--no-retry-failed, --upgrade-linked/--no-upgrade-linked, --apply/--no-apply, --library/--no-library. Passing -C replaces the profile’s collections and clears library unless you also pass --library.

Precedence¶

Builtin all policy (steps, try-all, retry-failed, upgrade-linked, apply) — only for all and profile show
[profiles.*] in config.toml
profiles/<name>.toml beside that config (same name overlays the table)
--profile NAME or -f / --run-config FILE
Explicit CLI flags

CLI always wins for a key you passed. Omitted flags keep the profile value, then the builtin all value when the command is all.

Docker¶

profiles/ is next to config.toml on the data dir (/data in Compose). Create it beside config, not inside the image:

mkdir -p ../paperful-data/profiles
docker compose run --rm paperful all --profile bbnj-journal --dry-run

Same commands as uv run. Headed session login stays on the host. For LLM / RAG / browser-agent inside the container, build heavy (PAPERFUL_IMAGE_MODE=heavy). See Docker.

4. Deep collection pass (example template)¶

Operator recipe for one collection slice: dedupe, aggressive PDF fetch (including the browser-agent lane), metadata write-back, grounded summaries, then manual handoff for whatever automation missed. Replace placeholders with your collection path (name or parent/subcollection), profile name, and pack label. Subcollections are included automatically when you pass a parent path.

Prerequisites: Zotero 10+ with write API, a real Unpaywall email, [llm].enabled for summarize and the auto browser_agent lane on run, uv sync --extra llm --extra browser-agent on the host (Python 3.11+) or a heavy Compose image, and headed paperful session login ezproxy / scholar when you use those sources. Optional: [camofox].enabled plus a running camofox-browser server for a scripted try before the LLM agent (camofox.md). Optional: [inbox].dir so --handoff tabs can continue into inbox watch. Campus-only policy: add --preset eoi on run / all.

paperful all forwards --handoff, --include-doi-tabs, --promote, --htmlpdf, and the other run-step flags to its internal run. Add --after-handoff-gaps (or a gaps_handoff step) to run gaps --list-missing with the same handoff after the chain. The shell sequence below stays useful when you want dedupe first, or a different handoff mode after run than after the leftover-gaps pass.

Review duplicates before merging:

uv run paperful dedupe -C COLLECTION --dry-run
# read state/dedupe-packs/*.md
uv run paperful dedupe -C COLLECTION --apply
# title+year merges (optional):
uv run paperful dedupe -C COLLECTION --apply --apply-medium

One-shot chain (dedupe → fetch with relogin + browser agent → lint → metadata → summarize → sequential handoff for stragglers). Add year/type filters to SCOPE when you need them (see scope filters):

SCOPE=(-C COLLECTION)
# SCOPE=(-C COLLECTION -T TYPE --year-from YYYY --year-to YYYY)

uv run paperful pack open --label PACK_LABEL

uv run paperful dedupe "${SCOPE[@]}" --apply && \
uv run paperful gaps "${SCOPE[@]}" && \
uv run paperful run "${SCOPE[@]}" \
  --try-all --retry-failed --upgrade-linked \
  --browser-agent \
  --handoff tabs && \
uv run paperful lint "${SCOPE[@]}" && \
uv run paperful fix-metadata "${SCOPE[@]}" --apply && \
uv run paperful summarize "${SCOPE[@]}" --apply && \
uv run paperful gaps "${SCOPE[@]}" --list-missing --handoff walk

uv run paperful pack close

Same policy as all, with dedupe first and handoff on the run step plus a leftover-gaps pass:

uv run paperful all -C COLLECTION \
  --steps dedupe,gaps,run,lint,fix-metadata,summarize \
  --apply --browser-agent \
  --handoff tabs --after-handoff-gaps --gaps-handoff walk

Handoff modes: tabs (batch open, optional inbox watch after tabs), watch (tabs then poll [inbox].dir), walk (one URL at a time via [gaps].downloads_dir). See Sources (manual handoff after soft blocks).

Save the slice for reruns. Profile TOML can store handoff / gaps_handoff; CLI --handoff still overrides one invocation:

uv run paperful profile save PROFILE_NAME \
  -C COLLECTION \
  --try-all --retry-failed --upgrade-linked \
  --apply \
  --steps dedupe,gaps,run,lint,fix-metadata,summarize \
  --description "DESCRIPTION"

uv run paperful all --profile PROFILE_NAME --browser-agent \
  --handoff tabs --after-handoff-gaps --gaps-handoff walk

Dry-run the automated half first:

uv run paperful all -C COLLECTION \
  --steps dedupe,gaps,run,lint,fix-metadata,summarize \
  --apply --browser-agent --dry-run

(summarize is skipped; dedupe and run do not apply merges or downloads.)

5. Research pack (cited works → parents → PDFs)¶

The operator playbook is research-pack. Short form:

Cited inside collection PDFs, not already in the library. Always dry-run the gap pack and ingest-dois classify before --apply. Then run (or inbox) fills PDFs. Then dedupe.

docker compose run --rm paperful refs gap -C BBNJ
# read state/refs-gaps/*/pack.md
docker compose run --rm paperful ingest-dois --from-pack state/refs-gaps/<stamp> -C BBNJ
docker compose run --rm paperful ingest-dois --from-file state/refs-gaps/<stamp>/dois.txt -C BBNJ --apply --tag bbnj
docker compose run --rm paperful run -C BBNJ
docker compose run --rm paperful inbox drain    # optional: PDFs dropped into [inbox].dir
docker compose run --rm paperful dedupe -C BBNJ --apply

Briefing / note mentions (DOIs or titles named in markdown or HTML, not bibliographies inside PDFs): coverage --from-file or --from-note writes state/coverage/<stamp>/ (paperful.coverage.pack.v1). Always dry-run. Feed missing rows through the same ingest-dois --from-pack path as refs gap. See commands.md (coverage, ingest-dois).

Contributors: the same verbs with uv run paperful ….

6. Field author packs (corpus frequency → author_site)¶

Rank creators already in -C, write a report, and seed a proposed pack for the author_site grey lane. Corporate name-only creators (FAO, …) stay in the report; only people go into the pack. Promote before fetch.

uv run paperful authors -C ocean/BBNJ                    # dry-run tables
uv run paperful authors -C ocean/BBNJ --apply            # report + proposed pack
uv run paperful snowball packs promote ocean-bbnj        # enable author_site
uv run paperful run -C ocean/BBNJ                        # grey:author_site when names match

This library’s BBNJ folder is ocean/BBNJ, not a top-level BBNJ. Dogfood on a small sibling: BBNJ author lanes.

Snowball --author-site-preflight is the frontier/co-author path to the same state/author-packs/ files; authors is the in-library frequency path. A CRM and a metasearch can feed the same packs later; that setup is Twenty and SearXNG.

7. Reachout (ask authors, do not fetch)¶

When you want contact data instead of grab modalities: no Unpaywall, no proxy, no Sci-Hub. Missing PDFs only. Emails come from the item (extra, mailto:, abstract). An optional CRM cache is Twenty and SearXNG. Paperful never sends mail.

uv run paperful reachout -C ocean/BBNJ --to reachout.csv
uv run paperful reachout -C ocean/BBNJ --non-oa-only --lookup --to reachout.csv
uv run paperful reachout -C ocean/BBNJ --request-rg --handoff tabs

--lookup is that optional CRM search. --handoff tabs opens existing ResearchGate publication URLs; you click Request full-text.

8. All-in E2E (NBA throwaway)¶

An opt-in harness that chains snowball → ORCID → full fetch → summarize → Twenty/reachout/handoff on -C e2e/NBA (not production BBNJ). Not in default CI. See E2E NBA and make e2e-nba.

What this is not¶

Not a workflow engine. Steps are a list, not a graph.
Not Rollup’s cron, and not TranscriptX’s saved GUI profiles.
Not a grey-lit playbook. URL → PDF rules stay in [[grey_playbooks]].
Not a pack. A pack records one run; a profile is the input you can run again.
Not a field author pack. state/author-packs/ is for author_site; state/packs/ is the run witness.
Not a fetch preset. reachout never downloads; --preset oa still grabs.