Snowball

Status: keyword search (multi-term AND / --or, trailing * stem expansion), multi-DOI / multi-ORCID / collection seeds, --seeds-file, hybrid (keyword hits, then one hop), refs, cited-by, and OpenAlex keywords, depth up to 5 under caps, gates dry-run / approve-each / approve-batch / auto, overlap ranking, --fetch-pdfs, --dedupe-scope per invocation, optional --dedupe-after, opt-in author-site preflight / snowball packs promote, and pull-only watch are implemented, plus a frontier digest on a saved queue or watch. Optional stopping rules are off until you set them: one hop, then Keep/Skip, then snowball continue. Config honors dedupe_scope, tag_prefix, default_tags, types, oa_only, venues, listed_in, languages, min_seed_citations, note_provenance, and backends. --refine writes query suggestions when [llm] is on and does not create items. expand = cited_authors stays off. Phases live in ROADMAP.

Snowball grows a library outward from a keyword (or several), one or more DOIs, one or more people (ORCID), or an existing collection. It proposes works, then creates items under an explicit gate. run fills PDFs for items that already exist. fetch_pdfs can do that in the same process, limited to the keys just created. How far one command reaches is drawn below.

paperful snowball search "area based management tools" --year-from 2018
paperful snowball search "marine spatial planning" --listed-in doaj,medline
paperful snowball search "marine spatial planning" "offshore renewable energy"
paperful snowball search msp ore --or
paperful snowball doi 10.1038/s41586-021-03819-2 --depth 2 --direction both
paperful snowball doi 10.1038/s41586-021-03819-2 10.1126/science.aao5646
paperful snowball orcid 0000-0002-9162-9618
paperful snowball orcid 0000-0002-9162-9618 0000-0002-1825-0097 --depth 1
paperful snowball collection "Inbox/Seeds" --direction refs
paperful snowball run --profile doi-refs-gated
paperful snowball apply <run-id> -C "Inbox/Snowball"
paperful snowball hybrid "high seas EIA" --hybrid-seeds 5 --direction refs
paperful snowball search "high seas EIA" --gate auto --fetch-pdfs fast -C "Inbox/Snowball"
paperful snowball orcid 0000-0002-9162-9618 --gate auto --fetch-pdfs full -C "Snowball/0000-0002-9162-9618"
paperful snowball search "high seas EIA" --gate dry-run --format json
paperful snowball apply <run-id> -C "Inbox/Snowball" --format json
paperful snowball watch save bbnj --profile keyword-scout
paperful snowball watch run bbnj --digest
paperful snowball watch run bbnj --refresh
paperful snowball watch digest bbnj
paperful snowball digest --run-id <run-id>
paperful snowball watch briefing bbnj
paperful snowball doi 10.1038/s41586-021-03819-2 --depth 3 --stop-rule marginal_yield
paperful snowball continue <run-id>

--format json on crawl (search / hybrid / doi / orcid / collection), run, resume, continue, and apply prints one paperful.agent.json.v1 object on stdout (run path + summary; progress on stderr). Mixed apply writes exit 3. Prefer that over MCP for shell agents.

search, hybrid, doi, orcid, and collection are seeds under one verb. search and hybrid take one or more keyword terms (AND by default; --or matches any). doi and orcid take one or more seeds in a single crawl (shared caps / gate / -C). hybrid is the keyword-then-hop job. There is no separate top-level harvest, crawl, or discover. watch re-runs a saved profile on a schedule you choose; see Watch.

Watch

A watch is a saved snowball profile plus a seen-set on disk. You run it when you want. Paperful does not schedule it (your own launchd or cron may call watch run). The first run records the current frontier and proposes nothing. Later runs write only works that were not in that set. Creating items stays on snowball apply (or a separate writing-gate crawl). Watch always forces gate = dry-run and fetch_pdfs = off, so a profile with gate = auto cannot create parents or download PDFs from a watch.

People you follow (ORCID lists, no hop) are authorwatch, not this profile watch.

paperful snowball watch save bbnj --profile keyword-scout
paperful snowball watch run bbnj          # first time: baseline, inbox empty
paperful snowball watch run bbnj --digest # later: N new, then digest.md
paperful snowball watch run bbnj --refresh  # skip last-run date cursor; rewrite responses.json
paperful snowball watch show bbnj
paperful snowball watch rebaseline --all  # dry run; --apply once after the Oct 2026 OpenAlex topic change
paperful snowball watch digest bbnj       # ranked rollup; --apply -C files a note
paperful snowball digest --run-id <run-id>
paperful snowball watch briefing bbnj     # thin markdown; same note tag
paperful snowball briefing --run-id <run-id>
paperful snowball apply <run-id> -C Inbox/Snowball   # only if you want parents

Oct 2026 topic change. OpenAlex reassigned work topics (and keywords) in October 2026, so a keyword watch baselined earlier can see old works reappear as “new”. snowball watch rebaseline (dry run by default; --apply) clears baseline_at for keyword-based watches baselined before 14 Oct 2026, so the next watch run absorbs the current hits as seen and proposes nothing. It keeps seen.json, applied.json, and the inbox, and marks the watch (rebaselined_for) so it is done once; --force repeats it. Every snowball run’s summary.json now records openalex_run_date and openalex_topic_epoch (pre-2026-10 or 2026-10), because keyword hops before the change cannot be reproduced.

Ledger under state/snowball/watches/<name>/:

File	Role
watch.json	paperful.snowball.watch.v1 — profile name, baseline_at, last_run_at, last_run_id
seen.json	Identities already recorded (doi: or openalex:)
inbox.jsonl	Append-only proposed status = new rows
responses.json	paperful.snowball.watch.responses.v1 — content hash per identity (new / unchanged / metadata_changed / reassigned_id)
briefing.md	Thin markdown export (watch briefing)
digest.md	Frontier digest (watch digest or watch run --digest)

Every crawl summary.json (and snowball apply / snowball watch run command reports) also carries additive PRISMA-style counts: identified, screened, kept, added. Counts only — no recall percentage. Watch runs add watch_drift (new / unchanged / metadata_changed / reassigned_id) from responses.json. --refresh skips the last-run from_created_date cursor and rewrites that cache; Paperful still does not schedule watches.

export-proposals reads inbox.jsonl here (or candidates.jsonl on a crawl run dir). See BibTeX / RIS from the queue.

paperful snowball briefing --run-id writes thin markdown next to candidates.jsonl. paperful snowball digest --run-id writes digest.md there: new / exists / version / deferred, overlap detail on the top 25 new rows (score, why, hop, overlap), a suggested -C from [snowball].target_collection (or the watch profile’s target_collection), and copy-paste paths for the queue, apply, and resume. A watch digest reads the last run queue. After watch run that file holds only the new proposals, so an exists split is empty there; already seen on the summary is the overlap with the prior frontier. If OpenAlex stopped early, summary.json still says deferred and the digest points at resume. snowball digest --run-id on a normal crawl queue still splits new / exists / version / deferred. The inbox count is separate, because the inbox is new-only and keeps older proposals. Neither command creates library items. --apply with -C files a collection note tagged paperful:frontier-briefing. OA / grey stamps already on the row (oa:…, grey:…, is_oa, oa_status) are printed; there is no extra API call. The alert loop (poll → ingest parents → briefing) and weekly what-cites digest are keepup. Standalone inbox apply:

paperful snowball watch apply bbnj -C Inbox/Snowball          # dry-run
paperful snowball watch apply bbnj -C Inbox/Snowball --apply  # parents + ingested.md

Schedule it yourself

Paperful does not run a timer. Prefer keepup run for the full loop. For watch-only digest:

Operators (Compose), from the repo:

15 7 * * 1 cd /path/to/paperful && docker compose run --rm paperful keepup run field --apply

Contributors, on a host checkout:

15 7 * * 1 cd /path/to/library && uv run paperful keepup run field --apply

launchd and systemd below are the host-checkout form (uv). A Compose operator uses the cron line above (docker compose run).

launchd (~/Library/LaunchAgents/paperful.keepup.field.plist):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>paperful.keepup.field</string>
  <key>WorkingDirectory</key>
  <string>/path/to/library</string>
  <key>ProgramArguments</key>
  <array>
    <string>/path/to/uv</string>
    <string>run</string>
    <string>paperful</string>
    <string>keepup</string>
    <string>run</string>
    <string>field</string>
    <string>--apply</string>
  </array>
  <key>StartCalendarInterval</key>
  <dict>
    <key>Weekday</key>
    <integer>1</integer>
    <key>Hour</key>
    <integer>7</integer>
    <key>Minute</key>
    <integer>15</integer>
  </dict>
</dict>
</plist>

systemd user units (~/.config/systemd/user/paperful-watch-bbnj.service and .timer):

# paperful-watch-bbnj.service
[Service]
Type=oneshot
WorkingDirectory=/path/to/library
ExecStart=/path/to/uv run paperful snowball watch run bbnj --digest

# paperful-watch-bbnj.timer
[Timer]
OnCalendar=Mon *-*-* 07:15:00
Persistent=true

[Install]
WantedBy=timers.target

Then systemctl --user enable --now paperful-watch-bbnj.timer. Read state/snowball/watches/bbnj/digest.md when it finishes.

Each watch run also writes a normal state/snowball/<run-id>/ queue. After baseline, that queue is empty (baseline N · proposed 0). After a later run, it holds only the new proposals (proposed N · already seen M), with keep = true so snowball apply can create them. OpenAlex keyword and cited-by calls pass from_created_date from the cursor (YYYY-MM-DD of the last run) so a work indexed after the last check can show up even when its publication year is old. The seen set still drops overlaps. A hybrid profile watches the keyword hit list only (depth 0); a refs hop of those hits stays a separate snowball run.

Snowball and run

Job	Command	What you are doing
Build	paperful snowball …	Find works and, if the gate says so, create parents
Fill	paperful run	PDFs and attach for items already in the library

A dry-run ends with “candidates ready”. A writing gate with fetch_pdfs = off ends with “items created (metadata only)”. Downloaded and attached appear in the summary only after fast or full has run.

A snowball profile is a named job in profiles/. paperful run and paperful all refuse it. Fetch profiles stay fetch profiles.

Seeds

Seed	What it collects	Default depth
Keyword	OpenAlex title/abstract search, plus (by default) works carrying matching OpenAlex keyword tags. Pass several terms for one boolean query (AND by default; --or for any). One term is passed through unchanged. Turn tags off with search_keywords = false or --no-search-keywords	0 — the hit list. Depth 1+ expands those hits and must be set explicitly. A global depth = 1 does not expand every keyword hit
Hybrid	That hit list, then one hop from the top hybrid_seeds DOIs (default 5). Same multi-term rules as Keyword	The hop is always 1. --depth does not add further hops
DOI	Each seed work (hop 0), then its neighbours (referenced_works and/or works that cite it). Pass several DOIs for one shared crawl	1, direction refs+keywords by default. Use --direction refs to drop tags, or cites / both for cited-by
ORCID	Those people’s works (ORCID public API, filled by OpenAlex author filter), then the same expander. Pass several ORCID iDs for one shared crawl	1. One hop out from the combined works
Collection	DOIs already in the seed collection path, then neighbours only	1. -C is the write target (defaults to the seed path)

expand = cited_authors (every paper by every cited author) stays off. It is a later, capped switch.

How a hop is cut

An ORCID run starts at those people’s own works (hop 0). Pass several ORCID iDs in one command (or repeat --orcid on profile save) to merge them into one crawl under shared caps. --depth 1 takes one hop out from those works and stops. It does not walk the neighbours’ neighbours. --direction both takes references and citing works. Each side is capped on its own: --per-hop-limit 25 keeps 25 references and 25 citing works per paper. --per-hop-rank chooses which ones: most-cited (default), least-cited, or random. Use all (or 0) on either cap for every neighbour or every row. After that hop, max_candidates (default 200) can still cut the new + exists list.

--cites-query keeps references and citing works whose title, abstract, or full text matches that OpenAlex search. Cited-by sends it with filter=cites:. Reference ids send it with filter=openalex:. The direction must include refs or cites, and depth must be at least 1. --per-hop-limit all and --max-candidates all still apply to the matches.

paperful snowball doi 10.1016/j.jclepro.2022.132764 --depth 2 --direction both --per-hop-limit all --max-candidates all --cites-query degrowth

paperful snowball orcid 0000-0002-9162-9618 --depth 1 --direction both --per-hop-limit 25 --per-hop-rank most-cited --max-candidates all

        flowchart TB
  person["ORCID"]
  own["Own works · hop 0"]
  person --> own
  paper["Each of those papers"]
  own --> paper
  refs["References · 25 kept"]
  cites["Citing works · 25 kept"]
  paper -->|"direction both"| refs
  paper -->|"direction both"| cites
  list["Own works plus that hop"]
  own --> list
  refs --> list
  cites --> list
  trimmed["Cut to 200 new and exists"]
  kept["Every new and exists row"]
  list -->|"max-candidates 200"| trimmed
  list -->|"max-candidates 0"| kept
    

DOI and collection seeds use the same hop. For DOI, the seed work itself is included as hop 0, then neighbours. Collection expands DOIs already in the seed path. Keyword search stays on the hit list unless you set depth. The usual numbers are in Stop rules. With a stopping rule on and depth greater than 1, that first command stops after hop 1 and waits. See Stopping rules.

direction including keywords adds another side (the built-in default is refs+keywords). OpenAlex stores at most five scored keywords on a work. The hop takes the top keyword_limit (default 3) at or above keyword_min_score and requests works with any of those slugs (keywords.id:a|b|c), then keeps keyword_hop_limit of them. all is refused on both keyword caps: the matching literature is not a closed neighbour list. Set direction = "refs" in config to restore reference-only hops.

Keyword search (and hybrid hop 0) also uses those tags when search_keywords is on (default): the query is resolved via OpenAlex /keywords, then tagged works are unioned with full-text search= hits and ranked by tag overlap. --no-search-keywords or search_keywords = false keeps full-text retrieval only.

Works with no topic have no keywords. Paperful does not quote a share: the old “about 12%” figure predates OpenAlex’s October 2026 topic reassignment and has not been re-measured. When keywords are in the direction, the run says how many seeds have none and will not expand on that side:

2 of 5 seeds have no OpenAlex keywords and will not expand on that side.

Those seeds are skipped on the keyword side only. References and cited-by still run. If keywords is the only side and every seed lacks keywords, the command exits 2 with that sentence and does not query.

Gates

The gate is a config value. The default for a new user is dry-run.

Gate	Library writes	When
dry-run	None	Always safe. fetch_pdfs is ignored
approve-batch	Only keep = true rows	snowball apply <run-id>. A second apply skips DOIs already created
approve-each	Each yes on status = new	Short lists, at most approve_each_max (default 20), and only on a terminal. No rows stay keep = false. Over the cap, or without a terminal, the command exits 2 and points at approve-batch
auto	Every status = new row under the caps	A named profile, after you have dry-run that job once

dry-run, approve-batch, approve-each, and auto are the gates. A writing gate (approve-batch, approve-each, auto) requires a target collection and stops before the crawl if it is missing. Paperful may create that collection path. It does not invent a silent default collection. auto is not a scheduler.

Dedupe runs before create: normalized DOI against dedupe_scope (library by default, or collection, or none logged loudly), then the existing title+year fingerprint. Only status = new rows can become parents. --dedupe-scope on search / doi / orcid / collection / run / resume / watch run overrides the profile for that invocation. snowball apply fingerprints the full library unless you pass --dedupe-scope on apply itself (a saved collection scope does not leak into delayed apply). After create it also writes a state/runs/<stamp>-snowball apply.json report (optional witness block) so an open pack can list the step — dry-run crawls stay queue-only under state/snowball/.

--seeds-file (or - for stdin) loads DOI or ORCID lists for doi / orcid. Trailing * on a keyword (polic*) expands client-side into an OR group; ? and ~ are stripped with a warning. After create, --dedupe-after classify|apply can write state/dedupe-packs/ (off by default).

Opt-in --author-site-preflight writes coauthors.json and a proposed pack under state/author-packs/. paperful authors -C … --apply also seeds a proposed pack from in-library creator frequency (see commands). snowball packs promote <slug> makes it available to the author_site grey lane (grey:author_site). Preflight tries ORCID researcher URLs first. A CRM and a metasearch can fill the rest; that is Twenty and SearXNG. On fetch, author_site is a late lane (after open access and campus, before Scholar). Promote before relying on the pack. Paperful does not send mail.

One-shot PDFs

fetch_pdfs is a mode. The default is off.

Mode	What happens after create
off	Metadata only. Fetch later with paperful run -C on that collection
fast	First pass on the new keys: configured sources, no vault browser. true means this
full	That first pass, then the same stack as run (browser lanes, and the recover agent when [llm] is on) for keys that still have no PDF

paperful snowball search "BBNJ EIA" --gate auto --fetch-pdfs fast -C "Inbox/Snowball"
paperful snowball orcid 0000-0002-9162-9618 --gate auto --fetch-pdfs full -C "Snowball/me"

Both modes stay on the keys snowball just created. Sci-Hub and Scholar run only when they are already in sources. full prints the Sci-Hub and browser-recovery lines when those lanes are in the second pass. If attach fails, the parent remains and the report counts attach_deferred.

Example profile keyword-library: gate = auto, fetch_pdfs = true (fast), target_collection set. keyword-scout is the dry-run twin. doi-refs-gated is approve-batch. orcid-ego-auto creates metadata only.

Stop rules

The picture is How a hop is cut.

Knob	Default	Meaning
depth	0 for keyword, 1 for DOI / ORCID / collection	Hops from the seed. Soft ceiling 5
direction	refs+keywords	refs, cites, both, keywords, similar, or combinations such as refs+similar and refs+cites+keywords. both stays references plus cited-by. similar is one ranked hop (shared references, plus Semantic Scholar recommendations), not a deeper crawl. Set refs to drop the keyword hop
search_keywords	true	On search / hybrid, also pull works tagged with matching OpenAlex keywords (union with full-text hits). --no-search-keywords or false keeps full-text only
keyword_limit	3	How many of a work’s OpenAlex keywords to expand (and how many tag slugs search resolves). Integer 1–5. all and 0 are errors. 5 uses every keyword OpenAlex stored (at most five)
keyword_hop_limit	50	Works kept per seed on the keyword side. A positive integer. all and 0 are errors. A keyword filter is an open query, so it never pages without a cap. Citation per_hop_limit = all still applies on a mixed run
keyword_min_score	0	Drop seed keywords below this similarity. 0 keeps whatever OpenAlex already assigned
max_candidates	200	Stops after this many new + exists rows. all (or 0) keeps every row
per_hop_limit	50	Neighbours kept per seed work per hop (references and cited-by). all (or 0) keeps every one OpenAlex returns
per_hop_rank	most-cited	How a numeric per_hop_limit picks neighbours: most-cited, least-cited, or random
cites_query	unset	OpenAlex search on every reference batch and every cited-by request (title, abstract, or full text). Needs refs or cites in the direction, and depth of at least 1
year_from / year_to	unset	Drop candidates outside the window
study_designs	unset	OpenAlex study design filter (filter=study_designs.id:…). OR several with `
listed_in	unset	OpenAlex trusted journal lists (filter=primary_location.source.listed_in:…). OR several with , / `
types	journal-article-shaped	OpenAlex / Zotero types
oa_only	false	Metadata filter only. It does not change the PDF chain
min_seed_citations	0	Skip cited-by expansion when the seed is below this count
languages, venue_include, venue_exclude	unset	Applied when the source has the field. A missing language does not drop the row
hybrid_seeds	5	How many top DOI hits hybrid expands
approve_each_max	20	Largest approve-each list. Above this, use approve-batch
stop_rule	off	off walks the whole depth in one shot. marginal_yield pauses after each hop until Keep/Skip, then snowball continue (Discover: Next hop) expands only from kept rows. Also asymptotic_coverage, budget, and any. Depth, per_hop_limit, and max_candidates stay as ceilings. See Stopping rules
stop_eps	0.01	marginal_yield: new Keeps divided by screened rows. Fires when that stays below this value
stop_k	2	How many low-yield hops in a row. The rule does not fire before iteration k + 1
stop_tau	0.95	asymptotic_coverage: fraction of a fitted Keep ceiling
stop_min_points	4	Labeled hops required before that fit
stop_max_screened	0	budget: screened new rows. 0 means no cap
refine	false	With [llm].enabled, write up to five query suggestions. They do not start a second crawl

A seed that fails to resolve is status = error. Other seeds continue. The process exits non-zero if any seed failed, using the existing exit ladder (2 for config / doctor).

Stopping rules

stop_rule is off until you set it. The hop caps above still bound every crawl. Precedence is the usual one: built-in default, then [snowball] in config.toml, then a snowball profile, then flags.

With a rule on and depth greater than 1, the first command fetches hop 1 and waits. Keyword depth 0 fetches the hit list only. hybrid is already one hop, so the rule scores that hop and does not walk further. Mark each new row Keep or Skip, then:

paperful snowball continue <run-id>

The next hop expands only from kept new rows that have an OpenAlex id or a DOI. Screened rows are status = new on the open hop. exists, filtered, and error are not that batch. approve-batch does not pre-mark those rows as Skip while a rule is on, so they stay unlabeled until you set Keep or Skip.

--gate auto on continue counts every unlabeled new row as Keep. Discover stays a dry-run search. Advanced Graph has Stop rule, Stop eps, Stop k, and Stop max screened. Next hop appears on the queue while the run is waiting.

CLI flags on search, hybrid, doi, orcid, collection, run, and profile save: --stop-rule, --stop-eps, --stop-k, --stop-max-screened. stop_tau and stop_min_points are config and profile keys only.

continue is not resume. resume finishes a crawl that OpenAlex paused (deferred.json). continue refuses while that file is present. If the pause happens on a later hop, resume first, then continue.

Rule	Fires when
marginal_yield	New Keeps / screened rows stay below stop_eps for stop_k hops, and not before hop k + 1
asymptotic_coverage	Cumulative Keeps reach stop_tau of a fitted ceiling, after stop_min_points hops
budget	Screened new rows reach stop_max_screened (0 means no cap)
any	The first of those three, or an empty kept frontier
exhaustion	No kept work left to expand. Recorded on every hop. Stops the crawl even when another rule is selected

Each hop appends one line per rule to state/snowball/<run-id>/stops.jsonl (paperful.snowball.stop.v1), including rules that did not fire. The stopping sentence is in stop.json (rationale) and is meant for a methods section. summary.json records stop_rule, stop_awaiting_labels, and stopped_by. A depth ceiling or a full max_candidates budget stops the walk as well and says so (stopped_by depth or max_candidates).

This is the saturation idea from SnowBallSLR, reimplemented here. It does not estimate how much of the literature you have found.

Progress

A long crawl keeps a progress bar at the bottom of the terminal, the same bar run and attach use. The label is coloured by step: cyan for hops and references, blue for cited-by, magenta for Crossref and Semantic Scholar, green while items are created, cyan while PDFs are fetched. When the step has a known length (reference ids, cited-by seeds, rows to fill, items to create, PDFs to fetch) the bar shows done/total. Otherwise it spins. A stage line is printed when the step changes (hop 1/2 · 40 seeds · both). When the run finishes, one summary line remains:

hop 1/2 references · 80 ids · 40 searches · 800 papers
crossref · 30 searches · 15 fields updated
fetching PDFs · 4 PDFs
creating · 12 created

Counts start over when the step changes, so a references hop does not repeat fill or PDF totals. searches counts reads in that step (OpenAlex, Crossref, or Semantic Scholar). papers counts OpenAlex works returned. fields updated counts empty title, year, venue, or author fields filled. PDFs counts files saved while fetch_pdfs runs. created counts items written while they are created.

Candidate record

Each run writes state/snowball/<run-id>/candidates.jsonl and summary.json. The summary counts requests, HTTP 429s, retries, and rows by status, backend, and hop.

Schema paperful.snowball.candidate.v1:

Field	Contents
schema, run_id	Version and run id
seed	{type, value} — keyword, doi, orcid, or openalex
hop	Integer. Search hits are 0. References of a seed are 1
direction	search, refs, cites, keywords, or orcid (the person’s own works)
ids	Normalized doi, openalex, s2, pmid, orcid, when known
biblio	Title, year, authors, venue, type, optional OA url, optional study_designs (OpenAlex slugs), optional listed_in (OpenAlex source-list ids)
why	Short reason, for example ref of 10.xxxx/yyyy
status	new, exists, filtered, or error
exists_match	When exists: item_key and doi or title_year
provenance	Backend, endpoint, retrieved_at
gate	The gate for this run
score	overlap * 1000 + cited_by_count for neighbours. Search hits stay on cited_by_count

The dry-run table shows title, year, DOI, why, in library, hop/direction, and backend, with new rows first. Created items are tagged paperful-snowball, paperful-snowball:<backend>, from-<seed-slug> from the candidate’s seed, [snowball].default_tags, and any --tag values. A child note holds the seed, direction, hop, why, run id, and schema version when note_provenance is on. API keys and the mailto address never go in that note.

A separate readable line follows [remarks].surface (a child note tagged paperful-linked by default):

• “Cited by 4 papers in this collection.” when at least one item already in the target collection lists this work in its references. Paperful fetches those OpenAlex reference lists once, caches them under state/cites/, and reuses the cache while the collection’s DOIs stay the same. If the OpenAlex budget runs out, the item is still created and this sentence is skipped.

• When the hop is at least 1 and at least two seeds from this run point at the work: “In the bibliography of 2 of the papers you started from.”, “Cites 2 of the papers you started from.”, or “Linked to 2 of the papers you started from.” Search hits (hop 0) get only the collection sentence.

BibTeX / RIS from the queue

Turn a dry-run queue or watch inbox into bibliography text without creating parents in Zotero:

paperful export-proposals state/snowball/<run-id> proposals.bib
paperful export-proposals state/snowball/watches/<name> watch-inbox.ris

Accepts a run directory (candidates.jsonl), a watch directory (inbox.jsonl), or a path to either file. Rows need at least a resolved DOI or a title in biblio. Agents can call MCP proposal_export with the same pack path and format bibtex or ris (bibliography text lives in the JSON envelope summary.bibliography). Library-scoped export of items you already own stays paperful export (or MCP export).

snowball search … --format json on a dry-run includes summary from summary.json when the in-memory crawl summary is empty (paths.run points at the queue folder).

Publication trends (read-only)

Count OpenAlex works by publication year for a keyword query or a saved search/hybrid profile seed. Does not open the library or create parents:

paperful snowball trends "high seas biodiversity"
paperful snowball trends --profile my-seed --format json

MCP snowball_trends uses the same envelope. Distinct from watch / digest (frontier arrivals since baseline).

Config

Precedence matches run profiles: built-in default, then [snowball] in config.toml, then profiles/<name>.toml, then flags on the command.

[snowball]
enabled = false          # doctor and the CLI refuse snowball until true
depth = 1                # keyword runs still default to depth 0
direction = "refs+keywords"
search_keywords = true   # search/hybrid also unions OpenAlex keyword tags
max_candidates = 200
per_hop_limit = 50         # all keeps every neighbour of a seed
per_hop_rank = "most-cited"  # most-cited | least-cited | random
gate = "dry-run"         # dry-run | approve-each | approve-batch | auto
target_collection = ""
dedupe_scope = "library" # library | collection | none
fetch_pdfs = "off"       # off | fast | full. true means fast
tag_prefix = "paperful-snowball"
default_tags = []
note_provenance = true
backends = ["openalex", "crossref", "semanticscholar", "orcid", "europepmc", "pdf"]
languages = []           # empty = do not filter; a missing language is kept
min_seed_citations = 0
hybrid_seeds = 5
approve_each_max = 20
refine = false           # suggestions only; needs [llm].enabled
dedupe_after = "off"     # off | classify | apply
author_site_preflight = false
author_site_max_authors = 15
author_site_max_queries = 20
stop_rule = "off"        # off | marginal_yield | asymptotic_coverage | budget | any
stop_eps = 0.01
stop_k = 2
stop_tau = 0.95          # config and profile only; no CLI flag
stop_min_points = 4
stop_max_screened = 0    # budget; 0 means no cap

[searxng].base_url (or SEARXNG_BASE_URL) is an optional metasearch for author-site remainder discovery. It is never in sources by default. What that engine is, and which hosts answer it: Twenty and SearXNG.

enabled = false until you opt in, the same posture as [llm]. doctor reports that flag, whether keys are present, and whether a backend answers. A missing OpenAlex key warns. A missing Semantic Scholar key does not, and neither missing key aborts the other backends.

email at the top of config.toml is the contact address for Unpaywall and Crossref. For heavy crawls, put a free OPENALEX_API_KEY in the environment (never in config.toml). Limits, resume after budget, Semantic Scholar keys, and an optional local or hosted OpenAlex parquet snapshot: Advanced.

Each crawl writes state/snowball/<run-id>/candidates.jsonl as it goes, for every configured backend. A rate limit, outage, or interrupt keeps that file and, when OpenAlex stops the crawl, deferred.json. Continue with:

paperful snowball resume <run-id>

If the queue is already complete and deferred.json is gone, the same command does not search again. With --gate auto, -C, and --fetch-pdfs, it creates any rows still missing from the library and fetches PDFs for keys that do not already have one.

A keyword search that asks for more than 5,000 works follows OpenAlex’s cursor instead of stopping at 50 pages.

Short 429s and 5xx responses retry with exponential backoff. A reset of a minute or more does not keep polling until midnight. snowball profile save writes seeds and knobs only, after a successful dry-run, or with --force. It refuses to store a key. Repeat --doi or --orcid for several seeds (dois = [...] / orcids = [...] in the profile).

Backends resolve in the configured order and emit each work once, keyed by DOI. The default is OpenAlex, then Crossref, Semantic Scholar (with or without a key), ORCID (the person’s own work list), Europe PMC, then an open-PDF bibliography. Reorder backends when a key or proxy makes a later source the better first try. OpenAlex wins when it and a later backend disagree on a field that both filled. A 429 or 5xx pauses that backend and the pass continues with the next one. After the pass, each paused backend is tried once more on only its remaining DOIs. DOIs still blocked stay in deferred.json. Items no paused backend still lists are created, and their PDFs are fetched when fetch_pdfs is fast or full. An open PDF already downloaded for the bibliography is written onto the item. Semantic Scholar responses are cached under state/snowball/cache/. Europe PMC misses and hits are cached under state/snowball/cache/europepmc/. Crossref, Semantic Scholar, Europe PMC, and the open-PDF bibliography ask only for hop-0 works that still have no outgoing references, and only when this crawl is hopping (depth 1 or more, or hybrid). A keyword search at depth 0 stays on the hit list: those backends may fill an empty field on a hit, and they do not import its references. Neighbours and citing works are left alone. A backend that adds references closes the gap, so the next one is not asked. A reference title is the structured article title. The raw citation string is not stored as the title. Before the year filter, a neighbour with a blank or citation-shaped title is filled from the OpenAlex work. If that still leaves no work title, the row is filtered and not created. Europe PMC searches those gaps in batches, and fetches a reference list only when the record has one. A MEDLINE id with no DOI is resolved when Europe PMC has a DOI for it. When a spread of the remaining queue is absent from the index, the rest of that pass is skipped.

backends must include openalex. An unknown name is a config error. Dropping orcid skips the public works list and keeps the OpenAlex author filter.

When OpenAlex lists no referenced_works for a seed, a refs hop does not treat that as “cites nothing”. It asks Semantic Scholar, then Europe PMC, then a publisher landing page (Notes / References HTML), then an open PDF bibliography, and resolves entries back to OpenAlex (DOI match, or a strict title match). The Europe PMC lookup is the same cached search as the fill pass: a list is fetched only when the record has one. A pause on one of those sources does not stop the hop. Cited-by stays OpenAlex-only. Book chapters often deposit zero Crossref / OpenAlex references while still listing footnotes on the public landing; the HTML stage recovers DOI links from that page without needing a PDF.

snowball run --profile NAME prints that profile’s one-line description before any request. mode is search, hybrid, doi, orcid, or collection. DOI profiles store dois = [...]; ORCID profiles store orcids = [...] (legacy singular orcid still loads). Keyword profiles store query = "..." for one term, or queries = [...] for several; query_op = "or" matches any term (default is AND). snowball profile save --query … --hybrid stores mode = "hybrid"; repeat --query and pass --or when saving.

--refine (or refine = true on a profile) asks the configured model for query strings and prints them. If [llm] is off, or the call fails, the crawl still finishes and summary.json records suggestions_error. Suggestions are not seeds.

Advanced

Rate limits, API keys, a CRM, a metasearch, and hosting a snapshot. Skip this for a small dry-run.

Twenty and SearXNG

Neither is on for a normal library fill. Use them when you already keep author pages in a CRM, or when you want a metasearch to find a personal site after ORCID has none.

Twenty

Twenty is an open-source CRM: people, companies, and notes. You can use Twenty Cloud or a workspace you host yourself. The product and the API are in the Twenty docs.

Paperful talks to People in a workspace you already run. It needs [twenty].enabled, [twenty].base_url (or TWENTY_BASE_URL), and env TWENTY_API_KEY. It never sends mail. Knobs: config.

• paperful twenty lookup reads People. --apply writes a proposed author pack and state/author-contacts/ only. It does not change the CRM.

• paperful twenty sync --apply creates a Person for a unique author who is not already a unique match, and enriches a unique match. A blank homepage or email is filled. Extra pages and emails are appended. A primary homepage or email you already set is left alone. Ambiguous names and corporate creators (FAO, and other single-field names) are skipped. New and updated People get the keyword paperful, the collection slug (for example ocean-bbnj), and a note titled Paperful. --limit caps the batch. Fifty or more creates ask on a terminal unless you pass --yes. A resume ledger under state/twenty-sync/ skips fingerprints already written.

• On fetch, author_site runs late (after open access and campus, before Scholar). The listing comes from a promoted pack, then the contact cache, then a capped live People lookup ([twenty].fetch_listing_max, default 20) for items that still have no PDF. That lookup does not create People.

• --twenty-writeback (or [twenty].writeback_listings) appends a personal page found by SearXNG or by a successful author_site fetch onto a unique Person. It does not create one, and it does not replace a primary link.

• [twenty].lookup_on_preflight can add CRM websites to a snowball --author-site-preflight proposed pack. It does not call sync.

reachout --lookup uses the same People search for a contact CSV. It still does not send mail.

SearXNG

SearXNG is a metasearch engine, the maintained fork of SearX. It asks other search engines and returns their hits. Paperful uses it in one place: author-site preflight, after ORCID researcher URLs and any Twenty or cache listing, to find a personal or faculty page. It is not a run source and it is not in sources by default.

Set [searxng].base_url or SEARXNG_BASE_URL to an instance that answers GET /search?format=json. Public instances are listed at searx.space. Many of them leave JSON off, so a host from that list works only when its search API includes json. Otherwise run your own (install) and enable the JSON format in its settings. Paperful does not ship an instance.

A hit is cached under state/snowball/cache/searxng/. With --twenty-writeback, a page found this way can be appended to a unique Twenty Person, as above.

OpenAlex and Semantic Scholar keys

OpenAlex ignores mailto for budget. Snowball calls OpenAlex without an API key first, so a small search needs no account. That free allowance is shared by everyone on the same public address (a campus network or VPN exit included) and is about a tenth of a free key. When it runs out, a configured OPENALEX_API_KEY takes over for the rest of the crawl. With no key, the partial queue is kept and paperful snowball resume continues after you add one. One key only: a second free key or a user+tag@gmail.com alias does not add budget. A free key is about $1/day. Past that, OpenAlex sells pay-as-you-go credit and subscriptions.

A list call that hits the daily budget of the key already in use stops the same way. The reset is midnight UTC, or sooner if you add credit on that key.

Semantic Scholar’s Academic Graph is public, so snowball calls it with no key. That unauthenticated pool is shared and can be throttled. Heavier use needs a private key: request one (it arrives by email; the introductory limit is 1 request per second). Put it in SEMANTIC_SCHOLAR_API_KEY. Like OPENALEX_API_KEY, it stays in the environment, never in config.toml.

OpenAlex parquet snapshot (optional)

If API limits or shared-IP budgets are the bottleneck, download the OpenAlex snapshot yourself — on your computer or a host you control — and configure [openalex_store]. Paperful does not ship the dump. Backends: local_duckdb (this machine) or ssh_duckdb (DuckDB on the data host). DOI / OpenAlex-id batches use the works parquet. Citing (citation-map expand, --direction cites) needs paperful openalex-store build-cites then cites_index_glob (resumable per works updated_date). Live API is first; the snapshot is the fallback when the daily budget is spent. Store cites-query matching is title-only (boolean OR / AND / NOT stay on the API). Fallback lookups that miss timeout_s (default 120s) count as failures; after fail_limit (default 3) the run skips further snapshot calls. Keywords / ORCID / full-text search still hit the live API. Citing with study_designs or listed_in also stays on the live API (older dumps may lack those fields). When the API daily budget is spent, DOI/id and citing keep using the snapshot unless those filters are set; the rest pause for resume.

Setup and TOML: config Advanced. Roadmap for full search parity and campus HTTP hosting: ROADMAP.

Where the crawl comes from

The personal site already does this hop for a people graph, not for Zotero:

Site	Snowball
processing/library/citations.py	One-hop referenced_works and filter=cites:
_data/openalex.yml	[snowball] plus profiles/*.toml
processing/library/s2_citations.py	Optional Semantic Scholar fill, disk cache under state/
processing/library/work_identity.py	DOI, ORCID, and title normalization, aligned with dedupe
No Zotero writes	create_parent and ensure_collection_path

Leave on the site: the people-graph JSON, the hard-coded ego slug, Jekyll outputs, and scholar_citations.py.

What to reuse elsewhere

OpenAlex is the graph (search, author, referenced_works, cites:). The ORCID public API is each person’s own works; those lists are often incomplete, so OpenAlex expands them. Several ORCID iDs in one command merge hop-0 works before a shared hop expand. Semantic Scholar references and citations fill gaps on the public API; a key is only for heavier use. Crossref, already used to verify DOIs, fills metadata gaps. It is not the forward-citation graph.

Adjacent tools, and the piece worth copying:

Tool	Copy	Leave there
findpapers, opencite	direction, depth, per-hop cap, dedupe before emit	The package, and any Scopus / Web of Science / IEEE requirement
paperscraper	—	DOI-list PDFs when you have no Zotero. See comparison
litsearch, lit-review-mcp, CoLRev	Flag ideas	The review project, the report, the screener
Citation Gecko	Overlap rank: neighbours score overlap * 1000 + cited_by_count	The network UI
zotero-snowball, Citegeist	—	In-Zotero one-hop dialogs
pyalex	—	A second HTTP client. Snowball extends the client Paperful already uses for OpenAlex (mailto, sleep, backoff)

ResearchRabbit, Litmaps, Connected Papers, Inciteful, Elicit, Consensus, Scite, and Lens.org stay outside. --refine can suggest further queries behind [llm]; it does not pick seed DOIs. Google Scholar stays the existing PDF lane. It is not a snowball source. Screening a RIS file stays with ASReview. Created items keep clean creators, DOI, and date so Better BibTeX citekeys still make sense. Snowball does not depend on that plugin.

Citation map

Explore (Discover panel) seeds a tiny queue from one DOI or title, auto-expands one hop, and opens Map for ad-hoc hops and Add to library. Seed search on a map node fills the Topic form (DOI kind + seeds) for a full snowball run.

Discover Map draws the current queue before you open PDFs. Node colour is year × cited_by_count:

Pattern	Role
Older + highly cited	foundational
Recent + highly cited	influential
Recent + fewer citations	emerging

Highly cited means at least 50 citations, or the median of the non-seed papers on this map when that median is higher. Recent means the last five years.

Literature categories are a second layer (badges and filter chips), orthogonal to roles. Defaults map OpenAlex study_designs and work type onto:

Category id	Matches
evidence-synthesis	study designs: systematic-review, meta-analysis
trials	study design: randomized-controlled-trial
clinical-trials	study design: clinical-trial (non-RCT interventional; RCTs may also carry this when OpenAlex dual-tags)
observational	study design: observational-study
case-reports	study design: case-report
protocols	study design: study-protocol
narrative-reviews	type: review
preprints	type: preprint, posted-content
books	type: book, book-chapter, edited-book
commentary	type: editorial, letter, paratext

Config [snowball].map_categories chooses which ids appear. [snowball].map_category_defs adds custom rules (any_study_designs and/or any_types). Graph JSON carries category_legend, and each node has study_designs plus categories. Untagged OpenAlex works stay uncategorised.

Click a paper to Keep or Skip it, or to expand one hop (papers it cites, or papers that cite it). Expand merges neighbours into the same dry-run queue. It does not create Zotero items. The drawing code follows the personal-site force layout (zoom, drag, hover focus, side panel); nodes here are works, not people.

Library on the map. Nodes already in the library show a LIB badge and a dark ring. The side panel lists item_key, whether a PDF is on disk, and (when [ui].citation_map_edit is not off) library actions:

Action	Behaviour
Add to library	Creates one parent (uses staged collection/tags/note, else the collection chip)
Find PDF	Runs Grab for that one item_key (Wanted path)
Remove from library	Trashes the item ([ui].citation_map_remove = trash)
Save tags / note / collection	Stages on the queue row; applies live when the paper is already in the library
Open in Zotero / Summary	Deep links when item_key / PDF exist

Defaults: citation_map_edit = both, citation_map_auto_search = false (Find PDF after Add is opt-in), citation_map_remove = trash.

state/snowball/<run-id>/citation_graph.json is paperful.citation_graph.v1, rebuilt from candidates.jsonl. Hops store biblio.graph_links so an edge knows which paper it came from. The map caps at 150 nodes. Keyword and similar neighbours appear on the map without citation edges — only refs/cites hops draw links. Use Repair links on Discover when an older queue is missing graph_links. Optional overlays: read-first triage, a year scrubber, path-between-two-papers inside the current graph, library neighbours (ghost nodes from your mirror, not added to the queue), and watch/compare badges when you open Discover with ?watch_name= or ?compare_run=.

Out of this lane

• A hosted discovery service or corpus-scale OpenAlex browser (Explore is one seed + hops, not the whole public graph)

• A built-in scheduler or cron helper (call watch run yourself), or a snowball step inside paperful all

• A dedicated screening TUI. Discover Keep/Skip and Next hop are the reviewer when a stopping rule is on

• Sci-Hub or Scholar as ways to discover works

• Pulling every publication of every cited author, until an explicit capped switch exists

Related docs

• ROADMAP — phases

• research-pack — refs gap → ingest → run / handoff → dedupe

• comparison — paperscraper, findpapers, in-Zotero plugins

• config — profile precedence this lane reuses

• config Advanced — fetch tuning, Twenty / SearXNG knobs, and OpenAlex snapshot store

• commands — run, which fetch_pdfs calls

• authorwatch — people lists, not a snowball hop