Configuration (config.toml)

Copy config.minimal.toml for a first run (email, paths, open-access sources) or config.example.toml for the full file. Both are tracked; config.toml is not.

Docker: prefer relative out_dir / state_dir ("out" / "state"). Paths with ~/… expand to the container user’s home (not the Compose /data mount), so host session login and docker compose run will disagree about where cookies live. paperful doctor ambers when that happens.

Grey literature and no-DOI items — Unpaywall and most DOI sources cannot resolve PrepCom papers, many DOALOS/UN docs, or undocs without a DOI. direct uses declarative grey playbooks (rewrite / scrape / synthesize). Grey-lit packs: UNGA/undocs · BBNJ/DOALOS · ISA (plus FAO/OECD/IEA/WHO examples). Add your own hosts in config.toml. Skip-host URLs (YouTube, Scholar, …) still synthesize from Extra/title when a playbook matches. Then htmlpdf can print DOI-less document / report pages. Otherwise the manifest records no_identifier. See Grey literature playbooks and architecture.

Key	Default	Meaning
email	""	Contact address for Unpaywall, Crossref, and NCBI. OpenAlex ignores mailto; its daily budget follows OPENALEX_API_KEY, or the client IP when that is unset
manager	zotero	Library adapter. zotero is well tested — use that. mendeley and endnote are in the tree and seeking testers. EndNote writes are an import bundle, not an edit of the .enl file
[mendeley].client_id / client_secret	""	Elsevier OAuth app from dev.mendeley.com/myapps.html. Or PAPERFUL_MENDELEY_CLIENT_*. See Mendeley
[mendeley].redirect_uri	http://127.0.0.1:8765/callback	Must match the app. Host-only (session login mendeley)
[endnote].library	(none)	Path to the .enl file. Matching .Data (with sdb/sdb.eni) must sit beside it. See EndNote
out_dir / state_dir	out / state	PDF tree; manifest, patches, PDF cache, run reports, and write key
[mirror].pdfs	additional	snapshot PDF policy: additional (fetched files only), all (also export Zotero PDFs), none (records and notes only). run always writes PDFs it downloads
[remarks].surface	note	Where the readable lines go: where a PDF came from, which duplicate to keep, and why a snowball hit belongs. note (child note), tag (parent tag), or off. The PDF attachment stamp stays the machine token
sources	unpaywall → openalex → arxiv → biorxiv → europepmc → semanticscholar → core → direct → ezproxy → htmlpdf	Source order; --sources overrides per run. scholar and scihub are not included unless you opt in. core is skipped until core_api_key is set
verify_doi	true	Check library DOIs against Crossref/OpenAlex before fetching; may swap DOI in memory for that run. false leaves an existing DOI as doi_verified=unknown and does not swap
doi_suspect_score	0.70	Title similarity below this marks a library DOI as suspect (eligible for in-memory swap). API failure is unknown and keeps the original DOI
core_api_key	""	CORE API bearer token; empty skips the core source
ezproxy_base	"" (disabled)	Campus proxy prefix ending in url= — see Campus EZProxy
ezproxy_cookies	state/ezproxy-cookies.txt	Compat Netscape dump after session login ezproxy
scholar_cookies	state/scholar-cookies.txt	Compat Netscape dump after session login scholar
grey_playbooks_builtin	true	Load the packaged ocean/governance example pack
grey_playbooks_dir	(none)	Directory of extra pack *.toml files (merged after builtin, before inline). Relative paths resolve against the config file’s folder
[[grey_playbooks]]	(none)	User rewrite/scrape/synthesize rules; same name overrides the pack
scihub_mirrors	built-in list	Hostnames tried in order
delay_scihub_s	[3, 8]	Random pause (seconds) before each Sci-Hub / EZProxy / htmlpdf page fetch
concurrency_oa	4	Parallel workers for open-access sources (Scholar, EZProxy, htmlpdf, and Sci-Hub are serial)
min_pdf_bytes	10000	Smaller downloads are rejected as error pages
crossref_min_score	0.90	Title-similarity threshold for accepting a title→DOI match (Crossref, then OpenAlex, then Semantic Scholar)
mirror_failures_before_skip	3	Network failures before a Sci-Hub mirror is skipped for the run
source_routing	true	Skip sources that look inapplicable from item metadata; use --try-all to override per run
circuit_breaker_threshold	3	Block-like failures (CAPTCHA, rate limits) before a source is skipped for the rest of the run
attach	true	Attach into Zotero after download (--no-attach overrides)
app_name	paperful	Name shown in Zotero’s authorisation dialog
user_agent	Chrome-like string	HTTP User-Agent for source and download requests

Leave ezproxy_base empty (or remove ezproxy from sources) if you do not use a library proxy. Google Scholar is off until you add scholar to sources (and usually run session login scholar). Sci-Hub is off until you add "scihub" to sources or pass --scihub — see Sci-Hub. Items dated after 2021 are not sent to Sci-Hub; a --year-from past that year drops it from the run list. Set source_routing = false (or pass --try-all) when Zotero fields are untrustworthy and you want every configured source tried anyway.

Run configs (profiles)

A profile stores one SCOPE plus the fetch/write flags you would otherwise repeat on the command line. It is not a grey-lit playbook and not a pack. Full recipes: Workflows.

Term	Meaning
Profile / run config	Named SCOPE + policy (collections, years, types, try_all, …)
paperful all	gaps → run → lint → fix-metadata → summarize
--preset oa	Open-access sources only (the default list without ezproxy). CLI flag or preset on a profile. config.minimal.toml is this list
--preset eoi	OA + EZProxy; no Scholar, no Sci-Hub. Same names as the default sources list today
Playbook	Grey-lit URL → PDF rule
Pack	Witness of one executed sequence under state/packs/
Snowball profile	kind = "snowball". run and all refuse it. Use paperful snowball run --profile or paperful snowball profile save

Where files live. Beside config.toml:

• [profiles.<name>] tables in that file

• profiles/<name>.toml in the same directory (overlay: file wins per key)

Relative profiles/ follows the config file, including Compose /data. Profiles are not written under state/ (that tree is backup-excluded).

Precedence for paperful all and paperful profile show:

1. Builtin all policy when the profile omits a key: steps gaps, run, lint, fix-metadata, summarize; try_all, retry_failed, upgrade_linked, and apply true

2. [profiles.*]

3. profiles/<name>.toml

4. --profile or -f / --run-config

5. Explicit CLI flags

A bare paperful run --profile does not inherit the builtin all flags. Omitted try_all stays off unless the profile sets it.

TOML key	CLI
collections	--collection / -C (repeatable; replaces the list)
library	--library / --no-library
types	--type / -T
year_from / year_to	--year-from / --year-to
limit	--limit / -n
try_all	--try-all / --no-try-all
retry_failed	--retry-failed / --no-retry-failed
upgrade_linked	--upgrade-linked / --no-upgrade-linked
no_attach	--no-attach / --attach
scihub	--scihub / --no-scihub
preset	--preset (eoi)
sources	--sources (comma-separated; TOML is an array of strings)
apply	--apply / --no-apply on all, fix-metadata, and summarize
overwrite	--overwrite / --no-overwrite on fix-metadata
steps	--steps (comma-separated). Optional extras: snapshot, dedupe, synthesize, ocr
require_summarize	--require-summarize — exit 1 if summarize cannot run because the LLM is off
description	profile save --description

profile save NAME writes profiles/NAME.toml from the flags on that invocation (plus --profile / -f if you are copying one). It refuses to overwrite unless --force. It does not edit config.toml.

[profiles.bbnj-journal]
description = "BBNJ journal articles 2021–2026"
collections = ["BBNJ"]
types = ["journalArticle"]
year_from = 2021
year_to = 2026
try_all = true
retry_failed = true
upgrade_linked = true
apply = true

Grey literature playbooks

Host-specific PDF rules are data, not forever-hardcoded Python. Kinds:

Kind	When	Example
rewrite	Zero-fetch URL → PDF (url_re + pdf_template, or parser = "undocs")	FAO /3/{code}/
scrape	Prefer matching hrefs on that host’s HTML landing	OECD /download/, WHO /iris/
synthesize	Extra/title (or skip-host URL + Extra) → PDF URL	UN document symbol → undocs

The packaged file paperful/data/grey_playbooks_ocean.toml is an ocean/governance example pack — grey-lit packs UNGA/undocs · BBNJ/DOALOS · ISA, plus FAO/OECD/IEA/WHO examples — on by default via grey_playbooks_builtin = true. Optionally set grey_playbooks_dir = "packs" to load every *.toml in that directory (same schema). Merge order: builtin → dir packs → inline [[grey_playbooks]] (same name replaces earlier entries). PMC / arXiv / HAL stay as core OA rewrites, not playbooks. See architecture § Grey literature for hosts and symbol patterns. If you use the optional Docker image, put packs next to config under /data (see Docker).

grey_playbooks_builtin = true
grey_playbooks_dir = "packs"

[[grey_playbooks]]
name = "my_org"
kind = "rewrite"
hosts = ["example.org"]
url_re = '(?i)example\\.org/docs/(?P<code>[a-z0-9]+)/?'
pdf_template = "https://example.org/docs/{code}/{code}.pdf"

LLM (optional, local-first)

Off by default. Install extras: uv sync --extra llm (LiteLLM for paid APIs), uv sync --extra browser-agent (Python 3.11+ only, for the browser_agent run lane and recover). Setup walkthrough, model advice, Docker networking, and troubleshooting: LLM.

Table / key	Default	Role
[llm].enabled	false	Master gate
[llm].provider	ollama	ollama or litellm
[llm].model	qwen2.5:7b	Model id
[llm].base_url	http://127.0.0.1:11434	Ollama API
[llm].api_base	""	OpenAI-compatible base when provider = litellm
[llm].allow_remote	false	Allow non-loopback Ollama (e.g. host.docker.internal from the image)
[llm].timeout_s	120	Per-completion timeout
[llm].max_num_ctx	32768	Cap on the Ollama context window for summarize and synthesize. The model tag must support it
[fix_metadata].llm_title	false	Grounded title proposals in fix-metadata
[lint].llm_pdf_match	false	pdf_identity_mismatch finding; summarize refuses flagged items unless --force
[lint].llm_pdf_match_min_confidence	0.6	A match: true below this confidence is still flagged
[summarize].prompt_template	default	Or path to a custom prompt file (relative to the config file); its SHA is stamped in the note footer
[summarize].max_context_chars	24000	Budget for PDF text sent to the model (head + headings + tail)
[summarize].tag	paperful-summary	Zotero child-note tag; re-runs update the note carrying it
[summarize].dest	both	disk (state/summaries/), zotero (child note), or both. --to overrides
[synthesize].prompt_template	default	Report prompt, or a path relative to the config file
[synthesize].max_context_chars	24000	Budget for summary text in one model call (room left for the prompt)
[synthesize].tag	paperful-report	Base tag on the collection note. A second tag tag:<slug> makes re-runs update
[synthesize].dest	both	disk (state/reports/), zotero (standalone note in the collection), or both
[synthesize].timeout_s	max([llm].timeout_s, 300)	Per-completion timeout for the report
[browser_agent].max_steps / max_wall_s	20 / 300	Step and wall-clock caps for recover (agent stops early once a valid PDF lands)
[ocr].languages	eng	Tesseract languages for paperful ocr (eng+fra or eng fra)
[ocr].timeout_s	600	Seconds allowed per PDF
[attachments].fix_broken	false	With attachments --apply, refill a ghost or broken link from out/ when the MD5 matches
[attachments].merge_files	false	With --apply, trash extra PDF children on the same parent that share an MD5
[attachments].rename	false	With --apply, rename files under out/ to the mirror stem
[attachments].link	false	With --apply, stored-to-linked under out/ (personal Zotero library only)
[browser_agent].during_run	true	When [llm].enabled and the extra is installed, run appends browser_agent after Scholar / EZProxy / htmlpdf
[browser_agent].model	([llm].model)	Larger model for browsing only; doctor warns under ~10B

API keys stay in the environment (never in config.toml).