Paperful architecture¶
Paperful does five jobs — library, find, completeness,
mirror, control — as a local workbench (GUI on
docker compose up, same verbs on the CLI). Fetch, identifier checks,
proposed metadata patches, summaries, and the per-item mirror happen on
disk (out/, state/). A library adapter reads the catalogue and,
separately, writes PDFs or field patches back. Zotero (local API on
localhost:23119) is the well-tested adapter. manager = "mendeley" talks
to the Mendeley REST API; manager = "endnote" reads a local .enl library
and stages an XML import bundle instead of editing the database. Both are
seeking testers — do not treat them as proven. The disk mirror is what
you keep if the manager changes. See Why Paperful.
Package layout. Flat modules under paperful/ hold one verb or one
concern each (*_cmd.py is the body of a CLI verb after flag parsing; the
module it names does the work). Subpackages: sources/ (fetch lanes),
resolve/ (identifiers), snowball/ (crawl, watches, citation graph),
search/ (SearXNG grey search), network/ (co-author SNA), rag/ (index,
Ask), sections/ (IMRaD extract), critique_checks/ (deterministic
manuscript checks), llm/ (providers), interop/ (RIS / BibTeX / EndNote
XML), sr/ (systematic-review protocol and screening ledger), ui/ (the
workbench), data/ (shipped grey-lit packs). The module-by-module map is in
developer.md.
run never rewrites bibliographic fields. On Zotero, attach,
fix-metadata --apply, and dedupe --apply use the Zotero 10+ write API.
Dedupe merges the extra parent’s PDF, notes, and better fields onto the keeper, then moves that parent to the Zotero trash. It does not delete files under out/.
Mirror first¶
Reference managers are flaky. A local API stalls, item keys change after a
sync, an attachment row outlives its bytes. Paperful’s answer is to copy the
user’s files and metadata into its own mirror (out/, state/) and work
from that copy. The manager API is a narrow door with three uses:
Refresh the mirror. Read catalogue rows, notes, and PDF bytes, and write them under
out/.Write back when asked. Attach a PDF, apply a patch, merge a duplicate, post a note. Always explicit, and the result is on disk first.
Check the door.
ping, write support, authorize.
Anything else is a call Paperful should not be making. Rules for new code:
A verb that only reads takes its items, PDFs, and notes from the mirror. A live listing is for refreshing the mirror, not for doing the work.
A manager that is not running costs write-back. It does not cost the read work.
A write to the manager also updates
record.json, so the mirror does not wait for the nextsnapshotto be true.A failed read is an error. It is never recorded as “no children” or “no PDF”.
Bytes exported from the manager go into the item folder, not a side cache.
pyzotero is called only from
zot.py,library.py, andattach.py. A newLibraryBackendmethod needs a reason the mirror cannot serve.
How the code does it:
Refresh. A command first asks Zotero what changed since the library version in
out/_sync.json, rewrites those items’ folders, marks items that left, and then records the new version (sync.py). Six requests when nothing changed. A first refresh reads the whole library once. A read that fails stops the refresh before the version moves.Read. The command then reads
out/(catalogue.py): collections, items, child notes, attachment rows, PDFs. With Zotero closed it reads the mirror as it is and says how old it is.--offline(orPAPERFUL_OFFLINE=1) never contacts the manager.Write through. A write goes to the manager; that item is then re-read into its folder in the same step (
MirroredBackendinlibrary.py). A trashed or merged parent is marked in its record. Nothing underout/is deleted.
Three verbs talk to the manager directly, because comparing it with the
mirror is their job: sync, snapshot, and restore. They exit 2 when it is
down. So does any --apply, attach, or note write. attachments reads each
item’s children from the mirror (record.json) and proves a stored file by
MD5 against the item folder; it asks the manager only about files the folder
does not hold, and with the manager down it does not call those ghosts.
Mendeley and EndNote implement changes(since) and take the same
mirror-first path as Zotero. They remain seeking testers; Zotero is the
well-tested adapter.
Data flow¶
flowchart LR
adapter[LibraryBackend]
resolve[prepare_identifiers]
sources[Source lanes]
disk[out/ and state/]
writeback[attach / fix-metadata --apply]
adapter -->|read| resolve --> sources --> disk --> writeback --> adapter
Refresh — bring
out/up to the manager’s current library version (skipped offline, or with[mirror].refresh = "manual").Scope — read from the mirror: collection subtree or whole library; optional
--year-from/--year-to(inclusive; undated items dropped) and--type/-T(Zotero item types);runskips items that already have an imported PDF (and by default skip items with only alinked_urlPDF).Prepare identifiers — verify library DOI; optional in-memory swap; PubMed PMID→DOI; title→DOI via Crossref / OpenAlex / Semantic Scholar. Skipped for web/blog/forum types.
Sources — ordered list (Unpaywall, OpenAlex, arXiv, …, CORE, EZProxy, HTML→PDF); per-item routing skips inapplicable sources unless
--try-all.Download — validate PDF magic and
min_pdf_bytes; probe page count and first-page word density (pdfid); write underout_dirinside<Author - Year - Title -- KEY>/; write or refreshrecord.json; extract PDF DOI (pdftotext, thenpypdf); append tostate/manifest.jsonl. Sparse one-pagers soft-reject so later sources can still run; denser one-pagers stayokwithreason=short_pdfuntilattach --allow-short-pdf.Attach — optional write-back through the adapter (Zotero
imported_fileupload; MendeleyPOST /files; EndNote stages a bundle untilflush_writes). Failures recorded asattach_failedwith typed reasons. Batchattachskipsreason=strict_pdf_doiandreason=short_pdfunless the matching allow flag is passed.
paperful lint runs step 2 (and PDF-text DOI) for items with and without PDFs. paperful fix-metadata writes state/metadata-patches.jsonl then, with --apply, pushes patches through the adapter.
Disk artifacts¶
JSON and ledger writes under out/ and state/ go through paperful.fsio
(unique temp file, then os.replace), so a crash or two concurrent writers
never leave a half-written record.json or run report.
Path |
Role |
|---|---|
|
Per-item restore folder |
|
|
|
PDF when |
|
Child-note HTML, including a copied summary when one exists |
|
The reader’s highlights and notes on the item’s attachments, as the manager holds them ( |
|
The library version the mirror was last refreshed to, when, and whether every manager PDF has been copied ( |
|
Folders of items that left the library, when |
|
Lookup rollup: item key, dirs, has_pdf, md5, child keys, and |
|
Collection tree ( |
|
Pointers at |
|
Append-only resume ledger. Latest line per item key wins. Fields include |
|
Proposed patches ( |
|
Duplicate review packs from |
|
Merge audit; appended only on |
|
OpenAlex reference lists for one DOI set, inverted so snowball can say how many items in the target collection cite a new work. Reused until that DOI set changes |
|
One crawl queue: |
|
Watch ledger: |
|
Web grey search pack ( |
|
Grey search watch: |
|
Preprint/published review packs ( |
|
One line per work updated by |
|
|
|
|
|
|
|
|
|
Gated inbox create/attach proposals ( |
|
Throwaway copies of manager PDFs, only when |
|
|
|
|
|
Advanced Ask passage summaries (chunk + question + model cache). Rebuildable |
|
|
|
|
|
GROBID TEI cache ( |
|
|
|
Operator-saved Ask system prompts from the Index form (“Save as…”); listed in the custom-prompt select |
|
Workbench uploads (Discover import, Index prompt/questions files); basename only |
|
|
|
|
|
|
|
|
|
|
|
Proposed / promoted field author packs ( |
|
Twenty People cache ( |
|
Resume ledger for |
|
ResearchGate handoff ledger (you click Request). |
|
Chromium profile + |
|
One line per vault or browser-agent PDF (host and path; query string dropped). Agent rows may include a |
|
Draft learned pack from |
|
Latest |
|
One report per |
|
Parent witness ( |
|
Mendeley tokens after |
|
|
|
Per-verb packs and ledgers; schema ids and writers in developer.md |
|
EndNote XML+PDF bundle for File → Import. Never an edit of |
doi_verified is ok (≥ crossref_min_score), suspect (< doi_suspect_score), swapped (in-memory replacement), unknown (API down, mid-range match, or verify_doi = false), or missing. unknown never swaps.
Library adapter¶
Commands hold a MirrorFirstBackend: the same protocol, with reads served
from out/ and writes passed to the manager adapter. The adapter itself is
below.
paperful/library.py defines LibraryBackend: report what changed since a library version (changes, Zotero only), list items, fetch one item by key (get_item), export a PDF onto disk, apply a field patch, merge a duplicate parent (children and better fields, then trash), trash a note (trash_note — never a parent), attach a file, create-or-update a tagged child note (create_or_update_note, used by summarize), create-or-update a standalone collection note (create_or_update_collection_note, used by synthesize), add an existing item to a collection (add_to_collection, used by collections add), and flush_writes() (EndNote stages state/endnote-import/<stamp>/; others no-op). The tag makes re-runs update instead of duplicate. Identifier and dedupe logic (resolve, lint, pdfid, metadata, dedupe, identity) must not import a manager except through this protocol. paperful/identity.py is the shared DOI then title+year fingerprint used by snowball, authorwatch, refs gap, ingest-dois, and inbox title-match. paperful/refs_gap.py and paperful/ingest_dois.py create review packs / parents through the adapter; they do not fetch PDFs. paperful/collections_add.py files existing keys into a collection (membership only). paperful/inbox_match.py is the drop-folder match ladder. Notes are skipped by items_in_scope, so a report note never enters run / lint / gaps. Canonical item types are Zotero ids; paperful/interop/ maps RIS / BibTeX / EndNote XML at the edge. paperful import / export use that layer. Zotero is well tested. Mendeley and EndNote are seeking testers.
LLM layer (optional, local-first)¶
paperful/llm/ is a thin transport: OllamaClient (httpx to a loopback daemon by default), LiteLLMClient (import-gated behind paperful[llm], keys from env), NullLLMClient when [llm].enabled = false. Verbs import only paperful.llm; validate.py / preflight.py fail before any network call. paperful/grounding.py supplies PDF text disk-first (out/, then state/pdf-cache/ via export_pdf) and a head+headings+tail budget slice.
Verb |
Gate |
Output |
Library write |
|---|---|---|---|
|
|
PDF in |
existing attach |
|
|
|
|
|
|
finding |
none |
|
|
|
child note unless |
|
|
|
standalone note in the scoped collection unless |
|
rules always; optional GROBID sidecar; optional |
|
none (disk only) |
|
|
|
none unless batch |
|
rules always; optional |
|
none |
|
checkers, metrics, heuristic AI-tells always (house glossary expansions when |
|
child note with |
|
apply an existing critique pack (never re-runs critique). Default area |
|
none (never overwrites the source) |
Opt-in camofox is a registered serial source (never in DEFAULT_SOURCES) that talks to an external camofox-browser REST server for a scripted DOI→PDF grab. run injects it when [camofox].enabled and [camofox].during_run (default on); --camofox / --no-camofox override one invocation. Policy order places it after vault lanes and before browser_agent. It does not reuse the vault Chromium SSO profile (see Camofox).
browser_agent is a registered serial source but never in DEFAULT_SOURCES. run auto-appends it when [llm].enabled and [browser_agent].during_run (default on) and the extra is importable. Default [fetch].order = "policy" places Scholar (if opted in) immediately before the agent, interleaved per item so a Scholar hit skips the agent (when camofox is also on, Scholar / camofox / agent run as separate phases so order stays scholar → camofox → agent); [fetch].order = "list" inserts the agent after the last of scholar / ezproxy / htmlpdf as before. The agent runs only if one of those vault lanes was tried and failed. --browser-agent / --no-browser-agent on run or all override during_run for that invocation. Before that phase the pipeline closes BrowserSession so browser-use can own the vault Chromium profile. Opt-in SerpApi ([serpapi].enabled, env SERPAPI_API_KEY) is a later serial Scholar-engine search; [serpapi].max_calls / --serpapi-max cap paid calls (see SerpApi). paperful recover --item (or --from-last-run with a mode such as browser_agent_miss) builds the pipeline with use_browser=False and only that source; batch selection reads state/last-run.json. Optional [browser_agent].fallback_model runs one retry per item before a final miss (skipped after captcha). Hard CAPTCHAs end as captcha, not auto-solved. summarize refuses items the gated identity check flags unless --force.
Identifiers and lint¶
paperful/resolve/ids.py prepare_identifiers is shared by run and lint. Swap is in memory only. PubMed uses NCBI ID Converter on PMID: / PubMed PMID: in Extra.
paperful/lint.py finding codes (manager-agnostic “library DOI”):
Code |
When |
|---|---|
|
Scholarly type, no DOI after prepare |
|
Library DOI fails title check, no swap candidate |
|
High-confidence replacement ≠ library DOI |
|
PMID present, converter failed |
|
PDF-text DOI ≠ library DOI and ≠ prepared DOI |
|
Scholarly title contains HTML tags or entities |
|
Scholarly title is mostly ALL CAPS ( |
|
Scholarly title looks like a filename or path (finding only) |
|
Scholarly title is blank or a citation string, not the work title |
|
No DOI, arXiv id, PMID, or URL |
|
Opt-in LLM says first pages do not match the record (or low confidence) |
--json prints only findings. Exit 0 unless --strict. Lint prefers a file already on disk (item.pdf_path or manifest path) and calls export_pdf only when has_pdf and nothing is on disk.
paperful/metadata.py whitelist: doi, title, date, publicationTitle. Default fills empty venue/date (richest Crossref/OpenAlex date available) and replaces a blank or citation-shaped title from the DOI work. --overwrite may replace title/date/venue when the candidate is at least as precise. Verified pdf_doi_mismatch can propose a DOI (source=pdf). HTML markup in titles is stripped into a title patch; ALL CAPS titles are recased to Title Case (source=title_case); tokens in state/acronyms/ stay uppercase; filename titles stay lint-only. The same ALL CAPS → Title Case pass runs when creating parents (parent_payload / snowball ingest) and when a DOI work title is written by fix-metadata, so new items do not land ALL CAPS in Zotero. Never invents creators. state/metadata-patches.jsonl is an append-only audit log (one patch per item key per invocation); not a curated re-apply queue.
PDF text¶
paperful/pdfid.py: pdftotext (Poppler) if on PATH, else pypdf (first two pages + /Title; max_pages=None reads the whole file for summarize). Also page_count, first-page word_count, and short_pdf_verdict for the one-page density gate on run (and sparse reject on handoff ingest). Manager fulltext is last-resort: export the file to state/pdf-cache/ first. paperful doctor reports amber if pdftotext is missing.
paperful/ocr.py: paperful ocr (dry-run unless --apply) runs ocrmypdf on image PDFs and replaces the file under out/. A manager-only PDF is exported into the item folder first; state/pdf-cache/ is never the file that gets the layer, because the next export would overwrite it. --attach uploads that file as a new attachment and leaves the scan in place. doctor is amber when ocrmypdf is missing. The step is optional on paperful all (--steps), not in the default chain.
Circuit breaker¶
Open-access sources run in parallel (concurrency_oa). Captcha and block pages increment a per-source counter; after circuit_breaker_threshold the source pauses, then one later item is probed. A 429 does not open the circuit. Scholar, Sci-Hub, EZProxy, and HTML→PDF stay serial (they share one Chromium profile lock). run probes EZProxy before batch 1 when configured. An expired session skips further proxy wraps; on a TTY, run offers re-login at the next batch boundary and again after the fetch (ezproxy_relogin, default on) to retry session-expired items. Publisher PDF URLs that 403 on httpx are retried in that profile (EZProxy-wrapped when configured). Direct PDF URLs are tried before landing pages.
Sci-Hub and presets¶
Sci-Hub is never in the default source list; opt in via config, --scihub, or --sources. --preset oa drops EZProxy. --preset eoi is open access plus campus EZProxy (no Scholar, no Sci-Hub), which matches the default list today. CORE is in the default list but skipped until core_api_key is set.
Disk mirror vs cloud quota¶
When Zotero cloud storage is full, attachments may fail with quota errors; PDFs still land on disk and can be attached later. Linked PDF URLs in Zotero are treated as “already covered” unless --upgrade-linked is set.
Quiet mirror: out/<collection>/<stem -- KEY>/ is a browsable restore
folder (dual store with Zotero storage/ after imported_file attach).
sync keeps a folder for every item. [mirror].pdfs chooses whether
existing Zotero PDFs are copied in (all, the default), copied the first
time a command needs one (lazy), or kept out (none). paperful restore --apply
creates missing items from those folders and does not overwrite fields that
are already in Zotero. Stance: quiet-mirror.md. House
folder sync is out of scope for this CLI.
Ghost attachments¶
Zotero can show The attached file could not be found for a path under the data directory’s storage/<key>/. The attachment record is there (MD5 and storage folder) but the bytes never landed on your computer. That is a ghost, not a file moved or deleted outside Zotero. paperful attachments reports it. --fix-broken --apply refills the row only when out/ already has a PDF with the same MD5.
Attachments created through the API as imported_url open that storage slot without always finishing a local download. linked_url attachments (including a quota-full open-access pass) do not use that path: they open in the browser and do not raise this dialog. Paperful’s attach path is imported_file: the PDF is already on disk, then uploaded through the local write API. The Zotero attachment note is a provenance stamp (paperful oa:unpaywall, campus:ezproxy, grey:<playbook>, pirate:scihub, …). Title stays Full Text PDF. The parent also gets a readable line (“Free copy from Unpaywall.”) unless [remarks].surface is off. Prefer paperful run / paperful attach for gap-fills so the file is written on your computer.
paperful/attach.py subclasses pyzotero._upload.Zupload so filename spaces are sent as %20. That module is private. The dependency is pinned to pyzotero>=1.15.1,<1.16. It is not vendored.
A refill from the attachment’s open-access URL is only good when the downloaded bytes match the stored MD5. Publishers that return 403 to a scripted download (Cambridge, Taylor & Francis, some institutional hosts, parliamentary briefings) will not refill that way. Open those in a browser, or with Paperful and EZProxy, and drop the PDF onto the parent item — or trash the empty attachment and re-attach.
In Zotero 10 the settings pane is Account (older builds still say Sync). Turn file sync on for this data directory, or right-click the attachment → Download File. Setup, write keys, and link modes: Zotero.
Operator tooling¶
paperful config check— static pass overconfig.toml: unknown keys and sections, secrets that belong in the environment. Exit 1 when anything is flagged. Run it beforedoctorwhen a setting seems ignored.paperful sr init/screen/status/export/import— systematic review understate/sr/<slug>/(protocol, append-only screening ledger, PRISMA counts and SVG). Exclude reasons come from the protocol list. Advanced/reviewrecords decisions. No scheduler and no recall percentage. See sr-mode.paperful doctor— preflight. Colours: green = ready; amber = usable with a degraded path (empty email, missing session, expired session when--probe, nopdftotext, missingvale(optional critique house style), rewrite areas other than style when[llm]is off, Playwright / Chromium not ready, Zotero 7–9 write API, incomplete grey-lit pack); red onZotero :23119/out_dir/state_diris fatal (doctorand any command that needs Zotero). Reports grey-lit packs (UNGA/undocs · BBNJ/DOALOS · ISA) when builtin is on. On a TTY, walks amber/red remediations (--guide/--no-guide).--probehits Scholar / EZProxysession_ok(network). See commands.paperful run --dry-run— no downloads. Per item: Would-hit is the routed source list in order (fullsourceswhen--try-all).paperful lint/paperful fix-metadata— identifier hygiene; apply is explicit.paperful critique— local manuscript pack (state/critiques/). Checkers, metrics, and heuristic AI-tells without an LLM; optional suggested comments (--llm/--advanced), Binoculars extra, and Vale house-style pass. Not a grade and not inall.paperful rewrite— apply a critique pack (state/rewrites/). Default area is style. Never overwrites the source. Not a generated paper and not inall.paperful dedupe/paperful gaps/paperful reachout/paperful refs gap/paperful coverage— duplicate packs, PDF/DOI counts, contact-only missing-PDF CSV (never fetches, never sends mail), cited-in-PDF missing-from-library packs, and briefing/note/file vs-Ccoverage.refs gapandcoverageare always dry-run (state/refs-gaps/,state/coverage/).ingest-doiscreates parents from those packs or a DOI file only with--apply.dedupewritesstate/dedupe-packs/and merges only with--apply(the spare-copy line is written then; title+year also needs--apply-medium).versionswritesstate/version-packs/and updates a preprint only with--apply. See dedupe.paperful report/paperful report --last-run— manifest totals plus the latest auditable run report (state/last-run.json, history understate/runs/). Eachrunprints a one-line banner (downloaded N · attached M · deferred K · not_found J · write-api yes|no) and then a Run summary table.deferredis manifest skips plus linked-URL skips.write-apiisunknownwhen the library was not probed.
When Zotero is unreachable and there is a mirror, read commands carry on
from it (attachments report-only included). sync, snapshot, restore,
attach, and any --apply exit 2 and print next steps (start Zotero, enable local API,
paperful doctor). With no mirror yet, every library command exits 2.
Report JSON (paperful.run_report.v1)¶
paperful report --json is { counts, by_source, no_identifier, no_doi, attach_failed_by_code, last_run? }. last_run (when present) is the same object
as state/last-run.json. The required key set below is frozen: a removed or
renamed required key is a break. Extra keys may still be added. Required
paperful.item.v1 keys are frozen the same way. Summary miss rollups
(retryable, browser_misses, not_downloaded, paywall_prices,
agent_after_playwright) are part of the required summary key set. Extra keys
may still be added.
Field |
Meaning |
|---|---|
|
Always |
|
|
|
ISO-8601 UTC |
|
Wall time, or |
|
Collection path(s) or library |
|
Source names for that run |
|
CLI flags ( |
|
Absolute paths |
|
Successful downloads ( |
|
Write-back counts |
|
Item outcomes (required on every |
|
Browser-lane miss labels rolled up ( |
|
Items with no PDF, one reason each. A page block (captcha, cloudflare, blocked, paywall, login) wins over a plain miss. Other reasons: session expired, paused, download failed, step budget, no identifier, error, not found |
|
Publisher prices the browser agent noted on items that were not saved, summed per currency ( |
|
Count of items where |
|
Not attempted this run |
|
In-memory DOI enrichments (not library writes) |
|
|
|
Hits per source name |
|
Per-source outcome tallies |
|
Typed errors |
|
|
|
Per-item required keys: |
|
|
--format json on run, refs gap, coverage, ingest-dois, collections add, inbox drain, gaps, lint,
fix-metadata, dedupe, snowball crawl (search / hybrid / doi / orcid /
collection) plus run / resume / apply, summarize, synthesize, restore,
import, recover, ocr, all, ask, inbox proposals list|apply|reject, and notes delete
prints one paperful.agent.json.v1 object on stdout (schema, command, exit, ok, partial,
summary, items, paths, flags, optional nested report). Progress and logs go to stderr.
Exit 3 is a mixed write batch (run uses attached vs attach_failed). Legacy --json on lint / gaps / dedupe keeps its old shape
unless --format json is also passed. paperful mcp is optional stdio sugar for the same
payloads on read-only tools: gaps, snowball_search (dry-run), snowball_trends
(publication-year counts), export, proposal_export, refs_gap (never creates
parents), and ask (index only; optional ask_mode). Agent and MCP export
build records in export_build.py (notes omitted, no PDF
copy). Proposal packs go through pack_bib.py. The CLI
export command overlays fields in export_cmd.py, including the EndNote XML bundle
and --pdfs. Prefer … --format json
from a shell. paperful serve (paperful[serve]) binds localhost HTTP: JSON
capability routes (health, doctor, collections, last-run, dry-run refs-gap /
ask) plus the server-rendered workbench (paperful.ui — Wanted, Discover,
Ask, Library, Activity, System; Advanced Repair / Mirror / Index / Briefs / Critique /
Suggest edits / Stuff / Settings). An origin guard (ui/origin_guard.py) refuses
cross-origin non-GET requests with 403 before any route runs; there is still no login.
mount_ui registers HTML/form routes from *_routes.py modules; shared helpers stay in app.py.
GUI long jobs enqueue under state/gui/commands/; Activity and post-submit pages
subscribe to GET /v1/runs/{id}/events (SSE) with GET /v1/runs/{id} as JSON
poll fallback. GUI writes use review tokens (state/gui/reviews/) then the same
domain entrypoints as the CLI. Grab fetches to out/. Attach writes the catalogue
when you press it (attachable pending only on the main Wanted button; mismatch
and short PDF need Advanced), or during Grab when [ui].attach_mode is auto
and the library can write. Apply writes the catalogue too. See gui.md.
collections add is CLI-only (--apply writes) and is not an MCP tool.
TTY-only paths (a GUI must not claim them): session login, doctor --guide,
mid-run EZProxy re-login, and snowball approve-each.
Child notes from summarize / synthesize / remarks / snowball / briefing / critique --apply start
with a scannable line and a <!-- paperful.note.v1 {…} --> comment (type,
verb, model, run_id, prompt_sha).
Ask follow-ups with --thread (or a TTY prompt loop) live in
state/rag/threads/<id>.json (paperful.rag.thread.v1). Retrieval uses a
rewritten standalone query; the chat model still sees the turns. Run reports
record the mode actually used (simple or advanced). In advanced, a
follow-up may cite only this turn’s excerpts.
Manifest counts keys match ledger statuses (ok, attached, not_found, …).
gaps, lint, screen, critique, rewrite, summarize, and fix-metadata (dry-run and --apply) write the same schema under state/runs/ and do not replace last-run.json. Their summary adds command-specific keys (no_stored_pdf, findings / findings_by_code, edits / skips, summarized / failed, patches_proposed). A dry-run fix-metadata report omits patches_applied.
Run packs (paperful.pack.v1)¶
paperful pack open writes state/packs/<id>.json and state/packs/current. Each later command that writes a run report appends a step {command, started_at, finished_at, report} — report is the child filename under state/runs/. When the report has a witness.config_sha256, the step also gets witness_id (first 12 hex chars). The first step that has a scope copies it onto the parent. paperful pack close sets status to closed and deletes current. A second open while one is open exits 1.
PAPERFUL_PACK=off writes the child report and does not append. Commands that exit before a report (bad flags, Zotero down, run --dry-run) are absent. paperful pack show reads disk only: the open pack, or the latest closed one. --json inlines each step’s summary, not the child items array.
Run configs¶
A profile is the input you can run again (paperful all --profile, or
--profile on one verb). A pack is the output of one sequence. Grey-lit
playbooks are URL → PDF rules. None of the three replaces the others.
Profiles are [profiles.*] in config.toml and profiles/*.toml beside
that file. They are not stored under state/. paperful all opens a pack
when none is open, runs the default chain (or the profile’s steps), and
closes the pack it opened. See Workflows.
Grey literature¶
direct and the vault browser share a playbook engine
(paperful/playbooks.py): declarative rewrite /
scrape / synthesize rules from config. The vault fetch
(paperful/session.py) applies those rewrites and
HTML PDF links before a fixed click list.
paperful/fetch_wins.py appends successes and
can write learned.toml under grey_playbooks_dir. That file is a normal
dir pack (names start with learned-); it is not shipped in the wheel.
Knobs and the propose/promote commands: Sessions and
Configuration. A builtin
ocean/governance example pack (paperful/data/grey_playbooks_ocean.toml)
ships the named grey-lit packs below, plus FAO, RFMO, IUCN/DOSI, and IDDRI/Pew.
IEA/IRENA and OECD/WHO/UNEP/UNDP are optional files in
paperful/data/grey_playbooks_examples/ (load them with grey_playbooks_dir).
The pack is not core product logic; set grey_playbooks_builtin = false or
override by name. A matching host is a URL→PDF recipe for an item already in
the library. Provenance stays grey:<playbook-name>.
paperful search / Discover → Web finds candidate URLs (SearXNG JSON);
playbooks still fetch PDFs after you apply metadata and run Wanted.
Skip-host item URLs (YouTube, Scholar, …) still allow Extra/title synthesize.
Domain-agnostic OA rewrites (PMC, arXiv, HAL) and DSpace/OAI stay in code.
DOI-less report / document items can fall through to htmlpdf. Campus
EZProxy is never used for these public hosts. Unpaywall/OpenAlex already skip
DOI-less items (no quota burn on institutional reports).
Named packs (BBNJ product)¶
Pack id |
Hosts / patterns |
Rules |
|---|---|---|
|
|
|
|
Extra/title symbols |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Optional example packs (not loaded unless grey_playbooks_dir includes them):
Pack id |
Hosts |
File |
|---|---|---|
|
|
|
|
|
same energy file |
|
|
|
|
|
same intl-orgs file |
|
|
same intl-orgs file |
|
|
same intl-orgs file |
un.org belongs to bbnj-doalos-prepcom. The intl-orgs file does not list it.
Smoke collections (dry-run): HKF7T7EI (UNGA/VME), 7R77ZJFH / XFD86ZFP (BBNJ/PrepCom), J2SEXDC5 (ISA).
Items with no DOI, arXiv id, PMID, URL, or matching synthesize playbook still
stop at no_identifier.