Dedupe

paperful dedupe finds duplicate parent items in a collection (or the whole library) and writes a review pack on disk. It does not change the library unless you pass --apply. --apply copies the extra parent’s PDF, notes, and better fields onto the keeper, then moves that emptied parent to the trash. It does not delete files under out/. Mendeley and EndNote cannot merge, so --apply is refused there.

Monday literature ingest stays outside Paperful. After that ingest, the local hygiene loop is:

uv run paperful dedupe -C BBNJ --dry-run
# read state/dedupe-packs/*.md
uv run paperful dedupe -C BBNJ --apply
uv run paperful gaps -C BBNJ
uv run paperful run -C BBNJ

# optional: same year / item-type filters as run
uv run paperful dedupe -C BBNJ --year-from 2023 --year-to 2026 -T journalArticle
uv run paperful gaps -C BBNJ -T journalArticle,report

-C accepts a path, a unique name, or a collection key (2DBKZRPC). Subcollections are included. --library is the whole library. --year-from / --year-to and --type / -T match Commands — Scope filters.

Phases

Default --phase all classifies both. --apply always does high_doi before medium_title_year.

high_doi. Parents that share a normalised DOI (casefold, strip https://doi.org/). One keep, the rest are merge candidates.

Keep rank, highest first:

  1. stored / imported PDF

  2. linked PDF URL only

  3. richer metadata (a real title, then a date or year, then more creators)

  4. older dateAdded, then item key

If any pair of titles in the group scores below 0.60 after HTML-unescape and punctuation stripping, the group is held (held_divergent_title). Nothing in that group is merged. Review it by hand — the same DOI string is attached to different works.

medium_title_year. Items that were not already in a multi-item DOI group. Same normalised title and the same year. Blank titles, the (untitled) placeholder, and missing years are skipped. These groups are marked needs_review. --apply skips them unless you also pass --apply-medium.

Flags

Flag

Effect

--dry-run

Write the pack only. This is the default. Do not combine with --apply.

--apply

Write a spare-copy line on each extra parent, then merge high_doi extras onto the keeper and trash the emptied parent. If that line cannot be written, the merge does not run. Needs Zotero 10+ (same gate as attach).

--apply-medium

Also merge title+year extras.

--phase

high_doi, medium_title_year, or all.

--limit / -n

Only the first N items in scope.

--year-from / --year-to

Inclusive publication-year range; undated items excluded.

--type / -T

Only these Zotero item types (repeatable or comma-separated).

--json

Pack paths and counts on stdout (dedupe), or the four gap counts (gaps).

The spare-copy line (“Same paper as …”) is written only with --apply, on each extra parent. [remarks].surface chooses a child note, a parent tag, or off. Omitting --apply and passing --dry-run both leave the library untouched.

Pack JSON is paperful.dedupe_pack.v1. Both commands exit 2 if Zotero is unreachable. --apply also needs the Zotero 10+ write API; without it, the pack is still written and the command exits 2. paperful doctor stays amber on the Write API check when trash, attach, and fix-metadata --apply cannot run.

Disk

  • state/dedupe-packs/<timestamp>-<scope>.json — groups, keep, trash, reason, scores

  • state/dedupe-packs/<timestamp>-<scope>.md — the same pack for reading

  • state/dedupe-applied.jsonl — one line per merged parent (keep, drop, moved children, fields filled), only after --apply

Better fields: a blank, (untitled), or a .pdf filename title loses to a real value. A longer abstract or Extra wins. A longer creator list wins only when it already contains the keeper’s surnames. A DOI is copied only into a blank. Other fields fill blanks only. The keeper keeps a real conflict. Identical imported PDFs (same MD5) collapse to one file, preferring the copy that has annotations; both copies stay if both are annotated.

Gaps

paperful gaps -C BBNJ (--library, --year-from / --year-to, --type / -T, --json) counts items with no stored PDF, a linked PDF URL only, or no DOI. It does not fetch or edit. Use run for PDFs and lint for identifiers.

Not this command

Adding new items from a Crossref year query (ingest-dois) is later. So are summary notes, moving items between collections, and Sci-Hub.