AI for notes and Ask: what is feasible

Paperful’s PDF loop (run / attach / lint / fix-metadata / dedupe) never needs a model. Optional AI covers grounded notes (summarize), a collection review (synthesize), Ask / RAG, title and identity helpers, the browser agent, suggested critique --llm / --advanced comments, and rewrite (apply a critique pack). Heuristic AI-prose tells and the optional Binoculars extra on critique are not authorship proof. Optional --vale is house-style lint, not a model. This page is the feasibility and scale guide: what works on ordinary academic hardware, what is a “try it” experiment, and what a large library actually costs in time or hosted tokens.

Command reference and install steps live in LLM setup. Ask / index details are in Ask your library. Recover model bands are in Browser agent models.

Two paths

Path

When to use

What leaves the machine

Local (Ollama) — default

Your computer can run the tags you need

Nothing (loopback)

Hosted (LiteLLM) — opt-in

You prefer an API, or local VRAM is too small for the job

Metadata + PDF text excerpts you send to the model

Install local with Ollama only. For hosted APIs, install paperful[llm] and set provider = "litellm". Keys stay in the environment, never in config.toml. Commands print a yellow notice when text leaves the machine.

# Local
[llm]
enabled = true
provider = "ollama"
model = "qwen2.5:7b"

# Or hosted (example: OpenRouter — see below)
# provider = "litellm"
# model = "openrouter/openai/gpt-4.1-mini"
# api_base = "https://openrouter.ai/api/v1"

Try it out vs library scale

These are different jobs. A model that feels fine on one paper can be unusable for a 1,500-PDF collection overnight.

Job

Scale

Feasible on typical academic hardware?

Notes

One summary

--item KEY

Yes (local 7B–12B, or any solid hosted chat model)

Acceptance test: four sections, no invented numbers

Small slice

-C … --limit 5 or one year / one type

Yes locally if a 7B tag fits; or cheap hosted

Use --order newest when the model is slow

Whole collection summarize

~10²–10³ PDFs

Local: only with patience or strong silicon; hosted: budget tokens + rate limits

Resume-safe; already-done items skip

synthesize over ~10³ summaries

One review from notes already written

Local: can take hours and many map-reduce chunks; hosted: far fewer tokens than summarizing every PDF again

Run synthesize --dry-run for the chunk plan first

Browser agent (recover)

Per stubborn landing

Needs ~14B+ local or a capable hosted tool-calling model

Not a free-tier hobby lane

Ask / RAG

After rag ingest

Embeddings dominate first ingest; chat is per question

simple (default): one completion. advanced: about evidence_k extra summary calls, cached under state/rag/evidence/. Local nomic-embed-text or hosted embed_provider = "litellm"

critique --lane reviewers

One manuscript

Yes (same 7B–12B band as summarize)

One completion per [[critique.reviewers]] row (--llm alias)

critique --lane marg

One manuscript

Yes locally; slower than reviewers

Three aspect groups (leader + chunk workers + expert) then refine. Caps in [critique]. Optional GROBID for PDF chunks (--advanced alias)

critique --lane adversary

One manuscript

Yes (same band as reviewers)

One JSON validity red-team turn (alt explanations, validity, falsifiers)

critique --lane outline

One outline / plan

Yes (same band as reviewers)

One JSON plan-review turn plus a section skeleton. Not a finished-paper verdict

critique --lane claims

One manuscript

Yes (same band as reviewers)

Claim extract + optional downgrade; support labels are deterministic overlap

critique --lane verify

One manuscript

Yes locally; slower (per-segment)

Density-budgeted section swarm + local synthesis. Not Deep Review

critique --lane context

One manuscript

Yes locally; several extra turns

Summary, library historian, baseline scout, Q&A. Local RAG; context-web may call OpenAlex / SearXNG

rewrite

One critique pack

Style-only: no model. Other areas: yes (same 7B–12B band)

One JSON call per area still lacking a glossary/Vale swap. Optional XtraGPT Ollama tag. Not a generated paper

Try it out means: one item, five papers, or a free hosted model while you watch for timeouts and bad grounding. Library scale means: hundreds or thousands of PDFs, overnight or multi-night runs, resumable batches, and a model you trust enough not to babysit every note.

What a ~1.5k collection actually looks like

Operator dogfood on a large ocean / BBNJ-style collection (thousands of mirror PDFs, on the order of ~1–1.5k summaries in scope for a review):

  • summarize is the expensive step. Default grounding sends up to [summarize].max_context_chars (24,000) of PDF text per paper. On a high-end Apple Silicon machine running a large local Qwen tag, night windows of several hours typically finish on the order of ~100–300 new notes per night, not the whole library in one sitting. Multiple overnight runs (and resume after skips / identity refusals / missing text) are normal. That hardware is not average laptop RAM/VRAM.

  • synthesize is cheaper than re-summarizing, but not instant. It reads existing summary notes only (PDFs are not opened again). Notes are packed under [synthesize].max_context_chars; overflow uses map-reduce. A corpus of ~1,500 summaries can produce dozens of first-level chunks (e.g. level 1 chunk 44/71 in a live run). Each chunk is one completion; then Paperful combines them (up to three reduce passes).

  • Average academic hardware (8–16 GB unified / VRAM, shared with Zotero and browser tabs) is a good fit for try-it and small slices, not for finishing 1.5k grounded summaries in a weekend with a large local model. Prefer a smaller local tag (qwen2.5:7b), a narrower -C / year filter, or a hosted mid-tier model with a spend cap.

paperful summarize skips items already summarized for the current model, so interrupted overnight runs are safe to resume. Changing the model id rewrites the audit trail.

Local models: pick by job, not by hype

Hardware (rough)

Sensible local tags

Realistic use

8 GB

qwen2.5:7b, llama3.1:8b

Try-it summaries; small --limit; Ask after a modest index

12–16 GB

7B–14B instruct

Summaries and synthesize on slices; recover is tight

24 GB+ / large unified memory

14B–32B, dogfood large Qwen tags

Multi-night library summarize; synthesize over large note sets

Browser agent floors and avoid-lists: browser-agent-models.md. Doctor ambers undersized recover tags by name.

Hosted models via LiteLLM

LiteLLM is the in-process adapter (paperful[llm]). Paperful does not require you to run a separate LiteLLM proxy. Point api_base at any OpenAI-compatible endpoint when you need one (OpenRouter, a campus proxy, OpenAI, Anthropic via LiteLLM’s provider ids).

uv sync --extra llm
# or: PAPERFUL_IMAGE_MODE=heavy docker compose build

Keys are provider-specific environment variables (see LiteLLM providers). Examples:

export OPENAI_API_KEY=…           # model = "openai/gpt-4.1-mini"
export ANTHROPIC_API_KEY=…        # model = "anthropic/claude-…"
export OPENROUTER_API_KEY=…       # model = "openrouter/…"

Rules that always apply:

  • Model ids use LiteLLM’s provider/model form. ollama/… is rejected on the LiteLLM path — use provider = "ollama" for local tags.

  • api_base must be http or https when set.

  • Title / abstract / PDF excerpts leave the machine; for recover, page text can too.

Full verb docs: llm.md § LiteLLM.

OpenRouter setup

OpenRouter aggregates many providers behind one OpenAI-compatible API. Useful when you want one key and easy model swaps.

  1. Create an account and an API key at openrouter.ai/keys.

  2. Install the LiteLLM extra (uv sync --extra llm or heavy Docker).

  3. Configure Paperful:

[llm]
enabled = true
provider = "litellm"
model = "openrouter/openai/gpt-4.1-mini"   # pin a concrete id; avoid random routers for batches
api_base = "https://openrouter.ai/api/v1"
timeout_s = 180                            # hosted + long prompts; raise if you see timeouts

export OPENROUTER_API_KEY=sk-or-…
uv run paperful doctor                     # LLM row should be green (import + config)
uv run paperful summarize --item ABCD1234  # one-paper acceptance test

Optional for RAG embeddings on the same key:

[rag]
enabled = true
embed_provider = "litellm"
embed_model = "openrouter/openai/text-embedding-3-small"
# embed_api_base empty → reuses [llm].api_base

Pin a fixed model id for library-scale runs. The openrouter/free router picks a free model per request — fine for curiosity, poor for consistent note quality across a collection. Browse current free and paid models at openrouter.ai/models and the free models collection.

Free OpenRouter lanes

Free :free endpoints can run a try-it summarize --item when they are up and rate limits allow. They are a weak default for library scale:

  • Daily / burst rate limits and empty responses are common.

  • Serving is often heavily quantized; grounding quality varies day to day.

  • A random free router makes provenance and quality inconsistent.

  • recover (tool calling, multi-step) is usually a poor free-tier fit.

Treat free lanes as experiments. For hundreds of papers, prefer a paid mid-tier chat model with a spending limit in the OpenRouter dashboard, or local Ollama.

Cost order of magnitude (hosted)

Paperful cannot quote a fixed dollar price: providers change rates, your PDF text length varies, and retries / identity failures change call counts. Use this order-of-magnitude sketch, then check the provider’s pricing page before a large run.

Rough tokens per grounded summary (defaults):

  • Input: up to ~24k characters of PDF text ≈ ~6k–10k tokens, plus prompt and metadata.

  • Output: under ~600 words ≈ ~0.5k–1.5k tokens.

  • Ballpark: ~8k–12k tokens per successful summary.

For ~1,500 papers (one call each, no retries): on the order of ~12–20 million tokens total.

Hosted tier (illustrative)

Ballpark for ~1.5k summaries

Role

Mid-tier / “mini” chat APIs

Often tens of USD, sometimes low double digits

Practical remote path for library summarize

Strong general / frontier chat

Higher by several× — can reach low hundreds USD

Only if you need that quality bar

OpenRouter :free

$0 listed, but rate limits and flaky uptime

Try-it only

Embeddings for a large rag ingest

Usually much cheaper per token than chat, but first ingest is many passages

Budget separately from summarize

synthesize over ~1.5k notes is typically one to two orders of magnitude cheaper than summarizing those PDFs: you send packed summary HTML in dozens of chunk completions (plus reduce passes), not 1.5k full PDF excerpts. Still check synthesize --dry-run for the chunk count before you pay for a long wall-clock run.

How to estimate for your library:

  1. paperful summarize -C … --limit 5 on a hosted model; note provider usage / cost for those five.

  2. Multiply by the number of not-yet-summarized items (summarize skips current-model footers).

  3. Add headroom for failures, --force regenerations, and identity retries.

  4. Set a spend cap on OpenRouter (or your provider) before an overnight batch.

  5. For synthesize: synthesize --dry-run → chunk count × similar per-call cost.

Pricing pages to re-check (numbers change):

Practical recipes

Laptop try-it (local)

ollama pull qwen2.5:7b
# [llm] enabled, provider = "ollama", model = "qwen2.5:7b"
uv run paperful summarize --item ABCD1234
uv run paperful synthesize -C "My Collection" --year-from 2024   # narrow scope
# Ask (after rag ingest): simple is one completion; advanced scores passages first
uv run paperful ask "What does this collection say about EIAs?" -C "My Collection"
uv run paperful ask "…" --ask-mode advanced

Small hosted batch (OpenRouter mid-tier)

uv sync --extra llm
export OPENROUTER_API_KEY=…
# config: provider = litellm, model + api_base as above
uv run paperful summarize -C "My Collection" --year-from 2024 --limit 20 --order newest

Library-scale summarize (local, multi-night)

  • Keep one model id for the whole campaign.

  • Prefer --order newest or year slices so useful notes land first.

  • Let night / cron windows stop cleanly; re-run to resume skips.

  • Fix could not extract PDF text with OCR before expecting a summary.

  • Do not expect a large Qwen tag on high-end silicon to finish 1.5k papers in one evening — plan for multiple overnight runs.

Library-scale synthesize

uv run paperful synthesize -C "My Collection" --dry-run   # sources + chunk plan
uv run paperful synthesize -C "My Collection"             # map-reduce over notes

Narrow with -T, --year-from / --year-to, or a smaller collection if the chunk plan is huge and wall time matters.

Privacy and grounding (unchanged)

  • Prompts only see item metadata plus text from PDFs already on disk (out/ or state/pdf-cache/).

  • Disk write first; Zotero notes are optional (--to disk|zotero|both).

  • Remote footers are stamped remote LLM so you can tell egress notes apart.

  • Terms: TERMS.

See also

Doc

Role

llm.md

Install, config keys, verb behaviour, troubleshooting

rag.md

Index, Ask, embedding providers

browser-agent-models.md

Recover / Ollama VRAM bands

example-setups.md

Workbench setup 4

config.md

Full key tables

docker.md

light vs heavy (LiteLLM + RAG)