Building a Local AI Assistant That Actually Knows Your Research

Back to Blog

Picture a theoretical physicist eighteen months from submitting. She has perhaps four hundred papers she's actually read, a decade of her own notes and derivations, and a thesis argument that depends on results published in the last two years. She asks a local model a question inside her subfield and gets back something fluent, confident, and subtly wrong — plausible notation, a citation that doesn't exist, an approximation used outside its regime. Her instinct, and everyone's, is that the model needs to be trained on her material. That instinct is the single most expensive mistake in this whole area, and it's worth understanding exactly why before spending a weekend on it.

The Four Levers, and What Each Actually Does

"Make the AI know my work" collapses four very different operations into one phrase. Separating them is most of the battle:

• Context / system prompt. You paste material in, or set standing instructions. Free, instant, no infrastructure. Limited by the context window and by you remembering to do it.
• Retrieval (RAG). The system searches your documents at question time and puts the relevant passages in front of the model. Facts stay in files you control. Add a paper and it's usable in seconds; delete one and it's gone.
• Fine-tuning (LoRA / QLoRA). You adjust the model's weights on examples. Teaches behavior — output format, notation conventions, house style, domain register. Does not reliably install facts.
• Full training. Millions of dollars and a cluster. Not a thing an individual does, and not what anyone actually wants.

For a frontier researcher, the answer is overwhelmingly the second one — with a possible thin layer of the third much later. The reason is worth stating precisely.

Why "Just Fine-Tune It on My Papers" Backfires

The clearest way to put it, and the framing the 2026 practitioner literature has converged on: fine-tuning is not a database write. When you fine-tune on a corpus of facts, you are not storing those facts. You are teaching the model to sound like something that knows them — and it will reproduce the tone, structure, and confidence of your material while inventing the specifics [1].

For our physicist this is actively dangerous. A model fine-tuned on four hundred papers in her subfield becomes more fluent in the notation and register of that subfield, which makes its fabrications harder to catch, not easier. She has traded obvious wrongness for plausible wrongness. Benchmark work on commercial fine-tuning APIs has found the same thing from the other direction: infusing genuinely new knowledge through fine-tuning is unreliable even when done well [2].

The sequence that current practice recommends is prompt → RAG → fine-tune → distill, in that order, stopping as soon as it's good enough [1]. Fine-tuning earns its place for narrow behavioral jobs: forcing a rigid output schema, matching a group's notation conventions, adopting the register of a specific journal. Even then the highest-value form is a thin LoRA or QLoRA adapter on a strong base model, paired with retrieval rather than replacing it. Most people never need it.

The Corpus Is the Product

Everything downstream is determined by what you put in and how carefully. The temptation is to index everything; resist it. A tight, curated corpus of material you have actually read and trust outperforms a vast one, because retrieval quality degrades when near-duplicates and low-quality sources compete for the same top-k slots.

For the thesis case, three tiers work well: papers she's read and annotated; her own notes, derivations and drafts (usually the highest-value and most overlooked material); and a small reference layer of textbooks and review articles for definitions. What stays out: the entire arXiv listing for her field, anything she hasn't evaluated, and — critically — anything under embargo or covered by a collaboration agreement she'd rather not have sitting in a vector database.

The Step Everyone Skips: Getting the PDF In

This is where most scientific RAG projects quietly fail, and it's the section a physicist should read twice. A PDF is a description of ink on a page, not a document structure. Extract naively and a displayed equation becomes garbage characters or vanishes entirely, and a two-column layout interleaves into nonsense.

Tool choice matters enormously here. On a controlled equation-extraction benchmark, Marker was the only open-source parser that consistently produced clean LaTeX, scoring about 10.5 out of 12, while Docling scored roughly 5 out of 12 and dropped display equations as placeholders [3]. Nougat produced good output when it worked but is effectively unmaintained, and GROBID converts formulas and tables to images — fine for bibliographic metadata, useless if you need the math as text.

One nuance that matters for equations specifically: preserve them as LaTeX in the extracted text rather than as images. An equation kept as `\frac{\partial \psi}{\partial t}` is searchable, quotable, and can be reasoned about. The same equation as a PNG is invisible to your entire pipeline.

Chunking: The 2026 Numbers Are Surprising

Chunking — cutting documents into retrievable pieces — is routinely treated as trivial preprocessing, and recent systematic evaluation suggests that's a mistake with real consequences [4]. A few findings worth knowing before you tune anything:

• Simple beats clever, usually. A February 2026 benchmark across 50 academic papers put recursive 512-token splitting first at 69% accuracy, with semantic chunking at 54% — despite semantic chunking being roughly 14× slower to compute [5].
• Overlap may be doing nothing. Systematic analysis found overlap provided no measurable benefit while increasing indexing cost — contradicting the near-universal default advice [4].
• There's a context cliff. Retrieval quality degrades beyond roughly 2.5k tokens of context. Stuffing more in is not free [4].
• But structure-aware chunking wins where structure exists. In a clinical decision-support study, chunking aligned to logical topic boundaries reached 87% accuracy against 13% for fixed-size baselines [4].

Reconciling those: start with recursive 512-token splitting because it's cheap and strong, then graduate to structure-aware chunking only where your documents have genuine structure to exploit — which scientific papers emphatically do. Respect section boundaries. Never split a derivation across chunks. Keep an equation with the sentences that define its terms, and keep captions attached to their figures and tables. The generic advice is written for prose; a paper is not prose.

Embeddings and the Notation Problem

The embedding model converts text into the vectors you search over. Good current local options: Qwen3-Embedding-0.6B offers strong quality per gigabyte — around 70.7 on MTEB-eng-v2 in about 1.5 GB of VRAM, Apache-2.0 licensed — with an 8B version if you have headroom. bge-m3 is the pick for multilingual work and hybrid dense/sparse retrieval, and all-MiniLM-L6-v2 remains the featherweight CPU-only fallback at roughly 0.1 GB [6].

Two caveats the leaderboards won't tell you. MTEB v2 scores aren't comparable to v1, and rankings shift between boards — so treat them as a shortlist, not a verdict, and benchmark on your own documents [6]. And more importantly for physics: use hybrid retrieval. Dense embeddings capture meaning but blur exact symbols, and in a field where σ, Σ, and s mean three different things, that blurring is fatal. Pairing vector search with keyword search (BM25) recovers exact-match behavior for notation, author names, and equation fragments. This single choice does more for scientific retrieval quality than most model upgrades.

Hardware, Honestly Sized

A useful rule of thumb at 4-bit quantization: VRAM in GB ≈ parameters in billions × 0.6. So a 30B model needs roughly 18 GB for weights, and a 32B at Q4_K_M lands around 18 GB — which is why 24 GB cards (RTX 3090/4090/5090 class) remain the 2026 sweet spot for serious local work [7].

Context costs extra, and researchers feel this more than most because papers are long. Going from 8K to 32K context adds roughly 1–2 GB of KV cache on a mid-sized model; 64K adds 2–4 GB; very long contexts can add 4–8 GB. Quantizing the KV cache to 8-bit roughly halves that penalty for a slight quality cost [7]. Our full hardware breakdown — storage, GPU tiers, whether to build a dedicated box — lives in the self-hosted AI build guide.

Local or Cloud? The Question That Isn't About Capability

Here's the question every reader arrives with: if Claude, GPT, Gemini and Grok are better, why run anything locally at all — and would a serious machine close the gap? The honest answer has three parts, and the first two are not the deciding one.

On raw capability, the cloud is still ahead. Open-weight models have closed roughly 70–90% of the gap at 5–10× lower inference cost, and on coding they've essentially caught up. But the remaining gap sits in general knowledge and hardest-case reasoning [8] — which is precisely where a physicist checking a subtle argument would notice it.

Throwing hardware at it doesn't change the verdict. Four RTX 5090s give you 128 GB of VRAM, enough for roughly 120B-class models at Q4 or a large mixture-of-experts model with community quantization. That is genuinely capable. It is also about $7,200–8,000 in cards alone, and four of them pull around 2,300 W, which means 4,000–5,000 W of power supply headroom [9]. For scale: a standard US 15-amp household circuit tops out near 1,800 W. This is a 240 V or multi-circuit machine with real cooling, not something that lives quietly under a desk.

So roughly $15,000 all-in buys you approximately the open-weight frontier — still a step behind a subscription costing $20–200 a month. On capability per dollar, it isn't close.

Which is why capability isn't the deciding variable — the constraint is. If unpublished thesis work genuinely cannot leave the machine, no amount of cloud capability is relevant, because the cloud isn't an option at any price. If it can leave, the cloud wins decisively on capability per dollar and keeps winning by more each year. Answer the confidentiality question first; the hardware question only matters afterwards.

Two things resolve this in practice. First, "we can't use cloud" is more often assumed than checked — zero-retention API tiers exist, and universities frequently hold institutional agreements that already cover the sensitivity in question. That's a conversation with your research office, and it's cheaper than a GPU. Second, and more useful: split by task rather than by principle.

• Local, modest model + excellent RAG for anything touching the confidential corpus — find the paper, cross-reference the approximation, surface the note from three years ago. Model size barely matters here; retrieval quality dominates.
• Cloud frontier model for hard reasoning over published, sanitized, or already-public material — the places where the reasoning gap is real and the confidentiality risk isn't.

That hybrid captures most of the value for a small fraction of the cost, and it degrades gracefully: as cloud models improve, the reasoning half improves for free, while the private half stays private.

Citations, or It Didn't Happen

This is the requirement that separates a research assistant from a toy, and it's non-negotiable for anyone whose name goes on the output. Configure the system so every answer returns the retrieved passages it used, with document and page. Then adopt a simple discipline: an assertion without a retrieved passage behind it is treated as unsupported, no matter how confident it sounds.

This matters more here than in ordinary use because the failure is asymmetric. A fabricated citation in a thesis is a career problem, not an inconvenience — and as we've documented in our piece on hallucinations, fabricated references are among the most persistent failure modes precisely because they look correct. The same rule applies to equations: never accept a paraphrased result without opening the source and reading it yourself. The assistant's job is to find the passage, not to be believed about it.

Evaluate It Like an Experiment

Researchers have an advantage here and mostly don't use it: they already know how to test something properly. Before tuning anything, write 25–30 questions from your own field where you already know the correct answer and where in your corpus it lives. That's your gold set.

Then measure the two stages separately, because they fail differently. Retrieval: did the right passage make it into the top-k results? Generation: given the right passage, did the model answer correctly from it? Most disappointing systems are failing at stage one while people spend weeks swapping out models to fix stage two. Change one variable at a time, re-run the gold set, keep the number. It's the same discipline as any other experiment.

What It Genuinely Cannot Do

An honest section, because overselling this helps nobody. A local RAG assistant is very good at: finding the paper where you half-remember seeing a result; surfacing your own note from three years ago; cross-referencing which of forty papers used a particular approximation; summarizing a subfield you're entering; checking notation consistency across a long draft; and answering "who has cited this idea and disagreed with it" over your own collection.

It cannot do novel physics. It cannot judge whether a frontier claim is correct, because that judgment is exactly what hasn't been settled in the literature it's retrieving from. It cannot replace reading the paper. And it will be least reliable precisely where your work is most original — the frontier is, by definition, the region where there's nothing to retrieve. Treat it as an exceptionally well-read assistant with no judgment, and it will save you real hours. Treat it as a collaborator and it will eventually embarrass you.

Back to Blog