The idea of owning your own AI — one that runs entirely on your hardware, knows only what you teach it, and never phones home with your data — has shifted from enthusiast fantasy to realistic weekend project. Ollama, Qdrant, LlamaIndex, and a wave of open-weight models have lowered the barrier dramatically. But the glossy tutorials leave out the honest stuff: how much storage you actually need, which GPU tier buys real capability versus frustrating limitations, and what risks come with running inference on hardware sitting in your living room. This guide covers all of it.
We will walk through the entire stack — from picking your GPU and sizing your storage, to choosing a vector database, wiring up RAG, and deciding whether to run everything on a dedicated machine or pay someone else to run the hardware for you. We will also be direct about where things go wrong, because understanding failure modes matters as much as understanding the happy path.
Before You Start: What Are You Actually Building?
A "personal AI" is not a single thing — it is a stack of components that work together. Most people conflate three distinct systems: a local LLM (the model that generates text), a vector database (the store that enables semantic search over your documents), and a RAG pipeline (the wiring that connects the two at query time). You can run any one of them independently, and the combination is where the real power emerges.
The local LLM handles open-ended conversations, code generation, summarization, and general reasoning — it is equivalent to ChatGPT or Claude, running on your own machine. The vector database transforms your private documents (PDFs, notes, emails, codebases) into searchable embeddings so the LLM can answer questions grounded in your data. The RAG pipeline orchestrates the handoff: when you ask a question, it retrieves the most relevant chunks from your vector store and injects them into the LLM's context window before the model generates its response.
Each component has its own hardware requirements, software options, and risk profile. Let us go through them in order of what you will need to decide first.
Storage: What You Will Actually Need
Storage is consistently underestimated by first-time builders, and running out of space mid-download is a miserable experience. There are three distinct storage needs to plan for: model files, vector database index, and your raw document corpus.
Model storage is the most immediately pressing concern. Model file sizes depend almost entirely on the number of parameters and the quantization level you use. Quantization compresses model weights to reduce VRAM usage at a modest quality cost — Q4 (4-bit) is the community standard for most use cases. Here are the real numbers you should plan around: a 7B model in Q4 occupies approximately 4–5 GB; a 13B model in Q4 takes 8–9 GB; a 34B model requires about 20–22 GB; and a 70B model in Q4 needs 40–45 GB. If you want full-precision weights (FP16) for maximum quality, multiply those figures by roughly 3.5. You will also likely want to keep several models on disk simultaneously — a coding model, a general-purpose model, and an embedding model at minimum.
VRAM is where models live while they run. Storage is where they live when they don't. Plan both, or your model won't fit in either place.
Vector database storage depends on your embedding dimensions and the number of chunks you store. At 1,536 dimensions (OpenAI-compatible size), one million vectors consume roughly 6–12 GB including index overhead — the exact figure varies by database and index type. If you use smaller 768-dimension embeddings (common in many local embedding models), you roughly halve that. A practical personal knowledge base of 10,000 document chunks sits comfortably under 1 GB. A small business's entire document archive at 500,000 chunks would need 3–6 GB of indexed vector storage.
Raw document corpus is whatever you are feeding the system — PDFs, markdown files, code repos, or scraped web content. This scales with your content library. For most individuals it is under 50 GB; for business use cases it can reach terabytes.
The practical minimum recommendation is a 2 TB NVMe SSD as your primary drive. NVMe matters for model loading speed — the difference between a slow SATA SSD and a fast NVMe drive is noticeable when loading a 40 GB model. For raw documents and backups, a secondary HDD of 4–8 TB provides affordable overflow storage without impacting inference performance.
GPU Tiers: Minimum, Sweet Spot, and Power Build
VRAM — the memory on your GPU — is the hard ceiling of your entire build. It dictates which models fit, how fast inference runs, and whether you can run embeddings in parallel. Every GPU tier recommendation below is built around VRAM first, then cost and power.
The RTX 3060 12 GB is the community-recommended entry point for good reason — it has more VRAM than the RTX 3080 10 GB despite being two generations older, and it can run Q4 7B models without splitting layers to system RAM. The RTX 4060 Ti 16 GB is the better modern choice if you can stretch the budget: those extra 4–6 GB of VRAM let you work with longer context windows and run embedding generation in parallel with inference.
Apple Silicon is worth mentioning here as an alternative path. A MacBook Pro with an M3 or M4 chip and 16–24 GB unified memory can run 7B models at impressive speeds because the GPU and CPU share the same memory pool — there is no VRAM vs. system RAM boundary. If you already own one, Ollama on Apple Silicon is a legitimate starter setup requiring zero additional hardware investment.
The RTX 3090 24 GB is the most popular choice in self-hosted AI communities for a reason that comes down to one number: 24 gigabytes. At that VRAM ceiling, you can run a 34B model in Q4 quantization entirely on-GPU with no CPU offloading — that means fast, responsive inference rather than the crawl that happens when layers spill into system RAM. It is a last-generation card with a used-market price that has dropped considerably, making it one of the best value propositions in the space.
Apple Silicon also becomes compelling at this tier: an M3 Max or M4 Max with 48–64 GB unified memory can run 34B models at usable speeds, making it viable for serious development work without a separate PC or dedicated GPU. The catch is cost — a maxed MacBook Pro reaches $3,000–4,000, which exceeds a purpose-built PC with the same effective model capacity.
The RTX 4090 is the best single-GPU option for consumer self-hosting. At 24 GB VRAM and with NVIDIA's fastest consumer-grade tensor cores, it runs 70B models in Q4 at two to three tokens per second — slow by cloud standards but entirely usable for interactive conversation. The dual RTX 3090 setup doubles your VRAM to 48 GB for roughly the same price, enabling 70B at Q5 or even Q8, though you will need a high-wattage power supply (1000W+) and a case with the airflow to match.
The AMD RX 7900 XTX deserves a mention: at 24 GB VRAM and around $950, it is significantly cheaper than the RTX 4090. ROCm (AMD's GPU compute platform) has improved substantially, and Ollama supports it natively. Performance trails NVIDIA on most workloads, but if the price delta matters more than the last 20% of speed, it is a legitimate option.
For Apple Silicon at this tier, the M3 Ultra or M4 Max with 96–192 GB unified memory can run 70B models comfortably and approaches 10 tokens per second — competitive with an RTX 4090 on many benchmarks. The cost is $4,000–8,000+, which places it firmly in professional workstation territory.
Choosing Your Self-Hosted Vector Database
A vector database stores your document chunks as high-dimensional numerical representations (embeddings) and retrieves the most semantically similar chunks when you run a query. Choosing the right one depends on whether you are building a quick local prototype or a production-grade system, and whether you want a managed server or an embedded library. For a deeper explanation of how vector databases work, see our Vector Database deep-dive.
Python-native, zero configuration, runs embedded in your app or as a server. The easiest entry point — import it, add documents, query. No Docker required. Performance limits kick in above ~100K chunks.
Rust-based for raw performance, excellent Docker support, built-in filtering, and a clean REST/gRPC API. Handles millions of vectors without breaking a sweat. The community's go-to recommendation for serious self-hosted deployments.
Combines semantic vector search with BM25 keyword search in a single query. GraphQL interface. Best when your use case benefits from both exact keyword matching and fuzzy semantic similarity — legal documents, code search, support tickets.
Meta's in-process similarity search library. No server to run — it lives inside your Python process. Maximum control, no overhead. The right choice when you are building a custom pipeline and do not need persistence or a query API.
For most personal and small-team use cases, the recommended path is: start with Chroma while learning, migrate to Qdrant once you need persistent storage, filtering, or more than a few tens of thousands of documents. Milvus (not listed above) is worth knowing about for enterprise-scale deployments requiring Kubernetes orchestration, but it is significantly more complex to operate.
Running a Local LLM with Ollama
Ollama has become the de facto standard for local LLM serving — and for good reason. It wraps llama.cpp's inference engine in a clean CLI and server, handles model downloads from a curated library, exposes an OpenAI-compatible REST API, and manages GPU acceleration automatically. Getting from zero to a running 8B model is genuinely under fifteen minutes on a supported system.
The local LLM ecosystem has converged around a few good tools, each with a different strength:
ollama run llama3.1, and get an OpenAI-compatible API on localhost:11434. Supports NVIDIA, AMD, and Apple Silicon automatically. Best overall choice for 90% of users.Wiring It All Together: Self-Hosted RAG
RAG (Retrieval-Augmented Generation) is the pipeline that transforms your local LLM from a general-knowledge chatbot into a system that can answer questions grounded in your private documents. The flow has four stages: ingest your documents by splitting them into chunks; convert each chunk into an embedding (a numerical vector) using an embedding model; store those vectors in your vector database; and at query time, embed the user's question, retrieve the top-K most similar chunks from the database, and inject those chunks into the LLM prompt before it generates a response. For a detailed explanation of why chunking matters at each stage, see our article on AI chunks.
The fastest paths to a working RAG setup are Open WebUI (which has native document upload and retrieval built in, backed by Ollama and a local Chroma store) and AnythingLLM (which handles the entire pipeline through a desktop GUI). For developers who want more control, LangChain or LlamaIndex paired with Qdrant and a local embedding model (such as nomic-embed-text or mxbai-embed-large, both available via Ollama) is the standard production-oriented approach. The key choices are your embedding model, your chunk size (512–1024 tokens is the typical starting range), and your retrieval strategy (naive top-K similarity, hybrid BM25+semantic, or reranking).
Should You Use a Dedicated PC?
Running your AI stack on your main computer works fine while you are learning, but it creates real friction in daily use. Every inference job competes with your browser, editor, and other applications for GPU memory. If your stack is always-on (a RAG server your team can query, for example), that means your primary workstation needs to stay on and undisturbed. Many people hit this friction wall within a few weeks and start thinking about a dedicated machine.
No extra upfront cost. If you already have a capable GPU, you can start immediately. Good for experimentation and learning the stack.
Shared resources. Gaming or video work while a 70B model is loaded will cause VRAM conflicts. You will be force-quitting one or the other constantly.
No always-on availability. If your machine sleeps, your AI goes offline. Fine for personal use; a problem for team setups.
Always-on access. A dedicated machine can run your Ollama server, Qdrant, and Open WebUI as background services without competing with your workflow.
Additional upfront cost. A used workstation with an RTX 3090 and 32 GB RAM will run $1,000–1,800 all-in. Factor in a UPS, case, and additional storage.
Ongoing power cost. A dedicated AI server draws 200–500 W under load. Running 24/7 adds $15–45/month to your electricity bill depending on usage patterns.
A practical middle path is a used workstation-class machine — a Dell Precision, HP Z-series, or similar — with a transplanted RTX 3090. These chassis are designed to run 24/7, have better airflow than consumer towers, and often come with server-grade components for under $400 before the GPU. Add a $700 used RTX 3090, 32 GB of ECC RAM, and a 2 TB NVMe, and you have a capable, always-on AI server for around $1,200–1,400 total.
Hosted Ollama: Let Someone Else Run the Hardware
Not everyone wants to manage physical hardware — the noise, the power draw, the maintenance, the upfront cost. Several cloud providers let you run Ollama (or an equivalent inference stack) on rented GPUs, paying only for what you use. This is often the smarter choice for workloads that are bursty rather than constant, for teams evaluating before committing to hardware, or for anyone who wants 70B-class inference without buying two RTX 3090s.
* Serverless pricing quoted as equivalent hourly rate; actual billing is per-second.
For most people getting started, RunPod is the recommended hosted path. The Ollama community templates reduce setup to selecting a GPU, clicking deploy, and SSHing into a running server. The community cloud tier (shared hardware) starts at $0.12/hr for an RTX 3080 — that is under $1.00 to run a full experiment. Vast.ai is worth checking when RunPod is out of stock on a particular GPU tier, as its marketplace pricing can be notably lower during off-peak periods.
The Risks Nobody Talks About
The tutorials make this look easy. Here is what they skip.
Self-hosted AI is not for everyone — but for the right use case, it is transformative. If you need absolute data privacy, want unlimited inference without per-token costs, or are building workflows that depend on custom knowledge bases, owning your stack gives you capabilities and control that no SaaS product can match.
The recommended starting point for most people is: Ollama on your existing hardware, nomic-embed-text as your embedding model, Chroma as your initial vector store, and Open WebUI as your chat interface. Get that running with one or two of your own documents first. Understand what chunking strategy works for your content. Feel the latency of your current GPU. Then decide whether to invest in dedicated hardware, upgrade VRAM, or move part of the stack to a hosted provider.
The worst approach is to spend $1,500 on an RTX 3090 build before you have run your first query. Start cheap, start now, and let your actual use case tell you where the bottlenecks are.
Sheldon writes about AI infrastructure, developer tools, and the practical reality of building with AI systems. Dear Tech covers the technology powering modern AI — without the hype.