Running Large Local Open Source and Open Weights LLM Models on Two Quad RTX-3060 12GB VRAM Threadripper 64GB AI Workstations with 48GB VRAM Each for CrewAI Agents running in a Home Lab

During my Summer break from teaching at Northwest University, I decided to upgrade my home AI lab to run larger local Open Source and Open Weights LLM Models using CrewAI.  I also wanted to bake lighting in very large Unity 3D open world environments using more Multi-GPU VRAM.

CrewAI is an open-source Python framework for building teams of AI agents that collaborate on a job, instead of stuffing everything into one giant prompt. You define people-like roles (“researcher,” “writer,” “reviewer”), give each a goal and tools, then CrewAI handles handoffs, context passing, and the process that runs them. It is not an LLM itself. It is the orchestration layer. The actual thinking is done by whatever model you point it at.

Since I didn’t want to purchase a very expensive brand-new RTX-5090 32GB VRAM card with very inflated street prices ($3,800+), I decided to purchase a couple of lightly used pre-owned AMD Threadripper AI Workstations both with Quad Multi-GPU Gigabyte RTX-3060 12GB VRAM video cards with a total of 48GB VRAM in each system.  I also upgraded both systems to 64GB of DDR4 System RAM so I could run multiple Crew AI Agents on both systems simultaneously.

This also allowed me to start baking my large Unity 3D open world scenes using Bakery with simultaneous multi-GPU light baking.  My Alienware R11 with a single RTX-3080 with 10GB could not complete the GPU baking with its limited VRAM memory.  Now I can bake large levels and render larger and longer cutscene videos faster. Basically, I got 2 lightly used AMD Threadrippers with a combined pool of 96 GB of VRAM memory running across 8 RTX-3060s for less than the price of 1 new RTX-5090 card with 32GB VRAM!

The largest models that fits comfortably on the Quad RTX-3060 with 48GB of VRAM are ~70B parameter model quantized to Q4_K_M or Q5_K_M.  That is roughly 35–45 GB total VRAM needed after overhead.  With 4 × 12GB = 48 GB total VRAM, you can run up to a 70B–90B model in Q4/Q5, or a smaller dense model like a good 34B–70B at higher quantization for better quality.

– Generation speed: 25–45 tokens/second (typical range)
– Optimistic (good prompt, context <8k): 35–45 t/s
– Average real-world (8k–16k context, complex prompts): 28–38 t/s

For a smaller but higher-quality model setup (e.g., 34B–40B Q6/Q8):

– 45–70+ tokens/second (much snappier for chat/coding)

The x399 Threadripper systems provide plenty of PCIe lanes for good multi-GPU scaling.  12GB per card is excellent for layer offloading using Ollama or LM Studio.  These factors will improve my local AI performance over my single GPU RTX-3080 10GB VRAM Alienware R11 system.  This should prove to be a very economical and capable local AI system for what I want to do over my summer break.  It is proving excellent for running large models offline, chatting, coding assistance, or light fine-tuning.  I am already seeing better speeds than my single RTX-3080 card for big models that could not fit on my single RTX-3080.

I plan to test the following AI Models locally on it:

• Primary: Qwen 2.5 Coder 32B or Llama 3.3 70B
• Great for Unity-specific tasks: Editor scripts, Addressables, DOTS/ECS, Shader Graph, UI Toolkit.

• Primary: Llama 3.3/4 70B or DeepSeek Coder
• Excellent for Unreal-specific: Niagara, Behavior Trees, GAS (Gameplay Ability System), Slate/UI, C++ performance optimization.

Here are the hardware components that make up the two Threadripper AI workstations:

I’ll follow up with another post at the end of summer to let you know how my local AI co-development went using these AMD Threadripper Quad RTX-3060 48GB VRAM workstations.