We tested 5 local models - what we picked and why

Avery NXR supports any model that Ollama can run. In practice, our customers run one of a small set of models that have proven themselves for operational AI work.

We tested many. We standardized our internal recommendations around 5. Here's the comparison.

The models tested

We focused on models in the 7B-14B parameter range — the sweet spot for laptop and mid-tier deployment. We tested:

→ Qwen2.5-Coder 7B (Alibaba) → Qwen2.5-Coder 14B (Alibaba) → DeepSeek-R1-Distill 8B (DeepSeek) → Llama 3.2 8B (Meta) → Mistral 7B (Mistral)

For each, we ran a fixed evaluation suite covering the operational tasks Avery NXR customers actually do — classification, extraction, drafting, summarization, routing logic.

The evaluation suite

5 task categories, each with 20 representative inputs:

→ Classification: support tickets, leads, emails into categories → Extraction: structured data from invoices, contracts, resumes → Drafting: emails, summaries, follow-ups in specific tones → Summarization: meeting transcripts, research documents, customer conversations → Routing: decision logic for multi-step workflows

Each output graded by a human reviewer (us) on a 1-10 scale based on: → Accuracy → Tone match → Edge case handling → Output formatting

100 outputs per model. ~25 hours of total reviewer time. Worth it for confident recommendations.

Results (averaged across 5 categories)

→ Qwen2.5-Coder 7B: 8.4/10 → Qwen2.5-Coder 14B: 8.9/10 → DeepSeek-R1-Distill 8B: 8.6/10 → Llama 3.2 8B: 7.9/10 → Mistral 7B: 7.6/10

The spread is smaller than people often assume. All five are deployable. The differences are real but not chasms.

Performance by task category

Classification: Qwen2.5-Coder 7B and 14B both excellent (8.7-8.9). DeepSeek-R1-Distill close behind (8.5). Llama and Mistral slightly weaker on nuanced classifications.

Extraction: Qwen2.5-Coder 14B clearly best (9.1) for complex extraction. Qwen 7B and DeepSeek 8B both good (8.5). Llama and Mistral struggle with edge cases in structured extraction.

Drafting: Qwen models excellent at structured drafting. DeepSeek slightly stronger on conversational tone. Llama and Mistral fine for simple drafts, weaker on specific brand voice matching.

Summarization: DeepSeek-R1-Distill leads (9.0) — reasoning-trained model helps with information density. Qwen models close (8.6-8.8). Llama solid (8.2). Mistral lags (7.8).

Routing: Qwen2.5-Coder 14B clear winner (9.2). DeepSeek 8B strong (8.8). Qwen 7B good (8.5). Llama and Mistral less reliable on multi-step decision logic.

Hardware requirements

→ Qwen2.5-Coder 7B: ~6GB VRAM/RAM. Runs on M1+ Mac, modern Windows laptop with integrated GPU or 16GB+ RAM.

→ Qwen2.5-Coder 14B: ~11GB VRAM/RAM. M3 Pro+ Mac, modern PC with 24GB+ RAM or dedicated GPU.

→ DeepSeek-R1-Distill 8B: ~7GB VRAM/RAM. M1+ Mac, 16GB+ RAM PC. Reasoning-trained variant adds slight overhead.

→ Llama 3.2 8B: ~7GB VRAM/RAM. Same hardware envelope.

→ Mistral 7B: ~5GB VRAM/RAM. Slightly lighter than others.

For most modern laptops (Apple Silicon M2+ or 16GB+ Windows/Linux), the 7-8B class is comfortable. 14B class needs more memory headroom.

Our recommendations

After all the testing, we've settled into specific recommendations for specific user profiles:

Default recommendation: Qwen2.5-Coder 7B

→ Best balance of quality, speed, hardware compatibility → Excellent across all task categories → Runs on most modern laptops → ~30 token/second inference on Apple Silicon M2+ → Quality at par with much larger cloud models on operational work

This is what we default new Avery NXR installs to. ~80% of our customers run this.

Power user recommendation: Qwen2.5-Coder 14B

→ Best quality across all categories → Worth it if your hardware supports it → Especially better at extraction and routing → Slower inference (~15 tokens/second on M2 Pro, faster on M3+) → ~10-15% of customers run this

For customers with M3 Pro+ Macs or PCs with 32GB+ RAM, this is the upgrade.

Reasoning-heavy recommendation: DeepSeek-R1-Distill 8B

→ Best for workflows with multi-step reasoning → Strongest at summarization and complex extraction → Especially good for agents that need to "think" before acting → Similar hardware requirements as Qwen 7B → ~5-10% of customers run this

For agents doing complex reasoning (legal document review, research synthesis, multi-step analysis), this is often the best choice.

What we don't recommend

We don't recommend Llama 3.2 8B or Mistral 7B for most Avery NXR users — not because they're bad, but because Qwen and DeepSeek are better for our specific operational use cases.

Llama and Mistral are still excellent general-purpose models. They might be the right choice for: → Conversational agents (chatbot-shaped use cases, where we're not the right platform anyway) → Creative writing (we're not optimized for this) → Use cases where you want a Meta or Mistral-aligned stack for other reasons

For the operational AI workloads Avery NXR targets, Qwen and DeepSeek win our internal evaluations consistently.

What this means for users

When you install Avery NXR, the installer recommends Qwen2.5-Coder 7B by default. For most users, this is the right answer.

If you have more powerful hardware and want maximum quality, switch to Qwen 14B.

If your agents do heavy reasoning, switch to DeepSeek-R1-Distill 8B.

Per-agent model selection means you can mix and match — light agents on Qwen 7B for speed, heavy agents on Qwen 14B or DeepSeek for quality.

What changes over time

The local model landscape moves fast. Every 3-6 months, new models appear that change recommendations. We re-test our standard suite quarterly and update default recommendations as needed.

Models we're watching for the next eval cycle:

→ Newer Qwen variants (likely incremental improvements) → Newer DeepSeek variants (likely meaningful improvements in reasoning) → Specialized small models (3B or smaller) that might handle specific tasks well → Multimodal models (vision + text) as those mature

If your Avery NXR install has been running for a while, periodically check whether you should upgrade your model. Quality improvements compound across thousands of agent executions.

The principle

When you can pick your model, picking right matters. Generic "use any model" advice underserves users. Specific recommendations based on real testing helps users actually get good results.

We do the testing so our users don't have to. They install Avery NXR, accept the default recommendation, and get strong outputs. Power users can deviate based on their specific needs.

This is the kind of "we did the work so you don't have to" that platforms should do for their users. Picking and supporting specific models is part of what makes a platform a platform vs. just a framework.

→ avery.software — Free Desktop tier. Installer recommends the right model for your hardware automatically.