Why we picked Ollama (and not the obvious choice)

Avery NXR runs local models through Ollama. That's a deliberate choice and it sometimes surprises engineers who'd expect us to use vLLM or another more "production-grade" runtime.

Here's the reasoning, because architectural choices matter and we want to be transparent about ours.

What we evaluated

Before settling on Ollama, we evaluated:

→ Ollama — easy install, good ecosystem, designed for local use → vLLM — high-throughput, designed for serving → llama.cpp — bare-metal performance, more configuration → LM Studio — GUI-first, less developer-friendly → Hugging Face Transformers — most flexible, most complex → MLX (Apple) — best Mac performance, Mac-only → Building our own runtime — most control, most cost

Each has merits. None was a clear winner on all axes.

The criteria that mattered

We picked criteria based on the audience Avery NXR is for: people who want local AI agents but aren't AI infrastructure engineers.

1. Install complexity. Can a non-ML-engineer get this running in under 10 minutes? Ollama wins decisively here. brew install ollama or download the installer; you're done. vLLM requires CUDA setup, dependency management. llama.cpp requires building from source for best results.

2. Model availability. Are the models we care about (Qwen2.5-Coder, DeepSeek-R1-Distill, Llama variants) easily accessible? Ollama's model library is curated and consistent. Other runtimes require you to know the right file path on Hugging Face.

3. Hardware coverage. Does this work on the laptops our users actually have? Ollama runs well on Apple Silicon (M-series), works on Intel Macs, works on Windows with NVIDIA GPUs, works on Linux with NVIDIA / AMD. The broadest coverage of consumer hardware.

4. Quantization handling. Quantized models (Q4, Q5, Q6, Q8 variants) need to be handled correctly for the quality/memory trade-off. Ollama handles this transparently — pick a model, it picks reasonable quantization. Other runtimes require you to specify.

5. Update path. When a new model lands, how soon is it available? Ollama's model library tends to add new models within days. Other runtimes require you to track HF, download manually, configure.

6. Service model. Does this stay running in the background, or do we need to manage process lifecycle? Ollama runs as a service. You don't think about it. Some other runtimes require explicit process management.

What we trade by picking Ollama

We're honest about the trade-offs:

Lower throughput than vLLM. For high-concurrency serving (many requests in parallel), vLLM is faster. For single-user laptop usage, the difference doesn't matter. Most Avery NXR users have 1-3 agents firing serially.

Less customization than llama.cpp. vLLM and llama.cpp expose more configuration knobs — batch sizes, KV cache configs, attention implementations. Ollama abstracts these. For users who want to tune deeply, Ollama is the wrong choice. For 95% of Avery NXR users, they don't want to tune; they want it to work.

Larger memory footprint per model than optimal. Ollama's defaults trade some memory for ease of use. A llama.cpp expert could squeeze more out of the same hardware. Most users have memory headroom.

Less Linux server-friendly. Ollama is great for laptops. For serving from a Linux server at scale, vLLM is the better choice. Avery NXR Pro/Enterprise has a different deployment path for that use case.

Why these trade-offs are right for Avery NXR's audience

Our target user is a knowledge worker, indie developer, small team operator, or mid-market team running agents on laptops or their own cloud. NOT an AI infrastructure team trying to maximize throughput on a GPU cluster.

For that audience:

→ Install complexity matters MORE than throughput → Model availability matters MORE than customization → Hardware coverage matters MORE than peak performance → Reliable defaults matter MORE than tunability

Ollama optimizes for exactly these. So do we.

What this enables architecturally

By standardizing on Ollama, we can:

Detect available models. Avery NXR can query the local Ollama instance and show users which models are available. No model paths to configure.

Recommend by hardware. We can detect available RAM and recommend models that fit. M2 Mac with 16GB → suggest Qwen2.5-Coder 7B Q4. M4 Pro with 48GB → suggest a 13B variant.

Hot-swap models per agent. Different agents can use different models. The CRM agent uses one optimized for tabular reasoning. The support agent uses one optimized for conversation. Ollama makes the swap fast.

Provide a consistent API. Every Avery NXR install talks to Ollama the same way. Cross-platform consistency.

What Ollama lets us NOT build

This is the underrated part. By using Ollama, we don't have to build:

→ Model downloading and caching → Quantization handling → Memory management for model loading → Cross-platform inference runtime → GPU acceleration on Apple Silicon, NVIDIA, AMD → Model API standardization

These are all hard problems that Ollama's team has solved well. We benefit from their work. We focus on the layer above — the agent platform.

This is the right kind of dependency. Critical infrastructure, well-maintained, open-source, with multiple commercial entities relying on it.

What changes if our calculus changes

We may add additional runtime support in the future. If Avery NXR moves toward enterprise deployment patterns where vLLM's throughput matters, we'll add it as an option.

But for the desktop tier where most users live? Ollama is the right choice. We'd make it again.

The broader lesson

When you build a product, the right infrastructure dependency is often the one that matches your USERS, not the one that's most impressive technically.

vLLM is more impressive. Ollama matches our users better.

That's the choice every product team makes a hundred times. We made it explicitly for the model runtime layer, and we'd make it the same way again.

→ avery.software — Free Desktop tier. Avery NXR + Ollama on your laptop. The whole local-first stack, installable in ~15 minutes.