From cloud to local: the AI migration playbook

You decided to migrate AI workloads from cloud to local. Either the bill got too big, or compliance asked the wrong question, or a vendor change forced your hand. Now you actually have to do it.

The migration is doable. It's also non-trivial. This post is the playbook in the order that works: audit, hardware, model selection, code migration, fallback design, parallel running, cutover, monitoring.

Done badly, a cloud-to-local migration takes six months and produces a janky system that nobody trusts. Done well, it takes six weeks and produces a system that's cheaper, faster, and more reliable than what it replaced.

Step 1: Audit your workloads

Before touching any code, list every AI call your product makes. Not just the obvious ones. The internal tools, the support workflows, the analytics queries that happen to use AI, the ad-hoc scripts that engineers run.

For each call, capture three things.

Privacy sensitivity. Does this call touch customer data, PII, PHI, MNPI, or anything else with compliance implications? Mark these candidates for migration regardless of cost.

Volume and recency. How many of these calls happen per month? When did volume start? Stable workloads are easier to migrate than ones that are still evolving rapidly.

Capability requirement. Does this call need the absolute best model, or would a competent local model work? Most calls in production are doing classification, summarization, extraction, or drafting. These are local-model tasks. The rare calls doing deep multi-step reasoning may need to stay on frontier.

The audit usually reveals that 70 to 90 percent of AI calls are migration candidates. Build the list. Sort by volume (highest first). That's your migration backlog.

Step 2: Pick your hardware

Hardware is the constraint. Choose it before you choose models.

For a solo developer or small team trying local AI for the first time: an M-series Mac with 32 to 64GB RAM is the simplest path. Runs 14B-class models comfortably. Develops and deploys on the same machine. Quiet, energy-efficient, and most people already have one.

For a production team running local AI as serious infrastructure: dedicated workstations or servers with NVIDIA GPUs. RTX 4090 (24GB VRAM) for development. RTX 5090 or workstation cards (RTX 6000 Ada with 48GB VRAM) for production inference. Or H100/H200 if you're going for serious enterprise-scale serving.

For air-gap or enterprise on-premise: server-grade hardware with the GPU choice based on model size requirements. 80GB-class GPUs (H100, H200) for 70B+ models. Multiple consumer cards in parallel for cost efficiency. Hardware security modules and TPM if compliance requires them.

The right hardware budget for most teams is a one-time spend of a few thousand to a few tens of thousands of dollars. The recurring cloud AI bill it replaces is typically much higher. The payback period is short.

Step 3: Pick local models

Once you know your hardware, choose models that fit.

For coding tasks (code generation, code review, refactoring, debugging): Qwen2.5-Coder. The 7B variant runs on 16GB+ Macs. The 14B variant runs on 32GB+ Macs or 24GB+ GPUs. The 32B variant runs on larger workstations.

For reasoning tasks (analysis, planning, multi-step problem solving): DeepSeek-R1-Distill. The 8B Llama distillation is the sweet spot for reasoning quality at modest hardware requirements.

For general tasks (chat, drafting, summarization, classification): Llama 3.3. The 8B variant is fast and competent for most tasks. The 70B variant is closer to frontier quality but requires workstation hardware.

For specialized tasks: there are domain-tuned models for healthcare, legal, finance. They're often outperformed by general models with good prompting, so test before committing.

Quantization matters. Q4_K_M is the standard sweet spot. Halves memory requirements with modest quality loss. Q8 if you have memory to burn and want maximum quality. Q2 only if you're desperate to fit a bigger model on smaller hardware.

Avery NXR ranks models for your specific hardware automatically. Skips the guesswork.

Step 4: Set up the local inference layer

The infrastructure for running models locally.

The simplest path: Ollama. Open source, OpenAI-compatible API, runs on Mac/Linux/Windows. Pull a model, start the server, point your app at localhost. Done in 30 minutes.

The slightly more sophisticated path: vLLM or llama.cpp directly. Better performance at scale, more configuration. Worth it for production serving but overkill for the first migration.

The Avery NXR path: integrated local AI runtime with model management, audit logging, and the agent platform on top. Worth using if you want the full stack, not just inference.

For your migration, start with Ollama. It's the lowest-risk choice. Switch to something more sophisticated only if you outgrow it.

Step 5: Replace the SDK calls

The code migration. Mostly mechanical, but with judgment calls.

If you're using the OpenAI SDK: Ollama exposes an OpenAI-compatible API. Change the base URL and the model name. Most code keeps working.

If you're using the Anthropic SDK: more rewriting required. Anthropic's prompt format is different. The completions API differs from chat APIs. Budget more time.

If you're using LangChain or LlamaIndex: these have abstractions for swapping providers. Configure the abstraction to point at the local model. Often a one-line change.

If you're using a custom HTTP client: rewrite the client to hit the local endpoint. Add proper retry logic since local errors look different from cloud errors.

The mechanical part is fast. The harder part is testing.

Step 6: Prompt engineering for smaller models

Local models in 2026 are roughly where frontier cloud models were in early 2024. They're competent but not as forgiving as GPT-5 or Claude Opus 5.

Prompts that worked perfectly on frontier cloud models often need refinement to work well on local. The patterns that matter:

More explicit instructions. Local models benefit from clearer task framing. "Summarize this document" might work on GPT-4. On a 14B local model, "Summarize this document in exactly 200 words. Focus on the three most important findings. Use bullet points." works better.

Few-shot examples. Including 2 to 3 examples of the desired output dramatically improves quality on local models. Costs more tokens, but the tokens are free locally.

Structured output. Asking for JSON with a schema works better than asking for "structured information." Tools like Pydantic, Zod, or Avery NXR's structured output mode reduce the post-processing burden.

Smaller, more focused tasks. A single complex prompt that works on frontier might need to be decomposed into three simpler prompts on local. The aggregate output is often better.

Budget two weeks of prompt iteration after the mechanical SDK migration. The iteration pays back because the prompts then work consistently in production.

Step 7: Build the Consult Mode fallback

For the 10 to 30 percent of calls that genuinely need frontier capability, build a controlled escalation pattern.

The pattern: try the local model first. If the output fails a validation check (low confidence, missing fields, malformed response), escalate to a frontier cloud model. The escalation uses BYOK keys (your customer's API keys, not yours). The payload is anonymized before it leaves. The user sees what would be sent and can decline.

This is what Avery NXR calls Consult Mode. The pattern works for any local-first system.

The benefits: you keep cloud capability for the cases that need it, you pay only for those specific calls, and you maintain audit control because escalations are explicit and logged.

The implementation: a validator function that checks local model output, a fallback path that calls the cloud API, anonymization middleware that strips PII before the cloud call, and audit logging for the entire flow.

Budget a week to build this carefully. It's the safety net that lets you migrate aggressively without breaking anything important.

Step 8: Run in parallel

Before cutover, run cloud and local in parallel for a few weeks.

Send each production AI call to both. Store both outputs. Compare them. Track where they agree, where they disagree, and where the local output is meaningfully worse.

Some calls will agree most of the time. Migrate those first.

Some calls will diverge frequently. Investigate why. Often it's prompt issues that more iteration can fix. Sometimes it's a genuine capability gap that means the call stays on cloud.

The parallel period is annoying because you're paying for cloud while building toward not paying for it. But it's the only way to migrate aggressively without breaking quality.

Two to four weeks of parallel running is typically enough to find the issues. After that, you can cutover with confidence.

Step 9: Cut over

Once you're confident in the parallel results, flip the dominant path.

Local becomes the default for the workloads you migrated. Cloud stays on standby through Consult Mode for the cases that need it.

Monitor closely for the first two weeks. Watch latency (should drop dramatically), cost (should drop dramatically), error rates (should be roughly flat), and any quality regressions in downstream metrics (customer satisfaction, support resolution times, etc.).

Have a rollback plan. If something breaks badly, flipping back to cloud should be a config change, not a re-architecture.

In practice, most teams don't need to roll back. The parallel running surfaces the issues before they become production incidents.

Step 10: Monitor and iterate

Post-migration, the work continues.

Cost monitoring. Your AI bill should drop significantly. Monitor to confirm. If it doesn't drop as expected, find out why. Usually it's a workload that still hits cloud through some path you missed.

Quality monitoring. Track the downstream metrics that matter (customer satisfaction, conversion rates, support handle time). Local AI should match or exceed cloud AI on these because the latency improvements compound.

Hardware utilization. Monitor your local inference machines. If they're underutilized, you over-provisioned. If they're saturated, you need more capacity. Adjust.

Model updates. Local models keep improving. Plan to test new model releases quarterly and upgrade when they meaningfully improve quality.

Prompt iteration. Production AI workloads evolve. Keep refining prompts as you learn what works.

Common migration failure modes

Migrating without auditing first. Teams that just start replacing SDK calls without understanding their workload end up with a mess. Audit first.

Picking wrong hardware. Buying a 16GB Mac for a workload that needs a 14B model is a path to frustration. Spec hardware based on the models you actually want to run.

Skipping the parallel run. The pressure to "just ship the migration" leads to quality regressions in production. Run parallel for a few weeks. It's worth the cost.

No Consult Mode fallback. Pure local works for 70 to 90 percent of calls. The remaining 10 to 30 percent need cloud. Without a fallback path, the migration produces a system that's worse than the cloud-only version it replaced.

Not iterating on prompts. Local models need prompt tuning that frontier models did not. Budget the time.

The honest timeline

For a small team migrating a moderate-scale AI workload:

Week 1: audit and hardware procurement.

Weeks 2-3: local inference setup, initial SDK swap, basic testing.

Weeks 4-5: prompt iteration for local models.

Weeks 6-7: Consult Mode fallback and parallel running infrastructure.

Weeks 8-10: parallel running and quality assessment.

Week 11: cutover.

Weeks 12-14: monitoring and iteration.

About three months end to end. The team is back at full velocity by week 14, with a system that costs significantly less to run going forward.

The payback period on the engineering effort is typically under six months for any team doing more than $5,000 per month in cloud AI spend.

When NOT to migrate

Honesty about the cases where staying on cloud is the right call.

Your team is two people and you do 100 AI calls per day. Just keep paying OpenAI. The migration effort isn't worth it at this scale.

Your AI workload is experimental and might be deleted in a month. Don't commit hardware to it. Use cloud, throw it away if it doesn't work, commit to local when the workload stabilizes.

Your specific tasks genuinely require frontier capability. Pay for the frontier model. Don't downgrade to a worse local model just to save money.

Your team has no engineering capacity to absorb a migration project. Migrating poorly is worse than not migrating. Wait until you have the resources, then do it well.

For everything else, the migration math works. The playbook above is the path that succeeds.

avery.software