Deploying LLM apps: the parts that aren't your model

Deploying an LLM app is mostly not deploying the model. The model is a managed API call, give or take. What you actually deploy is everything around it: the prompt management, the retrieval layer, the caching, the queue, the rate limiter, the observability stack. Most teams under-estimate this surface area until the second month, when the on-call rotation gets unpleasant.

The components that need real engineering

Prompt management deserves to be versioned and rolled out like code, with the ability to revert to a known-good version when a “small wording change” breaks twelve downstream tasks. The retrieval layer is a database problem with database problems: index freshness, embedding-model upgrades, partial-update races. The queue between user request and model call is where you handle backpressure and abort propagation, and getting it wrong is how a slow upstream becomes a melted application. The cost dashboard is operational, not optional — without it, the first time you’ll notice unbounded retry loops is on the bill.

What the deploy itself looks like

Canary your prompt changes the way you canary code changes — a small slice of traffic, with comparison metrics, before any full rollout. Treat the model API as a dependency that goes down: have a degraded mode that returns something useful or a graceful failure, not a 500. Keep secrets out of prompts; the model logs every prompt to your observability stack, and you do not want to be debugging a credential leak that lives in your trace store.

Deploying an LLM app is the part that the tutorials skip. It is also the part that determines whether your app is a demo or a product.