Self-Hosted Model Operations
Production operations for a self-hosted model — keeping it running, keeping it accurate, keeping the bill predictable. Real production, not a notebook.
<p>This is the operations engagement. The model is trained and deployed; what is missing is the production discipline that keeps it running through traffic spikes, drift, hardware failures, and the inevitable Sunday afternoon when a customer asks why the bot just refused fifty consecutive queries. The deliverable is the operations layer — inference serving, observability, drift monitoring, retraining triggers — that turns a model on a GPU into a production service.</p>
<p>The buyer for this page is a CTO or platform leader who has the model in production (or near it) and needs the operations layer to look like a real production system. Per-token-tax-free is not free; self-hosted inference still needs the runbook, the dashboards, and the alerts that any other production service needs. The parent — <a href="/services/customer-service-ai">Customer Service AI Models</a> — describes the model itself; this page is about keeping it running.</p>
<h2>Deployment topology</h2>
<p>On-prem GPU clusters where the buyer owns the hardware. Customer VPC on Azure, AWS, or GCP where the cloud is the platform. Hybrid where the inference lives on prem and the orchestration lives in the cloud. The topology is scoped against the customer's latency target, throughput profile, redundancy requirements, and existing platform discipline.</p>
<h2>Inference serving</h2>
<p>vLLM, TGI, TensorRT-LLM, or Ollama depending on the model architecture, the hardware, and the latency budget. Batching strategy tuned to the traffic shape — interactive web chat wants low p99 latency, batched email response generation wants throughput. The serving layer is monitored, the autoscaling is bounded, and the cost ceiling is enforced.</p>
<h2>Observability and cost</h2>
<p>Token-level metrics: prompt tokens, completion tokens, latency percentiles (p50, p95, p99), refusal rates, fallback triggers, cache hit rate. Cost dashboards in dollars per thousand conversations, not in vendor-API tokens. Alerting that pages a human on drift — not just on outages.</p>
<h2>Retraining cadence</h2>
<p>Written trigger policy. Corpus drift threshold (the embedding distribution between the current corpus and the training corpus passes a measured threshold). Eval-gate criteria (the new model must beat the production model on the customer's actual tasks, not on a public benchmark, before promotion). The retraining run is reproducible from the same pipeline the original training run used.</p>
<h2>What I ship</h2>
<ul>
<li><strong>Inference serving stack.</strong> Tuned to the customer's latency and throughput profile.</li>
<li><strong>Observability dashboards.</strong> Token-level metrics, latency percentiles, refusal rates, cost in dollars per thousand conversations.</li>
<li><strong>Alerting.</strong> Drift, outage, cost-ceiling breach, refusal-rate spike — paging a human, not just lighting up a dashboard.</li>
<li><strong>Retraining-trigger policy.</strong> Written, with the drift threshold and the eval gate named.</li>
<li><strong>Cost ceilings and rate limits.</strong> So a runaway loop or a viral mention does not burn the quarter.</li>
<li><strong>Runbook.</strong> A second engineer can run the system without a meeting.</li>
</ul>
<h2>Where it fits</h2>
<h3>From notebook to production</h3>
<p>The model is running on a single GPU under a Streamlit demo a data scientist set up. It works for the demo. It will not survive the traffic when the marketing team turns on the announcement. I engineer the production deployment — serving stack, observability, autoscaling, cost ceilings, alerting — so the model can be put in front of real customers.</p>
<h3>Predictable cost</h3>
<p>The CFO wants the cost of inference to be a known number per month, not a surprise on the cloud bill. I instrument the cost dashboard in dollars per thousand conversations, enforce the cost ceiling, and provide the cost-attribution view the finance team can reconcile against the customer-volume forecast.</p>
<h3>Drift is happening, no one is measuring it</h3>
<p>The model was good in February. It is March. The product team shipped a new SKU line, the support team adopted new vocabulary, and the refusal rate is creeping up. I wire drift monitoring against the corpus and the customer's actual KPIs, with alerts that fire before the dashboard turns red — not after.</p>
<h2>How I work</h2>
<p>Discovery scopes the deployment target, the latency and throughput profile, the cost ceiling, the existing observability stack, and the retraining requirements. Implementation runs in two-week sprints with the serving stack live at the end of sprint one and observability layered in over sprints two and three. The principal carrying the work is described <a href="/about">on the about page</a>.</p>
<h2>Engagement model</h2>
<p>Discovery runs two weeks. Implementation runs four to ten weeks depending on the deployment topology and existing platform maturity. Handover includes the serving stack, the dashboards, the alerting, the retraining policy, the runbook, and a 30-day support window. Retainer arrangements exist for organizations running the model continuously and needing ongoing drift triage. To scope an operations engagement, <a href="/contact">get in touch</a>.</p>