Field notes from
shipping AI at scale.

Most teams asking us "should we fine-tune or use RLHF" are asking the wrong question. The right one is: at what cost, at what scale, and against what failure mode?

Here's what we've seen across roughly forty production engagements over the last eighteen months.

LoRA and QLoRA

LoRA works when your domain shift is moderate, your data volume is in the thousands-to-low-tens-of-thousands of examples, and your latency budget doesn't require collapsing the adapter into the base weights. At our scale of consumer image GenAI, LoRA is the workhorse — brand adapters for Photoleaf, style adapters for LightX, vertical adapters for enterprise creative engines. Cost: typically <$2K per adapter on A100s. Quality ceiling: high for style and surface behavior, low for genuinely new capabilities.

When LoRA falls down: when the domain shift is large (e.g. radiology DICOMs vs natural images), when you need new tokenization (Indic scripts, code), or when the behavior you want isn't representable in your data — only in your preferences.

Full supervised fine-tuning

Full SFT is the right move when LoRA's representational capacity is the bottleneck — typically when you have >100K high-quality examples and a domain shift that LoRA can't bridge. Cost scales with parameter count: a 7B SFT run is usually $15–40K in compute; a 70B run is $200K+. Quality ceiling: significantly higher than LoRA on domain-specific accuracy, but it does nothing for behavior.

When full SFT falls down: when the failure mode isn't "wrong answer" but "right answer, wrong tone" or "right answer, unsafe phrasing." SFT can teach a model what to say. It cannot reliably teach it what not to say.

RLHF / DPO / ORPO

This is where most teams under-invest. The shift from "the model knows the domain" to "the model behaves correctly in deployment" almost always requires post-training alignment. We've shipped RLHF pipelines using PPO and reward models for two BFSI clients; DPO for three media and creative clients (faster to iterate, no separate reward model); and ORPO for one healthcare client where the preference data was scarce and we needed to combine SFT and preference optimization in a single pass.

Cost: surprisingly often dominated by preference data collection, not compute. Plan for $20–80K in human annotation for a serious RLHF pipeline. Compute itself is usually <$30K for a 7B model.

When RLHF falls down: when you don't actually have a behavioral failure mode — you have an accuracy failure mode. RLHF won't fix "the model doesn't know our product catalog." SFT will.

The decision framework we use with clients

1. What's the failure mode? Wrong answer → SFT or pretraining. Wrong behavior → alignment.
2. What's your data shape? <10K examples → LoRA. 10K–100K → LoRA or SFT depending on domain shift. >100K → SFT. Preferences instead of correct answers → alignment.
3. What's your latency budget? Sub-50ms → likely need distillation after any of the above.
4. What's your inference scale? At >10M requests/day, the cost of compression and routing dominates everything else.

The common mistake: treating these as alternatives. In production, the right answer is usually all three in sequence — LoRA or SFT first, then alignment, then compression. The packages we sell are shaped around exactly that sequence.

— Sharad Shankar, Founder

The expensive failures don't happen at the start of a training run. They happen forty hours in, on a Saturday, when no one's watching the dashboard. Loss is still going down. Token efficiency looks normal. Then somewhere between checkpoint 18 and checkpoint 19, gradient norm spikes by 12x, and by the time anyone notices on Monday, you've written eighteen hours of garbage to disk and the next four checkpoints are unrecoverable.

This is the failure mode we built our mid-training observability practice to catch. Here's what we instrument and what we look at.

What we track (beyond loss)

Loss is the dashboard everyone has. It's also the dashboard that will lie to you for the longest. By the time loss starts looking wrong, the divergence has been building for hours. The leading indicators we actually watch:

• Gradient norm — sudden 5x+ spikes are the canonical divergence signal. Set alerts at 3x rolling baseline.
• Gradient norm variance across the batch — rising variance with stable mean is often a sign of a few pathological examples poisoning the run.
• Token efficiency (useful tokens / total tokens) — drops here usually mean your data loader is starting to feed degenerate sequences.
• Hardware utilization — sudden drops mean you're stalling on I/O, which often means your dataset shard is corrupt.
• Activation norms at chosen layers — for long runs, these drift in characteristic ways before loss does.

Intervention playbook

When gradient norm spikes:

1. Pause the run (not kill — pause from a checkpoint).
2. Inspect the most recent batch. 90% of the time it's bad data.
3. If data looks clean, drop LR by 10x and resume from the pre-spike checkpoint.
4. If that fails twice, the issue is architectural — likely a numerical stability problem you should have caught in the ablation phase.

When token efficiency drops:

1. Check the shard your loader is currently on.
2. Validate against your data quality checksum (you do have one).
3. Skip the shard if necessary — never silently downsample.

When hardware utilization tanks:

1. Check disk I/O. Then network. Then your gradient synchronization step.
2. NCCL hangs are the silent killer. Set explicit timeouts and dump rank-level state on timeout.

The thing no one tells you

The most expensive mistake in long training runs isn't the divergence itself. It's not having a clean rollback target. Every team we've worked with that lost meaningful compute did so because their checkpointing strategy was "save every N steps" without retention policy — and by the time they noticed the run was diverging, the last clean checkpoint had been overwritten.

Save more checkpoints than you think you need. Tier them — keep every 5th checkpoint indefinitely, every checkpoint for the last 24 hours. Storage is cheap. Re-running pretraining is not.

— AndOr training infrastructure team

A capable model is not a deployable model. The gap between "passing eval" and "running profitably at scale" is closed by post-training compression — and most teams get this wrong by trying to do it all at once, or by treating it as an afterthought.

Here's what we've learned shipping compressed models to consumer-scale LightX traffic and to two BFSI clients running models in their VPCs.

Distillation

The highest-impact lever, by a wide margin. A well-distilled student model at 1/4 the parameter count of its teacher will often retain 92–97% of the teacher's domain accuracy at 4x the throughput. We use distillation in nearly every production engagement.

Two failure modes we see constantly:

• Distilling from a teacher that hasn't been aligned. The student inherits all the misalignment, plus a smaller capacity to recover from it. Align first, distill second.
• Distilling on the wrong data distribution. Your distillation corpus should look like production traffic, not like your training data. This is non-obvious and routinely overlooked.

Quantization

INT8 is essentially free for most modern architectures — do it. INT4 is where things get interesting. Quantization-aware training (QAT) almost always beats post-training quantization (PTQ) at INT4, but costs an additional fine-tuning pass.

When PTQ is enough: short-context inference, models where you can tolerate 1–2% accuracy loss, deployments where the cost saving matters more than the marginal quality.

When QAT is worth it: latency-critical deployments, models in regulated domains where accuracy regressions need to be defensible, long-context workloads where PTQ quality degrades nonlinearly.

Speculative decoding

Cheapest latency win in the stack if your traffic pattern fits. We've seen 2–3x throughput improvements on autoregressive workloads with no accuracy loss. The draft model can often be a 2x smaller version of your production model — even a quantized version of it.

When it doesn't help: very short outputs (the speculation overhead dominates), batch sizes >32 (you're already throughput-bound, not latency-bound), and workloads with high output entropy (acceptance rates collapse).

The order that works

We almost always run these in this sequence:

1. Align the full-size model
2. Distill to target size
3. Quantize (QAT if you can afford the training pass, PTQ if not)
4. Layer speculative decoding on top if traffic pattern fits

The order matters. Quantizing before alignment is a common and expensive mistake — alignment doesn't transfer cleanly across precisions. Same for distilling before alignment.

The cost calculation teams get wrong

Compression isn't about per-token cost. It's about end-to-end serving cost — including the engineering hours to maintain the compressed pipeline, the eval harness needed to detect quality regressions, and the rollback infrastructure for when compression turns out to have broken something subtle.

Budget for the operational tail, not just the compute saving.

— AndOr inference engineering team