Model Inferencing at Scale: How TinyAI Achieves 182ms Latency

Why Latency Is Everything in Voice AI

In a text based chatbot, 2 seconds of latency is acceptable. In a voice conversation, it's a dealbreaker. Humans perceive conversational pauses longer than 300ms as awkward. Anything above 500ms and the caller starts wondering if the line dropped.

Our target for model inferencing in voice AI: under 200ms end to end, measured from the moment the caller finishes speaking to the moment they hear the agent's first word. That 200ms budget covers:

• ASR (speech to text): ~40 to 60ms
• LLM inference (the "thinking"): ~60 to 80ms
• Tool calls (API lookups, CRM queries): ~20 to 40ms
• TTS (text to speech): ~30 to 50ms
• Network and audio buffering: ~10 to 20ms

Every millisecond matters. Here's how we hit our targets.

The Inference Stack

Layer 1: Model Architecture and Quantization

The single biggest lever in AI model inference optimization is model size. A 70B parameter model, even with aggressive optimization, will never hit 80ms inference on affordable hardware. That's why we start with a 3B parameter base model and fine tune it on customer data.

But even a 3B model needs optimization for real time voice:

• GPTQ 4 bit quantization: Reduces model memory footprint by 75% with less than 1% accuracy loss on domain specific tasks. A 3B model goes from ~6GB to ~1.5GB, fitting entirely in GPU VRAM.
• KV cache optimization: We use paged attention (inspired by vLLM) to efficiently manage the key value cache across concurrent conversations, reducing memory fragmentation.
• Speculative decoding: A tiny draft model (300M params) generates candidate tokens in parallel, which the main model verifies in a single forward pass. This gives us 2x to 3x faster token generation with zero quality loss.

The result: our fine tuned 3B TLM generates a full response (~40 to 60 tokens for a typical voice turn) in 60 to 80ms on a single consumer GPU.

Layer 2: Serving Infrastructure

The model is just one piece. The serving layer around it determines whether that raw inference speed translates to latency the user actually perceives.

Our model serving stack:

• Continuous batching: Instead of waiting for a batch to fill, we add new requests to the running batch on every iteration. This keeps GPU utilization above 85% while maintaining under 100ms queue time.
• Streaming inference: We don't wait for the full response to generate. Tokens stream to the TTS engine as they're produced, so speech synthesis starts while the model is still generating.
• Warm model pools: Customer specific models are preloaded in GPU memory during business hours. No cold starts. Model swapping takes ~200ms, which we do during call silence periods.
• Async tool execution: When the model decides to call a tool (e.g., look up an order), the tool call fires asynchronously. If the model is generating a preamble like "Let me check that for you...", the API call runs in parallel.

Layer 3: ASR Optimization

Automatic Speech Recognition is the first bottleneck in voice AI inference. The caller's audio needs to be transcribed to text before the LLM can process it.

Our approach:

• Streaming ASR: We don't wait for the caller to finish their entire sentence. Partial transcripts stream to the LLM in real time, so the model can begin "thinking" before the caller stops speaking.
• Endpointing: A lightweight voice activity detector (VAD) determines when the caller has finished speaking. We tuned this aggressively for Indian conversational patterns, including shorter pauses, more filler words, and frequent switching between languages.
• Fine tuned Whisper: We run a distilled, fine tuned Whisper model optimized for Indian English and Hindi. Runs in ~40ms for a typical 3 to 5 second utterance on a T4 GPU.

Layer 4: TTS Streaming

The last mile: converting the model's text response into natural sounding speech. Traditional TTS generates the entire audio file, then plays it. We don't have that luxury.

• Chunk based synthesis: As tokens stream from the LLM, we synthesize audio in small chunks of about 50 to 100ms. The first audio chunk plays while subsequent chunks are still being generated.
• Voice cloning per customer: Each deployment has a custom voice profile tuned for the brand. Warm, professional, and natural, not robotic.
• Prosody matching: Our TTS model adjusts tone, pace, and emphasis based on the content. Questions sound like questions. Confirmations sound confident.

The Full Pipeline: A Latency Breakdown

Here's an actual production trace from a logistics client deployment, a typical "where is my order?" call:

*Tool call runs in parallel with preamble generation, so its latency is mostly hidden.

P95 across all calls: 182ms. P50: 141ms. Faster than most human agents pick up the phone.

Cost Optimization: The Other Side of Inference

AI inference cost is the hidden killer of voice AI economics. Here's how the math works at scale:

• A 70B model on an A100 GPU: ~$2.50/hour, serving ~15 concurrent calls
• Our 3B quantized model on a T4 GPU: ~$0.30/hour, serving ~40 concurrent calls

That's a 22x cost difference per conversation. And because our fine tuned 3B model matches or exceeds the 70B model's accuracy on domain specific tasks, there's no quality tradeoff.

The key insight: model inferencing cost scales with model size, not with model quality on specific tasks. A small model that is carefully fine tuned is both cheaper AND better than a generic large model for defined use cases.

GPU Fleet Management

We run a mixed GPU fleet to optimize cost further:

• T4 GPUs for steady state inference: best cost/performance ratio for our quantized 3B models
• L4 GPUs for burst capacity: spun up during peak hours, spot instances for noncritical overflow
• CPU inference for lightweight models (VAD, intent classification, entity extraction): no GPU needed for these

Autoscaling based on call queue depth, not GPU utilization. If calls are waiting, we scale up. If GPUs are idle, we scale down. Response time: 30 seconds from scale trigger to first inference.

Edge Inference: The Next Frontier

Cloud inference works for most use cases, but some scenarios demand edge AI inference:

• Zero latency environments: Factory floors where even 20ms of network latency is too much
• Air gapped deployments: Government and defense clients with no cloud connectivity
• Bandwidth constrained locations: Rural healthcare centers with intermittent connectivity
• Data sovereignty: Scenarios where raw audio cannot leave the premises

We're actively deploying our quantized models on NVIDIA Jetson Orin devices for edge inference. The 3B model runs at ~120ms inference on Jetson Orin NX, fast enough for real time voice, entirely offline.

What We've Learned

After running model inferencing at scale for over a year, our key takeaways:

1. Quantization is free lunch: 4 bit quantization on fine tuned models loses less than 1% accuracy on domain tasks. There's no reason not to do it.
2. Streaming everything: Don't wait for any stage to complete before starting the next. Pipeline parallelism is the single biggest latency win.
3. Fine tuning beats scale: A 3B model with 6 months of customer data beats a 70B model with zero customer data. Every time, on every metric.
4. Latency budgets are real: Set a p95 target and measure every stage. You can't optimize what you don't measure.
5. Inference cost is the real moat: Anyone can fine tune a model. Running it at scale, profitably, at low latency, that's the hard part.

"The goal isn't to run the biggest model. It's to run the right model, fast enough and cheap enough that every business can afford it."

Related Posts

• How TinyAI Is Disrupting Voice AI in India with the Lowest Prices in the Industry: The founder story and business case behind our affordable voice AI platform.
• AI Image Analytics: How Computer Vision Is Transforming Business in 2026: Applying the same small model philosophy to computer vision and image analytics.