What we learned putting a local LLM on a phone line

Every managed voice AI platform bills by the minute. That sounds reasonable until you put one on a support line and it starts working, because the reward for a good agent is more calls answered, and the reward for more calls answered is a larger invoice. The better it gets, the more it costs. On a support line, which exists precisely to absorb volume, that is the wrong incentive to sign up to.

We built Vocale to answer a business's phone line: it picks up on the first ring, answers from the company's own manuals and policies, and opens a support ticket when the honest answer is that somebody needs to look at this. Early on we made a decision that shaped everything after it. The platform ships two engines behind one product. On the hosted tier, speech recognition, reasoning and voice synthesis come from a third party API and every minute is metered. On the self hosted tier, all three run on a GPU the business controls, no audio or document leaves the building, and a call costs electricity.

Both tiers behave identically from the caller's side. Getting them there was the interesting part, and this is what we learned.

What is actually in the loop

A voice agent is not a chatbot with a microphone bolted on. One conversational turn is a chain, and every link adds delay a human can hear.

Audio arrives over WebRTC into a LiveKit room. Voice activity detection (we use Silero) decides when the caller has actually started and stopped speaking. The buffered audio goes to Faster-Whisper and comes back as a transcript. That turn is written to Postgres, then the chat context and history go to the language model, which on our self hosted tier is Qwen3-8B served by vLLM. Tokens stream back, get normalised and aggregated into whole sentences, and only then reach XTTS-v2, which renders them as audio frames that travel back out over WebRTC.

Four models, one database write, and a round trip, all inside the pause a caller will tolerate before they say "hello?" into the silence.

Running that chain locally is not the hard part. Running it locally at conversational speed, on hardware a mid sized business would actually buy, is where the engineering lives.

Lesson one: interruption is a product feature, not an edge case

Callers talk over the agent. They do it constantly, and they do it the moment they have heard enough, which is usually about six words into a twenty word answer. An agent that keeps talking sounds broken in a way that is difficult to recover from, because the human instinct is to talk louder, and now two parties are speaking at once into a system that is listening to neither.

Cutting the outgoing audio is the obvious half. The half that is easy to miss is the transcript. If the agent's own record still contains the full sentence it intended to say, the conversation continues from a call that never happened. Ask a follow up question about "the second option you mentioned" and the agent will answer confidently about something the caller never heard.

So when speech is detected, we stop the audio and then truncate the agent's stored turn to what was actually spoken aloud. The conversation continues from the call the caller experienced, not the one the system had planned.

Lesson two: you cannot remove the latency, so you have to fill it

When a question needs a fact, the agent searches the indexed documents before it answers. That search takes long enough to read as a dropped line. Silence on a phone call is not neutral. It is alarming.

We fire a short, natural filler about half a second in, and cancel it if retrieval returns first. It is a small piece of behaviour with an outsized effect: it buys retrieval the time it needs, and none of that time lands on the caller as dead air.

This matters more on the self hosted tier, not less. A local model on a single GPU has different latency characteristics from a hyperscaler's API, and they are not uniformly worse. They are differently distributed. Designing the conversation to tolerate variable latency meant we could change where the models run without changing how the agent feels.

Lesson three: written text does not read aloud

Money amounts, dates as digits, times, acronyms, product codes. All of it is fine on a screen and wrong in a voice. Hand a speech model "1,250.00 by 15/03" and you will hear something no person has ever said out loud.

We put a normalisation layer between the language model and the speech model that rewrites amounts, dates, times and abbreviations into their spoken form, per language, before they reach synthesis. Vocale runs in six languages, and this layer is per language because the rules genuinely differ. There is no shortcut where you write it once for English and translate.

This is the least glamorous component in the system and one of the highest impact on whether the agent sounds like a person or a phone tree.

Lesson four: VRAM is the ceiling, not compute

Here is the one that surprised us most, and the one that most changes how you size hardware.

A live voice session holds its models in video memory for the duration of the call. Concurrency is therefore bounded by VRAM, not by raw throughput. You can have a card with compute to spare that cannot take another call, because there is nowhere to put the models.

Worse, if you let the framework pool GPUs by default, speech and language work land on the same card and starve each other. The symptom is not an error. It is a conversation that gets slower under load in a way that no single component owns. We now budget sessions per card and pin the workload across cards explicitly.

If you are planning a self hosted deployment, count concurrent calls against video memory first. Everything else follows from that number.

What does not change

Grounding does not change. On both tiers the agent answers from the business's uploaded documents rather than from the model's memory: a question that needs a fact triggers a search, and the reply is assembled from what came back. When nothing relevant is found, saying so and offering to open a ticket is the correct answer, and the one it gives. A smaller local model is not a licence to hallucinate more, because the model is not the source of truth in the first place. Retrieval is.

Handover does not change either. When the agent cannot help, it collects the subject, the contact and the priority while the caller is still on the line, files the ticket and attaches the transcript.

That is the point of building both tiers against one behaviour. Moving between them is configuration, not a rebuild.

So which one should you run?

Honestly: start hosted.

The hosted tier is the fastest way to find out whether an agent on your line is useful at all, and that is a product question, not an infrastructure one. Our own console metered it at about six cents per call minute. For a business taking a few dozen calls a day, that is not the line item worth optimising first.

Self hosting earns its keep on two thresholds, and they are different thresholds.

The first is volume. Metered cost scales with adoption, hardware cost does not, so there is a crossover point where the GPU is simply cheaper. Where that point sits depends on your call volume, and you should measure it rather than assume it. Vocale meters spend per model and per organisation for exactly this reason: the choice between an API and a card should be a decision with a number attached, not an argument.

The second threshold has nothing to do with money. Some businesses cannot send call audio, transcripts or internal documents to a third party, because of a regulator, a client contract or a policy. For them the local tier is not an optimisation. It is the only version of this product that is allowed to exist, and no per minute price makes the hosted tier acceptable.

Those two thresholds are worth separating when you evaluate any voice AI vendor. A platform that only offers metered inference has quietly decided the second one does not apply to you.

The short version

Local inference on a support line is practical today. The models are good enough, and the tooling (vLLM, Faster-Whisper, XTTS-v2, LiveKit) is mature enough that the pipeline is not the risk.

The risks are the ones nobody demos: being interrupted, sounding present while you think, reading numbers out loud like a human, and knowing how many conversations actually fit on a card. Get those right and the choice between a hosted API and your own GPU becomes what it should have been all along, which is a line in a budget rather than a limit on what you are allowed to build.

• Local LLM
• Voice AI
• Self-hosted
• RAG
• vLLM
• LiveKit
• Support automation