How the Voice AI Works

This page explains what the voice AI actually does on a call, and what it can and can't do. Read it before writing instructions or actions for phone calls: most surprises on calls follow from the points below. For how to put them into practice, see Voice AI best practices.

The voice AI is the same kind of AI as the chat bot, described in the Bot Operating Model, with speech added on both ends.

A call is a chain of three steps

1. Transcription. The caller's speech is turned into text as they talk.
2. The AI. When the caller finishes speaking, the AI reads the transcript of the call so far, may run actions, and writes a reply as text.
3. Speech. The reply is turned into speech with the voice set in the voice AI settings.

It is not a model that listens to audio. The AI only ever sees text, so everything depends on what the transcription produced:

• Transcripts of phone audio are imperfect. Names, email addresses, order numbers and spelled-out codes often come through wrong. When a sentence doesn't make sense, the AI infers the likely meaning or asks the caller to repeat. It doesn't repeat back a name the caller volunteers, because names are often transcribed wrongly.
• Tone of voice doesn't reach the AI. It sees the caller's words, not how they were said.
• Transcription keywords help with words the transcription wouldn't expect, such as brand and product names.

When the AI speaks

The AI never speaks on its own initiative. It speaks only when something happens:

Two consequences:

• The AI can't stay silent. Every time the caller finishes speaking, it replies. Instructions like "say nothing until the caller asks a question" don't work.
• It can't act later. It can't call back, wait for something to happen, or speak up after a pause unless the caller stays silent long enough for a check-in.

Interruptions

Callers can interrupt the AI while it speaks, as in a normal conversation. To keep background noise and "mm-hm" from cutting it off, by default:

• In the first 2 seconds of a reply, any speech from the caller stops the AI.
• After that, the caller has to say at least 10 characters (a few words) to stop it.

Both are settings. The caller can't interrupt the greeting, the short line before an action, the wait for an action's result, or the closing line when the call is ended or transferred.

When the AI is interrupted, it remembers only the part of its reply the caller actually heard.

Actions on a call

The AI uses the same actions as on other channels. On a call:

• The AI announces the first action of a turn. Unless it's already speaking, the AI says a short line it writes itself before the action runs, such as "Let me check that for you." Then it waits in silence for the result.
• Actions run one at a time, never in parallel, and at most eight in one turn. Each one adds silence for the caller.
• The caller can't interrupt while an action runs.
• An action that fails gives the AI its error message. One that takes longer than a minute, or can't be reached, gives it only "There was a system issue."
• An action can end or transfer the call. If the AI hasn't said anything yet, it says a short closing line first.

During a call, the conversation in Octocom contains only the greeting. Actions that need what was said so far read it from context["liveCallContext"].

The voice is fixed for the call

The voice and the background office noise are settings. The AI only writes text, so instructions like "speak slower" or "speak more quietly" don't change how it sounds.

Each voice AI speaks one language, set in its settings. The transcription expects that language too.

Settings are read when the call starts

The AI reads its settings, instructions, workflows and actions once, when the call connects. Changes apply to the next call, never to a call in progress.

What the AI knows on an inbound call

On a call to one of your phone lines, the AI gets:

• General Tone and Business Description from Settings -> Automation -> Configuration. The other fields there, such as writing styles and output syntax, don't apply to calls: calls have their own built-in instructions for spoken replies.
• Bot rules that apply to the Phone Call channel.
• Live workflows available on the Phone Call channel, with their actions.
• Knowledge base search, when the business has active knowledge.
• The caller's phone number, unless shareCallerNumberWithAi is turned off.
• The current date and time in UTC. This differs from chat, where the bot gets your business time zone. Give times with their time zone in your instructions.

It can always end the call. It can transfer the call when a workflow enables transferConversation, as on other channels, and the line has transfer destinations. Without destinations, the same action hands the conversation to your team for a follow-up instead; see Transfers.

Campaign calls work differently: the campaign's instructions replace everything above.

After the call

• The transcript reaches the help desk when the call ends, together with the recording. See Phone calls in the help desk.
• A call the AI handed off, or transferred to your team, stays open for your team. Other calls are closed.
• Data collection runs on the transcript after the call, and so does sentiment analysis if your organization has it enabled. Data collection prompts only run on calls when they include the Phone Call channel.