Until recently, building real-time voice AI for production was challenging. Developers faced hurdles such as managing audio streams and ensuring low-latency performance.

A Practical Application: The AI Sales Coach

Session Configuration

When the connection to Voice Live API is opened, the backend constructs a session configuration message. This sets up modalities, voice, avatar, and audio settings.

def _build_session_config(self) -> Dict[str, Any]:
    """Build the base session configuration."""
    return {
        "type": "session.update",
        "session": {
            "modalities": ["text", "audio"],
            "turn_detection": {"type": "azure_semantic_vad"},
            "input_audio_noise_reduction": {"type": "azure_deep_noise_suppression"},
            "input_audio_echo_cancellation": {"type": "server_echo_cancellation"},
            "avatar": {
                "character": "lisa",
                "style": "casual-sitting",
            },
            "voice": {
                "name": config["azure_voice_name"],
                "type": config["azure_voice_type"],
            },
        },
    }

This message is sent as the first step in establishing the conversation.

• modalities define if both text and audio are active.

• turn detection specifies how the system decides when the speaker has finished.

• noise reduction and echo cancellation improve audio quality.

• avatar and voice personalize the agent’s presence.

Adding Local Agent Behavior

The behavior of the agent is not controlled by session settings alone. Instead, the backend injects the model, instructions, temperature, and max tokens into the session.

def _add_local_agent_config(self, config_message: Dict[str, Any], agent_config: Dict[str, Any]) -> None:
    """Add local agent configuration to session config."""
    session = config_message["session"]
    session["model"] = agent_config.get("model", config["model_deployment_name"])
    session["instructions"] = agent_config["instructions"]
    session["temperature"] = agent_config["temperature"]
    session["max_response_output_tokens"] = agent_config["max_tokens"]

With this, we can dynamically swap out models (e.g., GPT-4o vs GPT-5), adjust creativity with temperature, and add prompt instructions that define how the AI should behave.

The Instructions

The instructions enforce conversational realism, ensuring the AI responds like a real person in a role-play setting.

BASE_INSTRUCTIONS = """
CRITICAL INTERACTION GUIDELINES:
- Keep responses SHORT and conversational (3 sentences max, as if speaking on phone)
- ALWAYS stay in character, never break role
- Simulate natural human speech patterns with pauses, um, well, occasional hesitation
- Respond as a real person would in this business context
- Use natural phone conversation style, direct but personable
- Show genuine emotions and reactions appropriate to the situation
- Ask follow-up questions to keep the conversation flowing naturally
- Avoid overly formal or robotic language
"""

These are appended to the role-play scenario instructions to guide the AI toward natural, in-character responses.

Feedback System

The Manifestation of AI

The Technical Architecture

Final Thoughts