A real-time voice intelligence API for AI voice agents.
VSIP sits between the microphone and an AI agent, receives raw audio streams, and returns structured events describing conversational turns, interruptions, speaker identity, verification decisions, and non-speech sounds.
Public showcase repository: This repository documents the product, public APIs, architecture, capabilities, integrations, and launch status of VSIP. The production API server, dashboard source code, infrastructure, proprietary processing logic, and internal services remain private.
- What VSIP Is
- The Problem It Solves
- How VSIP Fits Into a Voice Agent
- Public Product Capabilities
- Event Types
- Integration Flow
- SDKs and Adapters
- Product Suite
- Security and Privacy
- Public Performance Targets
- Pricing
- Product Status
- Ownership and My Role
- Private Source Disclosure
- Repository Scope
- Suggested Repository Structure
- Roadmap
- Website and Social Links
- Copyright
VoiceStream Intelligence Platform (VSIP) is a real-time voice intelligence API for AI voice agents.
It sits between the microphone and the existing AI-agent stack.
VSIP streams raw audio in and returns structured events describing:
- Who is speaking
- When a speech turn starts
- When a user has completed a thought
- Whether the user interrupted the agent
- Whether the current speaker matches the intended user
- Whether the detected sound is speech, noise, laughter, coughing, music, television, keyboard activity, or another non-speech event
- Whether the downstream agent should respond
VSIP is designed as a voice-events and identity layer. Developers continue using their preferred:
- Speech-to-text provider
- Large language model
- Text-to-speech provider
- Voice-agent framework
- Application logic
Voice agents often fail in real conversations because spoken interaction is more complex than sending text to an LLM.
Common problems include:
- The agent answers before the user has finished speaking
- A cough is treated as an interruption
- Background voices activate the agent
- Television or music opens a speech turn
- The agent talks over the user
- Another nearby speaker is treated as the intended user
- Fixed silence thresholds close turns during natural pauses
- The agent stops speaking but fails to process the interruption as a real user turn
VSIP gives voice agents more reliable conversational awareness.
It helps answer questions such as:
Who is speaking?
Has the user completed their thought?
Did the intended user interrupt the agent?
Was the detected sound actually speech?
Should the downstream AI agent respond?
The goal is to give voice agents reliable “ears” before speech reaches the reasoning layer.
A typical integration looks like this:
Microphone
│
▼
VSIP WebSocket
│
├── turn_start
├── turn_end
├── barge_in
├── speaker_verification
├── speaker_change
├── noise_event
└── lock_status
│
▼
Speech-to-Text
│
▼
Large Language Model
│
▼
Text-to-Speech
│
▼
Agent Response
VSIP does not replace the STT, LLM, or TTS layer.
It adds structured voice intelligence before the downstream agent decides how to respond.
VSIP determines whether a speaker has completed a thought rather than relying only on a fixed silence timeout.
This helps avoid prematurely ending a turn during:
- Natural pauses
- Hesitation
- Filler words
- Mid-sentence thinking
- Slow speech
- Multi-part questions
VSIP detects when the intended speaker interrupts the agent.
The goal is to prevent events such as:
- Coughing
- Television audio
- Background conversations
- Nearby colleagues
- Agent echo
- Random environmental sounds
from incorrectly stopping the agent.
VSIP can produce a speaker-verification result for a completed turn.
The result may include:
- Speaker identity
- Similarity information
- Verification decision
should_respondverdict
Applications can gate the downstream STT or LLM workflow using:
speaker_verification.should_respond
VSIP identifies non-speech events that should not be treated as user requests.
Examples include:
- Coughing
- Breathing
- Laughter
- Television
- Music
- Keyboard sounds
- Ambient noise
- Animal sounds
The first spoken interaction can be used both as:
- The user’s first actual request
- An in-session speaker enrollment sample
This removes the need for a separate onboarding phrase in supported workflows.
VSIP can support persistent voice profiles for returning users.
Profiles may be used for:
- Enrollment
- Verification
- Identification
- Retention control
- Reuse across sessions
VSIP can identify when a different person takes over the conversation.
This allows the application to:
- Pause responses
- Request verification
- Ignore unverified speakers
- Update the session state
VSIP is designed to handle scenarios where multiple speakers talk at the same time.
Supported workflows may isolate the enrolled speaker’s audio and provide per-speaker information, subject to audio quality and implementation constraints.
VSIP also supports offline audio processing.
Batch analysis can generate:
- Speech turns
- Speaker labels
- Start and end timestamps
- Talk-time ratios
- Interruption locations
- Silence segments
- Longest monologue
- Noise events
- Conversation statistics
The public Voice Events API includes the following event types.
Indicates that a speech turn has opened.
Possible fields include:
- Speaker identifier
- Audio start point
- Session context
Indicates that the speaker has completed a speech turn.
Possible fields include:
- Turn duration
- Endpoint decision
- Speaker identifier
- Timing information
Indicates that the verified user interrupted the agent while the agent was speaking.
Applications should normally stop TTS immediately after receiving this event.
Returns the speaker identity decision associated with a turn.
Important fields may include:
- Verification status
- Similarity information
should_respond
Signals that a different person has started speaking.
Reports a classified non-speech event.
Examples include:
- Cough
- Laughter
- Music
- Television
- Keyboard
- Ambient noise
Reports whether the current session has locked onto a primary speaker identity.
Connect to the VSIP streaming endpoint:
wss://api.vsip.online/v1/stream
One WebSocket connection is used per voice session.
Stream supported audio frames to VSIP.
A typical voice-agent pipeline may use:
16 kHz
Mono
PCM
20 ms audio frames
Supported formats and configuration may vary by integration.
VSIP returns events such as:
{
"event": "turn_start",
"speaker_id": "speaker_1"
}{
"event": "barge_in",
"speaker_id": "speaker_1"
}{
"event": "speaker_verification",
"verified": true,
"should_respond": true
}The examples above are illustrative. The live API documentation should remain the source of truth for the current schema.
Before the agent’s TTS begins:
{
"command": "set_agent_state",
"value": "speaking"
}After TTS finishes or is stopped:
{
"command": "set_agent_state",
"value": "idle"
}When a barge_in event arrives:
- Stop TTS playback
- Set the agent state to
idle - Preserve the interruption as a valid user turn
- Continue the downstream processing workflow
Before sending a user turn to the LLM, check:
speaker_verification.should_respond
This helps prevent responses to unverified or unintended speakers.
VSIP provides SDKs for Python and JavaScript/TypeScript.
pip install vsip-sdkExample import:
import vsipnpm install vsip-sdkThe SDK layer is designed to support:
- Connection management
- Automatic reconnect
- Session resume
- Typed event handling
- Audio lookback
- Turn-start recovery
- Queued turns
- Response gating
- Verification-aware callbacks
- Unknown-field tolerance
The Pipecat integration uses:
VSIPProcessor
It can sit between speech recognition and the LLM so that unverified speakers’ transcriptions do not reach the reasoning layer.
The LiveKit integration uses:
VSIPVAD
It can act as a custom voice activity and turn-detection component inside a LiveKit Agent session.
The Twilio integration uses:
TwilioBridge
It supports telephony audio workflows and interruption handling.
VSIP can run beside a speech-to-speech agent as a sidecar.
In sidecar mode, the existing platform continues handling:
- Speech recognition
- Reasoning
- Speech generation
VSIP adds:
- Speaker identity
- Verification decisions
- Barge-in detection
- Noise intelligence
- Response gating
VSIP currently includes four primary product areas.
A real-time WebSocket API that converts raw audio into structured conversational events.
Developer SDKs and framework integrations for Python, JavaScript, Pipecat, LiveKit, Twilio, and speech-to-speech agents.
A REST API for:
- Speaker enrollment
- One-to-one verification
- One-to-many identification
- Profile deletion
- Consent tracking
- Retention settings
- Reuse across voice sessions
An asynchronous API for processing recorded conversations and returning structured conversation timelines and statistics.
VSIP is designed to process audio frame by frame in memory.
The public product claim is that raw streaming audio is:
- Not written to disk
- Not retained as a recording
- Discarded after processing
VSIP does not provide speech-to-text as its core service.
Because it does not run the transcription layer, it does not need to retain conversation transcripts as part of the Voice Events API workflow.
Depending on the product and account configuration, VSIP may store:
- Voice embeddings
- Speaker labels
- Consent flags
- Retention settings
- Account information
- Hashed passwords
- Hashed API keys
- Billing and usage metadata
- Session identifiers
- Batch-analysis result timelines
Batch-analysis timelines may be retained for up to 30 days.
The source audio is not intended to be retained after processing.
VSIP uses tenant-scoped resources.
Tenant isolation applies to:
- API keys
- Speaker profiles
- Batch jobs
- Usage records
- Account data
API and WebSocket traffic use encrypted transport.
API keys are designed to be:
- Stored in hashed form
- Displayed in full only when created
- Revocable through account controls
- Sent through authorization headers rather than query strings
Voice profiles may include:
- Biometric voice embeddings
- User-provided labels
- Consent indicators
- Retention windows
Developers remain responsible for obtaining appropriate consent and meeting applicable legal requirements.
VSIP is not currently presented as SOC 2 or HIPAA certified.
Customers should not assume that using VSIP alone makes an application legally compliant.
Speaker similarity is not equivalent to liveness detection.
A replayed or synthetically generated voice may require separate anti-spoofing and liveness controls in security-sensitive workflows.
The following figures are public product targets and should not be interpreted as independently verified third-party benchmarks.
Approximately 250–450 ms
Approximately 300 ms p50
Approximately 700 ms p95
Approximately 250 ms p50
Approximately 450 ms p95
The public target is to enable a verified downstream response in under one second after the user stops speaking.
Actual performance can vary depending on:
- Network latency
- Audio quality
- Hardware
- Selected configuration
- Noise suppression
- Echo cancellation
- Downstream STT
- Downstream LLM
- Downstream TTS
Pricing is currently being finalized.
The public showcase repository will be updated when the official pricing structure is confirmed.
For current commercial information, visit:
https://vsip.online
VSIP is being prepared as a live developer platform.
The following product areas have been built, tested, and merged:
- Voice Events API
- SDKs and Adapters
- Speaker Profile API
- Batch Audio Analytics
Remaining launch and operational work includes:
- Production deployment validation
- Stripe integration
- Transactional email
- OAuth configuration
- Legal review
- Demo API-key rotation
- SDK publishing
- Load testing
- Backup verification
- Monitoring and operational readiness
- Documentation review
- Pricing confirmation
The platform should be described as:
A live developer platform with ongoing launch, infrastructure, and commercial readiness work.
VSIP is a product of The Three Tier.
The Three Tier holds:
- Product ownership
- Technical ownership
- Commercial ownership
My role includes:
- Product ownership
- Founder-level product direction
- Product vision
- Public positioning
- Launch readiness
- Product documentation
- Feature and claim validation
- Technical coordination
- Commercial readiness
- Brand and developer positioning
- Final review of public product claims
Additional engineering and operational responsibilities may be documented as the product and team structure evolve.
VSIP’s production source code is proprietary and confidential.
The following are not included in this repository:
- API server source code
- Dashboard source code
- Internal services
- Infrastructure configuration
- Deployment configuration
- Database schemas
- Authentication logic
- Billing implementation
- Detection algorithms
- Speaker-processing logic
- Model orchestration
- Internal monitoring
- Proprietary event-processing logic
- Private SDK implementation
- Security-sensitive configuration
VSIP exposes public interfaces through:
- APIs
- SDKs
- Documentation
- Product pages
- Pricing
- Demonstrations
- Public integration guides
The product should not be presented as open source unless it is explicitly relicensed by The Three Tier.
This public showcase repository may contain:
README.md
assets/
docs/
examples/
Approved materials may include:
- Public screenshots
- Product diagrams
- High-level architecture
- Public API examples
- Event examples
- Integration diagrams
- Security documentation
- Product roadmap
- Public performance targets
This repository must not contain:
- Production source code
- Private package files
- Docker files
- Environment files
- API keys
- Customer data
- Production logs
- Private endpoints
- Database credentials
- Internal model configuration
- Unpublished pricing
- Proprietary algorithms
voice-stream-intelligence-platform/
├── README.md
├── assets/
│ ├── vsip-cover.png
│ ├── product-dashboard.png
│ ├── live-session-console.png
│ ├── integration-flow.png
│ └── demo-thumbnail.png
├── docs/
│ ├── product-overview.md
│ ├── architecture.md
│ ├── event-model.md
│ ├── security.md
│ └── roadmap.md
└── examples/
└── example-events.json
Planned operational and product work includes:
- Production deployment validation
- Stripe billing integration
- Transactional email
- OAuth authentication
- SDK publishing
- Public package verification
- Load and concurrency testing
- Automated backups
- Monitoring and alerting
- Legal and privacy review
- API-key lifecycle improvements
- Developer onboarding improvements
- Demo environment hardening
- Public status monitoring
- Expanded integration guides
- Additional SDK examples
- Improved observability
- Enterprise readiness
- Security certification planning
LinkedIn page: TO_BE_ADDED
Website: TO_BE_ADDED
Substack: TO_BE_ADDED
X / Twitter: TO_BE_ADDED
LinkedIn product updates: TO_BE_ADDED
Documentation: TO_BE_ADDED
Muhammad Hamdan Rauf on GitHub
LinkedIn: TO_BE_ADDED
Copyright © 2026 The Three Tier. All rights reserved.
VoiceStream Intelligence Platform, VSIP, associated product documentation, screenshots, event definitions, architecture descriptions, branding, and related assets may not be copied, redistributed, sublicensed, reverse engineered, or used commercially without written permission from The Three Tier.