Ensemble/docs/remote-connection/WEBRTC_DISCONNECT_FIXES.md

WebRTC Data Channel Disconnect Fixes

Document Purpose

Track all fixes attempted for the WebRTC data channel disconnection issues in Ensemble's remote connection feature. This document should be referenced when debugging future disconnection problems.

---

Problem Summary

The WebRTC data channel between the Ensemble mobile app and Music Assistant server can fail in several ways:

1. One-way failure: Data channel appears "open" but only sends work, receives stop
2. Silent disconnect: No error events fire, channel just stops working
3. ICE failure: ICE connection fails, properly detected but causes disconnect
4. Signaling disconnect: Signaling WebSocket closes, causing ICE/WebRTC to fail

---

Key Files

File	Purpose
lib/services/remote/webrtc_connection.dart	WebRTC peer connection, data channel management
lib/services/remote/remote_bridge.dart	WebSocket bridge, health monitoring, reconnection logic
lib/services/remote/signaling_client.dart	Signaling server WebSocket connection

---

Implemented Fixes (Chronological)

Fix 1: Data Channel Keepalive (Original)

Problem: Data channel could silently die without any error events firing.

Solution: Added periodic keepalive ping/pong on the data channel itself.

Code (webrtc_connection.dart):

• _keepaliveTimer: Runs every 10 seconds
• _checkKeepalive(): Checks time since last data received
• _sendKeepalivePing(): Sends JSON ping message
• _recordDataReceived(): Called on every message receipt

Constants:

static const _keepaliveInterval = Duration(seconds: 10);
static const _keepaliveTimeout = Duration(seconds: 20);  // Reduced from 30
static const _maxKeepaliveMisses = 2;  // Reduced from 3

Fix 2: ICE Recovery

Problem: ICE connection could temporarily disconnect and might auto-recover.

Solution: Added ICE recovery timer before declaring failure.

Code (webrtc_connection.dart):

• _startIceRecovery(): Starts 5-second timer when ICE disconnects
• _cancelIceRecovery(): Cancels if ICE recovers on its own
• _attemptIceRestart(): Calls restartIce() if recovery timer expires

Fix 3: Flow Control (Buffer Overflow Prevention)

Problem: Fire-and-forget sends could overflow the 16MB SCTP buffer, causing silent channel death.

Solution: Added send-side flow control with buffering.

Code (webrtc_connection.dart):

static const _maxBufferedAmount = 1048576;  // 1MB threshold
static const _bufferLowThreshold = 65536;   // 64KB resume threshold
bool _sendPaused = false;
final _sendQueue = <String>[];

Modified sendRaw():

• Checks bufferedAmount before sending
• Queues messages if buffer > 1MB
• onBufferedAmountLow callback drains queue

Fix 4: Health Check Stale Detection

Problem: Health check could see "30s since last data" but not trigger reconnection.

Solution: Reduced stale detection threshold from 45s to 30s.

Code (remote_bridge.dart):

if (_state == RemoteBridgeState.connected && timeSinceLastData > 30 && !_isReconnecting) {
  // Force reconnection
}

Fix 5: Reduced Keepalive Timeouts

Problem: 30s timeout × 3 misses = 90s minimum before detection.

Solution: Reduced to 20s timeout × 2 misses = ~40s detection.

Fix 6: Recv Stagnation Detection

Problem: One-way failure where recv stops but sent continues not detected quickly enough.

Solution: Track recv count across health checks. If recv stuck while sent increases for 2 consecutive checks (20s), force reconnection.

Code (remote_bridge.dart):

// Track across health checks
int _lastHealthCheckRecvCount = 0;
int _lastHealthCheckSentCount = 0;
int _recvStagnationCount = 0;

// In health check:
if (recvStuck && sentIncreased) {
  _recvStagnationCount++;
  if (_recvStagnationCount >= 2) {
    // Force reconnection - channel is one-way
  }
}

Detection time: ~20 seconds (2 × 10s health check interval)

---

Log Patterns to Watch For

Healthy Connection

Health check - API sent:93 recv:88 ... lastRecv:0s ago

• sent and recv both increasing
• lastRecv low (< 5s typically)

One-Way Failure (CRITICAL)

Health check - API sent:206 recv:163 ... lastRecv:30s ago
Health check - API sent:218 recv:163 ... lastRecv:40s ago

• sent increases, recv STUCK
• lastRecv keeps increasing
• Data channel still shows "open"

Signaling Disconnect

WebRTC: Signaling disconnected
... (6-16 seconds later) ...
WebRTC: ICE connection state: RTCIceConnectionStateDisconnected
WebRTC: ICE connection state: RTCIceConnectionStateFailed

Network Failure (DNS)

Signaling: WebSocket closed
... (reconnection attempt) ...
Signaling: Failed to connect: SocketException: Failed host lookup: 'signaling.music-assistant.io'

This indicates the phone lost network connectivity entirely.

Reconnection Storm (Signaling Instability)

Signaling: Connecting to wss://signaling.music-assistant.io/ws
Signaling: Sending connect-request
Signaling: WebSocket closed  ← Closes within 100-500ms
WebRTC: Signaling disconnected
RemoteBridge: Scheduling reconnect attempt X/10 in Ys

Rapid reconnection loop - signaling never stays connected long enough.

Flow Control Engaged

WebRTC: ⚠️ Buffer full (1234567 bytes) or paused - queueing message
WebRTC: Buffer drained to 12345 bytes - draining 5 queued messages
WebRTC: ✅ Send queue drained (5 messages) - resuming normal sends

Keepalive Detection

WebRTC: ⚠️ Keepalive timeout! No data for 22s (missed: 1/2)
WebRTC: ❌ Data channel appears dead - triggering reconnection

---

Known Issues / Limitations

1. flutter_webrtc Limitations

• RTCDataChannel.state may show "open" when actually dead (Issue #843)
• bufferedAmount may not be accurate on all platforms
• onBufferedAmountLow callback support varies

2. SCTP Receive Window

• If server sends faster than client processes, receive window fills
• This causes server-side to stop sending, client sees no data
• No way to monitor receive-side buffer from client

3. TURN Server Issues

• aiortc (server-side) has known issues with TURN + DataChannel (#1156)
• Data channel may fail silently when using TURN relay

4. Network Transitions

• WiFi → Cellular or vice versa can cause ICE to fail
• App backgrounding on mobile can close WebSocket/WebRTC

---

Root Cause Categories

Category A: Network Issues (Not Code Bugs)

These are environmental issues, not bugs in our code:

• DNS failure: Failed host lookup - Phone lost internet
• WiFi/Cellular switch: Network changed, connections dropped
• Server unreachable: Signaling server down or blocked
• Firewall/NAT issues: TURN relay needed but unavailable

Evidence: Signaling WebSocket closes, DNS lookups fail, reconnection attempts fail repeatedly.

Category B: WebRTC/SCTP Issues (Partial Code Control)

These can be mitigated but not fully prevented:

• One-way channel death: Channel appears open but data stops flowing
• Buffer overflow: Too much data sent, SCTP buffer exceeded
• ICE failure: Network path lost, no recovery possible

Evidence: Data channel shows "open" but recv stuck, lastRecv increasing.

Category C: Code Bugs (Fixable)

Issues in our implementation:

• Detection too slow: Timeouts too long, misses too many
• Recovery not triggered: State checks wrong, reconnection not started
• Resource leaks: Timers not cancelled, connections not cleaned up

Evidence: Logs show problem but no action taken, or wrong action taken.

---

Debugging Checklist

When investigating a disconnect:

1. Find the last successful response:

   grep "WebRTC API → WS" logs.txt | head -5
   

2. Check health check progression:

   grep "Health check" logs.txt
   

   Look for recv getting stuck while sent increases.

3. Check for signaling issues:

   grep -i "signaling" logs.txt
   

4. Check ICE state changes:

   grep "ICE connection state" logs.txt
   

5. Check for keepalive detection:

   grep -i "keepalive" logs.txt
   

6. Check buffer status:

   grep -E "(buffer|queue|paused)" logs.txt
   

---

Potential Future Improvements

1. Server-Side Backpressure

MA protocol doesn't have explicit backpressure. Could add:

• Client-side request throttling
• Application-level ACKs for critical messages

2. More Aggressive Detection

• Track recv count across health checks, detect stagnation
• Shorter keepalive interval (5s instead of 10s)

3. Proactive Health Signals

• Monitor bufferedAmount trend (increasing = problem)
• Track round-trip latency of keepalive pings

4. Connection Quality Metrics

• Expose packet loss, jitter from WebRTC stats
• Use to predict imminent failure

---

Test Scenarios

Scenario	How to Test	Expected Behavior
Normal usage	Browse library, play music	No warnings
Heavy sync	Pull-to-refresh library	Buffer may queue, should drain
Long session	Leave connected 2+ hours	Should stay connected
Rapid navigation	Quick tab switching	Brief queue usage, recovers
Network change	WiFi → Cellular	Should detect and reconnect
App background	Background for 5+ min	May disconnect, should reconnect

---

Observed Failure Patterns

Pattern 1: One-Way Death After Library Sync

Observed: 2026-01-23 (multiple times)

17:13:06.175 - Last response (playlists sync)
17:13:06.264 - Local batch save
-- 5 seconds silence --
17:13:11.005 - Send players/all → NO RESPONSE
17:13:13.914 - ICE state: completed (looks healthy!)
17:13:28 - Keepalive detects timeout
17:13:38 - Reconnection triggered

Characteristics:

• Often happens after library sync (large batch responses)
• ICE shows "completed" even though data channel is dead
• Signaling ping/pong continues working
• Only data channel receive is affected

Hypothesis:

• Server-side SCTP buffer exhausted from sending large responses?
• Client receive window full, server blocks?
• flutter_webrtc onMessage callback stops firing?

Pattern 2: Reconnection Storm / Signaling Instability

Observed: 2026-01-23 17:30-17:31

17:31:05.093 - Signaling: Connecting
17:31:05.216 - Signaling: Sending connect-request
17:31:05.218 - Signaling: WebSocket closed (123ms later!)
17:31:05.218 - WebRTC: Signaling disconnected

Characteristics:

• Signaling WebSocket closes almost immediately after connect
• Multiple rapid reconnection attempts (storm)
• Connection never stable long enough for data flow
• Often happens after network instability
• Artist detail, albums etc don't load because connection keeps failing

Possible Causes:

• Network instability (WiFi/cellular issues)
• Signaling server rate limiting
• DNS resolution failures
• Server rejecting rapid reconnection attempts

Side Effects:

• UI shows errors like "Not connected to Music Assistant server"
• Artist/album detail screens fail to load
• Players list empty or stale

Pattern 3: API Channel Dies While Sendspin Lives

Observed: 2026-01-23 17:53

17:53:35.755 - Send album_tracks request → NO RESPONSE
17:53:40.596 - Health check: recv stuck at 197, sent increased
17:53:55.204 - Sendspin channel RECEIVES server/time response!
17:54:02.790 - Keepalive detects dead API channel

Characteristics:

• API data channel (ma-api) stops receiving
• Sendspin data channel continues working normally
• Same WebRTC peer connection, different data channels
• This proves it's NOT a network issue - Sendspin traffic flows fine

Hypothesis:

• flutter_webrtc bug where onMessage callback stops firing for one channel
• Possible memory/GC issue with the callback
• Server-side might be sending to wrong channel?

Key Evidence:

• Recv count stays at 197 for API
• Sendspin recv count increases (7→8)
• Both channels use same peer connection

---

Version History

Date	Build	Changes
2026-01-23	~v49	Initial flow control implementation
2026-01-23	~v49	Reduced keepalive timeout 30→20s, misses 3→2
2026-01-23	~v49	Reduced health check stale threshold 45→30s
2026-01-23	~v49	Fixed excessive "queue drained" logging
2026-01-23	~v49	Created WEBRTC_DISCONNECT_FIXES.md documentation
2026-01-23	~v49	Added recv stagnation detection (20s detection)
2026-01-23	~v49	Documented Pattern 2: Reconnection storm / signaling instability
2026-01-23	~v49	Fixed race condition in _scheduleReconnect (multiple simultaneous calls)
2026-01-23	~v49	Added WebSocket close code/reason logging to signaling
2026-01-23	~v49	CRITICAL FIX: _reconnectWebRTC() was returning immediately due to _isReconnecting already being true
2026-01-23	~v49	Fixed old WebRTC callbacks firing during reconnection cleanup (clear callbacks before disconnect)
2026-01-23	~v49	Fixed onStateChanged handler calling _handleWebRTCDisconnection during active reconnection

---

References

• flutter_webrtc Issue #843: Data channel state unreliable
• aiortc Issue #1156: TURN + DataChannel bugs
• WebRTC SCTP: Max buffer is 16MB, recommended threshold 1MB
• Music Assistant remote protocol: No explicit flow control