sf-app-platform/docs/videocall-juphoon-talking-issue.md

Videocall — Missing isTalking State Transition

Issue Summary

When the mobile app initiates a video call to a watch (RTOS or Android firmware), the call is answered on the watch and audio/video can be exchanged, but the Juphoon SDK on the mobile side never emits a callItemUpdateStream event with isTalking == true.

As a consequence the client-side state machine stays in outgoing (the UI keeps showing "Connecting…" / "Device ringing…") and never transitions to inCall. The _reportRoomCount notification (VIDEO_CALL_ROOM_COUNT_REQUEST) is therefore never sent, the local UI controls (mute, hang up, switch camera) stay in their pre-call state, and the call cannot be properly hung up by the user.

We need help from Juphoon to identify which side is failing to publish the "talking" state and what we need to send/configure for the SDK on the mobile to recognise the watch as connected.

---

Architecture Overview

The mobile side of the videocall feature is implemented in Flutter and wraps the Juphoon Android/iOS SDK through a dedicated videocall_sdk package.

modules/legacy/modules/device_management/lib/src/features/videocall/
├── domain/                       (entities, repositories interfaces)
├── data/                         (signaling datasource + repository impl)
├── providers/                    (Riverpod providers for signaling)
└── presentation/
    ├── videocall_screen.dart     (idle / outgoing / incoming / inCall / groupCall UI)
    ├── providers/
    │   ├── videocall_controller.dart   ← orchestrates SDK + signaling
    │   ├── videocall_state.dart
    │   ├── group_call_controller.dart  (group/conference mode)
    │   └── group_call_state.dart
    └── widgets/                  (UI for each call state)

packages/videocall_sdk/lib/src/
├── manager/videocall_sdk_manager.dart
├── services/
│   ├── videocall_client.dart           (login)
│   ├── videocall_call_service.dart     (start/answer/hangup, callItem*Stream)
│   ├── videocall_device_service.dart   (camera/mic/speaker)
│   ├── videocall_channel_service.dart  (group room)
│   ├── videocall_net_service.dart
│   ├── videocall_log_service.dart
│   └── videocall_push_service.dart
└── models/                             (CallParam, VideocallItem, CallDirection, MediaConfig, …)

---

Outgoing 1:1 Call — Expected Sequence

User taps the device card on the idle screen of the videocall feature → vm.startCall('') is invoked on the controller.

videocall_controller.dart:184 (startCall):

Step	Action	Code reference
1	Request runtime permissions for camera + microphone via permission_handler.	_requestMediaPermissions (line 167)
2	Resolve target Juphoon user ID. Defaults to w_${device.imei} if no explicit remoteUserId is provided.	line 191
3	Build chatType, roomNumber, sessionId. roomNumber = ${deviceId}_${sanitizedAppAccount} for 1:1, ${deviceId}_group for multi.	_buildRoomNumber (line 152), _buildSessionId (line 163)
4	Set screenMode = outgoing so the UI swaps to the "calling" view.	line 199
5	Send signaling to the watch via the backend: POST /commands with command: VIDEO_CALL_REQUEST, payload { chatType, appAccount, roomNumber, sessionId }. The backend forwards this as PRYVCALL to the watch firmware.	_signaling.initiateCall (line 206), see datasource videocall_signaling_datasource_impl.dart:11
6	Start local audio + camera on the SDK.	_deviceService.startAudio() / startCamera() (line 224-225)
7	Call _callService.startCall(userId, isVideo: true, callParam) to start the SIP/RTC session via the Juphoon SDK.	line 228

After step 7 the controller waits for two asynchronous signals:

• VideoCallRequestResponseEvent over WebSocket — the backend forwards a watch acknowledgement (OK or busy/refuse). When isOk == true the controller sets state.isDeviceRinging = true so the UI can show "Device ringing".
  Code: _onWebSocketEvent (line 525).

• callItemAddStream from the SDK — the SDK reports the outgoing call as a VideocallItem with direction == outgoing. The controller stores it in state.currentCall.
  Code: _onCallItemAdd (line 450).

• callItemUpdateStream from the SDK — once the remote party (watch) joins the call, the SDK should update the VideocallItem with isTalking == true, optionally with uploadVideoStreamSelf / uploadVideoStreamOther. The controller transitions to inCall and sends a VIDEO_CALL_ROOM_COUNT_REQUEST to inform the backend that the room now has 2 participants.
  Code: _onCallItemUpdate (line 466).

This last event is the one that never arrives in the buggy flow.

---

Watch Firmware Logs (Real Capture)

[3G*1106971865*0037*PRYVCALL,0,18606224072,1106971865_18606224072,123456789]
[3G*1106971865*000a*PRYVCALL,0]
[3G*1106971865*0028*UPRYROOMCOUNT,0,0,1106971865_18606224072]

• PRYVCALL,0,... — the watch received the request from the backend. The first arg is chatType (0 = single, 1 = multi), then appAccount, then roomNumber, then sessionId.
• PRYVCALL,0 (ack) — the watch replies acknowledging the request.
• UPRYROOMCOUNT,0,0,... — the watch updates the room count with count=0 (the watch is not in the room yet, or is reporting that nobody is connected).

Despite the watch beeping/ringing and the user pressing the answer button on the watch, the mobile-side SDK callback callItemUpdateStream never receives an event with isTalking == true.

---

Mobile SDK Configuration

videocall_controller.dart:118 (_configureDevice):

_deviceService.setDefaultSpeakerOn(true);
_deviceService.setCameraProperty(640, 360, 30);   // _cameraWidth, _cameraHeight, _cameraFps
_deviceService.setVideoAngle(0);
_callService.setMaxCallNum(1);
_callService.setTermWhenNetDisconnected(true);

final isRtos = device?.capabilities?.isRtos ?? false;
final mediaConfig = await VideocallCallService.generateMediaConfigByMode(
  isRtos
      ? MediaConfig.MODE_RTOS
      : MediaConfig.MODE_INTELLIGENT_HARDWARE,
);
_callService.updateMediaConfig(mediaConfig);

Login is performed in _autoLogin (line 93):

final userId = 'p_$sanitizedEmail';            // mobile user ID
await _client.login(userId: userId, password: user.id);

The watch is registered with Juphoon under another account (the watch firmware logs in by itself with its own credentials).

callParam for the outgoing call:

final callParam = CallParam(ticket: state.deviceId);
await _callService.startCall(
  userId: targetUserId,                           // 'w_<imei>'
  isVideo: true,
  callParam: callParam,
);

---

Commands Sent From the Mobile to the Watch (via backend)

Defined in videocall_signaling_datasource_impl.dart. They are sent as POST /commands to the SaveFamily backend, which forwards them to the watch as raw firmware commands.

Mobile event	Backend command	Payload	Watch firmware command
startCall	VIDEO_CALL_REQUEST	{chatType, appAccount, roomNumber, sessionId}	PRYVCALL,<chatType>,<appAccount>,<roomNumber>,<sessionId>
cancelCall, hangUp	VIDEO_CALL_CANCEL	{chatType}	(cancel)
rejectCall	VIDEO_CALL_REFUSE	{chatType, appAccount, roomNumber}	(refuse)
_reportRoomCount (after inCall)	VIDEO_CALL_ROOM_COUNT_REQUEST	{type, count, room_num}	UPRYROOMCOUNT,<type>,<count>,<roomNumber>

type in VIDEO_CALL_ROOM_COUNT_REQUEST is 0 for single and 1 for multi.

---

What We Have Verified

1. _callService.startCall(...) returns true (the SDK accepts the outgoing call).
2. callItemAddStream fires once with a VideocallItem whose direction == outgoing. We set it as state.currentCall.
3. The watch logs PRYVCALL,... immediately after we send VIDEO_CALL_REQUEST, so the watch receives the request.
4. The user can hear the watch ringing.
5. We confirm receiving VideoCallRequestResponseEvent via WebSocket with isOk: true — the backend says the watch acknowledged.
6. After the user answers on the watch, the SDK on the mobile does not emit any callItemUpdateStream event with isTalking == true.
7. No error is logged on the SDK side. Network is stable, login is loggedIn.

---

What We Need Help With (Asks for Juphoon)

1. Under which conditions is VideocallItem.isTalking == true published over callItemUpdateStream?
   We assume it is when both peers have joined the media session. Is there an additional handshake or media event that has to fire on the watch firmware before the mobile SDK considers the call as "talking"?

2. Does the watch firmware (RTOS and Android) need to call any specific SDK API after PRYVCALL is accepted by the user, in order for the mobile SDK to receive the talking event?
   The watch logs show PRYVCALL ack and UPRYROOMCOUNT,0,0,.... Is UPRYROOMCOUNT enough or are we missing a JCCallJoin / acceptCall equivalent on the watch side?

3. Should the mobile and watch use matching chatType?
   Today the mobile uses 0 = single for 1:1 calls regardless of the watch firmware. If the watch is Android (which according to the firmware vendor always operates in "room mode" / meeting_page), should the mobile send chatType = 1 (multi) and let both peers join the same room? Currently the watch logs PRYVCALL,0,... with a roomNumber of the form <deviceId>_<userPhone> even though the watch is Android — would that break the talking state propagation?

4. MediaConfig.MODE_RTOS vs MODE_INTELLIGENT_HARDWARE — we select the mode based on device.capabilities.system. Are these the correct mode mappings for the talking event to fire? Could the mode mismatch silently disable the update events?

5. UPRYROOMCOUNT returning count = 0 after the watch acknowledges PRYVCALL — is this expected before the watch joins the room, or does it mean the watch never actually joined the room from the SDK's perspective?

6. Reference to meeting_page — the firmware vendor mentioned that Android watches use meeting_page for any call. Is there a Juphoon-side equivalent we should be invoking from the mobile (e.g., joinChannel instead of startCall) when the peer is an Android watch?

---

Reproduction

• App build: apps/mobile_app flavour staging.
• Mobile: Android phone (Samsung, API 34+) signed in as testapps.savefamily@gmail.com.
• Watch: Android firmware, identificator 1106971865. The watch is logged into Juphoon as a separate account.
• The user opens the app → Device Management → Video call → taps the device card on the idle screen.

Steps:

1. Mobile sends VIDEO_CALL_REQUEST (visible in app logs).
2. Watch fires PRYVCALL,... and starts ringing.
3. User accepts on the watch.
4. Audio / video clearly flows (the watch shows the user's video preview, the user can hear the watch carrier speaking).
5. Mobile UI stays in outgoing ("Device ringing…") forever. _onCallItemUpdate is never triggered with isTalking == true.

---

Relevant Files

Feature

File	Purpose
modules/legacy/modules/device_management/lib/src/features/videocall/presentation/providers/videocall_controller.dart	Main controller. Holds the SDK lifecycle + signaling orchestration. The talking state transition is expected to happen in _onCallItemUpdate (line 466).
modules/legacy/modules/device_management/lib/src/features/videocall/presentation/providers/videocall_state.dart	Freezed state (screenMode, currentCall, isDeviceRinging, chatType, localUserId, remoteUserId, etc.).
modules/legacy/modules/device_management/lib/src/features/videocall/data/datasources/videocall_signaling_datasource_impl.dart	Sends the VIDEO_CALL_REQUEST, VIDEO_CALL_CANCEL, VIDEO_CALL_REFUSE, VIDEO_CALL_ROOM_COUNT_REQUEST commands to the backend.
modules/legacy/modules/device_management/lib/src/features/videocall/presentation/videocall_screen.dart	UI; reflects screenMode from the controller.

SDK Wrapper

File	Purpose
packages/videocall_sdk/lib/src/manager/videocall_sdk_manager.dart	Singleton orchestrator over the Juphoon SDK.
packages/videocall_sdk/lib/src/services/videocall_call_service.dart	Owns callItemAddStream, callItemUpdateStream, callItemRemoveStream, missedCallStream. The talking state should arrive via callItemUpdateStream.
packages/videocall_sdk/lib/src/services/videocall_client.dart	Login lifecycle.
packages/videocall_sdk/lib/src/services/videocall_device_service.dart	Camera/mic/speaker control.
packages/videocall_sdk/lib/src/services/videocall_channel_service.dart	Group/room mode (used in groupCall flow).
packages/videocall_sdk/lib/src/models/videocall_item.dart	Carries state, isTalking, uploadVideoStreamSelf, uploadVideoStreamOther.

Backend Side

Path	Purpose
packages/sf_infrastructure/lib/src/websocket/websocket_event.dart	VideoCallEvent, VideoCallRefusedEvent, VideoCallRoomCountEvent, VideoCallRequestResponseEvent.
packages/sf_infrastructure/lib/src/websocket/websocket_event_parser.dart	Parses incoming WebSocket frames into the events above.

---

Sample Mobile Logs During the Bug

[Videocall] build() called, scheduling _initSdk
[Videocall] _initSdk: isInitialized=true
[Videocall] _autoLogin: userId=p_testapps_savefamily_gmail_com, clientState=loggedIn
[Videocall] _autoLogin: already logged in, skipping login
… user taps device card …
[Videocall] _requestMediaPermissions: granted
… signaling.initiateCall succeeds (HTTP 201) …
[Videocall] WS video_call_request_response: ok
… (watch starts ringing, user accepts on watch) …
… (no further callItemUpdateStream events) …

Expected (but missing):

[Videocall] _onCallItemUpdate: state=<X>, isTalking=true, uploadSelf=true, uploadOther=true
[Videocall] screenMode -> inCall
[Videocall] _reportRoomCount → VIDEO_CALL_ROOM_COUNT_REQUEST {type:0, count:2, room_num:<…>}

---

Contact

Please reach out to the SaveFamily mobile team with any clarifications. We can provide additional logs from the Juphoon SDK if needed (the wrapper currently exposes videocall_log_service.dart but verbose logging is off by default; we can re-enable it on demand).