sf-app-platform/docs/videocall-juphoon-talking-issue.md

# Videocall — Missing `isTalking` State Transition

## Issue Summary

When the **mobile app initiates a video call to a watch** (RTOS or Android firmware), the call is answered on the watch and audio/video can be exchanged, but **the Juphoon SDK on the mobile side never emits a `callItemUpdateStream` event with `isTalking == true`**.

As a consequence the client-side state machine stays in `outgoing` (the UI keeps showing "Connecting…" / "Device ringing…") and never transitions to `inCall`. The `_reportRoomCount` notification (`VIDEO_CALL_ROOM_COUNT_REQUEST`) is therefore never sent, the local UI controls (mute, hang up, switch camera) stay in their pre-call state, and the call cannot be properly hung up by the user.

We need help from Juphoon to identify which side is failing to publish the "talking" state and what we need to send/configure for the SDK on the mobile to recognise the watch as connected.

---

## Architecture Overview

The mobile side of the videocall feature is implemented in Flutter and wraps the Juphoon Android/iOS SDK through a dedicated `videocall_sdk` package.

```
modules/legacy/modules/device_management/lib/src/features/videocall/
├── domain/                       (entities, repositories interfaces)
├── data/                         (signaling datasource + repository impl)
├── providers/                    (Riverpod providers for signaling)
└── presentation/
    ├── videocall_screen.dart     (idle / outgoing / incoming / inCall / groupCall UI)
    ├── providers/
    │   ├── videocall_controller.dart   ← orchestrates SDK + signaling
    │   ├── videocall_state.dart
    │   ├── group_call_controller.dart  (group/conference mode)
    │   └── group_call_state.dart
    └── widgets/                  (UI for each call state)

packages/videocall_sdk/lib/src/
├── manager/videocall_sdk_manager.dart
├── services/
│   ├── videocall_client.dart           (login)
│   ├── videocall_call_service.dart     (start/answer/hangup, callItem*Stream)
│   ├── videocall_device_service.dart   (camera/mic/speaker)
│   ├── videocall_channel_service.dart  (group room)
│   ├── videocall_net_service.dart
│   ├── videocall_log_service.dart
│   └── videocall_push_service.dart
└── models/                             (CallParam, VideocallItem, CallDirection, MediaConfig, …)
```

---

## Outgoing 1:1 Call — Expected Sequence

User taps the device card on the idle screen of the videocall feature → `vm.startCall('')` is invoked on the controller.

`videocall_controller.dart:184` (`startCall`):

| Step | Action | Code reference |
|------|--------|----------------|
| 1 | Request runtime permissions for camera + microphone via `permission_handler`. | `_requestMediaPermissions` (line 167) |
| 2 | Resolve target Juphoon user ID. Defaults to `w_${device.imei}` if no explicit `remoteUserId` is provided. | line 191 |
| 3 | Build `chatType`, `roomNumber`, `sessionId`. `roomNumber = ${deviceId}_${sanitizedAppAccount}` for 1:1, `${deviceId}_group` for multi. | `_buildRoomNumber` (line 152), `_buildSessionId` (line 163) |
| 4 | Set `screenMode = outgoing` so the UI swaps to the "calling" view. | line 199 |
| 5 | Send signaling to the watch via the backend: `POST /commands` with `command: VIDEO_CALL_REQUEST`, payload `{ chatType, appAccount, roomNumber, sessionId }`. The backend forwards this as `PRYVCALL` to the watch firmware. | `_signaling.initiateCall` (line 206), see datasource `videocall_signaling_datasource_impl.dart:11` |
| 6 | Start local audio + camera on the SDK. | `_deviceService.startAudio()` / `startCamera()` (line 224-225) |
| 7 | Call `_callService.startCall(userId, isVideo: true, callParam)` to start the SIP/RTC session via the Juphoon SDK. | line 228 |

After step 7 the controller waits for two asynchronous signals:

* **`VideoCallRequestResponseEvent` over WebSocket** — the backend forwards a watch acknowledgement (`OK` or busy/refuse). When `isOk == true` the controller sets `state.isDeviceRinging = true` so the UI can show "Device ringing".  
  Code: `_onWebSocketEvent` (line 525).

* **`callItemAddStream` from the SDK** — the SDK reports the outgoing call as a `VideocallItem` with `direction == outgoing`. The controller stores it in `state.currentCall`.  
  Code: `_onCallItemAdd` (line 450).

* **`callItemUpdateStream` from the SDK** — once the remote party (watch) joins the call, the SDK should update the `VideocallItem` with `isTalking == true`, optionally with `uploadVideoStreamSelf` / `uploadVideoStreamOther`. The controller transitions to `inCall` and sends a `VIDEO_CALL_ROOM_COUNT_REQUEST` to inform the backend that the room now has 2 participants.  
  Code: `_onCallItemUpdate` (line 466).

**This last event is the one that never arrives in the buggy flow.**

---

## Watch Firmware Logs (Real Capture)

```
[3G*1106971865*0037*PRYVCALL,0,18606224072,1106971865_18606224072,123456789]
[3G*1106971865*000a*PRYVCALL,0]
[3G*1106971865*0028*UPRYROOMCOUNT,0,0,1106971865_18606224072]
```

* `PRYVCALL,0,...` — the watch received the request from the backend. The first arg is `chatType` (`0 = single`, `1 = multi`), then `appAccount`, then `roomNumber`, then `sessionId`.
* `PRYVCALL,0` (ack) — the watch replies acknowledging the request.
* `UPRYROOMCOUNT,0,0,...` — the watch updates the room count with `count=0` (the watch is **not** in the room yet, or is reporting that nobody is connected).

Despite the watch beeping/ringing and the user pressing the answer button on the watch, the mobile-side SDK callback `callItemUpdateStream` never receives an event with `isTalking == true`.

---

## Mobile SDK Configuration

`videocall_controller.dart:118` (`_configureDevice`):

```dart
_deviceService.setDefaultSpeakerOn(true);
_deviceService.setCameraProperty(640, 360, 30);   // _cameraWidth, _cameraHeight, _cameraFps
_deviceService.setVideoAngle(0);
_callService.setMaxCallNum(1);
_callService.setTermWhenNetDisconnected(true);

final isRtos = device?.capabilities?.isRtos ?? false;
final mediaConfig = await VideocallCallService.generateMediaConfigByMode(
  isRtos
      ? MediaConfig.MODE_RTOS
      : MediaConfig.MODE_INTELLIGENT_HARDWARE,
);
_callService.updateMediaConfig(mediaConfig);
```

Login is performed in `_autoLogin` (line 93):

```dart
final userId = 'p_$sanitizedEmail';            // mobile user ID
await _client.login(userId: userId, password: user.id);
```

The watch is registered with Juphoon under another account (the watch firmware logs in by itself with its own credentials).

`callParam` for the outgoing call:

```dart
final callParam = CallParam(ticket: state.deviceId);
await _callService.startCall(
  userId: targetUserId,                           // 'w_<imei>'
  isVideo: true,
  callParam: callParam,
);
```

---

## Commands Sent From the Mobile to the Watch (via backend)

Defined in `videocall_signaling_datasource_impl.dart`. They are sent as `POST /commands` to the SaveFamily backend, which forwards them to the watch as raw firmware commands.

| Mobile event | Backend command | Payload | Watch firmware command |
|--------------|-----------------|---------|------------------------|
| `startCall` | `VIDEO_CALL_REQUEST` | `{chatType, appAccount, roomNumber, sessionId}` | `PRYVCALL,<chatType>,<appAccount>,<roomNumber>,<sessionId>` |
| `cancelCall`, `hangUp` | `VIDEO_CALL_CANCEL` | `{chatType}` | (cancel) |
| `rejectCall` | `VIDEO_CALL_REFUSE` | `{chatType, appAccount, roomNumber}` | (refuse) |
| `_reportRoomCount` (after `inCall`) | `VIDEO_CALL_ROOM_COUNT_REQUEST` | `{type, count, room_num}` | `UPRYROOMCOUNT,<type>,<count>,<roomNumber>` |

`type` in `VIDEO_CALL_ROOM_COUNT_REQUEST` is `0` for `single` and `1` for `multi`.

---

## What We Have Verified

1. `_callService.startCall(...)` returns `true` (the SDK accepts the outgoing call).
2. `callItemAddStream` fires once with a `VideocallItem` whose `direction == outgoing`. We set it as `state.currentCall`.
3. The watch logs `PRYVCALL,...` immediately after we send `VIDEO_CALL_REQUEST`, so the watch receives the request.
4. The user can hear the watch ringing.
5. We confirm receiving `VideoCallRequestResponseEvent` via WebSocket with `isOk: true` — the backend says the watch acknowledged.
6. After the user answers on the watch, the SDK on the mobile **does not** emit any `callItemUpdateStream` event with `isTalking == true`.
7. No error is logged on the SDK side. Network is stable, login is `loggedIn`.

---

## What We Need Help With (Asks for Juphoon)

1. **Under which conditions is `VideocallItem.isTalking == true` published over `callItemUpdateStream`?**  
   We assume it is when both peers have joined the media session. Is there an additional handshake or media event that has to fire on the watch firmware before the mobile SDK considers the call as "talking"?

2. **Does the watch firmware (RTOS and Android) need to call any specific SDK API after `PRYVCALL` is accepted by the user, in order for the mobile SDK to receive the talking event?**  
   The watch logs show `PRYVCALL` ack and `UPRYROOMCOUNT,0,0,...`. Is `UPRYROOMCOUNT` enough or are we missing a `JCCallJoin` / `acceptCall` equivalent on the watch side?

3. **Should the mobile and watch use matching `chatType`?**  
   Today the mobile uses `0 = single` for 1:1 calls regardless of the watch firmware. If the watch is Android (which according to the firmware vendor always operates in "room mode" / `meeting_page`), should the mobile send `chatType = 1` (multi) and let both peers join the same room? Currently the watch logs `PRYVCALL,0,...` with a `roomNumber` of the form `<deviceId>_<userPhone>` even though the watch is Android — would that break the talking state propagation?

4. **`MediaConfig.MODE_RTOS` vs `MODE_INTELLIGENT_HARDWARE`** — we select the mode based on `device.capabilities.system`. Are these the correct mode mappings for the talking event to fire? Could the mode mismatch silently disable the update events?

5. **`UPRYROOMCOUNT` returning `count = 0`** after the watch acknowledges `PRYVCALL` — is this expected before the watch joins the room, or does it mean the watch never actually joined the room from the SDK's perspective?

6. **Reference to `meeting_page`** — the firmware vendor mentioned that Android watches use `meeting_page` for any call. Is there a Juphoon-side equivalent we should be invoking from the mobile (e.g., `joinChannel` instead of `startCall`) when the peer is an Android watch?

---

## Reproduction

* App build: `apps/mobile_app` flavour `staging`.
* Mobile: Android phone (Samsung, API 34+) signed in as `testapps.savefamily@gmail.com`.
* Watch: Android firmware, identificator `1106971865`. The watch is logged into Juphoon as a separate account.
* The user opens the app → `Device Management` → `Video call` → taps the device card on the idle screen.

Steps:

1. Mobile sends `VIDEO_CALL_REQUEST` (visible in app logs).
2. Watch fires `PRYVCALL,...` and starts ringing.
3. User accepts on the watch.
4. Audio / video clearly flows (the watch shows the user's video preview, the user can hear the watch carrier speaking).
5. Mobile UI stays in `outgoing` ("Device ringing…") forever. `_onCallItemUpdate` is never triggered with `isTalking == true`.

---

## Relevant Files

### Feature

| File | Purpose |
|------|---------|
| `modules/legacy/modules/device_management/lib/src/features/videocall/presentation/providers/videocall_controller.dart` | Main controller. Holds the SDK lifecycle + signaling orchestration. The talking state transition is expected to happen in `_onCallItemUpdate` (line 466). |
| `modules/legacy/modules/device_management/lib/src/features/videocall/presentation/providers/videocall_state.dart` | Freezed state (`screenMode`, `currentCall`, `isDeviceRinging`, `chatType`, `localUserId`, `remoteUserId`, etc.). |
| `modules/legacy/modules/device_management/lib/src/features/videocall/data/datasources/videocall_signaling_datasource_impl.dart` | Sends the `VIDEO_CALL_REQUEST`, `VIDEO_CALL_CANCEL`, `VIDEO_CALL_REFUSE`, `VIDEO_CALL_ROOM_COUNT_REQUEST` commands to the backend. |
| `modules/legacy/modules/device_management/lib/src/features/videocall/presentation/videocall_screen.dart` | UI; reflects `screenMode` from the controller. |

### SDK Wrapper

| File | Purpose |
|------|---------|
| `packages/videocall_sdk/lib/src/manager/videocall_sdk_manager.dart` | Singleton orchestrator over the Juphoon SDK. |
| `packages/videocall_sdk/lib/src/services/videocall_call_service.dart` | Owns `callItemAddStream`, `callItemUpdateStream`, `callItemRemoveStream`, `missedCallStream`. The talking state should arrive via `callItemUpdateStream`. |
| `packages/videocall_sdk/lib/src/services/videocall_client.dart` | Login lifecycle. |
| `packages/videocall_sdk/lib/src/services/videocall_device_service.dart` | Camera/mic/speaker control. |
| `packages/videocall_sdk/lib/src/services/videocall_channel_service.dart` | Group/room mode (used in `groupCall` flow). |
| `packages/videocall_sdk/lib/src/models/videocall_item.dart` | Carries `state`, `isTalking`, `uploadVideoStreamSelf`, `uploadVideoStreamOther`. |

### Backend Side

| Path | Purpose |
|------|---------|
| `packages/sf_infrastructure/lib/src/websocket/websocket_event.dart` | `VideoCallEvent`, `VideoCallRefusedEvent`, `VideoCallRoomCountEvent`, `VideoCallRequestResponseEvent`. |
| `packages/sf_infrastructure/lib/src/websocket/websocket_event_parser.dart` | Parses incoming WebSocket frames into the events above. |

---

## Sample Mobile Logs During the Bug

```
[Videocall] build() called, scheduling _initSdk
[Videocall] _initSdk: isInitialized=true
[Videocall] _autoLogin: userId=p_testapps_savefamily_gmail_com, clientState=loggedIn
[Videocall] _autoLogin: already logged in, skipping login
… user taps device card …
[Videocall] _requestMediaPermissions: granted
… signaling.initiateCall succeeds (HTTP 201) …
[Videocall] WS video_call_request_response: ok
… (watch starts ringing, user accepts on watch) …
… (no further callItemUpdateStream events) …
```

Expected (but missing):

```
[Videocall] _onCallItemUpdate: state=<X>, isTalking=true, uploadSelf=true, uploadOther=true
[Videocall] screenMode -> inCall
[Videocall] _reportRoomCount → VIDEO_CALL_ROOM_COUNT_REQUEST {type:0, count:2, room_num:<…>}
```

---

## Contact

Please reach out to the SaveFamily mobile team with any clarifications. We can provide additional logs from the Juphoon SDK if needed (the wrapper currently exposes `videocall_log_service.dart` but verbose logging is off by default; we can re-enable it on demand).