Communicate with Video SDK

If your business requires not only high-quality audio and video transmission, but also easy integration, low memory usage, and low power consumption, for applications such as smart cameras and smart doorbells, Agora recommends that you use the Video SDK with the IoT SDK.

The following table shows the data formats supported when the IoT SDK communicates with the Video SDK (v4.x):

Video SDK Type	Supported formats
Native/third-party frameworks: Android iOS/macOS Windows Electron Unity Flutter React Native	Audio: G722, G711, Opus, AAC Video: H.264, JPEG
Web (v4.x)	Audio: G722, G711, Opus Video: H.264
Embedded systems	Audio: G711, Opus Video: H.264, JPEG

The sending and receiving ends can communicate with each other only if the set data format is consistent and complies with the above table.

The following sections describe how to set the audio and video data formats in the SDKs.

Audio

IoT SDK only supports mono audio data, and the data sending interval is 20 ms.

To send and receive AAC audio data, set audio_frame_info_t.data_type to AUDIO_DATA_TYPE_AACLC or AUDIO_DATA_TYPE_HEAAC and specify the corresponding data size when calling the agora_rtc_send_audio_data method. The size of AAC audio data needs to be resolved by the user.

Video

In IoT SDK, H.264 video data is sent and received by default.

To send and receive JPEG video data, when calling the agora_rtc_send_video_data method, set video_frame_info_t.data_type to VIDEO_DATA_TYPE_GENERIC_JPEG.

Audio

In Native/third-party framework SDKs (v4.x), Opus or AAC audio data is sent and received by default.

To send and receive G722 or G711 audio data, use the following methods:

Video SDK type	G722	G711 (PCMA)	G711 (PCMU)
Android	agoraEngine.setAudioProfile(6) or agoraEngine.setParameters("{\"che.audio. custom_payload_type\":9}")	agoraEngine.setParameters("{\"che.audio. custom_payload_type\":8}")	agoraEngine.setParameters("{\"che.audio. custom_payload_type\":0}")
iOS/macOS	agoraKit.setAudioProfile(6) or agoraKit.setParameters("{\"che.audio. custom_payload_type\":9}")	agoraKit.setParameters("{\"che.audio. custom_payload_type\":8}")	agoraKit.setParameters("{\"che.audio. custom_payload_type\":0}")
Windows	agoraEngine->setAudioProfile(6) or agora::base::AParameter apm(agoraEngine); apm->setParameters("{\"che.audio. custom_payload_type\":9}")	agora::base::AParameter apm(agoraEngine);apm->agoraEngine->setParameters("{\"che.audio. custom_payload_type\":8}")	agora::base::AParameter apm(agoraEngine);apm->agoraEngine->setParameters("{\"che.audio. custom_payload_type\":0}")
Electron	this.agoraEngine?.setAudioProfile(6) or this.agoraEngine?.setParameters("{\"che.audio. custom_payload_type\":9}")	this.agoraEngine?.setParameters("{\"rtc.audio. custom_payload_type\":8}")	this.agoraEngine?.setParameters("{\"rtc.audio. custom_payload_type\":0}")
Unity	await agoraEngine.setAudioProfile(profile: AudioProfileType.audioProfileIot) or await agoraEngine.setParameters("{\"che.audio. custom_payload_type\":9}")	await agoraEngine.setParameters("{\"che.audio. custom_payload_type\":8}")	await agoraEngine.setParameters("{\"che.audio. custom_payload_type\":0}")
Flutter	await agoraEngine.setAudioProfile(profile: AudioProfileType.audioProfileIot) or await agoraEngine.setParameters("{\"che.audio. custom_payload_type\":9}")	await agoraEngine.setParameters("{\"che.audio. custom_payload_type\":8}")	await agoraEngine.setParameters("{\"che.audio. custom_payload_type\":0}")
React Native	await agoraEngine.current?.setAudioProfile(6) or await agoraEngine.current?.setParameters("{\"che.audio. custom_payload_type\":9}")	await agoraEngine.current?.setParameters("{\"che.audio. custom_payload_type\":8}")	await agoraEngine.current?.setParameters("{\"che.audio. custom_payload_type\":0}")

Video

In Native/third-party framework SDKs (v4.x), H.264 video data is sent and received by default.

To send and receive JPEG video data, use the following methods:

Video SDK type	Method
Android	agoraEngine.setParameters("{\"engine.video.codec_type\": \"20\"}")
iOS/macOS	agoraKit.setParameters("{\"engine.video.codec_type\": \"20\"}")
Windows	agora::base::AParameter apm(agoraEngine); apm->setParameters("{\"engine.video.codec_type\": \"20\"}")
Electron	this.agoraEngine.setParameters("{\"engine.video.codec_type\": \"20\"}");
Unity	agoraEngine.SetParameters("{\"engine.video.codec_type\": \"20\"}");
Flutter	await agoraEngine.setParameters("{\"engine.video.codec_type\": \"20\"}");
React Native	this.engine.setParameters("{\"engine.video.codec_type\": \"20\"}");

Audio

In Video SDK for Web (v4.x), Opus audio data is sent and received by default.

To send and receive G722, G711 audio data, call the following method:

• G711 (PCMA): AgoraRTC.createClient({mode: "live", codec: "h264", audioCodec: "pcma"})
• G711 (PCMU): AgoraRTC.createClient({mode: "live", codec: "h264", audioCodec: "pcmu"})
• G722: AgoraRTC.createClient({mode: "live", codec: "h264", audioCodec: "g722"})

Only the Web SDK version 4.9.0 or higher supports setting the audio encoding format.

Video

In Web SDK (v4.x), VP8 video data is sent and received by default. You call createClient to set the video data format to "h264".

In Video SDK for embedded systems, Opus audio data and H.264 video data are sent and received by default. If you want to receive JPEG video data or send and receive G711 audio data, contact sales-us@agora.io for activation.

Video SDK for embedded systems does not support sending JPEG video data.