Customer satisfaction for your IoT SDK integrated app depends on the quality of video and audio it provides. Quality of audiovisual communication through your app is affected by the following factors:
Bandwidth of network connection: Bandwidth is the volume of information that an Internet connection can handle per unit of time. When the available bandwidth is not sufficient to transmit the amount of data necessary to provide the desired video quality, your users see jerky or frozen video along with audio that cuts in and out.
Stability of network connection: Network connections are often unstable with the network quality going up and down. Users get temporarily disconnected and come back online after an interruption. These issues lead to a poor audiovisual experience for your users unless your app is configured to respond to these situations and take remedial actions.
Hardware quality: The camera and microphone used to capture video and audio must be of sufficiently good quality. If the user's hardware does not capture the audiovisual information in suitably high definition, it limits the quality of audio and video that is available to the remote user.
Video and audio settings: The sharpness, smoothness, and overall quality of the video is directly linked to the frame rate, bitrate and other video settings. Similarly, the audio quality depends on the sample rate, bitrate, number of channels and other audio parameters. If you do not choose proper settings, the audio and video transmitted are of poor quality. On the other hand, if the settings are too demanding, the available bandwidth quickly gets choked, leading to suboptimal experience for your users.
Echo: Echo is produced when your audio signal is played by a remote user through a speakerphone or an external device. This audio is captured by the remote user's microphone and sent back to you. Echo negatively affects audio quality, making speech difficult to understand.
Multiple users in a channel: When multiple users engage in real-time audio and video communication in a channel, the available bandwidth is quickly used up due to several incoming audio and video streams. The device performance also deteriorates due to the excessive workload required to decode and render multiple video streams.
This page shows you how to use IoT SDK features to account for these factors and ensure optimal audio and video quality in your app.
To provide the best audio and video quality in your app:
Choose an audio codec to optimize bit rate and quality
In audio and video streaming, the choice of an encoding algorithm affects both quality and bit rate. Your goal is to lower the required bit rate while maintaining quality at the desired level. IoT SDK offers the following built-in audio codecs:
- G711A (PCMU)
- G711U (PCMU)
You specify an audio codec when you join a channel. Depending on your audio quality requirements you also set the sampling rate and the number of channels. If your app needs a custom encoding algorithm, you disable use of a built-in codec and implement your own encoder.
Adjust sending bit rate in real time
In order to optimize data transmission and avoid network congestion, best practice is to adjust the sending bit rate in real-time according to changes in network conditions.
You configure BandWidth Estimation (BWE) before joining a channel to set the minimum, maximum, and starting bit rate values according to the actual bandwidth and bit rate needs. When the network bandwidth changes, IoT SDK triggers an event to prompt your app to adjust the sending bit rate in real time. The bit rate returned by the callback is the maximum recommended encoding bit rate of the video encoder.
Change audio and video streaming status
Network traffic can be reduced by suspending data transmission when audio or video feed is not required by the receiver. After a user successfully connects to a channel and starts audio and video streaming, they can suspend sending streams to a specific connection or to all connections to flexibly manage the transmission status of audio and video streams. Similarly, the user may suspend receiving a specific or all streams as required.
When a user changes the transmission status of the local audio or video stream, the IoT SDK triggers a corresponding callback to prompt the remote user to suspend or resume sending their audio or video stream.
Request key frames
In video transmission, a frame containing complete image information is known as a key frame. Subsequent frames, known as delta frames, only include modifications to the previous frame. When a video streaming client experiences network congestion, the data loss in delta frames leads to visual inconsistencies in subsequent frames. IoT SDK enables you to request a key frame from the sender to resolve such issues. A fresh key frame resets the video stream to a known state. This feature allows the client to resynchronize with the video stream and resume playback without visual artifacts or distortion.
IoT SDK provides configuration options that you use to customize the location, content and size of log files containing key data of IoT SDK operation. When you set up logging, IoT SDK writes information messages, warnings, and errors regarding activities such as initialization, configuration, connection and disconnection to log files. Log files are useful in detecting and resolving channel quality issues.
The following figure shows the workflow you need to implement to ensure channel quality in your app:
In order to follow this procedure you must have:
- Implemented the SDK quickstart project for IoT SDK.
- Android Studio 4.1 or higher.
- Android SDK API Level 24 or higher.
- A mobile device that runs Android 4.1 or higher.
A computer with Internet access.
Ensure that no firewall is blocking your network communication.
To create the environment necessary to implement channel quality best practices into your app, open the SDK quickstart for IoT SDK project you created previously.
This section shows you how to integrate channel quality optimization features of IoT SDK into your app, step-by-step.
To enable app users to mute and unmute audio and video, add checkboxes to the user interface. To do this, open
/app/res/layout/activity_main.xml and add the following lines before
Import the necessary Android classes and access the UI elements.
Import supporting libraries
MainActivity.java, add the following to the list of
Define variables to access the checkboxes
/app/java/com.example.<projectname>/MainActivity, add the following declarations to the
To implement channel quality features, take the following steps:
Configure the IoT SDK log file
To customize the location, and content of the log file, add the following code to
agoraEngine.init(appId, agoraRtcEvents, options);
Specify the audio codec, sampling rate and the number of channels
You set the audio codec, sampling rate, and the number of channels in
ChannelOptionsthat you pass to the
agoraEngine.joinChannelmethod. To set these parameters, modify the following lines in the
joinChannel(View view)method according to your requirements:
G711Ubuilt-in audio codecs in the IoT SDK.
To use your own audio codec, set
AUDIO_CODEC_DISABLED. When you call
agoraEngine.sendAudioData, set the
AudioFrameInfoto your encoding format. This setting transmits the audio encoding format as is to the receiving end. When you disable use of a built-in audio codec, IoT SDK does not process the audio. The receiving end obtains the encoded audio data and the encoding format through the
onAudioDatacallback and decodes it using a custom decoder.
Configure bandwidth estimation parameters
You configure bandWidth estimation parameters before joining a channel. For example, based on the definition levels of a webcam, the minimum value can be set to 400 kbps, and the maximum value can be set to 4200 kbps. The starting value is between the minimum and the maximum value. If the initial encoding is SD, the initial bit rate can be set to 500 kbps, based on the table.
To specify these parameters, add the following code to the
joinChannel(View view)method before
Respond to target bit rate changes
When network bandwidth changes, IoT SDK triggers the
onTargetBitrateChangedcallback to prompt the app to adjust the sending bit rate. The
targetBpsvalue returned by the callback is the maximum recommended encoding bit rate of the video encoder.
In this example, you use definition levels of a webcam to switch resolution depending on the reported target bit rate. To do this, replace the
agoraRtcEventswith the following:
Manage audio and video streaming status
When a user taps a
Checkbox, you pause or resume sending local audio and video streams. To do this, add the following code to the
onCreatemethod in the
To pause and resume playing remote audio or video streams call
agoraEngine.muteRemoteAudio(connectionId, remoteUid, isMuted)or
agoraEngine.muteRemoteVideo(connectionId, remoteUid, isMuted).
Handle muting and unmuting notifications of remote streams
When a remote user mutes or unmutes their audio or video stream, you receive notification of these changes. In this example, you inform the user of these events by displaying a message. To do this, replace the
agoraRtcEventswith the following:
Request and send key frames
When you call
agoraEngine.sendVideoDatato send a video frame, you specify if this frame is a key frame by setting the
When the sender does not send a key frame for a long time or the key frame is lost or damaged during transmission, IoT SDK triggers the
onKeyFrameGenReqcallback to advise the sender to generate a key frame. In this example, you show a message when you receive the
onKeyFrameGenReqcallback. Add the following code to the
If the receiver encounters an error when decoding frames, it calls the
agoraEngine.requestVideoKeyFramemethod to request a remote user to generate a fresh key frame.
To ensure that you have implemented channel quality features into your app:
Generate a temporary token in Agora Console .
In your browser, navigate to the Agora Muting web demo and update App ID, Channel, and Token with the values for your temporary token, then click Join.
In Android Studio, open
app/java/com.example.<projectname>/MainActivity, and update
tokenwith the values for your temporary token.
Connect a physical Android device to your development device.
In Android Studio, click Run app. A moment later you see the project installed on your device. If this is the first time you run the project, grant microphone and camera access to your app.
When the app starts, it sets the log file location and logging level according to your preference.
Press Join. Your app does the following:
- Sets the audio codec, sampling rate and the number of channels
- Sets bandwidth estimation parameters
- Starts streaming audio and video
- Listens for notification of changes in network bandwidth to adjust video resolution and bit rate accordingly
- Listens for key frame request to notify the encoder to send a key frame
Check and uncheck Mute local audio and Mute local video boxes.
Audio and video are muted/unmuted in the web demo app.
Press Mute Audio and Mute Video buttons in the web demo app. You see messages in your Android app informing you of these events.
This section contains additional information that completes the content on this page, or points you to documentation that explains other aspects to this product.
The table below shows the recommended sampling rates and corresponding data sizes for Opus, G722, and G711 audio data.
|Audio format||Sampling rate (Hz)||Audio data size (Byte)|
|Gear||Resolution||Frame rate||Bit rate range (kbps)|
|SD||L1: 640*360||15||400 - 800|
|HD||L2: 1280*720||15||1130 - 2260|
|Ultra HD||L3: 1920*1080||15||1130 - 4160|