Dubbing AI Voice Changer

Dubbing AI Voice Changer extension enables you to change the audio timbre.

This page shows you how to integrate the Dubbing AI Voice Changer extension in your app.

Using the setExtensionProperty method provided in Agora SDK v4.x and passing in the specified key and value parameters, you can quickly integrate real-time AI voice conversion ability into your app.

The key parameter you specify in the setExtensionProperty method corresponds to the name of the API, and the value parameter wraps some or all of the parameters of the API in JSON format. By passing in the specified key and value parameters, you can call the corresponding API to realize the functions related to real-time AI sound conversion.

Ensure that your development environment meets the following requirements:

• Android Studio 4.1 or later.
• A physical device (not an emulator) running Android 5.0 or later.

• Dubbing AI Voice Changer is used with Agora Voice SDK v4.x.

  Refer to the SDK quickstart to integrate Voice SDK v4.x and implement basic voice calling.

This section shows you how to integrate the Dubbing AI Voice Changer extension, and call the core API to implement the voice changing function.

1. Integrate the Dubbing AI Voice Changer extension

   To integrate the extension in your project:

   1. Enter the Extensions Marketplace and download the Android Dubbing AI Voice Changer extension package. After unzipping, save all .aar files to the project folder /app/libs.

   2. Get the following resource files and save them in the same directory of the project folder. For example, a new vc_model directory:

      • License file and tone files: Contact Agora to obtain these files. The suffix of the tone file is .dat, and the tone file is issued according to the license.
      • Model file: Download the required resources.

   3. Open the app/build.gradle file and add the following lines under dependencies:

      
      

      _1

      implementation fileTree(dir: "libs", include: ["*.jar", "*.aar"])

      
      

2. Enable the plugin

   After creating and initializing RtcEngine, call enableExtension to enable the plug-in, before you call other APIs such as enableVideo and joinChannel.

   
   

   _22

   // Declare parameters

   _22

   private val EXTENSION_NAME = "dubbing_vc"

   _22

   private val EXTENSION_VENDOR_NAME = "Dubbing"

   _22

   private val EXTENSION_AUDIO_FILTER = "DubbingVC"

   _22

   _22

   private val changeSpeaker_ = "changeSpeaker"

   _22

   private val startRealTimeTranscribe_ = "startRealTimeTranscribe"

   _22

   private val stopRealTimeTranscribe_ = "stopRealTimeTranscribe"

   _22

   private val getSpeakersInfo_ = "getSpeakersInfo"

   _22

   _22

   val config = RtcEngineConfig()

   _22

   config.mContext = baseContext

   _22

   config.mAppId = appId

   _22

   config.mEventHandler = mRtcEventHandler

   _22

   // Load plugin

   _22

   config.addExtension(EXTENSION_NAME)

   _22

   config.mExtensionObserver = extensionObserver

   _22

   // Create and initialize RtcEngine

   _22

   mRtcEngine = RtcEngine.create(config)

   _22

   mRtcEngine.enableAudio()

   _22

   // Enable plugin

   _22

   mRtcEngine.enableExtension(EXTENSION_VENDOR_NAME, EXTENSION_AUDIO_FILTER, enable)

   
   

3. Set the resource file path

   When you download and integrate the plug-in, the license, sound, and model files are saved to the specified directory. In this step, you specify the path where these resource files are located.

   
   

   _1

   val modelPath: String = "${context.filesDir}${File.separator}vc_model"

   
   

4. Get the tone list

   After receiving the onExtensionStarted callback from Agora SDK, call getExtensionProperty passing in the getSpeakersInfo key to get the timbre list:

   
   

   _7

   val speakerList = mRtcEngine.getExtensionProperty(

   _7

   EXTENSION_VENDOR_NAME,

   _7

   EXTENSION_AUDIO_FILTER,

   _7

   getSpeakersInfo_

   _7

   )

   _7

   // Convert JSON to array

   _7

   val arr = JSONArray(speakerList)

   
   

   The timbre list is returned as JSON data that you parse yourself.

5. Start changing

   Call setExtensionProperty and pass in the corresponding key and value:

   
   

   _6

   mRtcEngine.setExtensionProperty(

   _6

   EXTENSION_VENDOR_NAME,

   _6

   EXTENSION_AUDIO_FILTER,

   _6

   startRealTimeTranscribe_,

   _6

   "true"

   _6

   )

   
   

6. Select timbre

   Pass in the timbre ID from the obtained timbre list to set the corresponding timbre:

   
   

   _6

   mRtcEngine.setExtensionProperty(

   _6

   EXTENSION_VENDOR_NAME,

   _6

   EXTENSION_AUDIO_FILTER,

   _6

   changeSpeaker_,

   _6

   id

   _6

   )

   
   

7. Stop changing

   Call the API to stop the voice changer:

   
   

   _6

   mRtcEngine.setExtensionProperty(

   _6

   EXTENSION_VENDOR_NAME,

   _6

   EXTENSION_AUDIO_FILTER,

   _6

   stopRealTimeTranscribe_,

   _6

   "true"

   _6

   )

   
   

8. Release resources

   Close the plug-in and release the resources used by the plug-in:

   
   

   _1

   mRtcEngine.enableExtension(EXTENSION_VENDOR_NAME, EXTENSION_AUDIO_FILTER, false)

   
   

This section contains information that completes the information in this page, or points you to documentation that explains other aspects to this product.

The Dubbing AI Voice Changer extension provides a GitHub sample project for testing. To clone and run the project:

1. Clone the repository

   
   

   _1

   git clone https://github.com/Dubbing-AI-Voice-Changer/DubbingAgoraDemo.git

   
   

2. Refer to the README.md file in the repository to run the demo.

• addExtension in the RtcEngineConfig class
• enableExtension in the RtcEngine class
• setExtensionProperty in the RtcEngine class