Raw video processing

In certain scenarios, it may be necessary to process raw video captured through the camera and microphone to achieve desired functionality or enhance the user experience. Video SDK provides the capability to pre-process and post-process the captured video data, allowing for the implementation of custom playback effects.

Video SDK enables you to pre-process the captured video frames before sending the data to the encoder or perform post-processing on the received video frames after sending the data to the decoder.

The following figure shows the video data processing flow in the SDK video module.

• Position (2) corresponds to the onCaptureVideoFrame callback.
• Position (3) corresponds to the onPreEncodeVideoFrame callback.
• Position (4) corresponds to theonRenderVideoFrame callback.

Ensure that you have implemented the SDK quickstart in your project.

To implement raw video data functionality in your project, refer to the following steps:

1. Before joining the channel, create an IVideoFrameObserver object and register the video observer by calling the registerVideoFrameObserver method.

   
   

   _1

   int ret = engine.registerVideoFrameObserver(iVideoFrameObserver);

   
   

2. Implement the onCaptureVideoFrame and onRenderVideoFrame callbacks. After obtaining the video data, process it according to your specific scenario.

   
   

   _72

   private final IVideoFrameObserver iVideoFrameObserver = new IVideoFrameObserver() {

   _72

   @Override

   _72

   public boolean onCaptureVideoFrame(VideoFrame videoFrame) {

   _72

   Log.i(TAG, "OnEncodedVideoImageReceived" + Thread.currentThread().getName());

   _72

   if (isSnapshot) {

   _72

   isSnapshot = false;

   _72

   _72

   // Get the image bitmap

   _72

   VideoFrame.Buffer buffer = videoFrame.getBuffer();

   _72

   VideoFrame.I420Buffer i420Buffer = buffer.toI420();

   _72

   int width = i420Buffer.getWidth();

   _72

   int height = i420Buffer.getHeight();

   _72

   _72

   ByteBuffer bufferY = i420Buffer.getDataY();

   _72

   ByteBuffer bufferU = i420Buffer.getDataU();

   _72

   ByteBuffer bufferV = i420Buffer.getDataV();

   _72

   _72

   byte[] i420 = YUVUtils.toWrappedI420(bufferY, bufferU, bufferV, width, height);

   _72

   _72

   Bitmap bitmap = YUVUtils.NV21ToBitmap(getContext(),

   _72

   YUVUtils.I420ToNV21(i420, width, height),

   _72

   width,

   _72

   height);

   _72

   _72

   Matrix matrix = new Matrix();

   _72

   matrix.setRotate(270);

   _72

   // Rotate around the center

   _72

   Bitmap newBitmap = Bitmap.createBitmap(bitmap, 0, 0, width, height, matrix, false);

   _72

   // Save to file

   _72

   saveBitmap2Gallery(newBitmap);

   _72

   _72

   bitmap.recycle();

   _72

   i420Buffer.release();

   _72

   }

   _72

   return false;

   _72

   }

   _72

   _72

   @Override

   _72

   public boolean onScreenCaptureVideoFrame(VideoFrame videoFrame) {

   _72

   return false;

   _72

   }

   _72

   _72

   @Override

   _72

   public boolean onMediaPlayerVideoFrame(VideoFrame videoFrame, int i) {

   _72

   return false;

   _72

   }

   _72

   _72

   @Override

   _72

   public boolean onRenderVideoFrame(String s, int i, VideoFrame videoFrame) {

   _72

   return false;

   _72

   }

   _72

   _72

   @Override

   _72

   public int getVideoFrameProcessMode() {

   _72

   return 0;

   _72

   }

   _72

   _72

   @Override

   _72

   public int getVideoFormatPreference() {

   _72

   return 1;

   _72

   }

   _72

   _72

   @Override

   _72

   public int getRotationApplied() {

   _72

   return 0;

   _72

   }

   _72

   _72

   @Override

   _72

   public boolean getMirrorApplied() {

   _72

   return false;

   _72

   }

   _72

   };

   
   
   Caution

   When modifying parameters in a VideoFrame, ensure that the updated parameters match the actual video frame in the buffer. Mismatches may cause issues like unexpected rotation, distortion, or other visual problems in the local preview and the remote video.

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Agora provides an open source sample project ProcessRawData on GitHub. Download it or view the source code for a more detailed example.

• registerVideoFrameObserver
• registerVideoEncodedFrameObserver
• onCaptureVideoFrame
• onRenderVideoFrame
• getVideoFrameProcessMode
• onEncodedVideoFrameReceived