如何在 iOS Swift 中为 WebRTC 视频通话添加画中画 (PIP)

Question

我们使用以下步骤为 WebRTC 视频通话集成 PIP（画中画）：

1. 我们正在项目中启用音频、Airplay 和画中画功能模式。

2. 我们添加了一个在多任务处理时访问相机的权利文件，请参阅访问多任务处理时使用相机。）

3. 从文档链接中，我们遵循：

   <块引用>

   配置您的应用

   您的帐户有权使用该权利后，您可以按照以下步骤使用该权利创建新的配置文件：

   1. 登录您的 Apple 开发者帐户。

   2. 转到证书、标识符和证书个人资料。

   3. 生成一个新的 为您的应用程序配置配置文件。

   4. 从您帐户的附加权利中选择多任务摄像头访问权利。

4. 我们还集成了以下链接，但是如何在这个SampleBufferVideoCallView中添加视频渲染层视图我们没有任何具体提示。 https://developer.apple.com/documentation/avkit/adopting_picture_in_picture_for_video_calls?changes=__8

5. 此外，RTCMTLVideoView 创建不支持 MTKView，但我们使用了 WebRTC 默认视频渲染视图，例如用于 GLKView 的 RTCEAGLVideoView视频渲染。

PIP 与 WebRTC iOS Swift 集成代码：

class SampleBufferVideoCallView: UIView {
    override class var layerClass: AnyClass {
        get { return AVSampleBufferDisplayLayer.self }
    }
    
    var sampleBufferDisplayLayer: AVSampleBufferDisplayLayer {
        return layer as! AVSampleBufferDisplayLayer
    }
}

func startPIP() {
    if #available(iOS 15.0, *) {
        let sampleBufferVideoCallView = SampleBufferVideoCallView()
        let pipVideoCallViewController = AVPictureInPictureVideoCallViewController()
        pipVideoCallViewController.preferredContentSize = CGSize(width: 1080, height: 1920)
        pipVideoCallViewController.view.addSubview(sampleBufferVideoCallView)
        
        let remoteVideoRenderar = RTCEAGLVideoView()
        remoteVideoRenderar.contentMode = .scaleAspectFill
        remoteVideoRenderar.frame = viewUser.frame
        viewUser.addSubview(remoteVideoRenderar)
        
        let pipContentSource = AVPictureInPictureController.ContentSource(
            activeVideoCallSourceView: self.viewUser,
            contentViewController: pipVideoCallViewController)
        
        let pipController = AVPictureInPictureController(contentSource: pipContentSource)
        pipController.canStartPictureInPictureAutomaticallyFromInline = true
        pipController.delegate = self
        
    } else {
        // Fallback on earlier versions
    }
}

如何将 viewUser GLKView 添加到 pipContentSource 以及如何将远程视频缓冲区视图集成到 SampleBufferVideoCallView 中？

是否可以通过这种方式或任何其他方式在 AVSampleBufferDisplayLayer 中视频渲染缓冲层视图？

Answer 1

代码级支持 当被问及这个问题时，Apple 给出了以下建议：

为了提出建议，我们需要详细了解您尝试渲染视频的代码。

正如您提到的文章中所讨论的，要提供画中画支持，您必须首先提供 源视图显示在视频通话视图控制器内 - 您需要将 UIView 添加到 AVPictureInPictureVideoCallViewController。系统支持显示来自 AVPlayerLayer 或 AVSampleBufferDisplayLayer 根据您的需要。不支持 MTKView/GLKView。视频通话应用需要显示远程视图，因此请使用 AVSampleBufferDisplayLayer 来执行此操作。

为了处理源视图中的绘图，您可以在将缓冲区流转换为 GLKView 之前获取对缓冲区流的访问权限，并将其提供给 AVPictureInPictureViewController 的内容。例如，您可以从视频源帧创建 CVPixelBuffers，然后从这些帧创建，创建 CMSampleBuffers 一旦您拥有 CMSampleBuffers，您可以开始将它们提供给 AVSampleBufferDisplayLayer 进行显示。查看那里定义的方法以了解这是如何完成的。有一些存档的 ObjC 示例代码 AVGreenScreenPlayer 您可能会看到它来帮助您开始使用 AVSampleBufferDisplayLayer （注意：它是Mac 代码，但 AVSampleBufferDisplayLayer API 跨平台相同）。

此外，为了实现画中画支持，您需要为 AVPictureInPictureControllerDelegate 和 AVSampleBufferDisplayLayer 提供委托方法 AVPictureInPictureSampleBufferPlaybackDelegate。请参阅最近的 WWDC 视频AVKit 中的新增功能，了解有关 AVPictureInPictureSampleBufferPlaybackDelegate 的更多信息iOS 15 中新增的代表。

Answer 2

要使用提供的代码在视频通话中通过 WebRTC 显示画中画 (PIP)，请按照以下步骤操作：

步骤 1：初始化 WebRTC 视频通话
确保您已经设置了 WebRTC 视频通话，并建立了必要的信令和对等连接。此代码假设您已经有一个代表从远程用户接收的视频流的remoteVideoTrack。

第2步：创建FrameRenderer对象
实例化 FrameRenderer 对象，该对象将负责渲染从远程用户接收的视频帧以进行 PIP 显示。

// 在初始化视频通话的位置添加此代码（渲染开始之前）

var frameRenderer: FrameRenderer?

步骤 3：将远程视频渲染到 FrameRenderer
在 renderRemoteVideo 函数中，将 RemoteVideoTrack 中的视频帧添加到 FrameRenderer 对象以在 PIP 视图中渲染它们。

func renderRemoteVideo(to renderer: RTCVideoRenderer) {
    // Make sure you have already initialized the remoteVideoTrack from the WebRTC video call.

    if frameRenderer == nil {
        frameRenderer = FrameRenderer(uID: recUserID)
    }

    self.remoteVideoTrack?.add(frameRenderer!)
}

步骤 4：从渲染远程视频中删除 FrameRenderer
在removeRenderRemoteVideo函数中，当您想要停止PIP显示时，从渲染视频帧中删除FrameRenderer对象。

func removeRenderRemoteVideo(to renderer: RTCVideoRenderer) {
    if frameRenderer != nil {
        self.remoteVideoTrack?.remove(frameRenderer!)
    }
}

第5步：定义FrameRenderer类
FrameRenderer 类负责在 PIP 视图中渲染从 WebRTC 接收的视频帧。

// Import required frameworks
import Foundation
import WebRTC
import AVKit
import VideoToolbox
import Accelerate
import libwebp

// Define closure type for handling CMSampleBuffer, orientation, scaleFactor, and userID
typealias CMSampleBufferRenderer = (CMSampleBuffer, CGImagePropertyOrientation, CGFloat, Int) -> ()

// Define closure variables for handling CMSampleBuffer from FrameRenderer
var getCMSampleBufferFromFrameRenderer: CMSampleBufferRenderer = { _,_,_,_ in }
var getCMSampleBufferFromFrameRendererForPIP: CMSampleBufferRenderer = { _,_,_,_ in }
var getLocalVideoCMSampleBufferFromFrameRenderer: 
CMSampleBufferRenderer = { _,_,_,_ in }

// Define the FrameRenderer class responsible for rendering video frames
class FrameRenderer: NSObject, RTCVideoRenderer {
// VARIABLES
var scaleFactor: CGFloat?
var recUserID: Int = 0
var frameImage = UIImage()
var videoFormatDescription: CMFormatDescription?
var didGetFrame: ((CMSampleBuffer) -> ())?
private var ciContext = CIContext()

init(uID: Int) {
    super.init()
    recUserID = uID
}

// Set the aspect ratio based on the size
func setSize(_ size: CGSize) {
    self.scaleFactor = size.height > size.width ? size.height / size.width : size.width / size.height
}

// Render a video frame received from WebRTC
func renderFrame(_ frame: RTCVideoFrame?) {
    guard let pixelBuffer = self.getCVPixelBuffer(frame: frame) else {
        return
    }

    // Extract timing information from the frame and create a CMSampleBuffer
    let timingInfo = covertFrameTimestampToTimingInfo(frame: frame)!
    let cmSampleBuffer = self.createSampleBufferFrom(pixelBuffer: pixelBuffer, timingInfo: timingInfo)!

    // Determine the video orientation and handle the CMSampleBuffer accordingly
    let oriented: CGImagePropertyOrientation?
    switch frame!.rotation.rawValue {
    case RTCVideoRotation._0.rawValue:
        oriented = .right
    case RTCVideoRotation._90.rawValue:
        oriented = .right
    case RTCVideoRotation._180.rawValue:
        oriented = .right
    case RTCVideoRotation._270.rawValue:
        oriented = .left
    default:
        oriented = .right
    }

    // Pass the CMSampleBuffer to the appropriate closure based on the user ID
    if objNewUserDM?.userId == self.recUserID {
        getLocalVideoCMSampleBufferFromFrameRenderer(cmSampleBuffer, oriented!, self.scaleFactor!, self.recUserID)
    } else {
        getCMSampleBufferFromFrameRenderer(cmSampleBuffer, oriented!, self.scaleFactor!, self.recUserID)
        getCMSampleBufferFromFrameRendererForPIP(cmSampleBuffer, oriented!, self.scaleFactor!, self.recUserID)
    }

    // Call the didGetFrame closure if it exists
    if let closure = didGetFrame {
        closure(cmSampleBuffer)
    }
}

// Function to create a CVPixelBuffer from a CIImage
func createPixelBufferFrom(image: CIImage) -> CVPixelBuffer? {
    let attrs = [
        kCVPixelBufferCGImageCompatibilityKey: false,
        kCVPixelBufferCGBitmapContextCompatibilityKey: false,
        kCVPixelBufferWidthKey: Int(image.extent.width),
        kCVPixelBufferHeightKey: Int(image.extent.height)
    ] as CFDictionary

    var pixelBuffer: CVPixelBuffer?
    let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(image.extent.width), Int(image.extent.height), kCVPixelFormatType_32BGRA, attrs, &pixelBuffer)

    if status == kCVReturnSuccess {
        self.ciContext.render(image, to: pixelBuffer!)
        return pixelBuffer
    } else {
        // Failed to create a CVPixelBuffer
        portalPrint("Error creating CVPixelBuffer.")
        return nil
    }
}

// Function to create a CVPixelBuffer from a CIImage using an existing CVPixelBuffer
func buffer(from image: CIImage, oldCVPixelBuffer: CVPixelBuffer) -> CVPixelBuffer? {
    let attrs = [
        kCVPixelBufferMetalCompatibilityKey: kCFBooleanTrue,
        kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue,
        kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue
    ] as CFDictionary

    var pixelBuffer: CVPixelBuffer?
    let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(image.extent.width), Int(image.extent.height), kCVPixelFormatType_32BGRA, attrs, &pixelBuffer)

    if status == kCVReturnSuccess {
        oldCVPixelBuffer.propagateAttachments(to: pixelBuffer!)
        return pixelBuffer
    } else {
        // Failed to create a CVPixelBuffer
        portalPrint("Error creating CVPixelBuffer.")
        return nil
    }
}

/// Convert RTCVideoFrame to CVPixelBuffer
func getCVPixelBuffer(frame: RTCVideoFrame?) -> CVPixelBuffer? {
    var buffer : RTCCVPixelBuffer?
    var pixelBuffer: CVPixelBuffer?
    
    if let inputBuffer = frame?.buffer {
        if let iBuffer = inputBuffer as? RTCI420Buffer {
            if let cvPixelBuffer = iBuffer.convertToCVPixelBuffer() {
                // Use the cvPixelBuffer as an RTCCVPixelBuffer
                // ...
                pixelBuffer = cvPixelBuffer
                return pixelBuffer
            }
            return convertToPixelBuffer(iBuffer)
        }
    }
    
    buffer = frame?.buffer as? RTCCVPixelBuffer
    pixelBuffer = buffer?.pixelBuffer
    return pixelBuffer
}
 /// Convert RTCVideoFrame to CMSampleTimingInfo
func covertFrameTimestampToTimingInfo(frame: RTCVideoFrame?) -> CMSampleTimingInfo? {
    let scale = CMTimeScale(NSEC_PER_SEC)
    let pts = CMTime(value: CMTimeValue(Double(frame!.timeStamp) * Double(scale)), timescale: scale)
    let timingInfo = CMSampleTimingInfo(duration: kCMTimeInvalid,
                                        presentationTimeStamp: pts,
                                        decodeTimeStamp: kCMTimeInvalid)
    return timingInfo
}

/// Convert CVPixelBuffer to CMSampleBuffer
func createSampleBufferFrom(pixelBuffer: CVPixelBuffer, timingInfo: CMSampleTimingInfo) -> CMSampleBuffer? {
    var sampleBuffer: CMSampleBuffer?
    
    var timimgInfo = timingInfo
    var formatDescription: CMFormatDescription? = nil
    CMVideoFormatDescriptionCreateForImageBuffer(kCFAllocatorDefault, pixelBuffer, &formatDescription)
    
    let osStatus = CMSampleBufferCreateReadyWithImageBuffer(
        kCFAllocatorDefault,
        pixelBuffer,
        formatDescription!,
        &timimgInfo,
        &sampleBuffer
    )
    
    // Print out errors
    if osStatus == kCMSampleBufferError_AllocationFailed {
        portalPrint("osStatus == kCMSampleBufferError_AllocationFailed")
    }
    if osStatus == kCMSampleBufferError_RequiredParameterMissing {
        portalPrint("osStatus == kCMSampleBufferError_RequiredParameterMissing")
    }
    if osStatus == kCMSampleBufferError_AlreadyHasDataBuffer {
        portalPrint("osStatus == kCMSampleBufferError_AlreadyHasDataBuffer")
    }
    if osStatus == kCMSampleBufferError_BufferNotReady {
        portalPrint("osStatus == kCMSampleBufferError_BufferNotReady")
    }
    if osStatus == kCMSampleBufferError_SampleIndexOutOfRange {
        portalPrint("osStatus == kCMSampleBufferError_SampleIndexOutOfRange")
    }
    if osStatus == kCMSampleBufferError_BufferHasNoSampleSizes {
        portalPrint("osStatus == kCMSampleBufferError_BufferHasNoSampleSizes")
    }
    if osStatus == kCMSampleBufferError_BufferHasNoSampleTimingInfo {
        portalPrint("osStatus == kCMSampleBufferError_BufferHasNoSampleTimingInfo")
    }
    if osStatus == kCMSampleBufferError_ArrayTooSmall {
        portalPrint("osStatus == kCMSampleBufferError_ArrayTooSmall")
    }
    if osStatus == kCMSampleBufferError_InvalidEntryCount {
        portalPrint("osStatus == kCMSampleBufferError_InvalidEntryCount")
    }
    if osStatus == kCMSampleBufferError_CannotSubdivide {
        portalPrint("osStatus == kCMSampleBufferError_CannotSubdivide")
    }
    if osStatus == kCMSampleBufferError_SampleTimingInfoInvalid {
        portalPrint("osStatus == kCMSampleBufferError_SampleTimingInfoInvalid")
    }
    if osStatus == kCMSampleBufferError_InvalidMediaTypeForOperation {
        portalPrint("osStatus == kCMSampleBufferError_InvalidMediaTypeForOperation")
    }
    if osStatus == kCMSampleBufferError_InvalidSampleData {
        portalPrint("osStatus == kCMSampleBufferError_InvalidSampleData")
    }
    if osStatus == kCMSampleBufferError_InvalidMediaFormat {
        portalPrint("osStatus == kCMSampleBufferError_InvalidMediaFormat")
    }
    if osStatus == kCMSampleBufferError_Invalidated {
        portalPrint("osStatus == kCMSampleBufferError_Invalidated")
    }
    if osStatus == kCMSampleBufferError_DataFailed {
        portalPrint("osStatus == kCMSampleBufferError_DataFailed")
    }
    if osStatus == kCMSampleBufferError_DataCanceled {
        portalPrint("osStatus == kCMSampleBufferError_DataCanceled")
    }
    
    guard let buffer = sampleBuffer else {
        portalPrint(StringConstant.samplbeBuffer)
        return nil
    }
    
    let attachments: NSArray = CMSampleBufferGetSampleAttachmentsArray(buffer, true)! as NSArray
    let dict: NSMutableDictionary = attachments[0] as! NSMutableDictionary
    dict[kCMSampleAttachmentKey_DisplayImmediately as NSString] = true as NSNumber
    
    return buffer
}

步骤 6：实现 PIP 功能
根据提供的代码，您似乎已经使用 AVPictureInPictureController 实现了 PIP 功能。当您想要在视频通话期间启用 PIP 时，请确保调用 startPIP 函数。 SampleBufferVideoCallView 用于显示从frameRenderer 接收到的PIP 视频帧。

/// start PIP Method
fileprivate func startPIP() {
    runOnMainThread() {
        if #available(iOS 15.0, *) {
            if AVPictureInPictureController.isPictureInPictureSupported() {
                let sampleBufferVideoCallView = SampleBufferVideoCallView()
                
                getCMSampleBufferFromFrameRendererForPIP = { [weak self] cmSampleBuffer, videosOrientation, scalef, userId  in
                    guard let weakself = self else {
                        return
                    }
                    if weakself.viewModel != nil {
                        if objNewUserDM?.userId != userId && weakself.viewModel.pipUserId == userId {
                            runOnMainThread {
                                sampleBufferVideoCallView.sampleBufferDisplayLayer.enqueue(cmSampleBuffer)
                            }
                        }
                    }
                }
                
                sampleBufferVideoCallView.contentMode = .scaleAspectFit
                
                self.pipVideoCallViewController = AVPictureInPictureVideoCallViewController()
                
                // Pretty much just for aspect ratio, normally used for pop-over
                self.pipVideoCallViewController.preferredContentSize = CGSize(width: 1080, height: 1920)
                
                self.pipVideoCallViewController.view.addSubview(sampleBufferVideoCallView)
                
                sampleBufferVideoCallView.translatesAutoresizingMaskIntoConstraints = false
                let constraints = [
                    sampleBufferVideoCallView.leadingAnchor.constraint(equalTo: self.pipVideoCallViewController.view.leadingAnchor),
                    sampleBufferVideoCallView.trailingAnchor.constraint(equalTo: self.pipVideoCallViewController.view.trailingAnchor),
                    sampleBufferVideoCallView.topAnchor.constraint(equalTo: self.pipVideoCallViewController.view.topAnchor),
                    sampleBufferVideoCallView.bottomAnchor.constraint(equalTo: self.pipVideoCallViewController.view.bottomAnchor)
                ]
                NSLayoutConstraint.activate(constraints)
                
                sampleBufferVideoCallView.bounds = self.pipVideoCallViewController.view.frame
                
                let pipContentSource = AVPictureInPictureController.ContentSource(
                    activeVideoCallSourceView: self.view,
                    contentViewController: self.pipVideoCallViewController
                )
                
                self.pipController = AVPictureInPictureController(contentSource: pipContentSource)
                self.pipController.canStartPictureInPictureAutomaticallyFromInline = true
                self.pipController.delegate = self
                
                print("Is pip supported: \(AVPictureInPictureController.isPictureInPictureSupported())")
                print("Is pip possible: \(self.pipController.isPictureInPicturePossible)")
            }
        } else {
            // Fallback on earlier versions
            print("PIP is not supported in this device")
        }
    }
}

注意：FrameRenderer 对象应该在您的应用程序中定义，并且您应该确保 PIP 视图的位置和大小设置适当，以实现所需的 PIP 效果。此外，请记住处理呼叫结束场景并优雅地释放frameRenderer和WebRTC连接。

请记住，提供的代码假设您已经拥有必要的 WebRTC 设置，并且此代码仅关注 PIP 渲染方面。此外，从 iOS 15.0 开始支持 PIP，因此请确保正确处理运行早期版本的设备。