AVCapture追加SampleBuffer

Question

我对这个简直要疯了——到处都看了，尝试了我能想到的一切。

我正在制作一个使用 AVFoundation - 特别是 AVCapture 来使用 iPhone 相机捕获视频的 iPhone 应用程序。

我需要一个自定义图像，覆盖在录音中包含的视频源上。

到目前为止，我已经设置了 AVCapture 会话，可以显示提要、访问框架、将其保存为 UIImage 并将覆盖图像标记到其上。然后将这个新的 UIImage 转换为 CVPixelBufferRef。为了仔细检查 bufferRef 是否正常工作，我将其转换回 UIImage，并且它仍然可以很好地显示图像。

当我尝试将 CVPixelBufferRef 转换为 CMSampleBufferRef 以附加到 AVCaptureSessions assetWriterInput 时，问题就开始了。当我尝试创建 CMSampleBufferRef 时，它总是返回 NULL。

这是 -(void)captureOutput 函数

- (void)captureOutput:(AVCaptureOutput *)captureOutput 
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer 
    fromConnection:(AVCaptureConnection *)connection
{ 

 UIImage *botImage = [self imageFromSampleBuffer:sampleBuffer];
 UIImage *wheel = [self imageFromView:wheelView];

 UIImage *finalImage = [self overlaidImage:botImage :wheel];
 //[previewImage setImage:finalImage]; <- works -- the image is being merged into one UIImage

 CVPixelBufferRef pixelBuffer = NULL;
 CGImageRef cgImage = CGImageCreateCopy(finalImage.CGImage);
 CFDataRef image = CGDataProviderCopyData(CGImageGetDataProvider(cgImage));
 int status = CVPixelBufferCreateWithBytes(NULL,
             self.view.bounds.size.width,
             self.view.bounds.size.height,
             kCVPixelFormatType_32BGRA, 
             (void*)CFDataGetBytePtr(image), 
             CGImageGetBytesPerRow(cgImage), 
             NULL, 
             0,
             NULL, 
             &pixelBuffer);
 if(status == 0){
  OSStatus result = 0;

  CMVideoFormatDescriptionRef videoInfo = NULL;
  result = CMVideoFormatDescriptionCreateForImageBuffer(NULL, pixelBuffer, &videoInfo);
  NSParameterAssert(result == 0 && videoInfo != NULL);

  CMSampleBufferRef myBuffer = NULL;
  result = CMSampleBufferCreateForImageBuffer(kCFAllocatorDefault,
            pixelBuffer, true, NULL, NULL, videoInfo, NULL, &myBuffer);
  NSParameterAssert(result == 0 && myBuffer != NULL);//always null :S

  NSLog(@"Trying to append");

  if (!CMSampleBufferDataIsReady(myBuffer)){
   NSLog(@"sampleBuffer data is not ready");
   return;
  }

  if (![assetWriterInput isReadyForMoreMediaData]){
   NSLog(@"Not ready for data :(");
   return;
  }

  if (![assetWriterInput appendSampleBuffer:myBuffer]){   
   NSLog(@"Failed to append pixel buffer");
  }



 }

}

我经常听到的另一个解决方案是使用 AVAssetWriterInputPixelBufferAdaptor，它消除了进行混乱的 CMSampleBufferRef 包装的需要。但是，我搜索了 stacked 和苹果开发者论坛和文档，但找不到有关如何设置或如何使用它的清晰描述或示例。如果有人有一个可行的例子，请告诉我或帮助我解决上述问题——我已经为此不间断地工作了一个星期，但现在束手无策。

如果您需要任何其他信息，请告诉我

提前致谢，

迈克尔

Answer 1

您需要 AVAssetWriterInputPixelBufferAdaptor，这里是创建它的代码：

    // Create dictionary for pixel buffer adaptor
NSDictionary *bufferAttributes = [NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithInt:kCVPixelFormatType_32BGRA], kCVPixelBufferPixelFormatTypeKey, nil];

// Create pixel buffer adaptor
m_pixelsBufferAdaptor = [[AVAssetWriterInputPixelBufferAdaptor alloc] initWithAssetWriterInput:assetWriterInput sourcePixelBufferAttributes:bufferAttributes];

以及使用它的代码：

// If ready to have more media data
if (m_pixelsBufferAdaptor.assetWriterInput.readyForMoreMediaData) {
    // Create a pixel buffer
    CVPixelBufferRef pixelsBuffer = NULL;
    CVPixelBufferPoolCreatePixelBuffer(NULL, m_pixelsBufferAdaptor.pixelBufferPool, &pixelsBuffer);

    // Lock pixel buffer address
    CVPixelBufferLockBaseAddress(pixelsBuffer, 0);

    // Create your function to set your pixels data in the buffer (in your case, fill with your finalImage data)
    [self yourFunctionToPutDataInPixelBuffer:CVPixelBufferGetBaseAddress(pixelsBuffer)];

    // Unlock pixel buffer address
    CVPixelBufferUnlockBaseAddress(pixelsBuffer, 0);

    // Append pixel buffer (calculate currentFrameTime with your needing, the most simplest way is to have a frame time starting at 0 and increment each time you write a frame with the time of a frame (inverse of your framerate))
    [m_pixelsBufferAdaptor appendPixelBuffer:pixelsBuffer withPresentationTime:currentFrameTime];

    // Release pixel buffer
    CVPixelBufferRelease(pixelsBuffer);
}

并且不要忘记释放您的 PixelBufferAdaptor。

Answer 2

我通过使用 CMSampleBufferCreateForImageBuffer() 来做到这一点。

OSStatus ret = 0;
CMSampleBufferRef sample = NULL;
CMVideoFormatDescriptionRef videoInfo = NULL;
CMSampleTimingInfo timingInfo = kCMTimingInfoInvalid;
timingInfo.presentationTimeStamp = pts;
timingInfo.duration = duration;

ret = CMVideoFormatDescriptionCreateForImageBuffer(NULL, pixel, &videoInfo);
if (ret != 0) {
    NSLog(@"CMVideoFormatDescriptionCreateForImageBuffer failed! %d", (int)ret);
    goto done;
}
ret = CMSampleBufferCreateForImageBuffer(kCFAllocatorDefault, pixel, true, NULL, NULL,
                                         videoInfo, &timingInfo, &sample);
if (ret != 0) {
    NSLog(@"CMSampleBufferCreateForImageBuffer failed! %d", (int)ret);
    goto done;
}